Archive for the ‘ASM’ Category

Removing Disk from a ASM diskgroup

Posted: November 22, 2013 in ASM

Steps to remove a disk from a ASM diskgroup

So we have the following disks and the plan is to remove VOLE

SQL> select group_number, name, TOTAL_MB, FREE_MB from V$asm_disk_stat;

GROUP_NUMBER NAME TOTAL_MB FREE_MB
------------ ------------------------------ ---------- ----------
0 1019 0
1 VOLA 1019 842
1 VOLB 1019 841
1 VOLC 1019 841
2 VOLD 1019 792
1 VOLE 1019 841

Drop the disk from the Disk group

SQL> alter diskgroup DATADG drop disk 'VOLE';

Diskgroup altered.

Check the re-balance status

SQL> select sysdate, GROUP_NUMBER, OPERATION, STATE, POWER, ACTUAL, SOFAR, EST_WORK
2 EST_RATE, EST_MINUTES from v$asm_operation;

SYSDATE GROUP_NUMBER OPERA STAT POWER ACTUAL SOFAR EST_RATE
--------- ------------ ----- ---- ---------- ---------- ---------- ----------
EST_MINUTES
-----------
12-SEP-08 1 REBAL RUN 1 1 1 193
3

Now the disk is not assigned to a disk group

SQL> select group_number, name, TOTAL_MB, FREE_MB from V$asm_disk_stat;

GROUP_NUMBER NAME TOTAL_MB FREE_MB
------------ ------------------------------ ---------- ----------
0 1019 0
1 VOLA 1019 783
1 VOLB 1019 783
1 VOLC 1019 782
2 VOLD 1019 792

Check the disk status now VOLE should be listed as FORMER and is ready to be re-used

SQL> select PATH, HEADER_STATUS, MODE_STATUS, STATE from v$asm_disk;
PATH HEADER_STATU MODE_ST STATE
-----------
ORCL:VOLE FORMER ONLINE NORMAL
ORCL:VOLF PROVISIONED ONLINE NORMAL
ORCL:VOLA MEMBER ONLINE NORMAL
ORCL:VOLB MEMBER ONLINE NORMAL
ORCL:VOLC MEMBER ONLINE NORMAL
ORCL:VOLD MEMBER ONLINE NORMAL
6 rows selected.

Moving ASM diskgroups for a RAC 11gR2

Posted: October 25, 2013 in ASM, RAC

Currently I have 2 disk groups DATA and FLASH for my RAC 11GR2 cluster I need to move them to a new storage array so here are the steps taken.

1. Create New Disk Groups FLASH2 & DATA2

Use asmca to create the new disks

Create a FLASH2 diskgroup

asm1

Create a DATA2 diskgroup

asm2

New disk groups

HA Resource Target State
———– —— —–
ora.DATA.dg ONLINE ONLINE on dba01tst
ora.DATA.dg ONLINE ONLINE on dba02tst
ora.DATA2.dg ONLINE ONLINE on dba01tst
ora.DATA2.dg ONLINE ONLINE on dba02tst
ora.FLASH.dg ONLINE ONLINE on dba01tst
ora.FLASH.dg ONLINE ONLINE on dba02tst
ora.FLASH2.dg ONLINE ONLINE on dba01tst
ora.FLASH2.dg ONLINE ONLINE on dba02tst

SQL> select name,state from v$asm_diskgroup;

NAME STATE
—————————— ———–
DATA2 MOUNTED
DATA MOUNTED
FLASH2 MOUNTED
FLASH MOUNTED

2. Moving OCR to new ASM diskgroup

[root@dba01tst /]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 192353601
Device/File Name : +DATA
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

First the new ASM diskgroup must be added and this will appear as the OCRMirror. Afterwards OCR location in the old diskgroup is removed (this will make the ocrmirror to become the ocr).

[root@dba01tst /]# ocrconfig -add +DATA2
[root@dba01tst /]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 192353601
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +DATA2
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@dba01tst /]# ocrconfig -delete +DATA
[root@dba01tst /]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 192353601
Device/File Name : +DATA2
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

Related metalink note : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

3. Moving Vote disks to new ASM diskgroup

[oracle@dba01tst ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 4d9126509abf4f03bf592190f8d0ae7b (/dev/asmdisk1_udev_p1) [DATA]
Located 1 voting disk(s).
[oracle@dba01tst ~]$ crsctl replace votedisk +DATA2
Successful addition of voting disk 4f4c817a9c5f4fa6bfb0ec9adc61ec39.
Successful deletion of voting disk 4d9126509abf4f03bf592190f8d0ae7b.
Successfully replaced voting disk group with +DATA2.
CRS-4266: Voting file(s) successfully replaced
[oracle@dba01tst ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 4f4c817a9c5f4fa6bfb0ec9adc61ec39 (/dev/asmdisk5_udev_p1) [DATA2]
Located 1 voting disk(s).

Related metalink note : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

4. Moving server side ASM SPfile to new ASM diskgroup

SQL> show parameter spfile;

NAME TYPE VALUE
———————————— ———– ——————————
spfile string +DATA/dbatst-scan/asmparameter
file/registry.253.817905393

[oracle@dba01tst ~]$ asmcmd
ASMCMD> spget
+DATA/dbatst-scan/asmparameterfile/registry.253.817905393

SQL> create pfile=’/tmp/asm_pfile.ora’ from spfile;

File created.

SQL> create spfile=’+DATA2′ from pfile=’/tmp/asm_pfile.ora’;

New SPfile location will be logged on the ASM alert log
NOTE: updated gpnp profile ASM SPFILE to +DATA2/dbatst-scan/asmparameterfile/registry.253.828801675

ASMCMD> spset +DATA2/dbatst-scan/asmparameterfile/registry.253.828801675

Related metalink note : How to move ASM spfile to a different disk group [ID 1082943.1]

5. Moving database related files to new ASM diskgroup

SQL> show parameter control

NAME TYPE VALUE
———————————— ———– ——————————
control_file_record_keep_time integer 7
control_files string +DATA/dbatst/controlfile/curre
nt.260.817912055, +FLASH/dbats
t/controlfile/current.256.8179
12055

SQL> create pfile=’/home/oracle/pfile.ora’ from spfile;

File created.

SQL> alter system set control_files=’+DATA2′,’+FLASH2′ scope=spfile sid=’*’;

System altered.

[oracle@dba01tst ~]$ srvctl stop database -d DBATST
[oracle@dba01tst ~]$ srvctl start instance -d DBATST -i DBATST1 -o nomount
[oracle@dba01tst ~]$ rman target /

Recovery Manager: Release 11.2.0.3.0 – Production on Mon Oct 14 15:10:44 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: DBATST (not mounted)

RMAN> restore controlfile from ‘+DATA/dbatst/controlfile/current.260.817912055’;

Starting restore at 14-OCT-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=8 instance=DBATST1 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+DATA2/dbatst/controlfile/current.256.828803469
output file name=+FLASH2/dbatst/controlfile/current.256.828803469
Finished restore at 14-OCT-13

RMAN> exit

Recovery Manager complete.
[oracle@dba01tst ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:11:35 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter database mount;

Database altered.

SQL> alter database open;

Database altered.

SQL> show parameter control

NAME TYPE VALUE
———————————— ———– ——————————
control_file_record_keep_time integer 7
control_files string +DATA2/dbatst/controlfile/curr
ent.256.828803469, +FLASH2/dba
tst/controlfile/current.256.82
8803469
control_management_pack_access string NONE
SQL>

Related metalink note : How to duplicate a controlfile when ASM is involved [ID 345180.1]
Useful metalink note : How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy) [ID 438580.1]

6. Moving SPfile to new ASM diskgroup

SQL> create pfile=’/tmp/pfile_copy.txt’ FROM SPFILE;

File created.

Edit the pfile /tmp/pfile_copy.txt to reflect the correct control files.

SQL> create spfile=’+DATA2′ FROM PFILE=’/tmp/pfile_copy.txt’;

File created.

Check the file in asmcmd and create an alias

ASMCMD> ls
DATA/
DATA2/
FLASH/
FLASH2/
ASMCMD> cd DATA2/
ASMCMD> cd DBATST/
ASMCMD> ls
CONTROLFILE/
PARAMETERFILE/
ASMCMD> cd PARAMETERFILE/
ASMCMD> ls
spfile.257.828803567
ASMCMD> pwd
+DATA2/DBATST/PARAMETERFILE
ASMCMD> cd ..
ASMCMD> mkalias +DATA2/DBATST/PARAMETERFILE/spfile.257.828803567 spfiledbatst.ora
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y CONTROLFILE/
Y PARAMETERFILE/
N spfiledbatst.ora => +DATA2/DBATST/PARAMETERFILE/spfile.257.828803567

[oracle@dba01tst ~]$ srvctl config database -d DBATST
Database unique name: DBATST
Database name: DBATST
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/DBATST/spfileDBATST.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DBATST
Database instances: DBATST1,DBATST2
Disk Groups: DATA,FLASH,DATA2,FLASH2
Mount point paths:
Services:
Type: RAC
Database is administrator managed

[oracle@dba01tst ~]$ srvctl config database -d DBATST
Database unique name: DBATST
Database name: DBATST
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/DBATST/spfileDBATST.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DBATST
Database instances: DBATST1,DBATST2
Disk Groups: DATA,FLASH,DATA2,FLASH2
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@dba01tst ~]$ srvctl modify database -d DBATST -p +DATA2/DBATST/spfiledbatst.ora
[oracle@dba01tst ~]$ srvctl config database -d DBATST
Database unique name: DBATST
Database name: DBATST
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA2/DBATST/spfiledbatst.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DBATST
Database instances: DBATST1,DBATST2
Disk Groups: DATA,FLASH,DATA2,FLASH2
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@dba01tst ~]$

Modify the pfiles that give reference to the SPFile in $ORACLE_HOME/dbs

[oracle@dba01tst dbs]$ cat initDBATST1.ora
SPFILE=’+DATA/DBATST/spfileDBATST.ora’

[oracle@dba02tst dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:21:25 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.
SQL>startup
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size 2234960 bytes
Variable Size 788530608 bytes
Database Buffers 247463936 bytes
Redo Buffers 5656576 bytes
Database mounted.
Database opened.

7. Moving data files to new ASM diskgroup

SQL> select ‘backup as copy datafile ‘||file#||’ format ”+DATA2”;’ from v$datafile;

‘BACKUPASCOPYDATAFILE’||FILE#||’FORMAT”+DATA2”;’
——————————————————————————–
backup as copy datafile 1 format ‘+DATA2’;
backup as copy datafile 2 format ‘+DATA2’;
backup as copy datafile 3 format ‘+DATA2’;
backup as copy datafile 4 format ‘+DATA2’;
backup as copy datafile 5 format ‘+DATA2’;

RMAN> run {
2> backup as copy datafile 1 format ‘+DATA2’;
3> backup as copy datafile 2 format ‘+DATA2’;
4> backup as copy datafile 3 format ‘+DATA2’;
5> backup as copy datafile 4 format ‘+DATA2’;
6> backup as copy datafile 5 format ‘+DATA2’;
7> }

Starting backup at 14-OCT-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=82 instance=DBATST1 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/dbatst/datafile/system.256.817911967
output file name=+DATA2/dbatst/datafile/system.258.828805189 tag=TAG20131014T153948 RECID=1 STAMP=828805204
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=+DATA2/dbatst/controlfile/backup.259.828805215 tag=TAG20131014T153948 RECID=2 STAMP=828805215
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 14-OCT-13
channel ORA_DISK_1: finished piece 1 at 14-OCT-13
piece handle=+DATA2/dbatst/backupset/2013_10_14/nnsnf0_tag20131014t153948_0.260.828805217 tag=TAG20131014T153948 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/dbatst/datafile/sysaux.257.817911967
output file name=+DATA2/dbatst/datafile/sysaux.261.828805219 tag=TAG20131014T154018 RECID=3 STAMP=828805244
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/dbatst/datafile/undotbs1.258.817911967
output file name=+DATA2/dbatst/datafile/undotbs1.262.828805255 tag=TAG20131014T154054 RECID=4 STAMP=828805263
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/dbatst/datafile/users.259.817911967
output file name=+DATA2/dbatst/datafile/users.263.828805271 tag=TAG20131014T154109 RECID=5 STAMP=828805270
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/dbatst/datafile/undotbs2.264.817912175
output file name=+DATA2/dbatst/datafile/undotbs2.264.828805271 tag=TAG20131014T154111 RECID=6 STAMP=828805276
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 14-OCT-13

RMAN>

[oracle@dba01tst dbs]$ srvctl stop database -d DBATST
[oracle@dba01tst dbs]$ srvctl start instance -d DBATST -i DBATST1 -o mount
[oracle@dba01tst dbs]$ sqlplus / as sysdba
SQL> select ‘switch datafile ‘||file#||’ to copy;’ from v$datafile;

‘SWITCHDATAFILE’||FILE#||’TOCOPY;’
—————————————————————–
switch datafile 1 to copy;
switch datafile 2 to copy;
switch datafile 3 to copy;
switch datafile 4 to copy;
switch datafile 5 to copy;

SQL> select ‘recover datafile ‘||file#||’;’ from v$datafile;

‘RECOVERDATAFILE’||FILE#||’;’
———————————————————-
recover datafile 1;
recover datafile 2;
recover datafile 3;
recover datafile 4;
recover datafile 5;
[oracle@dba01tst dbs]$ rman target /

Recovery Manager: Release 11.2.0.3.0 – Production on Mon Oct 14 15:45:00 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: DBATST (DBID=1790472310, not open)

RMAN> switch datafile 1 to copy;

using target database control file instead of recovery catalog
datafile 1 switched to datafile copy “+DATA2/dbatst/datafile/system.258.828805189”

RMAN> switch datafile 2 to copy;

datafile 2 switched to datafile copy “+DATA2/dbatst/datafile/sysaux.261.828805219”

RMAN> switch datafile 3 to copy;

datafile 3 switched to datafile copy “+DATA2/dbatst/datafile/undotbs1.262.828805255”

RMAN> switch datafile 4 to copy;

datafile 4 switched to datafile copy “+DATA2/dbatst/datafile/users.263.828805271”

RMAN> switch datafile 5 to copy;

datafile 5 switched to datafile copy “+DATA2/dbatst/datafile/undotbs2.264.828805271”

RMAN> recover datafile 1;

Starting recover at 14-OCT-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=196 instance=DBATST1 device type=DISK

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 14-OCT-13

RMAN> recover datafile 2;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 14-OCT-13

RMAN> recover datafile 3;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 14-OCT-13

RMAN> recover datafile 4;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 14-OCT-13

RMAN> recover datafile 5;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 14-OCT-13

[oracle@dba01tst dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:46:49 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter database open;

Database altered.

TAKE A FULL BACKUP

RMAN> delete datafilecopy all;

The backup cleared mine down on test

Useful metalink note : How to move a datafile from ASM to the file system [ID 390416.1]

8. Moving temp files to new ASM diskgroup

[oracle@dba01tst dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:50:54 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select file_name from dba_temp_files;

FILE_NAME
——————————————————————————–
+DATA/dbatst/tempfile/temp.263.817912069

SQL> alter tablespace temp add tempfile ‘+data2(tempfile)’;

Tablespace altered.

SQL> select file_name from dba_temp_files;

FILE_NAME
——————————————————————————–
+DATA/dbatst/tempfile/temp.263.817912069
+DATA2/dbatst/tempfile/temp.260.828805895

SQL> alter tablespace temp drop tempfile ‘+DATA/dbatst/tempfile/temp.263.817912069’;

Tablespace altered.

9. Moving online redo log files to new ASM diskgroup

SQL> col member format a60
SQL> select group#,member from v$logfile;

GROUP# MEMBER
———- ————————————————————
2 +DATA/dbatst/onlinelog/group_2.262.817912061
2 +FLASH/dbatst/onlinelog/group_2.258.817912063
1 +DATA/dbatst/onlinelog/group_1.261.817912059
1 +FLASH/dbatst/onlinelog/group_1.257.817912061
3 +DATA/dbatst/onlinelog/group_3.265.817912283
3 +FLASH/dbatst/onlinelog/group_3.259.817912285
4 +DATA/dbatst/onlinelog/group_4.266.817912285
4 +FLASH/dbatst/onlinelog/group_4.260.817912285

8 rows selected.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 1;

Database altered.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 2;

Database altered.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 3;

Database altered.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 4;

Database altered.

Drop the extra log file members

alter database drop logfile member ‘+FLASH/dbatst/onlinelog/group_3.259.817912285′;

SQL> select group#,member from v$logfile order by 1;

GROUP# MEMBER
———- ————————————————————
1 +DATA2/dbatst/onlinelog/group_1.259.828806063
1 +FLASH2/dbatst/onlinelog/group_1.257.828806065
2 +DATA2/dbatst/onlinelog/group_2.265.828806069
2 +FLASH2/dbatst/onlinelog/group_2.258.828806069
3 +DATA2/dbatst/onlinelog/group_3.266.828806077
3 +FLASH2/dbatst/onlinelog/group_3.259.828806077
4 +DATA2/dbatst/onlinelog/group_4.267.828806081
4 +FLASH2/dbatst/onlinelog/group_4.260.828806083

8 rows selected.

SQL> show parameter recover

NAME TYPE VALUE
———————————— ———– ——————————
db_recovery_file_dest string +FLASH
db_recovery_file_dest_size big integer 800M
db_unrecoverable_scn_tracking boolean TRUE
recovery_parallelism integer 0
SQL> alter system set db_recovery_file_dest=’+FLASH2′;

System altered.

SQL> show parameter recover

NAME TYPE VALUE
———————————— ———– ——————————
db_recovery_file_dest string +FLASH2
db_recovery_file_dest_size big integer 800M
db_unrecoverable_scn_tracking boolean TRUE
recovery_parallelism integer 0

SQL> show parameter db_create_file_dest

NAME TYPE VALUE
———————————— ———– ——————————
db_create_file_dest string +DATA
SQL> alter system set db_create_file_dest=’+DATA2’;

System altered.

SQL> show parameter db_create_file_dest

NAME TYPE VALUE
———————————— ———– ——————————
db_create_file_dest string +DATA2
SQL>

SQL> show parameter db_create_online_log_dest

NAME TYPE VALUE
———————————— ———– ——————————
db_create_online_log_dest_1 string
db_create_online_log_dest_2 string
db_create_online_log_dest_3 string
db_create_online_log_dest_4 string
db_create_online_log_dest_5 string

SQL> select max(sequence#),thread# from v$archived_log group by thread#;

MAX(SEQUENCE#) THREAD#
————– ———-
444 1
420 2

SQL> select name from v$archived_log where (sequence#=200 and thread#=1) or (sequence#=151 and thread#=2);

no rows selected

SQL> select name from v$archived_log where (sequence#=444 and thread#=1) or (sequence#=420 and thread#=2);

NAME
——————————————————————————–
+FLASH2/dbatst/archivelog/2013_10_14/thread_2_seq_420.262.828807191
+FLASH2/dbatst/archivelog/2013_10_14/thread_1_seq_444.263.828807193

10 . Dropping old disk groups

[root@dba01tst ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dba01tst’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.DATA2.dg’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.FLASH2.dg’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.cvu’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.dba01tst.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.dba01tst.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.FLASH2.dg’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dba02tst’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.DATA2.dg’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.FLASH2.dg’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.dba02tst.vip’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.dba02tst.vip’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.FLASH2.dg’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.DATA2.dg’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.asm’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.DATA2.dg’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.asm’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.ons’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.net1.network’ on ‘dba02tst’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dba02tst’ has completed
CRS-2673: Attempting to stop ‘ora.ons’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.ons’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.net1.network’ on ‘dba01tst’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dba01tst’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.crsd’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.evmd’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.asm’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.cssd’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.cssd’ on ‘dba01tst’ succeeded
[root@dba01tst ~]# crsctl start cluster -all
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dba02tst’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘dba01tst’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.cssd’ on ‘dba02tst’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dba02tst’
CRS-2676: Start of ‘ora.diskmon’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.diskmon’ on ‘dba02tst’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘dba01tst’
CRS-2676: Start of ‘ora.cssd’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘dba02tst’
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘dba02tst’
CRS-2676: Start of ‘ora.ctssd’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.ctssd’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.evmd’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.evmd’ on ‘dba02tst’
CRS-2676: Start of ‘ora.evmd’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘dba02tst’ succeeded
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘dba01tst’
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘dba02tst’
CRS-2676: Start of ‘ora.asm’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘dba01tst’
CRS-2676: Start of ‘ora.crsd’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.asm’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘dba02tst’
CRS-2676: Start of ‘ora.crsd’ on ‘dba02tst’ succeeded

Dismount the disks on one node

[oracle@dba02tst trace]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 16:40:20 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup data dismount;

Diskgroup altered.

SQL> alter diskgroup flash dismount;

Diskgroup altered.

Drop the disks from the other node

SQL> drop diskgroup flash including contents;

Diskgroup dropped.

SQL> drop diskgroup data including contents;

Diskgroup dropped.

SQL>

Remove the disks from OCR

[oracle@dba01tst ~]$ srvctl remove diskgroup -g DATA
PRCA-1002 : Failed to remove CRS resource ora.DATA.dg for ASM Disk Group DATA
PRCR-1028 : Failed to remove resource ora.DATA.dg
PRCR-1072 : Failed to unregister resource ora.DATA.dg
CRS-0222: Resource ‘ora.DATA.dg’ has dependency error.

[oracle@dba01tst ~]$ srvctl modify database -d DBATST -a “DATA2,FLASH2”
[oracle@dba01tst ~]$ srvctl remove diskgroup -g DATA
[oracle@dba01tst ~]$ srvctl remove diskgroup -g FLASH
[oracle@dba01tst ~]$ srvctl start database -d DBATST

Oracle 11.2.0.3 RAC DB backup failing with the following error:

Error

Mon Oct 07 01:03:28 2013
Errors in file /u01/app/oracle/diag/rdbms/testdb/TESTDB1/trace/TESTDB1_ora_25286.trc:
ORA-00245: control file backup failed; target is likely on a local file system

Cause

This is down to the snapshot controlfile being on a local disk and it needs to be on a shared disk for all of the cluster in our case we need to put it on our +FLASH ASM diskgroup

FIX

So to fix the issue we need to configure the SNAPSHOT CONTROLFILE to the +FLASH ASM diskgroup in RMAN, this is done as follows:

RMAN> show all;

using target database control file instead of recovery catalog
RMAN configuration parameters for database with db_unique_name TESTDB are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 3;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '/u01/app/oracle/product/11.2.0.3/dbs/snapcf_TESTDB1.f'; # default

RMAN> configure snapshot controlfile name to '+FLASH/snapcf_TESTDB1.f';

new RMAN configuration parameters:
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+FLASH/snapcf_TESTDB1.f';
new RMAN configuration parameters are successfully stored

RMAN>
RMAN> show all ;

RMAN configuration parameters for database with db_unique_name TESTDB are:
CONFIGURE RETENTION POLICY TO REDUNDANCY 3;
CONFIGURE BACKUP OPTIMIZATION ON;
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '%F'; # default
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO BACKUPSET; # default
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+FLASH/snapcf_TESTDB1.f';

Run a backup and check the file is created in the +FLASH diskgroup using asmcmd

oracle@testdb01tst /export/home/oracle $ asmcmd
ASMCMD> ls
DATA/
FLASH/
ASMCMD> cd FLASH/
ASMCMD> ls -l
Type         Redund  Striped  Time             Sys  Name
Y    TESTDB/
Y    TESTDBTST/
N    snapcf_testdb1.f => +FLASH/TESTDB/CONTROLFILE/Backup.302.828871329

Got the following error on a 11.2.0.3 RAC database, after investigation it turns out to be a Oracle Configuration Manager (OCM) error as we don’t use this product its safe to  just disable the jobs

Error

Sun Oct 13 06:00:01 2013
Errors in file /u01/app/oracle/diag/rdbms/test/TEST1/trace/TEST1_j004_20464.trc:
ORA-12012: error on auto execute of job “ORACLE_OCM”.”MGMT_CONFIG_JOB_2_1″
ORA-29280: invalid directory path
ORA-06512: at “ORACLE_OCM.MGMT_DB_LL_METRICS”, line 2436
ORA-06512: at line 1
Sun Oct 13 06:00:23 2013

Fix

Login to any instance in the cluster as sysdba and run

SQL> exec dbms_scheduler.disable(‘ORACLE_OCM.MGMT_CONFIG_JOB’);

PL/SQL procedure successfully completed.

SQL> exec dbms_scheduler.disable(‘ORACLE_OCM.MGMT_STATS_CONFIG_JOB’)

PL/SQL procedure successfully completed.

Oracle Reference documents – 1453959.1

The following is the steps needed to clear down a failed 11.2.0.3 RAC install, the root.sh part of the install can cause numerious issue so these steps have been used on several occasions

Run the rootcrs.pl to deconfig the cluster

Run on the first node

root@testnode1)$ ./rootcrs.pl -deconfig -verbose -force
Using configuration parameter file: ./crsconfig_params
Network exists: 1/172.23.72.0/255.255.248.0/bond0, type static
VIP exists: /testnode1-vip/172.23.72.23/172.23.72.0/255.255.248.0/bond0, hosting node testnode1
GSD exists
ONS exists: Local port 6100, remote port 6200, EM port 2016
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘testnode1’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.oc4j’ on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘testnode1’
CRS-2677: Stop of ‘ora.oc4j’ on ‘testnode1’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘testnode1’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘testnode1’
CRS-2677: Stop of ‘ora.asm’ on ‘testnode1’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘testnode1’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘testnode1’ succeeded
CRS-2673: Attempting to stop ‘ora.mdnsd’ on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.crf’ on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘testnode1’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘testnode1’
CRS-2677: Stop of ‘ora.mdnsd’ on ‘testnode1’ succeeded
CRS-2677: Stop of ‘ora.crf’ on ‘testnode1’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘testnode1’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘testnode1’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘testnode1’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘testnode1’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘testnode1’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘testnode1’
CRS-2677: Stop of ‘ora.cssd’ on ‘testnode1’ succeeded
CRS-2673: Attempting to stop ‘ora.gipcd’ on ‘testnode1’
CRS-2677: Stop of ‘ora.gipcd’ on ‘testnode1’ succeeded
CRS-2673: Attempting to stop ‘ora.gpnpd’ on ‘testnode1’
CRS-2677: Stop of ‘ora.gpnpd’ on ‘testnode1’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘testnode1’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

Run on the last node

root@testnode2)$ ./rootcrs.pl -deconfig -verbose -force -lastnode
Using configuration parameter file: ./crsconfig_params
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘testnode2’
CRS-5017: The resource action “ora.mdnsd start” encountered the following error:
Start action for daemon aborted. For details refer to “(:CLSN00107:)” in “/u01/app/11.2.0.3/grid/log/testnode2/agent/ohasd/oraagent_oracle/oraagent_oracle.log”.
CRS-2674: Start of ‘ora.mdnsd’ on ‘testnode2’ failed
CRS-2679: Attempting to clean ‘ora.mdnsd’ on ‘testnode2’
CRS-2681: Clean of ‘ora.mdnsd’ on ‘testnode2’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘testnode2’
CRS-5017: The resource action “ora.gpnpd start” encountered the following error:
Start action for daemon aborted. For details refer to “(:CLSN00107:)” in “/u01/app/11.2.0.3/grid/log/testnode2/agent/ohasd/oraagent_oracle/oraagent_oracle.log”.
CRS-2674: Start of ‘ora.gpnpd’ on ‘testnode2’ failed
CRS-2679: Attempting to clean ‘ora.gpnpd’ on ‘testnode2’
CRS-2681: Clean of ‘ora.gpnpd’ on ‘testnode2’ succeeded
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1119 : Failed to look up CRS resources of ora.cluster_vip_net1.type type
PRCR-1068 : Failed to query resources
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.gsd is registered
Cannot communicate with crsd
PRCR-1070 : Failed to check if resource ora.ons is registered
Cannot communicate with crsd

CRS-4689: Oracle Clusterware is already stopped
CRS-2672: Attempting to start ‘ora.mdnsd’ on ‘testnode2’
CRS-5017: The resource action “ora.mdnsd start” encountered the following error:
Start action for daemon aborted. For details refer to “(:CLSN00107:)” in “/u01/app/11.2.0.3/grid/log/testnode2/agent/ohasd/oraagent_oracle/oraagent_oracle.log”.
CRS-2674: Start of ‘ora.mdnsd’ on ‘testnode2’ failed
CRS-2679: Attempting to clean ‘ora.mdnsd’ on ‘testnode2’
CRS-2681: Clean of ‘ora.mdnsd’ on ‘testnode2’ succeeded
CRS-2672: Attempting to start ‘ora.gpnpd’ on ‘testnode2’
CRS-5017: The resource action “ora.gpnpd start” encountered the following error:
Start action for daemon aborted. For details refer to “(:CLSN00107:)” in “/u01/app/11.2.0.3/grid/log/testnode2/agent/ohasd/oraagent_oracle/oraagent_oracle.log”.
CRS-2674: Start of ‘ora.gpnpd’ on ‘testnode2’ failed
CRS-2679: Attempting to clean ‘ora.gpnpd’ on ‘testnode2’
CRS-2681: Clean of ‘ora.gpnpd’ on ‘testnode2’ succeeded
CRS-4000: Command Start failed, or completed with errors.
CSS startup failed with return code 1
Unable to communicate with the Cluster Synchronization Services daemon.
CRS-4000: Command Delete failed, or completed with errors.
crsctl delete for vds in DATA … failed
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on ‘testnode2’
CRS-2673: Attempting to stop ‘ora.cssdmonitor’ on ‘testnode2’
CRS-2677: Stop of ‘ora.cssdmonitor’ on ‘testnode2’ succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on ‘testnode2’ has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully deconfigured Oracle clusterware stack on this node

Clear down the install files

Node 1

root@testnode1)$ pwd
/u01/app
root@testnode1)$ rm -rf 11.2.0.3 oraInventory oracle
root@testnode1)$ rm -R /etc/oracle
root@testnode1)$ rm /etc/oraInst.loc
root@testnode1)$ rm /etc/oratab
root@testnode1)$ rm /usr/local/bin/dbhome
root@testnode1)$ rm /usr/local/bin/oraenv
root@testnode1)$ rm /usr/local/bin/coraenv
root@testnode1)$ chown oracle:dba /u01/app

Node 2

root@testnode2)$ cd /u01/app
root@testnode2)$ rm -rf 11.2.0.3 oraInventory oracle
root@testnode2)$ rm -R /etc/oracle
root@testnode2)$ rm /etc/oraInst.loc
root@testnode2)$ rm /etc/oratab
root@testnode2)$ rm /usr/local/bin/dbhome
root@testnode2)$ rm /usr/local/bin/oraenv
root@testnode2)$ rm /usr/local/bin/coraenv
root@testnode2)$ chown oracle:dba /u01/app

Clear the header from the disk that was used in the ASM disk group

oracle@testnode2.vcint.com /export/home/oracle $ dd if=/dev/zero of=/dev/asmdisk2_udev bs=1024 count=10000
10000+0 records in
10000+0 records out
10240000 bytes (10 MB) copied, 2.2165 s, 4.6 MB/s

Useful ASM Queries

Posted: March 5, 2013 in ASM

Disk Group Information

set pages 1000 lines 120
col NAME for a15
select GROUP_NUMBER DG#, name, ALLOCATION_UNIT_SIZE AU_SZ, STATE,
TYPE, TOTAL_MB, FREE_MB, OFFLINE_DISKS from v$asm_diskgroup;

DG# NAME                 AU_SZ STATE       TYPE     TOTAL_MB    FREE_MB OFFLINE_DISKS
———- ————— ———- ———– —— ———- ———- ————-
1 DATA               1048576 MOUNTED     EXTERN       5114       3353             0
2 FRA                1048576 MOUNTED     EXTERN       5114       2613             0

Disk Information

set pages 1000 lines 120
col PATH for a30
select DISK_NUMBER,MOUNT_STATUS,HEADER_STATUS,MODE_STATUS,STATE,
PATH FROM V$ASM_DISK;

DISK_NUMBER MOUNT_S HEADER_STATU MODE_ST STATE    PATH
———– ——- ———— ——- ——– ——————————
0 CACHED  MEMBER       ONLINE  NORMAL   /dev/asmdisk1
0 CACHED  MEMBER       ONLINE  NORMAL   /dev/asmdisk2

Rebalancing Information

select GROUP_NUMBER, OPERATION, STATE, ACTUAL, SOFAR, EST_MINUTES from v$asm_operation;

GROUP_NUMBER OPERA STAT    ACTUAL  SOFAR      EST_MINUTES
———— —– —- ———- ———- ———–
2            REBAL RUN           1         49          16

Add New ASM Diskgroup

Posted: April 19, 2012 in ASM

The system i’ll be adding the disk on is a linux system using asmlib.

Current Set-up

In the current set-up we have 3 disks with 1 disk in the FLASH disk group and the 2 remaning disks in the DATA diskgroup

SQL> select PATH, HEADER_STATUS, MODE_STATUS, STATE from v$asm_disk;

PATH HEADER_STATU MODE_ST STATE
————— ———— ——- ——–
ORCL:VOLA MEMBER ONLINE NORMAL
ORCL:VOLB MEMBER ONLINE NORMAL
ORCL:VOLC MEMBER ONLINE NORMAL
SQL> select name, type, total_mb, free_mb from v$asm_diskgroup;

NAME TYPE TOTAL_MB FREE_MB
—————————— —— ———- ———-
DATA EXTERN 4094 3273
FLASH EXTERN 2047 1820

Set-up the 2 new disks

We have 2 new disks added /dev/sde & /dev/sdf so we need to set-up a parttaion on these disks and configure them in asmlib

# fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-261, default 261):
Using default value 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.
# fdisk /dev/sdf
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won’t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-261, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-261, default 261):
Using default value 261

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

#
# /etc/init.d/oracleasm createdisk VOLD /dev/sde1
Marking disk “/dev/sde1” as an ASM disk: [ OK ]
# /etc/init.d/oracleasm createdisk VOLE /dev/sdf1
Marking disk “/dev/sdf1” as an ASM disk: [ OK ]
# /etc/init.d/oracleasm listdisks
VOLA
VOLB
VOLC
VOLD
VOLE
#

Create the new diskgroup

Now we can create the new diskgroup as we have the 2 new disks VOLD & VOLE

SQL> select PATH, NAME, HEADER_STATUS, MODE_STATUS, STATE from v$asm_disk;

PATH NAME HEADER_STATU MODE_ST STATE
————— —————————— ———— ——- ——–
ORCL:VOLD PROVISIONED ONLINE NORMAL
ORCL:VOLE PROVISIONED ONLINE NORMAL
ORCL:VOLA VOLA MEMBER ONLINE NORMAL
ORCL:VOLB VOLB MEMBER ONLINE NORMAL
ORCL:VOLC VOLC MEMBER ONLINE NORMAL
SQL> create diskgroup DATA2
2 external redundancy
3 disk ‘ORCL:VOLD’, ‘ORCL:VOLE’;

Diskgroup created.

SQL> select PATH, NAME, HEADER_STATUS, MODE_STATUS, STATE from v$asm_disk;

PATH NAME HEADER_STATU MODE_ST STATE
————— —————————— ———— ——- ——–
ORCL:VOLA VOLA MEMBER ONLINE NORMAL
ORCL:VOLB VOLB MEMBER ONLINE NORMAL
ORCL:VOLC VOLC MEMBER ONLINE NORMAL
ORCL:VOLD VOLD MEMBER ONLINE NORMAL
ORCL:VOLE VOLE MEMBER ONLINE NORMAL

SQL> select name, type, total_mb, free_mb from v$asm_diskgroup;

NAME TYPE TOTAL_MB FREE_MB
—————————— —— ———- ———-
DATA EXTERN 4094 3273
FLASH EXTERN 2047 1820
DATA2 EXTERN 4094 4042

DATA2 diskgroup created.