Moving ASM diskgroups for a RAC 11gR2

Posted: October 25, 2013 in ASM, RAC

Currently I have 2 disk groups DATA and FLASH for my RAC 11GR2 cluster I need to move them to a new storage array so here are the steps taken.

1. Create New Disk Groups FLASH2 & DATA2

Use asmca to create the new disks

Create a FLASH2 diskgroup

asm1

Create a DATA2 diskgroup

asm2

New disk groups

HA Resource Target State
———– —— —–
ora.DATA.dg ONLINE ONLINE on dba01tst
ora.DATA.dg ONLINE ONLINE on dba02tst
ora.DATA2.dg ONLINE ONLINE on dba01tst
ora.DATA2.dg ONLINE ONLINE on dba02tst
ora.FLASH.dg ONLINE ONLINE on dba01tst
ora.FLASH.dg ONLINE ONLINE on dba02tst
ora.FLASH2.dg ONLINE ONLINE on dba01tst
ora.FLASH2.dg ONLINE ONLINE on dba02tst

SQL> select name,state from v$asm_diskgroup;

NAME STATE
—————————— ———–
DATA2 MOUNTED
DATA MOUNTED
FLASH2 MOUNTED
FLASH MOUNTED

2. Moving OCR to new ASM diskgroup

[root@dba01tst /]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 192353601
Device/File Name : +DATA
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

First the new ASM diskgroup must be added and this will appear as the OCRMirror. Afterwards OCR location in the old diskgroup is removed (this will make the ocrmirror to become the ocr).

[root@dba01tst /]# ocrconfig -add +DATA2
[root@dba01tst /]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 192353601
Device/File Name : +DATA
Device/File integrity check succeeded
Device/File Name : +DATA2
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

[root@dba01tst /]# ocrconfig -delete +DATA
[root@dba01tst /]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 2964
Available space (kbytes) : 259156
ID : 192353601
Device/File Name : +DATA2
Device/File integrity check succeeded

Device/File not configured

Device/File not configured

Device/File not configured

Device/File not configured

Cluster registry integrity check succeeded

Logical corruption check succeeded

Related metalink note : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

3. Moving Vote disks to new ASM diskgroup

[oracle@dba01tst ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 4d9126509abf4f03bf592190f8d0ae7b (/dev/asmdisk1_udev_p1) [DATA]
Located 1 voting disk(s).
[oracle@dba01tst ~]$ crsctl replace votedisk +DATA2
Successful addition of voting disk 4f4c817a9c5f4fa6bfb0ec9adc61ec39.
Successful deletion of voting disk 4d9126509abf4f03bf592190f8d0ae7b.
Successfully replaced voting disk group with +DATA2.
CRS-4266: Voting file(s) successfully replaced
[oracle@dba01tst ~]$ crsctl query css votedisk
## STATE File Universal Id File Name Disk group
— —– —————– ——— ———
1. ONLINE 4f4c817a9c5f4fa6bfb0ec9adc61ec39 (/dev/asmdisk5_udev_p1) [DATA2]
Located 1 voting disk(s).

Related metalink note : OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

4. Moving server side ASM SPfile to new ASM diskgroup

SQL> show parameter spfile;

NAME TYPE VALUE
———————————— ———– ——————————
spfile string +DATA/dbatst-scan/asmparameter
file/registry.253.817905393

[oracle@dba01tst ~]$ asmcmd
ASMCMD> spget
+DATA/dbatst-scan/asmparameterfile/registry.253.817905393

SQL> create pfile=’/tmp/asm_pfile.ora’ from spfile;

File created.

SQL> create spfile=’+DATA2′ from pfile=’/tmp/asm_pfile.ora’;

New SPfile location will be logged on the ASM alert log
NOTE: updated gpnp profile ASM SPFILE to +DATA2/dbatst-scan/asmparameterfile/registry.253.828801675

ASMCMD> spset +DATA2/dbatst-scan/asmparameterfile/registry.253.828801675

Related metalink note : How to move ASM spfile to a different disk group [ID 1082943.1]

5. Moving database related files to new ASM diskgroup

SQL> show parameter control

NAME TYPE VALUE
———————————— ———– ——————————
control_file_record_keep_time integer 7
control_files string +DATA/dbatst/controlfile/curre
nt.260.817912055, +FLASH/dbats
t/controlfile/current.256.8179
12055

SQL> create pfile=’/home/oracle/pfile.ora’ from spfile;

File created.

SQL> alter system set control_files=’+DATA2′,’+FLASH2′ scope=spfile sid=’*’;

System altered.

[oracle@dba01tst ~]$ srvctl stop database -d DBATST
[oracle@dba01tst ~]$ srvctl start instance -d DBATST -i DBATST1 -o nomount
[oracle@dba01tst ~]$ rman target /

Recovery Manager: Release 11.2.0.3.0 – Production on Mon Oct 14 15:10:44 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: DBATST (not mounted)

RMAN> restore controlfile from ‘+DATA/dbatst/controlfile/current.260.817912055’;

Starting restore at 14-OCT-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=8 instance=DBATST1 device type=DISK

channel ORA_DISK_1: copied control file copy
output file name=+DATA2/dbatst/controlfile/current.256.828803469
output file name=+FLASH2/dbatst/controlfile/current.256.828803469
Finished restore at 14-OCT-13

RMAN> exit

Recovery Manager complete.
[oracle@dba01tst ~]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:11:35 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter database mount;

Database altered.

SQL> alter database open;

Database altered.

SQL> show parameter control

NAME TYPE VALUE
———————————— ———– ——————————
control_file_record_keep_time integer 7
control_files string +DATA2/dbatst/controlfile/curr
ent.256.828803469, +FLASH2/dba
tst/controlfile/current.256.82
8803469
control_management_pack_access string NONE
SQL>

Related metalink note : How to duplicate a controlfile when ASM is involved [ID 345180.1]
Useful metalink note : How To Move The Database To Different Diskgroup (Change Diskgroup Redundancy) [ID 438580.1]

6. Moving SPfile to new ASM diskgroup

SQL> create pfile=’/tmp/pfile_copy.txt’ FROM SPFILE;

File created.

Edit the pfile /tmp/pfile_copy.txt to reflect the correct control files.

SQL> create spfile=’+DATA2′ FROM PFILE=’/tmp/pfile_copy.txt’;

File created.

Check the file in asmcmd and create an alias

ASMCMD> ls
DATA/
DATA2/
FLASH/
FLASH2/
ASMCMD> cd DATA2/
ASMCMD> cd DBATST/
ASMCMD> ls
CONTROLFILE/
PARAMETERFILE/
ASMCMD> cd PARAMETERFILE/
ASMCMD> ls
spfile.257.828803567
ASMCMD> pwd
+DATA2/DBATST/PARAMETERFILE
ASMCMD> cd ..
ASMCMD> mkalias +DATA2/DBATST/PARAMETERFILE/spfile.257.828803567 spfiledbatst.ora
ASMCMD> ls -l
Type Redund Striped Time Sys Name
Y CONTROLFILE/
Y PARAMETERFILE/
N spfiledbatst.ora => +DATA2/DBATST/PARAMETERFILE/spfile.257.828803567

[oracle@dba01tst ~]$ srvctl config database -d DBATST
Database unique name: DBATST
Database name: DBATST
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/DBATST/spfileDBATST.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DBATST
Database instances: DBATST1,DBATST2
Disk Groups: DATA,FLASH,DATA2,FLASH2
Mount point paths:
Services:
Type: RAC
Database is administrator managed

[oracle@dba01tst ~]$ srvctl config database -d DBATST
Database unique name: DBATST
Database name: DBATST
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/DBATST/spfileDBATST.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DBATST
Database instances: DBATST1,DBATST2
Disk Groups: DATA,FLASH,DATA2,FLASH2
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@dba01tst ~]$ srvctl modify database -d DBATST -p +DATA2/DBATST/spfiledbatst.ora
[oracle@dba01tst ~]$ srvctl config database -d DBATST
Database unique name: DBATST
Database name: DBATST
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA2/DBATST/spfiledbatst.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: DBATST
Database instances: DBATST1,DBATST2
Disk Groups: DATA,FLASH,DATA2,FLASH2
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@dba01tst ~]$

Modify the pfiles that give reference to the SPFile in $ORACLE_HOME/dbs

[oracle@dba01tst dbs]$ cat initDBATST1.ora
SPFILE=’+DATA/DBATST/spfileDBATST.ora’

[oracle@dba02tst dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:21:25 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to an idle instance.
SQL>startup
ORACLE instance started.

Total System Global Area 1043886080 bytes
Fixed Size 2234960 bytes
Variable Size 788530608 bytes
Database Buffers 247463936 bytes
Redo Buffers 5656576 bytes
Database mounted.
Database opened.

7. Moving data files to new ASM diskgroup

SQL> select ‘backup as copy datafile ‘||file#||’ format ”+DATA2”;’ from v$datafile;

‘BACKUPASCOPYDATAFILE’||FILE#||’FORMAT”+DATA2”;’
——————————————————————————–
backup as copy datafile 1 format ‘+DATA2’;
backup as copy datafile 2 format ‘+DATA2’;
backup as copy datafile 3 format ‘+DATA2’;
backup as copy datafile 4 format ‘+DATA2’;
backup as copy datafile 5 format ‘+DATA2’;

RMAN> run {
2> backup as copy datafile 1 format ‘+DATA2’;
3> backup as copy datafile 2 format ‘+DATA2’;
4> backup as copy datafile 3 format ‘+DATA2’;
5> backup as copy datafile 4 format ‘+DATA2’;
6> backup as copy datafile 5 format ‘+DATA2’;
7> }

Starting backup at 14-OCT-13
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=82 instance=DBATST1 device type=DISK
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA/dbatst/datafile/system.256.817911967
output file name=+DATA2/dbatst/datafile/system.258.828805189 tag=TAG20131014T153948 RECID=1 STAMP=828805204
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:25
channel ORA_DISK_1: starting datafile copy
copying current control file
output file name=+DATA2/dbatst/controlfile/backup.259.828805215 tag=TAG20131014T153948 RECID=2 STAMP=828805215
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:03
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including current SPFILE in backup set
channel ORA_DISK_1: starting piece 1 at 14-OCT-13
channel ORA_DISK_1: finished piece 1 at 14-OCT-13
piece handle=+DATA2/dbatst/backupset/2013_10_14/nnsnf0_tag20131014t153948_0.260.828805217 tag=TAG20131014T153948 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:01
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA/dbatst/datafile/sysaux.257.817911967
output file name=+DATA2/dbatst/datafile/sysaux.261.828805219 tag=TAG20131014T154018 RECID=3 STAMP=828805244
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA/dbatst/datafile/undotbs1.258.817911967
output file name=+DATA2/dbatst/datafile/undotbs1.262.828805255 tag=TAG20131014T154054 RECID=4 STAMP=828805263
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:15
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA/dbatst/datafile/users.259.817911967
output file name=+DATA2/dbatst/datafile/users.263.828805271 tag=TAG20131014T154109 RECID=5 STAMP=828805270
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:01
Finished backup at 14-OCT-13

Starting backup at 14-OCT-13
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA/dbatst/datafile/undotbs2.264.817912175
output file name=+DATA2/dbatst/datafile/undotbs2.264.828805271 tag=TAG20131014T154111 RECID=6 STAMP=828805276
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:07
Finished backup at 14-OCT-13

RMAN>

[oracle@dba01tst dbs]$ srvctl stop database -d DBATST
[oracle@dba01tst dbs]$ srvctl start instance -d DBATST -i DBATST1 -o mount
[oracle@dba01tst dbs]$ sqlplus / as sysdba
SQL> select ‘switch datafile ‘||file#||’ to copy;’ from v$datafile;

‘SWITCHDATAFILE’||FILE#||’TOCOPY;’
—————————————————————–
switch datafile 1 to copy;
switch datafile 2 to copy;
switch datafile 3 to copy;
switch datafile 4 to copy;
switch datafile 5 to copy;

SQL> select ‘recover datafile ‘||file#||’;’ from v$datafile;

‘RECOVERDATAFILE’||FILE#||’;’
———————————————————-
recover datafile 1;
recover datafile 2;
recover datafile 3;
recover datafile 4;
recover datafile 5;
[oracle@dba01tst dbs]$ rman target /

Recovery Manager: Release 11.2.0.3.0 – Production on Mon Oct 14 15:45:00 2013

Copyright (c) 1982, 2011, Oracle and/or its affiliates. All rights reserved.

connected to target database: DBATST (DBID=1790472310, not open)

RMAN> switch datafile 1 to copy;

using target database control file instead of recovery catalog
datafile 1 switched to datafile copy “+DATA2/dbatst/datafile/system.258.828805189”

RMAN> switch datafile 2 to copy;

datafile 2 switched to datafile copy “+DATA2/dbatst/datafile/sysaux.261.828805219”

RMAN> switch datafile 3 to copy;

datafile 3 switched to datafile copy “+DATA2/dbatst/datafile/undotbs1.262.828805255”

RMAN> switch datafile 4 to copy;

datafile 4 switched to datafile copy “+DATA2/dbatst/datafile/users.263.828805271”

RMAN> switch datafile 5 to copy;

datafile 5 switched to datafile copy “+DATA2/dbatst/datafile/undotbs2.264.828805271”

RMAN> recover datafile 1;

Starting recover at 14-OCT-13
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=196 instance=DBATST1 device type=DISK

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 14-OCT-13

RMAN> recover datafile 2;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 14-OCT-13

RMAN> recover datafile 3;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 14-OCT-13

RMAN> recover datafile 4;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:01

Finished recover at 14-OCT-13

RMAN> recover datafile 5;

Starting recover at 14-OCT-13
using channel ORA_DISK_1

starting media recovery
media recovery complete, elapsed time: 00:00:00

Finished recover at 14-OCT-13

[oracle@dba01tst dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:46:49 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter database open;

Database altered.

TAKE A FULL BACKUP

RMAN> delete datafilecopy all;

The backup cleared mine down on test

Useful metalink note : How to move a datafile from ASM to the file system [ID 390416.1]

8. Moving temp files to new ASM diskgroup

[oracle@dba01tst dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 15:50:54 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> select file_name from dba_temp_files;

FILE_NAME
——————————————————————————–
+DATA/dbatst/tempfile/temp.263.817912069

SQL> alter tablespace temp add tempfile ‘+data2(tempfile)’;

Tablespace altered.

SQL> select file_name from dba_temp_files;

FILE_NAME
——————————————————————————–
+DATA/dbatst/tempfile/temp.263.817912069
+DATA2/dbatst/tempfile/temp.260.828805895

SQL> alter tablespace temp drop tempfile ‘+DATA/dbatst/tempfile/temp.263.817912069’;

Tablespace altered.

9. Moving online redo log files to new ASM diskgroup

SQL> col member format a60
SQL> select group#,member from v$logfile;

GROUP# MEMBER
———- ————————————————————
2 +DATA/dbatst/onlinelog/group_2.262.817912061
2 +FLASH/dbatst/onlinelog/group_2.258.817912063
1 +DATA/dbatst/onlinelog/group_1.261.817912059
1 +FLASH/dbatst/onlinelog/group_1.257.817912061
3 +DATA/dbatst/onlinelog/group_3.265.817912283
3 +FLASH/dbatst/onlinelog/group_3.259.817912285
4 +DATA/dbatst/onlinelog/group_4.266.817912285
4 +FLASH/dbatst/onlinelog/group_4.260.817912285

8 rows selected.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 1;

Database altered.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 2;

Database altered.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 3;

Database altered.

SQL> alter database add logfile member ‘+DATA2′,’+FLASH2’ to group 4;

Database altered.

Drop the extra log file members

alter database drop logfile member ‘+FLASH/dbatst/onlinelog/group_3.259.817912285′;

SQL> select group#,member from v$logfile order by 1;

GROUP# MEMBER
———- ————————————————————
1 +DATA2/dbatst/onlinelog/group_1.259.828806063
1 +FLASH2/dbatst/onlinelog/group_1.257.828806065
2 +DATA2/dbatst/onlinelog/group_2.265.828806069
2 +FLASH2/dbatst/onlinelog/group_2.258.828806069
3 +DATA2/dbatst/onlinelog/group_3.266.828806077
3 +FLASH2/dbatst/onlinelog/group_3.259.828806077
4 +DATA2/dbatst/onlinelog/group_4.267.828806081
4 +FLASH2/dbatst/onlinelog/group_4.260.828806083

8 rows selected.

SQL> show parameter recover

NAME TYPE VALUE
———————————— ———– ——————————
db_recovery_file_dest string +FLASH
db_recovery_file_dest_size big integer 800M
db_unrecoverable_scn_tracking boolean TRUE
recovery_parallelism integer 0
SQL> alter system set db_recovery_file_dest=’+FLASH2′;

System altered.

SQL> show parameter recover

NAME TYPE VALUE
———————————— ———– ——————————
db_recovery_file_dest string +FLASH2
db_recovery_file_dest_size big integer 800M
db_unrecoverable_scn_tracking boolean TRUE
recovery_parallelism integer 0

SQL> show parameter db_create_file_dest

NAME TYPE VALUE
———————————— ———– ——————————
db_create_file_dest string +DATA
SQL> alter system set db_create_file_dest=’+DATA2’;

System altered.

SQL> show parameter db_create_file_dest

NAME TYPE VALUE
———————————— ———– ——————————
db_create_file_dest string +DATA2
SQL>

SQL> show parameter db_create_online_log_dest

NAME TYPE VALUE
———————————— ———– ——————————
db_create_online_log_dest_1 string
db_create_online_log_dest_2 string
db_create_online_log_dest_3 string
db_create_online_log_dest_4 string
db_create_online_log_dest_5 string

SQL> select max(sequence#),thread# from v$archived_log group by thread#;

MAX(SEQUENCE#) THREAD#
————– ———-
444 1
420 2

SQL> select name from v$archived_log where (sequence#=200 and thread#=1) or (sequence#=151 and thread#=2);

no rows selected

SQL> select name from v$archived_log where (sequence#=444 and thread#=1) or (sequence#=420 and thread#=2);

NAME
——————————————————————————–
+FLASH2/dbatst/archivelog/2013_10_14/thread_2_seq_420.262.828807191
+FLASH2/dbatst/archivelog/2013_10_14/thread_1_seq_444.263.828807193

10 . Dropping old disk groups

[root@dba01tst ~]# crsctl stop cluster -all
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dba01tst’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN2.lsnr’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.cvu’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN1.lsnr’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER_SCAN3.lsnr’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.DATA2.dg’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.FLASH2.dg’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.cvu’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.LISTENER_SCAN2.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.scan2.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.LISTENER_SCAN3.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.scan3.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.LISTENER_SCAN1.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.scan1.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.dba01tst.vip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.scan2.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.scan3.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.scan1.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.dba01tst.vip’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.FLASH2.dg’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.crsd’ on ‘dba02tst’
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.DATA.dg’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.DATA2.dg’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.FLASH2.dg’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.LISTENER.lsnr’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.LISTENER.lsnr’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.dba02tst.vip’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.dba02tst.vip’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.FLASH2.dg’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.DATA.dg’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.DATA2.dg’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.asm’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.DATA2.dg’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.asm’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.ons’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.ons’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.net1.network’ on ‘dba02tst’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dba02tst’ has completed
CRS-2673: Attempting to stop ‘ora.ons’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.ons’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.net1.network’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.net1.network’ on ‘dba01tst’ succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on ‘dba01tst’ has completed
CRS-2677: Stop of ‘ora.crsd’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dba02tst’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.crsd’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.ctssd’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.evmd’ on ‘dba01tst’
CRS-2673: Attempting to stop ‘ora.asm’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.evmd’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.evmd’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.ctssd’ on ‘dba01tst’ succeeded
CRS-2677: Stop of ‘ora.asm’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dba02tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dba02tst’
CRS-2677: Stop of ‘ora.asm’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cluster_interconnect.haip’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.cssd’ on ‘dba02tst’ succeeded
CRS-2677: Stop of ‘ora.cluster_interconnect.haip’ on ‘dba01tst’ succeeded
CRS-2673: Attempting to stop ‘ora.cssd’ on ‘dba01tst’
CRS-2677: Stop of ‘ora.cssd’ on ‘dba01tst’ succeeded
[root@dba01tst ~]# crsctl start cluster -all
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.cssdmonitor’ on ‘dba02tst’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.cssd’ on ‘dba01tst’
CRS-2676: Start of ‘ora.cssdmonitor’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.cssd’ on ‘dba02tst’
CRS-2672: Attempting to start ‘ora.diskmon’ on ‘dba02tst’
CRS-2676: Start of ‘ora.diskmon’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.diskmon’ on ‘dba02tst’ succeeded
CRS-2676: Start of ‘ora.cssd’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘dba01tst’
CRS-2676: Start of ‘ora.cssd’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.ctssd’ on ‘dba02tst’
CRS-2672: Attempting to start ‘ora.cluster_interconnect.haip’ on ‘dba02tst’
CRS-2676: Start of ‘ora.ctssd’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.ctssd’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.evmd’ on ‘dba01tst’
CRS-2672: Attempting to start ‘ora.evmd’ on ‘dba02tst’
CRS-2676: Start of ‘ora.evmd’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.evmd’ on ‘dba02tst’ succeeded
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘dba01tst’
CRS-2676: Start of ‘ora.cluster_interconnect.haip’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.asm’ on ‘dba02tst’
CRS-2676: Start of ‘ora.asm’ on ‘dba01tst’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘dba01tst’
CRS-2676: Start of ‘ora.crsd’ on ‘dba01tst’ succeeded
CRS-2676: Start of ‘ora.asm’ on ‘dba02tst’ succeeded
CRS-2672: Attempting to start ‘ora.crsd’ on ‘dba02tst’
CRS-2676: Start of ‘ora.crsd’ on ‘dba02tst’ succeeded

Dismount the disks on one node

[oracle@dba02tst trace]$ sqlplus / as sysasm

SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 14 16:40:20 2013

Copyright (c) 1982, 2011, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production
With the Real Application Clusters and Automatic Storage Management options

SQL> alter diskgroup data dismount;

Diskgroup altered.

SQL> alter diskgroup flash dismount;

Diskgroup altered.

Drop the disks from the other node

SQL> drop diskgroup flash including contents;

Diskgroup dropped.

SQL> drop diskgroup data including contents;

Diskgroup dropped.

SQL>

Remove the disks from OCR

[oracle@dba01tst ~]$ srvctl remove diskgroup -g DATA
PRCA-1002 : Failed to remove CRS resource ora.DATA.dg for ASM Disk Group DATA
PRCR-1028 : Failed to remove resource ora.DATA.dg
PRCR-1072 : Failed to unregister resource ora.DATA.dg
CRS-0222: Resource ‘ora.DATA.dg’ has dependency error.

[oracle@dba01tst ~]$ srvctl modify database -d DBATST -a “DATA2,FLASH2”
[oracle@dba01tst ~]$ srvctl remove diskgroup -g DATA
[oracle@dba01tst ~]$ srvctl remove diskgroup -g FLASH
[oracle@dba01tst ~]$ srvctl start database -d DBATST

Comments
  1. Sammer (Oracle Database Admin.) says:

    Awesome Docs !!! It is cover most of the things in cluster environment. Really appriciated…..

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s