Steps:
1. Create a new shared virtual SCSI hard disk
a. Add a virtual disk (3GB) for RAC1
b. In the "Add Hardward Wizard" and "Hard Disk" windows for RAC2, Select the following option:
Use an Existing Virtual Disk - Choose this option to reuse or share a hard disk from another Virtual Machine
2. Modify virtual machine configuration file
Additional parameters are required to enable disk sharing between the two virtual RAC nodes. (Details see Vincent's article). Additonal reference is here.
3. Identify the device name for the disk
In RAC1 and RAC2, login as root, run command: fdisk -l, the following message can be seen:
Disk /dev/sdf: 3221 MB, 3221225472 bytes 255 heads, 63 sectors/track, 391 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk /dev/sdf doesn't contain a valid partition table
4. Create disk partitions
General reference about fdisk can be seen here.
At the end, I created partitions for /dev/sdf shown as follows:
Disk /dev/sdf: 3221 MB, 3221225472 bytes 255 heads, 63 sectors/track, 391 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Device Boot Start End Blocks Id System /dev/sdf1 1 63 506016 83 Linux /dev/sdf2 64 126 506047+ 83 Linux /dev/sdf3 127 131 40162+ 83 Linux /dev/sdf4 132 391 2088450 5 Extended /dev/sdf5 132 136 40131 83 Linux /dev/sdf6 137 141 40131 83 Linux /dev/sdf7 142 391 2008093+ 83 Linux
Note: in reality, OCR, OCRMIRROR and each votingdisk file should be placed on seperate physical disks. Here I placed them in the same disk but different partitions. This is just for learning purpose.
5. Binding the partition
To map the raw devices to the shared partitions created, I did the following:
a. Edit /etc/sysconfig/rawdevices on both nodes, add the following:
/dev/raw/raw4 /dev/sdf1 /dev/raw/raw5 /dev/sdf2 /dev/raw/raw6 /dev/sdf3 /dev/raw/raw7 /dev/sdf5 /dev/raw/raw8 /dev/sdf6 /dev/raw/raw9 /dev/sdf7
b. Issue: /sbin/service rawdevices restart
root@rac1:/dev/raw [devdb1]# /sbin/service rawdevices restart
Assigning devices:
/dev/raw/raw1 --> /dev/sdc1
/dev/raw/raw1: bound to major 8, minor 33
/dev/raw/raw2 --> /dev/sdd1
/dev/raw/raw2: bound to major 8, minor 49
/dev/raw/raw3 --> /dev/sde1
/dev/raw/raw3: bound to major 8, minor 65
/dev/raw/raw4 --> /dev/sdf1
/dev/raw/raw4: bound to major 8, minor 81
/dev/raw/raw5 --> /dev/sdf2
/dev/raw/raw5: bound to major 8, minor 82
/dev/raw/raw6 --> /dev/sdf3
/dev/raw/raw6: bound to major 8, minor 83
/dev/raw/raw7 --> /dev/sdf5
/dev/raw/raw7: bound to major 8, minor 85
/dev/raw/raw8 --> /dev/sdf6
/dev/raw/raw8: bound to major 8, minor 86
/dev/raw/raw9 --> /dev/sdf7
/dev/raw/raw9: bound to major 8, minor 87
done
Note: when issuing above command in RAC2, I recieve message: /dev/sdfn (n=1-7) is not a block device.
After reboot both node, everything is fine:
root@rac1:~ [devdb1]# raw -qa /dev/raw/raw1: bound to major 8, minor 33 /dev/raw/raw2: bound to major 8, minor 49 /dev/raw/raw3: bound to major 8, minor 65 /dev/raw/raw4: bound to major 8, minor 81 /dev/raw/raw5: bound to major 8, minor 82 /dev/raw/raw6: bound to major 8, minor 83 /dev/raw/raw7: bound to major 8, minor 85 /dev/raw/raw8: bound to major 8, minor 86 /dev/raw/raw9: bound to major 8, minor 87 [root@rac2 ~]# raw -qa /dev/raw/raw1: bound to major 8, minor 33 /dev/raw/raw2: bound to major 8, minor 49 /dev/raw/raw3: bound to major 8, minor 65 /dev/raw/raw4: bound to major 8, minor 81 /dev/raw/raw5: bound to major 8, minor 82 /dev/raw/raw6: bound to major 8, minor 83 /dev/raw/raw7: bound to major 8, minor 85 /dev/raw/raw8: bound to major 8, minor 86 /dev/raw/raw9: bound to major 8, minor 87
c. Edit /etc/udev/permissions.d/50-udev.permissions, add the following entries, and reboot
raw/raw1:oracle:dba:0660 raw/raw2:oracle:dba:0660 raw/raw3:oracle:dba:0660 raw/raw4:root:dba:0640 raw/raw5:root:dba:0640 raw/raw6:oracle:dba:0640 raw/raw7:oracle:dba:0640 raw/raw8:oracle:dba:0640 raw/raw9:oracle:dba:0660
Note: my intended use of raw devices is raw4 and raw5 for OCR and OCRMIRROR; raw6, raw7, raw8 for voting disk files.
6. Moving OCR file from OCFS to raw devices
a. shutdown CRS resource
b. Determine current OCR configuration:
[oracle@rac2 ~]$ ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 262144
Used space (kbytes) : 5364
Available space (kbytes) : 256780
ID : 645781380
Device/File Name : /ocfs/clusterware/ocr
Device/File integrity check succeeded
Device/File not configured
Cluster registry integrity check succeeded
c. Relocate ocr and add ocr mirror
root@rac1:~ [devdb1]# ocrconfig -replace ocrmirror /dev/raw/raw5 root@rac1:~ [devdb1]# ocrconfig -replace ocr /dev/raw/raw4
Note: if ocrmirror is not added first, the following errors will be seen:
root@rac1:~ [devdb1]# ocrconfig -replace ocr /dev/raw/raw4 PROT-16: Internal Error
d. Verify and start CRS resources
root@rac1:/ocfs/clusterware [devdb1]# ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 2
Total space (kbytes) : 505928
Used space (kbytes) : 5364
Available space (kbytes) : 500564
ID : 645781380
Device/File Name : /dev/raw/raw4
Device/File needs to be synchronized with the other device
Device/File Name : /dev/raw/raw5
Device/File integrity check succeeded
Cluster registry integrity check succeeded
[May 4,2010, update: I did not notice "Device/File needs to be synchronized ..." in the above ocrcheck output on Apr 28. CRS processes could be started despite of this. But today I have to fix this to get CRS started. see http://oracle-study-notes.blogspot.com/2010/05/play-with-my-toy-10g-rac-vi-failed-to.html ]
7. Moving voting disk files from OCFS to raw devices
a. Determine the current voting disk location
oracle@rac1:~ [devdb1]$ crsctl query css votedisk 0. 0 /ocfs/clusterware/votingdisk located 1 votedisk(s).
b. Shutdown Oracle Clusterware on both node;
oracle@rac1:~ [devdb1]$ sudo $ORA_CRS_HOME/bin/crsctl stop crs Password: Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. oracle@rac1:~ [devdb1]$ ssh rac2 sudo $ORA_CRS_HOME/bin/crsctl stop crs Password:vz123ys Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued.
c. Take a backup of all voting disk:
oracle@rac1:~ [devdb1]$ dd if=/ocfs/clusterware/votingdisk of=/home/oracle/backup/votingdisk_042810 20000+0 records in 20000+0 records out
d. Move
root@rac1:~ [devdb1]# crsctl delete votedisk css /ocfs/clusterware/votingdisk -force Unexpected parameter: votedisk root@rac1:~ [devdb1]# crsctl delete css votedisk /ocfs/clusterware/votingdisk -force only one votedisk /ocfs/clusterware/votingdisk configured. please add another disk before removing the last one. root@rac1:~ [devdb1]# crsctl add css votedisk /dev/raw/raw7 -force Now formatting voting disk: /dev/raw/raw7 successful addition of votedisk /dev/raw/raw7. root@rac1:~ [devdb1]# crsctl delete css votedisk /ocfs/clusterware/votingdisk -force successful deletion of votedisk /ocfs/clusterware/votingdisk. root@rac1:~ [devdb1]# crsctl add css votedisk /dev/raw/raw6 -force Now formatting voting disk: /dev/raw/raw6 successful addition of votedisk /dev/raw/raw6. root@rac1:~ [devdb1]# crsctl add css votedisk /dev/raw/raw8 -force Now formatting voting disk: /dev/raw/raw8 successful addition of votedisk /dev/raw/raw8. root@rac1:~ [devdb1]# crsctl query css votedisk 0. 0 /dev/raw/raw6 1. 0 /dev/raw/raw7 2. 0 /dev/raw/raw8 located 3 votedisk(s).
e. Start Oracle Clusterware and verify everything is ok
oracle@rac1:~ [devdb1]$ sudo $ORA_CRS_HOME/bin/crsctl start crs Password: Attempting to start CRS stack The CRS stack will be started shortly oracle@rac1:~ [devdb1]$ ssh rac2 sudo $ORA_CRS_HOME/bin/crsctl start crs Password:vz123ys Attempting to start CRS stack The CRS stack will be started shortly --- Wait for several min here ----------- oracle@rac1:~ [devdb1]$ oracle@rac1:~ [devdb1]$ ./crs_rep.sh HA Resource Target State ----------- ------ ----- ora.devdb.SLBA.cs ONLINE ONLINE on rac1 ora.devdb.SLBA.devdb1.srv ONLINE ONLINE on rac1 ora.devdb.SLBA.devdb2.srv ONLINE ONLINE on rac2 ora.devdb.SNOLBA.cs ONLINE ONLINE on rac1 ora.devdb.SNOLBA.devdb1.srv ONLINE ONLINE on rac1 ora.devdb.SNOLBA.devdb2.srv ONLINE ONLINE on rac2 ora.devdb.db ONLINE ONLINE on rac2 ora.devdb.devdb1.inst ONLINE ONLINE on rac1 ora.devdb.devdb2.inst ONLINE ONLINE on rac2 ora.rac1.ASM1.asm ONLINE ONLINE on rac1 ora.rac1.LISTENER_RAC1.lsnr ONLINE ONLINE on rac1 ora.rac1.gsd ONLINE ONLINE on rac1 ora.rac1.ons ONLINE ONLINE on rac1 ora.rac1.vip ONLINE ONLINE on rac1 ora.rac2.ASM2.asm ONLINE ONLINE on rac2 ora.rac2.LISTENER_RAC2.lsnr ONLINE ONLINE on rac2 ora.rac2.gsd ONLINE ONLINE on rac2 ora.rac2.ons ONLINE ONLINE on rac2 ora.rac2.vip ONLINE ONLINE on rac2 oracle@rac1:~ [devdb1]$ crsctl query css votingdisk Unexpected parameter: votingdisk oracle@rac1:~ [devdb1]$ crsctl query css votedisk 0. 0 /dev/raw/raw6 1. 0 /dev/raw/raw7 2. 0 /dev/raw/raw8 located 3 votedisk(s).
No comments:
Post a Comment