Play with my toy 10g RAC II - Familiar with SRVCTL

In this post, I continue to describe the experiences I have gained through playing with my little toy RAC. First of all, I found my RAC database could not fully function after restarting the VM servers, i.e. after my VM RAC1 and RAC2 nodes are powered on, the status of the cluster is usually shown as following:

[oracle@rac1 ~]$ ./crs_rep.sh
HA Resource                                   Target     State
-----------                                   ------     -----
ora.devdb.CRM.cs                              OFFLINE    OFFLINE
ora.devdb.CRM.devdb1.srv                      OFFLINE    OFFLINE
ora.devdb.db                                  ONLINE     ONLINE on rac1
ora.devdb.devdb1.inst                         ONLINE     ONLINE on rac1
ora.devdb.devdb2.inst                         ONLINE     ONLINE on rac2
ora.rac1.ASM1.asm                             ONLINE     ONLINE on rac1
ora.rac1.LISTENER_RAC1.lsnr                   ONLINE     UNKNOWN on rac1
ora.rac1.gsd                                  ONLINE     UNKNOWN on rac1
ora.rac1.ons                                  ONLINE     ONLINE on rac1
ora.rac1.vip                                  ONLINE     ONLINE on rac1
ora.rac2.ASM2.asm                             ONLINE     ONLINE on rac2
ora.rac2.LISTENER_RAC2.lsnr                   ONLINE     UNKNOWN on rac2
ora.rac2.gsd                                  ONLINE     UNKNOWN on rac2
ora.rac2.ons                                  ONLINE     ONLINE on rac2
ora.rac2.vip                                  ONLINE     ONLINE on rac2

It can be seen that the status of gsd and listener rescours are unknown. The service CRM is not started. I have to follow the following steps, which use SRVCTL exlusively to get the RAC work:

- Stop application resources

srvctl stop database -d devdb
srvctl stop asm -n rac1
srvctl stop asm -n rac2
srvctl stop nodeapps -n rac1
srvctl stop nodeapps -n rac2
- Start application resources

srvctl start nodeapps -n rac1

srvctl start nodeapps -n rac2
srvctl start asm -n rac1
srvctl start asm -n rac2
srvctl start database -d devdb
srvctl start service -d devdb -s CRM


Here I list some other srvctl commands which may be useful for reference:

- Start and stop instances
srvctl start instance -d devdb -i devdb1
srvctl stop instance -d devdb -i devdb1

- Start and stop listeners
srvctl start listener -n devdb1 -l LISTENER_RAC1
srvctl stop listener -n devdb2 -l LISTENER_RAC2

- View current configuration for the entire cluster database
srvctl config database -d devdb -a

[oracle@rac1 ~]$ srvctl config database -d devdb -a
rac1 devdb1 /u01/app/oracle/product/10.2.0/db_1
rac2 devdb2 /u01/app/oracle/product/10.2.0/db_1
DB_NAME: devdb
ORACLE_HOME: /u01/app/oracle/product/10.2.0/db_1
SPFILE: +DG1/devdb/spfiledevdb.ora
DOMAIN: null
DB_ROLE: null
START_OPTIONS: null
POLICY:  AUTOMATIC
ENABLE FLAG: DB ENABLED

- View the configuration for all node applications for node rac2
srvctl config nodeapps -n rac2

- Create a new service
srvctl add service -d devdb -s CRM

- Start a service on the whole cluster
srvctl start service -d devdb -s CRM

- Stop a service on both instance, force it to stop (-f)
srvctl stop service -d devdb -s CRM -i devdb1,devdb2 -f

Note: it does not work, I have to do:
srvctl stop service -d devdb -s CRM -i devdb1 -f
srvctl stop service -d devdb -s CRM -i devdb2 -f

- Enable a service on the devdb2 instance only
srvctl enable service -d devdb -s CRM -i devdb2

- Disable a service on the devdb1 instance only
srvctl disable service -d devdb -s CRM -i devdb1

- Modify a service so that devdb2 becomes preferred instance
srvctl modify service -d devdb -s CRM -i devdb2 -r

- Remove a service
srvctl remove service -d devdb -s CRM

- View config for a service
srvctl config service -d devdb -s CRM -a

Apr 17,2010 Update:

After all CRS resource offline, I was able to bring everything up by: crs_start -all

[oracle@rac1 ~]$ ./crs_rep.sh
HA Resource                                   Target     State
-----------                                   ------     -----
ora.devdb.SLBA.cs                             OFFLINE    OFFLINE
ora.devdb.SLBA.devdb1.srv                     OFFLINE    OFFLINE
ora.devdb.SLBA.devdb2.srv                     OFFLINE    OFFLINE
ora.devdb.SNOLBA.cs                           OFFLINE    OFFLINE
ora.devdb.SNOLBA.devdb1.srv                   OFFLINE    OFFLINE
ora.devdb.SNOLBA.devdb2.srv                   OFFLINE    OFFLINE
ora.devdb.db                                  OFFLINE    OFFLINE
ora.devdb.devdb1.inst                         OFFLINE    OFFLINE
ora.devdb.devdb2.inst                         OFFLINE    OFFLINE
ora.rac1.ASM1.asm                             OFFLINE    OFFLINE
ora.rac1.LISTENER_RAC1.lsnr                   OFFLINE    OFFLINE
ora.rac1.gsd                                  OFFLINE    OFFLINE
ora.rac1.ons                                  OFFLINE    OFFLINE
ora.rac1.vip                                  OFFLINE    OFFLINE
ora.rac2.ASM2.asm                             OFFLINE    OFFLINE
ora.rac2.LISTENER_RAC2.lsnr                   OFFLINE    OFFLINE
ora.rac2.gsd                                  OFFLINE    OFFLINE
ora.rac2.ons                                  OFFLINE    OFFLINE
ora.rac2.vip                                  OFFLINE    OFFLINE
oracle@rac1 ~]$ crs_start -all
Attempting to start `ora.rac1.vip` on member `rac1`
Attempting to start `ora.rac2.vip` on member `rac2`
Start of `ora.rac2.vip` on member `rac2` succeeded.
Start of `ora.rac1.vip` on member `rac1` succeeded.
Attempting to start `ora.rac1.ASM1.asm` on member `rac1`
Attempting to start `ora.rac2.ASM2.asm` on member `rac2`
Start of `ora.rac2.ASM2.asm` on member `rac2` succeeded.
Attempting to start `ora.devdb.devdb2.inst` on member `rac2`
Start of `ora.rac1.ASM1.asm` on member `rac1` succeeded.
Attempting to start `ora.devdb.devdb1.inst` on member `rac1`
Start of `ora.devdb.devdb2.inst` on member `rac2` succeeded.
Attempting to start `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2`
Start of `ora.rac2.LISTENER_RAC2.lsnr` on member `rac2` succeeded.
Start of `ora.devdb.devdb1.inst` on member `rac1` succeeded.
Attempting to start `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1`
Start of `ora.rac1.LISTENER_RAC1.lsnr` on member `rac1` succeeded.
Attempting to start `ora.devdb.SLBA.cs` on member `rac1`
Attempting to start `ora.devdb.SLBA.devdb2.srv` on member `rac2`
Attempting to start `ora.devdb.SLBA.devdb1.srv` on member `rac1`
Start of `ora.devdb.SLBA.devdb2.srv` on member `rac2` succeeded.
Attempting to start `ora.devdb.SNOLBA.cs` on member `rac1`
Attempting to start `ora.devdb.SNOLBA.devdb1.srv` on member `rac1`
Start of `ora.devdb.SLBA.cs` on member `rac1` succeeded.
Attempting to start `ora.rac1.gsd` on member `rac1`
CRS-1002: Resource 'ora.rac1.ons' is already running on member 'rac1'

Attempting to start `ora.devdb.SNOLBA.devdb2.srv` on member `rac2`
Start of `ora.devdb.SNOLBA.devdb2.srv` on member `rac2` succeeded.
Start of `ora.rac1.gsd` on member `rac1` succeeded.
CRS-1002: Resource 'ora.devdb.db' is already running on member 'rac1'

CRS-1002: Resource 'ora.rac2.ons' is already running on member 'rac2'

Start of `ora.devdb.SLBA.devdb1.srv` on member `rac1` succeeded.
Attempting to start `ora.rac2.gsd` on member `rac2`
Start of `ora.devdb.SNOLBA.cs` on member `rac1` succeeded.
Start of `ora.rac2.gsd` on member `rac2` succeeded.
Start of `ora.devdb.SNOLBA.devdb1.srv` on member `rac1` succeeded.
CRS-0223: Resource 'ora.devdb.db' has placement error.

CRS-0223: Resource 'ora.rac1.ons' has placement error.

CRS-0223: Resource 'ora.rac2.ons' has placement error.

[oracle@rac1 ~]$ ./crs_rep.sh
HA Resource                                   Target     State
-----------                                   ------     -----
ora.devdb.SLBA.cs                             ONLINE     ONLINE on rac1
ora.devdb.SLBA.devdb1.srv                     ONLINE     ONLINE on rac1
ora.devdb.SLBA.devdb2.srv                     ONLINE     ONLINE on rac2
ora.devdb.SNOLBA.cs                           ONLINE     ONLINE on rac1
ora.devdb.SNOLBA.devdb1.srv                   ONLINE     ONLINE on rac1
ora.devdb.SNOLBA.devdb2.srv                   ONLINE     ONLINE on rac2
ora.devdb.db                                  ONLINE     ONLINE on rac1
ora.devdb.devdb1.inst                         ONLINE     ONLINE on rac1
ora.devdb.devdb2.inst                         ONLINE     ONLINE on rac2
ora.rac1.ASM1.asm                             ONLINE     ONLINE on rac1
ora.rac1.LISTENER_RAC1.lsnr                   ONLINE     ONLINE on rac1
ora.rac1.gsd                                  ONLINE     ONLINE on rac1
ora.rac1.ons                                  ONLINE     ONLINE on rac1
ora.rac1.vip                                  ONLINE     ONLINE on rac1
ora.rac2.ASM2.asm                             ONLINE     ONLINE on rac2
ora.rac2.LISTENER_RAC2.lsnr                   ONLINE     ONLINE on rac2
ora.rac2.gsd                                  ONLINE     ONLINE on rac2
ora.rac2.ons                                  ONLINE     ONLINE on rac2
ora.rac2.vip                                  ONLINE     ONLINE on rac2