Wednesday, July 23, 2014

ORA-29702 While Starting Instance(s) on RAC/Clusterware

I was in process of upgrading my 3 nodes RAC database to RAC on Windows 2008. Since we had new hardware, we decided to build a new RAC and create a standby on the new RAC and then do a switch-over and immediately after that, perform the upgrade of switched over database. To accomplish this task, we followed following steps

1) Build a new 11.2.03 RAC by installing GI and RDBMS
2) Install on this new RAC
3) Build standby database on this new RAC setup
4) Perform switchover to this new RAC standby
5) Upgrade to
6) Perform any post upgrade tasks

On step 3 while building standby database, when I tried to start the instances, it failed to start with error "ORA-29702: error occurred in Cluster Group Service operation".

In Oracle 11.2 RAC, there is a concept of "pinning of nodes". By default, RAC nodes are not pinned, 
click here for the details.
10g is an older version in which, by default, nodes are actually "pinned" (having consistent configuration information), whereas in 11g; functionality is different and this error was appearing because of this reason.

Pinning of nodes resolved this issue. Following are the steps to check current nodes configuration and how to pin the nodes.

#####Getting information

%GRID_HOME%\bin\olsnodes -t
node01 Unpinned 
node02 Unpinned
node02 Unpinned 

#####Pinning the nodes by executing following from node1
crsctl pin css -n node01 node02 node03 
#####Getting information
%GRID_HOME%\bin\olsnodes -t
node01 1 Pinned
node02 2 Pinned
node02 3 Pinned 

After pinning the nodes, I was able to start all standby instances.

No comments: