Archive for the ‘OracleRAC’ Category

Add Node to Oracle RAC 11gR2(11.2.0.3) on Oracle Linux 6.5 2 Node RAC Infrastructure   1 comment

Here i am going to post the steps needed to add a node to the existing two node RAC 11.2.0.3 Infrastructure on Oracle Virtual BOX

Already this is two node RAC Infrastructure configured on OEL6.5 on Oracle Virtual BOX.

I Have used backup to create additional node rac3.

During the Installation of 11.2.0.3 RAC on Virtual BOX before the configuration of ASM disks I have taken backup of Virtual BOX VM.

Please see my previous post
https://balaoracledba.com/2013/12/23/step-by-step-install-of-oracle-rac-11gr2-on-oel-6-5-on-oracle-vm-virtual-box/
In the above post i have backed up the VM to some Backup location. To backup i have exported the present Appliance.
Backup VM. Click File export Appliance. 

Now to Create a New RAC node, i have imported the appliance which was backed up earlier at the time installation of RAC.

Click on File->Import Appliance

If you dont have any backup. then need to configure a fresh vm and  installation OEL6.5 with all pre-reqs steps which were mentioned in my previous post 11gRACINSTALL

01

Select the backup file .ova path which was taken earlier.

02

Change the name to rac3 and scroll down to select the location where you want to save the imported VM.

03

04

05

RAC3 vm is going to get created once we import the appliance

06

07

Add ASMDISKS click on storage.

07

Click on the icon show in below screenshot

08

Click on Add Hard Disk

09

And select “choose existing disk”

10

Go to the ASM disk 1 and select it and click ok

11

Repeat this step for the remaining ASMDISKs*

12

Start the RAC3 VM

13

15

hostname has to be changed. Change it to rac3.

16

17

Save and exit.

Now we need to configure virtual nics with correct IP’s for Private and Public

Follow the navigation as below

18

Change the IP address from 192.168.56.71 to 192.168.56.73 this is for PUBLIC

19

Now change for Private also

20

Add the entries in /etc/hosts file as below

21

Install oracleasmlib rpm and configure ASM

22

23

in the above screenshot i have mistakenly provided group as dba(which should be oinstall).

24

25

Also install the missing RPM’s

15 16 17 18 19 20

also Configure SSH between all the 3 Nodes. I followed http://oracle-base.com/articles/linux/user-equivalence-configuration-on-linux.php

it has ssh setup for 2 Node rac

Manual User Equivalence (Key-Based Authentication) Configuration

Assuming we have a two node cluster (rac1.localdomain, rac2.localdomain), log in as the “oracle” user and perform the following tasks on each node.

su - oracle
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa # Accept the default settings.

The RSA public key is written to the ~/.ssh/id_rsa.pub file and the private key to the ~/.ssh/id_rsa file.

Log in as the “oracle” user on rac1.localdomain, generate an “authorized_keys” file and copy it to rac2.localdomainusing the following commands.

su - oracle
cd ~/.ssh
cat id_rsa.pub >> authorized_keys
scp authorized_keys rac2.localdomain:/home/oracle/.ssh/

Next, log in as the “oracle” user on rac2.localdomain and perform the following commands.

su - oracle
cd ~/.ssh
cat id_rsa.pub >> authorized_keys
scp authorized_keys rac1.localdomain:/home/oracle/.ssh/

The “authorized_keys” file on both servers now contains the public keys generated on all nodes.

To enable SSH user equivalency on the cluster member nodes issue the following commands on each node.

ssh rac1 date ssh rac2 date ssh rac1.localdomain date ssh rac2.localdomain date exec /usr/bin/ssh-agent $SHELL /usr/bin/ssh-add

But we need to work on configure ssh on 3 node rac for oracle user!!!

Copy hosts file from rac3 to the rest of the 2 servers rac1 and rac2.

Copy nslookup file from /usr/bin from rac1 to rac3

cluvfy stage -pre nodeadd -n rac3 -fixup -verbose

26
Performing pre-checks for node addition
Checking node reachability...
Check: Node reachability from node "rac1"
 Destination Node Reachable? 
 ------------------------------------ ------------------------
 rac3 yes 
Result: Node reachability check passed from node "rac1"

Checking user equivalence...
Check: User equivalence for user "oracle"
 Node Name Status 
 ------------------------------------ ------------------------
 rac3 passed 
Result: User equivalence check passed for user "oracle"
Checking node connectivity...
Checking hosts config file...
 Node Name Status 
 ------------------------------------ ------------------------
 rac1 passed 
 rac2 passed 
 rac3 passed
Verification of the hosts config file successful

Interface information for node "rac1"
 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0 192.168.56.71 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.81 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.91 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth1 192.168.10.1 192.168.10.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth1 169.254.149.126 169.254.0.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth2 192.168.1.8 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:94:63:1F 1500

Interface information for node "rac2"
 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0 192.168.56.72 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.92 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.93 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.82 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth1 192.168.10.2 192.168.10.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth1 169.254.184.13 169.254.0.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth2 192.168.1.7 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:94:63:1F 1500

Interface information for node "rac3"
 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0 192.168.56.73 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth1 192.168.10.3 192.168.10.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth2 192.168.1.9 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:94:63:1F 1500

Check: Node connectivity for interface "eth0"
 Source Destination Connected? 
 ------------------------------ ------------------------------ ----------------
 rac1[192.168.56.71] rac1[192.168.56.81] yes 
 rac1[192.168.56.71] rac1[192.168.56.91] yes 
 rac1[192.168.56.71] rac2[192.168.56.72] yes 
 rac1[192.168.56.71] rac2[192.168.56.92] yes 
 rac1[192.168.56.71] rac2[192.168.56.93] yes 
 rac1[192.168.56.71] rac2[192.168.56.82] yes 
 rac1[192.168.56.71] rac3[192.168.56.73] yes 
 rac1[192.168.56.81] rac1[192.168.56.91] yes 
 rac1[192.168.56.81] rac2[192.168.56.72] yes 
 rac1[192.168.56.81] rac2[192.168.56.92] yes 
 rac1[192.168.56.81] rac2[192.168.56.93] yes 
 rac1[192.168.56.81] rac2[192.168.56.82] yes 
 rac1[192.168.56.81] rac3[192.168.56.73] yes 
 rac1[192.168.56.91] rac2[192.168.56.72] yes 
 rac1[192.168.56.91] rac2[192.168.56.92] yes 
 rac1[192.168.56.91] rac2[192.168.56.93] yes 
 rac1[192.168.56.91] rac2[192.168.56.82] yes 
 rac1[192.168.56.91] rac3[192.168.56.73] yes 
 rac2[192.168.56.72] rac2[192.168.56.92] yes 
 rac2[192.168.56.72] rac2[192.168.56.93] yes 
 rac2[192.168.56.72] rac2[192.168.56.82] yes 
 rac2[192.168.56.72] rac3[192.168.56.73] yes 
 rac2[192.168.56.92] rac2[192.168.56.93] yes 
 rac2[192.168.56.92] rac2[192.168.56.82] yes 
 rac2[192.168.56.92] rac3[192.168.56.73] yes 
 rac2[192.168.56.93] rac2[192.168.56.82] yes 
 rac2[192.168.56.93] rac3[192.168.56.73] yes 
 rac2[192.168.56.82] rac3[192.168.56.73] yes 
Result: Node connectivity passed for interface "eth0"

Check: TCP connectivity of subnet "192.168.56.0"
 Source Destination Connected? 
 ------------------------------ ------------------------------ ----------------
 rac1:192.168.56.71 rac1:192.168.56.81 passed 
 rac1:192.168.56.71 rac1:192.168.56.91 passed 
 rac1:192.168.56.71 rac2:192.168.56.72 passed 
 rac1:192.168.56.71 rac2:192.168.56.92 passed 
 rac1:192.168.56.71 rac2:192.168.56.93 passed 
 rac1:192.168.56.71 rac2:192.168.56.82 passed 
 rac1:192.168.56.71 rac3:192.168.56.73 passed 
Result: TCP connectivity check passed for subnet "192.168.56.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Checking CRS integrity...
Clusterware version consistency passed
The Oracle Clusterware is healthy on node "rac1"
The Oracle Clusterware is healthy on node "rac2"
CRS integrity check passed
Checking shared resources...
Checking CRS home location...
"/u01/app/11.2.0/grid" is shared
Result: Shared resources check for node addition passed

Checking node connectivity...
Checking hosts config file...
 Node Name Status 
 ------------------------------------ ------------------------
 rac1 passed 
 rac2 passed 
 rac3 passed
Verification of the hosts config file successful

Interface information for node "rac1"
 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0 192.168.56.71 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.81 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.91 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth1 192.168.10.1 192.168.10.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth1 169.254.149.126 169.254.0.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth2 192.168.1.8 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:94:63:1F 1500

Interface information for node "rac2"
 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0 192.168.56.72 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.92 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.93 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth0 192.168.56.82 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth1 192.168.10.2 192.168.10.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth1 169.254.184.13 169.254.0.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth2 192.168.1.7 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:94:63:1F 1500

Interface information for node "rac3"
 Name IP Address Subnet Gateway Def. Gateway HW Address MTU 
 ------ --------------- --------------- --------------- --------------- ----------------- ------
 eth0 192.168.56.73 192.168.56.0 0.0.0.0 192.168.1.1 08:00:27:CF:4E:CA 1500 
 eth1 192.168.10.3 192.168.10.0 0.0.0.0 192.168.1.1 08:00:27:3D:56:40 1500 
 eth2 192.168.1.9 192.168.1.0 0.0.0.0 192.168.1.1 08:00:27:94:63:1F 1500

Check: Node connectivity for interface "eth0"
 Source Destination Connected? 
 ------------------------------ ------------------------------ ----------------
 rac1[192.168.56.71] rac1[192.168.56.81] yes 
 rac1[192.168.56.71] rac1[192.168.56.91] yes 
 rac1[192.168.56.71] rac2[192.168.56.72] yes 
 rac1[192.168.56.71] rac2[192.168.56.92] yes 
 rac1[192.168.56.71] rac2[192.168.56.93] yes 
 rac1[192.168.56.71] rac2[192.168.56.82] yes 
 rac1[192.168.56.71] rac3[192.168.56.73] yes 
 rac1[192.168.56.81] rac1[192.168.56.91] yes 
 rac1[192.168.56.81] rac2[192.168.56.72] yes 
 rac1[192.168.56.81] rac2[192.168.56.92] yes 
 rac1[192.168.56.81] rac2[192.168.56.93] yes 
 rac1[192.168.56.81] rac2[192.168.56.82] yes 
 rac1[192.168.56.81] rac3[192.168.56.73] yes 
 rac1[192.168.56.91] rac2[192.168.56.72] yes 
 rac1[192.168.56.91] rac2[192.168.56.92] yes 
 rac1[192.168.56.91] rac2[192.168.56.93] yes 
 rac1[192.168.56.91] rac2[192.168.56.82] yes 
 rac1[192.168.56.91] rac3[192.168.56.73] yes 
 rac2[192.168.56.72] rac2[192.168.56.92] yes 
 rac2[192.168.56.72] rac2[192.168.56.93] yes 
 rac2[192.168.56.72] rac2[192.168.56.82] yes 
 rac2[192.168.56.72] rac3[192.168.56.73] yes 
 rac2[192.168.56.92] rac2[192.168.56.93] yes 
 rac2[192.168.56.92] rac2[192.168.56.82] yes 
 rac2[192.168.56.92] rac3[192.168.56.73] yes 
 rac2[192.168.56.93] rac2[192.168.56.82] yes 
 rac2[192.168.56.93] rac3[192.168.56.73] yes 
 rac2[192.168.56.82] rac3[192.168.56.73] yes 
Result: Node connectivity passed for interface "eth0"

Check: TCP connectivity of subnet "192.168.56.0"
 Source Destination Connected? 
 ------------------------------ ------------------------------ ----------------
 rac1:192.168.56.71 rac1:192.168.56.81 passed 
 rac1:192.168.56.71 rac1:192.168.56.91 passed 
 rac1:192.168.56.71 rac2:192.168.56.72 passed 
 rac1:192.168.56.71 rac2:192.168.56.92 passed 
 rac1:192.168.56.71 rac2:192.168.56.93 passed 
 rac1:192.168.56.71 rac2:192.168.56.82 passed 
 rac1:192.168.56.71 rac3:192.168.56.73 passed 
Result: TCP connectivity check passed for subnet "192.168.56.0"

Check: Node connectivity for interface "eth1"
 Source Destination Connected? 
 ------------------------------ ------------------------------ ----------------
 rac1[192.168.10.1] rac2[192.168.10.2] yes 
 rac1[192.168.10.1] rac3[192.168.10.3] yes 
 rac2[192.168.10.2] rac3[192.168.10.3] yes 
Result: Node connectivity passed for interface "eth1"

Check: TCP connectivity of subnet "192.168.10.0"
 Source Destination Connected? 
 ------------------------------ ------------------------------ ----------------
 rac1:192.168.10.1 rac2:192.168.10.2 passed 
 rac1:192.168.10.1 rac3:192.168.10.3 passed 
Result: TCP connectivity check passed for subnet "192.168.10.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "192.168.56.0".
Subnet mask consistency check passed for subnet "192.168.10.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.56.0" for multicast communication with multicast group "230.0.1.0" passed.
Checking subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0"...
Check of subnet "192.168.10.0" for multicast communication with multicast group "230.0.1.0" passed.
Check of multicast communication passed.
Check: Total memory 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 2.9401GB (3082932.0KB) 1.5GB (1572864.0KB) passed 
 rac3 2.9401GB (3082932.0KB) 1.5GB (1572864.0KB) passed 
Result: Total memory check passed
Check: Available memory 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 1.8159GB (1904084.0KB) 50MB (51200.0KB) passed 
 rac3 2.7753GB (2910084.0KB) 50MB (51200.0KB) passed 
Result: Available memory check passed
Check: Swap space 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 2.9531GB (3096572.0KB) 2.9401GB (3082932.0KB) passed 
 rac3 2.9531GB (3096572.0KB) 2.9401GB (3082932.0KB) passed 
Result: Swap space check passed
Check: Free disk space for "rac1:/u01/app/11.2.0/grid,rac1:/tmp" 
 Path Node Name Mount point Available Required Status 
 ---------------- ------------ ------------ ------------ ------------ ------------
 /u01/app/11.2.0/grid rac1 / 25.9531GB 7.5GB passed 
 /tmp rac1 / 25.9531GB 7.5GB passed 
Result: Free disk space check passed for "rac1:/u01/app/11.2.0/grid,rac1:/tmp"
Check: Free disk space for "rac3:/u01/app/11.2.0/grid,rac3:/tmp" 
 Path Node Name Mount point Available Required Status 
 ---------------- ------------ ------------ ------------ ------------ ------------
 /u01/app/11.2.0/grid rac3 / 35.8145GB 7.5GB passed 
 /tmp rac3 / 35.8145GB 7.5GB passed 
Result: Free disk space check passed for "rac3:/u01/app/11.2.0/grid,rac3:/tmp"
Check: User existence for "oracle" 
 Node Name Status Comment 
 ------------ ------------------------ ------------------------
 rac1 passed exists(1100) 
 rac3 passed exists(1100)
Checking for multiple users with UID value 1100
Result: Check for multiple users with UID value 1100 passed 
Result: User existence check passed for "oracle"
Check: Run level 
 Node Name run level Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 5 3,5 passed 
 rac3 5 3,5 passed 
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors" 
 Node Name Type Available Required Status 
 ---------------- ------------ ------------ ------------ ----------------
 rac1 hard 65536 65536 passed 
 rac3 hard 65536 65536 passed 
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors" 
 Node Name Type Available Required Status 
 ---------------- ------------ ------------ ------------ ----------------
 rac1 soft 4096 1024 passed 
 rac3 soft 4096 1024 passed 
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes" 
 Node Name Type Available Required Status 
 ---------------- ------------ ------------ ------------ ----------------
 rac1 hard 16384 16384 passed 
 rac3 hard 16384 16384 passed 
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes" 
 Node Name Type Available Required Status 
 ---------------- ------------ ------------ ------------ ----------------
 rac1 soft 2047 2047 passed 
 rac3 soft 2047 2047 passed 
Result: Soft limits check passed for "maximum user processes"
Check: System architecture 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 x86_64 x86_64 passed 
 rac3 x86_64 x86_64 passed 
Result: System architecture check passed
Check: Kernel version 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 3.8.13-16.3.1.el6uek.x86_64 2.6.32 passed 
 rac3 3.8.13-16.3.1.el6uek.x86_64 2.6.32 passed 
Result: Kernel version check passed
Check: Kernel parameter for "semmsl" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 250 250 250 passed 
 rac3 250 250 250 passed 
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 32000 32000 32000 passed 
 rac3 32000 32000 32000 passed 
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 100 100 100 passed 
 rac3 100 100 100 passed 
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 128 128 128 passed 
 rac3 128 128 128 passed 
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 68719476736 68719476736 1578461184 passed 
 rac3 68719476736 68719476736 1578461184 passed 
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 4096 4096 4096 passed 
 rac3 4096 4096 4096 passed 
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 4294967296 4294967296 2097152 passed 
 rac3 4294967296 4294967296 2097152 passed 
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 6815744 6815744 6815744 passed 
 rac3 6815744 6815744 6815744 passed 
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed 
 rac3 between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 & 65500.0 passed 
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 262144 262144 262144 passed 
 rac3 262144 262144 262144 passed 
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 4194304 4194304 4194304 passed 
 rac3 4194304 4194304 4194304 passed 
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 262144 262144 262144 passed 
 rac3 262144 262144 262144 passed 
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 1048586 1048586 1048576 passed 
 rac3 1048586 1048586 1048576 passed 
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr" 
 Node Name Current Configured Required Status Comment 
 ---------------- ------------ ------------ ------------ ------------ ------------
 rac1 1048576 1048576 1048576 passed 
 rac3 1048576 1048576 1048576 passed 
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "binutils" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed 
 rac3 binutils-2.20.51.0.2-5.36.el6 binutils-2.20.51.0.2 passed 
Result: Package existence check passed for "binutils"
Check: Package existence for "compat-libcap1" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 compat-libcap1-1.10-1 compat-libcap1-1.10 passed 
 rac3 compat-libcap1-1.10-1 compat-libcap1-1.10 passed 
Result: Package existence check passed for "compat-libcap1"
Check: Package existence for "compat-libstdc++-33(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed 
 rac3 compat-libstdc++-33(x86_64)-3.2.3-69.el6 compat-libstdc++-33(x86_64)-3.2.3 passed 
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "libgcc(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 libgcc(x86_64)-4.4.7-4.el6 libgcc(x86_64)-4.4.4 passed 
 rac3 libgcc(x86_64)-4.4.7-4.el6 libgcc(x86_64)-4.4.4 passed 
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 libstdc++(x86_64)-4.4.7-4.el6 libstdc++(x86_64)-4.4.4 passed 
 rac3 libstdc++(x86_64)-4.4.7-4.el6 libstdc++(x86_64)-4.4.4 passed 
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 libstdc++-devel(x86_64)-4.4.7-4.el6 libstdc++-devel(x86_64)-4.4.4 passed 
 rac3 libstdc++-devel(x86_64)-4.4.7-4.el6 libstdc++-devel(x86_64)-4.4.4 passed 
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 sysstat-9.0.4-22.el6 sysstat-9.0.4 passed 
 rac3 sysstat-9.0.4-22.el6 sysstat-9.0.4 passed 
Result: Package existence check passed for "sysstat"
Check: Package existence for "gcc" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 gcc-4.4.7-4.el6 gcc-4.4.4 passed 
 rac3 gcc-4.4.7-4.el6 gcc-4.4.4 passed 
Result: Package existence check passed for "gcc"
Check: Package existence for "gcc-c++" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 gcc-c++-4.4.7-4.el6 gcc-c++-4.4.4 passed 
 rac3 gcc-c++-4.4.7-4.el6 gcc-c++-4.4.4 passed 
Result: Package existence check passed for "gcc-c++"
Check: Package existence for "ksh" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 ksh-20120801-10.el6 ksh-20100621 passed 
 rac3 ksh-20120801-10.el6 ksh-20100621 passed 
Result: Package existence check passed for "ksh"
Check: Package existence for "make" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 make-3.81-20.el6 make-3.81 passed 
 rac3 make-3.81-20.el6 make-3.81 passed 
Result: Package existence check passed for "make"
Check: Package existence for "glibc(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 glibc(x86_64)-2.12-1.132.el6 glibc(x86_64)-2.12 passed 
 rac3 glibc(x86_64)-2.12-1.132.el6 glibc(x86_64)-2.12 passed 
Result: Package existence check passed for "glibc(x86_64)"
Check: Package existence for "glibc-devel(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 glibc-devel(x86_64)-2.12-1.132.el6 glibc-devel(x86_64)-2.12 passed 
 rac3 glibc-devel(x86_64)-2.12-1.132.el6 glibc-devel(x86_64)-2.12 passed 
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "libaio(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed 
 rac3 libaio(x86_64)-0.3.107-10.el6 libaio(x86_64)-0.3.107 passed 
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "libaio-devel(x86_64)" 
 Node Name Available Required Status 
 ------------ ------------------------ ------------------------ ----------
 rac1 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed 
 rac3 libaio-devel(x86_64)-0.3.107-10.el6 libaio-devel(x86_64)-0.3.107 passed 
Result: Package existence check passed for "libaio-devel(x86_64)"
Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed
Check: Current group ID 
Result: Current group ID check passed
Starting check for consistency of primary group of root user
 Node Name Status 
 ------------------------------------ ------------------------
 rac1 passed 
 rac3 passed
Check for consistency of root user's primary group passed
Checking OCR integrity...
OCR integrity check passed
Checking Oracle Cluster Voting Disk configuration...
Oracle Cluster Voting Disk configuration check passed
Check: Time zone consistency 
Result: Time zone consistency check passed
Starting Clock synchronization checks using Network Time Protocol(NTP)...
NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
 Node Name Running? 
 ------------------------------------ ------------------------
 rac1 no 
 rac3 yes 
Result: Liveness check failed for "ntpd"
PRVF-5508 : NTP configuration file is present on at least one node on which NTP daemon or service is not running.
Result: Clock synchronization check using Network Time Protocol(NTP) failed

Checking to make sure user "oracle" is not in "root" group
 Node Name Status Comment 
 ------------ ------------------------ ------------------------
 rac1 passed does not exist 
 rac3 passed does not exist 
Result: User "oracle" is not part of "root" group. Check passed
Checking consistency of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...
domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "localdomain" as found on node "rac1"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
 Node Name Status 
 ------------------------------------ ------------------------
 rac1 failed 
 rac3 failed 
PRVF-5636 : The DNS response time for an unreachable node exceeded "15000" ms on following nodes: rac3
File "/etc/resolv.conf" is not consistent across nodes

Pre-check for node addition was unsuccessful on all the nodes. 
[root@rac1 Desktop]#

The above errors can be ignored

[oracle@rac1 bin]$ export IGNORE_PREADDNODE_CHECKS=Y
[oracle@rac1 bin]$ export ORACLE_HOME=/u01/app/11.2.0/grid
[oracle@rac1 bin]$ d $ORACLE_HOME/oui/bin
-bash: d: command not found
[oracle@rac1 bin]$ cd $ORACLE_HOME/oui/bin
[oracle@rac1 bin]$ ./addNode.sh -silent “CLUSTER_NEW_NODES={rac3}” “CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac3-vip}” “CLUSTER_NEW_PRIVATE_NODE_NAMES={rac3-priv}”

27

30

31
32

Run the “root.sh” commands on the new rac3 node as directed

First run /u01/app/oraInventory/orainstRoot.sh

And then root.sh

33

Execute root.sh
34
[root@rac3 grid]# pwd
/u01/app/11.2.0/grid
[root@rac3 grid]# ./root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
 ORACLE_OWNER= oracle
 ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
 Copying oraenv to /usr/local/bin ...
 Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to upstart
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
grep for the services owned by oracle user on rac3 node
35
Now we will work on Extending the Oracle Database Software on rac3 node

From an existing node i.e rac1 – as the database software owner – run the following command to extend the Oracle database software to the new node “rac3”

[root@rac1 Desktop]# su - oracle
[oracle@rac1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/
[oracle@rac1 ~]$ cd $ORACLE_HOME/oui/bin 
[oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac3}"
38
[root@rac1 Desktop]# su - oracle
[oracle@rac1 ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1/
[oracle@rac1 ~]$ cd $ORACLE_HOME/oui/bin 
[oracle@rac1 bin]$ ./addNode.sh -silent "CLUSTER_NEW_NODES={rac3}"
Performing pre-checks for node addition
Checking node reachability...
Node reachability check passed from node "rac1"

Checking user equivalence...
User equivalence check passed for user "oracle"
WARNING: 
Node "rac3" already appears to be part of cluster
Pre-check for node addition was successful. 
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 3013 MB Passed
Oracle Universal Installer, Version 11.2.0.3.0 Production
Copyright (C) 1999, 2011, Oracle. All rights reserved.

Performing tests to see whether nodes rac2,rac3 are available
............................................................... 100% Done.
......
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
 Source: /u01/app/oracle/product/11.2.0/dbhome_1
 New Nodes
Space Requirements
 New Nodes
 rac3
 /: Required 4.09GB : Available 30.24GB
Installed Products
 Product Names
 Oracle Database 11g 11.2.0.3.0 
 Sun JDK 1.5.0.30.03 
 Installer SDK Component 11.2.0.3.0 
 Oracle One-Off Patch Installer 11.2.0.1.7 
 Oracle Universal Installer 11.2.0.3.0 
 Oracle USM Deconfiguration 11.2.0.3.0 
 Oracle Configuration Manager Deconfiguration 10.3.1.0.0 
 Oracle DBCA Deconfiguration 11.2.0.3.0 
 Oracle RAC Deconfiguration 11.2.0.3.0 
 Oracle Database Deconfiguration 11.2.0.3.0 
 Oracle Configuration Manager Client 10.3.2.1.0 
 Oracle Configuration Manager 10.3.5.0.1 
 Oracle ODBC Driverfor Instant Client 11.2.0.3.0 
 LDAP Required Support Files 11.2.0.3.0 
 SSL Required Support Files for InstantClient 11.2.0.3.0 
 Bali Share 1.1.18.0.0 
 Oracle Extended Windowing Toolkit 3.4.47.0.0 
 Oracle JFC Extended Windowing Toolkit 4.2.36.0.0 
 Oracle Real Application Testing 11.2.0.3.0 
 Oracle Database Vault J2EE Application 11.2.0.3.0 
 Oracle Label Security 11.2.0.3.0 
 Oracle Data Mining RDBMS Files 11.2.0.3.0 
 Oracle OLAP RDBMS Files 11.2.0.3.0 
 Oracle OLAP API 11.2.0.3.0 
 Platform Required Support Files 11.2.0.3.0 
 Oracle Database Vault option 11.2.0.3.0 
 Oracle RAC Required Support Files-HAS 11.2.0.3.0 
 SQL*Plus Required Support Files 11.2.0.3.0 
 Oracle Display Fonts 9.0.2.0.0 
 Oracle Ice Browser 5.2.3.6.0 
 Oracle JDBC Server Support Package 11.2.0.3.0 
 Oracle SQL Developer 11.2.0.3.0 
 Oracle Application Express 11.2.0.3.0 
 XDK Required Support Files 11.2.0.3.0 
 RDBMS Required Support Files for Instant Client 11.2.0.3.0 
 SQLJ Runtime 11.2.0.3.0 
 Database Workspace Manager 11.2.0.3.0 
 RDBMS Required Support Files Runtime 11.2.0.3.0 
 Oracle Globalization Support 11.2.0.3.0 
 Exadata Storage Server 11.2.0.1.0 
 Provisioning Advisor Framework 10.2.0.4.3 
 Enterprise Manager Database Plugin -- Repository Support 11.2.0.3.0 
 Enterprise Manager Repository Core Files 10.2.0.4.4 
 Enterprise Manager Database Plugin -- Agent Support 11.2.0.3.0 
 Enterprise Manager Grid Control Core Files 10.2.0.4.4 
 Enterprise Manager Common Core Files 10.2.0.4.4 
 Enterprise Manager Agent Core Files 10.2.0.4.4 
 RDBMS Required Support Files 11.2.0.3.0 
 regexp 2.1.9.0.0 
 Agent Required Support Files 10.2.0.4.3 
 Oracle 11g Warehouse Builder Required Files 11.2.0.3.0 
 Oracle Notification Service (eONS) 11.2.0.3.0 
 Oracle Text Required Support Files 11.2.0.3.0 
 Parser Generator Required Support Files 11.2.0.3.0 
 Oracle Database 11g Multimedia Files 11.2.0.3.0 
 Oracle Multimedia Java Advanced Imaging 11.2.0.3.0 
 Oracle Multimedia Annotator 11.2.0.3.0 
 Oracle JDBC/OCI Instant Client 11.2.0.3.0 
 Oracle Multimedia Locator RDBMS Files 11.2.0.3.0 
 Precompiler Required Support Files 11.2.0.3.0 
 Oracle Core Required Support Files 11.2.0.3.0 
 Sample Schema Data 11.2.0.3.0 
 Oracle Starter Database 11.2.0.3.0 
 Oracle Message Gateway Common Files 11.2.0.3.0 
 Oracle XML Query 11.2.0.3.0 
 XML Parser for Oracle JVM 11.2.0.3.0 
 Oracle Help For Java 4.2.9.0.0 
 Installation Plugin Files 11.2.0.3.0 
 Enterprise Manager Common Files 10.2.0.4.3 
 Expat libraries 2.0.1.0.1 
 Deinstallation Tool 11.2.0.3.0 
 Oracle Quality of Service Management (Client) 11.2.0.3.0 
 Perl Modules 5.10.0.0.1 
 JAccelerator (COMPANION) 11.2.0.3.0 
 Oracle Containers for Java 11.2.0.3.0 
 Perl Interpreter 5.10.0.0.2 
 Oracle Net Required Support Files 11.2.0.3.0 
 Secure Socket Layer 11.2.0.3.0 
 Oracle Universal Connection Pool 11.2.0.3.0 
 Oracle JDBC/THIN Interfaces 11.2.0.3.0 
 Oracle Multimedia Client Option 11.2.0.3.0 
 Oracle Java Client 11.2.0.3.0 
 Character Set Migration Utility 11.2.0.3.0 
 Oracle Code Editor 1.2.1.0.0I 
 PL/SQL Embedded Gateway 11.2.0.3.0 
 OLAP SQL Scripts 11.2.0.3.0 
 Database SQL Scripts 11.2.0.3.0 
 Oracle Locale Builder 11.2.0.3.0 
 Oracle Globalization Support 11.2.0.3.0 
 SQL*Plus Files for Instant Client 11.2.0.3.0 
 Required Support Files 11.2.0.3.0 
 Oracle Database User Interface 2.2.13.0.0 
 Oracle ODBC Driver 11.2.0.3.0 
 Oracle Notification Service 11.2.0.3.0 
 XML Parser for Java 11.2.0.3.0 
 Oracle Security Developer Tools 11.2.0.3.0 
 Oracle Wallet Manager 11.2.0.3.0 
 Cluster Verification Utility Common Files 11.2.0.3.0 
 Oracle Clusterware RDBMS Files 11.2.0.3.0 
 Oracle UIX 2.2.24.6.0 
 Enterprise Manager plugin Common Files 11.2.0.3.0 
 HAS Common Files 11.2.0.3.0 
 Precompiler Common Files 11.2.0.3.0 
 Installation Common Files 11.2.0.3.0 
 Oracle Help for the Web 2.0.14.0.0 
 Oracle LDAP administration 11.2.0.3.0 
 Buildtools Common Files 11.2.0.3.0 
 Assistant Common Files 11.2.0.3.0 
 Oracle Recovery Manager 11.2.0.3.0 
 PL/SQL 11.2.0.3.0 
 Generic Connectivity Common Files 11.2.0.3.0 
 Oracle Database Gateway for ODBC 11.2.0.3.0 
 Oracle Programmer 11.2.0.3.0 
 Oracle Database Utilities 11.2.0.3.0 
 Enterprise Manager Agent 10.2.0.4.3 
 SQL*Plus 11.2.0.3.0 
 Oracle Netca Client 11.2.0.3.0 
 Oracle Multimedia Locator 11.2.0.3.0 
 Oracle Call Interface (OCI) 11.2.0.3.0 
 Oracle Multimedia 11.2.0.3.0 
 Oracle Net 11.2.0.3.0 
 Oracle XML Development Kit 11.2.0.3.0 
 Database Configuration and Upgrade Assistants 11.2.0.3.0 
 Oracle JVM 11.2.0.3.0 
 Oracle Advanced Security 11.2.0.3.0 
 Oracle Internet Directory Client 11.2.0.3.0 
 Oracle Enterprise Manager Console DB 11.2.0.3.0 
 HAS Files for DB 11.2.0.3.0 
 Oracle Net Listener 11.2.0.3.0 
 Oracle Text 11.2.0.3.0 
 Oracle Net Services 11.2.0.3.0 
 Oracle Database 11g 11.2.0.3.0 
 Oracle OLAP 11.2.0.3.0 
 Oracle Spatial 11.2.0.3.0 
 Oracle Partitioning 11.2.0.3.0 
 Enterprise Edition Options 11.2.0.3.0 
-----------------------------------------------------------------------------

Instantiating scripts for add node (Tuesday, December 24, 2013 2:22:26 PM IST)
. 1% Done.
Instantiation of add node scripts complete
Copying to remote nodes (Tuesday, December 24, 2013 2:22:40 PM IST)
............................................................................................... 96% Done.
Home copied to new nodes
Saving inventory on nodes (Tuesday, December 24, 2013 2:36:10 PM IST)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each new cluster node. Each script in the list below is followed by a list of nodes.
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh #On nodes rac3
To execute the configuration scripts:
 1. Open a terminal window
 2. Log in as "root"
 3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.
39 40
41

Now run root.sh on rac3 node

42

Add Instance to Clustered Database
A database instance will be established on the new node. Specifically, an instance named “orcl3” will be added to “orcl” – a pre-existing clustered database.
Satisfy Node Instance Dependencies
Satisfy all node instance dependencies, such as passwordfile, init.ora parameters, etc.

From the new node “rac3”, run the following commands to create the passwordfile, “init.ora” file, and “oratab” entry for the new instance

[root@rac3 Desktop]# su - oracle
[oracle@rac3 ~]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
[oracle@rac3 ~]$ cd $ORACLE_HOME/dbs
[oracle@rac3 dbs]$ ls
hc_orcl1.dat  init.ora  initorcl1.ora  orapworcl1
[oracle@rac3 dbs]$ mv initorcl1.ora initorcl3.ora
[oracle@rac3 dbs]$ mv orapworcl1 orapworcl3
[oracle@rac3 dbs]$ echo "orcl3:$ORACLE_HOME:N" >> /etc/oratab
[oracle@rac3 dbs]$ 
43

From a node with an existing instance of “orcl” issue the following commands to create the needed public log thread, undo tablespace, and “init.ora” entries for the new instance


From RAC1 node


[oracle@rac1 bin]$ ps -ef | grep smon


root      2876     1  4 12:39 ?        00:05:59 /u01/app/11.2.0/grid/bin/osysmond.bin


oracle    3137     1  0 12:40 ?        00:00:00 asm_smon_+ASM1


oracle    3495     1  0 12:41 ?        00:00:03 ora_smon_orcl1


oracle   11417 10077  0 14:50 pts/2    00:00:00 grep smon


[oracle@rac1 bin]$ export ORACLE_SID=orcl1


[oracle@rac1 bin]$ export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1


[oracle@rac1 bin]$ export PATH=$ORACLE_HOME/bin:$PATH


[oracle@rac1 bin]$ sqlplus ‘/as sysdba’


SQL*Plus: Release 11.2.0.3.0 Production on Tue Dec 24 14:51:32 2013


Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:


Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production


With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,


Data Mining and Real Application Testing options


SQL>

44

SQL> alter database add logfile thread 3 group 5 ('+DATA') size 50M, group 6 ('+DATA') size 50M; 

Database altered. 

SQL> alter database enable public thread 3; 

Database altered. 

SQL> create undo tablespace undotbs3 datafile '+DATA' size 200M autoextend on; 

Tablespace created. 

SQL> alter system set undo_tablespace='undotbs3' scope=spfile sid='orcl3'; 

System altered. 

SQL> alter system set instance_number=3 scope=spfile sid='orcl3'; 

System altered. 

SQL> alter system set cluster_database_instances=3 scope=spfile sid='*'; 

System altered.
 45

Update Oracle Cluster Registry (OCR)
The OCR will be updated to account for a new instance – “orcl3” – being added to the “orcl” cluster database. Add “orcl3” instance to the “orcl” database and verify

46

Start the Instance
Now that all the prerequisites have been satisfied and OCR updated, the “orcl3” instance will be started. Start the newly created instance – “orcl3” – and verify

47

SQL> col host_name format a11 
SQL> set line 300 
SQL> select INSTANCE_NAME,HOST_NAME,VERSION,STARTUP_TIME,STATUS,ACTIVE_STATE,INSTANCE_ROLE,DATABASE_STATUS from gv$INSTANCE;
INSTANCE_NAME HOST_NAME VERSION STARTUP_TIME STATUS ACTIVE_ST INSTANCE_ROLE DATABASE_STATUS 
---------------- ----------- ----------------- ------------------- ------------ --------- ------------------ ----------------- 
orcl3 rac3.localdomain 11.2.0.3.0 2013-01-05 11:23:16 OPEN NORMAL PRIMARY_INSTANCE ACTIVE 
orcl1 rac1.localdomain 11.2.0.3.0 2013-01-05 09:53:24 OPEN NORMAL PRIMARY_INSTANCE ACTIVE 
orcl2 rac2.localdomain 11.2.0.3.0 2013-01-04 17:34:40 OPEN NORMAL PRIMARY_INSTANCE ACTIVE 48
49

Posted December 24, 2013 by balaoracledba.com in OracleRAC

Step by Step Install of Oracle RAC 11gR2 on OEL 6.5 on Oracle VM Virtual BOX   57 comments

Here I am listing out the steps to install 11gR2 RAC on OS OEL6.5 on Oracle VM Virtual BOX.

Step1:

Create new Virtual BOX VM. Click on new and provide Name as rac1 type as Linux and version as Oracle 64bit. Click Next

01

Step2: Set Base Memory to 3072 MB, then click the “Next” button

02

Step3:Accept the default option to create a new virtual hard disk by clicking the “Create” button

03

Step:4  Select the default VMDK type and click the “Next” button on the Create Virtual Hard Drive screen

04

Step5:  Select Dynamically Allocated and select option “split into files of less than 2gb” Click next

05

Step6: Provide the location where you want to save the VM files. In my case I have provided path as below. Also increase the size of your vm from default to say 50GB and  Click “Create”

06

Once we click on Create we get rac1 vm created as highlighted below:

07

Step7:Click on General->Advanced and change shared clipboard to Bidirectional and Change Drag ‘n’ Drop to Bidirectional and click Ok

08

Step8: Click on Network

09

Add three Adaptors.

Make sure “Adapter 1” is enabled, attached to “Internal Network” or “Host-only Adapter”. This interface will be used for public network, for example, for connection to the RAC datbase from other applications. More about networking will be explained later. On the screenshot below “Internal Network” is selected and name “pubnet” was given to this network

10

Then click on the “Adapter 2” tab. Make sure “Adapter 2” is enabled and attach to “Internal Network”. Name this network “privnet”:

11

Then finally click on the “Adapter 3”, enable it, and attach to “Bridged Adapter” or “NAT”. This adapter will be used for internet. Then press “OK” button

12

Step:9 Click on storage to load OEL6.5 iso

13

Step10: Select OEl6.5 ISO and click ok

14

Step11 Click on Shared Folder this will be helpful to get any windows folder mounted on OEL6.5.

15

16

17

18

Now we can see shared folders with 1 value. It Means we are sharing a folder/drive from Host machine to Guest.

Step12: Click on Start as shown below

19

Step13. After starting we get below screen and select Install or Upgrade an Existing System.

20

Step14: Select skip

21

Step15:Scroll Down and click NEXT

22

Step16: Click Next

23

Step17:Click Next

24

Step18: Select basic Storage and click Next

25

Step19: Select Discard any Data

26

Step20: Change the hostname as below(not mandatory that it should be same as below

27

Scroll down and click on “Configure Network”

28

Step 20: Select System eth0 and click on edit

29

Step21: Select Connect Automatically and select IPV4 Setting

30

Step22: Make sure that “Connect automatically” is checked. Select “IPv4 Settings” tab; change Method to “Manual”; Press “Add” and fill Address: 192.168.56.71; Netmask: 255.255.255.0; Gateway: 0.0.0.0. Press “Apply” then done:

31

Step23: Now select System eth1 and click  on edit

32

Step24: In the Network Connections screen select “System eth1” interface, this will be used for private network, then press “Edit”. Then check the box “Connect automatically”. Select “IPv4 Settings” tab; change Method to “Manual”. Press “Add” and fill Address: 192.168.10.1; Netmask: 255.255.255.0; Gateway: 0.0.0.0. When done, press “Apply”:

33

Step25: Finally select “System eth2” interface, this will be used for Internet, then press “Edit”. Check the box “Connect automatically”. Select “IPv4 Settings” tab make sure the Method is set to “Automatic (DHCP)”.Press “Apply” button:

34

35

Step26: Click Next

36

Step27:Give password for root

37

Step28:

38

Step29:Click next

39

Step30:Select on Format

40

Step31:Select Write Changes to disk

41

Step32:Click Next

42

Step33: Select Database Server option and select Customize now and Click Next

43

Step34:Select Tiger VNC (Not Manadatory). I sometimes prefer using vnc

44

Step35:Select options as below for Desktop and Click Next

45

48

49

NOW Sit and Relax.

50

We are done with the installation of OEL 6.5: Click Reboot.

36:After Reboot: Click on Forward

51

37:Licence Info: Click Forward

52

Step38:Click Forward

53

Step39:Select connect later

54

Step40:Click Forward

55

Step41:Click Forward

56

Step42:Select Finish

57

58

Step43:First of all check whether we can connect to the internet or not

59

Change the /etc/hosts entries as below

# Public

192.168.56.71    rac1.localdomain       rac1

192.168.56.72    rac2.localdomain       rac2

# Private

192.168.10.1    rac1-priv.localdomain   rac1-priv

192.168.10.2    rac2-priv.localdomain   rac2-priv

# Virtual

192.168.56.81    rac1-vip.localdomain   rac1-vip

192.168.56.82    rac2-vip.localdomain   rac2-vip

# SCAN

192.168.56.91    rac-scan.localdomain   rac-scan

192.168.56.92    rac-scan.localdomain   rac-scan

192.168.56.93    rac-scan.localdomain   rac-scan

61

Step44: Install Virtual BOX Software

62

Step45:Click Ok

63

Step47:Click on RUN

64

Step48: It is failed. We need to update packages.

65

Please follow steps provided in below link to resolve this issue

https://balaoracledba.com/category/virtualbox/

Once finishing the steps provided in above link Reboot the Server.

Once again try to install VM tools

01

02

Reboot.

Purpose we will be able to view the shared folders in Linux Guest.

Step51: Now we need to disable the firewall: Linux Main menu | System | Administration | Firewall. Click on “Disable” icon, then on “Apply”.

03

Stop and Disable

04

04

Step52: Copy the software from shared folder to tmp and change the permissions to read write execute on zip files

05

Step53: Install oracle-rdbms-server-11gR2-preinstall using yum

06

07

Install ASM packages as below

08

Create group and user

groupadd -g 1000 oinstall
groupadd -g 1200 dba
useradd -u 1100 -g oinstall -G dba oracle
passwd oracle

Passwd oracle

Create Passwd for oracle user

Step54: Create the directory in which the Oracle software will be installed.

mkdir -p  /u01/app/11.2.0.3/grid

mkdir -p /u01/app/oracle/product/11.2.0.3/db_1

chown -R oracle:oinstall /u01

chmod -R 775 /u01/

Step55: Disable secure linux by editing the “/etc/selinux/config” file, making sure the SELINUX flag is set as follows.

SELINUX=disabled

Change Enforcing to disabled

09

Add or amend the following lines to the “/etc/sysctl.conf” file.

fs.aio-max-nr = 1048576

fs.file-max = 6815744

#kernel.shmall = 2097152

#kernel.shmmax = 1054504960

kernel.shmmni = 4096

# semaphores: semmsl, semmns, semopm, semmni

kernel.sem = 250 32000 100 128

net.ipv4.ip_local_port_range = 9000 65500

net.core.rmem_default=262144

net.core.rmem_max=4194304

net.core.wmem_default=262144

net.core.wmem_max=1048586

Run the following command to change the current kernel parameters.

/sbin/sysctl –p

10

Add the following lines to the “/etc/security/limits.conf” file.

oracle              soft    nproc   2047

oracle              hard    nproc   16384

oracle              soft    nofile  4096

oracle              hard    nofile  65536

oracle              soft    stack   10240

# service iptables stop

# chkconfig iptables off

Step57: Either configure NTP, or make sure it is not configured so the Oracle Cluster Time Synchronization Service (ctssd) can synchronize the times of the RAC nodes. If you want to deconfigure NTP do the following.

# service ntpd stop
Shutting down ntpd:                                        [  OK  ]
# chkconfig ntpd off
# mv /etc/ntp.conf /etc/ntp.conf.orig
# rm /var/run/ntpd.pid

If you want to use NTP, you must add the “-x” option into the following line in the “/etc/sysconfig/ntpd” file.

OPTIONS="-x -u ntp:ntp -p /var/run/ntpd.pid"

Then restart NTP.

# service ntpd restart

11
12

Now we will shut down the server as we need to add asm disks. Before we add asm disks. We will backup our vm

Backup VM.

Click File export Appliance

13

14

Select our VM and click next Save to some location

15 15

Click next and click export

16

17

Ok We done with the backup now add disk’s for ASM storage.

TO add disks click on storage

01

02

03

04

06

06 08

Once this is done

Now Restore the Backup that was taken earlier and change it as Rac2

09

10

Change the name to RAC2

11

Scroll down

12

Click Import

After import we got RAC2 VM

13

Change the disks to shared  that were created earlier for ASM

14

15

16

Do the same for all the ASMDISK’s

Now once this is done click on RAC2 Vm and Storage

17

18

19

20

Repeat same for other 2 asm disks

21

Now start the Rac2 server Make sure sure RAC1 IS DOWN

22

The main purpose of starting the server is that we have to configure Network

If we see here it is still rac1. Hostname has to be changed

23

Change the hostsname

24

25

Change the IP addresses

01

02

03

04

Reboot

If there is any problem then restart network services

Service network restart

Start RAC1 and check whether you are able to ping the hosts as below from rac1

06

05

Install Oracleasmlib. I have downloaded file from

http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol6-1709075.html

07

08

09

10

11

12

Do the same with other storage devices i.e for /dev/sdc and /dev/sdd

Configure OracleASM

Configure ASMLib using the following command.

# oracleasm configure -i

13

/usr/sbin/oracleasm init

14

15

On RAC2

16

We are done with everything

now unzip the grid software

1

install cvqdisk* rpm

02

Execute the runInstaller.sh

03

04

05

06

Click on SSH Connectivity..

07

Give the password for grid and click next…

07 08

provide passwords in below screenshot.

09 10 11 12 13 14

Install missed RPM’s

Lets install one after the other using yum

15 16 17 18 19 20 21 22

24

This issue is due to wrong groups used. Change the groups as provided in the screenshot.

25

The other two issues can be ignored.

26 27 28 29 30 31 32 33 34 35 36

We are done with running root.sh on both nodes

37 38

Ignore error regarding SCAN- this can be resolved by performing action plan provide above error information Click Next and finish the installation.

SCAN Issue can be resolved using below provided below

 Install the DAtabase

01 02 03 04 05 06 07 08 09 10 11 12

SCAN Issue can be resolved using below

 [root@rac1]# mv /usr/bin/nslookup /usr/bin/nslookup.original

Next, create a new shell script named /usr/bin/nslookup as shown below while
replacing 24.154.1.34 with your primary DNS, rac-scan with your
SCAN host name, and 192.168.56.91 with your SCAN IP address:
#!/bin/bash
HOSTNAME=${1}
if [[ $HOSTNAME = "racnode-cluster-scan" ]]; then
 echo "Server: 24.154.1.34"
 echo "Address: 24.154.1.34#53"
 echo "Non-authoritative answer:"
 echo "Name: racn-scan"
 echo "Address: 192.168.56.91 "
else
 /usr/bin/nslookup.original $HOSTNAME
fi
Finally, change the new nslookup shell script to executable:
[root@rac1 ~]# chmod 755 /usr/bin/nslookup
Remember to perform these actions on both Oracle RAC nodes.
The new nslookup shell script simply echo's back your SCAN IP address whenever the CVU calls nslookup with your SCAN host name; otherwise, it calls the original nslookup binary.  The CVU will now pass during the Oracle grid infrastructure installation when it attempts to verify your SCAN: 

13 14 15 16 17 18 19 20 21 22 23 24 25 26

This finishes the installation of RAC.

You can create your own profiles for accessing grid and Oracle Database.


Posted December 23, 2013 by balaoracledba.com in OracleRAC

What is oratop   2 comments

OS: OEL 6.4

Oracle Software: 11gR2.

Oratop

oratop is a utility similar to the unix OS utility top which gives a overview of database performance and can be used in combination with top to get a more complete overview of system performance

Linux x86-64

NOTE: oratop can be used to monitor databases on other platforms but the executable must run on Linux with an Oracle client.

To monitor databases on other platforms simply define an alias in tnsnames.ora of the linux client and connect to the databases remotely as you would with sqlplus

Download the oratop executable for 11gR2 on Linux x86-64 here.

Download the oratop executable for 12c on Linux x86-64 for use with 12c clients here.

Download the oratop executable for 11gR2 on Linux x86-32 here.

Download the oratop executable for 12c on Linux x86-32 for use with 12c clients here.

Please download and review the oratop Users Guide for specific instructions on how to configure and run oratop.  A Quick Start Guide is provided in the Instructions section of this Note.

Running oratop

  1. Log in to the system as the Oracle RDBMS software installation owner
  2. Stage the oratop executable on the server on which the tool will be executed.  On a RAC system it is only necessary to stage the executable on one node as it is RAC aware.
  3. Change the name of the executable, eg.
  4. $ mv oratop* oratop
     Validate the permissions for oratop are 755 (-rwxr-xr-x). If the permissions are not currently set to 755, set the permissions on oratop as follows: 
  5. $ chmod 755 oratop
  6. Configure the execution environment as follows (if not already set accordingly)
  7. $ export TERM=xterm  #or vt100

    $ export ORACLE_HOME=<11.2 database home>
    $ export LD_LIBRARY_PATH=$ORACLE_HOME/lib
    $ export PATH=$ORACLE_HOME/bin:$PATH
    $ export ORACLE_SID=<local 11.2 database SID to be monitored>  #only needed if connecting to a local database
  8. Invoke the tool to monitor a local database as follows (interval can be any interval in seconds):
    $ ./oratop -i 10 / as sysdba
  9. Non-dba privileged user requires specific grants to use the tool.  See User Guide for grants required.
  10. Databases can be monitored remotely using a tns alias as described in the oratop Users Guide.
  11. $ ./oratop -i 10 username/password@tns_alias
    $ ./oratop -i 10 system/manager@tns_alias
  12. To exit the program simply press the keyboard key “q”.  To abort, CTRL-C.

33

22

 

In my case nothing is running so i got below screen

2

 

 

 

 

Posted December 17, 2013 by balaoracledba.com in 12cDatabase, Oracle DBA, Oracle10g/11g, OracleRAC

Applying PSU Patch on Two Node RAC GRID Oracle Home.   4 comments

Step-1
Take the backup of Grid Home
cd $GRID_HOME
tar -cvpf /u01/app/11.2.0/grid/gridhome_bkp.tar ./
Perform this Backup on Both the RAC Nodes
Step-2
Download  & Copy Latest OPatch
OPatch is the utility used to apply the patch. Download the latest OPatch utility from My Oracle Support patch 6880880. As software home owner, unzip the downloaded OPatch to the $GRID_HOME and $ORACLE_HOME
1
Take Backup of Original OPatch
Mv OPatch OPatch_bk
Now test for new opatch version
3
Step-3
OCM Configuration
During patch apply, OPatch will prompt for the location of the OCM (Oracle Configuration Manager) response file. If you do not have one, run the following command to create it.
cd /u01/app/11.2.0/grid/OPatch/ocm/bin
 4
Make a note of the response file location as we will need it. Or copy this ocm.rsp /tmp
Step-4
Validation of Oracle Inventory
 Before patch application, let check the consistency of the oracle inventory. If the below command succeeds, we are good.
 > $GRID_HOME/OPatch/opatch lsinventory -detail -oh $GRID_HOME
 5 6 



7



8
One-off Patch Conflict Detection and Resolution
 Check for conflicting patches by doing the following for each GI home you are planning to apply the patch on.
 [grid@rac1 db_grid_psu]$ opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir ./16902043
Oracle Interim Patch Installer version 11.2.0.3.5
Copyright (c) 2013, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/11.2.0/grid/oraInst.loc
OPatch version    : 11.2.0.3.5
OUI version       : 11.2.0.3.0
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-12-09_17-43-37PM_1.log
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.
[grid@rac1 db_grid_psu]$ 


Patch Installation  Note: -- Stop EM agent if it is running on your server before you apply the patch.  

emctl stop agent  

[root@rac1 db_grid_psu]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid 9[root@rac1 ~]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid/ 10 b1 b2 [root@rac1 ~]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid/ Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /tmp -patchn db_grid_psu -oh /u01/app/11.2.0/grid/ -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params /u01/app/11.2.0/grid/crs/install/crsconfig_params /u01/app/11.2.0/grid/crs/install/s_crsconfig_defs This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2013-12-09_19-13-07.log This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2013-12-09_19-13-07.report.log 2013-12-09 19:13:07: Starting Clusterware Patch Setup Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Enter 'yes' if you have unzipped this patch to an empty directory to proceed  (yes/no):yes OPatch  is bundled with OCM, Enter the absolute OCM response file path: /tmp/ocm.rsp CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.crsd' on 'rac1' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac1' CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac1' CRS-2673: Attempting to stop 'ora.prod.db' on 'rac1' CRS-2673: Attempting to stop 'ora.cvu' on 'rac1' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac1' CRS-2673: Attempting to stop 'ora.oc4j' on 'rac1' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac1' CRS-2677: Stop of 'ora.cvu' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.cvu' on 'rac1' CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac1' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.rac1.vip' on 'rac1' CRS-2677: Stop of 'ora.scan1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1' CRS-2677: Stop of 'ora.rac1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.rac1.vip' on 'rac1' CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded CRS-2677: Stop of 'ora.prod.db' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac1' CRS-2676: Start of 'ora.rac1.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1' CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded CRS-2677: Stop of 'ora.oc4j' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'rac1' CRS-2677: Stop of 'ora.FRA.dg' on 'rac1' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2676: Start of 'ora.oc4j' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rac1' CRS-2677: Stop of 'ora.ons' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rac1' CRS-2677: Stop of 'ora.net1.network' on 'rac1' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac1' has completed CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1' CRS-2673: Attempting to stop 'ora.evmd' on 'rac1' CRS-2673: Attempting to stop 'ora.asm' on 'rac1' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1' CRS-2677: Stop of 'ora.evmd' on 'rac1' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac1' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rac1' CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rac1' CRS-2677: Stop of 'ora.crf' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1' CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1' CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac1' has completed CRS-4133: Oracle High Availability Services has been stopped. Successfully unlock /u01/app/11.2.0/grid patch /tmp/db_grid_psu/16902043  apply successful for home  /u01/app/11.2.0/grid CRS-4123: Oracle High Availability Services has been started.  Do the same on rac2 node as well. Before applying PSU patch copy new opatch zip file to GRID Oracle Home  and take the backup of existing OPatch Directory and unzip the p6880880_11200*.zip file under Grid Oracle Home once it is done. rac2 [root@rac2 ~]# /u01/app/11.2.0/grid/OPatch/opatch auto /tmp/db_grid_psu/ -oh /u01/app/11.2.0/grid/ Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/11.2.0/grid/OPatch/crs/patch11203.pl -patchdir /tmp -patchn db_grid_psu -oh /u01/app/11.2.0/grid/ -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params /u01/app/11.2.0/grid/crs/install/crsconfig_params /u01/app/11.2.0/grid/crs/install/s_crsconfig_defs This is the main log file: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2013-12-09_19-13-07.log This file will show your detected configuration and all the steps that opatchauto attempted to do on your system: /u01/app/11.2.0/grid/cfgtoollogs/opatchauto2013-12-09_19-13-07.report.log 2013-12-09 19:13:07: Starting Clusterware Patch Setup Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params Enter 'yes' if you have unzipped this patch to an empty directory to proceed  (yes/no):yes OPatch  is bundled with OCM, Enter the absolute OCM response file path: /tmp/ocm.rsp CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac2' CRS-2673: Attempting to stop 'ora.crsd' on 'rac2' CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'rac2' CRS-2673: Attempting to stop 'ora.FRA.dg' on 'rac2' CRS-2673: Attempting to stop 'ora.prod.db' on 'rac2' CRS-2673: Attempting to stop 'ora.cvu' on 'rac2' CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac2' CRS-2673: Attempting to stop 'ora.oc4j' on 'rac2' CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'rac2' CRS-2677: Stop of 'ora.cvu' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.cvu' on 'rac1' CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac2' CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.rac2.vip' on 'rac2' CRS-2677: Stop of 'ora.scan1.vip' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac1' CRS-2677: Stop of 'ora.rac2.vip' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.rac2.vip' on 'rac1' CRS-2676: Start of 'ora.scan1.vip' on 'rac1' succeeded CRS-2677: Stop of 'ora.prod.db' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.DATA.dg' on 'rac2' CRS-2676: Start of 'ora.rac2.vip' on 'rac1' succeeded CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac1' CRS-2676: Start of 'ora.cvu' on 'rac1' succeeded CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac1' succeeded CRS-2677: Stop of 'ora.oc4j' on 'rac2' succeeded CRS-2672: Attempting to start 'ora.oc4j' on 'rac1' CRS-2677: Stop of 'ora.FRA.dg' on 'rac2' succeeded CRS-2677: Stop of 'ora.DATA.dg' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.asm' on 'rac2' CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded CRS-2676: Start of 'ora.oc4j' on 'rac1' succeeded CRS-2673: Attempting to stop 'ora.ons' on 'rac2' CRS-2677: Stop of 'ora.ons' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.net1.network' on 'rac2' CRS-2677: Stop of 'ora.net1.network' on 'rac2' succeeded CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'rac2' has completed CRS-2677: Stop of 'ora.crsd' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.ctssd' on 'rac2' CRS-2673: Attempting to stop 'ora.evmd' on 'rac2' CRS-2673: Attempting to stop 'ora.asm' on 'rac2' CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac2' CRS-2677: Stop of 'ora.evmd' on 'rac2' succeeded CRS-2677: Stop of 'ora.ctssd' on 'rac2' succeeded CRS-2677: Stop of 'ora.mdnsd' on 'rac2' succeeded CRS-2677: Stop of 'ora.asm' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac2' CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.cssd' on 'rac2' CRS-2677: Stop of 'ora.cssd' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.crf' on 'rac2' CRS-2677: Stop of 'ora.crf' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.gipcd' on 'rac2' CRS-2677: Stop of 'ora.gipcd' on 'rac2' succeeded CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac2' CRS-2677: Stop of 'ora.gpnpd' on 'rac2' succeeded CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac2' has completed CRS-4133: Oracle High Availability Services has been stopped. Successfully unlock /u01/app/11.2.0/grid patch /tmp/db_grid_psu/16902043  apply successful for home  /u01/app/11.2.0/grid CRS-4123: Oracle High Availability Services has been started. Finally I suggest bounce all the RAC nodes and check whether all the services are up or not. Cross check the patch [grid@rac1 ~]$ $GRID_HOME/OPatch/opatch lsinventory Oracle Interim Patch Installer version 11.2.0.3.5 Copyright (c) 2013, Oracle Corporation. All rights reserved. Oracle Home : /u01/app/11.2.0/grid Central Inventory : /u01/app/oraInventory from : /u01/app/11.2.0/grid/oraInst.loc OPatch version : 11.2.0.3.5 OUI version : 11.2.0.3.0 Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-12-11_20-11-30PM_1.log Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2013-12-11_20-11-30PM.txt -------------------------------------------------------------------------------- Installed Top-level Products (1): Oracle Grid Infrastructure 11.2.0.3.0 There are 1 products installed in this Oracle Home. Interim patches (1) : Patch 16902043 : applied on Mon Dec 09 18:27:57 IST 2013 Unique Patch ID: 16676143 Patch description: "Database Patch Set Update : 11.2.0.3.8 (16902043)" Created on 24 Sep 2013, 23:20:58 hrs PST8PDT Sub-patch 16619892; "Database Patch Set Update : 11.2.0.3.7 (16619892)" Sub-patch 16056266; "Database Patch Set Update : 11.2.0.3.6 (16056266)" Sub-patch 14727310; "Database Patch Set Update : 11.2.0.3.5 (14727310)" Sub-patch 14275605; "Database Patch Set Update : 11.2.0.3.4 (14275605)" Sub-patch 13923374; "Database Patch Set Update : 11.2.0.3.3 (13923374)" Sub-patch 13696216; "Database Patch Set Update : 11.2.0.3.2 (13696216)" Sub-patch 13343438; "Database Patch Set Update : 11.2.0.3.1 (13343438)" Bugs fixed: 13593999, 13566938, 10350832, 14138130, 12919564, 14198511, 13561951 13624984, 13588248, 16024441, 13080778, 17333199, 13914613, 13804294 16710324, 14258925, 12873183, 14755945, 13645875, 14472647, 12880299 14664355, 12998795, 14409183, 13719081, 14469008, 13492735, 13496884 12857027, 14263036, 13015379, 14263073, 14188650, 13742433, 13732226 16368108, 16314469, 12905058, 6690853, 13742434, 16212405, 12849688 12950644, 13742435, 13464002, 12899768, 12879027, 13063120, 13534412 13958038, 14613900, 12585543, 12535346, 13790109, 16382448, 12588744 13972394, 11877623, 13072654, 12395918, 13786142, 13814739, 12847466 13649031, 13855490, 13981051, 12582664, 12797765, 17333200, 14262913 17332800, 12923168, 14695377, 13384182, 12912137, 16279401, 13612575 13466801, 13484963, 14207163, 13724193, 13642044, 13772618, 11063191 16694777, 13945708, 13070939, 12797420, 12865902, 13041324, 15869211 13605839, 14003090, 16314467, 16279211, 16314468, 12976376, 12755231 11708510, 13680405, 13742437, 14589750, 13026410, 13737746, 11868640 13742438, 14644185, 13326736, 15841373, 15910002, 13596521, 16362358 14398795, 13001379, 13579992, 16344871, 13099577, 9873405, 13742436 14275605, 9858539, 16372203, 14841812, 16344758, 11715084, 9547706 16231699, 14040433, 9703627, 12662040, 12617123, 16530565, 13483354 14393728, 14207317, 12845115, 12764337, 13354082, 16902043, 14459552 13397104, 14191508, 13913630, 12964067, 12983611, 13550185, 12780983 13810393, 12583611, 14546575, 15862016, 13476583, 13489024, 11840910 13903046, 15862017, 13572659, 16294378, 13718279, 13657605, 14088346 13448206, 16314466, 13419660, 14480676, 13632717, 14668670, 14063281 14110275, 13430938, 13467683, 13420224, 13812031, 14548763, 12646784 16299830, 14512189, 12755116, 17230530, 13616375, 14035825, 13427062 12861463, 12834027, 13092220, 15862021, 13632809, 13377816, 13036331 14727310, 16619892, 13685544, 13499128, 15862018, 16175381, 13584130 12829021, 15862019, 12794305, 14546673, 12791981, 13561750, 13503598 13787482, 10133521, 12744759, 12718090, 13848402, 13399435, 14762511 14023636, 9095696, 13725395, 12401111, 13860201, 12796518, 13257247 13362079, 14176879, 12917230, 13923374, 16014985, 14220725, 12621588 12312133, 13524899, 14480675, 16306019, 13559697, 9706792, 12974860 12940620, 14751895, 14480674, 13916709, 13098318, 14076523, 13773133 16794244, 15905421, 13340388, 13528551, 13366202, 12731940, 12894807 12747437, 13343438, 13454210, 12748240, 14205448, 13385346, 17082364 14127231, 15853081, 14273397, 14467061, 12971775, 13923995, 13582702 14571027, 12784406, 13907462, 12748538, 10242202, 13493847, 13035804 13857111, 16710363, 13544396, 14128555, 13686047, 16382353, 8547978 14226599, 17333203, 14095982, 13591624, 17333197, 14523004, 13440516 9397635, 14007968, 16794241, 13035360, 14062795, 12693626, 12925089 14189694, 12815057, 13332439, 14038787, 13040943, 11071989, 13843646 14062796, 16794243, 12913474, 14841409, 14390252, 13370330, 16314470 14062797, 13059165, 14062794, 12959852, 12345082, 13358781, 17333202 12960925, 16703112, 9659614, 14546638, 13699124, 13936424, 17333198 16794240, 14301592, 13338048, 12938841, 12658411, 12620823, 12656535 14062793, 12678920, 13038684, 14791477, 14062792, 13807411, 16742095 16794238, 15862022, 12594032, 13250244, 12612118, 9761357, 14053457 13742464, 14052474, 13060271, 13911821, 13457582, 13527323, 7509451 13791364, 15862020, 13910420, 12821418, 12780098, 13502183, 13696216 13705338, 14841558, 10263668, 16794242, 15862023, 16056266, 13834065 14351566, 13723052, 16794239, 13554409, 15862024, 13103913, 13645917 14063280, 12772404, 13011409  Rac system comprising of multiple nodes Local node = rac1 Remote node = rac2 -------------------------------------------------------------------------------- OPatch succeeded.

Posted December 11, 2013 by balaoracledba.com in OracleRAC

How to Find OCR Master Node in RAC   Leave a comment

1. The default OCR master is always the first node that’s started in the cluster.

2. When OCR master (crsd.bin process) stops or restarts for whatever reason, the crsd.bin on surviving node with lowest node number will become new OCR master.

The following method can be used to find OCR master:

1. By searching crsd.l* on all nodes:

grep “OCR MASTER” $GRID_HOME/log/$HOST/crsd/crsd.l*

*In my case i have exported ORACLE_HOME to /u01/app/11.2.0/grid

so it is like

grep “OCR MASTER” $ORACLE_HOME/log/$HOST/crsd/crsd.l*

8

If there’s output, the last line will show who OCR master is. If the cluster has been up and running on all nodes for a long while, as crsd.log keeps rotating to crsd.l01, and crsd.l01 to crsd.l02 till crsd.l10, there’s a chance the above grep command will not return anything. If that’s the case, see Step 2.

2. By locating automatic OCR backup:

9

OCR automatic backup is done by OCR master every 4 hours. In above example, latest automatic backup was done around 2013/12/06 15:21:04  by racnode1 which means racnode1 was and may still be OCR master since 2013/12/06 15:21:04 .

Posted December 10, 2013 by balaoracledba.com in OracleRAC

Oracle RAC 11gR2 OCR manual Bakup and adding OCR Mirror device   1 comment

OCR BACKUP

To find OCR is backed up or not I am using ocrconfig –showbackup

1

 

Above screenshot depicts that we have only scheduled backups which were taken on 6th DEC 13. We don’t have fresh manual backup. To Backup manually I use as below

ocrconfig –manual backup

[grid@rac1 ~]$ ocrconfig -manualbackup
PROT-20: Insufficient permission to proceed. Require privileged user
Please note that if we execute backup with grid owner we will face above error. we need to use root to execute ocrconfig -manualbackup

2

Now I want to check all my OCR backups’s,

for this I use ocrconfig –showbackup

3

To see whether OCR is healthy, run an ocrcheck,

# ocrcheck

4

 


 

To add an OCRMIRROR device

In my case I will add +FRA which I configured for Archivelogs

5

 

Lets check in ASM path whether we got a OCR  backup file

6

 

 

Posted December 10, 2013 by balaoracledba.com in OracleRAC

RAC-Enable Archive Log-Configure to ASM Location   Leave a comment

I will List the steps to Enable Archive Log RAC Configured VM’s

Create  folders in ASM location.

a

Initially archives are not enabled

b

1. Login to one of the nodes (i.e. rac1) and disable the cluster instance parameter by setting cluster_database to FALSE from the current instance:

2. $ sqlplus “/ as sysdba”

SQL> alter system set cluster_database=false scope=spfile sid=’prod1′;

c

3. Shutdown all instances accessing the clustered database:

$ srvctl stop database -d prod

d

4.

We start in nomount mode one single instance of the RAC:

[oracle@rac1 ~]$ sqlplus ‘/as sysdba’

SQL> startup nomount;

SQL> alter system set log_archive_start=TRUE SCOPE=SPFILE;

System altered.

SQL> alter system set log_archive_dest_1=’LOCATION=+FRA/prod/archivelog’ SCOPE=SPFILE;

System altered.

e

alter system set cluster_database=true scope=spfile sid=’prod1′;

Now we mount the database instance:

SQL> alter database mount;

Database altered.
Now, we set the database to archive log mode:

SQL> alter database archivelog;
Database altered.
We can perform some checks after that…

srvctl start database -d prod

SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination +FRA/prod/archivelog
Oldest online log sequence 16
Next log sequence to archive 17
Current log sequence 17
SQL> alter system switch logfile;

System altered.

SQL> archive log list
Database log mode Archive Mode
Automatic archival Enabled
Archive destination +FRA/prod/archivelog
Oldest online log sequence 17
Next log sequence to archive 18
Current log sequence 18

SQL> alter system switch logfile;

Now Check

f

Posted December 5, 2013 by balaoracledba.com in OracleRAC