Upgrade 11.2.0.3 Real Application Cluster To 11.2.0.4

Introduction:
Today i will upgrade the clusterware from 11.2.0.3 to 11.2.0.4.

The aim of this project is to upgrade the clusterware in order to
support both 11.2.0.3 and 11.2.0.4 versions.The aim of this
document is not to upgrade database to 11.2.0.4 but to install a new
one. The environment Operating system us linux-86_64bit.
Contents of this document:
Pre upgrade Checks.
Apply the latest patchset update on the cluster 11.2.0.3.
The Upgrade Process.
Errors During upgrade process.
Rollup Upgrade.
Checks after the cluster upgrade.
Patch the new installed clusterware.
Installing a new Oracle Home for the new 11.2.0.4.
Patch the oracle Home with the latest patch update.
Pre upgrade Checks:
1-follow the Oracle note Doc ID 1363369.1 : Things to Consider

Before Upgrading to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM .
2-run "orachk" utility to check your environment for Upgrade
Readiness. Reference: ORAchk 11.2.0.3 Upgrade Readiness
Assessment ( Doc ID 1457357.1 ) .
3- Download the patchset 11.2.0.4 and unzip it.The zipped folder
will have a folder with name grid.
Cd <UNZIPPED PATCH LOCATION>/grid
Run the below command:
uncluvfy.sh stage -pre crsinst -upgrade -n

<racnode1>,<racnode2>,<racnode3> -rolling -src_crshome
/u01/app/11.1 -dest_crshome /u01/app/grid -dest_version
11.2.0.3.0 -fixup -fixupdir /tmp -verbose
where src_crs_home is the current grid Home and dest_crshome is
the new destination location.
Example will be like below:
[grid@CNCHPDB1 grid]$ ./runcluvfy.sh stage -pre crsinst -upgrade -n CNCHPDB1,CNCHPDB2 rolling -src_me /app/grid_11.2.0.3 -dest_crshome /newapp/grid -dest_version 11.2.0.4.0 verbose
Performing pre-checks for cluster services setup
Checking node reachability...
Check: Node reachability from node "CNCHPDB1"
Destination Node
Reachable?
------------------------------------ -----------------------CNCHPDB1
yes
CNCHPDB2
yes
Result: Node reachability check passed from node "CNCHPDB1"
Checking user equivalence...

Check: User equivalence for user "grid"
Node Name
Status
------------------------------------ -----------------------CNCHPDB2
passed
CNCHPDB1
passed
Result: User equivalence check passed for user "grid"
Checking CRS user consistency
Result: CRS user consistency check successful
Checking node connectivity...
Checking hosts config file...
Node Name
Status
------------------------------------ -----------------------CNCHPDB2
passed
CNCHPDB1
passed
Verification of the hosts config file successful
Interface information for node "CNCHPDB2"

Name IP Address Subnet
Gateway
Def. Gateway HW Address
MTU
------ --------------- --------------- --------------- --------------- ----------------- -----bond0 10.203.2.17 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:E0 1500
bond0 10.203.2.22 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:E0 1500
bond0 10.203.2.20 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:E0 1500
bond0 10.203.2.18 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:E0 1500
bond1 172.16.0.2 172.16.0.0 0.0.0.0
10.203.2.1 78:E7:D1:64:8E:B0 1500
bond1 169.254.250.230 169.254.0.0 0.0.0.0
10.203.2.1 78:E7:D1:64:8E:B0 1500
Interface information for node "CNCHPDB1"

Name IP Address Subnet
Gateway
Def. Gateway HW Address
MTU
------ --------------- --------------- --------------- --------------- ----------------- -----bond0 10.203.2.16 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:8E 1500
bond0 10.203.2.21 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:8E 1500
bond0 10.203.2.19 10.203.2.0 0.0.0.0
10.203.2.1 78:E7:D1:F7:3F:8E 1500
bond1 172.16.0.1 172.16.0.0 0.0.0.0
10.203.2.1 78:E7:D1:64:7E:F0 1500
bond1 169.254.13.64 169.254.0.0 0.0.0.0
10.203.2.1 78:E7:D1:64:7E:F0 1500
Check: Node connectivity for interface "bond0"

Source
Destination
Connected?
------------------------------ ------------------------------ ---------------CNCHPDB2[10.203.2.17]
CNCHPDB2[10.203.2.22]
CNCHPDB2[10.203.2.17]
CNCHPDB2[10.203.2.20]
CNCHPDB2[10.203.2.17]
CNCHPDB2[10.203.2.18]
CNCHPDB2[10.203.2.17]
CNCHPDB1[10.203.2.16]
CNCHPDB2[10.203.2.17]
CNCHPDB1[10.203.2.21]
CNCHPDB2[10.203.2.17]
CNCHPDB1[10.203.2.19]
CNCHPDB2[10.203.2.22]
CNCHPDB2[10.203.2.20]
CNCHPDB2[10.203.2.22]
CNCHPDB2[10.203.2.18]
CNCHPDB2[10.203.2.22]
CNCHPDB1[10.203.2.16]
CNCHPDB2[10.203.2.22]
CNCHPDB1[10.203.2.21]
CNCHPDB2[10.203.2.22]
CNCHPDB1[10.203.2.19]
CNCHPDB2[10.203.2.20]
CNCHPDB2[10.203.2.18]
CNCHPDB2[10.203.2.20]
CNCHPDB1[10.203.2.16]
CNCHPDB2[10.203.2.20]
CNCHPDB1[10.203.2.21]
CNCHPDB2[10.203.2.20]
CNCHPDB1[10.203.2.19]
CNCHPDB2[10.203.2.18]
CNCHPDB1[10.203.2.16]
CNCHPDB2[10.203.2.18]
CNCHPDB1[10.203.2.21]
CNCHPDB2[10.203.2.18]
CNCHPDB1[10.203.2.19]
CNCHPDB1[10.203.2.16]
CNCHPDB1[10.203.2.21]
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
yes
CNCHPDB1[10.203.2.16]
CNCHPDB1[10.203.2.19]
CNCHPDB1[10.203.2.21]
CNCHPDB1[10.203.2.19]
Result: Node connectivity passed for interface "bond0"
yes
yes
Check: TCP connectivity of subnet "10.203.2.0"

Source
Destination
Connected?
------------------------------ ------------------------------ ---------------CNCHPDB1:10.203.2.16
CNCHPDB2:10.203.2.17
passed
CNCHPDB1:10.203.2.16
CNCHPDB2:10.203.2.22
passed
CNCHPDB1:10.203.2.16
CNCHPDB2:10.203.2.20
passed
CNCHPDB1:10.203.2.16
CNCHPDB2:10.203.2.18
passed
CNCHPDB1:10.203.2.16
CNCHPDB1:10.203.2.21
passed
CNCHPDB1:10.203.2.16
CNCHPDB1:10.203.2.19
passed
Result: TCP connectivity check passed for subnet "10.203.2.0"
Check: Node connectivity for interface "bond1"

Source
Destination
Connected?
------------------------------ ------------------------------ ---------------CNCHPDB2[172.16.0.2]
CNCHPDB1[172.16.0.1]
yes
Result: Node connectivity passed for interface "bond1"
Check: TCP connectivity of subnet "172.16.0.0"

Source
Destination
Connected?
------------------------------ ------------------------------ ---------------CNCHPDB1:172.16.0.1
CNCHPDB2:172.16.0.2
passed
Result: TCP connectivity check passed for subnet "172.16.0.0"
Checking subnet mask consistency...
Subnet mask consistency check passed for subnet "10.203.2.0".
Subnet mask consistency check passed for subnet "172.16.0.0".
Subnet mask consistency check passed.
Result: Node connectivity check passed
Checking multicast communication...
Checking subnet "10.203.2.0" for multicast communication with multicast group "230.0.1.0"...
PRVG-11134 : Interface "10.203.2.22" on node "CNCHPDB2" is not able to communicate with
interface "1.2.16" on node "CNCHPDB1"

Check of subnet "10.203.2.0" for multicast communication with multicast group "224.0.0.251"
passed.
interface "170.1" on node "CNCHPDB1"
interface "170.2" on node "CNCHPDB2"
passed.
Check of multicast communication passed.
Checking OCR integrity...

OCR integrity check passed
Checking ASMLib configuration.
Node Name
Status
------------------------------------ -----------------------CNCHPDB2
passed
CNCHPDB1
passed
Result: Check for ASMLib configuration passed.
Check: Total memory
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 62.945GB (6.6002576E7KB) 1.5GB (1572864.0KB)
CNCHPDB1 62.945GB (6.6002576E7KB) 1.5GB (1572864.0KB)
Result: Total memory check passed
passed
passed
Check: Available memory

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 55.8544GB (5.8567544E7KB) 50MB (51200.0KB)
CNCHPDB1 55.9689GB (5.8687624E7KB) 50MB (51200.0KB)
Result: Available memory check passed
passed
passed
Check: Swap space

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 23.9987GB (2.5164432E7KB) 16GB (1.6777216E7KB)
CNCHPDB1 23.9987GB (2.5164432E7KB) 16GB (1.6777216E7KB)
Result: Swap space check passed
passed
passed
Check: Free disk space for "CNCHPDB2:/newapp/grid"

Path
Node Name Mount point Available Required Status
---------------- ------------ ------------ ------------ ------------ -----------/newapp/grid CNCHPDB2 /newapp
165.377GB 5.5GB
passed
Result: Free disk space check passed for "CNCHPDB2:/newapp/grid"
Check: Free disk space for "CNCHPDB1:/newapp/grid"
Path
---------------- ------------ ------------ ------------ ------------ -----------/newapp/grid CNCHPDB1 /newapp
164.0928GB 5.5GB
passed
Result: Free disk space check passed for "CNCHPDB1:/newapp/grid"
Check: Free disk space for "CNCHPDB2:/tmp"
Path
Node Name Mount point Available Required
---------------- ------------ ------------ ------------ ------------ ------------
Status
/tmp
CNCHPDB2 /
5.0244GB 1GB
passed
Result: Free disk space check passed for "CNCHPDB2:/tmp"
Check: Free disk space for "CNCHPDB1:/tmp"
Path
---------------- ------------ ------------ ------------ ------------ -----------/tmp
CNCHPDB1 /
4.7936GB 1GB
passed
Result: Free disk space check passed for "CNCHPDB1:/tmp"
Check: User existence for "grid"
Node Name Status
Comment
------------ ------------------------ -----------------------CNCHPDB2 passed
exists(505)
CNCHPDB1 passed
exists(505)
Checking for multiple users with UID value 505
Result: Check for multiple users with UID value 505 passed
Result: User existence check passed for "grid"
Check: Group existence for "oinstall"
Node Name Status
Comment
------------ ------------------------ -----------------------CNCHPDB2 passed
exists
CNCHPDB1 passed
exists
Result: Group existence check passed for "oinstall"
Check: Membership of user "grid" in group "oinstall" [as Primary]
Node Name
User Exists Group Exists User in Group Primary
Status
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
yes
yes
yes
yes
passed
CNCHPDB1
yes
yes
yes
yes
passed
Result: Membership check for user "grid" in group "oinstall" [as Primary] passed
Check: Run level
Node Name run level
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 5
3,5
passed
CNCHPDB1 5
3,5
passed
Result: Run level check passed
Check: Hard limits for "maximum open file descriptors"
Node Name
Type
Available Required Status
---------------- ------------ ------------ ------------ ---------------CNCHPDB2
hard
65536
65536
passed
CNCHPDB1
hard
65536
65536
passed
Result: Hard limits check passed for "maximum open file descriptors"
Check: Soft limits for "maximum open file descriptors"
Node Name
Type
---------------- ------------ ------------ ------------ ---------------CNCHPDB2
soft
1024
1024
passed
CNCHPDB1
soft
1024
1024
passed
Result: Soft limits check passed for "maximum open file descriptors"
Check: Hard limits for "maximum user processes"
Node Name
Type
---------------- ------------ ------------ ------------ ---------------CNCHPDB2
hard
16384
16384
passed
CNCHPDB1
hard
16384
16384
passed
Result: Hard limits check passed for "maximum user processes"
Check: Soft limits for "maximum user processes"
Node Name
Type
---------------- ------------ ------------ ------------ ---------------CNCHPDB2
soft
2047
2047
passed
CNCHPDB1
soft
2047
2047
passed
Result: Soft limits check passed for "maximum user processes"
There are no oracle patches required for home "/app/grid_11.2.0.3".
There are no oracle patches required for home "/newapp/grid".
Check: System architecture
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 x86_64
x86_64
passed
CNCHPDB1 x86_64
x86_64
passed
Result: System architecture check passed
Check: Kernel version
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 2.6.18-194.el5
2.6.18
passed
CNCHPDB1 2.6.18-194.el5
2.6.18
passed
Result: Kernel version check passed
Check: Kernel parameter for "semmsl"
Node Name
Current
Configured Required Status
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
250
250
250
passed
CNCHPDB1
250
250
250
passed
Result: Kernel parameter check passed for "semmsl"
Check: Kernel parameter for "semmns"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ ------------
CNCHPDB2
32000
32000
32000
passed
CNCHPDB1
320000
320000
32000
passed
Result: Kernel parameter check passed for "semmns"
Check: Kernel parameter for "semopm"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
100
100
100
passed
CNCHPDB1
100
100
100
passed
Result: Kernel parameter check passed for "semopm"
Check: Kernel parameter for "semmni"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
128
128
128
passed
CNCHPDB1
128
128
128
passed
Result: Kernel parameter check passed for "semmni"
Check: Kernel parameter for "shmmax"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
68719476736 68719476736 4294967295 passed
CNCHPDB1
68719476736 68719476736 4294967295 passed
Result: Kernel parameter check passed for "shmmax"
Check: Kernel parameter for "shmmni"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
4096
4096
4096
passed
CNCHPDB1
4096
4096
4096
passed
Result: Kernel parameter check passed for "shmmni"
Check: Kernel parameter for "shmall"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
4294967296 4294967296 2097152
passed
CNCHPDB1
4294967296 4294967296 2097152
passed
Result: Kernel parameter check passed for "shmall"
Check: Kernel parameter for "file-max"
Node Name
Current
Configured Required
---------------- ------------ ------------ ------------ -----------CNCHPDB2
6815744
6815744
6815744
CNCHPDB1
6815744
6815744
6815744
Result: Kernel parameter check passed for "file-max"
Check: Kernel parameter for "ip_local_port_range"
Node Name
Current
Configured Required
Status
Comment
-----------passed
passed
Status
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2

between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 &
65500.0 pa
CNCHPDB1
between 9000.0 & 65500.0 between 9000.0 & 65500.0 between 9000.0 &
65500.0 pa
Result: Kernel parameter check passed for "ip_local_port_range"
Check: Kernel parameter for "rmem_default"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
262144
262144
262144
passed
CNCHPDB1
262144
262144
262144
passed
Result: Kernel parameter check passed for "rmem_default"
Check: Kernel parameter for "rmem_max"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
4194304
4194304
4194304
passed
CNCHPDB1
4194304
4194304
4194304
passed
Result: Kernel parameter check passed for "rmem_max"
Check: Kernel parameter for "wmem_default"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
262144
262144
262144
passed
CNCHPDB1
262144
262144
262144
passed
Result: Kernel parameter check passed for "wmem_default"
Check: Kernel parameter for "wmem_max"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
1048576
1048576
1048576
passed
CNCHPDB1
1048756
1048756
1048576
passed
Result: Kernel parameter check passed for "wmem_max"
Check: Kernel parameter for "aio-max-nr"
Node Name
Current
Comment
---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2
4194304
4194304
1048576
passed
CNCHPDB1
4194304
4194304
1048576
passed
Result: Kernel parameter check passed for "aio-max-nr"
Check: Package existence for "make"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 make-3.81-3.el5
make-3.81
passed
CNCHPDB1 make-3.81-3.el5
make-3.81
passed
Result: Package existence check passed for "make"
Check: Package existence for "binutils"

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6
CNCHPDB1 binutils-2.17.50.0.6-14.el5 binutils-2.17.50.0.6
Result: Package existence check passed for "binutils"
Check: Package existence for "gcc(x86_64)"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 gcc(x86_64)-4.1.2-48.el5 gcc(x86_64)-4.1.2
CNCHPDB1 gcc(x86_64)-4.1.2-48.el5 gcc(x86_64)-4.1.2
Result: Package existence check passed for "gcc(x86_64)"
passed
passed
passed
passed
Check: Package existence for "libaio(x86_64)"

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 libaio(x86_64)-0.3.106-5 libaio(x86_64)-0.3.106 passed
CNCHPDB1 libaio(x86_64)-0.3.106-5 libaio(x86_64)-0.3.106 passed
Result: Package existence check passed for "libaio(x86_64)"
Check: Package existence for "glibc(x86_64)"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 glibc(x86_64)-2.5-49 glibc(x86_64)-2.5-24
CNCHPDB1 glibc(x86_64)-2.5-49 glibc(x86_64)-2.5-24
Result: Package existence check passed for "glibc(x86_64)"
passed
passed
Check: Package existence for "compat-libstdc++-33(x86_64)"

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 compat-libstdc++-33(x86_64)-3.2.3-61 compat-libstdc++-33(x86_64)3.2.3 passed
CNCHPDB1 compat-libstdc++-33(x86_64)-3.2.3-61 compat-libstdc++-33(x86_64)3.2.3 passed
Result: Package existence check passed for "compat-libstdc++-33(x86_64)"
Check: Package existence for "elfutils-libelf(x86_64)"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 elfutils-libelf(x86_64)-0.137-3.el5 elfutils-libelf(x86_64)-0.125 passed
CNCHPDB1 elfutils-libelf(x86_64)-0.137-3.el5 elfutils-libelf(x86_64)-0.125 passed
Result: Package existence check passed for "elfutils-libelf(x86_64)"
Check: Package existence for "elfutils-libelf-devel"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ----------
CNCHPDB2 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed

CNCHPDB1 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed
Result: Package existence check passed for "elfutils-libelf-devel"
Check: Package existence for "glibc-common"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 glibc-common-2.5-49
glibc-common-2.5
CNCHPDB1 glibc-common-2.5-49
glibc-common-2.5
Result: Package existence check passed for "glibc-common"
passed
passed
Check: Package existence for "glibc-devel(x86_64)"

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 glibc-devel(x86_64)-2.5-49 glibc-devel(x86_64)-2.5 passed
CNCHPDB1 glibc-devel(x86_64)-2.5-49 glibc-devel(x86_64)-2.5 passed
Result: Package existence check passed for "glibc-devel(x86_64)"
Check: Package existence for "glibc-headers"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 glibc-headers-2.5-49 glibc-headers-2.5
passed
CNCHPDB1 glibc-headers-2.5-49 glibc-headers-2.5
passed
Result: Package existence check passed for "glibc-headers"
Check: Package existence for "gcc-c++(x86_64)"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 gcc-c++(x86_64)-4.1.2-48.el5 gcc-c++(x86_64)-4.1.2
CNCHPDB1 gcc-c++(x86_64)-4.1.2-48.el5 gcc-c++(x86_64)-4.1.2
Result: Package existence check passed for "gcc-c++(x86_64)"
passed
passed
Check: Package existence for "libaio-devel(x86_64)"

Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 libaio-devel(x86_64)-0.3.106-5 libaio-devel(x86_64)-0.3.106 passed
CNCHPDB1 libaio-devel(x86_64)-0.3.106-5 libaio-devel(x86_64)-0.3.106 passed
Result: Package existence check passed for "libaio-devel(x86_64)"
Check: Package existence for "libgcc(x86_64)"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 libgcc(x86_64)-4.1.2-48.el5 libgcc(x86_64)-4.1.2
CNCHPDB1 libgcc(x86_64)-4.1.2-48.el5 libgcc(x86_64)-4.1.2
Result: Package existence check passed for "libgcc(x86_64)"
Check: Package existence for "libstdc++(x86_64)"
Node Name Available
Required
Status
passed
passed
------------ ------------------------ ------------------------ ---------CNCHPDB2 libstdc++(x86_64)-4.1.2-48.el5 libstdc++(x86_64)-4.1.2 passed

CNCHPDB1 libstdc++(x86_64)-4.1.2-48.el5 libstdc++(x86_64)-4.1.2 passed
Result: Package existence check passed for "libstdc++(x86_64)"
Check: Package existence for "libstdc++-devel(x86_64)"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 libstdc++-devel(x86_64)-4.1.2-48.el5 libstdc++-devel(x86_64)-4.1.2 passed
CNCHPDB1 libstdc++-devel(x86_64)-4.1.2-48.el5 libstdc++-devel(x86_64)-4.1.2 passed
Result: Package existence check passed for "libstdc++-devel(x86_64)"
Check: Package existence for "sysstat"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 sysstat-7.0.2-3.el5
sysstat-7.0.2
passed
CNCHPDB1 sysstat-7.0.2-3.el5
sysstat-7.0.2
passed
Result: Package existence check passed for "sysstat"
Check: Package existence for "ksh"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 ksh-20100202-1.el5
ksh-20060214
passed
CNCHPDB1 ksh-20100202-1.el5
ksh-20060214
passed
Result: Package existence check passed for "ksh"
Checking for multiple users with UID value 0
PRVF-4132 : Multiple users "root,ucmdb" with UID "0" exist on "CNCHPDB2".
PRVF-4132 : Multiple users "root,ucmdb" with UID "0" exist on "CNCHPDB1".
Result: Check for multiple users with UID value 0 failed
Check: Current group ID
Result: Current group ID check passed
Starting check for consistency of primary group of root user
Node Name
Status
------------------------------------ -----------------------CNCHPDB2
passed
CNCHPDB1
passed
Check for consistency of root user's primary group passed
Check: Package existence for "cvuqdisk"
Node Name Available
Required
Status
------------ ------------------------ ------------------------ ---------CNCHPDB2 cvuqdisk-1.0.9-1
cvuqdisk-1.0.9-1
passed
CNCHPDB1 cvuqdisk-1.0.9-1
cvuqdisk-1.0.9-1
passed
Result: Package existence check passed for "cvuqdisk"
Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
The NTP configuration file "/etc/ntp.conf" is available on all nodes
NTP Configuration file check passed
Checking daemon liveness...
Check: Liveness for "ntpd"
Node Name
Running?
------------------------------------ -----------------------CNCHPDB2
yes
CNCHPDB1
yes
Result: Liveness check passed for "ntpd"
Check for NTP daemon or service alive passed on all nodes
Checking NTP daemon command line for slewing option "-x"
Check: NTP daemon command line
Node Name
Slewing Option Set?
------------------------------------ -----------------------CNCHPDB2
yes
CNCHPDB1
yes
Result:
NTP daemon slewing option check passed
Checking NTP daemon's boot time configuration, in file "/etc/sysconfig/ntpd", for slewing option
"-x
Check: NTP daemon's boot time configuration
Node Name
Slewing Option Set?
------------------------------------ -----------------------CNCHPDB2
yes
CNCHPDB1
yes
Result:
NTP daemon's boot time configuration check for slewing option passed
Checking whether NTP daemon or service is using UDP port 123 on all nodes
Check for NTP daemon or service using UDP port 123
Node Name
Port Open?
------------------------------------ -----------------------CNCHPDB2
yes
CNCHPDB1
yes
NTP common Time Server Check started...
NTP Time Server "10.74.5.151" is common to all nodes on which the NTP daemon is running
NTP Time Server "10.74.5.150" is common to all nodes on which the NTP daemon is running
Check of common NTP Time Server passed

Clock time offset check from NTP Time Server started...
Checking on nodes "[CNCHPDB2, CNCHPDB1]"...
Check: Clock time offset from NTP Time Server
Time Server: 10.74.5.151
Time Offset Limit: 1000.0 msecs
Node Name Time Offset
Status
------------ ------------------------ -----------------------CNCHPDB2 3.701
passed
CNCHPDB1 4.506
passed
Time Server "10.74.5.151" has time offsets that are within permissible limits for nodes
"[CNCHPDB2, DB1]".
Time Server: 10.74.5.150
Time Offset Limit: 1000.0 msecs
Node Name Time Offset
Status
------------ ------------------------ -----------------------CNCHPDB2 3.763
passed
CNCHPDB1 3.698
passed
Time Server "10.74.5.150" has time offsets that are within permissible limits for nodes
"[CNCHPDB2, DB1]".
Clock time offset check passed
Result: Clock synchronization check using Network Time Protocol(NTP) passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.
Checking to make sure user "grid" is not in "root" group
Node Name Status
Comment
------------ ------------------------ -----------------------CNCHPDB2 passed
does not exist
CNCHPDB1 passed
does not exist
Result: User "grid" is not part of "root" group. Check passed
Check default user file creation mask
Node Name Available
Required
Comment
------------ ------------------------ ------------------------ ---------CNCHPDB2 0022
0022
passed
CNCHPDB1 0022
0022
passed
Result: Default user file creation mask check passed
Checking consistency of file "/etc/resolv.conf" across nodes
Checking the file "/etc/resolv.conf" to make sure only one of domain and search entries is
defined
File "/etc/resolv.conf" does not have both domain and search entries defined
Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...

domain entry in file "/etc/resolv.conf" is consistent across nodes
Checking if search entry in file "/etc/resolv.conf" is consistent across the nodes...
search entry in file "/etc/resolv.conf" is consistent across nodes
Checking file "/etc/resolv.conf" to make sure that only one search entry is defined
All nodes have one search entry defined in file "/etc/resolv.conf"
Checking all nodes to make sure that search entry is "eg01.etisalat.net" as found on node
"CNCHPDB2"
All nodes of the cluster have same value for 'search'
Checking DNS response time for an unreachable node
Node Name
Status
------------------------------------ -----------------------CNCHPDB2
passed
CNCHPDB1
passed
The DNS response time for an unreachable node is within acceptable limit on all nodes
File "/etc/resolv.conf" is consistent across nodes
UDev attributes check for OCR locations started...

Result: UDev attributes check passed for OCR locations
UDev attributes check for Voting Disk locations started...

Result: UDev attributes check passed for Voting Disk locations
Check: Time zone consistency
Result: Time zone consistency check passed
Checking VIP configuration.
Checking VIP Subnet configuration.
Check for VIP Subnet configuration passed.
Checking VIP reachability
Checking Oracle Cluster Voting Disk configuration...
ASM Running check passed. ASM is running on all specified nodes
Oracle Cluster Voting Disk configuration check passed
Clusterware version consistency passed
Pre-check for cluster services setup was unsuccessful.
Checks did not pass for the following node(s):
CNCHPDB2,CNCHPDB1
In the environment i was working on get the below error:
Checking multicast communication...

Checking subnet "10.203.2.0" for multicast communication with multicast group

"224.0.0.251"...
passed.
The above error can be ignored according to the below document:

Doc ID 1372372.1 : PRVG-11134 : Interface "n.n.n.n" on node
"racnode1" is not able to communicate with interface "n.n.n.n" on
node "racnode2"
4-As per oracle documentation in order to upgrade the clusterware
you should have at least the first patchset update level 11.2.0.3.1
applied. You must check that you meet this requirements and make
sure to apply it.
In my case there was no patchset update applied so I installed the
latest patchset for both the cluster and database and applied it.
5-In grid infrastructure home make sure that the Opatch version
is 11.2.0.3.6.
You have to backup the $GRID_HOME/OPatch directory then
download the patch 6880880 and execute the unzip command
below:
unzip 6880880 d $GRID_HOME
6-Undefine all the defined oracle parameters. In my case i
undefined the below parameters:
unset ORACLE_BASE
unset ORACLE_SID
unset ORA_NLS10
unset TNS_ADMIN
unset ORACLE_HOME
unset ORA_CRS_HOME
Apply the latest patchset update on the cluster 11.2.0.3.
On both cluster nodes apply the latest patchset update as per

below:
[root@CNCHPDB1 grid_11.2.0.3]# /app/grid_11.2.0.3/OPatch/opatch auto
/home/grid/patchset/patchset/ -oh /app/grid_11.2.0.3/ -ocmrf
/app/grid_11.2.0.3/OPatch/ocm/bin/response20102014.rsp
Executing /app/grid_11.2.0.3/perl/bin/perl /app/grid_11.2.0.3/OPatch/crs/patch11203.pl patchdir /home/grid/patchset -patchn patchset -oh /app/grid_11.2.0.3/ -ocmrf
/app/grid_11.2.0.3/OPatch/ocm/bin/response20102014.rsp -paramfile
/app/grid_11.2.0.3/crs/install/crsconfig_params
This is the main log file: /app/grid_11.2.0.3/cfgtoollogs/opatchauto2014-10-21_00-07-35.log
This file will show your detected configuration and all the steps that opatchauto attempted to
do on your system:
/app/grid_11.2.0.3/cfgtoollogs/opatchauto2014-10-21_00-07-35.report.log
2014-10-21 00:07:35: Starting Clusterware Patch Setup
Using configuration parameter file: /app/grid_11.2.0.3/crs/install/crsconfig_params
Stopping CRS...
Stopped CRS successfully
patch /home/grid/patchset/patchset/17592127 apply successful for home /app/grid_11.2.0.3
patch /home/grid/patchset/patchset/19121548 apply successful for home /app/grid_11.2.0.3
Starting CRS...
CRS-4123: Oracle High Availability Services has been started.
The Upgrade Process:

1-In the downloaded folder grid that you unzipped earlier run the below:
./RunInstaller
2-Follow the steps as below:
As per the above requirement you have to run rootupgrade on

each node:
Start with the primary node which you run the RunInstaller
from.In our case it is node CNCHPDB1.
[root@CNCHPDB1 ~]# cd /newapp/grid
[root@CNCHPDB1 grid]# ./rootupgrade.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /newapp/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /newapp/grid/crs/install/crsconfig_params
Creating trace directory
Installing Trace File Analyzer
ASM upgrade has started on first node.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on
'cnchpdb1'
CRS-2673: Attempting to stop 'ora.crsd' on 'cnchpdb1'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.AGILINK_ARCH.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.AGILINK_DATA.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.AGILINK_REDO1.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.AGILINK_REDO2.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.AGILINK_SYSTEM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.ARCH_JUPITER.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.HPARCH2.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.HPREDO2.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.OCR_VOT.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.SIS_ARCH.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.SIS_REDO.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.hpdb.eims.svc' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.oimdb.db' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.HPARCH.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.HPDATA.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.HPDATA2.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.HPREDO.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.jupiter.roaming.svc' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.sisdb.db' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'cnchpdb1'
CRS-2677: Stop of 'ora.jupiter.roaming.svc' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.jupiter.db' on 'cnchpdb1'
CRS-2677: Stop of 'ora.hpdb.eims.svc' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.hpdb.db' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'cnchpdb1'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'cnchpdb1'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.cnchpdb1.vip' on 'cnchpdb1'
CRS-2677: Stop of 'ora.oimdb.db' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.ARCH_FRA_OIM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.DATA_OIM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.REDO1_OIM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.REDO2_OIM.dg' on 'cnchpdb1'

CRS-2673: Attempting to stop 'ora.SYSTEM_OIM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.TEMP_OIM.dg' on 'cnchpdb1'
CRS-2677: Stop of 'ora.scan1.vip' on 'cnchpdb1' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'cnchpdb2'
CRS-2677: Stop of 'ora.sisdb.db' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.SIS_DATA.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.SIS_SYSTEM.dg' on 'cnchpdb1'
CRS-2677: Stop of 'ora.registry.acfs' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.cnchpdb1.vip' on 'cnchpdb1' succeeded
CRS-2672: Attempting to start 'ora.cnchpdb1.vip' on 'cnchpdb2'
CRS-2677: Stop of 'ora.AGILINK_DATA.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.AGILINK_REDO1.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.SIS_REDO.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.AGILINK_ARCH.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.SIS_ARCH.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.HPARCH.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.hpdb.db' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.ARCH_FLASH.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.REDO.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.SYSTEM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.TEMP.dg' on 'cnchpdb1'
CRS-2677: Stop of 'ora.DATA.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.SYSTEM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.ARCH_FLASH.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.SIS_SYSTEM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.jupiter.db' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.JUPITER.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.NIM.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.REDO1_JUPITER.dg' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.REDO2_JUPITER.dg' on 'cnchpdb1'
CRS-2677: Stop of 'ora.SIS_DATA.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.DATA_OIM.dg' on 'cnchpdb1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'cnchpdb2' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'cnchpdb2'
CRS-2677: Stop of 'ora.TEMP.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.NIM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.JUPITER.dg' on 'cnchpdb1' succeeded
CRS-2676: Start of 'ora.cnchpdb1.vip' on 'cnchpdb2' succeeded
CRS-2677: Stop of 'ora.AGILINK_REDO2.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.AGILINK_SYSTEM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.REDO1_OIM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.REDO2_OIM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.REDO2_JUPITER.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.REDO.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.HPREDO2.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.REDO1_JUPITER.dg' on 'cnchpdb1' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'cnchpdb2' succeeded

CRS-2677: Stop of 'ora.SYSTEM_OIM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.TEMP_OIM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.ARCH_FRA_OIM.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.ARCH_JUPITER.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.HPREDO.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.HPDATA.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.HPDATA2.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.HPARCH2.dg' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.OCR_VOT.dg' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'cnchpdb1'
CRS-2677: Stop of 'ora.asm' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'cnchpdb1'
CRS-2677: Stop of 'ora.ons' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'cnchpdb1'
CRS-2677: Stop of 'ora.net1.network' on 'cnchpdb1' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'cnchpdb1' has
completed
CRS-2677: Stop of 'ora.crsd' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.evmd' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.asm' on 'cnchpdb1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'cnchpdb1'
CRS-2677: Stop of 'ora.evmd' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.asm' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'cnchpdb1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'cnchpdb1'
CRS-2677: Stop of 'ora.cssd' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.crf' on 'cnchpdb1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'cnchpdb1' succeeded
CRS-2677: Stop of 'ora.crf' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'cnchpdb1'
CRS-2677: Stop of 'ora.gipcd' on 'cnchpdb1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'cnchpdb1'
CRS-2677: Stop of 'ora.gpnpd' on 'cnchpdb1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'cnchpdb1' has
completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@CNCHPDB1 grid]#
The same Action will be applied on the other node:

[main] [ 2014-11-02 23:35:36.402 EET ] [RuntimeExec.runCommand:164] srvm
[main] [ 2014-11-02 23:35:36.402 EET ] [RuntimeExec.runCommand:164] sysman
[main] [ 2014-11-02 23:35:36.403 EET ] [RuntimeExec.runCommand:164] timingframework
[main] [ 2014-11-02 23:35:36.403 EET ] [RuntimeExec.runCommand:164] ucp
[main] [ 2014-11-02 23:35:36.403 EET ] [RuntimeExec.runCommand:164] uix
[main] [ 2014-11-02 23:35:36.403 EET ] [RuntimeExec.runCommand:164] usm
[main] [ 2014-11-02 23:35:36.403 EET ] [RuntimeExec.runCommand:164] utl
[main] [ 2014-11-02 23:35:36.404 EET ] [RuntimeExec.runCommand:164] wwg
[main] [ 2014-11-02 23:35:36.404 EET ] [RuntimeExec.runCommand:164] xdk
[main] [ 2014-11-02 23:35:36.404 EET ] [RuntimeExec.runCommand:170] RunTimeExec: error>
[main] [ 2014-11-02 23:35:36.404 EET ] [RuntimeExec.runCommand:192] Returning from
RunTimeExec.runCommand
[main] [ 2014-11-02 23:35:36.406 EET ] [Utils.getLocalHost:481] Hostname retrieved:
CNCHPDB2, returned: CNCHPDB2
[main] [ 2014-11-02 23:35:36.834 EET ] [UnixSystem.isNodeAccessible:1015] Checking
accessibility of node: cnchpdb2
[main] [ 2014-11-02 23:35:36.834 EET ] [UnixSystem.isNodeAccessible:1021] isNodeAccessible::
Running Unix command: /usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o
StrictHostKeyChecking=yes -o NumberOfPasswordPrompts=0 cnchpdb2 -n /bin/true
[main] [ 2014-11-02 23:35:36.834 EET ] [RuntimeExec.runCommand:75] Calling Runtime.exec()
with the command
[main] [ 2014-11-02 23:35:36.834 EET ] [RuntimeExec.runCommand:77] /usr/bin/ssh
[main] [ 2014-11-02 23:35:36.834 EET ] [RuntimeExec.runCommand:77] -o
[main] [ 2014-11-02 23:35:36.835 EET ] [RuntimeExec.runCommand:77] FallBackToRsh=no
[main] [ 2014-11-02 23:35:36.835 EET ]
[RuntimeExec.runCommand:77] PasswordAuthentication=no
[main] [ 2014-11-02 23:35:36.835 EET ]
[RuntimeExec.runCommand:77] StrictHostKeyChecking=yes
[main] [ 2014-11-02 23:35:36.835 EET ]
[RuntimeExec.runCommand:77] NumberOfPasswordPrompts=0
[main] [ 2014-11-02 23:35:36.836 EET ] [RuntimeExec.runCommand:77] cnchpdb2
[main] [ 2014-11-02 23:35:36.836 EET ] [RuntimeExec.runCommand:77] -n
[main] [ 2014-11-02 23:35:36.836 EET ] [RuntimeExec.runCommand:77] /bin/true
[Thread-280] [ 2014-11-02 23:35:36.843 EET ] [StreamReader.run:61] In StreamReader.run

[main] [ 2014-11-02 23:35:36.842 EET ] [RuntimeExec.runCommand:142] runCommand:
Waiting for the process
[main] [ 2014-11-02 23:35:37.171 EET ] [RuntimeExec.runCommand:144] runCommand: process
returns 0
[main] [ 2014-11-02 23:35:37.171 EET ] [RuntimeExec.runCommand:161] RunTimeExec:
output>
[main] [ 2014-11-02 23:35:37.172 EET ] [NativeResult.<init>:91] NativeResult: The String
obtained is1|
[main] [ 2014-11-02 23:35:37.173 EET ] [NativeResult.<init>:99] The status string is: 1
[main] [ 2014-11-02 23:35:37.174 EET ] [GetActiveNodes.create:226] Returning an existing
instance of GetActiveNodes
[main] [ 2014-11-02 23:35:37.175 EET ]
[Cluster.verifyNodeList:1148] clusterNodes[0]=cnchpdb1
[main] [ 2014-11-02 23:35:37.175 EET ]
[main] [ 2014-11-02 23:35:37.175 EET ] [Cluster.verifyNodeList:1153] nodeList[0]=cnchpdb1
[main] [ 2014-11-02 23:35:37.176 EET ] [Cluster.isSharedPath:1072] NodeList=[cnchpdb1,
cnchpdb2]
[main] [ 2014-11-02 23:35:37.176 EET ]
[Cluster.isSharedPath:1108] pathName=/app/oracle/product/agent11g dirPath =
/app/oracle/product/agent11g
[main] [ 2014-11-02 23:35:37.176 EET ]
[UnixSystem.isSharedPath:1592] dirPath=/app/oracle/product/agent11g
[main] [ 2014-11-02 23:35:37.176 EET ] [UnixSystem.isSharedPath:1597] UnixSystem.isShared:
pathName=/app/oracle/product/agent11g tempFile=CFSFileName1414964137176
dirPathName is /app/oracle/product/agent11g/CFSFileName1414964137176
removing file /app/oracle/product/agent11g/CFSFileName1414964137176on cnchpdb2
with the command
[main] [ 2014-11-02 23:35:37.177 EET ] [RuntimeExec.runCommand:77] /bin/rm
[main] [ 2014-11-02 23:35:37.177 EET ] [RuntimeExec.runCommand:77] -f
[main] [ 2014-11-02 23:35:37.177 EET ]
[RuntimeExec.runCommand:77] /app/oracle/product/agent11g/CFSFileName1414964137176

returns 0
output>
[main] [ 2014-11-02 23:35:37.189 EET ] [NativeSystem.isCmdScv:502] isCmdScv: cmd=[/bin/rm f /app/oracle/product/agent11g/CFSFileName1414964137176]
[main] [ 2014-11-02 23:35:37.189 EET ] [NativeSystem.isCmdScv:571] isCmdScv: returned false
[main] [ 2014-11-02 23:35:37.190 EET ]
[NativeSystem.rununixcmd:1152] NativeSystem.rununixcmd: RetString 1| :successful
removing file /app/oracle/product/agent11g/CFSFileName1414964137176on cnchpdb1
with the command
[main] [ 2014-11-02 23:35:37.191 EET ]
[main] [ 2014-11-02 23:35:37.191 EET ]
[main] [ 2014-11-02 23:35:37.192 EET ]
[main] [ 2014-11-02 23:35:37.193 EET ]
[RuntimeExec.runCommand:77] /app/oracle/product/agent11g/CFSFileName1414964137176
returns 0
output>
[main] [ 2014-11-02 23:35:37.441 EET ] [NativeSystem.isCmdScv:502] isCmdScv:

cmd=[/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o
StrictHostKeyChecking=yes -o NumberOfPasswordPrompts=0 cnchpdb1 -n /bin/rm -f
/app/oracle/product/agent11g/CFSFileName1414964137176]
[main] [ 2014-11-02 23:35:37.441 EET ] [NativeSystem.isCmdScv:552] isCmdScv: /usr/bin/ssh is
present.
a file.
[main] [ 2014-11-02 23:35:37.442 EET ] [NativeSystem.isCmdScv:571] isCmdScv: returned true
[main] [ 2014-11-02 23:35:37.442 EET ]
creating file /app/oracle/product/agent11g/CFSFileName1414964137176
[main] [ 2014-11-02 23:35:37.442 EET ] [UnixSystem.isSharedPath:1620] Failed to create the
test CFS file at /app/oracle/product/agent11g/CFSFileName1414964137176
with the command
[main] [ 2014-11-02 23:35:37.444 EET ]
[main] [ 2014-11-02 23:35:37.444 EET ]
[main] [ 2014-11-02 23:35:37.445 EET ]
returns 0
output>

obtained is1|
[main] [ 2014-11-02 23:35:37.779 EET ] [UnixSystem.listDirectory:4154] unixcmd=/bin/ls -1A
/app/oracle/product/agent11g
with the command
[main] [ 2014-11-02 23:35:37.779 EET ] [RuntimeExec.runCommand:77] /bin/ls
[main] [ 2014-11-02 23:35:37.779 EET ] [RuntimeExec.runCommand:77] -1A
[main] [ 2014-11-02 23:35:37.780 EET ]
[RuntimeExec.runCommand:77] /app/oracle/product/agent11g
[Thread-288] [ 2014-11-02 23:35:37.792 EET ] [StreamReader.run:65] OUTPUT>agentpatch
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>Apache
returns 0
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>bin
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>ccr
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>cfgtoollogs
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>CNCHPDB2
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>config
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>crs
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>css
[Thread-288] [ 2014-11-02 23:35:37.793 EET ] [StreamReader.run:65] OUTPUT>diagnostics
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>EMStage
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>install
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>install.platform
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>inventory
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>jdbc
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>jdk
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>jlib
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>ldap
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>lib
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>lib32
[Thread-288] [ 2014-11-02 23:35:37.794 EET ] [StreamReader.run:65] OUTPUT>mesg
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>modules
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>network
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>nls
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>ons
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>OPatch
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>OPatch.orig
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>opmn

[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>oracore
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>oraInst.loc
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>oui
[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>owm
[Thread-288] [ 2014-11-02 23:35:37.796 EET ]
[StreamReader.run:65] OUTPUT>p6880880_111000_Linux-x86-64.zip
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>.patch_storage
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>perl
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>plsql
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>precomp
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>rdbms
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>relnotes
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>root.sh
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>slax
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>sqlplus
[Thread-288] [ 2014-11-02 23:35:37.796 EET ] [StreamReader.run:65] OUTPUT>srvm
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>sysman
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>tg4ifmx
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>tg4ingr
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>tg4sybs
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>tg4tera
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>userSetup.sh
[Thread-288] [ 2014-11-02 23:35:37.797 EET ] [StreamReader.run:65] OUTPUT>xdk
output>
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] agentpatch
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] Apache
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] bin
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] ccr
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] cfgtoollogs
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] CNCHPDB2
[main] [ 2014-11-02 23:35:37.798 EET ] [RuntimeExec.runCommand:164] config
[main] [ 2014-11-02 23:35:37.799 EET ] [RuntimeExec.runCommand:164] crs
[main] [ 2014-11-02 23:35:37.799 EET ] [RuntimeExec.runCommand:164] css
[main] [ 2014-11-02 23:35:37.799 EET ] [RuntimeExec.runCommand:164] diagnostics
[main] [ 2014-11-02 23:35:37.799 EET ] [RuntimeExec.runCommand:164] EMStage
[main] [ 2014-11-02 23:35:37.799 EET ] [RuntimeExec.runCommand:164] install
[main] [ 2014-11-02 23:35:37.799 EET ] [RuntimeExec.runCommand:164] install.platform
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] inventory
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] jdbc
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] jdk
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] jlib
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] ldap
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] lib
[main] [ 2014-11-02 23:35:37.800 EET ] [RuntimeExec.runCommand:164] lib32
[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] mesg
[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] modules
[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] network

[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] nls
[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] ons
[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] OPatch
[main] [ 2014-11-02 23:35:37.802 EET ] [RuntimeExec.runCommand:164] OPatch.orig
[main] [ 2014-11-02 23:35:37.802 EET ] [RuntimeExec.runCommand:164] opmn
[main] [ 2014-11-02 23:35:37.802 EET ] [RuntimeExec.runCommand:164] oracore
[main] [ 2014-11-02 23:35:37.802 EET ] [RuntimeExec.runCommand:164] oraInst.loc
[main] [ 2014-11-02 23:35:37.802 EET ] [RuntimeExec.runCommand:164] oui
[main] [ 2014-11-02 23:35:37.802 EET ] [RuntimeExec.runCommand:164] owm
[main] [ 2014-11-02 23:35:37.802 EET ]
[RuntimeExec.runCommand:164] p6880880_111000_Linux-x86-64.zip
[main] [ 2014-11-02 23:35:37.803 EET ] [RuntimeExec.runCommand:164] .patch_storage
[main] [ 2014-11-02 23:35:37.803 EET ] [RuntimeExec.runCommand:164] perl
[main] [ 2014-11-02 23:35:37.803 EET ] [RuntimeExec.runCommand:164] plsql
[main] [ 2014-11-02 23:35:37.803 EET ] [RuntimeExec.runCommand:164] precomp
[main] [ 2014-11-02 23:35:37.803 EET ] [RuntimeExec.runCommand:164] rdbms
[main] [ 2014-11-02 23:35:37.804 EET ] [RuntimeExec.runCommand:164] relnotes
[main] [ 2014-11-02 23:35:37.804 EET ] [RuntimeExec.runCommand:164] root.sh
[main] [ 2014-11-02 23:35:37.804 EET ] [RuntimeExec.runCommand:164] slax
[main] [ 2014-11-02 23:35:37.804 EET ] [RuntimeExec.runCommand:164] sqlplus
[main] [ 2014-11-02 23:35:37.804 EET ] [RuntimeExec.runCommand:164] srvm
[main] [ 2014-11-02 23:35:37.805 EET ] [RuntimeExec.runCommand:164] tg4ifmx
[main] [ 2014-11-02 23:35:37.805 EET ] [RuntimeExec.runCommand:164] tg4ingr
[main] [ 2014-11-02 23:35:37.805 EET ] [RuntimeExec.runCommand:164] tg4sybs
[main] [ 2014-11-02 23:35:37.805 EET ] [RuntimeExec.runCommand:164] tg4tera
[main] [ 2014-11-02 23:35:37.805 EET ] [RuntimeExec.runCommand:164] userSetup.sh
with the command

[main] [ 2014-11-02 23:35:38.799 EET ]
[main] [ 2014-11-02 23:35:38.800 EET ]
[main] [ 2014-11-02 23:35:38.800 EET ]
returns 0
output>
obtained is1|
[main] [ 2014-11-02 23:35:39.137 EET ]
[main] [ 2014-11-02 23:35:39.137 EET ]
[main] [ 2014-11-02 23:35:39.138 EET ] [Cluster.isSharedPath:1072] NodeList=[cnchpdb1,
cnchpdb2]
[main] [ 2014-11-02 23:35:39.138 EET ]
[Cluster.isSharedPath:1108] pathName=/app/oracle/product/11.2.0.3/dbhome_1 dirPath =
/app/oracle/product/11.2.0.3/dbhome_1
[main] [ 2014-11-02 23:35:39.138 EET ]
[UnixSystem.isSharedPath:1592] dirPath=/app/oracle/product/11.2.0.3/dbhome_1
pathName=/app/oracle/product/11.2.0.3/dbhome_1 tempFile=CFSFileName1414964139139

dirPathName is /app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139
removing file
/app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139on cnchpdb2
with the command
[main] [ 2014-11-02 23:35:39.140 EET ]
[RuntimeExec.runCommand:77] /app/oracle/product/11.2.0.3/dbhome_1/CFSFileName141496
4139139
returns 0
output>
[main] [ 2014-11-02 23:35:39.150 EET ] [NativeSystem.isCmdScv:502] isCmdScv: cmd=[/bin/rm f /app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139]
[main] [ 2014-11-02 23:35:39.150 EET ] [NativeSystem.isCmdScv:571] isCmdScv: returned false
[main] [ 2014-11-02 23:35:39.150 EET ]
removing file
/app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139on cnchpdb1
with the command
[main] [ 2014-11-02 23:35:39.152 EET ]
[main] [ 2014-11-02 23:35:39.152 EET ]
[main] [ 2014-11-02 23:35:39.153 EET ]

[main] [ 2014-11-02 23:35:39.154 EET ]
[RuntimeExec.runCommand:77] /app/oracle/product/11.2.0.3/dbhome_1/CFSFileName141496
4139139
returns 0
output>
[main] [ 2014-11-02 23:35:39.402 EET ] [NativeSystem.isCmdScv:502] isCmdScv:
cmd=[/usr/bin/ssh -o FallBackToRsh=no -o PasswordAuthentication=no -o
StrictHostKeyChecking=yes -o NumberOfPasswordPrompts=0 cnchpdb1 -n /bin/rm -f
/app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139]
present.
a file.
[main] [ 2014-11-02 23:35:39.403 EET ] [NativeSystem.isCmdScv:571] isCmdScv: returned true
[main] [ 2014-11-02 23:35:39.403 EET ]
creating file /app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139
[main] [ 2014-11-02 23:35:39.403 EET ] [UnixSystem.isSharedPath:1620] Failed to create the
test CFS file at /app/oracle/product/11.2.0.3/dbhome_1/CFSFileName1414964139139
with the command
[main] [ 2014-11-02 23:35:39.405 EET ]
[main] [ 2014-11-02 23:35:39.405 EET ]

[main] [ 2014-11-02 23:35:39.406 EET ]
returns 0
output>
obtained is1|
[main] [ 2014-11-02 23:35:39.738 EET ] [UnixSystem.listDirectory:4154] unixcmd=/bin/ls -1A
/app/oracle/product/11.2.0.3/dbhome_1
with the command
[main] [ 2014-11-02 23:35:39.738 EET ] [RuntimeExec.runCommand:77] /bin/ls
[main] [ 2014-11-02 23:35:39.738 EET ] [RuntimeExec.runCommand:77] -1A
[main] [ 2014-11-02 23:35:39.739 EET ]
[RuntimeExec.runCommand:77] /app/oracle/product/11.2.0.3/dbhome_1
[Thread-300] [ 2014-11-02 23:35:39.751 EET ] [StreamReader.run:65] OUTPUT>apex
[Thread-300] [ 2014-11-02 23:35:39.751 EET ] [StreamReader.run:65] OUTPUT>assistants
returns 0
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65] OUTPUT>bin
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65] OUTPUT>ccr
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65] OUTPUT>cdata
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65] OUTPUT>cfgtoollogs
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65] OUTPUT>clone
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65] OUTPUT>config
[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65]

2014.bkp
OUTPUT>crs
OUTPUT>csmig
OUTPUT>css
OUTPUT>ctx
OUTPUT>cv
OUTPUT>dbs
OUTPUT>dc_ocm
OUTPUT>deinstall
OUTPUT>demo
OUTPUT>diagnostics
OUTPUT>dv
OUTPUT>emcli
OUTPUT>EMStage
OUTPUT>has
OUTPUT>hs
OUTPUT>ide
OUTPUT>install
OUTPUT>instantclient
OUTPUT>inventory
OUTPUT>j2ee
OUTPUT>javavm
OUTPUT>jdbc
OUTPUT>jdev
OUTPUT>jdk
OUTPUT>jlib
OUTPUT>ldap
OUTPUT>lib
OUTPUT>log
OUTPUT>md
OUTPUT>mesg
OUTPUT>mgw
OUTPUT>network
OUTPUT>nls
OUTPUT>oc4j
OUTPUT>odbc
OUTPUT>olap
OUTPUT>OPatch
OUTPUT>OPatch-20-10OUTPUT>opmn
OUTPUT>oracore
OUTPUT>oraInst.loc
OUTPUT>ord
OUTPUT>oui
OUTPUT>owb
OUTPUT>owm
OUTPUT>patch
OUTPUT>.patch_storage
[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>perl

[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>plsql
[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>precomp
[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>psu
[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>racg
[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>rdbms
[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>relnotes
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>root.sh
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>scheduler
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>slax
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>sqldeveloper
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>sqlj
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>sqlplus
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>srvm
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>sysman
[Thread-300] [ 2014-11-02 23:35:39.758 EET ]
[StreamReader.run:65] OUTPUT>timingframework
[Thread-300] [ 2014-11-02 23:35:39.758 EET ] [StreamReader.run:65] OUTPUT>ucp
[Thread-300] [ 2014-11-02 23:35:39.759 EET ] [StreamReader.run:65] OUTPUT>uix
[Thread-300] [ 2014-11-02 23:35:39.759 EET ] [StreamReader.run:65] OUTPUT>usm
[Thread-300] [ 2014-11-02 23:35:39.759 EET ] [StreamReader.run:65] OUTPUT>utl
[Thread-300] [ 2014-11-02 23:35:39.759 EET ] [StreamReader.run:65] OUTPUT>utlu112.lst
[Thread-300] [ 2014-11-02 23:35:39.759 EET ] [StreamReader.run:65] OUTPUT>wwg
[Thread-300] [ 2014-11-02 23:35:39.759 EET ] [StreamReader.run:65] OUTPUT>xdk
output>
[main] [ 2014-11-02 23:35:39.759 EET ] [RuntimeExec.runCommand:164] apex
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] assistants
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] bin
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] ccr
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] cdata
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] cfgtoollogs
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] clone
[main] [ 2014-11-02 23:35:39.760 EET ] [RuntimeExec.runCommand:164] config
[main] [ 2014-11-02 23:35:39.761 EET ] [RuntimeExec.runCommand:164] crs
[main] [ 2014-11-02 23:35:39.761 EET ] [RuntimeExec.runCommand:164] csmig
[main] [ 2014-11-02 23:35:39.761 EET ] [RuntimeExec.runCommand:164] css
[main] [ 2014-11-02 23:35:39.761 EET ] [RuntimeExec.runCommand:164] ctx
[main] [ 2014-11-02 23:35:39.761 EET ] [RuntimeExec.runCommand:164] cv
[main] [ 2014-11-02 23:35:39.761 EET ] [RuntimeExec.runCommand:164] dbs
[main] [ 2014-11-02 23:35:39.762 EET ] [RuntimeExec.runCommand:164] dc_ocm
[main] [ 2014-11-02 23:35:39.762 EET ] [RuntimeExec.runCommand:164] deinstall
[main] [ 2014-11-02 23:35:39.762 EET ] [RuntimeExec.runCommand:164] demo
[main] [ 2014-11-02 23:35:39.762 EET ] [RuntimeExec.runCommand:164] diagnostics
[main] [ 2014-11-02 23:35:39.762 EET ] [RuntimeExec.runCommand:164] dv
[main] [ 2014-11-02 23:35:39.762 EET ] [RuntimeExec.runCommand:164] emcli
[main] [ 2014-11-02 23:35:39.763 EET ] [RuntimeExec.runCommand:164] EMStage
[main] [ 2014-11-02 23:35:39.763 EET ] [RuntimeExec.runCommand:164] has
[main] [ 2014-11-02 23:35:39.763 EET ] [RuntimeExec.runCommand:164]

2014.bkp
hs
ide
install
instantclient
inventory
j2ee
javavm
jdbc
jdev
jdk
jlib
ldap
lib
log
md
mesg
mgw
network
nls
oc4j
odbc
olap
OPatch
OPatch-20-10opmn
oracore
oraInst.loc
ord
oui
owb
owm
patch
.patch_storage
perl
plsql
precomp
psu
racg
rdbms
relnotes
root.sh
scheduler
slax
sqldeveloper
sqlj
sqlplus
srvm

[main] [ 2014-11-02 23:35:39.771 EET ] [RuntimeExec.runCommand:164] timingframework
[main] [ 2014-11-02 23:35:39.772 EET ] [RuntimeExec.runCommand:164] ucp
[main] [ 2014-11-02 23:35:39.772 EET ] [RuntimeExec.runCommand:164] uix
[main] [ 2014-11-02 23:35:39.772 EET ] [RuntimeExec.runCommand:164] usm
[main] [ 2014-11-02 23:35:39.772 EET ] [RuntimeExec.runCommand:164] utl
[main] [ 2014-11-02 23:35:39.772 EET ] [RuntimeExec.runCommand:164] utlu112.lst
[main] [ 2014-11-02 23:35:39.773 EET ] [RuntimeExec.runCommand:164] wwg
[main] [ 2014-11-02 23:35:40.232 EET ] [SQLEngine.done:2189] Done called
ASM upgrade has finished on last node.

Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Errors During upgrade process:

Error1:
I Got the below issue when i was upgrading the cluster:
This issue is handled in the below note for bug:

BUG 17666448 - INS-06006 OCCURS IF HOSTNAME IS IN CAPITAL
LETTERS FOR GI INSTALLATION
11.2.0.4 runInstaller: [INS-06006] Passwordless SSH connectivity
not set up between the following nodes(s) ( Doc ID 1597212.1)
Error in the log file should be similar to the below:
INFO: Remote node(s) not configured for passwordless SSH
connectivity : [cnchpdb1]
INFO: Details about the list of remote node(s), not configured for
passwordless SSH connectivity
--------------------------------NODE REMOTE NODES
--------------------------------cnchpdb1 [cnchpdb1, cnchpdb2]
I have followed the below steps in order to solve the issue:
[grid@CNCHPDB1 grid_11.2.0.3]$ /app/grid_11.2.0.3/oui/bin/runInstaller -updateNodelist
ORACLE_HOME=/app/grid_11.2.0.3 "CLUSTER_NODES={cnchpdb1,cnchpdb2}" CRS=true
Starting Oracle Universal Installer...
Checking swap space: must be greater than 500 MB. Actual 18473 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /app/oracle/oraInventory
'UpdateNodeList' was successful.
11.2.0.4 runInstaller: [INS-06006] Passwordless SSH connectivity not set up between the
following nodes(s) (Doc ID 1597212.1)
$ORACLE_HOME/oui/bin/runInstaller -updateNodelist
ORACLE_HOME=/app/grid_11.2.0.3 "CLUSTER_NODES={cnchpdb1,cnchpdb2}"
CRS=true
Error2:
Users With Same UID - This test checks that multiple users do not
exist with user id as "0".
Check Failed on Nodes: [cnchpdb2, cnchpdb1]
Verification result of failed node: cnchpdb2
Details:
PRVF-4132 : Multiple users "root,ucmdb" with UID "0" exist on
"cnchpdb2". - Cause: More than one user is found to have the
same user ID as specified in the message. - Action: Ensure that

no two users share the same UID.
Back to Top
Verification result of failed node: cnchpdb1
Details:
PRVF-4132 : Multiple users "root,ucmdb" with UID "0" exist on
"cnchpdb1". - Cause: More than one user is found to have the
same user ID as specified in the message. - Action: Ensure that
no two users share the same UID.
I asked the system admin to change the user ucmdb id and the
issue was resolved.
Rollup Upgrade:
While upgrading I made the upgrade in rollup upgrade way. In this scenario we
run the rootupgrade.sh on node by node. It is mandatory to run the
rootupgrade.sh on node by node and not in parallel.
After upgrading the first node please execute the below on the first node and
you will find that it is still the same 11.2.0.3.
[grid@CNCHPDB1 bin]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]
Above output showing 11.2.0.3.0, is because clusterware's active version is still
11.2.0.3, because other node, CNCHPDB2 is still running in 11.2.0.3. Till you
successfully upgrade the other node, CNCHPDB2 to 11.2.0.4, system would
show activeversion as 11.2.0.4.
You could check the software version and/or releaseversion on CNCHPDB1
and they should show 11.2.0.4
crsctl query crs softwareversion
The above command will show 11.2.0.4 on the first node while it will show
11.2.0.3 on the second node.
Run the following after you run rootupgrade.sh on node2, CNCHPDB2

crsctl query crs softwareversion
If we finish the upgrade the cluster version will become 11.2.0.4.
Checks after the cluster upgrade:
After finishing the upgrade we have to check the software versions:

You will need to run the below commands to check that everything is ok:
[grid@CNCHPDB1 bin]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.4.0]
[grid@CNCHPDB1 bin]$ crsctl query crs softwareversion
Oracle Clusterware version on node [cnchpdb1] is [11.2.0.4.0]
[grid@CNCHPDB1 bin]$ crsctl stat res -t -init
-------------------------------------------------------------------------------NAME
TARGET STATE
SERVER
STATE_DETAILS
-------------------------------------------------------------------------------Cluster Resources
-------------------------------------------------------------------------------ora.asm
1
ONLINE ONLINE
cnchpdb1
Started
ora.cluster_interconnect.haip
1
ONLINE ONLINE
cnchpdb1
ora.crf
1
ONLINE ONLINE
cnchpdb1
ora.crsd
1
ONLINE ONLINE
cnchpdb1
ora.cssd
1
ONLINE ONLINE
cnchpdb1
ora.cssdmonitor
1
ONLINE ONLINE
cnchpdb1
ora.ctssd
1
ONLINE ONLINE
cnchpdb1
OBSERVER
ora.diskmon
1
OFFLINE OFFLINE
ora.drivers.acfs
1
ONLINE ONLINE
cnchpdb1
ora.evmd
1
ONLINE ONLINE
cnchpdb1
ora.gipcd
1
ONLINE ONLINE
cnchpdb1
ora.gpnpd
1
ONLINE ONLINE
cnchpdb1
ora.mdnsd
1
ONLINE ONLINE
cnchpdb1
[grid@CNCHPDB1 bin]$ crsctl stat res -t
-------------------------------------------------------------------------------NAME
TARGET STATE
SERVER
STATE_DETAILS
-------------------------------------------------------------------------------Local Resources
-------------------------------------------------------------------------------ora.AGILINK_ARCH.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.AGILINK_DATA.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.AGILINK_REDO1.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.AGILINK_REDO2.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.AGILINK_SYSTEM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.ARCH_FLASH.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.ARCH_FRA_OIM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.ARCH_JUPITER.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.DATA.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.DATA_OIM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.HPARCH.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
ora.HPARCH2.dg
ONLINE ONLINE
ONLINE ONLINE
ora.HPDATA.dg
ONLINE ONLINE
ONLINE ONLINE
ora.HPDATA2.dg
ONLINE ONLINE
ONLINE ONLINE
ora.HPREDO.dg
ONLINE ONLINE
ONLINE ONLINE
ora.HPREDO2.dg
ONLINE ONLINE
ONLINE ONLINE
ora.JUPITER.dg
ONLINE ONLINE
ONLINE ONLINE
ora.LISTENER.lsnr
ONLINE ONLINE
ONLINE ONLINE
ora.NIM.dg
ONLINE ONLINE
ONLINE ONLINE
ora.OCR_VOT.dg
ONLINE ONLINE
ONLINE ONLINE
ora.REDO.dg
ONLINE ONLINE
ONLINE ONLINE
ora.REDO1_JUPITER.dg
ONLINE ONLINE
ONLINE ONLINE
ora.REDO1_OIM.dg
ONLINE ONLINE
ONLINE ONLINE
ora.REDO2_JUPITER.dg
ONLINE ONLINE
ONLINE ONLINE
ora.REDO2_OIM.dg
ONLINE ONLINE
ONLINE ONLINE
ora.SIS.dg
ONLINE OFFLINE
ONLINE OFFLINE
ora.SISDBARCH.dg
ONLINE OFFLINE
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb2
cnchpdb1
ONLINE OFFLINE cnchpdb2

ora.SIS_ARCH.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.SIS_DATA.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.SIS_REDO.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.SIS_SYSTEM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.SYSTEM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.SYSTEM_OIM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.TEMP.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.TEMP_OIM.dg
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.asm
ONLINE ONLINE
cnchpdb1
Started
ONLINE ONLINE
cnchpdb2
Started
ora.gsd
OFFLINE OFFLINE cnchpdb1
OFFLINE OFFLINE cnchpdb2
ora.net1.network
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.ons
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
ora.registry.acfs
ONLINE ONLINE
cnchpdb1
ONLINE ONLINE
cnchpdb2
-------------------------------------------------------------------------------Cluster Resources
-------------------------------------------------------------------------------ora.LISTENER_SCAN1.lsnr
1
ONLINE ONLINE
cnchpdb2
ora.LISTENER_SCAN2.lsnr
1
ONLINE ONLINE
cnchpdb1
ora.LISTENER_SCAN3.lsnr
1
ONLINE ONLINE
ora.cnchpdb1.vip
1
ONLINE ONLINE
ora.cnchpdb2.vip
1
ONLINE ONLINE
ora.cvu
1
ONLINE ONLINE
ora.hp1cloud.db
1
OFFLINE OFFLINE
2
OFFLINE OFFLINE
ora.hp2cloud.db
1
ONLINE OFFLINE
2
ONLINE OFFLINE
ora.hpdb.db
1
ONLINE ONLINE
2
ONLINE ONLINE
ora.hpdb.eims.svc
1
ONLINE ONLINE
2
ONLINE ONLINE
ora.jupiter.db
1
ONLINE OFFLINE
2
ONLINE OFFLINE
ora.jupiter.roaming.svc
1
ONLINE OFFLINE
2
ONLINE OFFLINE
ora.oc4j
1
ONLINE ONLINE
ora.oimdb.db
1
ONLINE ONLINE
2
ONLINE ONLINE
ora.scan1.vip
1
ONLINE ONLINE
ora.scan2.vip
1
ONLINE ONLINE
ora.scan3.vip
1
ONLINE ONLINE
ora.sisdb.db
1
ONLINE ONLINE
2
ONLINE ONLINE
[grid@CNCHPDB1 bin]$
cnchpdb1
cnchpdb1
cnchpdb2
cnchpdb1
cnchpdb1
cnchpdb2
Open
Open
cnchpdb1
cnchpdb2
Instance Shutdown
cnchpdb2
cnchpdb1
cnchpdb2
Open
Open
cnchpdb2
cnchpdb1
cnchpdb1
cnchpdb1
cnchpdb2
Open
Open
Patch the new installed clusterware:

Now the cluster upgrade was done successfully done. An additional task
that is recommended to do as per oracle recommendation is to apply the
latest patch set update available to the newly upgraded cluster home as
per oracle recommendation.
[root@CNCHPDB1 grid]# /newapp/grid/OPatch/opatch auto
/home/grid/patchset/patchset2/19380115 -oh /newapp/grid/ -ocmrf
/newapp/grid/OPatch/ocm/bin/ocm.rsp
Executing /newapp/grid/perl/bin/perl /newapp/grid/OPatch/crs/patch11203.pl -patchdir
/home/grid/patchset/patchset2 -patchn 19380115 -oh /newapp/grid/ -ocmrf
/newapp/grid/OPatch/ocm/bin/ocm.rsp -paramfile /newapp/grid/crs/install/crsconfig_params
This is the main log file: /newapp/grid/cfgtoollogs/opatchauto2014-11-03_02-35-57.log
This file will show your detected configuration and all the steps that opatchauto attempted to
do on your system:
/newapp/grid/cfgtoollogs/opatchauto2014-11-03_02-35-57.report.log
2014-11-03 02:35:57: Starting Clusterware Patch Setup
Using configuration parameter file: /newapp/grid/crs/install/crsconfig_params
Stopping CRS...
Stopped CRS successfully
patch /home/grid/patchset/patchset2/19380115/19121551 apply successful for
home /newapp/grid
home /newapp/grid
home /newapp/grid
Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
opatch auto succeeded.
Installing a new Oracle Home for the new 11.2.0.4:
Follow the below steps to install the new home software only with version
11.2.0.4.
Patch the oracle Home with the latest patch update.

[oracle@CNCHPDB1 oracle]$ pwd
/newapp/oracle
[oracle@CNCHPDB1 oracle]$ cd 19121551
[oracle@CNCHPDB1 19121551]$ ls
17478514 18031668 18522509 19121551 patchmd.xml README.html README.txt
[oracle@CNCHPDB1 19121551]$ /newapp/oracle/product/11.2.0.4/dbhome_1/OPatch/opatch
apply
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home
: /newapp/oracle/product/11.2.0.4/dbhome_1
Central Inventory : /app/oracle/oraInventory
from
: /newapp/oracle/product/11.2.0.4/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version
: 11.2.0.4.0
Log file location :
/newapp/oracle/product/11.2.0.4/dbhome_1/cfgtoollogs/opatch/opatch2014-11-05_14-1101PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches: 17478514 18031668 18522509 19121551
Do you want to proceed? [y|n]
y
User Responded with: Y
All checks passed.
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: y
This node is part of an Oracle Real Application Cluster.
Remote nodes: 'cnchpdb2'

Local node: 'cnchpdb1'
Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/newapp/oracle/product/11.2.0.4/dbhome_1')
Is the local system ready for patching? [y|n]

y
Backing up files...
Applying sub-patch '17478514' to OH '/newapp/oracle/product/11.2.0.4/dbhome_1'
Patching component oracle.rdbms, 11.2.0.4.0...
Patching component oracle.rdbms.rsf, 11.2.0.4.0...
Patching component oracle.sdo, 11.2.0.4.0...
Patching component oracle.sysman.agent, 10.2.0.4.5...
Patching component oracle.xdk, 11.2.0.4.0...
Patching component oracle.rdbms.dbscripts, 11.2.0.4.0...
Patching component oracle.sdo.locator, 11.2.0.4.0...
Patching component oracle.nlsrtl.rsf, 11.2.0.4.0...
Patching component oracle.xdk.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.rman, 11.2.0.4.0...
Verifying the update...
Patching component oracle.ldap.rsf, 11.2.0.4.0...
Patching component oracle.rdbms.crs, 11.2.0.4.0...
Patching component oracle.precomp.common, 11.2.0.4.0...
Patching component oracle.ldap.rsf.ic, 11.2.0.4.0...
Patching component oracle.rdbms.deconfig, 11.2.0.4.0...

Patching component oracle.rdbms.deconfig, 11.2.0.4.0...
Patching component oracle.sysman.console.db, 11.2.0.4.0...
Patching component oracle.ordim.client, 11.2.0.4.0...
Patching component oracle.ordim.jai, 11.2.0.4.0...
OPatch found the word "warning" in the stderr of the make command.
Please look at this stderr. You can re-run this make command.
Stderr output:
ins_emagent.mk:113: warning: overriding commands for target `nmosudo'
ins_emagent.mk:52: warning: ignoring old commands for target `nmosudo'
/newapp/oracle/product/11.2.0.4/dbhome_1/sysman/lib/ins_emagent.mk:113: warning:
overriding commands for target `nmosudo'
/newapp/oracle/product/11.2.0.4/dbhome_1/sysman/lib/ins_emagent.mk:52: warning:
ignoring old commands for target `nmosudo'
Patching in rolling mode.
The node 'cnchpdb2' will be patched next.
Please shutdown Oracle instances running out of this ORACLE_HOME on 'cnchpdb2'.

(Oracle Home = '/newapp/oracle/product/11.2.0.4/dbhome_1')
Is the node ready for patching? [y|n]
y
Updating nodes 'cnchpdb2'
Apply-related files are:
FP = "/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_1411-01PM/rac/copy_files.txt"
DP = "/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_1411-01PM/rac/copy_dirs.txt"
MP = "/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_1411-01PM/rac/make_cmds.txt"
RC = "/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_1411-01PM/rac/remote_cmds.txt"
Instantiating the file
"/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_14-1101PM/rac/copy_files.txt.instantiated" by replacing $ORACLE_HOME in
"/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_14-1101PM/rac/copy_files.txt" with actual path.
Propagating files to remote nodes...
"/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_14-1101PM/rac/copy_dirs.txt.instantiated" by replacing $ORACLE_HOME in
"/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_14-1101PM/rac/copy_dirs.txt" with actual path.
Propagating directories to remote nodes...
"/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_14-1101PM/rac/make_cmds.txt.instantiated" by replacing $ORACLE_HOME in
"/newapp/oracle/product/11.2.0.4/dbhome_1/.patch_storage/NApply/2014-11-05_14-1101PM/rac/make_cmds.txt" with actual path.
Running command on remote node 'cnchpdb2':
cd /newapp/oracle/product/11.2.0.4/dbhome_1/rdbms/lib; /usr/bin/make -f ins_rdbms.mk
client_sharedlib ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
REMOTE_MAKE_FAILED::>&2
iamdu ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
ikfod ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
irenamedg ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
ikfed ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
ioracle ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
cd /newapp/oracle/product/11.2.0.4/dbhome_1/network/lib; /usr/bin/make -f
ins_net_client.mk client_sharedlib
ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
iorion ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
liborasdksh ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
cd /newapp/oracle/product/11.2.0.4/dbhome_1/sysman/lib; /usr/bin/make -f ins_emagent.mk
e2eme ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo

irman ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
cd /newapp/oracle/product/11.2.0.4/dbhome_1/precomp/lib; /usr/bin/make -f
ins_precomp.mk proc ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 || echo
jox_refresh_knlopt ioracle ORACLE_HOME=/newapp/oracle/product/11.2.0.4/dbhome_1 ||
echo REMOTE_MAKE_FAILED::>&2
RC file not exist. There are no commands to be run on the remote nodes.
The node 'cnchpdb2' has been patched. You can restart Oracle instances on it.
Composite patch 19121551 successfully applied.
OPatch Session completed with warnings.
Log file location: /newapp/oracle/product/11.2.0.4/dbhome_1/cfgtoollogs/opatch/opatch201411-05_14-11-01PM_1.log
OPatch completed with warnings.
[oracle@CNCHPDB1 19121551]$ /newapp/oracle/product/11.2.0.4/dbhome_1/OPatch/opatch

lsinventory
Oracle Interim Patch Installer version 11.2.0.3.6
Copyright (c) 2013, Oracle Corporation. All rights reserved.
Oracle Home
: /newapp/oracle/product/11.2.0.4/dbhome_1
Central Inventory : /app/oracle/oraInventory
from
: /newapp/oracle/product/11.2.0.4/dbhome_1/oraInst.loc
OPatch version : 11.2.0.3.6
OUI version
: 11.2.0.4.0
Log file location :
/newapp/oracle/product/11.2.0.4/dbhome_1/cfgtoollogs/opatch/opatch2014-11-05_14-1526PM_1.log
Lsinventory Output file location :
/newapp/oracle/product/11.2.0.4/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2014-1105_14-15-26PM.txt
-------------------------------------------------------------------------------Installed Top-level Products (1):

Oracle Database 11g
11.2.0.4.0
There are 1 product(s) installed in this Oracle Home.
Interim patches (1) :

Patch 19121551 : applied on Wed Nov 05 14:12:06 EET 2014
Unique Patch ID: 17949166
Patch description: "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Created on 6 Oct 2014, 10:07:57 hrs PST8PDT
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Bugs fixed:
17288409, 17205719, 17811429, 17754782, 17726838, 13364795, 17311728
17284817, 17441661, 13645875, 18199537, 16992075, 16542886, 17446237
14565184, 17071721, 17610798, 17375354, 17449815, 17397545, 19463897
18230522, 17235750, 16360112, 13866822, 17982555, 17478514, 12905058
14338435, 13944971, 16929165, 12747740, 17546973, 14054676, 17088068
18264060, 17343514, 17016369, 17042658, 14602788, 14657740, 17332800
19211724, 13951456, 16315398, 17186905, 18744139, 16850630, 17437634
19049453, 18673304, 17883081, 18641419, 17296856, 18262334, 17006183
18277454, 17232014, 16855292, 10136473, 17705023, 17865671, 18554871
19121551, 17588480, 17551709, 17344412, 17842825, 18681862, 17390160
13955826, 13609098, 18139690, 17501491, 17239687, 17752121, 17299889
17602269, 18673325, 17313525, 17242746, 19544839, 17600719, 18191164
17571306, 19466309, 17951233, 18094246, 17165204, 17011832, 17040527
16785708, 16180763, 17477958, 17174582, 17465741, 18522509, 17323222
19463893, 16875449, 16524926, 17237521, 17596908, 17811438, 17811447
18031668, 16912439, 16494615, 18061914, 17545847, 17082359, 19554106
17614134, 17341326, 17891946, 19458377, 17716305, 17752995, 16392068
19271443, 17767676, 17614227, 17040764, 17381384, 18973907, 18673342
14084247, 17389192, 17006570, 17612828, 17721717, 13853126, 18203837
17390431, 17570240, 14245531, 16043574, 16863422, 19727057, 17468141
17786518, 17037130, 17267114, 18203838, 16198143, 16956380, 17478145
14829250, 17394950, 17027426, 16268425, 18247991, 19584068, 14458214
18436307, 17265217, 13498382, 16692232, 17786278, 17227277, 16042673
16314254, 17443671, 16228604, 16837842, 17393683, 17787259, 18009564
15861775, 16399083, 18018515, 16472716, 17050888, 14010183, 17325413
16613964, 17080436, 17036973, 17761775, 16721594, 18280813, 15979965
18203835, 17297939, 16731148, 17811456, 14133975, 17385178, 17586955
16450169, 17655634, 9756271, 17892268, 17648596, 16220077, 16069901
11733603, 16285691, 17587063, 18180390, 17393915, 18096714, 17238511
17824637, 14285317, 19289642, 14764829, 18328509, 17622427, 16943711
17346671, 18996843, 14852021, 17783588, 16618694, 17672719, 17546761
Rac system comprising of multiple nodes

Local node = cnchpdb1
Remote node = cnchpdb2
-------------------------------------------------------------------------------OPatch succeeded.
References:
Quick Reference For Patchset Numbers/Availability And Download [ID
554256.1] .
Quick Reference to Patchset Patch Numbers [ID 753736.1] .
Doc ID 1363369.1 : Things to Consider Before Upgrading

to 11.2.0.3/11.2.0.4 Grid Infrastructure/ASM .
Passwordless SSH connectivity not set up between the
following nodes(s) ( Doc ID 1597212.1)

Upgrade 11.2.0.3 Real Application Cluster To 11.2.0.4

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Upgrade 11.2.0.3 Real Application Cluster To 11.2.0.4

Diunggah oleh

Hak Cipta:

Format Tersedia

Introduction:

Today i will upgrade the clusterware from 11.2.0.3 to 11.2.0.4.

1-follow the Oracle note Doc ID 1363369.1 : Things to Consider

uncluvfy.sh stage -pre crsinst -upgrade -n

Checking user equivalence...

Verification of the hosts config file successful

Interface information for node "CNCHPDB2"

Interface information for node "CNCHPDB1"

Check: Node connectivity for interface "bond0"

Check: TCP connectivity of subnet "10.203.2.0"

Check: Node connectivity for interface "bond1"

Check: TCP connectivity of subnet "172.16.0.0"

PRVG-11134 : Interface "10.203.2.20" on node "CNCHPDB2" is not able to communicate with

Checking OCR integrity...

Check: Available memory

Check: Swap space

Check: Free disk space for "CNCHPDB2:/newapp/grid"

---------------- ------------ ------------ ------------ ------------ -----------CNCHPDB2

Check: Package existence for "binutils"

Check: Package existence for "libaio(x86_64)"

Check: Package existence for "compat-libstdc++-33(x86_64)"

CNCHPDB2 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.125 passed

Check: Package existence for "glibc-devel(x86_64)"

Check: Package existence for "libaio-devel(x86_64)"

------------ ------------------------ ------------------------ ---------CNCHPDB2 libstdc++(x86_64)-4.1.2-48.el5 libstdc++(x86_64)-4.1.2 passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

Check of common NTP Time Server passed

Checking if domain entry in file "/etc/resolv.conf" is consistent across the nodes...

UDev attributes check for OCR locations started...

UDev attributes check for Voting Disk locations started...

In the environment i was working on get the below error:

Checking multicast communication...

Checking subnet "10.203.2.0" for multicast communication with multicast group

The above error can be ignored according to the below document:

Apply the latest patchset update on the cluster 11.2.0.3.

On both cluster nodes apply the latest patchset update as per

The Upgrade Process:

As per the above requirement you have to run rootupgrade on

Entries will be added to the /etc/oratab file as needed by

CRS-2673: Attempting to stop 'ora.REDO2_OIM.dg' on 'cnchpdb1'

CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'cnchpdb2' succeeded

The same Action will be applied on the other node:

[Thread-280] [ 2014-11-02 23:35:36.843 EET ] [StreamReader.run:61] In StreamReader.run

[Thread-282] [ 2014-11-02 23:35:37.185 EET ] [StreamReader.run:61] In StreamReader.run

[main] [ 2014-11-02 23:35:37.441 EET ] [NativeSystem.isCmdScv:502] isCmdScv:

[main] [ 2014-11-02 23:35:37.778 EET ] [NativeResult.<init>:91] NativeResult: The String

[Thread-288] [ 2014-11-02 23:35:37.795 EET ] [StreamReader.run:65] OUTPUT>opmn

[main] [ 2014-11-02 23:35:37.801 EET ] [RuntimeExec.runCommand:164] network

[main] [ 2014-11-02 23:35:38.799 EET ] [RuntimeExec.runCommand:77] -o

[main] [ 2014-11-02 23:35:39.139 EET ] [UnixSystem.isSharedPath:1602] UnixSystem.isShared:

[main] [ 2014-11-02 23:35:39.153 EET ] [RuntimeExec.runCommand:77] cnchpdb1

[main] [ 2014-11-02 23:35:39.405 EET ]

[Thread-300] [ 2014-11-02 23:35:39.752 EET ] [StreamReader.run:65]

[Thread-300] [ 2014-11-02 23:35:39.757 EET ] [StreamReader.run:65] OUTPUT>perl

[main] [ 2014-11-02 23:35:39.763 EET ] [RuntimeExec.runCommand:164]

[main] [ 2014-11-02 23:35:39.771 EET ] [RuntimeExec.runCommand:164] sysman

ASM upgrade has finished on last node.

Errors During upgrade process:

This issue is handled in the below note for bug:

same user ID as specified in the message. - Action: Ensure that

Run the following after you run rootupgrade.sh on node2, CNCHPDB2

If we finish the upgrade the cluster version will become 11.2.0.4.

Checks after the cluster upgrade: