11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

Oracle-Ninja.com
Andy Colvin's Orac le Blog

Database Shutdown With crsctl

By Andy Colvin | April 6, 2014

1 Comment

I noticed something for the rst time tonight when I was playing around in the Enkitec lab something that I have been doing wrong for a while.
When working in the lab, I often rely on the crsctl command to shut down the entire cluster stack for me. Its really easy to use crsctl stop
cluster -all followed by dcli -l root -g ~/dbs_group /u01/app/11.2.0.4/grid/bin/crsctl stop crs to get everything down quickly. As it turns out,
thats not really the best course of action.
Lets start by looking at our dbm01 database to see how it is con gured.
?

View Code NONE

[enkdb01:oracle:dbm011] /home/oracle
> srvctl config database -d dbm01
Database unique name: dbm01
Database name: dbm01
Oracle home: /u01/app/oracle/product/11.2.0.4/dbhome_1
Oracle user: oracle
Spfile: +DATA/dbm01/spfiledbm01.ora
Domain:
Start options: open
http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

1/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

Stop options: immediate

Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: dbm01
Database instances: dbm011,dbm012
Disk Groups: DATA,RECO
Mount point paths:
Services: ANDY,martin,soe
Type: RAC
Database is administrator managed

As you can see, we have this database con gured to use a shutdown immediate when the database is shut down. When we issue a simple
srvctl stop database -d dbm01 we get the following in the third line of the alert log snippet below:
?

View Code NONE

Sun Apr 06 18:50:05 2014

ALTER SYSTEM SET service_names='SYS$SYS.SCHEDULER$_EVENT_QUEUE.DBM01' SCOPE=MEMORY SID='dbm011';
Shutting down instance (immediate)
Shutting down instance: further logons disabled
Sun Apr 06 18:50:05 2014
Stopping background process CJQ0
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
License high water mark = 16
ALTER DATABASE CLOSE NORMAL /* db agent *//* {1:971:290} */
Sun Apr 06 18:50:11 2014
SMON: disabling tx recovery
Stopping background process RCBG
SMON: disabling cache recovery
Sun Apr 06 18:50:13 2014
Block change tracking service stopping.

http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

2/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

Block change tracking service stopping.

NOTE: Deferred communication with ASM instance
NOTE: deferred map free for map id 51
Stopping background process CTWR
Sun Apr 06 18:50:14 2014
NOTE: Deferred communication with ASM instance
NOTE: deferred map free for map id 32
Redo thread 1 internally disabled at seq 445 (LGWR)
Shutting down archive processes
Archiving is disabled
Sun Apr 06 18:50:14 2014
ARCH shutting down
ARC2: Archival stopped
Sun Apr 06 18:50:14 2014
ARCH shutting down
Sun Apr 06 18:50:14 2014
ARC3: Becoming the 'no FAL' ARCH
ARC0: Archival stopped
ARC3: Becoming the 'no SRL' ARCHSun Apr 06 18:50:14 2014
ARCH shutting down
ARC3: Archiving disabled
ARCH shutting down
ARC1: Archival stopped
ARC3: Archival stopped
Sun Apr 06 18:50:14 2014
NOTE: Deferred communication with ASM instance
Thread 1 closed at log sequence 445
Successful close of redo thread 1
Sun Apr 06 18:50:15 2014
NOTE: Deferred communication with ASM instance
NOTE: deferred map free for map id 4
Sun Apr 06 18:50:16 2014
Completed: ALTER DATABASE CLOSE NORMAL /* db agent *//* {1:971:290} */

This is what you would expect. The database is receiving a shutdown immediate command from srvctl. Now, lets try this as part of stopping the

http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

3/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

This is what you would expect. The database is receiving a shutdown immediate command from srvctl. Now, lets try this as part of stopping the
clusterware stack:
?

View Code NONE

[root@enkdb01 ~]# crsctl stop cluster -all

CRS-2673:
CRS-2790:
CRS-2673:
CRS-2673:
CRS-2673:

Attempting to stop 'ora.crsd' on 'enkdb01'

Starting shutdown of Cluster Ready Services-managed resources on 'enkdb01'
Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'enkdb01'
Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'enkdb01'
Attempting to stop 'ora.oc4j' on 'enkdb01'

CRS-2673:
CRS-2673:
CRS-2673:
CRS-2673:
CRS-2673:

Attempting
Attempting
Attempting
Attempting
Attempting

CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2673:
CRS-2677:

Stop of 'ora.dbm01.andy.svc' on 'enkdb01' succeeded

Stop of 'ora.dbm01.martin.svc' on 'enkdb01' succeeded
Stop of 'ora.cvu' on 'enkdb01' succeeded
Stop of 'ora.LISTENER_SCAN3.lsnr' on 'enkdb01' succeeded
Attempting to stop 'ora.scan3.vip' on 'enkdb01'
Stop of 'ora.LISTENER_SCAN2.lsnr' on 'enkdb01' succeeded

CRS-2673:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2673:
CRS-2672:
CRS-2672:

Attempting to stop 'ora.scan2.vip' on 'enkdb01'

Stop of 'ora.LISTENER.lsnr' on 'enkdb01' succeeded
Stop of 'ora.scan3.vip' on 'enkdb01' succeeded
Stop of 'ora.scan2.vip' on 'enkdb01' succeeded
Stop of 'ora.dbm01.soe.svc' on 'enkdb01' succeeded
Attempting to stop 'ora.enkdb01.vip' on 'enkdb01'
Attempting to start 'ora.dbm01.andy.svc' on 'enkdb02'
Attempting to start 'ora.dbm01.martin.svc' on 'enkdb02'

CRS-2676:
CRS-2676:
CRS-2672:
CRS-2674:
CRS-2673:

Start of 'ora.dbm01.andy.svc' on 'enkdb02' succeeded

Start of 'ora.dbm01.martin.svc' on 'enkdb02' succeeded
Attempting to start 'ora.dbm01.martin_preconnect.svc' on 'enkdb02'
Start of 'ora.dbm01.martin_preconnect.svc' on 'enkdb02' failed
Attempting to stop 'ora.crsd' on 'enkdb02'

to
to
to
to
to

stop
stop
stop
stop
stop

'ora.LISTENER.lsnr' on 'enkdb01'
'ora.dbm01.andy.svc' on 'enkdb01'
'ora.dbm01.soe.svc' on 'enkdb01'
'ora.dbm01.martin.svc' on 'enkdb01'
'ora.cvu' on 'enkdb01'

CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'enkdb02'

http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

4/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

CRS-2790:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2673:
CRS-2673:
CRS-2673:
CRS-2673:

Starting shutdown of Cluster Ready Services-managed resources on 'enkdb02'

Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'enkdb02'
Stop of 'ora.enkdb01.vip' on 'enkdb01' succeeded
Attempting to stop 'ora.registry.acfs' on 'enkdb01'
Attempting to stop 'ora.DBFS_DG.dg' on 'enkdb01'
Attempting to stop 'ora.blah.db' on 'enkdb01'
Attempting to stop 'ora.colvin.db' on 'enkdb01'
Attempting to stop 'ora.dbm01.db' on 'enkdb01'

CRS-2673:
CRS-2673:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2673:
CRS-2673:

Attempting to stop 'ora.LISTENER.lsnr' on 'enkdb02'

Attempting to stop 'ora.dbm01.andy.svc' on 'enkdb02'
Attempting to stop 'ora.dbm01.martin.svc' on 'enkdb02'
Stop of 'ora.LISTENER_SCAN1.lsnr' on 'enkdb02' succeeded
Attempting to stop 'ora.scan1.vip' on 'enkdb02'
Stop of 'ora.LISTENER.lsnr' on 'enkdb02' succeeded
Stop of 'ora.oc4j' on 'enkdb01' succeeded
Stop of 'ora.dbm01.andy.svc' on 'enkdb02' succeeded
Stop of 'ora.dbm01.martin.svc' on 'enkdb02' succeeded
Attempting to stop 'ora.enkdb02.vip' on 'enkdb02'
Attempting to stop 'ora.DBFS_DG.dg' on 'enkdb02'

CRS-2673:
CRS-2673:
CRS-2673:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2677:

Attempting to stop 'ora.blah.db' on 'enkdb02'

Attempting to stop 'ora.colvin.db' on 'enkdb02'
Attempting to stop 'ora.dbm01.db' on 'enkdb02'
Stop of 'ora.registry.acfs' on 'enkdb01' succeeded
Stop of 'ora.blah.db' on 'enkdb01' succeeded
Stop of 'ora.scan1.vip' on 'enkdb02' succeeded
Stop of 'ora.colvin.db' on 'enkdb01' succeeded
Stop of 'ora.colvin.db' on 'enkdb02' succeeded
Stop of 'ora.enkdb02.vip' on 'enkdb02' succeeded
Stop of 'ora.blah.db' on 'enkdb02' succeeded

CRS-2677:
CRS-2673:
CRS-2673:
CRS-2677:
CRS-2677:
CRS-2677:
CRS-2673:
CRS-2673:
CRS-2677:

Stop of 'ora.dbm01.db' on 'enkdb01' succeeded

Attempting to stop 'ora.RECO.dg' on 'enkdb01'
Attempting to stop 'ora.DATA.dg' on 'enkdb01'
Stop of 'ora.RECO.dg' on 'enkdb01' succeeded
Stop of 'ora.DATA.dg' on 'enkdb01' succeeded
Stop of 'ora.dbm01.db' on 'enkdb02' succeeded
Attempting to stop 'ora.RECO.dg' on 'enkdb02'
Attempting to stop 'ora.DATA.dg' on 'enkdb02'
Stop of 'ora.RECO.dg' on 'enkdb02' succeeded

CRS-2677: Stop of 'ora.DATA.dg' on 'enkdb02' succeeded

http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

5/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

CRS-2677:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2677:

Stop of 'ora.DATA.dg' on 'enkdb02' succeeded

Stop of 'ora.DBFS_DG.dg' on 'enkdb01' succeeded
Attempting to stop 'ora.asm' on 'enkdb01'
Stop of 'ora.asm' on 'enkdb01' succeeded
Stop of 'ora.DBFS_DG.dg' on 'enkdb02' succeeded

CRS-2673:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2792:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2677:

Attempting to stop 'ora.asm' on 'enkdb02'

Stop of 'ora.asm' on 'enkdb02' succeeded
Attempting to stop 'ora.ons' on 'enkdb02'
Stop of 'ora.ons' on 'enkdb02' succeeded
Attempting to stop 'ora.net1.network' on 'enkdb02'
Stop of 'ora.net1.network' on 'enkdb02' succeeded
Shutdown of Cluster Ready Services-managed resources on 'enkdb02' has completed
Attempting to stop 'ora.ons' on 'enkdb01'
Stop of 'ora.ons' on 'enkdb01' succeeded
Attempting to stop 'ora.net1.network' on 'enkdb01'
Stop of 'ora.net1.network' on 'enkdb01' succeeded

CRS-2792:
CRS-2677:
CRS-2673:
CRS-2673:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2673:
CRS-2673:
CRS-2677:
CRS-2677:

Shutdown of Cluster Ready Services-managed resources on 'enkdb01' has completed

Stop of 'ora.crsd' on 'enkdb02' succeeded
Attempting to stop 'ora.ctssd' on 'enkdb02'
Attempting to stop 'ora.evmd' on 'enkdb02'
Attempting to stop 'ora.asm' on 'enkdb02'
Stop of 'ora.crsd' on 'enkdb01' succeeded
Attempting to stop 'ora.ctssd' on 'enkdb01'
Attempting to stop 'ora.evmd' on 'enkdb01'
Attempting to stop 'ora.asm' on 'enkdb01'
Stop of 'ora.evmd' on 'enkdb01' succeeded
Stop of 'ora.evmd' on 'enkdb02' succeeded

CRS-2677:
CRS-2677:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2673:
CRS-2677:
CRS-2673:

Stop of 'ora.ctssd' on 'enkdb01' succeeded

Stop of 'ora.ctssd' on 'enkdb02' succeeded
Stop of 'ora.asm' on 'enkdb02' succeeded
Attempting to stop 'ora.cluster_interconnect.haip' on 'enkdb02'
Stop of 'ora.asm' on 'enkdb01' succeeded
Attempting to stop 'ora.cluster_interconnect.haip' on 'enkdb01'
Stop of 'ora.cluster_interconnect.haip' on 'enkdb01' succeeded
Attempting to stop 'ora.cssd' on 'enkdb01'
Stop of 'ora.cluster_interconnect.haip' on 'enkdb02' succeeded
Attempting to stop 'ora.cssd' on 'enkdb02'

CRS-2677: Stop of 'ora.cssd' on 'enkdb01' succeeded

CRS-2673: Attempting to stop 'ora.diskmon' on 'enkdb01'
http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

6/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

CRS-2673: Attempting to stop 'ora.diskmon' on 'enkdb01'

CRS-2677: Stop of 'ora.cssd' on 'enkdb02' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'enkdb02'
CRS-2677: Stop of 'ora.diskmon' on 'enkdb02' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'enkdb01' succeeded

This command stops CRS resources on each of the nodes in the cluster, including the dbm01 database. To look in the alert log, we can see that
the database received a shutdown abort command. Once again, the third line tells us what we want to know:
?

View Code NONE

Sun Apr 06 19:02:11 2014

ALTER SYSTEM SET service_names='SYS$SYS.SCHEDULER$_EVENT_QUEUE.DBM01' SCOPE=MEMORY SID='dbm011';
Shutting down instance (abort)
License high water mark = 9
USER (ospid: 51156): terminating the instance
Instance terminated by USER, pid = 51156
Sun Apr 06 19:02:16 2014
Instance shutdown complete

As it turns out, Oracle says that this is expected behavior. In MOS note #1274270.1, Oracle makes their case for crsctl issuing a shutdown abort
when shutting down:

In other words: crsctl stop crs is a command to stop a cluster environment, more precisely Oracle Clusterware here on one node. This
command should only be issued either in cases of planned maintenance or in cases of emergency.For a case of emergency, it is reasonable for
such a command to take down the stack as fast as possible. As fast as possible regarding databases, however, means shutdown abort, as any
other shutdown option can lead to unwanted delays (immediate / transactional for example). This is what we do. Besides, we have been doing
this for most versions, not only for 11g Rel. 2.
http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

7/10

11/23/2014

this for most versions, not only for 11g Rel. 2.

Database Shutdown With crsctl | Oracle-Ninja.com

Before shutting down a cluster, its best to utilize srvctl to stop database instances. In order to shut down everything in a home, use the srvctl
stop home command:
?

View Code NONE

[enkdb01:oracle:dbm011] /home/oracle
> srvctl stop home -o /u01/app/oracle/product/11.2.0.4/dbhome_1 -s ~/stophome_11.2.0.4.txt -n enkdb01

The le ~/stophome_11.2.0.4.txt is used to designate all of the resources that were shut down by this command:
?

View Code NONE

[enkdb01:oracle:dbm011] /home/oracle
> cat ~/stophome_11.2.0.4.txt
db-colvin
svc-ANDY
svc-soe
db-dbm01

From looking at this, we can see that the colvin and dbm01 databases were shut down along with the ANDY and soe services. When we want to
start the resources up, change the above command from srvctl stop home to srvctl start home

View Code NONE

http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

8/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

View Code NONE

[enkdb01:oracle:dbm011] /home/oracle
> srvctl start home -o /u01/app/oracle/product/11.2.0.4/dbhome_1 -s ~/stophome_11.2.0.4.txt -n enkdb01

Be Sociable, Share!

Tw eet

Like

Share

StumbleUpon

Category: Oracle Tags: Enkitec , Exadata , oracle , RAC , troubleshooting

One thought on Database Shutdown With crsctl

Guido Campra

I noticed the change of that behavior upgrading from 11.2.0.1 to 11.2.0.3.

April 7, 2014

Doing some research I discovered that was a default before 11.2:

crsctl stop crs tries to shutdown abort DB and ASM instance

In 11.2.0.1 there is a bug (8702731) that change in shutdown immediate the action used to stop DB and ASM during CRS stack shutdown.
Bug 8702731 ctsctl stop crs waits for DB shutdown immediate (Doc ID 8702731.8)
http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

9/10

11/23/2014

Database Shutdown With crsctl | Oracle-Ninja.com

That Bug has been xed in 11.2.0.2.

Regards.
Guido

Iconic One Theme | Powered by Wordpress

http://blog.oracle-ninja.com/2014/04/database-shutdown-with-crsctl/

10/10