Are you using Oracle 12c Flex ASM? If yes - do you have ASM passwor...

http://blog.trivadis.com/b/robertbialek/archive/2014/10/26/are-you-using...

Are you using Oracle 12c Flex ASM? If yes - do you have ASM password file backup?
Robert Bialek So, Okt 26 2014 22:56

With a 12c Standard ASM (or a pre 12c Grid Infrastructure) all ASM clients connect locally using Oracle Bequeath NT Protocol (bypassing the
listener process). Example from a 2-node 12c Standard Cluster with Standard ASM (OCR client):
SELECT s.inst_id, s.osuser, s.username, s.program, i.network_service_banner
FROM gv$session s, gv$session_connect_info i
WHERE s.sid = i.sid
AND
s.inst_id = i.inst_id
AND
s.program like 'crsd.bin%'
AND
i.network_service_banner like '%Protocol%'
ORDER BY 1;
INST_ID OSUSER USERNAME PROGRAM
NETWORK_SERVICE_BANNER
---------- ------ -------- ------------------------------------------------------------------------------------------------------------------------------1
root
SYS
crsd.bin@clstdb01.trivadis.com (TNS V1-V3)
Oracle Bequeath NT Protocol
Adapter for Linux: Version 12.1.0.2.0 - Production
2
root
SYS
crsd.bin@clstdb02.trivadis.com (TNS V1-V3)
Oracle Bequeath NT Protocol
Adapter for Linux: Version 12.1.0.2.0 - Production
As you see in the example above, in a non-Flex ASM environment the ASM password file generally does not play an important role for the cluster
and can be easily re-created.
But things change with Flex ASM... Because the ASM Flex instance is not always available on all cluster nodes, there is a change in the way the
clients connect to ASM instance.
With 12c Flex ASM all local clients still connect to ASM using the Oracle Bequeath NT Protocol (bypassing the listener process). However, all remote
clients connect using the TCP/IP NT Protocol via a new ASM Listener process (a new resource started on every HUB node in a cluster). For this
architecture to work, the ASM password file needs to be placed in a disk group (shared storage) and the disk group ASM compatibility
(compatible.asm) need to be set to at least the version 12.1.0.0.0.
srvctl config asm
ASM home: <CRS home>
Password file: +GRID/orapwASM
ASM listener: LISTENER
ASM instance count: 3
Cluster ASM listener: ASMNET1LSNR_ASM
With the Flex ASM configuration the ASM password file plays a substantial role for the client connectivity and is a critical cluster component which is not the case with older releases (or 12c Standard ASM). During the cluster start, the crsd.bin process on all nodes (but the OCR Writer), will
connect remotely to the ASM Flex instance. Example from a 4-node Standard Cluster with Flex ASM (the OCR Writer is active on the
cldb04.trivadis.com cluster node):
SELECT s.inst_id, s.osuser, s.username, s.program, i.network_service_banner
FROM gv$session s, gv$session_connect_info i
WHERE s.sid = i.sid
AND
s.inst_id = i.inst_id
AND
s.program like 'crsd.bin%'
AND
i.network_service_banner like '%Protocol%'
ORDER BY 1;
INST_ID OSUSER USERNAME PROGRAM
NETWORK_SERVICE_BANNER
---------- ------ -------- ------------------------------------------------------------------------------------------------------------------------------4
root
SYS
crsd.bin@cldb04.trivadis.com (TNS V1-V3)
Oracle Bequeath NT Protocol Adapter
for Linux: Version 12.1.0.2.0 - Production
4
root
SYS
crsd.bin@cldb02.trivadis.com (TNS V1-V3)
TCP/IP NT Protocol Adapter for
Linux: Version 12.1.0.2.0 - Production
4
root
SYS
crsd.bin@cldb01.trivadis.com (TNS V1-V3)
TCP/IP NT Protocol Adapter for
Linux: Version 12.1.0.2.0 - Production
4
root
SYS
crsd.bin@cldb03.trivadis.com (TNS V1-V3)
TCP/IP NT Protocol Adapter for
Linux: Version 12.1.0.2.0 - Production
What happens internally during the cluster start ?
During the start the crsd.bin process on all nodes (but the OCR Writer), will connect remotely to the ASM Flex instance using a new ASM user
crsuser__asm_001:

1 of 3

6/10/2015 10:15 PM

Are you using Oracle 12c Flex ASM? If yes - do you have ASM passwor...

http://blog.trivadis.com/b/robertbialek/archive/2014/10/26/are-you-using...

SQL> SELECT * FROM v$pwfile_users;

USERNAME
---------------SYS
CRSUSER__ASM_001
ASMSNMP
ORACLE_001

SYSDB
----TRUE
TRUE
TRUE
TRUE

SYSOP
----TRUE
FALSE
FALSE
FALSE

SYSAS
----TRUE
TRUE
FALSE
FALSE

SYSBA
----FALSE
FALSE
FALSE
FALSE

SYSDG
----FALSE
FALSE
FALSE
FALSE

SYSKM
CON_ID
----- ---------FALSE
0
FALSE
0
FALSE
0
FALSE
0

The credentials are stored not only in the ASM password file, but also in the OLR (Oracle Local Registry). With the current implementation (version
12.1.0.2 plus PSU 12.1.0.2.1) re-creating the ASM password file after a failure (e.g. re-creating the ASM disk group hosting the password file) will not
suffice to recover the system. There is currently also no procedure to re-create the credentials for the crsuser__asm_001 user (ASM password file as
well as OLR). In such a case, you will be able to start the cluster only on the OCR writer node, on other cluster nodes during the unsuccessful tries to
start the ora.storage resource, you will see the following errors reported in the ohasd_orarootagent_root.trc trace file:
2014-10-26 21:31:55.380587*:kgfn.c@1684: kgfnRecordErrPriv: rec=1
2014-10-26 21:31:57.010607 : default:2260707072: clsCredDomClose: Credctx deleted 0x7fd1603ed920
2014-10-26 21:31:57.010607*:kgfn.c@4868: kgfnConnect2: failed to connect
2014-10-26 21:31:57.010607*:kgfn.c@4887: kgfnConnect: conn=(nil)
2014-10-26 21:31:57.010607*:kgfp.c@669: kgfpInitComplete2 hdl=0x7fd1601f4ff8 conn=0x7fd1601f5010 ok=0
2014-10-26 21:31:57.010607*:kgfo.c@947: kgfo_kge2slos error stack at kgfoAl06: ORA-01017: invalid
username/password; logon denied
2014-10-26 21:31:57.163377*:kgfn.c@6356: kgfnGetNodeType: flags=0x10
2014-10-26 21:31:57.163377*:kgfn.c@6369: kgfnGetNodeType: ntyp=1
2014-10-26 21:31:57.163377*:kgfn.c@4644: kgfnConnect2: kgfnGetBeqData failed
2014-10-26 21:31:57.163377*:kgfn.c@4680: kgfnConnect: srvr valid
2014-10-26 21:31:57.163377*:kgfn.c@5972: kgfnConnect2Int: sysasm=0 envflags=0x10 srvrflags=0x1
unam=crsuser__asm_001 password is NOT NULL pstr=_ocr
2014-10-26 21:31:57.163377*:kgfn.c@6121: kgfnConnect2Int: hosts=1
2014-10-26 21:31:57.163377*:kgfn.c@6134: kgfnConnect2Int: cstr=(DESCRIPTION=(TRANSPORT_CONNECT_TIMEOUT=60)
(EXPIRE_TIME=1)(LOAD_BALANCE=ON)(ADDRESS_LIST=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.0.17)(PORT=1522)))
(CONNECT_DATA=(SERVICE_NAME=+ASM)))
Conclusion
In case you use a Flex ASM configuration, make sure that you incorporate the ASM password file into your Grid Infrastructure backup/recovery
strategy, as it cannot be simply re-created with the necessary credentials. The password file can be backed up with the following command:
ASMCMD> pwcopy +GRID/orapwASM <Backup_Location>/orapwASM_<timestamp>
copying +GRID/orapwASM -> <Backup_Location>/orapwASM_<timestamp>
For the restore use the pwcopy ASMCMD command (use the --asm parameter, especially if you change the ASM password file location - the
corresponding Clusterware resources will be updated automatically):
ASMCMD> pwcopy --asm <Backup_Location>/orapwASM_<timestamp> +GRID/orapwASM
copying <Backup_Location>/orapwASM_<timestamp> -> +GRID/orapwASM
An enhancement request (Bug 19585275 - 12C_ASM_X64: NEED A WAY TO RE-SETUP THE CREDENTIAL FOR CRSUSER__ASM_001 USER)
has been already filed. With the future implementation the ASM password file should be backed up automatically, just like OCR and voting disks.
Update on 2014.12.09
If you were forced to re-create the ASM password file and do not have a valid backup of it, there is still a way to recover. You need to find out from the
cluster configuration the automatically generated password for the user crsuser__asm_001.
First of all, you need to find out the correct path to the credential:
grid@cldb01:~/ [+ASM1] crsctl query credmaint -path ASM/Self
Path
Credtype
ID
/ASM/Self/159b7866d7ca5f63bf3e6099f0abf432

userpass

Attrs
create_time=2014
-10-10 18:42:58,
modify_time=2014
-10-10 18:42:58,
expiration_time=
NEVER,bootstrap=
FALSE

With the correct path you can query the user password:
grid@cldb01:~/ [+ASM1] crsctl get credmaint -path \
> /ASM/Self/159b7866d7ca5f63bf3e6099f0abf432 \
> -credtype userpass -id 0 -attr passwd -local

2 of 3

6/10/2015 10:15 PM

Are you using Oracle 12c Flex ASM? If yes - do you have ASM passwor...

http://blog.trivadis.com/b/robertbialek/archive/2014/10/26/are-you-using...

W0mqtCgA8Zj33h6GDC6gFvHlo4OhA

"W0mqtCgA8Zj33h6GDC6gFvHlo4OhA" string is in my test environment the automatically generated password for the user crsuser__asm_001
Knowing the correct password allows you to re-create correctly the user crsuser__asm_001 (after re-creating the ASM password). At the end don't
forget to grant the roles sysasm and sysdba to the user crsuser__asm_001.

3 of 3

6/10/2015 10:15 PM