Using Udev RH

Using udev with Oracle Architecture (RAC & ASM)
Providing persistent names on Linux 2.6 Red Hat

An Oracle Technical Paper
May 2006
Reviewed Sep 2008
Using udev with Oracle Architecture (RAC & ASM) ............. 1

Introduction..................................................................................................................... 2
What is UDEV ................................................................................................................ 3
Configuring udev ........................................................................................................ 3
Setting the rules........................................................................................................... 4
Setting the permissions. .............................................................................................. 5
Changes on OEL 5, RHEL 5 or SLES 10................................................................... 6
Additional configuration................................................................................................. 7
Additional notes .............................................................................................................. 9
Configuration .............................................................................................................. 9
Installting CRS and not recognizing the voting disk and ocr ..................................... 9
Appendix A. Files examples. ....................................................................................... 10
/etc/udev/rules.d/50-udev.rules (before OEL5/RHEL5/SLES10) ............................ 10
/etc/udev/permissions.d/50-udev.permissions (before OEL5/RHEL5/SLES10)...... 10
/etc/udev/rules.d/50-udev.rules under OEL5/RHEL5/SLES10 ................................ 10
Using udev with Oracle Architecture (RAC & ASM) - 1 -

Introduction
For Linux platforms the LUNs presented from a SAN, NAS or SCSI devices are
identified with the path /dev/sdX. (/dev/sda, dev/sdb, etc). There is a common
issue when disks are removed from the system (due to I/O failures), causing a
change in the name (path) or changing the permissions.
Example: having /dev/sda, /dev/sdb, /dev/sdc, if disk /dev/sdb fails on the next
reboot the system will discover /dev/sda and /dev/sdb, but /dev/sdb had the
content originally referenced by /dev/sdc.
Single instance or RAC environments are exposed to this situation. If the

devices are binded to raw, it won’t change because there is a static file
(/etc/sysconfig/rawdevices or /etc/rc.local), where the bindind is executed. At
the application level it could be a problem. For example, Oracle, will reference
/dev/raw/raw2 for the OCR or Voting disk, but starting CRS will fail because of
the content of the device is not the OCR or the voting disk.
Similar situation could apply to disks used by ASM where after the reboot the
disk is not an ASM disk or because the permissions were not correct and the
device can not be read.
ASMLIB is not a problem because it reads the content of the header and creates
the block device under /dev/oracleasm/disks after finding the correct ASMLIB
label and adjust the correct permissions.
The solution to avoid this type of situations is to guarante concistent names for
the devices and correct permissions through reboots. Linux 2.6 introduced udev
as a mechanism to dynamically manage all type of devices, including disks.
This document presents how to configure udev and create persistent names and
use these names in a RAC configuration, for OCR, Voting disks and ASM disks.

What is UDEV
The best definition comes from the man page:
udev provides a dynamic device directory containing only the files for actually
present devices. It creates or removes device node files usually located in the
/dev directory, or it renames network interfaces.
As part of the hotplug subsystem, udev is executed if a kernel device is added or

removed from the system. On device creation, udev reads the sysfs directory of
the given device to collect device attributes like label, serial number or bus
device number.
These attributes may be used as keys to determine a unique name for the
device. udev maintains a database for devices present on the system. On
device removal, udev queries its database for the name of the device file to be
deleted.
Configuring udev
Udev is configured basically changing two major components, the rules

configuration and permissions configuration.
We will find these files under /etc/udev
[root@jfrac2 udev]# ls
devices permissions.d rules.d scripts udev.conf
The general configuration information is defined on file udev.conf where it is

defined the location of the configuration files, udev database,logging and other
attributes.
# udev_root - where in the filesystem to place the device nodes

udev_root="/dev/"
# udev_db - The name and location of the udev database.
udev_db="/dev/.udev.tdb"
# udev_rules - The name and location of the udev rules file
udev_rules="/etc/udev/rules.d/"
# udev_permissions - The name and location of the udev permission file
udev_permissions="/etc/udev/permissions.d/"
# udev_log - set to "yes" if you want logging, else "no"
udev_log="yes"

Setting the rules
The rules basically determine how to identify the disks and how to assign the
name that will be persistent through reboots or disk changes. The rule consist of
different fields:
BUS Match the bus type of the device. (The sysfs device bus must be able to
be determined by a "device" symlink. Examples of BUS will be scsi, usb
KERNEL Match the kernel device name. This is basically the name assgined by
the system to the device. LUNs usually are named /dev/sdX.
PROGRAM Call external program. This key is valid if the program returns
successful. The environment variables of udev are also available for the
program.
RESULT Match the returned string of the last PROGRAM call. This key may be
used in any following rule after a PROGRAM call.
NAME The name of the node to be created, or the name, the network interface
should be renamed to.
The following is a line from file 50-udev.rules:
KERNEL="sd?10", BUS="scsi", PROGRAM="/sbin/scsi_id",

RESULT="SSEAGATE ST39173WC LMB9126900007922HQKH",
NAME="ocr"
The program used to identify the unique disk is scsi_id which will report the
UUID name of the disk. This is a unique identification name for every disk.
To obtain this unique identifier, execute scsi_id -g -s /block/sdc. This is the value
to be used on the RESULT field. Replace sdc with every disk present on the
box.
The other field BUS, for LUNS presented to the machine (SAN,NAS,scsi) the
value is scsi.
The KERNEL field has the name used by the system. This example will search
all the sd devices, specifically the 10th partition. When the value specified under
result filed matches the value returned by scsi_id command, then a block device
/dev/ocr will be created.
There are external references to Udev rules

(http://reactivated.net/writing_udev_rules.html#tips) like :
“If applicable, make sure you identify the difference between sdX and
sdX1. This can be done with the key KERNEL="sd?1" to match sdX1, or
KERNEL=="sd?" to match sdX.”
In this test were used 4 disks of 9gb each, creating around 10 slices of 1gb on
each disk. In a production environment there is not recommended having
multiple slices on the same disk, so the value for kernel field will be sd?1 or sd?
only.
Setting the permissions.
A common error is that after rebooting the server, the database or CRS can not
be started and the diagnostic shows the cause was permissions of the devices.
Udev protects against this type of situations.
Under /etc/udev/permissions.d there is a file 50-udev.permissions, where you

define the permissions applied to the devices discovered by udev.
The enty has the following syntax
<name>:<owner>:<group>:<xxxx> where xxxx is the bits for the r/w permissions.
Example:
voting:usupport:dba:0660
ocr:root:dba:0640
asmdisk*:usupport:dba:0660

Changes on OEL 5, RHEL 5 or SLES 10
Starting on this release, there is not need to manage the rules and permissions
on different directories/files. The permissions are defined as part of the rule
under /etc/udev/rules.d.
Also there was a change on how to define the rules. Now, UDEV clauses with
"==" are attribute comparisions and clauses with "=" are attribute assignments.
Here an example of the rule on RHEL 5:
KERNEL=="sd?7", BUS=="scsi", PROGRAM="/sbin/scsi_id",

RESULT=="14f70656e66696c000000000001000000780d00000f000000",
NAME="asm1_disk3",GROUP="dba",OWNER="usupport",MODE="0660"

Additional configuration
In order to obtain the UUID when using command scsi_id, it is required to

modify /etc/scsi_id config file, This tells udev to assume that all the SCSI
devices on the system will return a UUID.
# The first step would be to acquire UUIDs. Open the /etc/scsi_id.config

file.
# Comment out the line:
# options=-b
# In its place, add the following line:
options=-g
Before creating the rules, obtain the UUID for each device:
# To get the UUID, run this command:
# scsi_id -g -s /block/sdc or (scsi_id –p 0x80 –s /block/sdc)
To discover new udev devices:

#start_udev
When logging is enabled after changing parameter udev_log="yes within

file /etc/udev/udev.conf, the following lines will be present in
/var/log/messages file:
May 30 02:03:24 jfrac2 udev[18105]: configured rule in

'/etc/udev/rules.d/50-udev.rules' at line 60 applied, 'sdd10' becomes 'ocr'
That message will be repeated for every rule specified within file
/etc/udev/rules.d/50-udev.rules file.

As we know the persistent names to be created by udev, use ls command
to verify if the devices were created under /dev directory. For this
scenario:
[root@jfrac2 udev]# ls -l /dev/asm*

brw-rw---- 1 usupport dba 8, 1 May 29 22:12 /dev/asmdisk1
[root@jfrac2 udev]# ls -l /dev/ocr

brw-rw---- 1 root dba 8, 58 May 29 22:12 /dev/ocr
[root@jfrac2 udev]# ls -l /dev/voting
brw-rw---- 1 usupport dba 8, 43 May 29 22:12 /dev/voting

Additional notes
Configuration
The storage array is a Sun scsi array holding 4 disks of 9gb.

To simulate multiple disks for ASM, the disks were divided using 1gb
partitions.
When having multiple partitions on the disk, all will return same UUID.
That is why it requires using the specific syntax for the KERNEL field. For
example, for the OCR and Voting disk, we picked the 10th partition of the
device. That is the reason to use KERNEL="sd?10. It will read always
the 10th partition of the device and if the UUID is the value specified in
result field, then the /dev/ocr path will be created.
Installting CRS and not recognizing the voting disk and ocr
Installing Oracle Clusterware get an error because the API used by Oracle
Installer does not recognize block devices, only character devices (/dev/raw/raw).
Bug 5196176
The workaround is this:
Before starting the installation, bind the paritition used by OCR and Voting
disks to the raw device.
At the last stage of the installation, it requires executing shell script
$ORA_CRS_HOME/root.sh. Before executing this shell, modify file
$ORA_CRS_HOME/install/rootconfig.sh.
This file has a reference to the Voting disk and OCR:
CRS_OCR_LOCATIONS=/dev/raw/raw1
CRS_VOTING_DISKS=/dev/raw/raw2
Modify this lines with the names created by udev.
CRS_OCR_LOCATIONS=/dev/ocr
CRS_VOTING_DISKS=/dev/voting
Now, execute root.sh

Appendix A. Files examples.
/etc/udev/rules.d/50-udev.rules (before OEL5/RHEL5/SLES10)
KERNEL="sd?10", BUS="scsi", PROGRAM="/sbin/scsi_id", RESULT="SSEAGATE ST39173WC

LMB9126900007922HQKH", NAME="ocr"
LMB7488600007923H2WH", NAME="voting"
KERNEL="sd?1", BUS="scsi", PROGRAM="/sbin/scsi_id", RESULT="SIBM DNES-318350Y
AKG1E241", NAME="asmdisk1"
LMB7488600007923H2WH", NAME="asmdisk6"
/etc/udev/permissions.d/50-udev.permissions (before
OEL5/RHEL5/SLES10)
voting:usupport:dba:0660
ocr:root:dba:0640
asmdisk*:usupport:dba:0660
/etc/udev/rules.d/50-udev.rules under OEL5/RHEL5/SLES10

RESULT=="14f70656e66696c000000000001000000780d00000f000000",
NAME="ocr1",GROUP="dba",OWNER="root",MODE="0640"
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
NAME="voting1",GROUP="dba",OWNER="usupport",MODE="0644"
RESULT=="14f70656e66696c000000000001000000780d00000f000000",

RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
RESULT=="14f70656e66696c000000000001000000780d00000f000000",
May 2006
Author: Jaime Figueroa
Oracle Corporation
World Headquarters
500 Oracle Parkway
Redwood Shores, CA 94065
U.S.A.
Worldwide Inquiries:
Phone: +1.650.506.7000
Fax: +1.650.506.7200
oracle.com
Copyright © 2006, Oracle. All rights reserved.
This document is provided for information purposes only and the
contents hereof are subject to change without notice.
This document is not warranted to be error-free, nor subject to any
other warranties or conditions, whether expressed orally or implied
in law, including implied warranties and conditions of merchantability
or fitness for a particular purpose. We specifically disclaim any
liability with respect to this document and no contractual obligations
are formed either directly or indirectly by this document. This document
may not be reproduced or transmitted in any form or by any means,

electronic or mechanical, for any purpose, without our prior written permission.
Oracle, JD Edwards, and PeopleSoft, are registered trademarks of Oracle
Corporation and/or its affiliates. Other names may be trademarks
of their respective owners

Using Udev RH

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Using Udev RH

Diunggah oleh

Hak Cipta:

Format Tersedia

Using udev with Oracle Architecture (RAC & ASM)

Providing persistent names on Linux 2.6 Red Hat

Using udev with Oracle Architecture (RAC & ASM) ............. 1

Using udev with Oracle Architecture (RAC & ASM) - 1 -

Single instance or RAC environments are exposed to this situation. If the

Using udev with Oracle Architecture (RAC & ASM) - 2 -

The best definition comes from the man page:

As part of the hotplug subsystem, udev is executed if a kernel device is added or

Udev is configured basically changing two major components, the rules

We will find these files under /etc/udev

The general configuration information is defined on file udev.conf where it is

# udev_root - where in the filesystem to place the device nodes

Using udev with Oracle Architecture (RAC & ASM) - 3 -

The following is a line from file 50-udev.rules:

KERNEL="sd?10", BUS="scsi", PROGRAM="/sbin/scsi_id",

There are external references to Udev rules

Using udev with Oracle Architecture (RAC & ASM) - 4 -

Setting the permissions.

Under /etc/udev/permissions.d there is a file 50-udev.permissions, where you

The enty has the following syntax

<name>:<owner>:<group>:<xxxx> where xxxx is the bits for the r/w permissions.

Using udev with Oracle Architecture (RAC & ASM) - 5 -

Here an example of the rule on RHEL 5:

KERNEL=="sd?7", BUS=="scsi", PROGRAM="/sbin/scsi_id",

Using udev with Oracle Architecture (RAC & ASM) - 6 -

In order to obtain the UUID when using command scsi_id, it is required to

# The first step would be to acquire UUIDs. Open the /etc/scsi_id.config

# In its place, add the following line:

# To get the UUID, run this command:

# scsi_id -g -s /block/sdc or (scsi_id –p 0x80 –s /block/sdc)

To discover new udev devices:

When logging is enabled after changing parameter udev_log="yes within

May 30 02:03:24 jfrac2 udev[18105]: configured rule in

Using udev with Oracle Architecture (RAC & ASM) - 7 -

[root@jfrac2 udev]# ls -l /dev/asm*

[root@jfrac2 udev]# ls -l /dev/ocr

Using udev with Oracle Architecture (RAC & ASM) - 8 -

The storage array is a Sun scsi array holding 4 disks of 9gb.

The workaround is this:

This file has a reference to the Voting disk and OCR:

Modify this lines with the names created by udev.

Now, execute root.sh

Using udev with Oracle Architecture (RAC & ASM) - 9 -

/etc/udev/rules.d/50-udev.rules (before OEL5/RHEL5/SLES10)

KERNEL="sd?10", BUS="scsi", PROGRAM="/sbin/scsi_id", RESULT="SSEAGATE ST39173WC

/etc/udev/rules.d/50-udev.rules under OEL5/RHEL5/SLES10

KERNEL=="sd?14", BUS=="scsi", PROGRAM="/sbin/scsi_id",

Using udev with Oracle Architecture (RAC & ASM) - 10 -

Using udev with Oracle Architecture (RAC & ASM) - 11 -

Using udev with Oracle Architecture (RAC & ASM) - 12 -

Anda mungkin juga menyukai