Automatic Storage Management (ASM) Best Practices OUGN 27.05.

2009
Marie-Louise Lindstrm Service Delivery Engineer Oracle Support ACS

Agenda

Concepts, Architecture New Features in Database Release 11g Tips and Techniques Examples, Demo

ASM Architecture - Overview

1.

Automatic Storage Management (ASM) instance Oracle instance that manages the diskgroup metadata ASM Disk Groups Logical grouping of disks Determines file mirroring options ASM Disks LUNs presented to ASM ASM Files Files that are stored in ASM disk groups are called ASM files, this includes database files

2.

3.

4.

ASM Architecture - Overview

ASM provides filesystem and volume manager capabilities built into the Oracle Database kernel Provides a simple storage management interface across all server and storage platforms Discover and associate (adding) disks under ASM management A disk can be partition of a physical spindle, the entire spindle, or a LUN in a RAID group set Logical volumes can be presented to ASM as disks, but not a good practice, since an extra layer is added to IO code path, and thus some level of overhead

Key Benefits of ASM

I/O is spread evenly across all available disk drives to prevent hot spots and maximize performance ASM eliminates the need for over provisioning and maximizes storage resource utilization facilitating database consolidation Performs automatic online redistribution after the incremental addition or removal of storage capacity Maintains redundant copies of data to provide high availability, or leverage 3rd party RAID functionality Capable of leveraging 3rd party multi-pathing technologies

Key Benefits of ASM (cont.)

An Oracle Database can contain ASM and non-ASM files. Any new files can be created as ASM files whilst existing files can also be migrated to ASM RMAN commands enable non-ASM managed files to be relocated to an ASM disk group Enterprise Manager can be used to manage ASM disk and file management activities ASM reduces Oracle Database cost and complexity without compromising performance or availability

Software Stack With and Without ASM

APPL1

APPL2

APPL3 DB2

APPL1

APPL2

APPL3

DB1 File System

DB1 ASM Logical Volume Manager

DB2

OPERATING SYSTEM

OPERATING SYSTEM

Disk Group Dynamic Rebalancing

Automatic online rebalancing whenever storage configuration changes Only move data proportional to storage added No need for manual I/O tuning Online migration to new storage

Mirroring And Striping

Mirroring : Redundant copies of data over multiple disks Primary data extents Secondary data extents

Striping : Spreading data over multiple disks

Unstriped Disks

Striped Disks

ASM Instance

Oracle instance to manage the metadata for disk groups All metadata modifications are done by the ASM instance Database instances connect to ASM instance to create, delete, resize, open, or close files Database instances read/write directly to the disks One ASM instance is required per node The ASM instance comprises shared memory structure (SGA) and background processes The ASM instance contains no physical files other than the password file and the parameter file Instance alert log file and trace files in the admin directories (release 10g), or diag directory (release 11g)

ASM Instance (cont.)

Works with the RAC architecture to support clustered storage environments In RAC environments, there will be one ASM instance per clustered node, and the ASM instances communicate with each other on a peer-to-peer basis using the interconnect Generally named +ASM, in a RAC environment +ASM1, +ASM2, etc

ASM Instance (cont.)

instance_type=asm asm_diskstring asm_diskgroups asm_power_limit sga_target or memory_target ; used in ASM release 11.1 db_cache_size=64M ; The buffer cache area is used to cache metadata blocks shared_pool=12M ; Used for standard memory usage (control structures, etc.) to manage the instance large_pool=128M ; Used to store extent maps processes Background processes
RBAL This process performs global opens of all the disks in the disk groups, manages rebalancing operations ASMB Periodic messages are exchanged to update statistics and provide a heartbeat mechanism O00x A group of slave processes establish connections to the ASM instance

ASM Diskgroups

Highest-level data structure in ASM Comparable to a LVMs volume group

SQL> create diskgroup <name> disk <path>,<path>; When the diskgroup has been created, metadata information, which includes creation date, diskgroup name, and redundancy type, is stored in the SGA and on each disk (in disk header) within the diskgroup A disk and a database file can only be part of one diskgroup A database can store its files in multiple diskgroups Recommended to have two diskgroups: Database area Flash recovery area

ASM Disks
Unit of persistent storage for a disk group Accessed through normal OS interfaces Oracle must have read/write access Must be accessible to all nodes of a cluster May have different names on different nodes Cannot be a local filesystem file Must not include partition table Unit that fails if using redundancy Disks in a disk group are the same percent full Best practice is to have disks of same capacity (size, throughput) belonging to the same diskgroup ASM must scale to support at least 10,000 disks in a disk group

Allocation Unit

ASM Disk

1 MB

ASM disks are divided into allocation units Unit of allocating storage to a file 1 MB in size ; release 10g 1, 2, 4, 8, 16, 32, 64 MB ; release 11g Small enough to not be hot Large enough for efficient sequential access Many allocation units in a single file

Redundancy and Failure Groups

External redundancy No mirroring, no failure groups Normal redundancy ; default Files are two-way mirrored, requiring at least two failure groups High redundancy Using three-way mirroring, requiring at least three failure groups Redundancy level set at creation time A failure group, a subset of a diskgroup, is a collection of disks that can become unavailable due to a failure of one of its associated components; e.g., controllers or entire arrays Disks in two separate failure groups (for the same diskgroup) must not share a common failure component The primary extent and its mirror copy are always stored in separate failure groups

Redundancy Levels

External redundancy data exists only once Normal redundancy two-way mirroring one additional extent copy at least two failure groups High redundancy three-way mirroring two additional copies at least three failure groups

ASM Files

Datafiles, control files, spfiles, and redo log files, incl archived log files, can be stored in ASM diskgroups Rman backups ASM file names are in the format of

+diskgroup_name/database_name/file_type/tag_name.file_n umber.incarnation ASM Alias name format: instead of a file and incarnation number, a user-friendly string name is used For best practices, implement the Oracle Managed File (OMF) feature Log files, trace files and executables are stored outside ASM diskgroups

ASM Rebalance and Redistribution

ASM allocates space in units called allocation units (AU) A database file is broken up into file extents (coarse and fine distribution) Changing the storage configuration, for example adding a disk to a diskgroup, will trigger a rebalance operation The objective of the rebalance operation is to provide an even distribution of file extents and space usage across all disks in the diskgroup Rebalancing involves physical movement of file extents If removing or adding several disks, it is best to add or remove drives all at once, this will reduce the number rebalance operations that are needed for storage changes

ASM Views

V$ASM_DISK V$ASM_DISKGROUP V$ASM_DISK_STAT V$ASM_DISKGROUP_STAT V$ASM_OPERATION V$ASM_CLIENT V$ASM_FILE V$ASM_ALIAS V$ASM_TEMPLATE

ASMCMD Utility
Command line interface providing access to the files and directories within the ASM diskgroups

$ export ORACLE_SID=+ASM $ asmcmd ASMCMD> lsdg State Type Rebal Sector Block AU Usable_file_MB Offline_disks Name MOUNTED NORMAL N 512 4096 1048576 111 0 DATA1/ MOUNTED EXTERN N 512 4096 1048576 885 0 FRA1/ ASMCMD> cd data1 ASMCMD> ls X1020/ X1110/ ASMCMD> cd X1110/ ASMCMD> ls CONTROLFILE/ DATAFILE/ ONLINELOG/ PARAMETERFILE/ TEMPFILE/ spfileX1110.ora ASMCMD> ls DATAFILE/ SYSAUX.260.687886011 SYSTEM.261.687886193 UNDOTBS1.262.685476185 USERS.264.681997355 ASMCMD>

Total_MB 8188 2047

Free_MB 2269 885

Req_mir_free_MB 2047 0

Migrating to ASM

ASM migration using Data Guard physical standby Use this method if your requirement is to minimize downtime during the migration. It is possible to reduce total downtime to just seconds by using the best practices described in the white paper ASM migration using Rman A simpler approach, but one that can result in downtime measured in minutes to hours, depending on the method used for migration ASM migration using DBMS_FILE_TRANSFER

Copying Files To and From ASM Diskgroups

1. 2. 3. 4.

Across ASM diskgroups Between the OS file system and ASM diskgroups Using Rman Using the DBMS_FILE_TRANSFER package Through ftp to the XML DB Virtual Folder Using the ASMCMD command cp in release 11g

ASM New Features in Release 11g

ASM Fast Mirror Resync ASM Preferred Mirror Read ASM Rolling Upgrades and Patches ASM Scalability and Performance Enhancements Multiple Allocation Unit Size SYSASM Privilege ASMCMD Extensions To use the new diskgroup features, the disk group compatibility attribute must be set to 11.1

Without ASM Fast Mirror Resync

1

ASM redundancy used

Disk access failure

Oracle Database 10g and 11g

Disk added back: Extents rebalanced

Disk automatically dropped: All dropped extents re-created

With ASM Fast Mirror Resync

1

ASM redundancy used

Disk access failure

Oracle Database 11g

Disk again accessible: Auto resync modified extents

4

Failure time < DISK_REPAIR_TIME> Access other extents

ASM Fast Mirror Resync

Define DISK_REPAIR_TIME
ALTER DISKGROUP dgroupA SET ATTRIBUTE 'DISK_REPAIR_TIME'='3H'; ALTER DISKGROUP dgroupA OFFLINE DISKS IN FAILGROUP contrl2 DROP AFTER 5H;

Check time left before ASM drops offlined disk

SELECT REPAIR_TIMER FROM V$ASM_DISK;

Online disk after completing maintenance

ALTER DISKGROUP dgroupA ONLINE DISKS IN FAILGROUP contrl2 POWER 2 WAIT;

ASM Preferred Mirror Read

Allow local mirror read operations Eliminate network latencies in extended clusters
Site A Site B

Extended Cluster

P
ASM_PREFERRED_READ_ FAILURE_GROUPS=DATA .SITEA

S
ASM_PREFERRED_READ_ FAILURE_GROUPS=DATA .SITEB

SELECT preferred_read FROM v$asm_disk; SELECT * FROM v$asm_disk_iostat;

ASM Rolling Upgrades and Patches

Maximizes database availability in a cluster How it works Place cluster in Rolling Migration mode Bring down ASM on a cluster node Upgrade or patch software Restart ASM Stop Rolling Migration mode after upgrading all nodes

ASM Scalability and Performance Enhancements

Variable size extents Grows automatically with file size Benefits Increase ASM file size Reduce memory utilization in SGA 100% automatic
63 disk groups 10,000 ASM disks 4 petabyte per ASM disk 40 exabyte of storage 1 million files per disk group Maximum file size: External redundancy: 140 PB Normal redundancy: 42 PB High redundancy: 15 PB

SYSASM Privilege
SYSASM role to manage ASM instances separates duty of DBAs and storage administrators
SQL> CONNECT / AS SYSASM SQL> CREATE USER ossysasmusername IDENTIFIED by passwd; SQL> GRANT SYSASM TO ossysasmusername; SQL> CONNECT ossysasmusername / passwd AS SYSASM; SQL> DROP USER ossysasmusername;

SYSDBA will be deprecated for ASM management: Oracle Database 11g Release 1 behaves as in 10g In future releases SYSDBA privileges restricted in ASM instances

Multiple Allocation Unit Size

Allocation Unit (AU) selected at disk group creation time and may be 1,2,4,8,16,32,64 MB Set Oracle MAX_IO_SIZE = AU size Default MAX_IO_SIZE is 1MB MAX_IO_SIZE determines default AU size Striping coarse stripe size = 1 AU fine stripe size = 128KB Higher performance for large sequential I/O (DW) Better leverage of Hardware RAID read-ahead

ASMCMD Extensions
User created directories Templates Disk group compatibility Disk group name Disk names and failure groups

md_backup md_restore

repair

$ asmcmd help

cp

Repair Bad Blocks

Copy files
lsdsk

List ASM disks

Tips and Techniques Best Practices

Should ASM be installed in a separate Oracle_Home? Yes, it is recommended to install and run ASM in a separate Oracle_Home. Use the latest RDBMS version, currently version 11.1.0.7. For release 10g, use version 10.2.0.4 plus patch for defect 6163771. Where do I run the database listener from, the ASM Oracle_Home or the Database Oracle_Home? It is recommended to run the listener from the ASM Oracle_Home, or a separate Oracle_Home. This is particularly important for RAC installations since the listener is a node-level resource.

Tips and Techniques Best Practices (cont.)

How to separate ASM administrators from DBAs? Choose a different OSDBA group for the ASM installation (asmdba) than for the Database installation (dba). The Database instance connects to ASM using OSDBA group of the ASM instance, which means that the software owner for each Database instance connecting to ASM must be a member of ASM's OSDBA group. In 11g, ASM administrator has to be member of a separate SYSASM group to separate ASM administrators and DBAs.

Tips and Techniques Best Practices (cont.)

Can the Database instance and the ASM instance run different versions? Yes, the ASM instance can be at a higher version or a lower version than its client databases. There are two components of compatibility: 1. Software compatibility The compatible instance parameter 2. Diskgroup compatibility attributes: compatible.asm compatible.rdbms

Tips and Techniques Best Practices (cont.)

What instance parameters need to be configured for ASM instances? The default parameter settings work perfectly for ASM release 11g. The only parameters needed for ASM release 11g: instance_type asm_diskstring asm_diskgroups

Does the FILESYSTEMIO_OPTIONS parameter need to be configured in the Database instance when using ASM? No, the Database instance does I/O directly to the raw disk devices, whereas the FILESYSTEMIO_OPTIONS parameter is only for file systems

Tips and Techniques Best Practices (cont.)

How many diskgroups should I have? The general recommendation is two diskgroups: 1. Database area 2. Flash recovery area What redundancy level should be used? Use normal redundancy level, or high redundancy level, if the underlying storage system does not provide mirroring. You can rely on external redundancy level if the underlying storage system provides mirroring and you determine that failgroups are not needed.

Tips and Techniques Best Practices (cont.)

What size would be a good allocation unit (AU) for disk groups? In 10g, the allocation unit size is, by default, set to 1 MB. In 11g, it is recommended to use 4 MB. What is the best LUN size for ASM? There is no generic best size, in most cases the storage team will dictate based on their standardized LUN size. Communicate the ASM Best Practices and application characteristics to the storage team: Need equally sized / performance LUNs Do not use LUNs larger than 2 TB (minimum of 4 MB) The capacity requirement The workload characteristic (random r/w, sequential r/w) & any response time SLA

Tips and Techniques Best Practices (cont.)

Would it be better to use BIGFILE tablespaces, or default (SMALLFILE) tablespaces for ASM? The use of BIGFILE tablespaces has no bearing on ASM, or vice versa. In fact, most database object related decisions are transparent to ASM. How to backup the ASM instance? All the metadata the ASM instance needs to startup is on on-disk structures (in the disk headers). There are no datafiles or control files that need to be backed up. The only file you can consider to back up is the instance parameter file. In 11g, use the ASMCMD md_backup to take a copy of the metadata.

Tips and Techniques Best Practices (cont.)

When should I use RMAN and when should I use ASMCMD copy? RMAN is the recommended and most complete and flexible method to backup and transport database files in ASM. ASMCMD cp is good for copying single files Supports all Oracle file types Can be used to instantiate a Data Guard environment Does not update the controlfile Does not create OMF files

Tips and Techniques Best Practices (cont.)

Is it possible to unplug an ASM disk group from one platform and plug into a server on another platform, for example from Solaris to Linux)? No, cross-platform disk group migration is not supported. To move datafiles between endian-ness platforms, you need to use XTTS, Datapump or Streams. How does ASM work with multipathing software? Oracle does not certify or qualify multi-pathing tools. ASM does, however, leverage multi-pathing tools. Use multi-pathing if the underlying storage supports it. Multipathing software is at a layer lower than ASM, and thus transparent to ASM. You may need to adjust the ASM_DISKSTRING parameter to specify only the path to the multipathing pseudo devices.

Tips and Techniques Best Practices (cont.)

AIX specific information: Do not assign a PVID to ASM disks Assigning a Physical Volume ID (PVID) to an existing ASM disk will destroy the ASM disk header rendering the ASM disk unusable. How to repair damaged disk headers? If you encounter an error situation caused by corrupt disk headers leading to invalid ASM metadata, open a service request with Oracle Support to get technical assistance from the ASM team. Depending on the extent of the damage, it may be possible to manually repair the disk headers. However, if internal structures cannot be patched, the only solution will be to recreate the diskgroup and restore data from backup. Make sure you always have a good backup of your data. Also, store backup copies outside ASM diskgroups.

Tips and Techniques Best Practices (cont.)

Should ASMLib be used on Linux installations? Yes, this is the recommendation. The ASMLib interface is not required to run ASM, however, it provides easier management of disk discovery and provisioning. Set the ASM_DISKSTRING parameter to ORCL:.

References
Oracle Database Storage Administrator's Guide 11g Release 1 (11.1) B31107-05
http://download.oracle.com/docs/cd/B28359_01/server.111/b31 107.pdf

ASM Collateral and Content

http://www.oracle.com/technology/asm

Best Practices for Minimal Downtime Migration to ASM Oracle 10g Release 2
http://www.oracle.com/technology/deploy/availability/pdf/ma a_wp_10gr2_asmmigrationwithdg.pdf

References (cont.)
ASM and Multipathing Generic Best Practices and Information Matrix March 2008
http://www.oracle.com/technology/products/database/asm/pdf/ asm%20and%20multipathing%20best%20practices%20info%20matrix %203-5-08.pdf

MyOracleSupport Note:265633.1 - ASM Technical Best Practices MyOracleSupport Note:751300.1 - How to perform Rolling Upgrade/Downgrade in 11g ASM