Compellent Corporate Office Compellent Technologies 7625 Smetana Lane Eden Prairie, Minnesota 55344
www.compellent.com
Contents
Contents .............................................................................................................................. 2 Preface ................................................................................................................................ 3 Customer Support ........................................................................................................... 3 Disclaimers ...................................................................................................................... 3 General Syntax ................................................................................................................ 3 Document Revision ......................................................................................................... 3 AIX 5.2, 5.3, 6.1 and 7.1 Best Practices ............................................................................. 4 IBM AIX Overview ........................................................................................................... 4 IBM AIX and Compellent Storage Center ....................................................................... 4 Fiber Channel Switches .................................................................................................. 5 Fiber Channel Connectivity ............................................................................................. 5 Dynamic Tracking with IBM Fiber Channel Cards .......................................................... 5 Compellent Storage Center Legacy vs. Virtual Ports and AIX ........................................ 6 Fiber Channel Boot from Storage Center ........................................................................ 6 MPIO Multipath (MPIO debuted in AIX BOS 5L 5200-01................................................ 8 AIX with Storage Center Volumes without a Compellent ODM/PCM ............................. 8 AIX with Storage Center Volumes with the Compellent ODM/PCM ............................... 9 iSCSI Software Initiator Connectivity............................................................................. 10 Add iscsi file sets for aix 6.1 .......................................................................................... 10 Logical Volume Management ........................................................................................ 15 Migration Options .......................................................................................................... 16 Create a fiber channel SAN boot disk via migratepv..................................................... 17 AIX alt_disk_copy .......................................................................................................... 17 Clean off source server H/W information to create a Gold Copy .................................. 18 Making Storage Center Volumes Visible to AIX on the fly ............................................ 19 Mirror Creation and usage via AIX ................................................................................ 20 Migratepv usage in AIX ................................................................................................. 21 Growing File Systems online ......................................................................................... 22 Discovering the new space on a SAN volume which has been expanded from the Storage Center .............................................................................................................. 22 Replay Creation and Mapping a Local Recovery back to the same server. ................. 23 Replay Creation and Mapping a Local Recovery to a different server.......................... 23 Advanced POWER Virtualization (APV) Virtual Input Output Server (VIOS) ............... 24
Compellent Technologies
Page 2
Preface
Customer Support
Compellent provides live support 1-866-EZSTORE (866.397.8673), 24 hours a day, 7 days a week, and 365 days a year. For additional support, email Compellent at support@compellent.com. Compellent responds to emails during normal business hours.
Disclaimers
Information in this document is subject to change without notice. 2011 Compellent Technologies. All rights reserved. Reproduction in any manner without the express written permission of Compellent Technologies is strictly prohibited. Trademarks used in this text are property of Compellent Technologies, or their respective owners.
General Syntax
Item Menu items, dialog box titles, field names, keys Mouse click required User Input User typing required Website addresses Email addresses Convention Bold Click: Monospace Font Type: http://www.compellent.com info@compellent.com
Document Revision
Date March 2011 Revision 4.0 Description Fiber Channel and iSCSI Updates
Compellent Technologies
Page 3
IBM provides the P-Series of RISC (Reduced Instruction Set) CPU equipped servers along with the AIX operating system to provide robust and resilient server environments in a wide variety of enterprise configurations. AIX has a robust Logical Volume Manager built in to the operating system which provides a capability to manipulate and modify the hard disks the server is connected to. These disks may be local SCSI, Fiber Channel or iSCSI volumes presented from a variety of sources. The Compellent Storage Center provides AIX compatible disk volumes which appear as the familiar hdisk# when viewed from within AIX. The full range of AIX supplied utilities such as mirroring, backup, multiple file system types, multipath, boot from SAN and disaster recovery are used with Compellent volumes.
Compellent Technologies
Page 4
Static Domain IDs and Persistent FC IDs for Storage Center Ports
Cisco fabrics
Cisco fabric switches have Persistent FCID enabled by default and do not require any administrator intervention to use. https://www.cisco.com/en/US/docs/storage/san_switches/mds9000/sw/sanos/quick/guide/qcg_ids.html
Brocade fabrics
8Gb capable Brocade Fabric switches do not have Persistent WWN based PID assignment enabled by default and the minimum version to use is the 6.4 version of the Brocade Fabric Operating System (FOS).
Compellent Technologies
Page 5
Events that can cause an N_Port ID to change include moving a cable between a switch and storage device from one switch port to another, connecting two separate switches using an inter-switch link (ISL), rebooting a switch or during a Storage Center failover event for maintenance or due to a failure within the Storage Center. The dynamic tracking feature has a hard coded fifteen second timeout value set on the AIX server. Testing in the Compellent labs has shown that a heavily loaded Storage Center (More than 400 objects defined) can exceed the fifteen second timeout value of dynamic tracking resulting in lost connections between the AIX server and the Storage Center providing the volumes.
Fiber Channel Boot from Storage Center Multiple SAN boot options
IBM AIX provides the ability to install/boot from a single path to a Storage Center volume without the presence of the Compellent MPIO ODM Path Control Module. This section assumes there is not an existing internal disk on the AIX server. If there is an existing local hard disk which the server is currently using to run AIX, there are multiple options to either change to a Storage Center volume as the only boot volume, to migrate from an existing internal hdisk or to mirror the existing boot disk with a volume from the Storage Center. Please refer to the sections in this document on the migratepv, alt_disk_copy, and mirrorvg commands for AIX to understand some of the options available to an administrator.
Compellent Technologies
Page 6
Connections check box and allow the Storage Center to display all HBAs that the Storage Center has seen, whether the connection is active or not. Once the HBA has been mapped to the server object and an appropriately size Storage Center volume, the AIX installation media will list the Storage Center as a valid installation option via the normal installation utilities. Please note that at this point, the Storage Center is treated as a basic third party storage device. Do not map a second path to the Storage Center volume until after the operating system installation is completed successfully and the Compellent MPIO ODM Path Control Module is installed on the operating system boot disk.
Compellent Technologies
Page 7
Compellent Technologies
Page 8
Once the ODM/PCM is initially installed on the server and the server is rebooted, the output of the same command will show the Storage Center volumes as Compellent FC SCSI Disk Drive(s). #> lsdev -Cc disk hdisk0 Available 01-08-00-1,0 SCSI Disk Drive hdisk1 Available 01-10-01 Compellent FC SCSI Disk Drive The attributes of any future Storage Center Volume(s) mapped to this particular server have the queue_depth value set to 32 and the rw_timeout value set to 60 to allow time for the Storage Center Front End Port movement during a Storage Center maintenance event or in the unlikely event of a Storage Center controller failure. #> lsattr -HEl hdisk1 attribute value PCM algorithm queue_depth rw_timeout PCM/friend/compellent_sc round_robin 32 60
description Path Control Module Algorithm Queue DEPTH READ/WRITE time out
Compellent Technologies
Page 9
Verify the iSCSI target Node Name on the Storage Center by selecting Controllers, and then select the controller you want to use, select IO Cards, select iSCSI, select the iSCSI card you want to use, finally select General from the right window tab. Add the IP Address of the Compellent iSCSI card, the default iSCSI port number (3260 for all Compellent Devices), and iSCSI Node Name of the Storage Center to the end of the /etc/iscsi/targets file on the AIX server. When finished write and close the file. EXAMPLE: 172.31.32.103 3260 iqn.2002-03.com.compellent:5000d31000019301 Verify the iSCSI device is available in the kernel: # lsdev -C | grep iscsi iscsi0 Available iSCSI Protocol Device Verify the iSCSI Node Name of the AIX server: # lsattr -El iscsi0 disc_filename disc_policy initiator_name max_targets num_cmd_elems
/etc/iscsi/targets Configuration file False file Discovery Policy True iqn.hostid.0a3cc1dc iSCSI Initiator Name True 16 Maximum Targets Allowed True 200 Maximum number of commands to queue to driver True
Scan via the iSCSI port using the new information from the /etc/iscsi/targets file. # cfgmgr -l iscsi0 Create a new server and map the new iSCSI port of the AIX server to the new server. NOTE: you may need to unselect the Only Show Active/UP Connections box to see the new iSCSI item.
Compellent Technologies
Page 10
Create a Volume on the Storage Center and map it to the new server. Rescan the AIX iSCSI subsystem for new devices: # cfgmgr -l iscsi0 Verify that the iSCSI block devices have been created on the AIX server: # lsdev -Cc disk hdisk0 Available 10-60-00-4,0 16 Bit SCSI Disk Drive hdisk1 Available 10-60-00-5,0 16 Bit SCSI Disk Drive hdisk2 Available Other iSCSI Disk Drive
Software Initiator
Discover the iSCSI Control Port IP Address and iSCSI Virtual Port iqn on the Storage Center by selecting Controllers, and then select the controller you want to use, select IO Cards, select iSCSI. Record the IP address shown for the iSCSI Control Port and the iqn for the iSCSI Virtual Port. Edit the /etc/iscsi/targets file on the server with the IP address of the New Domain 6 iSCSI Control Port, the default port of 3260 and the iqn name of the New Domain 6 iSCSI Virtual Port Repeat for the second entry which consists of the IP address of the New Domain 7 iSCSI Control Port, the default port of 3260 and the iqn name of the New Domain 7 iSCSI Virtual Port as shown below. NOTE: Your Domain numbers and iqn numbers will be different than this example. Do not use the iSCSi physical adapter address(s) for this purpose.
#New Domain 6 iSCSI Control Port address 172.17.32.4 #Port is 3260 #New Domain 6 iSCSI Virtual Port iqn iqn.2002-03.com.compellent:5000d310000199a8 #172.17.32.4 3260 iqn.2002-03.com.compellent:5000d310000199a8 #New Domain 7 iSCSI Control Port address 172.17.34.4 #Port is 3260 #New Domain 7 iSCSi Virtual Port iqn.2002-03.com.compellent:5000d310000199a9 # 172.17.32.4 3260 iqn.2002-03.com.compellent:5000d310000199ac 172.17.32.4 3260 iqn.2002-03.com.compellent:5000d310000199a8 172.17.34.4 3260 iqn.2002-03.com.compellent:5000d310000199ad 172.17.34.4 3260 iqn.2002-03.com.compellent:5000d310000199a9
Compellent Technologies
Page 11
When finished write and close the file. Scan via the iSCSI port using the new information from the /etc/iscsi/targets file. # lsdev | grep iscsi iscsi0 iscsi1 Available 02-09-01 Available 03-09-01 iSCSI Protocol Device iSCSI Protocol Device
# cfgmgr -l iscsi0 Method error (/usr/lib/methods/cfgqliscsi -l iscsi0 ): 0514-061 Cannot find a child device. # cfgmgr -l iscsi1 Method error (/usr/lib/methods/cfgqliscsi -l iscsi1 ): 0514-061 Cannot find a child device. On the Storage Center, create a new server and map the new iSCSI port of the AIX server to the new server. NOTE: you may need to unselect the Only Show Active/UP Connections box to see the new iSCSI item. Create a Volume on the Storage Center and map it to the new server. Rescan the AIX iSCSI subsystem for new devices: # cfgmgr -l iscsi0 Verify that the iSCSI block devices have been created on the AIX server: # lsdev -Cc disk hdisk0 Available 10-60-00-4,0 16 Bit SCSI Disk Drive hdisk1 Available 10-60-00-5,0 16 Bit SCSI Disk Drive
Compellent Technologies
Page 12
To see the iqn name and current settings of the iSCSI TOE cards: # lsattr -HE -l ics0
Copy the /etc/iscsi/targetshw file to the /etc/iscsi/targetshw0 to configure the first iSCSI TOE card called iscsi0. Set the following: Discovery Filename to Discovery Policy to Adapter IP Address Adapter Subnet Mask Adapter Gateway
/etc/iscsi/targetshw0 file Desired network address Desired Netmask address Desired Gateway
Compellent Technologies
Page 13
Copy the /etc/iscsi/targetshw file to the /etc/iscsi/targetshw1 to configure the second iSCSI TOE card called iscsi1. Set the following: Discovery Filename to Discovery Policy to Adapter IP Address Adapter Subnet Mask Adapter Gateway
/etc/iscsi/targetshw1 file Desired network address Desired Netmask address Desired Gateway
Compellent Technologies
Page 14
Compellent Technologies
Page 15
mode of SMIT (which supports a mouse and point-and-click operations) can be run only on a graphical display running an X Window manager. The ASCII mode is often the preferred method to run SMIT because it can be run from any machine.
Migration Options
UNIX cp command
The most basic way to move smaller multiple static data sets into the Storage Center is with the AIX command. The files written to the Storage Center volumes are thin provisioned. The R and p options are recursive and maintain file attributes. Please consult your AIX documentation for various cp options. # cp Rp /source /target The procedure is to create a new target volume on the Storage Center, mount the new volume on the AIX server and then copy the data to the new volume. Once the data has been copied, the source volume is unmounted from the server, the target volume is unmounted from the server and then the target is mounted into the original mount point of the source volume. The use of the original mount point for the new volume allows for the use of any existing path variables without modification. The original data can then be kept in reserve as a backup until the testing of the new data set is completed to the satisfaction of the administrator. Be aware that the copy command is a single threaded process and may take a substantial amount of time to complete based on the amount of data to be copied.
AIX dd command
The use of an image level copy technique such as the dd command will consume the same amount of the disk space on the Storage Center target volume which was allocated in the source volume whether there is data in the space or not. Therefore, to preserve the Dynamic Capacity advantages of the Storage Center, it is not recommended to use image level data transfer techniques when moving data from less advanced storage to the Storage Center.
Compellent Technologies
Page 16
# migratepv -l hd5 hdisk0 hdisk2 0516-1246 migratepv: If hd5 is the boot logical volume, please run 'chpv -c hdisk0' as root user to clear the boot record and avoid a potential boot off an old boot image that may reside on the disk from which this logical volume is moved/removed. # chpv -c hdisk0 # bosboot -ad /dev/hdisk2 bosboot: Boot image is 29924 512 byte blocks. # migratepv hdisk0 hdisk2 # reducevg -d rootvg hdisk0 # bootlist -m normal hdisk2 Reboot server
AIX alt_disk_copy
Clone local boot disk (hdisk#) to Compellent Storage Center volume (hdisk1) and run the bootlist command to change the boot order so the san volume is the boot disk (Five minutes to complete in our test bed) # alt_disk_copy -d hdisk1 Calling mkszfile to create new /image.data file. Checking disk sizes. Creating cloned rootvg volume group and associated logical volumes. Creating logical volume alt_hd5. Creating logical volume alt_hd6. Creating logical volume alt_paging00. Creating logical volume alt_hd8. Creating logical volume alt_hd4. Creating logical volume alt_hd2. Creating logical volume alt_hd9var. Creating logical volume alt_hd3. Creating logical volume alt_hd1.
Compellent Technologies
Page 17
Creating logical volume alt_hd10opt. Creating logical volume alt_lg_dumplv. Creating /alt_inst/ file system. /alt_inst filesystem not converted. Small inode extents are already enabled. Creating /alt_inst/home file system. /alt_inst/home filesystem not converted. Small inode extents are already enabled. Creating /alt_inst/opt file system. /alt_inst/opt filesystem not converted. Small inode extents are already enabled. Creating /alt_inst/tmp file system. /alt_inst/tmp filesystem not converted. Small inode extents are already enabled. Creating /alt_inst/usr file system. /alt_inst/usr filesystem not converted. Small inode extents are already enabled. Creating /alt_inst/var file system. /alt_inst/var filesystem not converted. Small inode extents are already enabled. Generating a list of files for backup and restore into the alternate file system... Backing-up the rootvg files and restoring them to the alternate file system... Building boot image on cloned disk. forced unmount of /alt_inst/var forced unmount of /alt_inst/var forced unmount of /alt_inst/usr forced unmount of /alt_inst/usr forced unmount of /alt_inst/tmp forced unmount of /alt_inst/tmp forced unmount of /alt_inst/opt forced unmount of /alt_inst/opt forced unmount of /alt_inst/home forced unmount of /alt_inst/home forced unmount of /alt_inst forced unmount of /alt_inst Changing logical volume names in volume group descriptor area. Fixing LV control blocks... Fixing file system superblocks... Bootlist is set to the boot disk: hdisk1 blv=hd5 Reboot the server to use the Storage Center volume as the boot disk.
Compellent Technologies
Page 18
generate a local volume recovery on the Storage Center for use as a bootable hdisk for other servers. As always, make sure you have the proper licenses for your AIX operating system. The -B (as in Bravo) option tells the AIX system to NOT change the current boot disk. This allows for the current boot disk to remain the active boot disk and to retain all of its hardware and user defined specifics. To create a hardware neutral image on Storage Center volume hdisk7 and to NOT change the existing boot disk parameters, the following command can be run: # alt_disk_copy B O d hdisk7
Compellent Technologies
Page 19
Compellent Technologies
Page 20
You can use the migratepv command to move data from one physical volume to another physical volume within the same volume group. Note: You can specify more than one destination physical volume. First, identify the source disk from which you want to migrate the data. Then, identify the target disk to which you want to migrate the data. You can only migrate to disks that are already in the rootvg volume group. To get a list of disks in the rootvg volume group, run the lsvg -p rootvg command. # lsvg -p rootvg rootvg: PV_NAME PV STATE TOTAL PPs FREE PPs FREE DISTRIBUTION hdisk0 active 515 116 57.00..00..00..59 hdisk1 active 515 515 00.00..00..00..00
Now, determine the space that is currently in use on the disk that you want to migrate. This is the total physical partitions (PPs) value minus the free PPs value for the desired disk. In the preceding example, refer to hdisk0, which is using (515 - 116) PPs or 399 physical partitions. Next, find a disk or disks that have the available space. In this case, hdisk1 has 515 free physical partitions, which is more than the required space of 399 physical partitions.
Compellent Technologies
Page 21
Discovering the new space on a SAN volume which has been expanded from the Storage Center
#chvg g volumegroup
Compellent Technologies
Page 22
Replay Creation and Mapping a Local Recovery back to the same server.
1. 2. 3. 4. 5. 6. 7. 8. Create a replay on the Storage Center Perform a local recovery of the replay to create a new volume. Map it to the AIX server. Run the cfgmgr command to discover the new volume. Run the lspv command to display the PVID of the new volume. Note: That the new volume has the same PVID as the original volume. The new volume PVID must be changed. For example if the new volume shows up as hdisk4. 9. Run the chdev l hdisk4 a pv=clear command to clear the PVID. 10. Run the chdev l hdisk4 a pv=yes command to auto-assign a new PVID. 11. Run the recreatevg -y newname hdisk4 command to recreate the Volume Group present on the replay volume
#> mkvg -S -y vg500GB hdisk2 0516-1254 mkvg: Changing the PVID in the ODM. vg500GB #> lspv hdisk2 PHYSICAL VOLUME: PV IDENTIFIER: VG IDENTIFIER PP SIZE: TOTAL PPs: FREE PPs: hdisk2 VOLUME GROUP: vg500GB 0000093e9d100998 0000093e0000d700000001259d100b35 256 megabyte(s) LOGICAL VOLUMES: 0 1999 (511744 megabytes) VG DESCRIPTORS: 2 1999 (511744 megabytes) HOT SPARE: no
#> mklv -t jfs2 -y lv500GB vg500GB 1999 lv500GB #> mkdir /500gb #> crfs -v jfs2 -a log=INLINE -d lv500GB -m /500gb File system created successfully. 523485388 kilobytes total disk space. New File System size is 1048051712 #> mount /dev/lv500GB /500gb #> df -g Filesystem /dev/lv500GB GB blocks 499.75 Free %Used 499.17 1% Iused %Iused Mounted on 4 1% /500gb
Compellent Technologies
Page 23
#> varyonvg vg500GB Create a replay of source volume (tyrant_500gb_Lun1) and create a local recovery using the default name (tyrant_500gb_Lun1 View1) on the Storage Center. Map the local recovery to the second server and run cfgmgr on the second server to discover new volume 2ndServer#> cfgmgr #> lsdev -Cc disk hdisk0 Available 01-08-00-1,0 hdisk1 Available 01-10-01 hdisk2 Available 01-11-01 # mkdir /500gbreplay # importvg -y vg500GB hdisk2 #> mount /dev/lv500GB /500gb Replaying log for /dev/lv500GB. #> df -g /500gb Filesystem GB blocks /dev/lv500GB 499.75 Free %Used 499.17 1% Iused %Iused Mounted on 5 1% /500gb SCSI Disk Drive Compellent FC SCSI Disk Drive Compellent FC SCSI Disk Drive
Compellent Technologies
Page 24
Compellent Technologies
Page 25
PdAt: uniquetype = "disk/iscsi/compellent_sc" attribute = "reserve_policy" deflt = "no_reserve" values = "PR_exclusive,no_reserve,single_path" width = "" type = "R" generic = "DU" rep = "sl" nls_index = 96
2. Delete the old entries in the ODM by executing the following commands:
# odmdelete -o PdAt -q "uniquetype = disk/fcp/compellent_sc and attribute = reserve_policy" # odmdelete -o PdAt -q "uniquetype = disk/iscsi/compellent_sc and attribute = reserve_policy"
3. Create the new entries in the ODM by executing the following commands:
# odmadd reserve.odmadd
4. Verify the new entries in the ODM by executing the following commands:
# odmget -q "uniquetype = disk/iscsi/compellent_sc and attribute = reserve_policy" PdAt # odmget -q "uniquetype = disk/fcp/compellent_sc and attribute = reserve_policy" PdAt
You should notice the deflt value under the PdAT output as no_reserve. Note that after you make this change to the ODM, the default behavior will be changed from PR_exclusive to no_reserve for any new LUNs created and mapped to this server (a reboot is not required to take effect). All existing LUNs settings before the change will remain the intact.
Compellent Technologies
Page 26