x
10/4/05
Corporate Headquarters Cisco Systems, Inc. 170 West Tasman Drive San Jose, CA 95134-1706 USA http://www.cisco.com Tel: 408 526-4000 800 553-NETS (6387) Fax: 408 526-4100
CCSP, the Cisco Square Bridge logo, Cisco Unity, Follow Me Browsing, FormShare, and StackWise are trademarks of Cisco Systems, Inc.; Changing the Way We Work, Live, Play, and Learn, and iQuick Study are service marks of Cisco Systems, Inc.; and Aironet, ASIST, BPX, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Empowering the Internet Generation, Enterprise/Solver, EtherChannel, EtherFast, EtherSwitch, Fast Step, GigaDrive, GigaStack, HomeLink, Internet Quotient, IOS, IP/TV, iQ Expertise, the iQ logo, iQ Net Readiness Scorecard, LightStream, Linksys, MeetingPlace, MGX, the Networkers logo, Networking Academy, Network Registrar, Packet, PIX, Post-Routing, Pre-Routing, ProConnect, RateMUX, Registrar, ScriptShare, SlideCast, SMARTnet, StrataView Plus, SwitchProbe, TeleRouter, The Fastest Way to Increase Your Internet Quotient, TransPath, and VCO are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries. All other trademarks mentioned in this document or Website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0406R)
Cisco MDS 9000 Family Cookbook Copyright 2004 Cisco Systems, Inc. All rights reserved.
Comments: mds-cookbook@cisco.com
Preface
This document provides a reference guide to the configuration and implementation of fabrics using the MDS-9000 family of switches. The configurations and components used herein have been tested and validated by Ciscos Solution-Interoperability Engineering for supporting risk-free deployment of fabrics using Ciscos MDS 9000 family of Fibre Channel Switch and Director Class products.
Audience
This reference document is designed for use by Cisco TAC, Sales, Support Engineers, Professional Service Partners, Systems Administrators and others, responsible for the design and deployment of Storage Area Networks in the data center environment.
Inquiries
Please send your comments via email to: Email: mds-cookbook@cisco.com State the document name, page number, and details of the request. Thank you. Latest version of this document is available on Ciscos Partner website at: http://www.cisco.com
THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT T CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS. THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY. The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCBs public domain version of the UNIX operating system. All rights reserved. Copyright 1981, Regents of the University of California. NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED AS IS WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED ORIMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE. IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
iii
Preface
iv
78-xxxxx-xx
CONTENTS
1
CHAPTER
Management
13
CFS: Cisco Fabric Services 13 Fabric Manager and CFS 15 Which Switch does FM Distribute from? 15 CFS CLI Commands 16 What Switches are CFS capable? 16 What CFS applications do I have and what is their scope? Locked out of an application by CFS 17 Command Scheduler 17 Regularly Backing up the Switch Configuration Copying files to or from the MDS
19 21 18
16
Firmware Upgrades 22 Upgrading firmware with the CLI 22 Downgrading firmware with the CLI 24 Upgrading Firmware with Fabric Manager Password Recovery
28
25
Installing a License 30 Using the CLI to Install a License: 31 Using Fabric Manager to Install a License 32 What Feature is Enabling the License Grace Period? Copying core files from the MDS Fixed Switch Config Restoration
35 36
34
Configuring an NTP Server 38 Configuring NTP with CFS 38 Without CFS 40 Steps to Perform Before Calling TAC How to Disable the Webserver
41 42
Device Aliases 43 Device Aliases with the CLI 43 Examples of Devices Aliases Displayed in the CLI: Creating Device Aliases with the CLI 44
44
Contents
Converting FC Aliases into Device Aliases 45 Device Aliases with FM 46 Enabling Fabric Manager to use Device Alias 46 Creating a Device Alias for an Existing Device 47 Managing Fabric Manager 48 Operating Fabric Manager Through a Firewall using SNMP Proxy Configuration using a non-NAT Packet Filter 49
2
49
CHAPTER
Account Management
51 52
Creating a MDS User Role 52 Creating a Role with Device Manager Creating Roles with CLI 56
52
Configuring TACACS+ with Cisco SecureACS 57 Authentication and Authorization with TACACS+ Configure SecureACS Server 57 Configure TACACS+ on the MDS 62 Accounting with TACACS+ 63 Configuring the MDS 63 Configuring SecureACS 64 Providing Passwordless Access Leveraging SSH
3
66
57
CHAPTER
Physical Interfaces
69
Configuring FC ports 69 Port Description 69 Port speed 69 Port mode (auto) 70 Port mode (E) 70 Port mode (F) 70 Port mode (FL) 71 Port mode (Fx) 71 Port mode (SD) 71 Port mode (ST) 71 Port mode (TL) 72 Configuring Trunking E ports 72 Trunk port mode 72 Configuring trunk ports to filter specific VSANs Enabling port beaconing 73 Configuring Gigabit Ethernet Ports
Cisco MDS 9000 Family Cookbook for SAN-OS 2.x
72
73
vi
Contents
Configuring VRRP
73
Implementing WWN Based VSANs (DPVM) 75 Adding Existing Devices to DPVM 77 Adding New Devices to DPVM 79 Modify the VSAN Assignment of a DPVM Entry DPVM Conflicting Entries 82 DPVM with the CLI 84 Adding Existing Devices to DPVM 84 Adding New Devices to DPVM 85 Modify the VSAN Assignment of a DPVM Entry
4
81
86
CHAPTER
VSANs
87 87
Modifying VSAN Attributes with Fabric Manager 88 Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using Fabric Manager. 89 Modifying VSAN Attributes with the CLI 91 Creating a VSAN on a single switch and adding an Interface 91 Setting VSAN Interop Mode 92 Changing Load-balancing scheme 93 Sequence Level load-balancing (Source_ID, Destination_ID) 93 Exchange level load balancing (S_ID, D_ID, OX_ID) 93 Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using CLI. 93 Restarting a VSAN 94 Assigning a Predetermined FCID to a PWWN 95
5
CHAPTER
Zoning
97
Enhanced Zoning 97 How to Enable Enhanced Zoning 99 Enabling Enhanced Zoning with the CLI 99 Enabling Enhanced Zoning with Fabric Manager 99 How to Display which user has obtained the lock 100 ZoneSets 101 Zoneset Distribution 101 Automatic Zoneset Distribution 101 Manual Zoneset Distribution: 102 Zones 102 Zone Creation and Adding it to a Zoneset with Fabric Manager
103
vii
Contents
How to Create Non-PWWN based zones 108 Creating a zone and adding it to a zoneset (CLI standalone method) 109 Creating a zone and adding it to a zoneset (CLI inline method) 110 Creating a FC Alias based zone (CLI) 111 Creating an Interface based zone (CLI method) 113
6
CHAPTER
Inter-VSAN Routing 115 IVR with CFS 115 Configuring a Three Switch, Two Transit VSAN Topology with CFS IVR-1 121 Enable IVR-1 121 To enable IVR-1 via the CLI: 121 To enable IVR-1 via FM: 122 Single Switch, Two VSANs 122 IVR-2: FC-NAT 125 Enabling IVR-2 125 How to Upgrade from IVR-1 to IVR-2: 128 How to Configure Persistent FCIDs in IVR 129 Single Switch: Two VSANs 131 Adding a new IVR Enabled Switch 134
117
CHAPTER
FCIP
Configuring multiple FCIP tunnels using a single gigE port Configuring FCIP with IPsec using FM
149
Tuning FCIP 158 TCP Tuning: Latency and Available Bandwidth FCIP Write Acceleration 159 FCIP Compression 159 Enabling Tape Acceleration 161 SAN Extention Tuner (SET)
8
164
158
CHAPTER
iSCSI
Configuring iSCSI on MDS in Transparent Mode Configuring iSCSI on MDS in Proxy initiator mode Configuring iSCSI Client (initiators) on hosts
Cisco MDS 9000 Family Cookbook for SAN-OS 2.x
180
viii
Contents
ix
Contents
Introduction
While deploying or configuring the MDS platform, a task may come up that the administrator does not know how to perform and requires a simple recipe to complete it. This guide wont replace the MDS 9000 Family Configuration Guides, but compliment them with simple, concise procedures to accomplish a specific task. This is field driven book, meaning that the intended audience, the storage administrators, technical support engineers, SEs and CEs are the source for these procedures. Their requirements for a procedure to perform XYZ are what determine the content. If there are procedures that should be covered in this book, please notify by Email: mds-cookbook@cisco.com. Within this book, there are sections that look like this:
Tip
These tips refer to a MDS best practice towards implementing the MDS. They come from in depth knowledge of the platform as well as years of experience of implementing SANs.
xi
Introduction
xii
78-xxxxx-xx
C H A P T E R
Management
This chapter will provide recipes for the aspects of managing the MDS as an entity. These non-datapath areas include access control, accounting, event resolution and monitoring.
Pending Database: When configuration changes are made to a CFS application, they are first made to the pending database which is always distributed to all switches in the fabric.To activate these changes into the MDSs running-configuration, an explicit commit command must be executed. Alternatively, the user can clear the applications pending database by issuing an explicit abort command. Locking. Prior to modifying the pending database, the application via the CFS transport obtain a lock, thus preventing other users and switches from modifying the pending database. Applications outside the scope of the lock can still be modified.
Upon the start of configuration, the application implicitly attempts to obtain a lock. The CFS
infrastructure knows which switch and user has obtained the lock.
Scope. The scope of an application can be either Physical or Logical. This determines if other users can be modifying the same application simultaneously.
A physical scope encompasses all the switches in the physical fabric such as NTP. While a NTP
lock has been obtained, no other user may modify NTP within the physical fabric.
1-13
Management
A logical scope encompasses only the VSAN that the application is being configured for such
as Port-Security. While a port-security lock is obtained, no other user can modify port-security in the VSAN for which the lock is applied to. However, port-security could be modified for another VSAN as it is outside of the scope of the lock.
Merge Control. Each application is responsible for the merging of its configuration with that of the same application in another physical fabric if two fabrics are merged. The basic rule for merging is that a union of the two configurations will be performed however, conflicting entries will not be merged. Those conflicting entries will have to be manually created in the resulting configuration.
Note
Failure to fully merge a CFS application when two merging fabrics, will NOT isolate the ISL.
Tip
If CFS is to be used with an application, all the MDSs in the fabric should be configured to use CFS for that application. For example, if there are five switches in a fabric, and NTP will be configured leveraging CFS, all five switches should have NTP leverage CFS, not just a subset of them. As illustrated in Figure 1-1, a CFS application would follow the below procedure:
Step 1 Step 2
Prior to the first Administrator enables CFS distribution for the application. Then enters configuration mode for the specified application. Local MDS implicitly attempts to obtain a an application lock from the other switches in the fabric according to the scope of the application (VSAN or physical). If available, other MDSs grant the lock to the local MDS. If the lock is not granted, the admin cannot make changes to the applications pending database. Admin makes configuration changes to the pending database and when that is completed, the changes are explicitly committed or aborted by the admin. Local MDS informs the other MDSs in the scope to commit the changes and the lock is released. Until the lock is released by the application, other users on other switch cannot make changes to the locked application. However, other applications can still be modified.
1-14
78-xxxxx-xx
Chapter 1
Figure 1-1
Tip
If an application is configured to use CFS in the fabric, CFS should be enabled for that application on all switches in the fabric. FM will use either CFS or the legacy method (send multiple requests), not both.
1-15
Management
Figure 1-2
1-16
78-xxxxx-xx
Chapter 1
No No No No Yes No
Note
Remember that a physical scope spans all switches physically connected together, regardless of VSAN configuration. Logical scope applies only to the VSAN that the configuration applies to. SFM - SCSI Flow Manager for use with monitoring scsi flows with the Storage Services Module FSCM - Fabric Startup Configuration Manager enables copy running-config startup-config
fabric
Until that user commits their changes to the database or their lock expires another user cannot modify the pending database. However, if its required, then to break a lock. This will clear the pending-database and all pending changes for the specified application will be lost.
172.22.36.9# clear ntp session
Command Scheduler
The following section provides recipes leveraging the MDSs command scheduler, which provides the user with the ability to schedule tasks for the MDS.
1-17
Management
Step 2
Define the job to be run. The job will consist of saving the running configuration and then copying it to a tftp server. the {config-job} prompt is the same as the MDS exec-mode prompt. Thus any command on the MDS could be executed.
ca-9506(config)# scheduler job name backup_config ca-9506(config-job)# copy running-config startup-config ca-9506(config-job)# copy startup-config tftp://171.71.58.69/ca-9506_config
Step 3
Step 4
Create the schedule, assign the time (20:00) and the job that will be assigned to it (backup_config).
ca-9506# conf terminal Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# scheduler schedule name nightly_10pm ca-9506(config-schedule)# time daily 20:00 ca-9506(config-schedule)# job name backup_config
Step 5
Step 6
After the job has run, the status of the job and the details of the execution can be examined:
ca-9506# show scheduler schedule name nightly_10pm
1-18
78-xxxxx-xx
Chapter 1
Schedule Name : nightly_10pm ---------------------------------User Name : admin Schedule Type : Run every 0 Days 0 Hrs 1 Mins Start Time : Fri Apr 22 20:00:00 2005 Last Execution Time : Fri Apr 22 20:00:00 2005 Last Completion Time: Fri Apr 22 20:00:15 2005 Execution count : 1 ----------------------------------------------Job Name Last Execution Status ----------------------------------------------backup_config Success (0) ==============================================================================
Detailed Log:
ca-9506# show scheduler logfile ============================================================================== Job Name : backup_config Job Status: Success (0) Schedule Name : nightly_10pm User Name : admin Completion time: Fri Apr 22 20:00:15 2005 --------------------------------- Job Output --------------------------------`copy running-config startup-config ` [#### ] 7% [####### ] 14% [########## ] 23% [############# ] 30% [################ ] 37% [################### ] 46% [###################### ] 53% [######################### ] 60% [############################ ] 69% [############################### ] 76% [################################## ] 84% [##################################### ] 92% [########################################] 100% `copy startup-config tftp://171.71.58.69/ca-9506_config` Trying to connect to tftp server...... TFTP put operation was successful ==============================================================================
Tip
Some tftp servers may have to be configured to allow the overwriting of files. To avoid overwriting the previous night configuration, create multiple jobs. Specify each job using a different destination filename (ca-9506_monday, ca-9506-tuesday... etc).
1-19
Management
The CLI offers a broad range of protocols to use for copying to or from the MDS. Note that the MDS always acts as a client, such that an ftp/scp/tftp session will always originate from the MDS and either push files to an external system or pull files from an external system. File Server: 172.22.36.10 File to be copied to the switch: /etc/hosts The MDSs copy command supports 4 transfer protocols and 12 different sources for files.
ca-9506# copy ? bootflash: core: debug: ftp: licenses log: modflash: nvram: running-config scp: sftp: slot0: startup-config system: tftp: volatile:
Select source filesystem Select source filesystem Select source filesystem Select source filesystem Backup license files Select source filesystem Select source filesystem Select source filesystem Copy running configuration to destination Select source filesystem Select source filesystem Select source filesystem Copy startup configuration to destination Select source filesystem Select source filesystem Select source filesystem
To use scp (Secure copy) as the transfer mechanism, the following syntax would be used: "scp:[//[username@]server][/path]" To copy /etc/hosts from 172.22.36.10 using the user user1, and the destination would be hosts.txt use:
switch# copy scp://user1@172.22.36.10/etc/hosts bootflash:hosts.txt user1@172.22.36.10's password: hosts 100% |*****************************| 2035 00:00
Tip
Backing up the startup-configuration to a server should be done on a daily basis and prior to any changes. A short script could be written to be run on the MDS to perform a save and then backup of the configuration. The script only needs to contain two commands: copy running-configuration startup-configuration and then copy startup-configuration tftp://<server>/<name>. To execute the script use: run-script <filename>.
1-20
78-xxxxx-xx
Chapter 1
Note
This recipe is most often invoked when a firmware upgrade fails because there is not be enough free bootflash: capacity on the standby supervisor for the firmware images. Determine which supervisor is the standby. In this case it is module 6 as displayed below.
switch# show module Mod Ports Module-Type --- ----- ------------------------------1 16 1/2 Gbps FC Module 2 16 1/2 Gbps FC Module 3 8 IP Storage Services Module 4 0 Caching Services Module 5 0 Supervisor/Fabric-1 6 0 Supervisor/Fabric-1
Step 1
Step 2
Next, connect to the standby supervisor using the attach command. Note how the prompt displays the word standby.
ca-9506# attach module 6 Attaching to module 6 ... To exit type 'exit', to abort type '$.' Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2004, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by Andiamo Systems, Inc. and/or other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. ca-9506(standby)#
Step 3
Step 4
1-21
Management
Step 5
To return to the active supervisor, type exit, and the prompt will also return to the active supervisors prompt:
ca-9506(standby)# exit rlogin: connection closed. ca-9506#
Firmware Upgrades
Within SAN-OS 2.x, the process of upgrading has not changed from within SAN-OS 1.x. However, if the need to downgrade from SAN-OS 2.x to 1.x is required, special attention needs to be paid to the procedure.
Tip
Always carefully read the output of install alls compatibility check. This will tell you exactly what needs to be upgraded (BIOS, loader, firmware) and what modules are not hitless. If there are any questions or concerns about the results of the output, select n to not proceed with the installation and contact the next level of support.
ca-9506# install all system scp://setmason@dino/tftpboot/rel/qa/2_1_1a/final/m95 00-sf1ek9-mz.2.1.1a.bin kickstart scp://setmason@dino/tftpboot/rel/qa/2_1_1a/fin al/m9500-sf1ek9-kickstart-mz.2.1.1a.bin For scp://setmason@dino, please enter password: For scp://setmason@dino, please enter password: Copying image from scp://setmason@dino/tftpboot/rel/qa/2_1_1a/final/m9500-sf1ek9 -kickstart-mz.2.1.1a.bin to bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Copying image from scp://setmason@dino/tftpboot/rel/qa/2_1_1a/final/m9500-sf1ek9 -mz.2.1.1a.bin to bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Verifying image bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin [####################] 100% -- SUCCESS Verifying image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin [####################] 100% -- SUCCESS
1-22
78-xxxxx-xx
Chapter 1
Extracting "slc" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "ips" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "svclc" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "system" version from image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "kickstart" version from image bootflash:///m9500-sf1ek9-kickstart-mz .2.1.1a.bin. [####################] 100% -- SUCCESS Extracting "loader" version from image bootflash:///m9500-sf1ek9-kickstart-mz.2. 1.1a.bin. [####################] 100% -- SUCCESS
Compatibility check is done: Module bootable Impact ------ -------- -------------1 yes non-disruptive 2 yes non-disruptive 3 yes disruptive 4 yes disruptive 5 yes non-disruptive 6 yes non-disruptive
Reason ------
Images will be upgraded according to following table: Module Image Running-Version New-Version ------ ---------- -------------------- -------------------1 slc 2.0(2b) 2.1(1a) 1 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 2 slc 2.0(2b) 2.1(1a) 2 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 3 ips 2.0(2b) 2.1(1a) 3 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 4 svclc 2.0(2b) 2.1(1a) 4 svcsb 1.3(5m) 1.3(5m) 4 svcsb 1.3(5m) 1.3(5m) 4 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 5 system 2.0(2b) 2.1(1a) 5 kickstart 2.0(2b) 2.1(1a) 5 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 5 loader 1.2(2) 1.2(2) 6 system 2.0(2b) 2.1(1a) 6 kickstart 2.0(2b) 2.1(1a) 6 bios v1.1.0(10/24/03) v1.1.0(10/24/03) 6 loader 1.2(2) 1.2(2) Do you want to continue with the installation (y/n)? Install is in progress, please wait. [n] y
Syncing image bootflash:///m9500-sf1ek9-kickstart-mz.2.1.1a.bin to standby. [####################] 100% -- SUCCESS Syncing image bootflash:///m9500-sf1ek9-mz.2.1.1a.bin to standby.
1-23
Management
[####################] 100% -- SUCCESS Setting boot variables. [####################] 100% -- SUCCESS Performing configuration copy. [####################] 100% -- SUCCESS Module 5: Waiting for module online. 2005 May 20 15:46:03 ca-9506 %KERN-2-SYSTEM_MSG: mts: HA communication with standby terminated. Please check the standby supervisor. -- SUCCESS "Switching over onto standby".
The %Kern-2-SYSTEM_MSG displayed at the end is displayed when the standby supervisor is rebooted as part of the rolling upgrade of the supervisor modules. It is expected and upon the next message, the user should reconnect (telnet/ssh) back into the switch on the new active supervisor. The continued progress of the installation can be watched on the new active supervisor using the show install all status command.
ca-9506# show install all status There is an on-going installation... Enter Ctrl-C to go back to the prompt. Continue on installation process, please wait. The login will be disabled until the installation is completed. Trying to start the installer... Module 5: Waiting for module online. -- SUCCESS Module 1: Non-disruptive upgrading. -- SUCCESS
Verify there are no features enabled that are not supported in the lower level which should be disabled.
Warning
Failure to disable a feature listed in the incompatibility check can result in a disruptive firmware downgrade.
1-24
78-xxxxx-xx
Chapter 1
If downgrading from 2.1(x) to 2.0(x) as IVR with FC-Nat was not available in SAN-OS 2.0(x).
ca-9506# show incompatibility system bootflash:m9500-sf1ek9-mz.2.0.2b.bin The following configurations on active are incompatible with the system image 1) Service : ivr , Capability : CAP_FEATURE_IVR_FCID_NAT_ENABLED Description : ivr fcid-nat mode is enabled Capability requirement : STRICT 2) Service : ivr , Capability : CAP_FEATURE_IVR_AUTO_VSAN_TOPOLOGY_ENABLED Description : ivr auto vsan-topology mode is enabled Capability requirement : STRICT
Step 2
Step 3
Step 4
Proceed with the downgrade procedure using the install all command as described in Upgrading firmware with the CLI, page 1-22.
Step 1
Select the Software Install Wizard from the toolbar in Fabric Manager.
Figure 1-3 Image Installation with Fabric Mgr
Step 2
1-25
Management
Figure 1-4
Step 3
Fill in the file information to transfer the file from the server to the switch. If the files are to be downloaded during the install, the path and filename of the images are required to be filled in as well. By selecting Skip Image Download, an upgrade can be performed using images which are already located on the supervisors bootflash.
1-26
78-xxxxx-xx
Chapter 1
Figure 1-5
Step 4
Select Next. Depending on the method for installing (already downloaded to bootflash or download during the install), the wizard may prompt for additional file locations. The fourth and final screen will provide a summary and enable the user to start the install. During the install, a compatibility screen will pop-up displaying the same version compatibility information that was displayed in the CLI upgrade. The user must select Yes to continue with the upgrade.
Note
Unlike the CLI, the FM maintains connectivity to the switch and provides detailed information during the entire upgrade sequence without requiring the user to manually reestablish connectivity to the switch during the supervisor switchover. If there is a failure, the last screen will display any reasons for a failed upgrade.
1-27
Management
Password Recovery
If there are no accounts accessible on the MDS that have either network-admin or user account creation privileges, due to lost passwords, it may be required to perform a password recovery on the admin account.
Warning
This procedure requires console access to the switch and will require the switch to be rebooted.
Tip
It is possible for another CLI user with network-admin privileges to change the password of the admin user, which can alleviate reloading the switch.
To recover the admin accounts password the following procedure can be used:
Step 1
Step 2
1-28
78-xxxxx-xx
Chapter 1
Step 3 Step 4
Attach the RS-232 end of the console cable to a PC. Configure Hyperterm or similar terminal emulation software for 9600 baud, 8 data bits, no parity, 1 stop bit and no flow control:
Figure 1-7 HyperTerm Terminal Settings.
Step 5 Step 6
Establish a connection to the switch if possible or enough to display the login prompt if no user accounts are available. For a multi supervisor switch, MDS-9509 or MDS-9506, physically remove the standby supervisor. It is not necessary to remove it from the chassis, just enough such that it does not make contact with the backplane.
1-29
Management
Reboot the switch either by cycling the power or issuing the reload command. Press the Ctrl-] key sequence (when the switch begins its SAN-OS software boot sequence) to enter the switch(boot)# prompt. Enter configuration mode:
switchboot# config terminal
Step 10
Step 11
Step 12
Log into the switch using the admin account and the temporary password.
switch login: admin Password: Cisco Storage Area Networking Operating System (SAN-OS) Software TAC support: http://www.cisco.com/tac Copyright (c) 2002-2004, Cisco Systems, Inc. All rights reserved. The copyrights to certain works contained herein are owned by Andiamo Systems, Inc. and/or other third parties and are used and distributed under license. Some parts of this software are covered under the GNU Public License. A copy of the license is available at http://www.gnu.org/licenses/gpl.html. switch#
Step 13
Step 14
Installing a License
In order to run the advanced features of the MDS platform, a license key is required for permanent usage. A 120 day grace period allows the user to try out the feature or if to resume operation in the case where a chassis is replaced until a license key is installed.
Warning
If a licenses grace period expires, all of the features that depend on that license will be disabled even if they are currently running or in production.
1-30
78-xxxxx-xx
Chapter 1
To install a license key there are two methods available, using either the CLI or Fabric Manager.
Step 2
Step 3
Cross reference the switchs host-id (VDH=FOX0713037X) with that which is listed in the license file:
ca-9506# show license host-id License hostid: VDH=FOX0713037X
Step 4
Step 5
To display a summary of the installed licenses use the command show license summary:
switch# show license usage Feature Insta License Status Expiry Date Comments lled Count ----------------------------------------------------------------FM_SERVER_PKG Yes In use never MAINFRAME_PKG No Unused -
1-31
Management
ENTERPRISE_PKG Yes In use never SAN_EXTN_OVER_IP Yes 2 In use never SAN_EXTN_OVER_IP_IPS4 No 0 Unused -----------------------------------------------------------------
To display which features within a license package are being used, specify the package name. In this case QOS is using the Enterprise package.:
ca-9506# show license usage ENTERPRISE_PKG Application ----------Qos Manager -----------
Step 2
This will bring up the following screen which will allow the user to specify how to install the keys based upon whether or not the user has already obtained the license key files or if they only have a Product Authorization Key (PAK). If the user has a PAK then the license file can be downloaded and installed from Ciscos website.
1-32
78-xxxxx-xx
Chapter 1
Figure 1-9
a.
If the keys already exist on a server then the following dialog box will be provided to specify the name and location of the license key files:
b.
If the license files are not already available, and the user only has the PAK numbers, then Fabric Manager can obtain the license files directly from Cisco.com.
1-33
Management
At this point, the license keys can be installed and the licensable feature can be used.
With Fabric Manager, in the Physical Attributes Pane, select Switches, then Licenses. In the resulting middle pane, select the Usage tab:
1-34
78-xxxxx-xx
Chapter 1
2.
With the CLI, run the command show license usage <license name>:
switch1# show license usage ENTERPRISE_PKG Application ----------SFM Qos Manager Port Security -----------
At this point the feature has been identified and it can be either disabled or if the feature is needed then a license should be installed.
Note
The destination system can any of the possible destinations including tftp, scp, sftp or slot0. Before copying a core file to another server, the pid of the core file must be identified.
switch# show cores Module-num Process-name PID Core-create-time
Step 1
1-35
Management
Step 2
Then to copy the core file using, for example, ftp issue the following command syntax:
"core://<module-number>/<process-id>" switch# copy core://5/1524 ftp://172.22.36.10/tmp/fspfcore
The file can now be sent to Cisco TAC according to the TAC engineers directions.
Note
Only restore a switch configuration to a switch which has the exact same firmware version on it as which was used to create the switch configuration. If an upgrade is required, restore the configuration then upgrade the firmware. Save the running configuration:
switch1# copy running-config startup-config [########################################] 100%
Step 1
Step 2
Copy the startup-configuration to the file server using any of the available methods on the MDS (ftp, tftp, sftp, scp):
switch1# copy startup-config scp://user@host1/switch1.config user@switch1's password: sysmgr_system.cfg 100% |*****************************| 10938 switch1#
00:00
Step 3
Capture the port assignments using the FLOGI database. This will be used to verify that all of the cables are placed in the correct locations.
switch1# show flogi database --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------fc1/8 600 0x7c0007 50:05:07:63:00:ce:a2:27 50:05:07:63:00:c0:a2:27 fc1/13 1001 0xef0001 50:06:0e:80:03:4e:95:13 50:06:0e:80:03:4e:95:13 fc1/15 600 0x7c0004 50:06:0b:00:00:13:37:ae 50:06:0b:00:00:13:37:af
1-36
78-xxxxx-xx
Chapter 1
Note
At this point the old switch is no longer needed and its mgmt0 port should be disconnected from the LAN.
Step 4
Log on to the new switch using the console connection and clear the switchs configuration. Do not run the setup script, if prompted. The write erase command will erase the switchs configuration.
switch2# write erase Warning: This command will erase the startup-configuration. Do you wish to proceed anyway? (y/n) [n] y
Step 5
Step 6
The switch will come up in its factory default mode and will prompt for the Basic System Configuration Dialog, skip it, as all the configuration options are contained in the old switchs startup-config. Manually configure the IP address.
switch2# config terminal Enter configuration commands, one per line. End with CNTL/Z. switch2(config)# int mgmt 0 switch2(config-if)# ip address 172.22.36.8 255.255.254.0 switch2(config-if)# no shut
Step 7
If interface (fcX/Y) based zoning was done, obtain the new switchs wwn. Otherwise, this step can be skipped.
switch2# show wwn switch Switch WWN is 20:00:00:0d:ec:02:1d:40
Step 8
On the file server, make a copy of the configuration file and then open it in a text editor (notepad, vi).
a.
Remove the lines that contain the snmp user accounts, as the encrypted passwords are tied to the MAC address of the chassis.:
$ cp switch1.config switch1.config.orig $ vi switch1.config
The user accounts are all grouped together and begin with the command snmp-server user
snmp-server user admin network-admin auth md5 0x46694cac2585d39d3bc00c8a4c7d48a6 localizedkey snmp-server user guestadmin network-admin auth md5 0xcae40d254218747bc57ee1df348 26b51 localizedkey
b.
If interface (fcX/Y) zoning was not done skip this step. Otherwise, replace the old switchs wwn in the zone member commands with the the new switchs wwn:
zone name Z_1 vsan 9 member interface fc1/9 swwn 20:00:00:0d:ec:02:1d:40
c. Step 9
From the new switch, copy the modified config file from the fileserver onto the new switchs running configuration. As the file is copied it will be executed on the switch as the configuration is applied. The commands being applied with be contained in single-quotes. Any errors due to applying the commands will be displayed immediately after the error causing command is executed. The prompt will change to reflect the new switchname.
switch2# copy scp://user@host1/switch1.config running-config user@host1's password: switch1.config 100% |*****************************| 10938
00:00
1-37
Management
Step 10
Step 11
At this point, thecan be accessed via the CLI and the following remaining items will need to be performed:
a. b. c.
SNMP user accounts will need to be recreated. Also, If the switch is accessed via ssh, the hosts known_hosts file will need to have the MDS switchs entry removed from it, because the switchs public key is different. License keys, if required, will need to be installed.
Step 12 Step 13
Move the cables from the old switch to the new switch, using the old switchs show flogi database as a reference to verify that each cable is in the correct location. Verify that all devices have logged in and all features are running is as they are supposed to be, save the running-configuration to the startup-config with the command copy running-config startup-config. Reload the switch to verify that it boots correctly with the configuration.
Step 14
Note
NTP will not set the timezone (or offset from UTC) for the switch, it must be set manually using for example for Eastern Standard Time and Eastern Daylight-Savings Time: clock timezone EST -5.0 clock summer-time EDT 1 Sunday Apr 02:00 5 Sunday Oct 02:00 60 For this example:
1-38
78-xxxxx-xx
Chapter 1
Switch #1 IP Address: 172.22.36.142 Switch #2 IP Address: 172.22.36.8 NTP Server: 171.69.16.26 To configure ntp for switch1, the following procedure would be used:
Step 1
switch1:
switch1# conf t Enter configuration commands, one per line. switch1(config)# ntp distribute End with CNTL/Z.
b.
switch2
switch2# conf t Enter configuration commands, one per line. switch2(config)# ntp distribute End with CNTL/Z.
Step 2
Add the ntp server (this and the following steps can be done from either switch1 or switch2).
switch1# conf t switch1(config)# ntp server 171.69.16.26
Step 3
Step 4
At this point, NTP is configured and the switch will slowly adjust to the new time. To view the NTP configuration on the local switch:
switch1# show ntp peers -------------------------------------------------Peer IP Address Serv/Peer -------------------------------------------------172.22.36.142 Peer 171.69.16.26 Server 172.22.36.8 Peer To view the NTP configuration on the remote switch: switch2# show ntp peers -------------------------------------------------Peer IP Address Serv/Peer -------------------------------------------------172.22.36.142 Peer 171.69.16.26 Server 172.22.36.8 Peer
Step 5
Save the configuration fabric wide leveraging the fabric keyword. Which instructs all CFS capable switches to perform copy running-config startup-config. See Saving the configuration across the fabric, page 1-42
1-39
Management
Without CFS
Configuring NTP without CFS will require the user to log onto every switch in the fabric and configure NTP. This is the same procedure that would be used in a SAN-OS 1.x environment.
Note
NTP will not set the timezone (or offset from UTC) for the switch, it must be set manually using for example for Eastern Standard Time and Eastern Daylight-Savings Time: clock timezone EST -5.0 clock summer-time EDT 1 Sunday Apr 02:00 5 Sunday Oct 02:00 60 For this example: Switch #1 IP Address: 172.22.36.142 Switch #2 IP Address: 172.22.36.9 NTP Server: 171.69.16.26 To configure ntp for switch1, the following procedure would be used:
Step 1
Step 2
At this point, NTP is configured and the switch will slowly adjust to the new time.
1-40
78-xxxxx-xx
Chapter 1
Step 1 Step 2
Do not reload the line card or the switch until atleast Step 2has been done. Some logs and counters are kept in volatile storage and will not survive a reload. Collect switch information and configuration. This should be done before and after the issue has been resolved. The three methods below each provide the same information.
a. b.
CLI: Configure the telnet/ssh application to log the screen output to a text file and issue the command show tech-support details. CLI: Issue the command tac-pac <filename> (ex: tac-pac bootflash://showtech.switch1). The tac-pac command will redirect the output of show tech-support details to a file then gzip the file. If no filename is specified, the file created is volatile:show_tech_out.gz. The file should then be copied off the MDS using the procedure outlined inCopying files to or from the MDS on page 19. Fabric Manager: Select Tools, then select Show tech support. Fabric manager can capture switch configuration information from multiple switches simultaneously. The file can be saved on the local PC. For IVR related issues capture the output of show ivr tech-support If the error occurs in Fabric Manager, take a screen shot of the error. In windows, to capture the active window use ALT+PrintScreen, and to capture the entire desktop hit just PrintScreen. Then paste this into a new MSpaint.exe (or similiar program) session. Copy the error from the message log which can be displayed using either show logging log or to view the last X lines of the log show logging last #
c.
d. Step 3
b.
The following questions should be answered prior to placing the call to TAC:
1.
2. 3. 4. 5. 6. 7. 8. 9.
What is the network topology? (FM/tools ->show tech & save map) Were any changes being made to the environment (zoning, adding linecards, upgrades) prior to or at the time of this event? Are there other similarly configured devices that could have this problem but do not have? Where this problematic device connected to (MDS switch Z, interface x/y)? When did this problem first occur? When did this problem last occur? How often does this problem occur? How many devices have this problem? have already been done? Using such tools as:
10. Were any traces or debug outputs captured during the problem time? What troubleshooting steps
1-41
Management
a. Fcanalyzer, PAA-2, Ethereal, local or remote SPAN b. CLI debug commands c. FC traceroute, FC ping d. FM/DM
This command will take slightly longer than the local-only copy running-config startup-config would.
Note
IP ACLs have an implicity DENY ALL added to the end of the users ACL, so atleast one permit statement must be used otherwise all traffic will be denied.
Tip
Whenever using ACLs on mgmt0, make sure that it is tested out on a switch that has console access in case a typo is made. When deploying multiple MDSs, the commands in this recipe can be integrated into a script, by taking the bolded commands and pasting them into a CLI session.
The following CLI statements will create an IP ACL filter and then apply it to the mgmt0 interface:
Step 1
First create the access list called disable_webserver. Notice that the second entry for the access list is a permit any statement.
switch1#conf t
1-42
78-xxxxx-xx
Chapter 1
switch1(config)#ip access-list disable_webserver deny tcp any any eq port www switch1(config)#ip access-list disable_webserver permit ip any any
Step 2
Device Aliases
Device aliases provide a plain text name to pwwn mapping. This one to one mapping was brought upon by the need to be able to identify devices within an MDS environment by easier to understand labels rather than pwwns. The device alias can be used in areas such as, zoning, QOS, IVR, and throughout Fabric Manager and the Performance Manager utilities. This CFS aware application extends to the various CLI output by inserting the Device Alias where appropriate.
Note
It is a one to one mapping of pwwn to plain text name. The CFS scope is physical. A Device Alias database is not tied to a VSAN so if you move an end device (ex: hba) from one VSAN to another the device alias still applies. The merging of two device alias databases produces a union of the two databases. Conflicts are not imported into the resulting database and must be manually resolved. This will not keep the non-conflicting entries from being merged into the new database. The merge failure will not isolate an ISL. The CLI manages Device Aliases in a central database while Fabric Manager manages them under their respective type (Hosts and Storage). For services that leverage Device Alias (zoning, IVR, QOS etc), updating the Device Alias with a new pwwn, will not automatically propagate to the services. Those services should have the device removed, then update the device alias, and then re-add the device alias to the service.
Tip
Use the plain text name to describe the hostname and hba instance of the device. For example, host123_lpfc0 or SYMM7890_FA14ab is preferrable to just host123 and SYMM7890. The device alias can then become the basis for a Fabric Manager Enclosure. Device Aliases SYMM7890_FA14ab and SYMM7890_FA14ba should be part of the enclosure SYMM7890.
1-43
Management
ca-9506# show fcns database VSAN 1: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x620000 N 10:00:00:00:c9:32:8b:a8 (Emulex) scsi-fcp:init [ca-sun1_lpfc0] 0x65000a N 10:00:00:00:c9:34:a6:3e (Emulex) [ca-aix1_fcs0]
2.
zoneset name nozoneset vsan 501 zone name IVRZ_IvrZone1 vsan 501 pwwn 50:06:0e:80:03:4e:95:23 [HDS20117-c20-9] pwwn 10:00:00:00:c9:32:8b:a8 [ca-sun1_lpfc0] zone name ca_aix2_HDS vsan 600 pwwn 10:00:00:00:c9:34:a5:94 [ca-aix2_fcs1] pwwn 50:06:0e:80:03:4e:95:23 [HDS20117-c20-8]
3.
ca-9506# show flogi database --------------------------------------------------------------------------INTERFACE VSAN FCID PORT NAME NODE NAME --------------------------------------------------------------------------fc2/5 1000 0xef0008 50:06:0e:80:03:4e:95:23 50:06:0e:80:03:4e:95:23 [HDS20117-c20-9]
Device Alias is enabled by default, so first enter the Device Alias database:
ca-9506# conf terminal Enter configuration commands, one per line. ca-9506(config)# device-alias database ca-9506(config-device-alias-db)# End with CNTL/Z.
Step 2
Create the entry for the Device Alias using a meaingful name.
ca-9506(config-device-alias-db)# device-alias name ca-aix1_fcs0 pwwn 10:00:00:00 :c9:34:a6:3e
Step 3
Display the pending changes. A + means that the entry is being added to the database, while a - means that it will be removed from the database, upon performing a CFS commit.
1-44
78-xxxxx-xx
Chapter 1
ca-9506(config-device-alias-db)# do show device-alias database pending-diff + device-alias name ca-aix1_fcs0 pwwn 10:00:00:00:c9:34:a6:3e
Step 4
Note
While the Device Alias database is maintained on all switches as it has a physical scope, FC Aliases have a vsan scope and may not be present or replicated to all switches if Full Zoneset Distribution is not enabled. It may be required to log onto multiple switches to import all of the FC Aliases. Importing FC Aliases does not automatically update any zones that are based upon FC Aliases. The zones will need to manually converted to Device Alias based zones. Import FC Aliases does not automatically delete the FC Aliases. The FC Aliases will need to be manually deleted later.
In this example the following FC Aliases will be imported and converted into Device Aliases:
fcalias name alias123 vsan 1 fcalias name temphost vsan 1 pwwn 11:11:11:11:11:11:11:11 fcalias name temphost vsan 2 pwwn 11:11:11:11:11:11:99:99 fcalias name temphost2 vsan 2 pwwn 11:11:11:11:11:22:22:22
Ineligible FC Aliases are listed in vsan 1 and 2 to illustrate how to determine which FC Aliases failed to be imported.
Step 1
Step 2
1-45
Management
a.
If a Warning is displayed then examine the logs to determine which FC Aliases did not import:
switch(config)# do show device-alias internal errors 1) Event:E_DEBUG, length:111, at 608209 usecs after Wed Sep 14 16:25:13 2005 [109] ddas_import_fcalias(703): CONFLICT: fcalias temphost on vsan 2 has same name as fcalias temphost on vsan 1 2) Event:E_DEBUG, length:127, at 607826 usecs after Wed Sep 14 16:25:13 2005 [109] ddas_import_getnext_alias_resp_handler(1119): not importing alias alias123 from vsan 1 as the number of members are not 1.
Examining the contents of the FC Alias alias123 shows that it is not a valid Device Alias because it doesnt have any pwwn members, and that vsan 2s FC Alias temphost will not be imported as the same name was imported from vsan 1.
b.
At this point the conflicts should be resolved, by deleting the FC Alias (as in the case of alias123) or renaming one of the conflicting aliases (as in case of temphost).
Note
FC Aliases might be part of an existing zone and so the appropriate zone(s) should be updated accordingly. Display the pending Device Alias Database to display the newly imported Device Aliases
switch(config)# do show device-alias database pending-diff + device-alias name temphost2 pwwn 11:11:11:11:11:22:22:22
Step 3
Step 4
1-46
78-xxxxx-xx
Chapter 1
To enable during installation check the box marked Use Global Device Aliases in place of FC Aliases, at the initial installation screen. For SAN-OS 2.0 through 2.1(1a):
Step 1
In MDS SAN-OS 2.1(1a) and below, to enable after installation, modify the server.properties file (default location: C:\Program Files\Cisco Systems\MDS 9000\server.properties) entry to true:
# Specify whether to discover device aliases from a global alias server. # Default is false if unspecifed. fabric.globalAlias = true
Step 2 Step 3
Restart the Fabric Manager service. Note: Logging out and back into FM will not reread the file. Log back into FM.
FM can be configured to use Device Aliases after installation by selecting the Device Alias checkbox under the Admin -> Server menus.
1-47
Management
Prior to performing any Device Alias work in Fabric Manager, FM should be configured to leverage Device Aliases as in Enabling Fabric Manager to use Device Alias, page 1-46. In Fabric Managers Physical Attributes pane, expand End Devices then since the the wwn corresponds to a host, select the Hosts menu item. For a storage device select the Storage menu item. Find the pwwn and in the Device Alias column enter in the device alias for the referenced pwwn. Press the Apply changes button.
Figure 1-14 Create Device Alias with Fabric Manager
Step 2 Step 3
1-48
78-xxxxx-xx
Chapter 1
TCP port 9198.. The FM Client uses this TCP port to send the encapsulated SNMP packets to the FM Server. TCP ports 9099-9200. TCP port 9099 is used for java.rmi.registry.port while the other ports are used for java.rmi.server.remoteObjectPort. TCP port 80. Used to view the Performance Monitors web based statistics.
Additionally, for some features the Fabric Manager Client uses the CLI to obtain information on the switch directly. Therefore one of the two options should be used to allow the Fabric Manager Client to access the switch.
1. 2.
TCP port 23. When the Fabric Manager Client has been configured to use telnet for CLI access into the Cisco MDS. This is the default setting for the Fabric Manager Client. TCP port 22. When the Fabric Manager Client has been configured to use SSH for CLI access into the Cisco MDS. This is the preferred setting for this environment, and the MDS will need to have SSH enabled.
Note
All of the ports that Fabric Manager Client and Server use and which if need be modified are contained in the server.properties file, which may be found the following directory C:\Program Files\Cisco Systems\MDS 9000.
1-49
Management
Step 1
TCP port 9198. The FM Client uses this TCP port to send the encapsulated SNMP packets to the FM Server. TCP ports 9099-9200. TCP port 9099 is used for java.rmi.registry.port while the other ports are used for java.rmi.server.remoteObjectPort. TCP port 23. Some features of the FabricManager client require telnet access into the MDS. However, if SSH is enabled on the MDS, the FM Client can leverage SSH instead. If SSH is used, then this port is not required. TCP port 80. Used to view the Performance Monitors web based statistics. TCP port 22. Should be enabled if telnet on the MDS is disabled and SSH is enabled.
4. 5. Step 2
Configure the host running the FM Client how to reach the network behind the firewall where the FM Server resides. The 10.0.0.0 network is behind the firewall while 157.11.22.99 is the external IP address of the firewall. This is not the mgmt0 address of the firewall. On a windows host this may be done by:
C:\>route add 10.0.0.0 mask 255.255.255.0 157.11.22.99
Step 3
Configure the MDS to be able to reach the Fabric Manager Client (157.11.1.50) which is outside of the firewall. The internal IP address (the side which faces the MDS) assigned to the firewall is 10.0.0.44.
switch(config)# ip route 157.11.1.50 255.255.255.0 10.0.0.44
Step 4
Launch the Fabric Manager Client, and in the login screen, select the Use SNMP Proxy checkbox and enter the IP addresses of the Fabric Manager Server and the MDS Switch.
1-50
78-xxxxx-xx
C H A P T E R
Account Management
This chapter provides recipes for managing users and their accounts. In MDS firmware versions prior to 2.0, a separate account was required for both SNMP and CLI access. After 2.0, a single username grants access to both CLI and SNMP.
Note
A new MDS with SAN-OS 2.x preinstalled will not have a existing admin password. The user will need to choose one the first time they run the setup script. Existings accounts will not be forced to change their password. Upgrading from 1.x to 2.x will cause the 1.x CLI password to be used for both accounts where there are different passwords for the same user with regard to CLI and SNMP access. SAN-OS 2.x enforces strong passwords for all accounts created after the installation/upgrading to 2.x. A strong password must have the following characteristics
Must be atleast 8 characters long. Should not have too many consecutive characters. Should not have too many repeated characters. Should not contain easily guessable dictionary words.
Tip
The admin account should only be used during initial setup. Afterwards, other user accounts should be created. Each administrator should have their own individual account. Always change the admin password from the factory default value. Users should be granted the minimum amount of rights or abilities required to perform their job function. Implementing TACACS+, Configuring TACACS+ with Cisco SecureACS, page 2-57 eliminates the need to perform password recovery on the MDS.
2-51
Account Management
Step 2
Create the CLI user with using the following syntax: username <username> password <password> role <role>
switch(config)# username user1 password sox2004ch@mps role network-admin
At this point the user user1 can access the switch using the password sox2004ch@mps via console, ssh, telnet or snmp.
Network-admin is the role assigned to the predefined user admin. It has the ability to perform any modification to the MDS platform. There are no restrictions on this user. Network-operator is a predefined read-only role. It does not have the ability to make modifications to the MDS. There are no predefined users assigned to this role.
Tip
Provide each user with a role that provides them with the minimum amount of abilities to perform their job function. Leverage the read only role network-operator for those users who do not require the ability to modify the MDS. VSAN based role based access can allow administrators to have complete control over their VSANs while having read only or no access to other VSANs.
2-52
78-xxxxx-xx
Chapter 2
Step 1
In Fabric Manager, enable role distribution for all switches in the fabric. In the Physical Attributes pane, select Switches, Security then SNMP. Under the Enable Admin column, change the cells to enable. Then hit the Apply button. For this procedure, Fabric Manager is nolonger needed.
Figure 2-1 Enabling CFS distribution for Roles
Step 2
Open up Device Manager (DM) from any one of the switches that were just enabled for CFS distribution of roles, select Security and then Roles. DM will inform you that CFS is enabled, click continue.
Figure 2-2 Initial Roles Screen
Step 3
Select Create
2-53
Account Management
Figure 2-3
Fill in a name and description (do not use spaces) for the role. Optionally, a VSAN scope can be specified limiting this specific role to a subset of VSANs. Such that a zoning admin role could be created for zone admins that could only modify VSANs 1-10. This example will not specify a VSAN scope.
Step 4
To define what this role can and cannot do within the optional VSAN scope, select Rules
Figure 2-4 Create Rules
Click the CLI Command column to sort the table by CLI command. Select the show checkbox at the top and all of the zone, zone-attribute-group and zoneset commands. Additionally the copy Exec command should be checked so that the zoning admin can save the configuration.
Step 5
Select Apply. This will save the role configuration to CFSs pending database. Not until the CFS commit command has been executed will this role be part of the running-configuration of the switches in the fabric. This will bring up the previous dialog box.
2-54
78-xxxxx-xx
Chapter 2
Figure 2-5
Step 6
Select Create to add the role to the CFS pending database for Roles and then Close this will bring up the original Roles screen. The ZoningAdmin role only exists in the pending-database.
Figure 2-6 Display Role
Step 7
To commit the changes select the CFS button and then commit. To abort the changes and flush the pending-database, choose abort. To see the status of the CFS operation, from the main window, select Admin then CFS.
2-55
Account Management
Figure 2-7
Enable CFS distribution for roles on all switches that should receive this role. This step should be done the first time for all switches. The other steps should be do not need to be repeated on each switch.
switch# conf t Enter configuration commands, one per line. switch(config)# role distribute End with CNTL/Z.
Step 2
Step 3
Commit the role with CFS and distribute it to the other switches.
switch(config)# role commit
Step 4
switch# config terminal switch(config)# username zoning_user password g0s0x456 role ZoningAdmin
Step 5
2-56
78-xxxxx-xx
Chapter 2
Note
It is recommended that a tacacs+ server be used for both authentication, authorization and accounting. Implementing TACACS+ prevents having to perform password recovery on the MDS.
Prior to configuring the MDS, the SecureACS server must be configured. The first step is to configure SecureACS to allow the modification of advanced tacacs settings.
a. b.
On the main screen, on the left hand pane, select Interface Configuration, In the resulting screen select TACACS+ (Cisco IOS).
2-57
Account Management
c. d.
At the bottom of the screen, check Advanced TACACS+ Features and Display a window...attributes. This should result in the following screen. Select the Submit button at the bottom to save the changes.
SecureACS Configure Display
Figure 2-8
Step 2
Next is to define the MDS-9506 to the tacacs+ server, so that the MDS can be authenticated by the tacacs+ server. On the left hand pane, select Network Configuration and then Add Entry.
a. b.
Fill in the MDSs ip address, 172.22.36.142, and shared secret key WarEagle. Select Submit to save the information.
2-58
78-xxxxx-xx
Chapter 2
Figure 2-9
Step 3
Next is to define a group. These groups should be used to provide an easy way to assign the same role to multiple users without having to modify the attributes of each user individually. On the left hand pane, select Group Setup.
2-59
Account Management
Step 4
Select an available group and select Rename Group. In the resulting box, choose a new name for this group.
Tip
Use the same SecureACS group name as the MDS role, this will ease creation of tacacs based users. Click Submit to save the name change. On the left hand pane, select Group Setup, then select the newly renamed group and click Edit Settings. Scroll to the section labeled TACACS+ Settings, check Shell and Custom attributes. In the Custom attributes field, input the av-pair string corresponding to the role that is defined on the MDS for users. The syntax is: cisco-av-pair=shell:roles=<role> Select Submit + Restart to save and apply the configuration.
Step 5 Step 6
2-60
78-xxxxx-xx
Chapter 2
Step 7 Step 8
Next is to define a user. Select User Setup from the left hand pane. Then type in a new or existing username and select Add/Edit. In the resulting window, fill in the fields for Password, Confirm Password and Group to which the user is assigned as illustrated below.
2-61
Account Management
Configuring the SecureACS server has been completed and he remaining steps involve configuring the MDS Switch itself. It can be done from the CLI or SNMP.
Enable tacacs+
ca-9506# conf t Enter configuration commands, one per line. ca-9506(config)# tacacs+ enable
Step 2
Define the tacacs+ server and the shared secret key to use with it.
ca-9506(config)# tacacs-server host 172.22.36.10 key WarEagle
Step 3
Define a group of authentication servers to use and add the tacacs+ server to the group.
2-62
78-xxxxx-xx
Chapter 2
Step 4
Define the methods for the switch to perform authentication for the telnet/ssh/snmp access.
ca-9506# show tacacs-server timeout value:5 total number of servers:1 following TACACS+ servers are configured: 172.22.36.10: available on port:49 TACACS+ shared secret:******** ca-9506# show aaa authentication default: group tacacs-group1 console: local iscsi: local dhchap: local ca-9506# show user-account user:admin this user account has no expiry date roles:network-admin user:seth expires on Fri Jun 18 23:59:59 2004 roles:network-admin account created through REMOTE authentication Local login not possible
Note
The user seth is not available locally on the switch and yet is a member of the group/role network-admin. This means the user was authenticated by the tacacs+ server and not by the switch.
2-63
Account Management
Step 1
Configure the MDS to use the tacacs-group1 server group. The local keyword denotes to log locally on the switch if all servers listed in the server group are unavailable. If the server group is available, commands/events will not be logged locally.
switch# conf t switch(config)# aaa accounting default group tacacs-group1 local
Configuring SecureACS
Since this procedure builds on the configuration defined in Authentication and Authorization with TACACS+, page 2-57, only small modifications need to be made:
Step 1 Step 2 Step 3
Configure the SecureACS server to monitor for Update/Watchdog packets, by modifying the clients configuration. In SecureACS, in the left pane, click on Network Configuration. Select the client to be modified. Check the Log Update/Watchdog Packets from this AAA Client checkbox.
Figure 2-13 Enabling Accounting on the SecureACS server
Next is to configure the SecureACS to display the commands. Click on System Configuration in the left pane. Click on Logging. Click on CSV TACACS+ Accounting. Add the column err_msg, and check the Log to CSV TACACS+ Accounting report box.and click submit at the bottom.
2-64
78-xxxxx-xx
Chapter 2
To view the accounting report, click Reports and Activity in the left pane. Click TACACS+ Accounting. In the right hand pane, select the day to view. The current day is called TACACS+ Accounting active.csv.
Figure 2-15 SecureACS Accounting Log
2-65
Account Management
Tip
Passwordless logins should be assigned to either a read only role like network-operator or to one with a minimal set of privileges.
Warning
Only having the public key will not cause the MDS to grant access to a user the private key is required to be on the host. The private key should be guarded or treated like a password.
This procedure will cover the process of setting up a read-only (network-operator) based account which will only allow access if the user comes from a host that knows both the public and private keys.
Step 1
$ /usr/bin/ssh-keygen -t rsa1 Generating public/private rsa1 key pair. Enter file in which to save the key (/users/testuser/.ssh/identity): /users/setmason/.ssh/identity already exists. Overwrite (y/n)? y Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /users/testuser/.ssh/identity. Your public key has been saved in /users/testuser/.ssh/identity.pub. The key fingerprint is: c2:4d:6d:26:21:9d:79:9b:c3:86:dc:a5:07:d2:62:d4 testuser@host
On the host, the file /users/testuser/.ssh/identity.pub is the ssh public key that is encrypted using the rsa1 algorithm. The contents of this file will be used in the creation of the MDS user. In this example, the file looks like this:
$ cat /users/testuser/.ssh/identity.pub 1024 35 139198677264732164858153476357747926024656548233745027006381178621992083524037906211714241 450436547019604214530354070873624269283640613058470615170649963414635036859628344005142227 886318134122126153182906740418449098047827961768214148936752631482459130056603268404256522 191410368204629699075809390037814979061 testuser@host
Step 2
On the MDS create all of the ssh keys, even though in this case the client is using rsa1.
2-66
78-xxxxx-xx
Chapter 2
172.22.36.11# conf t Enter configuration commands, one per line. 172.22.36.11(config)# ssh key rsa1 generating rsa1 key(1024 bits)..... generated rsa1 key ca-9506(config)# ssh key dsa generating dsa key(1024 bits)..... generated dsa key ca-9506(config)# ssh key rsa generating rsa key(1024 bits)..... generated rsa key
Step 3
Step 4
On the MDS create the user, pasting in the contents of the identity.pub file after the sshkey parameter:
172.22.36.11# conf t Enter configuration commands, one per line. End with CNTL/Z. 172.22.36.11(config)# username testuser role network-operator warning: password for user:setmason not set. S/he cannot login currently 172.22.36.11(config)# username testuser sshkey 1024 35 139198677264732164858153476357747926024656548233745027006381178621992083524037906211714241 450436547019604214530354070873624269283640613058470615170649963414635036859628344005142227 886318134122126153182906740418449098047827961768214148936752631482459130056603268404256522 191410368204629699075809390037814979061 testuser@host 172.22.36.11(config)# end
Step 5
172.22.36.11# show user-account testuser user: testuser this user account has no expiry date roles:network-operator no password set. Local login not allowed Remote login through RADIUS/TACACS+ is possible ssh public key: 1024 35 139198677264732164858153476357747926024656548233 74502700638117862199208352403790621171424145043654701960421453035407087362426928 36406130584706151706499634146350368596283440051422278863181341221261531829067404 18449098047827961768214148936752631482459130056603268404256522191410368204629699 075809390037814979061 testuser@host
Step 6
2-67
Account Management
http://www.gnu.org/licenses/gpl.html. 172.22.36.11#
If the same user tries logging in from another host which does not have both the private key file (/users/testuser/.ssh/identity) and the public key (/users/testuser/.ssh/identity), then it will be denied access to the MDS. The fact that the public key has testuser@host at the end of it, does not tie it to a specific host but allows an admin to determine which host it was generated from.
Tip
One simple method to leverage this feature is to set up a scheduled, for example using cron, to backup the switchs configuration on a nightly basis using ssh. This could be done with the following commands executed on a host, provided the user that is used has the privileges to issue the copy command:
#!/bin/sh ##################################################### # #/usr/local/bin/backup_mds_config.sh # This is used for a cron entry. No arguments are # allowed in cron.Absolute paths to commands must # be specified to ssh for it to work properly # ssh key exchange must be separately configured # for the account "USER" # # Adjust the variables for your host and switch ###################################################### DIR=/mds_config DATE=`date "+%m%d%y_%H%M%S"` SWITCH_NAME=beat_bama FILE=$SWITCH_NAME"_run_cfg_"$DATE USER=cwilliams COMMAND1="copy running-config startup-config COMMAND2="show startup-config" #Copy running to startup-config /usr/local/bin/ssh -l $USER $SWITCH_NAME $COMMAND1 #Backup MDS config to local file /usr/local/bin/ssh -l $USER $SWITCH_NAME $COMMAND2 > $DIR/$FILE
Then setup cron to execute the script, the cron job must be run by the user specified in the script. Configure the crontab for the user. This example runs at 11:00pm every Sunday.
#Backup MDS config: 00 23 * * 0 /usr/local/bin/backup_mds_config.sh > /mds_logs/beat_bama1
2-68
78-xxxxx-xx
C H A P T E R
Physical Interfaces
The MDS is a multi protocol switch, and in this section various options will be used to configure the FibreChannel (FC) and Gigabit Ethernet ports. The recipes below show how to configure various parameters and modes for a physical port on the MDS
Configuring FC ports
Port Description
The port description allows a user to provide a description of plain text name for the interface. In the below example the fibre channel interface fc 1/1 is given a description storage array 17 port 1. This is just naming the port on the switch.
switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# interface fc 1/1 switch(config-if)# switchport description storage array 17 port 1 switch(config-if)# end switch#
Port speed
This example show how to the set the port speed for fc 1/1 to 1Gb, 2Gb or auto negotiate speed.
Note
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1
3-69
Physical Interfaces
<- port speed set to 1 GB <- port speed set to 2 GB <- port speed set to auto negotiate
A FC port can be set to only one port mode at any given time. The default is auto on the 16 port linecards and FX on the 32 port linecards.
Setting the port mode to auto, allows the port to negotiate to either a F, FL or E port. fc 1/1 is set to auto port mode. It can not negotiate to either ST, SD or TL port modes. This is the default setting for all ports on a 16 port linecard.
switch# conf t Enter configuration commands, one per line. switch(config)# interface fc 1/1 switch(config-if)# switchport mode auto switch(config-if)# end switch#
3-70
78-xxxxx-xx
Chapter 3
3-71
Physical Interfaces
Note
These same principals apply to port-channels. Just specify the port channel interface, int port-channel 1, rather than an individual link, interface fc 1/1.
with CNTL/Z. auto negotiates trunk port mode sets trunk port mode to on sets trunk port mode to off
3-72
78-xxxxx-xx
Chapter 3
switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# interface fc 1/1 switch(config-if)# switchport trunk allowed vsan all < all VSAN traffic switch(config-if)# switchport trunk allowed vsan add 2 < only VSAN 2 traffic switch(config-if)# switchport trunk allowed vsan add 2-4 < VSAN 2 to 4 traffic switch(config-if)# ^Z switch#
Note
To have one interface become the master interface whenever it is online, also known as premption, set the gigabit Ethernet interface to have the same IP address as the VRRP IP address.
In this example, the configuration is as follows: VRRP ID: 1 VRRP IP address: 192.168.1.40 Switch 1: Interface gige3/3 (192.168.1.20)
3-73
Physical Interfaces
At this point it is a good idea to verify that a host on the local subnet can ping both IP addresses (192.168.1.20 and 192.168.1.30). Alternatively, the ips measure-rtt command could be used to ping from one GigE port to the other.
Switch1# ips measure-rtt 192.168.1.30 interface gigabitethernet 3/3 Round trip time is 172 micro seconds (0.17 milli seconds)
Step 2
Configure the VRRP session on both switches using the VRRP id (1)
Switch1# conf t Enter configuration commands, one per line. End with CNTL/Z. Switch1(config)# interface gigabitethernet 3/3 Switch1(config-if)# vrrp 1 Switch1(config-if-vrrp)# address 192.168.1.40 Switch1(config-if-vrrp)# no shut Switch1(config-if-vrrp)# end Switch2# conf t Enter configuration commands, one per line. End with CNTL/Z. Switch2(config)# interface gigabitethernet 4/1 Switch2(config-if)# vrrp 1 Switch2(config-if-vrrp)# address 192.168.1.40 Switch2(config-if-vrrp)# no shut Switch2(config-if-vrrp)# end
Step 3
Verify that the VRRP session is up and which interface has become the master perform the following:
Switch2# show vrrp vr 1 Interface VR Status ------------------------------------------------------GigabitEthernet3/3 1 backup
switch1# show vrrp vr 1 interface gig3/3 status vr id 1 status MAC address 00:00:5e:00:01:01 Operational state: master Up time 8 sec
3-74
78-xxxxx-xx
Chapter 3
Note
A DPVM configuration will override the VSAN that is assigned to the port. Therefore, changing the VSAN membership of an interface that has a DPVM configured device attached will have no effect on the VSAN of the device. DPVMs CFS scope is physical. DPVM can work in conjunction with Persistent FCIDs, however if the device moves to another switch it will be assigned a different WWN.
After configuring DPVM, if the DPVM assigned VSAN is different from the port assigned VSAN, the operational value for the VSAN will be that of the DPVM assigned value:
3-75
Physical Interfaces
Figure 3-1
Alternatively, in the CLI the operational VSAN assignment will be displayed in the Port vsan field. The configured vsan displays the VSAN that the port would be in if DPVM was not configured.
switch# show int fc2/5 fc2/5 is up Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN) Port WWN is 20:45:00:0c:85:e9:d2:c0 Admin port mode is auto, trunk mode is on Port mode is F, FCID is 0xef0008 Configured Port vsan is 1 Port vsan is 1000 Speed is 2 Gbps Transmit B2B Credit is 7 Receive B2B Credit is 16 Receive data field Size is 2112
Prior to any actual configuration activities take place, DPVM must be enabled via CLI or Fabric Manager:
switch# conf terminal Enter configuration commands, one per line. switch(config)# dpvm enable End with CNTL/Z.
Alternatively, in Fabric Manager, under Logical Domains, select All VSANs, then DPVM. Under the Control tab, set the command field to enable and press the Apply Changes button as in Figure 3-2.
Figure 3-2
After enabling DPVM, proceed to either Adding Existing Devices to DPVM on page 77 or Adding New Devices to DPVM on page 79 for the rest of the recipe.
3-76
78-xxxxx-xx
Chapter 3
Enable DPVM as per Implementing WWN Based VSANs (DPVM) on page 75. In Fabric Manager, select Tools, Other, and then DPVM Setup.
Figure 3-3
Select any of the switches that have been enabled for DPVM. DPVM is a CFS aware application and so the DPVM configuration will be propagated to all switches. If a switch is listed as not having DPVM configured then the wizard will automatically enable DPVM on those switches. Press Next.
Step 3 Step 4
Since this is a new DPVM configuration, select the Create Configuration From Currently Logged In End Devices checkbox and press Next. If an existing DPVM configuration exists, do not check this box. At this point, FM will determine the VSAN assignment of all the devices in the fabric and present a table listing the proposed configuration:
3-77
Physical Interfaces
If additional entries are desired, select Insert... whereby a dialog box will prompt for a WWN and VSAN. If an above entry should be deleted. Select the row to be removed and select Delete.
When the above changes are completed, select Finish which will CFS commit the action. To view the DPVM configuration in Fabric Manager, in the Logical Domains pane, select the Fabric to view, All VSANs and then DPVM. Click the CFS tab, which will activate the other CFS tabs, and then select the Active Database tab:
3-78
78-xxxxx-xx
Chapter 3
Figure 3-5
In the above output, Device Aliases (see Device Aliases, page 1-43) have been enabled therefore the pWWN column displays the Device Alias of the device rather than the WWNs.
Enable DPVM as per Implementing WWN Based VSANs (DPVM) on page 75. To modify the DPVM configuration in Fabric Manager, in the Logical Domains pane, select the Fabric to view, All VSANs and then DPVM. Click the CFS tab, which will activate the other CFS tabs, and then select the Config Database tab:
3-79
Physical Interfaces
Click the Create Row... button, located above the CFS tab. Either select from the list the device or wwn to be added or manually type in the wwn. Then indicate the VSAN to be assigned. Click Create. When all entries have been created, click Close. In this example, Device Aliases have been configured (see Device Aliases, page 1-43), therefore devices aliases are displayed instead of the WWNs.
Next, select the Actions tab. Change the Action column to activate and click the green Apply Changes icon. Using the activate action will ensure that a device that is currently logged into the fabric does not get accidentally moved into another VSAN which could potentially disrupt IO. To determine which devices causing the CFS commit to fail see section on DPVM Conflicting Entries, page 3-82. If the commit should ignore this safety check, then use the activate force action. Since DPVM is CFS enabled, a CFS commit is still required. Also, at this point the Config and Active Databases are different.
3-80
78-xxxxx-xx
Chapter 3
Step 7 Step 8
To Commit the changes click the CFS button and and select Commit. If the commit succeeds, in the bottom left corner of Fabric Manager will display CFS(dpvm):Committed.. At this point the Active Database will now contain the new entry.
To modify the DPVM configuration in Fabric Manager, in the Logical Domains pane, select the Fabric to view, All VSANs and then DPVM. Click the CFS tab, which will activate the other CFS tabs, and then select the Config Database tab:
Select the cell corresponding to the Login VSAN Id that will be modified, and then enter in the new VSAN ID (1000). Then click the Apply Changes button.
3-81
Physical Interfaces
Next, select the Actions tab. Change the Action column to activate and click the green Apply Changes icon. Using the activate action will ensure that a device that is currently logged into the fabric does not get accidentally moved into another VSAN which could potentially disrupt IO. To determine which devices causing the CFS commit to fail see section on DPVM Conflicting Entries, page 3-82. If the commit should ignore this safety check, then use the activate force action. Since DPVM is CFS enabled, a CFS commit is still required. Also, at this point the Config and Active Databases are different. To Commit the changes click the CFS button and and select Commit. If the commit succeeds, in the bottom left corner of Fabric Manager will display CFS(dpvm):Committed.. At this point the Active Database will now contain the new entry, and the CFS tab will display a success.
Step 5 Step 6
3-82
78-xxxxx-xx
Chapter 3
This procedure follows the procedure outlined in Adding New Devices to DPVM, page 3-79. To determine what entries caused the error perform the following:
Step 1
Access the MDS via the CLI using either telnet or SSH. Connect to the switch listed in the CFS tab that contains the erroneous result:
Run the following command to see the conflict, the VSAN listed is the new VSAN for the listed device.
172.22.36.11# show dpvm pending-diff Session is on, Lock Taken DPVM Pending Status ------------------Active DB : Activate Auto Learn : None Pending Database Diff --------------------Legend: "+" New Entry, "-" Missing Entry, "*" Possible Conflict Entry --------------------------------------------------------------------* pwwn 50:06:0e:80:03:4e:95:33 vsan 1000
Step 3
To determine the physical location of this device, in Fabric Manager in the Physical Attributes pane, select the End Devices folder. Then sort by pwwn:
3-83
Physical Interfaces
At this point, if its determined that the port(s) can be safely moved to a new VSAN, then the force activate action can be used, or the conflicting entry should be removed from the config database.
3-84
78-xxxxx-xx
Chapter 3
After logging into the switch enter config mode and the dpvm database:
ca-9506# config terminal Enter configuration commands, one per line. ca-9506(config)# dpvm database ca-9506(config-dpvm-db)# End with CNTL/Z.
Step 2 Step 3
Activate the changes. This can be done from the current CLI prompt or the previous prompt which can be reached by typing exit.:
ca-9506(config-dpvm-db)#dpvm activate
Step 4
Prior to activating the changes, the proposed changes can be seen with the following show command. A conflicting device will cause the CFS commit to fail.
Note
The + represents devices that are being added to the database. The - represents devices that are being removed from the database. The * represents devices that are being modified in the database, including those that are
currently logged into the fabric and are changing VSAN assignment.
Step 5
3-85
Physical Interfaces
3-86
78-xxxxx-xx
C H A P T E R
VSANs
VSANs (Virtual SAN) are a logical fabric. It is a logical grouping of ports in a single switch or across multiple switches that function like a single fabric. Each VSAN has all the required fabric services independent of the other VSANs configured on the same switch or set of switches. VSANs also provide SAN island consolidation on a higher port density physical switch, traffic isolation and increased security. VSANs can be numbered from 1 4094. VSAN 1 and VSAN 4094 are predefined and have very specific roles. VSAN 1 is the default VSAN which holds all the ports by default and the VSAN 4094 is the isolated VSAN into which orphaned ports are assigned.
Note
Moving a port from one VSAN to another will not change its configuratin (F, FL, TL), its speed or its administrative state (shut/noshut). However, any device attached to the port will need to FLOGI back into the switch. In Fabric Manager, on the toolbar select the Create VSAN icon. Select the switch(es) to create the VSAN on. Enter the VSAN ID. Enter in a name for the VSAN. If the VSAN will be attaching to a third party switch select the appropriate Interop mode.
Note
In SAN-OS 2.1(2), a static domainID can be specified for the VSAN at the time of creation. Otherwise the recipe in Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using CLI., page 4-93 should be followed to specify the new domainID. Press Create.
Step 6
4-87
VSANs
In Fabric Manager, in the Physical Attributes pane, expand Switches. Expand Interfaces. Select FC Physical. Modify the Port VSAN Config field for the switchs interface that should be moved to the specified VSAN. Optionally, If the port has an Admin Status of down, it can be enabled by changing the value to up. Press the green Apply Changes... button.
Note
A warning will be displayed informing that moving a port between two VSANs can be disruptive to that port as it will have to relogin to the fabric and will nolonger have access to resources in the previous VSAN. Press Yes to move the port to the new VSAN.
Step 7
Access the VSAN attributes pane by selecting the Fabric in the Logical Domains pane. Expand the VSAN to be modified, in the below example, VSAN 3001 was chosen. Select VSAN Attributes. Make the appropriate changes to the desired fields.
Note
Standard editting keyboard shortcuts (ctrl-x to cut; ctrl-c to copy; ctrl-v to paste) can be used to edit the text fields. Press the green Apply Changes... button.
Step 5
4-88
78-xxxxx-xx
Chapter 4
Figure 4-1
Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using Fabric Manager.
Within a VSAN, the domain manager process on the principal switch in a fabric is responsible for assigning a domain_ID to switch that is joining the fabric. When a switch boots up or joins a new fabric it can request a specific domain_ID or take any available domain_ID. A domain_ID can be configured in one of two ways:
Preferred: The new switch will request a specific domain_ID, however, if it receives a different
domain_ID, it will isolate itself from the fabric. This is the case when the same domain_ID must be maintained under all circumstances. After obtaining the domain_ID from the principal switch in the VSAN, the local switch will assign Fibre Channel Identifiers (FC_IDs) to each end device as they log in to the fabric. A process known as FLOGI (Fabric Login).
Tip
HPUX and AIX are two operating systems that utilize the FC_ID in the device path to the storage. In order to have the switch always assign the same FC_ID to a device across switch reboots, persistent FC_ID and static domain_ID must be configured for the VSAN. If an FC_ID changes for a device accessed by either an AIX or a HPUX host, the host may lose access to the device. By default, the switch will assign the same FC_ID to a device, however, if the switch is rebooted this database of pwwn/FC_ID mapping is not maintained. Enabling persistent FC_IDs will make this database persistent across reboots.
4-89
VSANs
FC_ID database is manually purged by the user this entry will be deleted. These entries are persistent across reboots of the switch and are VSAN specific.
Static: The FC_ID is determined by the user prior to attaching the device to the switch. If the
persistent FC_ID database is manually purged by the user these entries will not be removed. These entries are persistent across reboots of the switch and are VSAN specific. When persistent FC_ID is enabled, the MDS will make persistent all of the devices in that VSAN, therefore the admin is not required to manually type in devices entering that VSAN.
In this procedure an existing VSAN (3000) on switch 172.22.36.11 with domain_ID 239 will be statically statically configured and then persistent FCID will be enabled using FM. This recipe does not alter the running domain_ID.
In Fabric Manager, in the Logical Domains pane, expand the VSAN to be modified and then select the Domain Manager entry. Select the Configuration tab. Enter the domain_ID that is in the Running domain_ID (239) column into the Config domain_ID column (239). Change the IdType field to static.
Figure 4-2 Enabling Static domain_ID
4-90
78-xxxxx-xx
Chapter 4
Press the green Apply Changes... button. Select the Persistent Setup tab. Select the Enable checkbox. Press the green Apply Changes... button. At this point, the domain_ID has been statically set and FC_IDs will remain persistent across reboots for VSAN 3000 on the switch 172.22.36.11. The persistent FC_ID database can be viewed in the Persistent FCIDs tab.
Figure 4-3 Persistent FC_ID database
4-91
VSANs
switch(config-vsan-db)# vsan 200 name TapeVSAN switch(config-vsan-db)# vsan 200 interface fc 1/1 switch(config-vsan-db)# ^Z switch#
This recipe shows the steps to set the interop modes 1, 2, 3 for a VSAN. Interop mode 1 (ensure that the domain ID of the VSAN is between 97 127 for this mode to work). To change the interop mode of VSAN 200 to interop mode 1:
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 interop 1 switch(config-vsan-db)# ^Z switch#
Interop mode 2 (for Brocade switches running corePID mode 0). To change the interop mode of VSAN 200 to interop mode 2.
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 interop 2 switch(config-vsan-db)# ^Z switch#
Interop mode 3 (for Brocade switches running corePID mode 1). To change the interop mode of VSAN 200 to interop mode 3.
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 interop 2 switch(config-vsan-db)# ^Z switch#
4-92
78-xxxxx-xx
Chapter 4
switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# vsan database switch(config-vsan-db)# vsan 200 loadbalancing src-dst-id switch(config-vsan-db)# ^Z switch#
Converting an Existing VSAN to Static DomainID and Enabling Persistent FCID using CLI.
This recipe shows the steps to configure static Domain_ID for a VSAN and also how to enable persistent FC_ID for the same VSAN. In this procedure an existing VSAN (3000) on switch 172.22.36.11 with domain_ID 239 will be statically statically configured and then persistent FCID will be enabled using FM. This recipe does not alter the running domain_ID.
Step 1
4-93
VSANs
Step 2
Step 3
Step 4
Note
If the domain ID of VSAN 200 is different than what is currently running (22 in this case) then the VSAN will have to be restarted for the configuration changes to the Domain_ID and FC_ID persistence to take effect. Changing Domain_IDs and hence FC_IDs for a device is disruptive as an end device will have to relogin to the fabric (FLOGI) to obtain a new FCID.
Warning
Changing Domain_IDs and therefore FC_IDs for a device is disruptive, as an end device will have to relogin to the fabric (FLOGI) to obtain a new FCID. However, making a Domain_ID static without changing its value is not disruptive.
Restarting a VSAN
Sometimes the VSAN on a switch will need to be restarted. For example, after changing the Domain_ID of a VSAN, the VSAN should be restarted for the new Domain_ID to take effect. The recipe below shows how a VSAN (200) can be restarted.
switch# conf t Enter configuration commands, one per line. switch(config)# vsan database switch(config-vsan-db)# vsan 200 suspend switch(config-vsan-db)# no vsan 200 suspend switch(config-vsan-db)# end
4-94
78-xxxxx-xx
Chapter 4
Note
A new FCID cannot be assigned to a pwwn which is already logged into the fabric. Log the device out of the fabric, for example by shutting the FC interface before assigning the new FCID. FC_ID 0x160000 will be assigned to pwwn 50:06:0b:82:bf:d1:db:cd permanently. Therefore when the pwwn logs into the switch (FLOGI) it will get this assigned FC_ID.
Note
The FC_ID to be assigned (0x160000) should contain the same Domain_ID (0x16) as the currently running domain in the VSAN.
switch# conf t Enter configuration commands, one per line. End with CNTL/Z. switch(config)# fcdomain fcid database switch(config-fcid-db)# vsan 22 wwn 50:06:0b:82:bf:d1:db:cd fcid 0x160000 dynamic switch(config-fcid-db)# ^Z switch#
4-95
VSANs
4-96
78-xxxxx-xx
C H A P T E R
Zoning
Zones and zonesets are the basic form of datapath security within a fibre channel environment. A zoneset is a collection of zones which in turn have individual members in them. Only those members within the same zone can communicate with each other. A device can be a member of multiple zones and those devices not in a zone are in the default-zone. The policy for the default-zone can either be permit (devices can see each other) or deny (devices in the default-zone cannot see each other). The basic flow of zoning is to:
1. 2. 3. 4. 5.
Create zones Add hosts and storage to the zones. Create Zonesets. Add the zones to Zonesets Activate the Zoneset.
This chapter will focus on the creation of zones, zonesets and how to manipulate them. In both basic zoning (FC-GS-3) and Enhanced Zoning (FC-GS-4) which was introduced in SAN-OS 2.0, both use the concepts of zones and zonesets.
Enhanced Zoning
Introduced in SAN-OS 2.0, Enhanced Zoning, defined in the FC-GS-4 and FC-SW-3 standards provides significant enhancement to basic zoning.
Has a VSAN scope, so that while VSAN X is using enhanced zoning. Other VSANs can continue to use basic zoning. IVR compatible. Provides session locking so that two SAN administrators cannot modify the zoning database within a VSAN at the same time, Provides implicit full zoneset distribution so that the zoneset database that is local to each switch remain in sync when a zoneset is modified. Full zoneset changes can be distributed without having to activate a zoneset. This can be leveraged if changes want to be readied during the daytime, and then have the zoneset activated during the night.
5-97
Zoning
Modifications are staged and have to be explicity committed or aborted, allowing the SAN admin to review all changes prior to them going into production. Can control how a zone merge should be done. Either by performing a union of the two zonesets according to the same rules as basic zoning, or by a strict policy in which if the two candidate active zonesets are not exactly the same, the two zonesets will not merge. Thereby preventing accidental merging. Enhanced Zoning has a similar behavior to CFS: Cisco Fabric Services, page 1-13.
Enhanced zoning continues to use the same techniques and tools as basic zoning with a few added commands which will be covered in the subsequent recipes. However, the flow of using enhanced zoning is slightly different. First, a VSAN wide lock, if available is implicitly obtained. Next, all zone and zoneset modifications are done including activation. Last, the changes are either committed (put into production) or aborted (pending changes are scrapped). The flow is illustrated below:
Figure 5-1 Enhanced Zoning Flowchart
5-98
78-xxxxx-xx
Chapter 5
Enhanced zoning only needs to be enabled on one switch in the VSAN. Attempting to enable it on multiple switches in the same VSAN via Fabric Manager can result in a failure to activate. Enabling enhanced zoning does not cause a zoneset activation. The switch that is chosen to initiate the migration to enhanced zoning will distribute its full zone database to the other switches in the VSAN. Thereby overwriting the destination switches full zoneset database. Therefore it is critical that you select the correct switch to enable enhanced zoning on to prevent deleting your full zoneset database.
Step 2
In the Logical Domains pane select the VSAN to enable VSANs and then the folder corresponding to the name of the active zoneset. If no active zoneset exists choose default zone instead. Select the Enhanced Tab. In the action column for the switch you want to enable enhanced zoning from, change the cell to Enhanced. Remember that for this VSAN, this switch will distribute its full zone database thus overwriting other switches full zone database.
5-99
Zoning
Figure 5-2
Step 4
Altneratively in Fabric Manager to display the user with the lock, select the Logical Domains Pane, the VSAN to be investigated, the name of the Zoneset (or Default Zone if there is no active zoneset), then the Enhanced Tab. The user is displayed in the Config DB Locked By column:
5-100
78-xxxxx-xx
Chapter 5
Zoning ZoneSets
Figure 5-3
ZoneSets
Zonesets are containers of zones and of which there are two types on the MDS platform:
1.
Active Zoneset: is the rules by which the MDS platform enforces its zoning security policy.It cannot be modified and is distributed to all switches in the VSAN. There are specific rules to merging the active zoneset when two switches are connected by an ISL as set by the FC standards. Local Zoneset: The local zonesets are contained in the full zoneset database on the switch. The zonesets are editted directly and then activated to become the active zoneset. They can optionally be distributed to other switches, either manually or when a zoneset is activated.
2.
Zoneset Distribution
The zoneset in the full zoneset database can be distributed to other switches either during activation or manually when basic zoning is enabled. When enhanced zoning is enabled, the full zoneset is always distributed upon committing any changes to the full zoneset database. It is always enabled for enhanced zoning.
5-101
Chapter 5 Zones
Zoning
Tip
This feature should be enabled on all switches in the fabric, and can be initially specified in the initial setup script. This is the default behavior when enhanced zoning is enabled.
ca-9506# conf t Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zoneset distribute full vsan 804
Zones
In order for two devices to communicate, they must be in the same zone. Valid members of a zone can be:
Port WWN FC Alias FCID FWWN (WWN of a fc interface) Switch Interface (fc X/Y) Symbolic nodename Distributed Device Alias
The three most common types of zone members are the pwwn, Device Alias, FC alias and the switch interface.
Tip
It is recommended that pwwn or Device Alias be used for zoning as it provides hardware enforced zoning and ties a zone member to a specific hba rather than to the switch port. Also, Device Aliases have the added benefit of being VSAN independent and are based on an easy to understand name rather than a cryptic pwwn. Equally important is the name of the zone. Many environments use different zone names, however, all name formats should provide relevant information as to their contents. Names like Zone1 or TapeZone do not provide sufficient information about their contents.
Tip
A zone name, should contain two members and within the zone name contain identifiers related to the two devices (eg: Z_testhost_fcaw0_symm13FA3aa). It may be longer than Z_testhost_hba0, but provides detailed information about the contents without having to consult further documentation.
5-102
78-xxxxx-xx
Chapter 5
Zoning Zones
Zoneset: ZS_SJC_Bldg6 Solaris Host: ca-sun1,hba lpfc0 HDS Array: HDS20117-c20-8 First a zoneset will need to be created.
Step 1
In the Logical Domains pane, right click on the VSAN, and select Edit Local Full Zone Database.
5-103
Chapter 5 Zones
Zoning
Figure 5-5
Note
The VSAN field displays the VSAN to which the zone database will be modified and The Switch field shows which switch is being editted. The Name column represents either FC Aliases or Global Device Aliases, (Device Aliases, page 1-43) if they are used. If Full Zoneset Distribution is enabled, and there are existing zones then the left hand column should contain the existing zonesets and zones. If Active Zoneset Distribution is enabled, then choose the switch that contains the Full Zone Database.
In the left pane, Right click on Zonesets. Select Insert... Enter a Zoneset name such as ZS_SJ_Bldg6 and press OK.
At this point a zoneset has been created. The next phase is to create a zone and add members to it.
Step 1 Step 2 Step 3
Right click on Zones. Select Insert... Enter a meaningful zone name such as ca-sun1_lpfc0_HDS20117-c20-8 to represent both the initiator and the target in the zones name.
5-104
78-xxxxx-xx
Chapter 5
Zoning Zones
Step 4
Step 5
Next, drag the two end devices from the bottom pane into the zone that was just created. This will create a pwwn based zone. If non-pwwn zone members are desired, such as interface, FCID or Global Device Alias see How to Create Non-PWWN based zones, page 5-108 to specify these member types before continuing to the next step.
5-105
Chapter 5 Zones
Zoning
Figure 5-7
This completes the steps to create the zone. Next, the zone must be added to the zoneset.
In the left hand pane, drag the zone (ca-sun1_lpfc0_HDS20117-c20-8) into the zoneset (ZS_SJC_Bldg6). The zonesets icon will change by appending a folder icon and it will expand with the newly added zone underneath it.
5-106
78-xxxxx-xx
Chapter 5
Zoning Zones
Figure 5-8
The last phase is to Activate the Zoneset. This will instruct the switch program its Access Control Lists, modify the running configuration of the zoneserver to finally allow the two devices to communicate. When enhanced zoning is enabled, the activate button will commit the changes as well.
Step 1 Step 2
In the left pane, right click the Zoneset (ZS_SJC_BLDG6). Choose Activate...
a. b.
If the current Active Zoneset is empty, then press Continue Activation... If the current Active Zoneset is not empty, pressing Proposed Changes will display what is being added or removed from the active zoneset. Then press Continue Activation...
Zoneset Activation Proposed Changes
Figure 5-9
5-107
Chapter 5 Zones
Zoning
After activating the zoneset, the two end devices will be able to communicate.
In the Logical Domains pane, right click on the VSAN, and select Edit Local Full Zone Database. In the resulting dialog box, Right click on Zones and select Insert... Specify a zone name and press OK. Right click on the newly created zone and select Insert...
Figure 5-10 Possible zone member types
Step 5
Select the button for the type of zone member required. The labels for the three text boxes will change depending on the selection in the Zone By panel. For example, if Switch & Port is selected, then the text boxes change to Switch Interface (e.g: fc1/1) and Switch Address (e.g. 192.168.1.2). Also the meaning of the ... and pulldown buttons change depending on the selection in the Zone By panel.
Note
Domain & Port zoning should only be done when working in interop mode 2 or 3. See section Setting VSAN Interop Mode, page 4-92 for more information about interop modes.
5-108
78-xxxxx-xx
Chapter 5
Zoning Zones
Alias refers to both FC Alias and Global Device Alias depending on the mode that Fabric Manager is running in.
Step 6
The resulting zone still needs to be added to a zoneset and then the zoneset needs to be activated, which is outlined in Zone Creation and Adding it to a Zoneset with Fabric Manager, page 5-103.
Step 1
Create the zone, building a zone name that reflects the names of the members.
ca-9506# config terminal Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zone name Z_host1_fcaw0_symm78FA03ab vsan 804 ca-9506(config-zone)# member pwwn 22:35:00:0c:85:e9:d2:c2 ca-9506(config-zone)# member pwwn 10:00:00:00:c9:32:8b:a8
Step 2
Step 3
5-109
Chapter 5 Zones
Zoning
Step 4
Finally, to put the zoneset into production activate the zoneset using zoneset activate name ZS_Engr_primary vsan 804. This will activate all the zones in the zoneset, not just the one that was just added. For enhanced zoning, in addition to activating the zoneset, it needs to be committed.
ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804
Step 5 Step 6
Step 7
If enhanced zoning is enabled, then the changes to the pending database should be committed and therefore be distributed to the other switchs full zone database:
ca-9506(config)# zone commit vsan 804
Step 1
5-110
78-xxxxx-xx
Chapter 5
Zoning Zones
Step 2
Step 3
Step 4
Step 5
For basic zoning, put the zoneset into production activate the zoneset using zoneset activate name ZS_Engr_primary vsan 804. This will activate all the zones in the zoneset, not just the one that was just added. For enhanced zoning, in addition to activating the zoneset, it needs to be committed.
ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804
Step 6
Step 7
Tip
Aliases are distributed with the full zoneset database, thus if zoning is going to be editted on multiple switches, full zoneset distribution should be enabled. An alias can be mapped to more than one device, however it is recommended that a one to one mapping be used.
5-111
Chapter 5 Zones
Zoning
Solaris host1, hba instance fcaw0: 22:35:00:0c:85:e9:d2:c2 Symmetrix 78, FA port 03ab: 10:00:00:00:c9:32:8b:a8
Figure 5-13 Alias Zoning Topology
Step 1
Step 2
Step 3
Step 4
Step 5
5-112
78-xxxxx-xx
Chapter 5
Zoning Zones
Step 6
Tip
Interface based zoning should be done when a zone needs to be created prior to the hba being connected to the fabric. After, the hba is connected to the fabric, the zone member should be converted to a pwwn based member.
Figure 5-14 Interface Zoning Topology
Step 1
Step 2
Note
The swwn is the switchs wwn as displayed by the show wwn switch command:
ca-9506# show wwn switch Switch WWN is 20:00:00:0c:85:e9:d2:c0
Step 3
5-113
Chapter 5 Zones
Zoning
ca-9506# conf t Enter configuration commands, one per line. End with CNTL/Z. ca-9506(config)# zoneset activate name ZS_Engr_primary vsan 804 Zoneset activation initiated. check zone status
Step 4
5-114
78-xxxxx-xx
C H A P T E R
Inter-VSAN Routing
Inter-VSAN Routing (IVR), first introduced into the MDS platform in SAN-OS 1.3(1), provided the ability for devices in different VSANs to communicate. This facilitated the ability to create VSANs consisting of devices which could be shared with devices in different VSANs. The classic example is to share a tape library with many hosts which are in different VSANs, or to allow disk subsystems to communicate over WAN distances without having to merge large zoneset databases. This initial release of IVR (IVR-1) suffered from two deficiencies in that it required unique domain IDs in the source and destination VSANs and secondly that the two VSANs could not be the same number. Although, these two issues could be easily solved with some simple planning. If the requirement was needed to merge two large fabrics and then IVR between them, there maybe duplicate VSAN IDs or domain IDs in the two fabrics. Introduced in SAN-OS 2.1, IVR obtained the ability to perform Network Address Translation (NAT). IVR with FCNAT (IVR-2) solved both of the issues with IVR-1, as it eliminated the need for unique domain IDs and VSAN IDs. IVR-2 continues to use the same basic principles as in IVR-1 such as IVR topologies, IVR zones and zonesets as well as transit VSANs. In addition to the FCNAT capabilities, IVR also gained the ability to leverage CFS (introduced in SAN-OS 2.0(1)) and auto-topology (introduced in SAN-OS 2.1(1)). Although these two technologies make implementing IVR easier and faster, careful planning should be done prior to configuration.
Tip
The preferred method of configuring IVR either with or without NAT is with Cisco Fabric Services (CFS). This will ease topology configuration and reduce the number of configuration steps and potential configuration errors. See IVR with CFS, page 6-115
6-115
Chapter 6
Inter-VSAN Routing
Modifying the topology is still a manual operation, but the user nolonger needs to go to every switch (using either FM or CLI) and modify those switches. For example, a fabric with three switches in it would require nine entries into the topology database (3 switches with IVR * 3 VSAN Route Switches). Adding a fourth IVR switch would increase this from nine to sixteen (4 switches with IVR * 4 VSAN Route Switches). Using the following topology as a reference:
Figure 6-1 CFS Reference Topology
In the below example, which only contains two switches, Figure 6-2 on page 6-116 shows two switches, 172.22.36.9 and 172.22.36.11 which have duplicate entries for the VSAN Route Switch. Thus if a VSAN needed to be added to the VSAN route switch 172.22.36.11, I would need to modify two rows. One for each unique switch in the master column.
Note
If CFS is to be used with IVR, all IVR enabled switches must have CFS distribution for IVR enabled. Conversely, If CFS is not going to be used for IVR, then all of the IVR enabled switches should have CFS distribution for IVR disabled.
Figure 6-2
6-116
78-xxxxx-xx
Chapter 6
Inter-VSAN Routing
If the same configuration was done with CFS, then I would have one row for each VSAN Route Switch and would not have to enter the same information for each switch in the fabric as in Figure 6-3 on page 6-117.
Figure 6-3 IVR Topology with CFS
It turns out that only one row per IVR enabled switch is displayed making the topology table easier to comprehend. Looking only at columns two and three. Its easy to tell that 172.22.36.11 routes between VSANs 1, 3000-3002, while switch 172.22.36.9 routes between 1, 3001 and 3003. All of the duplicate information is removed. The first column represents the switch that FM will use to perform any CFS operations from.
Tip
Instead of using multiple transit VSANs, look to use a single transit VSAN extended over multiple switches. It will simplify the topology while providing the isolation of a transit VSAN.
6-117
Chapter 6
Inter-VSAN Routing
Figure 6-4
Prior to configuring the topology, the topology needs to be determined. By examining the diagram, the following entries will need to be configured.
Table 6-1 IVR Topology Table
Tip
Creating a table such as Table 6-1 provides simple documentation and can be easily added to a proper implementation plan or detailed design document. In FM, in the Logical Domains pane, select the Fabric, All VSANs then IVR. Click the CFS tab to activate the other tabs, and then Local Topology.
Step 1 Step 2
6-118
78-xxxxx-xx
Chapter 6
Inter-VSAN Routing
Figure 6-5
Click the blue Create Row... button. In the resulting pop up box, reference Table 6-1 on page 6-118, in the Switch pulldown menu, select the first switch (172.22.36.11) and its associated VSAN list. Fill in the switchs VSAN List (3000, 3001).
Figure 6-6 Create Topology
Step 6 Step 7
Click Create. Repeat this procedure for the second and third switch. When the third switch has been created. The dialog box can be dismissed by selecting Close. The local topology should look like Figure 6-7 on page 6-120.
6-119
Chapter 6
Inter-VSAN Routing
Figure 6-7
Local Topology
Select the Action tab. Select the Activate Local checkbox, and press the green Apply Changes button. Select the CFS pull down menu and select Commit. The message CFS(ivr):Committed should display in the bottom left hand corner of Fabric Manager. To verify, select the Active Topology tab and the following should be displayed:
Figure 6-8 Active Topology
At this point, the topology is correctly defined and the Active Topology will contain the correct information. Correct configuration and distribution via CFS will result in the Discrepancies tab containing no entries. From this point IVR Zonesets can be defined providing connectivity between the two end devices.
6-120
78-xxxxx-xx
Chapter 6
IVR-1
IVR-1 is the method of Inter-VSAN Routing that has existed since SAN-OS 1.3 and does not perform any Network Address Translation (NAT) functions. It requires unique VSAN and domain IDs across the IVR topology. It has the ability to leverage CFS for its configuration distribution and application locking. Although IVR-1 can be accomplished without the use of CFS, the recipes contained in this chapter will focus on using CFS.
Note
Enable IVR-1
Prior to configuring IVR-1, it needs to be enabled, this can be done via CLI or FM. Any switch that is acting as a border switch or one that will route frames between VSANs should be enabled in the same manner. This includes CFS, such that either all the IVR switches have CFS enabled for IVR, or none of them do.
Step 2
By default, CFS is not enabled for IVR-1, so enable CFS distribution. Notice that the scope is physical as IVR crosses VSANs.
172.22.36.11(config)# ivr distribute 172.22.36.11(config)# do show cfs application name ivr Enabled Timeout Merge Capable Scope : : : : Yes 30s Yes Physical
Step 3
Verify that all the switches are recognized by IVR using CFS.
172.22.36.11# show cfs peers name ivr Scope : Physical -------------------------------------------------Switch WWN IP Address -------------------------------------------------20:00:00:05:30:00:68:5e 172.22.36.11 [Local] Total number of entries = 1
6-121
Chapter 6 IVR-1
Inter-VSAN Routing
In the Logical Domains pane, select All VSANs, then IVR. Under the control tab, change the command to enable for the switches that should have IVR enabled. Then click the Apply Changes button. The status field should change from disabled to enabled.
Figure 6-9 Enable IVR in Fabric Manager
Step 2
Next, enable CFS distribution for IVR. Click the CFS tab, then under the Enable Admin column, change the column from noSelection to enable and press the green Apply Changes button. The Last result column should display success.
Figure 6-10 Enable CFS Distribution for IVR
6-122
78-xxxxx-xx
Chapter 6
Enable IVR as per Enable IVR-1, page 6-121. In FM, in the Logical Domains window, select the fabric, All VSANs and then IVR. In the Local Topology tab click the Create Row... button. Enter in the VSANs to be routed in the VSAN list: 3000,3002, and select the switch that will perform the routing from the pull down menu. The click Create. The word success should be displayed in the botton of the window, then click Close.
Figure 6-12 Single Switch IVR-1 Create Topology
Step 5 Step 6
To activate the topology, click the Action tab, click the checkbox Activate Local and click the Apply Changes button. The topology is not active until it has been CFS committed. To CFS commit the topology, click the CFS pulldown menu button and select Commit. In the bottom left hand corner of FM, the message CFS(ivr):Committed should be displayed. Also, the Active Topology tabs contents should match that of the Local Topology tabs.
The next phase is to create the IVR zoneset and accompaning zones:
Step 1 Step 2
In FMs Logical Domains pane, select the fabric, All VSANs and then right click on IVR. Select Edit Local Full Zone Database In the resulting dialog box, to create the IVR zoneset right click in the left hand pane on Zonesets and choose Insert. Pick a name for the zoneset and click OK. The resulting zoneset will appear in the left hand pane.
6-123
Chapter 6 IVR-1
Inter-VSAN Routing
To create the IVR Zone, in the left hand pane, right click on Zones and select Insert. Enter a meaningful zonename and click OK. Add the members to be zoned from the bottom pane into the newly created zone by dragging them into the zone. In the left hand pane, drag the zone into the zoneset:
Figure 6-13 Single Switch IVR-1, Create Zoneset
Step 6 Step 7
To activate the zoneset and implicitly CFS commit it, right click on the zoneset and select Activate. Then Continue Activation. Click Close to return to the main FM window. The IVR members can be displayed on the map by selected the IVR Zone in the Logical Domains pane.
6-124
78-xxxxx-xx
Chapter 6
IVR-2: FC-NAT
In SAN-OS 2.1(1a), Cisco introduced IVR with Fibre Channel Network Address Translation (NAT). This provides the user with the ability to route between fabrics containing duplicate domainIDs between VSANs, and routing between two VSANs of the same VSAN ID. In addition, instead of representing each native VSANs domain with a unique virtual domain, a single virtual domain is used to represent all of the domains in the native VSAN. Lastly, IVR-FCNAT, leveraging the CFS infrastructure can automatically discover the topology to use thus alleviating the process of configuring and maintaining the IVR topology.
Enabling IVR-2
This recipe provides the steps to enable IVR-2, with CFS and Auto Topology discovery. Prior to using IVR-2, it must be enabled using the following procedure.
Note
In a multi switch topology, these steps should be followed for all the switches in the fabric. Remember, that CFS commands (for example the adding of the NAT and Auto Topology) only have to be done from one switch as CFS will instruct the other switches.
6-125
Inter-VSAN Routing
Step 1
In the Logical Domains pane, select All VSANs, then IVR. Under the control tab, change the command to enable for the switches that should have IVR enabled. Then click the Apply Changes button. The status field should change from disabled to enabled.
Figure 6-15 Enable IVR in Fabric Manager
Step 2
Next, enable CFS distribution for IVR. Click the CFS tab, then under the Enable Admin column, change the column from noSelection to enable and press the green Apply Changes button. The Last result column should display success.
Figure 6-16 Enable CFS Distribution for IVR
Note
Auto topology is not required for IVR-2 with FC-NAT. If it is not enabled, then the topology will need to be manually defined. If Auto Topology is not used then Step 4 can be skipped. Next to enable the FC-NAT and auto topology discovery functions of IVR, select the Action tab and check the Enable IVR Nat checkbox
Step 3
6-126
78-xxxxx-xx
Chapter 6
Step 4 Step 5
Check the Automatically Discover Topology check box. Press the Apply Changes button. The configuration is not yet active until it has been CFS committed.
Figure 6-17 Enabling IVR-2s FC-NAT and Auto Topology Discovery
Step 6 Step 7
Goto the CFS tab, and in the Config Changes Action column, change the selection to Commit. Then press the Apply Changes button. If you switch to the Active Topology tab, the current active topology will be displayed:
Figure 6-18 IVR-2 FC-NAT Auto Discovered Topology
At this point IVR-2 is enabled for CFS distribution, NAT and automatic topology discovery.
6-127
Inter-VSAN Routing
Warning
Upgrading from IVR-1 to IVR-2 will be disruptive to IVR based traffic but is not disruptive to non-IVR traffic, for example non IVR zones contained in a VSAN. Upgrading from IVR-1 to IVR-2 with NAT may cause virtual devices to change their FCIDs, thus requiring FCID dependant hosts such as HPUX and AIX to rescan for the devices.
Note
Within a single physical fabric, IVR-1 cannot coexist with IVR-2. Either the border switches running IVR are in IVR-1 mode or IVR-2 mode. Mixed configurations are not supported. CFS can be leveraged to make sure all switches are running the same configuration.
Step 1 Step 2
Backup the configuration for all the of MDS switches using a procedure similar to the recipe in Copying files to or from the MDS, page 1-19. Upgrade all IVR border switches to SAN-OS 2.1(1a) or later, as IVR-2 with FC-NAT was first introduced in SAN-OS 2.1(1a). Switches that are not acting as IVR border switches are not required to be upgraded. However, it is recommended that they be upgraded. For the recipe on upgrading SAN-OS firmware see Firmware Upgrades, page 1-22. Deactivate the IVR Zoneset. This will not delete the IVR zoneset from the local database as it will be reactivated once IVR-2 has been enabled. In FM, in the Logical Domains pane, select your Fabric, All VSANs, and then right-click on IVR. Select Deactivate Zoneset from the resulting pop-up menu. Upon selecting OK, all devices will now be isolated from devices in other VSANs. Upon successful deactivation, the pop-up box can be dismissed by clicking Close. Enable CFS for IVR for all switches that will perform IVR. In the Logical Domains pane, select the Fabric, All VSANs, then IVR. Click the CFS tab in the top pane. In the Enable Admin column, change the field to enable and press the green Apply Changes button.
Figure 6-19 Enable CFS
Step 3
Step 4
6-128
78-xxxxx-xx
Chapter 6
Step 5
Enable FC-Nat and Auto Topology. First select the CFS tab to determine the CFS master switch. Then select the Action tab. Only one switch should be listed here, it is the switch that FM will use to perform the configuration distribution from. Select both the Enable IVR Nat and the Automatically Discover Topology checkboxes. Press the green Apply Changes button.
Note
Although there may still be a Local Topology present on the switch, Auto Topology takes care of the need to modify the Local Topology database. Select the CFS button and choose Commit. In the bottom left corner of FM, the message CFS(ivr) Committed should be displayed. Activate the IVR Zoneset. In the Logical Domains pane, select the Fabric, All VSANs, then right click on IVR. Select Edit Local Full Zone Database from the resulting menu. Right click on the zoneset that was previously deactivated and select Activate. Select Continue Activation in the resulting confirmation window. It will take a few seconds to commit the changes and save the running-configuration to startup. Click Close to return to the main FM window.
At this point the MDSs have been upgrade to IVR-2 with FC-NAT, CFS and Auto Topology. There may be hosts such as HPUX and AIX that have their disks tied to the FCID. Since NAT is being performed, appropriate action should be performed to have the hosts rescan for the new FCIDs.
Tip
HPUX and AIX are two operating systems that utilize the FC_ID in the device path to the storage. In order to have the switch always assign the same FC_ID to a virtual device across switch reboots, persistent FC_ID for IVR must be configured. If the FC_ID changes for a device accessed by either an AIX or a HPUX host, the host may lose access to the device. This example will be used to illustrate how to configure a storage device to use a specific FCID in the hosts VSAN. The actual devices are already logged into the fabric and the IVR topology has already been created to include VSANs 3000 and 3002. IVR Features Enabled: CFS IVR with NAT Host PWWN: 10:00:00:00:c9:32:8b:a8 VSAN 3002
6-129
Inter-VSAN Routing
Storage PWWN: 50:06:0e:80:03:4e:95:33 VSAN 3000 Real FCID: 0xef0002 FCID to be configured in Host VSAN: 0x630063 IVR Topology:
AFID SWITCH WWN Active Cfg. VSANS -------------------------------------------------------------1 20:00:00:05:30:00:68:5e * yes yes 3000,3002
Step 1
Enter the IVR fcdomain configuration mode for the AFID and VSAN for the location of the virtual device. In this case its the AFID and VSAN where the storage will be virtual.
switch# conf t switch(config)# ivr fcdomain database autonomous-fabric-num 1 vsan 3002 switch(config-fcdomain)#
Step 2
Enter the native AFID, VSAN of the storage device and domain to be used in the hosts VSAN. CFS has been enabled for IVR so any changes will need to be committed later. The domainID in this command is in decimal format.
switch(config-fcdomain)# native-autonomous-fabric-num 1 native-vsan 3000 domain 99 fabric is locked for configuration. Please commit after configuration is done. switch(config-fcdomain-fcid)#
Step 3
Specify the pwwn and FCID to be used. Alternatively, a device-alias can be used instead of a pwwn. The virtual domain and FCIDs will not be created until the zoneset is activated.
switch(config-fcdomain-fcid)# pwwn 50:06:0e:80:03:4e:95:33 fcid 0x630063 The FCID should correspond to virtual domain 99 specified earlier for this mode switch(config-fcdomain-fcid)#
Step 4
Step 5
CFS commit the changes for IVR to activate the IVR zoneset and the modifications to the IVR persistent FCID database.
switch(config)# ivr commit commit initiated. check ivr status
Step 6
6-130
78-xxxxx-xx
Chapter 6
---------------------------------------------------1 3002 50:06:0e:80:03:4e:95:33 0x630063 [HDS20117-c20-8] Number of Virtual-fcid entries: 1 switch# show fcns database vsan 3002 VSAN 3002: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x1c0003 N 10:00:00:00:c9:32:8b:a8 (Emulex) scsi-fcp [ca-sun1_lpfc0] 0x630063 N 50:06:0e:80:03:4e:95:33 scsi-fcp [HDS20117-c20-8] Total number of entries = 2 switch# show zoneset active vsan 3002 zoneset name ZS_test vsan 3002 zone name zone1 vsan 3002 pwwn 50:06:0e:80:03:4e:99:99 pwwn 50:06:0e:80:03:4e:98:98 zone name IVRZ_IVRZ_host1_lpfc0_Array1_port12 vsan 3002 * fcid 0x630063 [pwwn 50:06:0e:80:03:4e:95:33] [HDS20117-c20-8] * fcid 0x1c0003 [pwwn 10:00:00:00:c9:32:8b:a8] [ca-sun1_lpfc0]
6-131
Inter-VSAN Routing
Step 1 Step 2
Enable IVR-2 with CFS, FC-NAT and automatic topology discovery as per Enabling IVR-2, page 6-125. Create the IVR Zones and Zoneset. In the Logical Domains pane, select the Fabric, All VSANs and then right click on IVR and select Edit Full Local Zone Database. The resulting dialog box will look exactly like a non-IVR zoning dialog box:
Figure 6-21 IVR-2 Create Zoneset
Step 3
In the resulting dialog box, to create the IVR zoneset right click in the left hand pane on Zonesets and choose Insert. Pick a name for the zoneset and click OK. The resulting zoneset will appear in the left hand pane. To create the IVR Zone, in the left hand pane, right click on Zones and select Insert. Enter a meaningful zonename and click OK. Add the members to be zoned from the bottom pane into the newly created zone by dragging them into the zone. In the left hand pane, drag the zone into the zoneset:
6-132
78-xxxxx-xx
Chapter 6
Step 7
To activate the zoneset and implicitly CFS commit it, right click on the zoneset and select Activate. Then Continue Activation. Click Close to return to the main FM window.
6-133
Inter-VSAN Routing
At this point, the host ca-sun1 can access the storage on HDS20017-c20-8.
6-134
78-xxxxx-xx
Chapter 6
In this configuration, the switch 172.22.36.11 is currently performing IVR-FCNat between the host in VSAN 3000 and the storage in VSAN 3002. IVR w/FC-NAT using CFS and auto topology will be enabled on 172.22.36.9 without any impact to the currently running configuration.
Tip
By having multiple IVR-2 capable switches configured, it is possible for a second switch to take over the routing functionality of the other provided it can directly see both the source and destination VSANs. For this recipe, if IVR-2 was disabled on 172.22.36.11, 172.22.36.9 could automatically take over the routing. This is not possible with IVR-1. Enable IVR in FM, by in the Logical Domains pane, select Fabric, All VSANs then IVR. In the control tab, under the Command column, change the 172.22.36.9 switch to enable and press the green Apply Changes button. The status column should change from a yellow progess box, to the word success. Select the CFS tab. In the Enable Admin column, change the field for the 172.22.36.9 switch to enable, and press the green Apply Changes button. The Last Result column should change from inProgress to Success. Click the Active Topology column, and a resulting topology will be displayed that will include both switches, even though only the 172.22.36.11 switch is performing any IVR. Save the configuration of both switches
Step 1 Step 2
6-135
Inter-VSAN Routing
Note
By having CFS already enabled on the first switch, 172.22.36.11, by just enabling IVR and CFS on the second switch, 172.22.36.9, the second switch will learn what configuration parameters namely FC-NAT and auto topology, should be enabled. Therefore, there was no need to enable FC-NAT and auto topology on the second switch.
Tip
This is one of the primary advantages of CFS as the configuration of the first switch could have been very complex and accurately configuring additional switches may be considerably tougher as the number of switches, devices and VSANs grow within the topology. Adding a single switch to an N IVR enabled configuration will require changes to be made to all N switches, therefore leaving room for human error if done manually.
6-136
78-xxxxx-xx
C H A P T E R
FCIP
FCIP (Fibre Channel over IP) is a IETF standards based protocol for connecting Fibre Channel SANs over IP networks. FCIP encapsulates the FCP frames in a TCP/IP packet which is then sent across an IP network. FCIP is used to interconnect geographically dispersed SANs using the IP backbone network. In short FCIP creates a FC tunnel over an existing IP network. The MDS 9200 and 9500 series switches support FCIP using the IPS-8, IPS-4 and the 14+2 blades. The chapter below documents recipes to create and manage FCIP links between MDS switches.
If the fcip enable command is not run, further FCIP configuration is not possible. This command will enable further FCIP configuration options in the CLI.
MDS1# conf t Enter configuration commands, one per line. MDS1(config)# FCIP enable MDS1(config)#^Z MDS1#
7-137
FCIP
The topology consists of two 9509 with one IPS-8 blade in each of the switches. The MDS switches are both connected to Cisco Catalyst 6509 switches the gigE port 8/1 on each of the MDS switches. The gigE port 8/1 on the switch sjc7-9509-5 has an ip address 172.22.34.83 with a subnet mask of 255.255.254.0 and the gateway address 172.22.34.1. The gigE 8/1 port on the switch sjc7-9509-6 has an ip address 172.22.36.58 with a subnet 255.255.254.0 and the gateway address 172.22.34.1. In the recipe the a FCIP tunnel will be established between the switch sjc7-9509-5 (gigE 8/1) and sjc7-9509-6 (gigE 8/1).
Warning
The ip address for the ports on the IPS blade should be in a different subnet than the management interface. This is critical to get FCIP working on the MDS.
Configure the gigE interface on the MDS switches The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnet mask. This allows the gigE interface to communicate with the network.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface gigabitethernet 8/1 sjc7-9509-5(config-if)# ip address 172.22.34.83 255.255.254.0 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# end sjc7-9509-5#
The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ip address 172.22.36.98 255.255.254.0 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end sjc7-9509-5#
Step 2
Configure ip route so that the 2 gigE interfaces can communicate with each other IP route needs to be configured to allow the two gigE ports on switches sjc7-9509-5 and sjc7-9509-6 to communicate with each other. In the above recipe the gigE ports in two different subnet. This does require explicit routes to be able to communicate with each other.
Note
The recommended best practice is just to have a host route to each of the two gigE interfaces. Therefore, only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. For the gigE port 8/1 on the switch sjc7-9509-5 to communicate with the port gigE 8/1 on switch sjc7-9509-6 the following is the route configuration that needs to be done on switch sjc7-9509-5.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# ip route 172.22.36.98 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# end sjc7-9509-5#
7-138
78-xxxxx-xx
Chapter 7
The above configuration translates as, in order to reach 172.22.36.98 use the gateway 172.22.34.1 and interface gigE 8/1 on switch sjc7-9509-5. Similarly for the gigE port 8/1 on switch sjc7-9509-6 to communicate with gigE port 8/1 on switch sjc7-9509-5 the following route configuration needs to be done on switch sjc7-9509-6
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# ip route 172.22.34.83 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config-if)# end sjc7-9509-6#
The above configuration translates as, in order to reach 172.22.34.83 use the gateway 172.22.36.1 and interface gigE 8/1 on switch sjc7-9509-6.
Step 3
Ping the gigE interfaces to ensure that the gigE ports can communicating with each other From the switch sjc7-9509-5 ping the ip address of the gigE interface 8/1 on switch sjc7-9509-6. Similarly ping the ip address of the gigE interface 8/1 on switch sjc7-9509-5 from switch sjc7-9509-6. This can be done from the switch prompt.
sjc7-9509-5# ping 172.22.36.98 PING 172.22.36.98 (172.22.36.98) 56(84) bytes of data. 64 bytes from 172.22.36.98: icmp_seq=1 ttl=254 time=0.504 64 bytes from 172.22.36.98: icmp_seq=2 ttl=254 time=0.404 64 bytes from 172.22.36.98: icmp_seq=3 ttl=254 time=0.414 64 bytes from 172.22.36.98: icmp_seq=4 ttl=254 time=0.416 64 bytes from 172.22.36.98: icmp_seq=5 ttl=254 time=0.449 64 bytes from 172.22.36.98: icmp_seq=6 ttl=254 time=0.399 64 bytes from 172.22.36.98: icmp_seq=7 ttl=254 time=0.411
ms ms ms ms ms ms ms
--- 172.22.36.98 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5999ms rtt min/avg/max/mdev = 0.399/0.428/0.504/0.036 ms sjc7-9509-5# sjc7-9509-6# ping 172.22.34.83 PING 172.22.34.83 (172.22.34.83) 56(84) bytes of data. 64 bytes from 172.22.34.83: icmp_seq=1 ttl=254 time=0.492 64 bytes from 172.22.34.83: icmp_seq=2 ttl=254 time=0.401 64 bytes from 172.22.34.83: icmp_seq=3 ttl=254 time=0.438 64 bytes from 172.22.34.83: icmp_seq=4 ttl=254 time=0.440 64 bytes from 172.22.34.83: icmp_seq=5 ttl=254 time=0.408 64 bytes from 172.22.34.83: icmp_seq=6 ttl=254 time=0.400 64 bytes from 172.22.34.83: icmp_seq=7 ttl=254 time=0.389
ms ms ms ms ms ms ms
--- 172.22.34.83 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5995ms rtt min/avg/max/mdev = 0.386/0.412/0.492/0.037 ms sjc7-9509-6#
Note
It is critical to check the connectivity between the host NIC card and the gigE port on the MDS IPS blade before proceeding further. A successful ping test should be sufficient to check the connectivity. Measure the round trip time between the two gigE interfaces. It is important to measure the round trip time between the interfaces. This value is required for later during the configuration.
sjc7-9509-5# ips measure-rtt 172.22.36.98 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-5#
Step 4
7-139
FCIP
sjc7-9509-6# ips measure-rtt 172.22.34.83 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-6#
Note
FCIP uses port 3225. If there is a firewall between the two switches connected through FCIP then port 3225 need to be opened for the FCIP tunnel to come up between the two switches. The VSAN (defaults to VSAN 1) to which the FCIP interfaces belongs to cannot be suspended. This will prevent the FCIP tunnel from coming up. Configure the FCIP profile on both the switches. FCIP profile needs to be created. The profile defines the characteristics for the FCIP tunnel that is setup. The round trip time measured is needed during the profile configuration. In this case the 407 micro seconds. The ip address used in the profile config below is the ip address assigned to the gigE interface on the switch that will be associated with the profile.
Step 5
Tip
It is recommended that the FCIP profile and FCIP interface numbers number be kept same on both the switches for the given FCIP tunnel for easier management.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# fcip profile 100 sjc7-9509-5(config-profile)# ip address 172.22.34.83 sjc7-9509-5(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-5(config-profile)# end sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# fcip profile 100 sjc7-9509-6(config-profile)# ip address 172.22.36.98 ssjc7-9509-6(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-6(config-profile)# end sjc7-9509-6#
At this stage of the configuration the FCIP profile 100 has been defined on both the switches sjc7-9509-5 and sjc7-9509-6.
Step 6
Configure the fcip interface on both the switches. The fcip interfaces need to be configured on both the switches. The configuration is as shown below.In the fcip interface configuration the profile to used and the peer information (remote gigEs ip address are specified.)are specified. Additionally compression and write acceleration can also be configured.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config-if)# peer-info ipaddr 172.22.36.98 sjc7-9509-5(config-if)# use-profile 100 sjc7-9509-5(config-if)# no shutdown sjc7-9509-5(config-if)# end sjc7-9509-5# sjc7-9509-6# conf t
7-140
78-xxxxx-xx
Chapter 7
Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# int fcip 100 sjc7-9509-6(config-if)# use-profile 100 sjc7-9509-6(config-if)# peer-info ipaddr 172.22.34.84 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end sjc7-9509-6#
The FCIP tunnel should be up and running now. a show interface fcip 100 brief should show the status of the FCIP link between the two switches.
sjc7-9509-5# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -sjc7-9509-5# sjc7-9509-6# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -sjc7-9509-6#
7-141
FCIP
Figure 7-2
The first FCIP link is configured between gigE port 8/1 on switch sjc7-9509-6 and the gigE port 8/1 on switch sjc7-9509-5. This FCIP connection used the default FCIP TCP port 3225. The second FCIP link between gigE port 8/1 on switch sjc7-9509-5 and gigE port 9/1 on switch sjc7-9509-3 is configured to use an alternate TCP port 3500.
Note
The down side to this configuration is that the bandwidth of the gigE port 8/1 on the switch sjc7-9509-6 has to be shared between the 2 tunnels that will be set up to switch sjc7-9509-3 and sjc7-9509-5. The configuration summary on the three MDS switches are as follows.
Configure FCIP tunnel fcip100 between sjc7-9509-5 and sjc7-9509-6 using the default FCIP port. Configure FCIP tunnel fcip100 between sjc7-9509-5 and sjc7-9509-3 using TCP port 3500.
Configure the gigE interface on the MDS switches sjc7-9509-6 and sjc7-9509-5 The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface gigabitethernet 8/1 sjc7-9509-5(config-if)# ip address 172.22.34.83 255.255.254.0 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# ^Z sjc7-9509-5#
7-142
78-xxxxx-xx
Chapter 7
The gigE interface on the MDS switch sjc7-9509-5 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ip address 172.22.36.98 255.255.254.0 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# ^Z sjc7-9509-5#
Step 2
Configure ip route so that the two gigE interfaces can communicate with each other IP route needs to be configured to allow the two gigE ports on switches sjc7-9509-5 and sjc7-9509-6 to communicate with each other. In the above recipe the gigE ports in two different subnets. This does require explicit routes to be able to communicate with each other.
Note
The recommended best practice is just to have a host route to each of the two gigE interfaces. I.e. only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. Syntax for configuring a route is shown below. For the gigE port 8/1 on the switch sjc7-9509-5 to communicate with the port gigE 8/1 on switch sjc7-9509-6 the following is the route configuration needs to be done on switch sjc7-9509-5.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# ip route 172.22.36.98 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# ^Z sjc7-9509-5#
The above configuration translates as, in order to reach 172.22.36.98 use the gateway 172.22.34.1 and interface gigE 8/1 on switch sjc7-9509-5. Similarly for the gigE port 8/1 on switch sjc7-9509-6 to communicate with gigE port 8/1 on switch sjc7-9509-5 the following route configuration needs to be done on switch sjc7-9509-6.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# ip route 172.22.34.83 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ^Z sjc7-9509-6#
The above configuration translates as, in order to reach 172.22.34.83 use the gateway 172.22.36.1 and interface gigE 8/1 on switch sjc7-9509-6.
Step 3
Ping the gigE interfaces to ensure that the gigE ports can communicating with each other From the switch sjc7-9509-5 ping the ip address of the gigE interface 8/1 on switch sjc7-9509-6. Similarly ping the ip address of the gigE interface 8/1 on switch sjc7-9509-5 from switch sjc7-9509-6. This can be done from the switch prompt.
sjc7-9509-5# ping 172.22.36.98 PING 172.22.36.98 (172.22.36.98) 56(84) bytes of data. 64 bytes from 172.22.36.98: icmp_seq=1 ttl=254 time=0.504 64 bytes from 172.22.36.98: icmp_seq=2 ttl=254 time=0.404 64 bytes from 172.22.36.98: icmp_seq=3 ttl=254 time=0.414 64 bytes from 172.22.36.98: icmp_seq=4 ttl=254 time=0.416
ms ms ms ms
7-143
FCIP
64 bytes from 172.22.36.98: icmp_seq=5 ttl=254 time=0.449 ms 64 bytes from 172.22.36.98: icmp_seq=6 ttl=254 time=0.399 ms 64 bytes from 172.22.36.98: icmp_seq=7 ttl=254 time=0.411 ms --- 172.22.36.98 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5999ms rtt min/avg/max/mdev = 0.399/0.428/0.504/0.036 ms sjc7-9509-5# sjc7-9509-6# ping 172.22.34.83 PING 172.22.34.83 (172.22.34.83) 56(84) bytes of data. 64 bytes from 172.22.34.83: icmp_seq=1 ttl=254 time=0.492 64 bytes from 172.22.34.83: icmp_seq=2 ttl=254 time=0.401 64 bytes from 172.22.34.83: icmp_seq=3 ttl=254 time=0.438 64 bytes from 172.22.34.83: icmp_seq=4 ttl=254 time=0.440 64 bytes from 172.22.34.83: icmp_seq=5 ttl=254 time=0.408 64 bytes from 172.22.34.83: icmp_seq=6 ttl=254 time=0.400 64 bytes from 172.22.34.83: icmp_seq=7 ttl=254 time=0.389
ms ms ms ms ms ms ms
--- 172.22.34.83 ping statistics --7 packets transmitted, 7 received, 0% packet loss, time 5995ms rtt min/avg/max/mdev = 0.386/0.412/0.492/0.037 ms sjc7-9509-6#
Step 4
Measure the round trip time between the two gigE interfaces. It is important to measure the round trip time between the interfaces. This value is required for later during the configuration.
sjc7-9509-5# ips measure-rtt 172.22.36.98 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-5# sjc7-9509-6# ips measure-rtt 172.22.34.83 interface gigabitethernet 8/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-6#
Note
FCIP uses port 3225. If there is a firewall between the two switches connected through FCIP then port 3225 need to be opened for the FCIP tunnel to come up between the two switches. The VSAN (defaults to VSAN 1) to which the FCIP interfaces belongs to cannot be suspended. This will prevent the FCIP tunnel from coming up. Configure the FCIP profile on both the switches. FCIP profile needs to be created. The profile defines the characteristics for the FCIP tunnel that is setup. The round trip time measured is needed during the profile configuration. In this case the 407 micro seconds. The ip address used in the profile config below is the ip address assigned to the gigE interface on the switch that will be associated with the profile.
Step 5
Tip
It is recommended that the FCIP profile and FCIP interface numbers number be kept same on both the switches for the given FCIP tunnel for easier management.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# fcip profile 100 sjc7-9509-5(config-profile)# ip address 172.22.34.83 sjc7-9509-5(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-5(config-profile)# ^Z
7-144
78-xxxxx-xx
Chapter 7
sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# fcip profile 100 sjc7-9509-6(config-profile)# ip address 172.22.36.98 ssjc7-9509-6(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-6(config-profile)# ^Z sjc7-9509-6#
At this stage of the configuration the FCIP profile 100 has been defined on both the switches sjc7-9509-5 and sjc7-9509-6.
Step 6
Configure the fcip interface on both the switches. The fcip interfaces need to be configured on both the switches. The configuration is as shown below.In the fcip interface configuration the profile to used and the peer information (remote gigEs ip address are specified.)are specified. Additionally compression and write acceleration can also be configured.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config-if)# peer-info ipaddr 172.22.36.98 sjc7-9509-5(config-if)# use-profile 100 sjc7-9509-5(config-if)# no shutdown sjc7-9509-5(config-if)#^Z sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# int fcip 100 sjc7-9509-6(config-if)# use-profile 100 sjc7-9509-6(config-if)# peer-info ipaddr 172.22.34.84 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end sjc7-9509-6#
Step 7
Check to see if the first FCIP tunnel is up and running between switches sjc7-9509-6 and sjc7-9509-5. The FCIP tunnel should be up and running now. a show interface fcip 100 brief should show the status of the FCIP link between the two switches.
sjc7-9509-5# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -sjc7-9509-5# sjc7-9509-6# sh int fcip 100 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 --
7-145
FCIP
sjc7-9509-6#
Configure the gigE interface on the MDS switches sjc7-9509-3 The gigE interface on the MDS switch sjc7-9509-3 is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# interface gigabitethernet 9/1 sjc7-9509-3(config-if)# ip address 172.22.36.109 255.255.254.0 sjc7-9509-3(config-if)# no shut sjc7-9509-3(config-if)# end sjc7-9509-3#
Step 2
Configure ip route so that the two gigE interlaces can communicate with each other IP route needs to be configured to allow the two gigE ports on switches sjc7-9509-3 and sjc7-9509-6 to communicate with each other. In the above recipe the gigE ports in two different subnets. This does require explicit routes to be able to communicate with each other.
Tip
The recommended best practice is just to have a host route to each of the two gigE interfaces. Therefore only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. Syntax for configuring a route is shown below. For the gigE port 8/1 on the switch sjc7-9509-5 to communicate with the port gigE 9/1 on switch sjc7-9509-3 the following is the route configuration that needs to be done on switch sjc7-9509-5.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# ip route 172.22.36.109 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# ^Z sjc7-9509-5#
The above configuration translates as, in order to reach 172.22.36.109 use the gateway 172.22.36.1 and interface gigE 8/1 on switch sjc7-9509-5. Similarly for the gigE port 9/1 on switch sjc7-9509-3 to communicate with gigE port 8/1 on switch sjc7-9509-5 the following route configuration needs to be done on switch sjc7-9509-3
sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# ip route 172.22.34.83 255.255.255.255 172.22.36.1 interface gigabitethernet 9/1 sjc7-9509-3(config)# end sjc7-9509-3#
The above configuration translates as, in order to reach 172.22.34.83 use the gateway 172.22.36.1 and interface gigE 9/1 on switch sjc7-9509-3.
Step 3
Ping the gigE interfaces to ensure that the gigE ports can communicating with each other
7-146
78-xxxxx-xx
Chapter 7
From the switch sjc7-9509-5 ping the ip address of the gigE interface 9/1 on switch sjc7-9509-3. Similarly ping the ip address of the gigE interface 8/1 on switch sjc7-9509-5 from switch sjc7-9509-3. This can be done from the switch prompt.
sjc7-9509-5# ping 172.22.36.109 PING 172.22.36.109 (172.22.36.109) 56(84) bytes 64 bytes from 172.22.36.109: icmp_seq=1 ttl=254 64 bytes from 172.22.36.109: icmp_seq=2 ttl=254 64 bytes from 172.22.36.109: icmp_seq=3 ttl=254 64 bytes from 172.22.36.109: icmp_seq=4 ttl=254 64 bytes from 172.22.36.109: icmp_seq=5 ttl=254 64 bytes from 172.22.36.109: icmp_seq=6 ttl=254 of data. time=0.351 time=0.399 time=0.405 time=0.408 time=0.398 time=0.410
ms ms ms ms ms ms
--- 172.22.36.109 ping statistics --6 packets transmitted, 6 received, 0% packet loss, time 4998ms rtt min/avg/max/mdev = 0.398/5.525/26.018/10.246 ms sjc7-9509-5# sjc7-9509-3# ping 172.22.34.83 PING 172.22.34.83 (172.22.34.83) 56(84) bytes of data. 64 bytes from 172.22.34.83: icmp_seq=1 ttl=254 time=0.542 64 bytes from 172.22.34.83: icmp_seq=2 ttl=254 time=0.408 64 bytes from 172.22.34.83: icmp_seq=3 ttl=254 time=0.424 64 bytes from 172.22.34.83: icmp_seq=4 ttl=254 time=0.430 64 bytes from 172.22.34.83: icmp_seq=5 ttl=254 time=0.404 64 bytes from 172.22.34.83: icmp_seq=6 ttl=254 time=0.448
ms ms ms ms ms ms
--- 172.22.34.83 ping statistics --6 packets transmitted, 6 received, 0% packet loss, time 4995ms rtt min/avg/max/mdev = 0.404/0.442/0.542/0.052 ms sjc7-9509-3#
Step 4
Measure the round trip time between the two gigE interfaces. It is important to measure the round trip time between the interfaces. This value is required for later during the configuration.
sjc7-9509-5# ips measure-rtt 172.22.36.109 interface gigabitethernet 8/1 Round trip time is 408 micro seconds (0.41 milli seconds) sjc7-9509-5# sjc7-9509-3# ips measure-rtt 172.22.34.83 interface gigabitethernet 9/1 Round trip time is 407 micro seconds (0.41 milli seconds) sjc7-9509-3#
Tip
FCIP uses port 3225 by default. Since the FCIP tunnel to sjc7-9509-6 already is up and is using TCP port 3225 this tunnel will use port 3500. Configure the FCIP profile on both the switches. FCIP profile needs to be created. The profile defines the characteristics for the FCIP tunnel that is setup. The round trip time measured is needed during the profile configuration. In this case the 407 micro seconds. The ip address used in the profile config below is the ip address assigned to the gigE interface on the switch that will be associated with the profile.
Step 5
Tip
It is recommended that the FCIP profile and FCIP interface numbers number be kept same on both the switches for the given FCIP tunnel for easier management.
sjc7-9509-5# conf t
7-147
FCIP
Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# fcip profile 200 sjc7-9509-5(config-profile)# port 3500 sjc7-9509-5(config-profile)# ip address 172.22.34.83 sjc7-9509-5(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-5(config-profile)# end sjc7-9509-5# sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# fcip profile 200 sjc7-9509-3(config-profile)# port 3500 sjc7-9509-3(config-profile)# ip address 172.22.36.109 sjc7-9509-3(config-profile)# tcp max-bandwidth-mbps 1000 min-available-bandwidth-mbps 500 round-trip-time-us 407 sjc7-9509-3(config-profile)# end sjc7-9509-3#
At this stage of the configuration the FCIP profile 200 has been defined on both the switches sjc7-9509-5 and sjc7-9509-3 on TCP port 3500.
Step 6
Configure the fcip interface on both the switches. The fcip interfaces need to be configured on both the switches. The configuration is as shown below.In the fcip interface configuration the profile to used and the peer information (remote gigEs ip address are specified.)are specified. Additionally compression and write acceleration can also be configured.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface fcip 200 sjc7-9509-5(config-if)# peer-info ipaddr 172.22.36.109 port 3500 sjc7-9509-5(config-if)# use-profile 200 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# end sjc7-9509-5# sjc7-9509-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-3(config)# interface fcip 200 sjc7-9509-3(config-if)# peer-info ipaddr 172.22.34.83 port 3500 sjc7-9509-3(config-if)# use-profile 200 sjc7-9509-3(config-if)# no shut sjc7-9509-3(config-if)# end sjc7-9509-3#
Step 7
Check to see if the first FCIP tunnel is up and running between switches sjc7-9509-6 and sjc7-9509-5. The FCIP tunnel should be up and running now. The command show interface fcip 100 brief should show the status of the FCIP link between the two switches.
sjc7-9509-5# show interface fcip 100-200 brief ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip100 1 auto on trunking TE 100 GigabitEthernet8/1 -fcip200 1 auto on trunking TE 200 GigabitEthernet8/1 -sjc7-9509-5 sjc7-9509-3# show interface fcip 200 brief
7-148
78-xxxxx-xx
Chapter 7
------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip200 1 auto on trunking TE 200 GigabitEthernet9/1 -sjc7-9509-3#
Warning
If the fcip enable command is not run, further FCIP configuration is not possible. This command will enable further FCIP configuration options in the CLI.
Step 1
Enable FCIP on the involved switches. In this example sjc7-9216-3 and sjc7-9216-6 are the switches involved. Using Fabric Manager select FCIP in the Physical Attributes on the Lower Left hand Pane. This selection is encircled using a blue box as shown in Figure 7-4. Once this is selected in the upper right hand side pane is populated with the switches seen by FM and the state of FCIP in each of the switches in the Fabric. Then working in the control TAB select the switch sjc7-9216-3 on which the FCIP feature needs to be enabled. Using the drop down box in the command column enable FCIP. This is step is marked using a red box in Figure 7-4. Repeat the above process for the switch sjc7-9216-6 and FCIP on it.
7-149
FCIP
Figure 7-4
Once FCIP is enabled on the two switches it needs to be applied to configuration on the switches. Selecting the green button (highlighted in a blue) applies the changes to the switch.
Figure 7-5 FM Apply Changes to switches.
This is shown in Figure 7-5. Once the command is successfully completed the output is as shown in the red box shown in Figure 7-6.
7-150
78-xxxxx-xx
Chapter 7
Figure 7-6
Enable results
Note
IPSec is available only when 14 + 2 module is used in MDS 9216, MDS 9506, MDS 9509 or a MDS 9216i is used. Configure IP address for the gigE interfaces on switches sjc7-9216-3 and sjc7-9216-6. This is done from Physical Attributes --> Switches --> Interfaces --> Gigabit Ethernet. This is shown in Figure 7-7 and is high lighted using a red. Select gigE port 2/1 on sjc7-9216-3 and gigE 2/1 on sjc7-9216-6 and in the ipaddress/Mask column to configure the ip address and the subnet mask of the gigE ports. The gigE 2/1 on sjc7-9216-3 has an ipaddress of 172.22.34.82 and a mask 255.255.254.0 (/23) and gigE 2/1 on sjc7-9216-6 is assigned 172.22.36.108 and a mask of 255/255/254/0 (/23). Also from the pull down menu in the Admin column set the port state to up. This is highlighted in blue in Figure 7-7.
Step 2
Warning
The ip address for the ports on the ips blade should be in a different subnet than the management interface. This is critical to get FCIP working on the MDS.
Then apply the changes to the switches using the green button above the General tab. This then configures the ip address and the maks for the gigE interfaces on the switches.
7-151
FCIP
Figure 7-7
Step 3
Configure ip routes between the two gigE interfaces so that hey can communicate with each other.
Note
The recommended best practice is just to have a host route to each of the two gigE interfaces. I.e. only allow the two gigE interfaces to communicate with each other, This is done by having a subnet mask of 255.255.255.255 when the route is added. On Switch sjc7-9216-3
sjc7-9216-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-3(config)# ip route 172.22.36.108 255.255.255.255 172.22.34.1 interface gigabitethernet 2/1 sjc7-9216-3(config)#end sjc7-9216-3#
The above configuration translates as, in order to reach 172.22.36.108 use the gateway 172.22.34.1 and interface gigE 2/1 on switch sjc7-9216-3. On switch sjc7-9216-6
sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config)#ip route 172.22.34.82 255.255.255.255 172.22.36.1 interface gigabitethernet 2/1 sjc7-9216-6(config)# end sjc7-9216-6#
The above configuration translates as, in order to reach 172.22.34.82 use the gateway 172.22.36.1 and interface gigE 2/1 on switch sjc7-9216-6.
Step 4
Ping the gigE interfaces to ensure that the gigE ports can communicating with each other
7-152
78-xxxxx-xx
Chapter 7
From the switch sjc7-9216-3 ping the ip address of the gigE interface 2/1 on switch sjc7-9216-6. Similarly ping the ip address of the gigE interface 2/1 on switch sjc7-9509-5 from switch sjc7-9509-6. This can be done from the switch prompt.
sjc7-9216-3# ping 172.22.36.108 PING 172.22.36.108 (172.22.36.108) 56(84) bytes of data. 64 bytes from 172.22.36.108: icmp_seq=1 ttl=63 time=0.495 64 bytes from 172.22.36.108: icmp_seq=2 ttl=63 time=0.524 64 bytes from 172.22.36.108: icmp_seq=3 ttl=63 time=0.488 64 bytes from 172.22.36.108: icmp_seq=4 ttl=63 time=0.496 64 bytes from 172.22.36.108: icmp_seq=5 ttl=63 time=0.467
ms ms ms ms ms
--- 172.22.36.108 ping statistics --5 packets transmitted, 5 received, 0% packet loss, time 4012ms rtt min/avg/max/mdev = 0.467/0.493/0.524/0.034 ms sjc7-9216-3# sjc7-9216-6# ping 172.22.34.82 PING 172.22.34.82 (172.22.34.82) 56(84) bytes of data. 64 bytes from 172.22.34.82: icmp_seq=1 ttl=254 time=0.495 64 bytes from 172.22.34.82: icmp_seq=2 ttl=254 time=0.461 64 bytes from 172.22.34.82: icmp_seq=3 ttl=254 time=0.450 64 bytes from 172.22.34.82: icmp_seq=4 ttl=254 time=0.468
ms ms ms ms
--- 172.22.34.82 ping statistics --4 packets transmitted, 4 received, 0% packet loss, time 2997ms rtt min/avg/max/mdev = 0.450/0.468/0.495/0.027 ms sjc7-9216-6#
Note
It is critical to check the connectivity between the host NIC card and the gigE port on the MDS IPS blade before proceeding further. A successful ping test should be sufficient to check the connectivity. Create the FCIP tunnel between the two GigE interfaces.
Figure 7-8 FCIP Tunnel Icon in FM
Step 5
In FM select the FCIP tunnel highlighted in blue in Figure 7-8. This brings up a FCIP config dialogue. The FCIP Tunnel Wizard start screen is shown below in Figure 7-9.
7-153
FCIP
Figure 7-9
In the start screen select the two switches that are going to be interconnected using FCIP. In the topology used for this recipe the switches are sjc7-9216-3 and sjc7-9216-6, Select the next button after the switches are selected.
Figure 7-10 Configured and available gigE ports.
The next screen displays the available gigE ports on the two switches as well as the ip address of the gigE port if configured. In the topology gigE 2/1 on both switches are used. This is shown in Figure 7-10. To protect the FCIP tunnel using IPSec select Enforce IPSEC Security check box and then supply IKE Auth Key (a pass phrase or key). These are high lighted in blue boxes in Figure 7-10. Click on the next button to get to the next step.
Tip
For the IKE Auth Key use at least a 8 character pass phrase or key.
7-154
78-xxxxx-xx
Chapter 7
In the specify Tunnel properties screen the Max bandwidth and the min bandwidth of the link needs to be set. this is shown in Figure 7-11. The Max Bandwidth is the set to a value that the FCIP tunnel is allowed to use. If the tunnel is dedicated for FCIP then the max bandwidth is set to the MAX value of the bandwidth available. Min Value is adjusted based on whether the link is shared or is a dedicate for FCIP. The estimated round trip time can be measures using the measure button high lighted using a brown box in Figure 7-11. Selecting the measure button brings up a screen which measures the RTT between the two gigE interfaces. This is shown in Figure 7-12.
Figure 7-12 Measure RTT output screen.
Based on the RTT output the Value is changed on the Estimated RTT. In this topology the RTT was measured to be 423 micro seconds. This is shown using a red circle in Figure 7-11. Also the write acceleration and compression check box is selected to turn these features on. The next button will bring up the Create FCIP ISL screen.
7-155
FCIP
In this screen verify the switches and the interfaces that are being interconnected. Also turn on trunk mode so that VSANs can flow through the link. This selection is shown using a red ellipse in. Figure 7-13. In this screen the VSANS allowed through the FCIP link can be changes in the VSAN list dialogue. Selecting finish will bring up another screen prompting to enable ipsec on the two switches. This is shown in Figure 7-14. Select Yes and the tunnel creation will complete. The FCIP link is configured and up. This can be seen either through FM or through command line.
sjc7-9216-3# sh int fcip2 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip2 1 auto on trunking TE 1 GigabitEthernet2/1 -sjc7-9216-3# sjc7-9216-6# sh int fcip 2 br ------------------------------------------------------------------------------Interface Vsan Admin Admin Status Oper Profile Eth Int Port-channel Mode Trunk Mode Mode ------------------------------------------------------------------------------fcip2 1 auto on trunking TE 1 GigabitEthernet2/1 -sjc7-9216-6#
7-156
78-xxxxx-xx
Chapter 7
In FM Physical Attributes --> ISLs -->FCIP and on the right hand side select Tunnels (Advanced) You should see the IP compression mode(brown box), Write Acceleration(blue box) has been turned on on both switches and IPSec set to true (red box). This is shown in Figure 7-15. The link should also show up in FM map topology.
7-157
FCIP
Tuning FCIP
Configuring and bringing up a FCIP tunnel establishes a ISL between the two interconnected switches. To achieve greater efficiency and utilization from the link, some link parameters may need to be optimized. This optimization is specific to each FCIP link between the two connected switches. The optimization for a FCIP link over a slow 1.54 Mb/s connection will be done differently than a FCIP link over a 1Gb/s connection with very low latency. This section will look to provide insight into what parameters can be optimized to increase efficiency and utilization of the FCIP link.
Note
Individual results may vary due to network conditions (existing link utilization / quality) as well as storage array and host type using the FCIP tunnel.
The maximum available bandwidth value is the maximum amount of bandwidth that the FCIP link should use on the network. The minimum available bandwidth value is used as a guideline for the minimum value. Of course, if there are serious problems on the network (dropped packets, congestion), the link will go slower than the minimum value. It is recommended that the minimum value be set at least to what the minimum acceptable bandwidth that the application (EMC SRDF, IBM PPRC etc.) requires.
Tip
If the underlying link is dedicated for FCIP, the minimum and maximum available bandwidth values should be set the same. Table 7-1 contains some common WAN links and their speeds. These circuits are most often used as the underlying network for a FCIP link. For example, the underlying network may be a OC3 but you may only be able to use 100Mb of that link.
Tip
When deploying FCIP, you should always involve the LAN and WAN teams to find out about the connectivity that they are providing you. If there are performance issues, they can often help you troubleshoot them from the networks standpoint. Involve them earlier rather than later.
Table 7-1 Common WAN circuit speeds
Circuit Name T1 T3
7-158
78-xxxxx-xx
Chapter 7
Link Speed 155 Megabits/sec 622 Megabits/sec 1.244 Gigabits/sec 2.488 Gigabits/sec 10 Gigabits/sec 40 Gigabits/sec
Note
Only in MDS version 2.0.1b and higher should write acceleration be used with Port-Channels. Enable Write Acceleration.
sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# interface fcip 100 sjc7-9509-5(config)# write-accelerator sjc7-9509-5(config)# end sjc7-9509-5# sjc7-9509-6# conf t Enter configuration commands, one per line. sjc7-9509-6(config)# interface fcip 100 sjc7-9509-6(config)# write-accelerator sjc7-9509-6(config)# end sjc7-9509-6# End with CNTL/Z.
Step 1
FCIP Compression
The IPS-8 and IPS-4 blades support software based compression. The 14+2 blades supports hardware based compression. The IPS-8 and IPS-4 which do software based compression can do compression at the rate of 155 Mb/sec per port. While the 14+2 blades 2 IPS ports does hardware compression can do compression at line rate(1 Gb/s) per port. In SAN-OS version 2.0, three new modes of compression were introduced along with hardware based compression using the 14+2 module. These modules are summarized in Figure 7-16. IP compression modes.
1. 2. 3.
auto (default) mode picks the appropriate compression scheme based on the bandwidth of the link (the bandwidth of the link configured in the FCIP profiles TCP parameters) mode1 is a fast compression mode for high bandwidth links (> 25 Mbps) mode2 is a moderate compression mode for moderately low bandwidth links (between 10 and 25 Mbps)
7-159
FCIP
4.
mode3 is a high compression mode for low bandwidth links (< 10 Mbps)
Warning
The compression and write acceleration has to be enabled on both sides and also the compression modes on both sides should also match for the link to come up.
Note
The numbers displayed in Figure 7-16 are derived using the Canterbury Corpus test suite. Your actual performance number may vary depending on device type and data pattern. However, these should be used as a guide to determine which compression mode is most appropriate.
Figure 7-16 Approximate application throughput with MDS 2.0 compression
Step 1
7-160
78-xxxxx-xx
Chapter 7
sjc7-9509-6#
Warning
FCIP Tape acceleration will not work if the fcip port is part of a port channel or even if there are multiple paths with equal costs between backup host and the tape device.
Tip
It is recomended as a best practice to have multiple paths between the backup serve and the tape device but configre the multiple paths with varying costs (i.e. FSPF cost of the fcip link). This will ensure that only one link is used at a given time for tape acceleration and has other higher cost links to fail over is the current link fails. It is enabled in CLI as shown below on both switches..
sjc7-9216-3# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-3(config)# int fcip2 sjc7-9216-3(config-if)# write-accelerator tape-accelerator sjc7-9216-3(config-if)#shut sjc7-9216-3(config-if)#no shut sjc7-9216-3# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config)# int fcip2 sjc7-9216-6(config-if)# write-accelerator tape-accelerator sjc7-9216-6(config-if)#shut sjc7-9216-6(config-if)#no shut sjc7-9216-6#
The show interface fcip command shows the status of the link and the configuration of the FCIP tunnel.
sjc7-9216-3# sh int fcip 2 fcip2 is trunking Hardware is GigabitEthernet Port WWN is 20:50:00:0d:ec:02:31:c0 Peer port WWN is 20:50:00:0d:ec:01:b0:80 Admin port mode is auto, trunk mode is on Port mode is TE Port vsan is 1 Speed is 1 Gbps Trunk vsans (admin allowed and active) (1,10,304) Trunk vsans (up) (1,10) Trunk vsans (isolated) (304) Trunk vsans (initializing) () Using Profile id 1 (interface GigabitEthernet2/1) Peer Information Peer Internet address is 172.22.36.108 and port is 3225 FCIP tunnel is protected by IPSec Write acceleration mode is on Tape acceleration mode is on
7-161
FCIP
Tape Accelerator flow control buffer size is automatic IP Compression is enabled and set for mode1 Special Frame is disabled Maximum number of TCP connections is 2 Time Stamp is disabled QOS control code point is 0 QOS data code point is 0 B-port mode disabled TCP Connection Information 2 Active TCP connections Control connection: Local 172.22.34.82:60584, Remote 172.22.36.108:3225 Data connection: Local 172.22.34.82:60586, Remote 172.22.36.108:3225 2476 Attempts for active connections, 71 close of connections TCP Parameters Path MTU 2200 bytes Current retransmission timeout is 200 ms Round trip time: Smoothed 10 ms, Variance: 5 Advertized window: Current: 148 KB, Maximum: 146 KB, Scale: 6 Peer receive window: Current: 118 KB, Maximum: 118 KB, Scale: 6 Congestion window: Current: 19 KB, Slow start threshold: 211 KB Current Send Buffer Size: 146 KB, Requested Send Buffer Size: 149878 KB CWM Burst Size: 50 KB 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 15561 frames input, 1705428 bytes 15502 Class F frames input, 1699056 bytes 59 Class 2/3 frames input, 6372 bytes 0 Reass frames 0 Error frames timestamp error 0 15895 frames output, 1652940 bytes 15836 Class F frames output, 1646568 bytes 59 Class 2/3 frames output, 6372 bytes 0 Error frames sjc7-9216-6# sh int fcip 2 fcip2 is trunking Hardware is GigabitEthernet Port WWN is 20:50:00:0d:ec:01:b0:80 Peer port WWN is 20:50:00:0d:ec:02:31:c0 Admin port mode is auto, trunk mode is on Port mode is TE Port vsan is 1 Speed is 1 Gbps Trunk vsans (admin allowed and active) (1,5,10,39,200,301) Trunk vsans (up) (1,10) Trunk vsans (isolated) (5,39,200,301) Trunk vsans (initializing) () Using Profile id 1 (interface GigabitEthernet2/1) Peer Information Peer Internet address is 172.22.34.82 and port is 3225 FCIP tunnel is protected by IPSec Write acceleration mode is on Tape acceleration mode is on Tape Accelerator flow control buffer size is automatic IP Compression is enabled and set for mode1 Special Frame is disabled Maximum number of TCP connections is 2 Time Stamp is disabled QOS control code point is 0 QOS data code point is 0 B-port mode disabled TCP Connection Information 2 Active TCP connections Control connection: Local 172.22.36.108:3225, Remote 172.22.34.82:60584
7-162
78-xxxxx-xx
Chapter 7
Data connection: Local 172.22.36.108:3225, Remote 172.22.34.82:60586 2458 Attempts for active connections, 72 close of connections TCP Parameters Path MTU 2200 bytes Current retransmission timeout is 200 ms Round trip time: Smoothed 8 ms, Variance: 6 Advertized window: Current: 118 KB, Maximum: 26 KB, Scale: 6 Peer receive window: Current: 148 KB, Maximum: 148 KB, Scale: 6 Congestion window: Current: 14 KB, Slow start threshold: 211 KB Current Send Buffer Size: 26 KB, Requested Send Buffer Size: 26948 KB CWM Burst Size: 50 KB 5 minutes input rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 5 minutes output rate 0 bits/sec, 0 bytes/sec, 0 frames/sec 15336 frames input, 1606556 bytes 15277 Class F frames input, 1600184 bytes 59 Class 2/3 frames input, 6372 bytes 0 Reass frames 0 Error frames timestamp error 0 15565 frames output, 1705796 bytes 15506 Class F frames output, 1699424 bytes 59 Class 2/3 frames output, 6372 bytes 0 Error frames sjc7-9216-6#
In the above outputs from sjc7-9126-3 and sjc7-9216-6 the link properties have been highlighted. In FM select Physical Attributes --> ISLs -->FCIP and on the right hand side select Tunnels (Advanced). Then select the chek box on both the switches for Tape Acceleration(highlighted in a blue box) and then apply the config to the switches by selecting the green apply changes button(highlighted in a red box) in Figure 7-17.
Figure 7-17 Activatiing Tape Acceleration in FM
The tape flowctrl buffer size can be tuned from auto (0 == auto) using the brown box in Figure 7-17.
7-163
FCIP
Tip
SET requires the SAN_EXTN_OVER_IP license to work. SET requires the following.
Two IPS modules. FCIP link between the switches. One unused gigabit ethernet port per switch to act as a initiator or target. Physical layer of the second gigabit ethernet port should be up. Enabling ISCSI on both the switches. Enabling SAN-EXT-TUNER.
SET works by creating virtual initiators and targets behind two gigabit ethernet ports. These virtual devices are created in a VSAN, they have pwwns and obtain FCIDs just like real end devices do. They are required to be zoned together to communicate, and they send standard FCP commands to each other which are handled by the fibre channel infrastructure as they are no different from normal FC traffic. The frames are routed via FSPF to their destination and can travel on E and TE ports through MDS and non-MDS switches. If the minimum requirements are met, SET can be used to test optical networks and not just FCIP links. While the same gigabit ethernet port that is configured for FCIP can be used as a target or initiator, it is not recommended as this may interfere with the ability to generate sufficient bandwidth. Always use an unused gigabit ethernet port to create the initiator and target on.
Note
This SET recipe assumes that the FCIP link is already up and functional. For assistance with creating a FCIP tunnel see Configuring FCIP on MDS, page 7-137 For this recipe the following resources will be used: Switch: sjc7-9216-5 Additional gigE port: gig2/2 VSAN 1000
7-164
78-xxxxx-xx
Chapter 7
Virtual nwwn: 10:00:00:00:00:00:00:00 Virtual pwwn: 20:00:00:00:00:00:00:01 Switch: sjc7-9216-6 Additional gigE port: gig2/2 VSAN 1000 Virtual nwwn 11:00:00:00:00:00:00:00 Virtual pwwn: 30:00:00:00:00:00:00:01
Step 1
Enable the second gigabit ethernet port on both the switches. In this recipe gigE port 2/2 will be used on both the switches for SET.
Tip
This additional gigE port does not require an IP address to be assigned. Only the physical layer is required to be up.
sjc7-9216-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-5(config-if)#interface gigabitethernet 2/2 sjc7-9216-5(config-if)# no shut sjc7-9216-5(config-if)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config-if)#interface gigabitethernet 2/2 sjc7-9216-6(config-if)# no shut sjc7-9216-6(config-if)# end sjc7-9216-6#
Check to ensure that the physical layer of the gigE port is up and running.
sjc7-9216-5# show interface gigabitethernet 2/2 brief ------------------------------------------------------------------------------Interface Status IP Address Speed MTU Port Channel ------------------------------------------------------------------------------GigabitEthernet2/2 up -1 Gbps 1500 --
sjc7-9216-6# sh int gigabitethernet 2/2 brief ------------------------------------------------------------------------------Interface Status IP Address Speed MTU Port Channel ------------------------------------------------------------------------------GigabitEthernet2/2 up -1 Gbps 1500 --
Step 2
Enable ISCSI on both the switches sjc7-9216-5 and sjc7-9216-6 if not already enabled.
sjc7-9216-5# conf t Enter configuration commands, one per line. sjc7-9216-5(config)# enable iscsi sjc7-9216-5(config)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z.
7-165
FCIP
Step 3
Enable the iscsi on the second gigE interface on both the switches.
sjc7-9216-5# conf t Enter configuration commands, one per line. sjc7-9216-5(config)# interface iscsi 2/2 sjc7-9216-5(config-if)# no shut sjc7-9216-5(config-if)# end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. sjc7-9216-6(config)# interface iscsi 2/2 sjc7-9216-6(config-if)# no shut sjc7-9216-6(config-if)# end sjc7-9216-6# End with CNTL/Z.
Step 4
Tip
Using a seperate VSAN for SET ensures that other devices in other VSANs are not impacted by the SET traffic.
Step 5
7-166
78-xxxxx-xx
Chapter 7
sjc7-9216-5(config-vsan-db)#vsan 1000 name SETVSAN sjc7-9216-5(config-vsan-db)#end sjc7-9216-5# sjc7-9216-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9216-6(config)#vsan database sjc7-9216-6(config-vsan-db)#vsan 1000 name SETVSAN sjc7-9216-6(config-vsan-db)#end sjc7-9216-6#
Step 6
Configure nWWN and nport on both the switches. The nport pwwn on switch sjc7-9216-5 is configured as 20:00:00:00:00:00:00:01with a nWWN of 10:00:00:00:00:00:00:00. Similarly on switch sjc7-9216-6 the nport pwwn is configured as 30:00:00:00:00:00:00:01 and nWWN of 11:00:00:00:00:00:00:00. Both the nports are made a part of VSAN 1000 which is created just for SET.
sjc7-9216-5# san-ext-tuner sjc7-9216-5(san-ext)# nwwN 10:00:00:00:00:00:00:00 sjc7-9216-5(san-ext)# nport pWWN 20:00:00:00:00:00:00:01 vsan 1000 interface gigabitethernet 2/2 sjc7-9216-5(san-ext-nport)#end sjc7-9216-5# sjc7-9216-6# san-ext-tuner sjc7-9216-6(san-ext)# nwwN 11:00:00:00:00:00:00:00 sjc7-9216-6(san-ext)# nport pWWN 30:00:00:00:00:00:00:01 vsan 1000 interface gigabitethernet 2/2 sjc7-9216-6(san-ext-nport)#end sjc7-9216-6#
The command show fcns database vsan 1000 should display both the pwwns in VSAN 1000.
sjc7-9216-5# show fcns database vsan 1000 VSAN 1000: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x0c0003 N 30:00:00:00:00:00:00:01 scsi-fcp 0x410002 N 20:00:00:00:00:00:00:01 scsi-fcp Total number of entries = 2 sjc7-9216-5#
7-167
FCIP
VSAN 1000: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x0c0003 N 30:00:00:00:00:00:00:01 scsi-fcp 0x410002 N 20:00:00:00:00:00:00:01 scsi-fcp Total number of entries = 2 sjc7-9216-6#
Step 7
Create a zoneset and a zone in vsan 1000 so that both SET nports can communicate with each other. It is not required that the CLI be used for zoning these devices.
sjc7-9216-5# conf t Enter configuration commands, one sjc7-9216-5(config)# zoneset name sjc7-9216-5(config-zoneset)# zone sjc7-9216-5(config-zoneset-zone)# sjc7-9216-5(config-zoneset-zone)# sjc7-9216-5(config-zoneset-zone)# sjc7-9216-5(config-zoneset)# exit per line. End with CNTL/Z. ZS_santune_vsan1000 vsan 1000 name Z_SET_VSAN100 member pwwn 20:00:00:00:00:00:00:01 member pwwn 30:00:00:00:00:00:00:01 exit
Step 8
Activate the zoneset. If enhanced zoning is enabled for this VSAN, then the zone changes will need to be committed as well.
sjc7-9216-5(config)# zoneset activate name ZS_santune_vsan1000 vsan 1000 Zoneset activation initiated. check zone status sjc7-9216-5(config)# end sjc7-9216-5#
Verify that the zoneset is active and the two nports are able to communicate with each other.
sjc7-9216-5# show zoneset active vsan 1000 zoneset name santune vsan 1000 zone name Z_SET_VSAN100 vsan 1000 * fcid 0x410002 [pwwn 20:00:00:00:00:00:00:01] * fcid 0x0c0003 [pwwn 30:00:00:00:00:00:00:01] sjc7-9216-5#
Step 9
Create tasks that will either read or write from one nport to another. Two tasks, one to read and one to write will be created as examples. In this example the nport on switch sjc7-9216-5 will act as an initiator and the nport on switch sjc7-9216-6 will act as a traget.
Note
By default, an all-zero pattern is used as the pattern for data generated by the virtual N ports. One can optionally specify a file as the data pattern to be generated by selecting a data pattern file.
sjc7-9216-5# san-ext-tuner sjc7-9216-5(san-ext)# nport pWWN 20:00:00:00:00:00:00:01 vsan 1000 interface gigabitethernet 2/2 sjc7-9216-5(san-ext-nport)# read command-id 1 target 30:00:00:00:00:00:00:01 transfer-size 1024000 outstanding-ios 5 continuous <-- read command sjc7-9216-5(san-ext-nport)# write command-id 2 target 30:00:00:00:00:00:00:01 transfer-size 1024000 outstanding-ios 5 continuous <-- write command sjc7-9216-5(san-ext-nport)#
Note
The transfer size should be a multiple of 512. The test can be continuous or can be limited to a certain number of transactions. The the throughput and performance data can be gathered on the switch as shown below.
7-168
78-xxxxx-xx
Chapter 7
sjc7-9216-5# show san-ext-tuner interface gigabitethernet 2/2 nport pWWN 20:00:00:00:00:00:00:01 vsan 1000 counters Statistics for nport Node name 10:00:00:00:00:00:00:00 Port name 20:00:00:00:00:00:00:01 I/Os per sec : 18 Reads : 50% Writes : 50% Egress throughput : 9.06 MBs/sec (Max - 9.12 MBs/sec) Ingress throughput : 8.62 MBs/sec (Max - 10.45 MBs/sec) Average response time : Read - 572450 us, Write - 568564 us Minimum response time : Read - 343728 us, Write - 331788 us Maximum response time : Read - 1350666 us, Write - 990794 us Errors : 0 sjc7-9216-5 sjc7-9216-6# show san-ext-tuner interface gigabitethernet 2/2 nport pWWN 30:00:00:00:00:00:00:01 vsan 1000 counters Statistics for nport Node name 11:00:00:00:00:00:00:00 Port name 30:00:00:00:00:00:00:01 I/Os per sec : 17 Reads : 58% Writes : 41% Egress throughput : 8.84 MBs/sec (Max - 10.47 MBs/sec) Ingress throughput : 9.02 MBs/sec (Max - 9.42 MBs/sec) Average response time : Read - 447611 us, Write - 424872 us Minimum response time : Read - 36986 us, Write - 124183 us Maximum response time : Read - 1165843 us, Write - 1386055 us Errors : 0 sjc7-9216-6#
Collecting the above data over the specific period of time will help calibrate the link and further tune the link for optimal throughput and performance.
Step 10
7-169
FCIP
7-170
78-xxxxx-xx
C H A P T E R
iSCSI
iSCSI is a transport protocol which is used to transport SCSI packets over TCP/IP. It is an internet protocol based standard that allows hosts to connect and to access storage over a network interface card. iSCSI is used to facilitate data transfers over intranets and to manage storage over long distances. Since iSCSI runs over IP it can be used to transmit data over LAN, WAN, MAN etc. there by enabling data access independent of the location of the storage sub system. The MDS 9200 and 9500 series switch support iSCSI. This is done using the IPS-8, IPS-4 and the 14+2 blades.
If the iscsi enable command is not run further iSCSI configuration is not possible. This command will enable further iSCSI configuration options in the CLI
sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# iscsi enable sjc7-9509-5(config)# end sjc7-9509-5#
8-171
iSCSI
Figure 8-1
iSCSI topology.
The topology consists of a Windows 2003 server using a gigabit ethernet NIC for iSCSI. It is connected to a port on the catalyst 6509. The iSCSI interface on the host is assigned the ip address 172.22.38.45. The IPS port 8/1 on the switch sjc7-9509-5 is also connected to the Catalyst and has an ip address 172.22.36.98. The storage port from the array is connected to the FC port 1/16 on sjc7-9509-5. This example shows how to configure an iSCSI initiator using iqn (iSCSI qualified name).
Warning
The ip address for the ports on the ips blade should be in a different subnet than the management interface. This is critical to get iSCSI working on the MDS.
Configure the gigE interface on the MDS switch. The gigE interface on the MDS switch is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# interface gigabitethernet 8/1 sjc7-9509-5(config-if)# ip address 172.22.34.83 255.255.254.0 sjc7-9509-5(config-if)# end sjc7-9509-5#
Step 2
Configure IP routes as required. In the recipe the initiator sjc7-pc-1 is in the 172.22.38.0 subnet while the gigE interface is in the 172.22.34.0 sub net. In order to allow the initiator and the gigE port to communicate ip route has to be configured on the switch.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)#ip route 172.22.38.45 255.255.255.255 172.22.34.1 interface gigabitethernet 8/1 sjc7-9509-5(config)# end sjc7-9509-5#
Note
It is critical to check the connectivity between the host NIC card and the gigE port on the MDS IPS blade before proceeding further. A successful ping test should be sufficient. Ping the gigE interface from the host. Similarly ping the host from the switch.
sjc7-9509-5#ping 172.22.38.45 PING 172.22.38.45 (172.22.38.45) 56(84) bytes of data. 64 bytes from 172.22.38.45: icmp_seq=1 ttl=127 time=1.18 ms 64 bytes from 172.22.38.45: icmp_seq=2 ttl=127 time=0.483 ms 64 bytes from 172.22.38.45: icmp_seq=3 ttl=127 time=0.479 ms --- 172.22.38.45 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2006ms
Step 3
8-172
78-xxxxx-xx
Chapter 8
Step 4
Enable the iSCSI interface on the switch sjc7-9509-5. The iSCSI interface 8/1 (same port as the gigE interface) needs to be enabled. Along with the enabling of the iSCSI interface additional iSCSI related TCP tuning can also be done. In this recipe the default values are used.
sjc7-9509-5# conf t Enter configuration commands, one per line. sjc7-9509-5(config)# interface iscsi 8/1 sjc7-9509-5(config-if)# no shut sjc7-9509-5(config-if)# end sjc7-9509-5# End with CNTL/Z.
Step 5
Configure the iSCSI initiator on the switch sjc7-9509-5; This can be done in multiple ways. It can be done using the ip address of the initiator or as a proxy initiator or using an iqn (iSCSI Qualified Name), This example will use iqn name. Most iSCSI drivers / clients can automatically assign an iqn name to the on the host. The iqn name has to be at least 16 characters long. The iqn name can also be manually assigned. If it is being manually assigned care has to taken to ensure that the iqn name is unique. The win2k server used in the The Microsoft driver pre configures the iqn name at the time of install. This can be viewed as shown below. In the iSCSI utility select Initiator Settings, this shows the iqn name for the host. The driver assigned initiator node name is iqn.1991-05.com.microsoft:sjc7-pc-1.
Figure 8-2 iSCSI initiator properties
The Figure 8-2 shows the iqn name in the iSCSI driver interface. it is highlighted in a red box. For linux OS this can be found in the /etc/initiatorname.iscsi file. This is required to configure the iSCSI initiator on the MDS.
8-173
iSCSI
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi initiator name iqn.1991-05.com.microsoft:sjc7-pc-1 sjc7-9509-5(config-(iscsi-init))# static pWWN system-assign 1 <-- system assigned sjc7-9509-5(config-(iscsi-init))# vsan 1 <-- Must be a member in the Targets VSAN sjc7-9509-5(config-(iscsi-init))# end sjc7-9509-5#
Tip
If an iSCSI initiator needs to have the same pwwn as was previously used by a host, as in the case if you were converting a host from fibre channel to iSCSI, the pwwn can be manually assigned. This would alleviate the need to modify the lun masking database on the array.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi initiator name iqn.1991-05.com.microsoft.sjc7-pc-1 sjc7-9509-5(config-iscsi-init)# static pWWN 21:05:00:0d:ec:02:2d:82 <-- manually assigned sjc7-9509-5(config-iscsi-init)# vsan 1 <-- Must be a member of the Target VSAN sjc7-9509-5(config-iscsi-init)# end sjc7-9509-5#
In the recipe the iqn assigned by the driver is used. If it needs to be changed make sure that it is unique and is at least 16 characters long. Optionally a pwwn can also be assigned to the initiator. The pwwn can be statically assigned by the administrator as shown above or the system can automatically assign one. The initiator can be part of multiple VSANs. In order to talk to the Target it has to be a member of the targets VSAN. In the above example the target belongs to VSAN 1.
Note
Alternatively the IP address of the iSCSI initiator can be used in all the configuration. Also assigning a static pwwn is also optional. While doing zoning an iscsi interface, its IP address can be used in place of its pwwn or the iqn name. An example of the using ip address is shown below.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi initiator ipaddress 172.22.38.45 sjc7-9509-5(config-(iscsi-init))# static pWWN system-assign 1 <-- system assigned sjc7-9509-5(config-(iscsi-init))# vsan 3003 <-- Must be a member in the Targets VSAN sjc7-9509-5(config-(iscsi-init))# end sjc7-9509-5#
Note
For the iSCSI initiator to communicate with a target port, the iSCSI initiator has to made a member of the target ports VSAN. iSCSI initiators can be members of multiple VSANs. Configure the virtual target on the switch sjc7-9509-5.
sjc7-9509-5# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# iscsi virtual-target name iscsi-sjc7-jbod-1 sjc7-9509-5(config-iscsi-tgt)# pWWN 22:00:00:20:37:5a:40:26 sjc7-9509-5(config-iscsi-tgt)# end sjc7-9509-5#
Step 6
8-174
78-xxxxx-xx
Chapter 8
The virtual target is a name assigned to the storage device. This name has to be 16 characters long. Then the pwwn of the storage port is assigned to the virtual target as shown above. This completes the configuration of a virtual target.
Step 7
The virtual target can be configured to allow all initiators to communicate with the virtual-target. In the above command the virtual-target is only configured to communicate with the initiator iqn.1991-05.com.microsoft.sjc7-pc-1. This permits the initiator iqn.1991-05.com.microsoft:sjc7-pc-1 to communicate with the virtual-target iscsi-sjc7-jbod-1.
Step 8
Next, create a zone with the initiator and the virtual target configured. Add the zone to the zoneset and activate the zoneset. Create a zone with the iSCSI initiator and the virtual targets as members.This enables the initiator and the target to communicate with each other. The zone can be created either with the iqn name or with the ip address or with the pwwn that was assigned to the initiator. In this recipe it is created using the pwwn. Zoning with pwwn of the virtual-target and initiator.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zone name Z_iscsi_tst vsan 3003 sjc7-9509-5(config-zone)# mem pwwn 22:00:00:20:37:39:9c:1f sjc7-9509-5(config-zone)# mem pwwn 21:05:00:0d:ec:02:2d:82 sjc7-9509-5(config-zone)#end sjc7-9509-5# sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zoneset name ZS_ISCSI vsan 3003 sjc7-9509-5(config-zoneset)#member Z_iscsi_tst sjc7-9509-5(config-zoneset)end sjc7-9509-5#
Alternatively: Zoning with the pwwn of the virtual target and the iqn of the iSCSI initiator.
sjc7-9509-5# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-5(config)# zone name Z_iscsi_tst vsan 3003 sjc7-9509-5(config-zone)# member pwwn 22:00:00:20:37:39:9c:1f sjc7-9509-5(config-zone)# member symbolic-nodename iqn.1991-05.com.microsoft:sjc7-pc-1 sjc7-9509-5(config-zone)# end sjc7-9509-5#
8-175
iSCSI
8-176
78-xxxxx-xx
Chapter 8
Step 1
Configure the gigE interface on the MDS switch The gigE interface on the MDS switch is given an ip address, a subnetmask. This allows the gigE interface to communicate with the network.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface gigabitethernet 8/1 sjc7-9509-6(config-if)# ip address 172.22.36.98 255.255.254.0 sjc7-9509-6(config-if)# end sjc7-9509-6#
Step 2
Configure ip routes if required In the recipe the initiator sjc7-pc-1 and sjc7-pc-2 are in the 172.22.38.0 subnet while the gigE interface is in the 172.22.36.0 sub net. In order to allow the initiator and the gigE port to communicate ip route has to be configured on the switch. A host based route will be configured to allow the gigE port to communicate with sjc7-pc-1 and sjc7-pc-2 hosts.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# ip route 172.22.38.45 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config)# ip route 172.22.38.46 255.255.255.255 172.22.36.1 interface gigabitethernet 8/1 sjc7-9509-6(config)# end sjc7-9509-6#
Step 3
Ping the gigE interface from the hosts. Similarly ping the host from the switch.
sjc7-9509-6#ping 172.22.38.45 PING 172.22.38.45 (172.22.38.45) 56(84) bytes of data. 64 bytes from 172.22.38.45: icmp_seq=1 ttl=127 time=1.18 ms 64 bytes from 172.22.38.45: icmp_seq=2 ttl=127 time=0.483 ms 64 bytes from 172.22.38.45: icmp_seq=3 ttl=127 time=0.479 ms --- 172.22.38.45 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2006ms rtt min/avg/max/mdev = 0.479/0.716/1.186/0.332 ms sjc7-9509-6# sjc7-9509-6#ping 172.22.38.46 PING 172.22.38.45 (172.22.38.45) 56(84) bytes of data. 64 bytes from 172.22.38.46: icmp_seq=1 ttl=127 time=1.18 ms 64 bytes from 172.22.38.46: icmp_seq=2 ttl=127 time=0.483 ms 64 bytes from 172.22.38.46: icmp_seq=3 ttl=127 time=0.479 ms --- 172.22.38.46 ping statistics --3 packets transmitted, 3 received, 0% packet loss, time 2006ms rtt min/avg/max/mdev = 0.479/0.716/1.186/0.332 ms sjc7-9509-6#
Step 4
Enable the iSCSI interface on the switch sjc7-9509-6. The iSCSI interface 8/1 (same port as the gigE interface) needs to be enabled. Along with the enabling of the iSCSI interface additional iSCSI related TCP tunning can also be done. In this recipe the default values are used. The switch port command is used to enable proxy-initiator mode for the iscsi interface 8/1. It is also used to assign pwwn to the interface as shown in the recipe below.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# interface iscsi 8/1 sjc7-9509-6(config-if)# switchport proxy-initiator nWWN 21:05:00:0d:ec:02:2d:82 pwwn 21:05:00:0d:ec:02:2d:82 sjc7-9509-6(config-if)# no shut sjc7-9509-6(config-if)# end
8-177
iSCSI
sjc7-9509-6#
Step 5
Add the iSCSI interface to the required VSANs. This needs to be done to allow the iscsi interface to communicate with the virtual target to see the luns in different VSANs. The commands below add the iSCSI interface 8/1 into the 3003 VSAN. This is VSAN to which the FC target is connected. Once this is done the interface iSCSI 8/1 will log on to the fabric.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi interface vsan-membership sjc7-9509-6(config)# vsan database sjc7-9509-6(config-vsan-db)# vsan 3003 interface iscsi 8/1 sjc7-9509-6(config-vsan-db)#end sjc7-9509-6#
Note
The command iscsi interface vsan-membership is required to make the iscsi interface as a part of the a VSAN. Configure the virtual target on the switch sjc7-9509-6.
sjc7-9509-6# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi virtual-target name sjc7-jbod-iscsi-1 sjc7-9509-6(config-iscsi-tgt)# pWWN 22:00:00:20:37:39:9c:1f sjc7-9509-6(config-iscsi-tgt)# end sjc7-9509-6#
Step 6
The virtual target is a name assigned to the storage device. This name has to be 16 characters long. Then the pwwn of the storage port is assigned to the virtual target as shown above. This completes the configuration of a virtual target.
Step 7
Create a zone with the initiator and the virtual target configured. Add the zone to the zoneset and activate the zoneset. Create a zone with the iSCSI initiator and the virtual targets as members.This enables the initiator and the target to communicate with each other. The zone can be created either with the iqn name or with the ip address or with the pwwn that was assigned to the initiator. In this recipe it is created using the pwwn. Zoning with pwwn of the virtual-target and initiator.
sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# zone name Z_iscsi_tst vsan 3003 sjc7-9509-6(config-zone)# mem pwwn 22:00:00:20:37:39:9c:1f sjc7-9509-6(config-zone)# mem pwwn 21:05:00:0d:ec:02:2d:82 sjc7-9509-6(config-zone)#end sjc7-9509-6# sjc7-9509-6# conf t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# zoneset name ZS_ISCSI vsan 3003 sjc7-9509-6(config-zoneset)#member Z_iscsi_tst sjc7-9509-6(config-zoneset)end sjc7-9509-6#
8-178
78-xxxxx-xx
Chapter 8
sjc7-9509-6(config)# zoneset activate name ZS_ISCSI vsan 3003 Zoneset activation initiated. check zone status sjc7-9509-6# sjc7-9509-6# show zoneset active vsan 3003 zoneset name ZS_iscsi vsan 3003 zone name Z_iscsi_proxy_8-1 vsan 3003 * fcid 0xd90002 [pwwn 21:05:00:0d:ec:02:2d:82] * fcid 0xd90000 [pwwn 22:00:00:20:37:39:9c:1f] sjc7-9509-6#
Tip
As a best practice, to achieve lun security a virtual target with access to specific luns needs to be created for each initiator. Configure a virtual target for each initiator and configure lun masking for the initiator Once the zone is successfully activated the luns made available on the storage port should now be visible to the iscsi interface. As this interface could be a proxy iscsi interface for many iscsi initiators, some form of lun security needs to enabled. To achieve lun security a virtual target with access to specific luns needs to be created for each initiator. In this recipe the iscsi interface can see 10 luns (lun 11 to lun 20 in decimal). The configuration below will allow the host sjc7-pc-1 to see luns 11 - 14 (decimal) on the array.
sjc7-9509-6# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi virtual-target name sjc7-jbod-iscsi-1 sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun b iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun b iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun d iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun e iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# initiator ip address 172.22.38.45 permit sjc7-9509-6(config-(iscsi-tgt))# end sjc7-9509-6#
Step 8
1 2 3 4
Similarly for the hosts sjc7-pc-2, the following needs to be done to make it see luns 16 - 20 (decimal).
sjc7-9509-6# config t Enter configuration commands, one per line. End with CNTL/Z. sjc7-9509-6(config)# iscsi virtual-target name sjc7-jbod-iscsi-1 sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 10 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 11 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 12 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 13 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# pwwN 22:00:00:20:37:39:9c:1f fc-lun 14 iscsi-lun sjc7-9509-6(config-(iscsi-tgt))# initiator ip address 172.22.38.46 permit sjc7-9509-6(config-(iscsi-tgt))# end sjc7-9509-6#
1 2 3 4 5
After the above configuration changes both the hosts should be able to see the luns allocated to them through the virtual -target created for each. As can be seen there is no need to create additional zones when new iSCSI client are added. If the iSCSI clients need access additional targets the iSCSI interface needs to be zoned to the other targets as shown in Step 7 in Configuring iSCSI on MDS in Proxy initiator mode.
8-179
iSCSI
Step 1
In the iSCSI initiators properties select the Discovery tab. This is shown using a red circle in Figure 8-4. In this tab select the Add button. This brings up the Add Target Portal dialogue. The ip address of the gigE interface on the switch sjc7-9509-5 which is 172.22.34.83 and select OK. This will add the target in the iSCSI client. At this stage the show fcns database command should show the iSCSI initiator logged on to the fabric in the specified vsan. In this case it is vsan 3003. This is shown in the output below. If the initiator does not show up in the output check the and see if the initiator is properly configured. The output in brackets (ex: [dev1-1]) are examples of how Device Aliases are used, see Device Aliases, page 1-43 for more information.
sjc7-9509-5# show fcns database vsan 3003 VSAN 3003: -------------------------------------------------------------------------FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE -------------------------------------------------------------------------0x900002 N 21:05:00:0d:ec:02:2d:82 (Cisco) scsi-fcp:init isc.w 0xd900cb NL 22:00:00:20:37:39:9c:1f (Seagate) scsi-fcp:target
8-180
78-xxxxx-xx
Chapter 8
[dev1-1] 22:00:00:20:37:38:84:9e [dev3-1] 0xd900ce NL 22:00:00:20:37:18:16:c9 [dev5-1] 0xd900d1 NL 22:00:00:20:37:38:7c:a4 [dev6-2] 0xd900d2 NL 22:00:00:20:37:36:00:92 [dev4-1] 0xd900d4 NL 22:00:00:20:37:5a:40:26 [dev7-2] 0xd900d5 NL 22:00:00:20:37:38:88:9c [dev2-1] Total number of entries = 8 sjc7-9509-5# 0xd900cd NL
Step 2
Select the target tab in the iSCSI initiator properties. This is shown using a blue ellipse in Figure 8-5.
Figure 8-5 Target Tab in the iSCSI properties
If the virtual-target is properly configured and the initiator is permitted to access the virtual-target, the configured virtual target should be visible under targets and it status should be in active as seen in Figure 8-5 high lighted in a brown square.
Step 3
Select the Log On button. This brings up the Log On to Target window. This is shown in Figure 8-6.
8-181
iSCSI
Figure 8-6
Here there is an option of making the system automatically log on to the target automatically and also to enable multipath if multipathing software is installed on the server. Clicking on OK starts the iSCSI login and storage discovery process for the target listed in the window.An iSCSI initiator can see multiple targets.
Figure 8-7 Target Logged on status
On a successful log on to the target the status of the target changes for inactive to connected as seen in Figure 8-7 highlighted in a brown rectangle. At this stage the host should be able to see the storage made available (directly assigned or made available through LUN masking or otherwise) to the iSCSI initiator. Selecting the Details button in the Targets tab brings up the Target properties window. In this window selecting the Devices Tab lists the disk available to the host on the storage array. This is shown in Figure 8-8.
8-182
78-xxxxx-xx
Chapter 8
Figure 8-8
Now from the MS windows Start -->Settings -->Control Panel --> Administrative Tools --> Computer management -->Disk Management the user can scan for new disk and initialize them for use on the system.
iSCSI on Linux
This section talks about the configuration that needs to be performed on a linux client to get iSCSI client going on a Linux server. To enable iSCSI to work on a linux host the iSCSI driver has to be downloaded and configured. The linkdriver is currently maintained on sourceforge and can be downloaded from http://sourceforge.net/project/showfiles.php?group_id=26396&release_id=177564. In some instances the driver may need to be compiled. Once the driver is installed on the host it can be configured by editing the iscsi.conf file. This is usually present in the /etc directory on the linux server. Again as in case of the windows configuration the target portal has to be added and the iSCSI initiator needs to be configured on the MDS. Then the iSCSI initiator on the linux host has to be restarted to log on to the target to access the LUNs. As with the case of the Windows iSCSI driver, the iSCSI initiators name is auto generated by the driver subsystem. The auto generated name can be found in the /etc/initiatorname.iscsi file. The process of configuring the iSCSI initiator and virtual target on the MDS switch and creating zones and activating the zoneset is as shown in the Configuring iSCSI on MDS in Transparent Mode section. The iSCSI initiator name is auto generated by the iSCSI driver in linux and is stored /etc/initiatorname.iscsi. file.
8-183
iSCSI
Step 1
The /etc/iscsi.conf file has to be edited to make iSCSI configuration changes on a linux server. Under the DiscoverAddress Settings section add the address of the target portal, in this case it would be 172.22.34.83.
# DiscoveryAddress Settings # ------------------------# Add "DiscoveryAddress=xxx" entries for each iSCSI router instance. # The driver will attempt to discover iSCSI targets at that address # and make as many targets as possible available for use. # 'xxx' can be an IP address or a hostname. A TCP port number can be # specified by appending a colon and the port number to the address. # All entries have to start in column one and must not contain any # whitespace. # # Example: # DiscoveryAddress=172.22.34.83
This is all that is needed to changed to get basic iscsi configured on the linux server.
Step 2
Restaring the iSCSI process. Once the target portal address has been updated the iscsi processes have to be restarted to force the iscsi client to log on to the target and to discover the luns. This is done using the rc scripts. The rc scripts are located in /etc/rc3.d directory. It is S25iscsi script. This script needs to be run with a restart option after the changes to the /etc/iscsi.conf are made. The script also have a status option to check the status of the iscsi processes on the system
[root@sjc7-pc-6 Stopping iSCSI: Starting iSCSI: [root@sjc7-pc-6 iSCSI driver is [root@sjc7-pc-6 rc3.d]# /etc/rc3.d/S25iscsi restart sync umount sync iscsid iscsi iscsi iscsid rc3.d]# /etc/rc3.d/S25iscsi status loaded rc3.d]#
At this stage the iscsi initiator should log on to the target and discover the luns that have been assigned to it.
8-184
78-xxxxx-xx