Anda di halaman 1dari 144

IBM Storage Area Network

Data collection Guide v. 21


2 | IBM Storage Area Network Data Collection Guide | TOC

Contents
Customer Information..........................................................................................................5
Introduction to the SAN data collection Guide..................................................................6
How to use the SAN data collection Guide..............................................................................................6
SAN Components.................................................................................................................. 7
Where to send the gathered data.........................................................................................8
Files to EMEASAN..................................................................................................................................8
ECuRep procedure................................................................................................................................... 8
Additional information required for data collection....................................................... 11
Data collection for Host platforms.................................................................................... 12
RS/6000 / AIX (UNIX).......................................................................................................................... 12
NOVELL................................................................................................................................................ 13
iSeries OS/400........................................................................................................................................14
iSeries Linux.......................................................................................................................................... 15
Management Module in BladeCenter.................................................................................................... 15
BladeCenter Modules............................................................................................................................. 16
BladeServer HSxx/LSxx (Intel/AMD) blades and general JSxx(PPC)................................................. 17
BladeCenter JS20 (type 8842) on PowerPC Architecture..................................................................... 17
XSeries/Netfinity/Intel/W2K3/W2K8....................................................................................................18
X-series Linux (SuSE and RedHat)....................................................................................................... 19
HP / HP-UX........................................................................................................................................... 20
SUN / Solaris..........................................................................................................................................21
Dec / Tru64.............................................................................................................................................23
Vmware.................................................................................................................................................. 24
SDD Enhanced trace capability..............................................................................................................24
SDD - using sddgetdata..........................................................................................................................24
SDDPCM Enhanced trace capability..................................................................................................... 25
SDDPCM - using sddpcmgetdata.......................................................................................................... 25
SDDSM Enhanced trace capability........................................................................................................25
SDDSM - using sddgetdata.................................................................................................................... 26
Special data collection for DS8000 attachments....................................................................................26
Performance problems ...................................................................................................... 28
iSeries OS/400 Performance.................................................................................................................. 28
SUN / Solaris Performance.................................................................................................................... 28
Performance Questionnaire.................................................................................................................... 28
Data collection for SAN connectivity components...........................................................33
IBM 2108-G07 Gateway (Obsolete due to EOS 1-1-2009)................................................................... 33
IBM 2108-R03 Gateway (Obsolete due to EOS 1-1-2009)................................................................... 33
IBM 7139/7140 Vicom Slic Router (SSA-FC Converter) (Obsolete due to EOS 1-1-2008)................ 34
IBM TotalStorage SANxxB-R (2109-Axx)........................................................................................... 36
IBM TotalStorage SAN04M-R (2027-R04) and SAN16M-R (2027-R16)........................................... 37
IBM 2006-L10 switch (Emulex)............................................................................................................ 46
IBM Storage Area Network Data Collection Guide | TOC | 3

IBM 3722 (Nexus 5000)........................................................................................................................ 47


IBM 3758 (Brocade B8000)...................................................................................................................48
IBM 2109-F16 Switch and 3534-F08 codelevel 3.x.............................................................................. 48
IBM 2109, 2498, 2499and 2005 codelevel 5.x and above..................................................................... 49
DCFM.....................................................................................................................................................51
McData 2034, 2032, 2031, 2026 and 2027 all models...........................................................................55
CNT/Inrange 2042 FC Director (FC9000), CNT UMD and IBM 2045-N16. (EOS at 1-1-2011) ........ 63
For Code Levels 3.0 and below..................................................................................................65
For Code Levels 3.2.1 and higher up to EM8006...................................................................... 65
For EM8006 and higher............................................................................................................. 65
FOR ALL CODELEVELS below 9.x........................................................................................65
CNT FC/9000, CNT UMD, IBM 2045-N16 and inVSN 9.0 code............................................ 69
Cisco 2061-0xx, 2061-420 and 2062-Dxx FC Switches and Director (MDS 9000)............................. 74
Data collection for Virtualisation components.................................................................77
2145 (SVC)............................................................................................................................................ 77
SVC Data Collection via SVC CLI............................................................................................77
SVC Data Collection via SVC GUI........................................................................................... 81
ICAT, masterconsole (GUI) and CIMOM datacollection..........................................................89
SVC performance data collection by using SVC Stats.............................................................. 97
SVC performance data collection by using TPC..................................................................... 102
4146 (SAN-FS).................................................................................................................................... 111
For SAN-FS higher Version2 Release1 (V2R1):..................................................................... 111
For SAN-FS lower Version2 Release1 (V2R1)....................................................................... 111
Data collection for Storage components......................................................................... 113
2105 and 2107 (DS8x00)..................................................................................................................... 113
IBM 2106 Modular Storage Server...................................................................................................... 113
All IBM DS4000 and DS5000 models.................................................................................................113
Performance problems..............................................................................................................113
Storage manager 8.x and lower................................................................................................ 114
Storage Manager 9.x and higher.............................................................................................. 119
First additional action plan....................................................................................................... 121
Second additional action plan.................................................................................................. 121
Third additional action plan..................................................................................................... 123
Fourth additional action plan....................................................................................................125
DS3000 Data collection with Storage Manager 2.x / 9.x.....................................................................126
DS300 and DS400................................................................................................................................ 128
DS6000.................................................................................................................................................129
NAS General........................................................................................................................................ 130
IBM 5190 Total Storage NAS 100 (Obsolete due to EOS 1-1-2007)..................................................131
IBM 5194 Total Storage NAS 200, 201, 225 and 226 (Obsolete due to EOS 1-1-2008).................... 132
IBM 5195 Total Storage NAS 300, 301, 325 and 326 (EOS 1-1-2011).............................................. 133
IBM 5196 Total Storage NAS 300G G00 and G25 (EOS 1-1-2011).................................................. 133
IBM 5198 Total Storage NAS 500 (EOS 1-1-2011)............................................................................134
XIV.......................................................................................................................................................136
IBM 4125 Total Storage IP Storage (all models) (Obsolete due to EOS 1-1-2009)............................137
4 | IBM Storage Area Network Data Collection Guide | TOC

N-Series - All models........................................................................................................................... 139


N-Series Gateway models.................................................................................................................... 140
N-Series dump analysis due to filer panic............................................................................................141
N-Series Performance Problems.......................................................................................................... 141
N-Series Snapdrive and Snapmanager Problems................................................................................. 142
N-Series issues with connected Tapes................................................................................................. 142
Tape ==> See Tape Cookbook.............................................................................................................142
General..................................................................................................................................... 143
Appendix A. Special notices............................................................................................. 144
IBM Storage Area Network Data Collection Guide | Customer Information | 5

Customer Information

Name: ANAF
Contact Person: Nimeni Altu
PMR: 23456,826,826
Date: 10.11.2010
Employee: Zoltan
e-Mail: zoltan_vodinszki@ro.ibm.com
6 | IBM Storage Area Network Data Collection Guide | Introduction to the SAN data collection Guide

Introduction to the SAN data collection Guide

The sometimes-complex structure of Storage Area Networks and the various different SAN devices used in such
environments has triggered the development of this document. It provides a step-by-step instruction on how to gather
information needed for troubleshooting SAN devices. It is designed to support service technicians and all other
persons involved in solving problems in the Storage Area Network neighbourhood.
"It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of
theories to suit facts."
Sherlock Holmes

How to use the SAN data collection Guide


The SAN Troubleshooting Guide contains information structured by SAN product
• Modular datacollection items per machinetype, model and firmware/software level.
• Step-by-step instruction on how to perform data collection on Hosts, Storage, SAN and SVC devices.
When troubleshooting a Storage Area Network the user of this guide is required to perform all steps described
in this document in order to collect all necessary data needed for data analysis. Therefore the user is asked to
prepare an inventory list containing all devices used in the SAN. Next he will visit all product sections in the SAN
Troubleshooting Guide suitable for the SAN in question and perform the required steps for data collection and data
analysis.
IBM Storage Area Network Data Collection Guide | SAN Components | 7

SAN Components

For an overview use http://www-03.ibm.com/servers/storage/ here you will find the latest data of all the components
8 | IBM Storage Area Network Data Collection Guide | Where to send the gathered data

Where to send the gathered data

Files to EMEASAN
Emails not containing the PMR number or containing attachments > 5 MB will be deleted unread.
So in nearly all cases you must use ECuRep (see below) as all logs are greater than 5MB
Lotus Notes address is EMEASAN@de.ibm.com
The subject should reflect the PMH number, branch number, country number and customer (e.g.
PMH12345,B678,C724 Customer XYZ)

ECuRep procedure
Files to big for E-Mail
In case the gathered files are too large (more than 5 MB) to be sent via E-mail (i.e. snap from AIX), please ftp the
files to ECuRep (EMEA Centralized Customer Data Repository).
What's the purpose of this ftp site?
Customers are not able to send data through the IBM firewall. Therefore, there is an IBM internet-server that
customers can access directly (ftp.emea.ibm.com).
Data on this server is mirrored every 10 minutes to an IBM intranet server.
How customers upload data?
If you request data from the customer, here's what the customer has to do. All data must be compressed before it is
sent to the FTP.EMEA.IBM.COM server (exceptions as desribed in the specific chapters). MVS customers should use
the TRSMAIN program to compress their file [SPACK parameter] which can be obtained from the IBM site: ftp://
ftp.software.ibm.com/s390/mvs/tools/packlib/ . The installation instructions are in the README file. By compressing
their data, they improve data transfer time and convert their files into a compatible record length and record format.
VM users should compress the file to be ftp'ed using the CMS COPYFILE command with the PACK option.'

Note: The FTP procedure (see below) is the preferred way to upload the files. If the customer can not use this
because the FTP possibilty is blocked then he can use:
• http://www.ecurep.ibm.com/app/upload for standard upload
• https://www.ecurep.ibm.com/app/upload for secure upload
IBM Storage Area Network Data Collection Guide | Where to send the gathered data | 9

Write down the PMR number


Use Hardware in the Upload is for field
Use your OWN email address
EcuRep will add the PMR number at the front on the file so NO need to rename the file as for normal FTP procedure.

If a customer is told to upload any data to the EMEA FTP server, he or she has to perform the following steps:
FTP to 'ftp.emea.ibm.com'
login as 'anonymous'
enter the email ID as password
> bin
> cd toibm
> cd hw
10 | IBM Storage Area Network Data Collection Guide | Where to send the gathered data

Type to upload the data :


> put xxxxx.bbb.ccc.yyy.yyy

Note:
Customers must use the appropriate naming conventions as shown in the examples below:
xxxxx.bbb.ccc.yyy.yyy
where the parts have the meaning:
xxxxx =PMR number
bbb = Branch Office (if known)
ccc = IBM Country Code (e.g. Germany 724)
yyy.yyy = Short description for the filetype, e.g. tar.Z, restore.Z, terse

Because login by the customer was done with an anonymous user ID, no file listing is allowed on the server.
Customers cannot check what files they have already uploaded, which means that this has to be done carefully.
INTRANET users, for example IBM employees, MUST use the 'ftp.ncs.mainz.ibm.com' URL to access the FTP
server.

Note: An automatic update to the PMR will only work with a correct file name.
IBM Storage Area Network Data Collection Guide | Additional information required for data collection | 11

Additional information required for data collection

To determine what you need to collect you can ask the customer the following questions:
• What is the customer's platform? (Z-series, P-series, I-series, X-series, Linux, HP,?)
• Which platform has the problem or is it a general problem?
• Is the problem on one or on more servers?
• Is it a solid or an intermittent problem?
• Did the customer “fixed” the problem and if yes how? (reboot a component, enable/disable a path,....)
• What components are in the SAN fabric? (2109, 2005, 2006, 2026, 2027,...., long distance solution, data
gateway,.....)
• Is the problem on one component, on more or do not know? (Data needed from one, more or every SAN
component).
• What kind of I/O has the problem? (DS4000, 2105, NAS, Tape)
• A complete SAN layout as described below.
• Detailed history of the problem giving dates and times when known or estimated
• Support actions already taken
• Time difference between SVCs, hosts, SAN, etc...
Drawing of his SAN layout
Including the following information:
• Cabling diagram including cabling from Host to Switch/Gateway to the ESS/Tape
• Port numbers and Domain-id of the switches
• WWNN or WWPN
• Number of cables connect to each of the components
• Are certain servers used in a clustered environment?
• Volume assignment
• How many volumes should each host see?
• Do any of the hosts share volumes?
• Zoning information (if used)
The drawing of the SAN layout can also be uploaded to PFE Datacase or ECurep
12 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

Data collection for Host platforms

RS/6000 / AIX (UNIX)


Run the following commands and save the output to a file:

Note: be sure that PTF U488916 is installed on version 5.2

> snap -r
> snap -gfikLGc
or
> Go to /tmp/ibmsupt
> rm -r*
> snap -gfikLGc

Also collect the following output:


Command fcstat fcsx (x = 0 to n) for ALL FC HBAs
This displays Fibre Channel adapter device driver statistics
• - Speed and other link stats
• - Transmit/receive counts
• - Driver internal statistics
Limitations
• - Link must be up and running
• - Not intended to manage SAN configuration (won't display zoning information, etc)
If SDD installed:
Run the following commands if available. (This depends on AIX and SDD version)
> showvpaths output
> datapath query adapter
> datapath query device
For SDDPCM/MPIO installations
We need the output from the following commands,
> pcmpath query adapter
> pcmpath query adaptstats
> pcmpath query device
> pcmpath query devstats
> pcmpath query essmap
> pcmpath query portmap
> pcmpath query wwpn
Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
• Type ps -ef |grep sddsrv to verify if the SDD server has automatically started and on what port (sddsrv starts with
20001 as the default port number; the customer may use a different port)
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 13

• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser (as stated before: 20001 can be
different)
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• aSubsystem information
• Datapath device information
> datapath query adapter
> datapath query device
> datapath query adaptstats
> datapath query devstats
> adapter performance
> device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace

NOVELL
Always request the following data from all involved NetWare servers:
CONFIG.txt (Server Configuration)
Execute from console screen
> LOAD CONFIG
==> This will create a current CONFIG.TXT which will be in SYS:\SYSTEM directory on the server.
ABEND.LOG
An ABEND.LOG is created in the SYS:\SYSTEM directory on a server whenever it abends.
SYS$LOG.ERR
Location of file: SYS:\System\SYS$LOG.ERR
LOGGER.TXT
The LOGGER.TXT contains the latest history of the logger screen, e.g. modules loaded, etc. + output of SDD
"datapath query" commands.
Hit F2-key to save the logger screen to C:\NWSERVER\LOGGER.TXT
If SDD is installed
Provide the outputs of the following SDD commands (can be included in LOGGER.TXT):
datapath query adapter
datapath query device
When DS3000/4000/5000 is connected to Novell NetWare we need:
14 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

1. config.txt This is to check the Novell Netware version and Support Pack (SP),? the correct driver settings are
loaded (Startup.ncf), the HBA adapter number and PCI slot, the HBA device driver version and the required
MPIO-driver and -module versions
2. list failover devices This is to check the preferred paths (priority) per HBA and Lun
3. cfg.nlm (this is a QLogic Module which comes with the IBM device driver package. It is a MSJ adequate for the
Novell console command line.) This is to check for HBA device driver version, the HBA WWN and BIOS, the
DS3000/4000/5000 controllers seen by the singel HBA and to check the switch domain and port where the HBA
and the DS4000/FAStT controller is inserted.
4. If you use the NetWare MPIO failover driver attached to the DS3000/4000/5000 run the commands
list partitions
list storage adapter
list failover devices

iSeries OS/400
Copies of the following print outputs...
1. System Configuration List
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 7 "Hardware Service Manager" from the "Start a Service Tool" screen
• Press F6 "Print configuration" from the "Hardware Service Manager" screen
2. System Configuration List -- Logical HW Resources:
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 7 "Hardware Service Manager" from the "Start a Service Tool" screen
• Select option 2 "Logical Hardware resources" from the "Hardware Service Manager" screen
• Press F6 "Print configuration" from the "Logical Hardware Resources" screen
3. World Wide Port Name Information (FiberChannel only)
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 7 "Hardware Service Manager" from the "Start a Service Tool" screen
• Select option 2 "Logical Hardware resources" from the "Hardware Service Manager" screen
• Select option 1 "System bus resources" from the "Logical Hardware Resources screen
• For each iSeries FiberChannel Disk Adapter #2766 or #2787:
• Select option 9 for the corresponding IOP #2843 or #2844 of the IOA #2766 or #2787 whose WWPN is of
interest
• Select option 5 "Display detail" for the IOA #2766 or #2787 and note the "Resource name" together with its
"Port worldwide name" (WWPN)
4. Product Activity Log :
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 1 "Product activity log" from the "Start a Service Tool" screen
• Select option 1 "Analyze log" from the "Product Activity Log" screen
• Enter "3" for Log (3 = Magnetic media log) and timeframe of log in the "Select Subsystem Data" screen
• Enter "3" for Report type (3 = Print options) and "Y" for including optional statistical entries in the "Select
Analysis Report Options" screen
• Enter "4" for Report type (4 = Print full report) and "Y" for including hexadecimal data in the "Select Options for
Printed Report" screen
5. System Operator Message Queue:
• Run the "DSPMSG MSGQ(QSYSOPR) OUTPUT(*PRINT)" command to get a spool file of the system operator
message queue contents
6. List of installed PTFs
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 15

• Issue command "DSPPTF OUTPUT(*PRINT)"

iSeries Linux
Copies of the following print outputs...
1. Linux Partition Allocated I/O Resources
• Access system service tools by using the command STRSST
• Select option 5 "Work with system partitions" from the "System Service Tools (SST)" screen
• Select option 1 "Display partition information" from the "Work with System Partitions" screen
• Select option 3 "Display allocated I/O resources" from the "Display Partition Information" screen
• Enter the corresponding Linux partition number for "System partition(s) to display"
• Press F6 "Print" from the "Display Allocated I/O Resources" screen
2. Linux Partition Reference Code History
• Access system service tools by using the command STRSST
• Select option 5 "Work with system partitions" from the "System Service Tools (SST)" screen
• Select option 1 "Display partition information" from the "Work with System Partitions" screen
• Select option 8 "Display secondary partition reference code history" from the "Display Partition Information"
screen
• Enter the corresponding Linux partition number for "System partition(s) to display"
• Press F6 "Print" from the "Display Secondary Partition Reference Code History" screen
3. Startup Console Log
• Issue command "dmesg > /var/log/dmesg.txt" and obtain the output file
• Alternatively from the managing OS/400 partition:
• Access system service tools by using the command STRSST
• Select option 5 "Work with system partitions" from the "System Service Tools (SST)" screen
• Select option 1 "Display partition information" from the "Work with System Partitions" screen
• Select option 10 "Display guest environment console log" from the "Display Partition Information" screen
4. System Message Log
• Obtain the syslogd system message text file "/var/log/messages"
5. List of installed PTFs on the OS/400 managing partition.
• Issue command "DSPPTF OUTPUT(*PRINT)" and obtain the generated spool-file

Management Module in BladeCenter


For the Management Module, the MM EventLog, Hard-/Firmware VPD and MM Configuration File
Collect the EventLog of the webfronted Management Module :
• access the MM webfronted w/ the MM-Servers IP address
• in the left MM pane under 'Monitor' select 'Event Log'
• in the main MM pane the EventLog info shows up
• scroll down to the bottom of the EventLog table
• press 'Save Log as Text File'
• send this *txt text file as described in section ECuRep procedure on page 8
Collect the Hardware VPD section of the webfronted Management Module :
• access the MM webfronted w/ the MM-Servers IP address
• in the left MM pane under 'Monitor' select 'Hardware VPD'
• in the main MM pane the Hardware VPD info shows up
• in the browser do a File/Save As... and select 'Archive Single File (*.mht)' as save type.
• send this archive *.mht file as described in section ECuRep procedure on page 8
Collect the Firmware VPD section of the webfronted Management Module :
• access the MM webfronted w/ the MM-Servers IP address
• in the left MM pane under 'Monitor' select 'Firmware VPD'
16 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

• in the main MM pane the Firmware VPD info shows up


• in the browser do a File/Save As... and select 'Archive Single File (*.mht)' as save type.
• send this archive *.mht file as described in section ECuRep procedure on page 8
Collect the MM Configuration File content of the webfronted Management Module :
• access the MM webfronted w/ the MM-Servers IP address
• in the left MM pane under 'MM Control' select 'Configuration File'
• in the main MM pane the Configuration File info shows up
• in the upper 'Backup Configuration' section click on the? 'view the current config summary' text link
• select the content of the config summary popup window (CTRL-A) and copy it (CTRL-C) to a texteditor ( CTRL-
V)
• save this texteditor content to a txt file
• send this *.txt text file described in section ECuRep procedure on page 8

BladeCenter Modules
a) DataCollection for BladeCenter Ethernet Switch Module
- Perl is required to run a support script, therefor Perl needs to be installed on the MM Server
- Perl is open source and can be downloaded at http://www.perl.org
- request the 'dlink_dump.zip' package to generate html Ethernet Switch Module data from IBM
- this package can be download at http://bisc.raleigh.ibm.com
- from there go to 'Resources -> Utilities -> Dlink Module Dump Tool' and save this .zip file
- out of this zip archive you need 2 files to generate the output file we need :

the perl script : get_dlink.pl


and the ascii input file : dlink.txt
- change the first 3 lines of the ascii input file to your ESM's IP, userid and password, for example:
host 192.168.70.127
userid USERID
password PASSW0RD ( the O is a zero ! )
- the command syntax to generate the html output of your ESM is?
perl get_dlink.pl input_file > output_file
- therefore issue the command
perl get_dlink.pl dlink.txt > dlink.html
- send this 'dlink.html' output file to ECUREP as described in section ECuRep procedure on page 8
b) Data Collection for BladeCenter Fibre Channel Switch Modules
6Generate a 'show support' ( Qlogic FCSM ) resp. a 'supportSave ' ( Brocade FCSM ) logfile :
- easiest way is to use Windows HyperTerminal service:
- go to 'Start -> Programs -> Accessoires -> Communications -> HyperTerminal'
- a 'New Connection' popup asks for a connection name
- enter a name describing you FCSM module best ( which bladecenter, which bay, ... )
- in the 'Connect To' popup, change the 'Connect using' field to 'TCP/IP ( Winsock )'
- the 'Connect To' popup changes and now asks for 'Host Address' and 'Port Number'
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 17

- enter the IP address of your switch module to the 'Host Address' field,
( when in Bay3 of the BladeCenter, the IP address defaults to 192.168.70.129,
when in Bay4,? it defaults to 192.168.70.130 )
- enter a '23' to the 'Port Number' field
- press 'OK'
- in the HyperTerminal main window, you now can logon to the switch :
• for Qlogic SanBox2 use : sanbox2 login=USERID and password=PASSW0RD (0=zero!)
• for Brocade 3016 use : user= USERID and password= PASSW0RD (0=zero!)
- start recording of logging data with HyperTerminal menu 'Transfer -> Capture Text -> Start',
define file location and file name for the logging data of the switch and press 'Start'
- in the HyperTerminal main window, send the correct FCSM cmd to generate the data we need
- stop recording
- send the generated log file to ECUREP as described in section ECuRep procedure on page 8
- if you have multiple FCSMs, repeat the above process while using the appropriate IP address.

BladeServer HSxx/LSxx (Intel/AMD) blades and general JSxx(PPC)


For each of the HS20/HS40 blades :
Running WINDOWS :
• Run Windows DSA See section >DSA (Dynamic System Analysis)< for DSA procedure
• send output data file(s) as described in section >DSA (Dynamic System Analysis)<
Running LINUX :
- run LINUX DSA :
• Log in as root
• Download or copy latest egather2-x.xx.linux to a directory of your choice ( /tmp )
• Enter the command: chmod u+x egatherer2-x.xx.linux
• Enter the command: ./egatherer2-x.xx.linux to run eGatherer.
• eGatherer completes with an EG2 file in the same directory.
• send output data file(s) as described in section >DSA (Dynamic System Analysis)<

BladeCenter JS20 (type 8842) on PowerPC Architecture


For each of the JS20 blades :
- install SUSE linux packages for JS20 Service Aids and Hardware Diagnostics :
a) lsvpd-x.xx.x-x.ppc.rpm.gz (where x.xx.x-x is the version number)
b) ppc64-utils-x.x-x.ppc64.rpm.gz (where x.x-x is the version number)
c) diagela-x.x.x.x-x.ppc.rpm.gz (where x.xx.x-x is the version number)

- collect JS20 data by following the 'Linux Aid Install Instructions for JS20' :
http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/images/JS20_Install.pdf
• generate lvcfg output : lvcfg > filename.txt
• generate snap data : snap (generates snap.tar.gz )
• send generated filename.txt and snap.tar.gz to ECUREP as described in section ECuRep procedure
on page 8
18 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

XSeries/Netfinity/Intel/W2K3/W2K8
Customer needs to provide the following information:
• Manufacturer of FC Adapter/SCSI Adapter
• BIOS level of the FC Adapter/SCSI Adapter
For data collection use the DSA tool. See section for procedure.
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If SDD is installed:
Run the following commands
> datapath query adapter
> datapath query device
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
Verifying that the SDD server has started
W2K3 and W2K8
Click Start > Programs > Administrative Tools > Computer Management.
Expand the Services and Applications tree.
Click Services.
Right-click SDD_Service.
Click Start. The status of SDD Service should be Started if the SDD server has automatically started.
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• Subsystem information
• Datapath device information

> datapath query adapter


> datapath query device
> datapath query adaptstats
> datapath query devstats
> adapter performance
> device performance

• Server trace
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 19

• get server trace


• start detailed server trace
• start normal server trace
• Device driver trace

X-series Linux (SuSE and RedHat)


For data collection use the DSA tool See section >DSA (Dynamic System Analysis)< for procedure
Run the following commands and save the output to the displayed filenames
> df -a > df.txt
> iostat -t > iostat.txt
> uname -a > uname.txt
and send the following files as well.
> /var/log/messages
> /proc/scsi/scsi
> /proc/scsi/<adaptertyp>/<adapternumbers> (e.g. /proc/scsi/qla2200/2)
> /proc/partitions
> etc/modules.conf

If SDD installed: Run the following commands:


> datapath query adaptstats > adaptstats.txt
> datapath query adapter > adapter.txt
> datapath query device > device.txt
> addpaths debug > addpaths_debug.txt
> lsvpcfg
If RDAC installed: Run the following script:
> /opt/mpp/mppSupport
This collects all data for RDAC (failover) troubleshooting with DS4000
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
Type ps wax |grep sddsrv to verify if the SDD server has automatically started.
command output looks like this:
31616 S 0:00 /opt/IBMsdd/bin/sddsrv
31617 S 0:00 /opt/IBMsdd/bin/sddsrv
31618 S 0:00 /opt/IBMsdd/bin/sddsrv
31619 S 0:10 /opt/IBMsdd/bin/sddsrv
31620 S 0:00 /opt/IBMsdd/bin/sddsrv
31621 S 0:00 /opt/IBMsdd/bin/sddsrv
31622 S 0:00 /opt/IBMsdd/bin/sddsrv
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser
- Enter "Server trace" and provide "get server trace"
20 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

- Provide "Device driver trace"


Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• Subsystem information
• Datapath device information
• adapter status (datapath query adapter)
• device status (datapath query device)
• adapter statistics (datapath query adaptstats)
• device statistics (datapath query devstats)
• adapter performance
• device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace

HP / HP-UX
Question to customer:

Is the EMS (Event Monitoring Service) DISABLED on all HPUX attached to ESS?
Run this commands:
> swlist > output.txt
> swlist -l patch
> ioscan -fnC disk
> dmesg >> output.txt
> diskinfo /dev/rdsk/c#t#d#
> pvdisplay
> uname -m
> uname -r
> lsdev

Get the following file : var/adm/syslog.log

Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If SDD is installed
Run the following commands
> showvpaths output
> datapath query adapter
> datapath query device
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 21

Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
• Type ps -ef |grep sddsrv to verify if the SDD server has automatically started and on what port (sddsrv starts with
20001 as the default port number; the customer may use a different port)
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser (as stated before: 20001 can be
different)
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• Subsystem information
• Datapath device information
• adapter status (datapath query adapter)
• device status (datapath query device)
• adapter statistics (datapath query adaptstats)
• device statistics (datapath query devstats)
• adapter performance
• device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace

SUN / Solaris
There is a data collection tool available in the internet.
Go to http://sunsolve.sun.com/search/document.do?assetkey=1-9-82329-1
Customer must be registered for this tool
Download the SunTM Explorer Data Collector
Run the collector

Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Collect also the data for SDD or veritas (see next page)
OR
For data collection of Solaris Systems provide the following information:
1. modinfo > modinfo.output
2. showrev -p > showrev.output will show the installed patch level
Update the installed patch level if needed. Patches for Solaris 2.6 can
be downloaded at:
22 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

http://sunsolve.sun.com/pub-cgi/show.pl?target=patches/patch-access
3. Send the file "/etc/systems"
4. Send the file "/kernel/drv/sd.conf" if connected to disk
Send the file "/kernel/drv/st.conf" if connected to tape
5. If FC adapters are used
Send the file "/kernel/drv/fcaw.conf" for JNI/JAYCOR - FC adapter
or

Send the file "/kernel/drv/ql2100.conf" forQLOGIC 2100 - FC adapter


or
Send the file "/kernel/drv/ql2200.conf" forQLOGIC 2200 - FC adapter
or
Send the file "/kernel/drv/lpfc.conf" for PCI EMULEX lp8000 - FC adapter
or
Send the file "/kernel/drv/lpfs.conf" for SBUS EMULEX lp8000 - FC adapter
6. Send "dmesg" output (redirect to file "dmesg > dmesg.output")
Send all files named " /var/adm/messages.* " (* = 0,1,2,....)
7. Send iostat -nE output (shows reference between 2105 S/N and SUN disks, shows also ESS microcode)
8. We need to get some more details about the current SCSI parameters:
Enter the debugger tool in kernel mode "adb -k". Within the tool type in the following parameters (one after the
other):
- "scsi_options /X"
- "sd_io_time /X"
- "sd_retry_count /X"
- "sd_max_throttle /X"
Note down the output (hex values) of the parameters above.
Exit the tool with "$q"
9. SUN settings need to be double-checked:
sd_retry_count and sd_max_throttle
See ESS host attachement guide for the setting formula.
10. output of command "diskinfo" (to see relation of ESS volumes S/N to SUN physical disks)
Need to check the following patches in the showrev -p output:
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 23

Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If SDD is installed:
Run the following commands
> showvpaths output
> datapath query adapter
> datapath query device

Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

If the customer use Veritas Volume Manager :


> vxdisk list
> vxdisk list <Nbr of the disk>
> vxdmpadm listenclosure all
> dmpadm getdmpnode enclsure=shark0

Additional SDD action plan (see also >SDD, SDDPCM and SDDDSM data collection<)
• Type ps -ef |grep sddsrv to verify if the SDD server has automatically started and on what port (sddsrv starts with
20001 as the default port number; the customer may use a different port)
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser (as stated before: 20001 can be
different)
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
• Server status
• Device information
• Subsystem information
• Datapath device information
• adapter status (datapath query adapter)
• device status (datapath query device)
• adapter statistics (datapath query adaptstats)
• device statistics (datapath query devstats)
• adapter performance
• device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace

Dec / Tru64
24 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

Use the tool Sys_Check-Tool to gather all relevant system information.


Sys_Check can be downloaded at:
http://h30097.www3.hp.com/sys_check/
The Sys_Check Tool generates a HTML file of the Tru64 UNIX system configuration.
One important section is the system.log, it shows the errors.
Send the HTML output file to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

Vmware
Run the following script on the service console:
> /usr/bin/vm-support
This script collects all relevant ESX Server system and configuration information and ESX Server log files.
Send the requested data and files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

SDD Enhanced trace capability


Starting with SDD 1.5.1.0 (and later), SDD has enhanced its trace capability for problem determination. SDD has
been enhanced to collect SDD trace data periodically and write the trace data to the system local hard drive. SDD
maintains 4 files for its trace data:
• sdd.log
• sdd_bak.log
• sddsrv.log
• sddsrv_bak.log
These files can be found in the following directories:
• AIX - /var/adm/ras
• HP-UX - /var/adm
• Linux - /var/log
• Solaris - /var/adm
• Windows 2000 and Windows NT - \WINNT\system32
• Windows Server 2003 - \Windows\system32

SDD - using sddgetdata


SDD provides the sddgetdata script to collect information used for problem determination. For UNIX
platforms, sddgetdata creates a tar file or a compressed tar file at the current directory with the current
date and time as a part of the file name (for example, sdddata_hostname_yyyymmdd_hhmmss.tar or
sdddata_hostname_yyyymmdd_hhmmss.tar.Z, where yyyymmdd_hhmmss is the timestamp of the file creation).
For Windows, you can run the sddgetdata script from any directory to collect the data for problem determination.
sddgetdata creates a cab file in the %root%\Program Files\IBM\Subsystem Device Driver directory with the
current date and time as part of the file name (for example, sdddata_hostname_yyyymmdd_hhmmss.cab), where
yyyymmdd_hhmmss is the timestamp of the file creation).
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 25

When you report an SDD problem, it is essential to run this script and send this output file for problem determination.
Steps within the sddgetdata script might fail depending on the problem and the system condition. Is this case, you
might have to execute manual commands.
Here is an example output for the AIX platform:
/tmp/sdd_getdata>sddgetdata
/tmp/sdd_getdata>ls
./ ../ sdddata_host1_20050315_122521.tar
Send the requested data and files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

SDDPCM Enhanced trace capability


Starting with SDDPCM 2.1.0.7 ( and later), SDDPCM has enhanced its trace capability for problem determination.
SDDPCM has been enhanced to collect SDDPCM trace data periodically and write the trace data to the system local
hard drive. SDDPCM maintains 4 files for its trace data:
• pcm.log
• pcm_bak.log
• pcmsrv.log
• pcmsrv_bak.log
These files can be found in the /var/adm/ras directory.
When you report an SDDPCM problem, the following information is essential for problem determination:
• pcm.log
• pcm_bak.log
• pcmsrv.log
• pcmsrv_bak.log
• Output of the pcmpath query adapter command
• Output of the pcmpath query device command
• AIX system error log

SDDPCM - using sddpcmgetdata


SDDPCM provides the sddpcmgetdata script to collect information used for problem determination. sddpcmgetdata
creates a tar file at the current directory with the current date and time as a part of the file name (for example,
sddpcmdata_hostname_yyyymmdd_hhmmss.tar, where yyyymmdd_hhmmss is the timestamp of the file creation).
When you report an SDDPCM problem, it is essential to run this script and send this tar file for problem
determination.
For example:
/tmp/sddpcmgetdata>sddpcmgetdata
/tmp/sddpcmgetdata>ls
./ ../ sddpcmdata_test1_20050315_122521.tar
Send the requested data and files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

SDDSM Enhanced trace capability


26 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms

Perfomed by SDD service


SDDDSM log:
%Windir%\system32\sdd.log
%windir%\system32\sdd_bak.log
SDD service log:
%windir%\system32\sddsrv.log
%windir%\system32\sddsrv.log

SDDSM - using sddgetdata


SDDDSM also provides the sddgetdata script to collect information used for problem Determination:
SDDGETDATA.BAT
Generates sddgetdata_%host%_%date%_%time%.cab
SDD\SDDSrv logs
Datapath output
Event Logs
Cluster log
SDD specific registry entry
HBA information
Send the requested data and files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

Special data collection for DS8000 attachments


DSCLI related problems
• Run the failing command in verbose-mode (dscli> setoutput -v on or edit dscli.profile under <Home Dir>/dscli/
profile, change verbose: on)
• Run the scenario and save the output (copy into text file)
• Try to get timestamp of the HMC when command executed
• DSCLI-Client: Collect >/dscli/log (niCA.log, niClient.log)
Collect logs under <Home Dir>/dscli/log (dscli_x_x.log)

Remarks:
Home Dir is in case of WinXP: C:\Documents and Settings\USERID
Install Dir defaults: WIN = c:\Program Files\IBM\DSCLI\...
UNIX based systems: \opt\ibm\dscli\...
DS8000 Storage manager related problems:
If you have the Storage manger installed on a separate client gather the following logs:
(Paths are only valid, if the default install directories got used during installation):
Windows client:
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 27

C:\Program Files\IBM\SMServer\AppServer\logs\*.*
C:\Program Files\IBM\SMServer\AppServer\log\*.*
C:\Program Files\IBM\dsniserver\logs
C:\Program Files\IBM\SMServer\logs
C:\Program Files\IBM\DS8000StorageManager\logs
C:\Program Files\IBM\DS8000StorageManager\SM\logs
Linux client:
/opt/IBM/SMServer/logs/*.*
/opt/IBM/SMServer/AppServer/logs/server1/*.*
/opt/IBM/SMServer/AppServer/logs/*.*
28 | IBM Storage Area Network Data Collection Guide | Performance problems

Performance problems

iSeries OS/400 Performance


The following procedure requires that OS/400 "Performance Tools" licensed program feature "5722PT1" (*BASE
and option 1) is installed -- for verification enter "GO LICPGM" and select option 10 "Display installed licensed
programs"
Start settting up iSeries Collection Services:
• Access "IBM Performance Tools for iSeries" by using the command "GO PERFORM"
• Select option 2 "Collect performance data" from the "IBM Performance Tools for iSeries" screen
• Select option 1 "Start collecting performance data" from the "Collect Performance Data" screen and confirm the
default settings via pressing ENTER
After granting a reasonable time for performance data collection create an iSeries Performance Tools disk utilization
report:
• Go back to the "IBM Performance Tools for iSeries" screen and select option 3 "Print performance report"
• Select option 1 "System report" for the corresponding member object from the "Print Performance Report -
Sample data" screen
• Select option 1 "Disk Utilization" from the "Select Sections for Report" screen
• Press F6 to print the entire report from the "Select Categories for Report" screen
• Select an appropriate start/stop time interval to specify the data basis for the report to be generated from the
"Specify Report Options" screen

SUN / Solaris Performance


• Enter the following command: /usr/bin/iostat -xnc 5 12 (This starts 12 probes with a time difference of 5
seconds.)
• Save the output.

Performance Questionnaire
SVC L2 Support EMEA’s Performance Questionaire

Note: To get a clear picture of the situation, the involved devices and the impact, fill out the below questions as
precise as possible.

1. General Questions :

1.1 Problem occurrence


1.1.1 When exactly did the problem start (date/time) ?
1.1.2 Is it persistent now and impacting customers productive env ?
1.1.3 Is it of transient or reoccuring for a certain time frame or
static behaviour ?
1.1.4 Does it shows up when any kind of (copy) services are started ?
IBM Storage Area Network Data Collection Guide | Performance problems | 29

1.1.5 If the problem is transient, did customer perform any recovery


action or is the recovery automatic ?
.
1.2 When did the performance problem initially start ?
1.2.1 After any kind of SAN issue ?
1.2.2 After upgrading drivers or firmware on any product ?
1.2.3 After expanding the SAN ?
1.2.4 After changing anything in the config (storage/host/SVC/SAN) ?
1.2.5 After other changes / incidents ?
1.2.6 Power outages ?

1.3 What is the impact to the business?


1.3.1 Applications and/or services slow ?
1.3.2 Applications and/or services crash ?
.
1.4 Does the customer use management or monitoring tools like SAN,
storage or host management tools ?
.
1.5 Have performance stats been gathered / how is "bad performance" measured ?
1.5.1 With/for SVC IO Stats ?
1.5.2 With/for TPC ?
1.5.3 Any other tool ?
1.5.4 What are the performance expectations ?

1.6 Is time synchronization between all involved devices done ?


If not provide time shift !

2. Host specific questions :

2.1 Can the performance problem be bound to a single host ?

2.2 Provide a list of affected and non-affected host systems


( each with OS type, OS level, host type, model ) !

2.3 Details about the affected hosts ( OS type / OS level ) ?


2.3.1 Provide OS and host name
2.3.2 Provide HBA driver and firmware levels
2.3.3 Provide failover/multipathing sw + version
2.3.4 Do host logs show any errors/events related to the problem ?
2.3.5 Do host specific and recommended sw updates / code fixes exist
which are not installed ?
30 | IBM Storage Area Network Data Collection Guide | Performance problems

2.3.6 In questions of sw levels and device drivers, the affected host(s) is ...
... pretty much uptodate ???
... very outdated ???
.
2.4 Can the performance problem be bound to a single or to
multiple applications ?
2.4.1 What applications ?
2.4.2 What does application(s) logs show ?

3. SVC specific questions :

3.1 Describe SVC environment/setup


3.1.1 Cluster size(s)
3.1.2 Cluster versions(s)
3.1.3 Impacted nodes

3.2 Does SVC report any specific errors which indicates I/O issues ?

3.3 Are SVC CopyServices configured ?


3.3.1 How well performed the env BEFORE CopyServices were started ?
3.3.2 If stopping CopyServices, does the problem still exist ?
3.3.3 Are consitency grps defined and do they start at the same time ?
3.3.4 Has connectivity between SVC clusters been checked to be ok ?

4. Storage Controller specific questions :

4.1 Can the performance problem be bound to a single or multiple


storage subsystems ?
4.1.1 Which storage subsystem(s) is/are affected ?
4.1.2 What code is installed on storage subsystem ?
4.1.3 What do storage subsystem logs show ?

4.2 Provide appropriate datacollections for the involved IBM


storage components, if storage is non-IBM, advice customer
to check with the non-IBM storage vendor to analyze the box
and come back with results.

5. SAN specific questions :

5.1 How does the SAN layout look like ? Core-edge ? Fully-meshed ?
5.1.1 Provide a detailed SAN topology with distances, port numbers,
attached devices, etc !
5.1.2 Describe SAN devices and which are affected by the problem !
IBM Storage Area Network Data Collection Guide | Performance problems | 31

5.2 Have error counters for all ports on all boxes been cleared and
monitored and - if needed - have appropriate actions been taken
( replace SFPs / cables / etc ) to solve obvious issues ?

5.3 Do we see any obvious error/event messages in the switch logs

5.3 Have the ISLs been checked for enough bandwidth ?

5.4 Are specific SAN switch features in use ( Preferred Path, Trunking,
traffic isolation ... ) ?
.
5.5 What about he SAN details :
5.5.1 Are long distance links in use ?
5.5.2 Are DWDMs / dark fiber / IP Routing / etc ... in use ?
5.5.3 If there is an FCIP WAN network, describe network and Service
Level Agreement SLA (bandwidth, MTU, delay) and TELCO provider ?
5.5.4 If an IP network used, is any other type of traffic mixed with
storage traffic ?

6. Next actions if not already done :

.6.1 provide datacollections of affected host(s) (see SAN-CookBook !)


6.1.1 one or two if possible
6.1.2 if possible when same OS: one of a "bad" / one of a "good" host

6.2 Provide full SVC datacollections (see SAN-CookBook !)

6.3 Provide a meaningful SAN layout (see section 5.1)

6.4 Provide SAN data collections (see SAN-CookBook !)


.
6.5 Provide performance data (see SAN-CookBook !)
6.5.1 Covering the time frame of the impact.
6.5.2 If possible also for a time frame when production works without
performance impact.
6.5.3 If possible, provide TPC performance data
6.5.4 For SVC perform stats collection (see San CookBook)
6.5.5 Use host/OS specific performance collecting tools to provide
performance data
.
6.6 Involve support structures for all products ( host / SAN / SVC /
storage ) and ask them to check THEIR part of the bad performing
32 | IBM Storage Area Network Data Collection Guide | Performance problems

storage environment for any obvious or eye-catching issue in regard


to the problem description !
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 33

Data collection for SAN connectivity components

IBM 2108-G07 Gateway (Obsolete due to EOS 1-1-2009)


Connect to the 2108-G07 gateway using the RS 232 interface or the Ethernet port on the backside of the machine.
Information on how to establish a connection to the gateway can be obtained in Appendix B. Serial port connection
using NetTerm
Start the terminal program "Net Term" which can be downloaded at:
http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session login
Login into the gateway via the RS232 interface or through the Ethernet port
The default login is set to:
Login: "admin"
Pass: "password"
If the customer has changed the default login settings he needs to provide the current values.
Run the following commands (case sensitive) and send the log file to ECUREP as described in section ECuRep
procedure on page 8:
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If this 2108 is an integrated SAN Data Gateway in a LTO library:
> supportDump
In all other cases :
> version
> fcShow
> hostTypeShow
> sysConfigShow
> fcShowDevs
> mapShowDatabase
> scsiShow
> scsiRescan
> scsiTermGet
> scsiHostidGet
> scsiHostChanGet
> loggerDump
> csEtimeShow

IBM 2108-R03 Gateway (Obsolete due to EOS 1-1-2009)


Connect to the 2108-R03 gateway using the RS 232 interface or the Ethernet port on the backside of the machine.
Information on how to establish a connection to the gateway can be obtained in Appendix B. Serial port connection
using NetTerm
Start the terminal program "Net Term" which can be downloaded at:
http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000078
34 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Enable session logging


Login into the gateway via the RS232 interface or through the Ethernet port
The default login is set to:
Login: "admin"
Pass: "password"
If the customer has changed the default login settings he needs to provide the current values.
Run the following commands (case sensitive) and send the log file to ECUREP as described in section ECuRep
procedure on page 8:
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
> version
> fcShow
> hostTypeShow
> sysConfigShow
> fcShowDevs
> mapShowDatabase
> scsiShow
> scsiRescan
> scsiTermGet
> scsiHostidGet
> scsiHostChanGet
> loggerDump
> supportDump
> csEtimeShow

IBM 7139/7140 Vicom Slic Router (SSA-FC Converter) (Obsolete due to


EOS 1-1-2008)
1.Provide the installed firmware level
Therefore you need to connect to the SLIC Router using the serial port interface.

From the Serial Port Service Utility Key Assignments select Display VPD
Capture the window and send it to ECUREPas described in section ECuRep procedure on page 8
2.Provide the settings of DIP Switch 1 and DIP Switch 2
Both switches are located on the side of the unit
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 35

3.Provide information on the LED Status

The front panel contains three LED?s


Power LED (green)
Status LED (green)
Fault LED (yellow)
Select the LED Status out of the following table

?
Slow Blink: LED is ON 90% and OFF 10%
Steady Blink: LED is ON 50% and OFF 50%
If the Status LED Blinks Code refer to the following section in order to decipher the LED Blinks
The Blink Code of the Status LED presents digital numbers as follows:
36 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Numerical Digits
Presentation of
0 = short, fast blink
1 = LED blinks once
2 = LED blinks twice with one short duration between blinks
3 = LED blinks three times with one short duration between each blink
And so on
Decimal Numbers
Each digit can be obtained by the number of blinks in series separated by only a short duration of LED Off
Decimal Digits are separated by a medium duration of LED OFF (two seconds)
Status Code is separated by a long duration of LED Off (four seconds) before the code is repeated
Example:

Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout including the Fibre Channel Side and the SSA side as described in section
Additional information required for data collection on page 11

IBM TotalStorage SANxxB-R (2109-Axx)


See Appendix C for the data collection procedure using the AIX platform.
Connect to the 2109-Axx switch using the RS 232 interface or the Ethernet port on the backside of the machine.
Information on how to establish a connection to the switch can be obtained in Appendix B. Serial port connection
using NetTerm
Start the terminal program "Net Term" which can be downloaded at:
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 37

http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session logging
Login into the switch via the RS232 interface or through the Ethernet port
The default login is set to:
Login: "admin"
Pass: "password"
If the customer has changed the default login settings he needs to provide the current values.
Run the command 'supportShow' and send the log file to ECUREP as described in section ECuRep procedure on
page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

IBM TotalStorage SAN04M-R (2027-R04) and SAN16M-R (2027-R16)


For the following procedure you need the "Enterprise Edition" of SANVergence. If customer is using the "Standard
Edition" of SANVergence , the way to gather data is to use the CLI thru telnet and enter the command (archive
feature not available):
> dump all <ip address of tftp server>
On the management console you must start the TFTP Server

Configure TFTP Server


38 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Set TFTP Server root directory

Checkmark TFTP Server transmit receive under Security

press OK
Verify the TFTP root in Windows Explorer
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 39

Go to the SANvergence Manager

Open all items under mSAN pane and select the multiprotocol router (in our case 2640-122)

Select Preferences
40 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Be sure that c:\TFTP-Root is selected in the TFTP rootpath. If not use browse to get it in.
Select OK
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 41

Open the "mSAN Configuration" Window

Select Tools at the top than navigate over reports and select Configuration Archive

Be sure that all items are ticked. And select Browse


42 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Go to Local Disk C
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 43

Go to TFTP-Root
44 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Name the file (in our case it is the TCP/IP name of the multiprotocol Router)

select open
Check the results in this panel under "Archive location and file name"
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 45

Press Start
It is possible that you get the following message if you did not follow the procedure correctly

(You maybe forgot to open the mSAN configuration window in a previous step)

You should see the status bar increase to 100%

Check for status "configuration archive completed"


46 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Verify on TFTP window the progress

When you see that all .log files are received than you must go to the file (in our case c:\TFTPRoot
\9.155.69.119.zip) and send this file to ECuRep

IBM 2006-L10 switch (Emulex)


• Connect to the 2106-L10 switch using the RS 232 interface or the Ethernet port (CLI).
• Be sure to use the capture text option and name the file according to the PMH number
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 47

• Enter password enter Logon (default is password).


Now the root menu is displayed.

• From root menu enter show


• Then enter dump or 15 this will take about 3 minutes to get all info
send the log file to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

IBM 3722 (Nexus 5000)

Capture to terminal emulator buffer: (CLI Session, Telnet).

Switch# terminal length 0


Switch# show tech-support
Switch# show tech-support fcdomain
Switch# show tech-support fcoe
Switch# show tech-support zone

Or:

Capture to file in volatile:

Switch# tac-pac (only the show tech-support)

Upload the file:


48 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Switch# dir volatile


12345 Date Time Year show_tech_out.gz
Switch# copy volatile:show_tech_out.gz sftp://username@Server /path

Send the output files to ECUREP as described in section ECuRep procedure on page 8

IBM 3758 (Brocade B8000)

Telnet to the switch and capture the Console-Log:

Switch:admin> supportsave
Switch:admin> cmsh
Switch:cmsh> terminal length 0
Switch:cmsh> show tech-support

Run the command and send the log file to ECUREP as described in section ECuRep procedure on page 8:

IBM 2109-F16 Switch and 3534-F08 codelevel 3.x


See Appendix C for the data collection procedure using the AIX platform.
Connect to the 2109-F16 switches using the Ethernet port on the backside of the machine. Information on how to
establish a connection to the switch can be obtained in Appendix B. Serial port connection using NetTerm
Start the terminal program "Net Term" which can be downloaded at:
http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session logging
If you use an AIX or UNIX server be sure the buffer size is big enough. To set it use
dtterm -sb -sl 50000 command (50000 = number of lines)
or run: telnet address.of.a.device | tee /path/to/logfile
where address.of.a.device is an address (a resolvable name or an IP address) of the device you want to collect data
from and /path/to/logfile is a path to a file you want to log your output to. This command will save the whole telnet
session to the file named logfile
Login into the switch through the Ethernet port
The default login is set to:
Login: "admin"
Pass: "password"
If the customer has changed the default login settings he needs to provide the current values.
Run the command and send the log file to ECUREP as described in section ECuRep procedure on page 8:
> supportShow
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If you have a problem on one or more ports and you can recreate the problem perform the additional Action Plan
below.
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 49

Additional Action Plan


Run the command to reset the port errors:
> portLogClear
Run the command:
> supportShow
Recreate the problem
Run the command 'supportShow' and send the log files to ECUREP as described in section ECuRep procedure on
page 8
In case the customer has installed more than one Switch:
If multiple switches are installed in the SAN-Network it is required to run the command 'supportShow' on every
single switch.
Procedure for Lost password-Request
If customer has lost his password we need the following:
• Model of the switch. (2109 model F16 or .....)
• Codelevel on the switch. (3.x.x.)
• WWN from the switch. (10:00:00:60:69:xx:xx:xx)
• If customer knows his ROOT password. (Yes or No)

IBM 2109, 2498, 2499and 2005 codelevel 5.x and above


See Appendix C for the data collection procedure using the AIX platform.
The M12 and has TWO separate 64 port switches in one chassis.
If FICON is used on the switches you must collect data on the MVS (Z-series) console.
Let customer provide the screen output in ASCII for the following z/OS (OS/390) commands.
These commands? are very helpful for analyzing any kind of errors in a FICON environment.
On the affected LPAR (logical Partition) start the z/OS Commands:
D M=CHP command displays the status and type of CHPIDs
D M=CHP(cc) where cc is the affected CHPID.
D U,,,dddd,1 where dddd is the affected device address
DS P,dddd where dddd is the affected device address
D M=DEV (dddd) where dddd is the device address
F IO,D D dddd * where dddd is the device address
Connect to the 2109-M12/M14 or 2005-H08/H16 switches using the Ethernet port on the front of the machine.
Information on how to establish a connection to the switch can be obtained in Appendix B. Serial port connection
using NetTerm
Start the terminal program "Net Term" which can be downloaded at:
http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session logging
If you use an AIX or UNIX server be sure the buffer size is big enough. To set it use
dtterm -sb -sl 50000 command (50000 = number of lines)
50 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

or run: telnet address.of.a.device | tee /path/to/logfile


where address.of.a.device is an address (a resolvable name or an IP address) of the device you want to collect data
from and /path/to/logfile is a path to a file you want to log your output to. This command will save the whole telnet
session to the file named logfile
Login into the switch through the Ethernet port
The default login is set to:
Login: "admin"
Pass: "password"
If the customer has changed the default login settings he needs to provide the current values.
In FICON environment run the following commands before running the supportSave all:
supportshowcfgenable ficon
ficoncupset fmsmode enable
Run the command and send the log file to ECUREP as described in section ECuRep procedure on page 8:
> supportSave all
In case of a director (2 CP cards) you must also get a 'supportSave all' from the standby CP
To perform this you must login to the TCP/IP address of the standby CP
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If you have a problem with unexpected 2109 reboots and system crashes, do:
Run the command:
> pdshow
and
Run the command and collect the files which are ftp-ed to an ip host:
> savecore
If you have a problem on one or more ports and you can recreate the problem perform the additional Action Plan
below.
Additional Action Plan
Run the command to reset the port errors:
> portLogClear
Run the command:
> supportSave all
Recreate the problem
Run the command and send the log files to ECUREP as described in section >ECUREP<:
> supportSave all
In case the customer has installed more than one Switch:
If multiple switches are installed in the SAN-Network it is required to run the command
'supportSave all' on every single switch.
Procedure for Lost password-Request
If customer has lost his password we need the following:
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 51

• Model of the switch. (2109/2005 model M12 or .....)


• Codelevel on the switch. (5.x.x.)
• WWN from the switch. (10:00:00:60:69:xx:xx:xx or? 10:00:00:05:1E:xx:xx:xx)
• If customer knows his ROOT password. (Yes or No)

DCFM
If client and server are running on the same system you need to perform part 1 only
If client and server are running on different system you need to perform part 1 and 2.
1/ DCFM Server data collection
Starting DCFM Server Console using SMC Batch File
Double click on <install directory>\DCFM 10.0.X\bin\smc.bat
Or From command line
Form DOS
C:\Program Files\DCFM 10.X.X\bin\smc.bat or
From Solaris and Linux
/opt/DCFM10_X_X/bin/smc
The process status should be started (if not, try to start all the process using the start button)

Using the "technical support information tab" choose a output path and click capture
52 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

When DCFM supportSave completes you will see the following message with location of the file

-If Client and Server running same system, the server supportSave will capture both Client and Server supportSave
data (DCFM client supportSave, HTTP capture, Java Console)
If client and server are not running on the same system we need to perform the client data collection as follow
2/ DCFM client data collection
In this part we will collect: client supportSave, HTTP capture and Java console output
Client supportSave
On Microsoft Windows client system you will find "clientsupportSave.bat"file at:
C:\Documents and Settings\<User currently logged in Windows Host>\DCFM\<Server IP>".
-The DCFM Client collection will take a few minutes
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 53

HTTP capture
We also need to capture the HTTP on MS Windows
How to get HTTP Capture on MS Windows
- Only Available on Microsoft Windows, not available on Unix
- Go to the "%DCFM_HOME%\bin"directory
--C:\Program Files\DCFM 10.0.2\bin (default location)
- Double click on the "httpcapture.bat"file.
- The HTTP Capture Utility will capture all required HTML pages ofthe Fabrics and Switches currently monitored in
DCFM.
- The HTTP Capture will be captured under "%DCFM_HOME%\support\HttpCapture-<TIMESTAMP>"folder.
Please zip the corresponding older when sending it
Then we will need to capture the JAVA console
Java Console output
-If there is a problem starting the Client collect the Java Console output
Click on the JAVA icon the windows control panel
54 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Enable java console output (use control panel, java, advanced, enable logging and show console.)

Starting the client will open the JAVA consol


IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 55

Debug information will appear in the java window. Just cut and paste this information.
If some data collection are not possible one alternative would be to collect
DCFM_10.X.X_InstallLog.log file under
x:\Program Files\DCFM 10.x.x
=> And in addition the content of conf and logs directory (Under x:\Program Files\DCFM 10.x.x

Send the output files to ECUREP as described in section ECuRep procedure on page 8:

McData 2034, 2032, 2031, 2026 and 2027 all models.


If FICON is used on the switches you must collect data on the MVS (Z-series) console.
Let customer provide the screen output in ASCII for the following z/OS (OS/390) commands.
These commands are very helpful for analyzing any kind of errors in a FICON environment.
On the affected LPAR (logical Partition) start the z/OS Commands:
D M=CHP command displays the status and type of CHPIDs
D M=CHP(cc) where cc is the affected CHPID.
D U,,,dddd,1 where dddd is the affected device address
DS P,dddd where dddd is the affected device address
D M=DEV (dddd) where dddd is the device address
F IO,D D dddd * where dddd is the device address
Preferred to collect data is option 1 (using EFCM
56 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

1
a EFCM 8.7 and higher without group manager license
b EFCM 8.7 and higher with group manager license

2 Collecting data using web GUI interface (EFCM basic)


a Data collection for EFCM Basic
b CLI procedure
I CLI for all M-series classical switches except IBM 2027-256 (McData I10K)
II CLI for 2027-256 (McData I10 K)

Note: For older code you can refer to previous version of SAN cookbook as these are not supported anymore.

1) Collecting data Using EFCM


a) EFCM 8.7 and higher without group manager license
Select the switch, right mouse click and launch element manager.
On element manager choose the following menu.
Maintenance| Data collection| extended and save the Zip file in the local disk

Send the file toEMEASAN@de.ibm.comor ECuRep .


Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
b. EFCM 8.7 and higher with group manager licence

Note: This allows DC of multiple switch a the same time

On EFCM main windows, click on Configure pulldown menu, Select Group Manager
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 57

Select Select Action on the left


Select radiobutton in front of Run data collection
Click on Next at the bottom

Select Select Switches on the left


Use > and < arrows to select or deselect the switches to collect the data from.
Click on Next at the bottom
58 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Select Output Options on the left


Use radiobutton Zip files into a single file
Fill in a filename that reflect the PMH number or RCMS number
Use Browse to put the file into a folder

Select a folder and click on OK


Select Execution options on the left
Do not tick the options
Click on Next on the bottom
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 59

Select Data Collection on the left


Select Start on the right

See the status change to In Progress


60 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

When status is Completed select Finish

Use explorer on the EFCM and get the file you just created.
Send the file ECuRep as described in ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
2) Collecting data using web GUI interface (EFCM basic formerly SANpilot)
For McData E/OS firmware 08.00.00 and above go to subsection a
For McDATA E/OS firmware below 08.00.00 collect data according subsection b
a) Data collection for EFCM Basic with Eos 08.00.00 and above (Web GUI)
Choose Logs| all
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 61

And save the logs as text file

Chose maintenance | product information

Save the product info text file

Chose maintenance | system file and click on data collection


62 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Save Data collection (and dumps if this apply)


b) CLI procedure
As alternative if the GUI is not available, you can collect the command line interface. (CLI)
I) CLI for all switches except IBM 2027-256 (McData I10K)
• Telnet to the switch (eg. using command window) and logon as Administrator (case sensitive!). Default pw is
password.
• Type and capture the following command at root prompt:
> show all
> show eventLog
> show auditlog
> show fabric node
> show fabricLog noWrap
> show fabricLog Wrap
> show linkincidentlog
• Send the file toECuRep as described in ECuRep procedure on page 8
• Provide also the complete SAN Layout as described in section Additional information required for data
collection on page 11
II)CLI for 2027-256 (McData I10 K)
Capture the output of the following command
use min 120 char per line to avoid wrapped text
> stty pagination off (to avoid to Press any key to continue (Q to quit))

> fc show switch 1

> version

> system config

> system status

> system fru

> card status

> card show

> system inventory

> port opticsInfo


IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 63

> port pom

> port status

> port technology

> fc interopMode 1

> fc interopMode 1

> fc config 1

> fc show fabric 1

> fc show login

> fc show nameserver 1

> system log show audit

> system log show event

> system log show link

> system log show opentrunking

> system log show threshold

> system log show fabric wrapping 1

> system log show fabric non-wrapping 1

> system log show port wrapping 1

> system log show port non-wrapping 1

> system log show security

> system log show trap

and in addition ftp additional log to


> system log upload advanced [Ftp hostname] [userName] [password] [fileName]
Send the file toECuRep as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

CNT/Inrange 2042 FC Director (FC9000), CNT UMD and IBM 2045-N16.


(EOS at 1-1-2011)
If FICON is used on the switches you must collect data on the MVS (Z-series) console.
Let customer provide the screen output in ASCII for the following z/OS (OS/390) commands.
These commands are very helpful for analyzing any kind of errors in a FICON environment.
On the affected LPAR (logical Partition) start the z/OS Commands:
D M=CHP command displays the status and type of CHPIDs
64 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

D M=CHP(cc) where cc is the affected CHPID.


D U,,,dddd,1 where dddd is the affected device address
DS P,dddd where dddd is the affected device address
D M=DEV (dddd) where dddd is the device address
F IO,D D dddd * where dddd is the device address
• For CNT UMD and IBM 2045-N16 follow chapter CNT FC/9000, CNT UMD, IBM 2045-N16 and inVSN 9.0
code on page 69
• For FC/9000 (2042) update PMH with the following information. This is also available as a form on our web site.
http://web.mainz.de.ibm.com/e_dir/esssstosol.nsf/DocumentsSortedTitle/-Homepage
SAN
FC switches and directors
FC/9000 and 2042 Problem Determination Form
• Than follow the correct data collection procedure depending of the code level (starting at For Code Levels 3.0
and below on page 65)
Problem Description: Brief description of problem
Analysis of Problem: Your understanding of problem after speaking with the customer
Customer Information/Severity:
1) Account: Name
2) Contact: Name and Number
3) Business Impact (ie. Switch down, degraded, Applications down)
4) Where is Debugbackup.zip file located
5) Dial in # (if available)
6) When does Customer expect an update?
Director Information:
1) Director and fabric name
2) Serial number
3) Director size
4) Director firmware / EM version
5) Are ISL s attached Y/N
Problem:
1) Were there any recent network changes prior to the problem being reported?
2) What were they doing at the time of problem?
4) Did the problem occur as part of an install, move, add, change and so did it occur during/after the IMAC?
5) Visual examination of director (LED s, blink codes, red boxed on GUI)
6) If problem is failed board, please record / capture the results of resetting and reseating the board
7) Any errors in Host System logs and if so what are they?
Environment: (Only required if applicable to problem.)
1) Is network diagram available
2) Ficon or Open systems?
3) What types of devices are being impacted and on which ports?
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 65

4) What applications are running?


5) What types of HBA s and drivers are being used?

For Code Levels 3.0 and below


1) Using the Primary FCM's Serial / Debug / Hyperterminal Port:
Capture all options and all PB's (Port Blocks) on the following screens:
2a, 2b, 2c, 2d, 2j, 2s & 2z. *** Not all of these options will be available, depending on the level of code running.
2) With the Client and Server shutdown, run the "Debugbackup.bat" program in the C:\Inrange\FC9000 folder.
Collect the resultant "Debugbackup.jar" (or .zip) file.
3) Any configuration information available that addresses what servers and devices are attached to the FC9000.
Including Driver levels and Port numbers affected.
4)Any Zoning information.
5)A detailed description of the problem.
6)Goto FOR ALL CODELEVELS below 9.x on page 65 For all codelevels

For Code Levels 3.2.1 and higher up to EM8006


1) Using the Primary FCM's Serial / DeBug / Hyperterminal Port:
Capture all options and all PB's (Port Blocks) on the following screens:
2a, 2b, 2c, 2d, 2j, 2s, 2z & 6x.
2) From the Manager-Server go to Maintenance on the tool bar and select Debugbackup. Collect the resultant
"Debugbackup.zip" file from the Inrange\fc9000 folder.
3) Any configuration information available that addresses what servers and devices are attached to the FC9000.
Including Driver levels and Port numbers affected.
4) Any Zoning information to include active Zoneset.
5) A detailed description of the problem.
6) Goto FOR ALL CODELEVELS below 9.x on page 65 For all codelevels

For EM8006 and higher


1) From the EM Server screen: Select "Maintenance" - Select "Debug Backup" option
2) When the window opens asking about collecting data; select all of the installed directors to gather serial port
data from.*** Depending on the number of directors installed on the EM, this operation may possibly take up to 20
minutes to complete.
3) Any configuration information available that addresses what servers and devices are attached to the FC9000.
Including Driver levels and Port numbers affected.
4) A detailed description of the problem.
5)Goto FOR ALL CODELEVELS below 9.x on page 65 For all codelevels

FOR ALL CODELEVELS below 9.x


Collect also the following screen shots
Start the IN-VSN Enterprise Manager
Start the IN-VSN Client
The default logon is: User Name: admin
66 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Password: admin
In the IN-VSN Client click Fabrics > Fabric Name (here TIC) and double click on the director's picture as show
below

The Enterprise Manager will display the general view of the director

Select System Configuration


Make a screen shot of this image
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 67

Select Version and click Export

The Export save menu wills pop-up. Specify a directory and file name to store the file. Make sure that the files
extension is set to .txt (Text file)
Click on Export
Click on Audit Trail
68 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

Click on Export
The Export Save Menu wills pop-up. Specify a directory and file name to store the file. Make sure that the files
extension is set to .csv (Comma Separated Values Files)
Click on Export
Click on EventLog

Click on Export
The Export Save Menu wills pop-up. Specify a directory and file name to store the file. Make sure that the files
extension is set to .csv (Comma Separated Values Files)
Click on Export
Send all files and screen shots to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 69

CNT FC/9000, CNT UMD, IBM 2045-N16 and inVSN 9.0 code
For UMD and 2045 you must complete a special form and do the data collection below (1. or 2.). Send it to ECuREP
or EMEASAN@de.ibm.com
Form is available on our web site. http://web.mainz.de.ibm.com/e_dir/esssstosol.nsf/DocumentsSortedTitle/-
Homepage
SAN
FC switches and directors
UMD and 2045 Problem Determination Form
For the IN-VSN 9.0 code you can collect the Debug Backup file in two ways. Server (1) or Client (2)
1. Using the inVSN Enterprise Manager - Server
1) Go to the inVSN Enterprise Manager - Server

2) Select Maintenance
3) Select Debug Backup
4) Give the .zip file a name and navigate in the folder structure to the place where you want to create the xxx.zip file
and click on Create.
70 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

5) Select from which UMD's or FC/9000 you want to create this debug
backup file. Multiple machines are possible. (Default is all machines)

6) Select OK
7) Check if create was successful and click OK

8) Click Close
9) Send all files and screen shots to EMEASAN@de.ibm.comor ECuRep
10) Provide also the complete SAN Layout as described in section Additional information required for data
collection on page 11
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 71

2. Using the inVSN Enterprise Manager Client 9.x.x.x code


1) Go to the inVSN Enterprise Manager Client. You can do this from every server that is in the same network as the
IN-VSN Enterprise Manager Server by using Internet Explorer and the TCP/IP address of the IN-VSN Enterprise
Manager Server. Than click on Launch! and enter the username and password (default is admin - admin).
2) Select SAN and than the fabric.

3) or Select Directors and than the UMD or FC/9000.

4) Select File in the top left corner.


5) Click on Debug Backup.
72 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

6) Select from which UMD's or FC/9000 you want to create this debug
backup file. Multiple machines are possible. (Default is all machines)

7) Click on Save As and give the .zip file a name and navigate in the folder structure to the place where you want to
create the xxx.zip file.
8) Select OK
9) Select OK
10) Select Yes
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 73

11) Check if create was successful and click OK

12) Click Close


13) Send all files and screen shots to EMEASAN@de.ibm.comor ECuRep
14) Provide also the complete SAN Layout as described in section Additional information required for data
collection on page 11
Collect also the event log
1) Go to the inVSN Enterprise Manager Client

Select View in
the top left corner
74 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

2) Select Events

3) Select Export on the top left corner


4) Give the .txt file a name and navigate in the folder structure to the place where you want to create the .txt file.
5) Select Export
6) Select OK
7) Send all files and screen shots to ECUREP as described in section ECuRep procedure on page 8:
8) Provide also the complete SAN Layout as described in section Additional information required for data collection
on page 11

Cisco 2061-0xx, 2061-420 and 2062-Dxx FC Switches and Director (MDS


9000)
IMPORTANT:
Provide CISCO Serial Number in every PMH you open to EBESAN.
Serial number is located on a label on machine chassis.
CISCO Serial number is 11 characters starting with JAB .
Where Cisco serial = JAB0YZZ1234, IBM Serial = 13-Y1234.
You can use the CISCO Product identification tool ( need CCO id for access , use the following url http://
tools.cisco.com/Support/CPI/index.do) help for label location on MDS9216, MDS9506, MDS9509. etc.
If FICON is used on the switches you must collect data on the MVS (Z-series) console.
Let customer provide the screen output in ASCII for the following z/OS (OS/390) commands.
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 75

These commands are very helpful for analyzing any kind of errors in a FICON environment.
On the affected LPAR (logical Partition) start the z/OS Commands:
D M=CHP command displays the status and type of CHPIDs
D M=CHP(cc) where cc is the affected CHPID.
D U,,,dddd,1 where dddd is the affected device address
DS P,dddd where dddd is the affected device address
D M=DEV (dddd) where dddd is the device address
F IO,D D dddd * where dddd is the device address
Connect to 2061 or 2062 using Ethernet or Serial Port (9600 8-N-1, no flow control). Further information on how to
establish a connection to the switch can be obtained in Chapter 3 'Connecting the Cisco MDS 9500' of the Cisco
MDS 9500 Series Hardware Installation Guide.
Start the terminal program "Net Term" which can be downloaded at:
http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session logging
If you use an AIX or UNIX server be sure the buffer size is big enough. To set it use
dtterm -sb -sl 50000 command (50000 = number of lines)
or run: telnet address.of.a.device | tee /path/to/logfile
where address.of.a.device is an address (a resolvable name or an IP address) of the device you want to collect data
from and /path/to/logfile is a path to a file you want to log your output to. This command will save the whole telnet
session to the file named logfile
Login into the switch
The default login is set to:
Login: "admin"
Pass: "admin"
If the customer has changed the default login settings he needs to provide the current values.
Run the command:
> terminal length 0
Run the command:
> show tech-support detail
Run the command:
> terminal length 24
Run the these commands on every single switch of the fabric and send the log file to ECUREP as described in
section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11 You may print a copy
of the map provided by Cisco Fabric Manager as a SAN Layout.
Other commands that can be done comparing to 2109:
2109---------------------mds9000
cfgshow(zoneshow)---***--sh zoneset active [vsan vsan#]
76 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components

----------------- sh zoneset (local defined in running config)


configshow---------------sh run (included in sh tech)
--------------- sh fctimer
diagshow------------***--sh int brief
errdump------------------sh log (recent events)
------------------sh log nvram (older major events)
------------------sh ver (included in sh tech)
fabricshow----------***--sh fcs ie vsan vsan#
--------------sh fcdomain domainlist [vsan vsan#]
nsallshow-----------***--sh fcns data [vsan vsan#]
nsshow-------------------sh fcns data local [vsan vsan#]
Qlshow-------------------sh TLport int <x/y> topology
porterrshow---------------sh interface counters brief
portflagshow--------------sh fcs <?>
portlogdump--------------debug fcns all (switch off with 'un all'
psshow-------------------sh hard (volle info über dir gesamte HW)
-------------------sh env (included in sh tech)
supportshow---------***--sh tech
switchshow----------***--sh fcs database
---------------sh fcdomain [vsan vsan#]
topologyshow-------------sh fspf database
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 77

Data collection for Virtualisation components

2145 (SVC)
Minimum requirements
• open a PMS (compid: SANVCNTL1), not a PMH and send it to EBESAN,25V
• SVC software level (CLI command: svcinfo lscluster <cluster_name>)
• Detailed history of the problem including dates and times when known or estimated
• Upload requested logs to DUMPDECODER (See procedure in the datacollection parts)
• Support actions already taken
• Time difference between SVCs, hosts, SAN, ...

SVC Data Collection via SVC CLI


Note: If CLI is not usable go to SVC Data Collection via SVC GUI on page 81
a) PUTTY session ; collect data onto config node
• Go to the Master console
• Open a Putty session to the SVC you want to collect the data
78 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

• login as 'admin'

• Depend on SVC code, perform the following commands to capture the data collection onto the Config node:
• SVC code 4.1 and above use step 3 and 4 (a)
• SVC code less as 4.1 use step 1 to 4 (b or c)
1. IBM_2145:admin>svcinfo lsnode -delim :
Write down the node id's of all Non config node's (1 to 7)

1. IBM_2145:admin>svctask cpdumps -prefix /dumps <node_id_of_non-config_node>


Perform this step for ALL non config node's
1. IBM_2145:admin>svcconfig backup
1. Choose a, b or c depending on code level.
a) for SVC code level 4.1.0.1 and above: *
IBM_2145:admin>svc_snap dumpall ; will take some time
Output format is snap_data collected in
/dumps/snap.FrontPanel_ID.yymmdd.hhmm.tgz
(eg : snap.106176.080116.125027.tgz)
b) for SVC code level 3.1.0.2 - 3.1.0.5:
IBM_2145:admin>svc_snap dumpall ; will take some time
Output format is snap_data collected in
/dumps/snap.yymmdd.hhmm.tgz
(eg : snap.060117.0723.tgz)

c) for SVC code level 2.1.0.3 - 3.1.0.1:

IBM_2145:admin>svc_snap ; will take some time


Output format is snap_data collected in
/dumps/snap.yymmdd.hhmm.tgz
(eg : snap.060117.0723.tgz)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 79

b) Transfer collect data from the config node onto Master Console
On the SVC Master Console open a DOS window:
Create a new directory in root of master console with commands
• cd\ ; change directory to root
• md data ; make directory

• IF putty less as 0.60 is installed, you will find it under:


• cd support utils ; This brings you into the sub directory "support utils"
• cd putty ; This brings you into the sub directory "support utils\/putty"
• If putty 0.60 and later is installed, you will find it under:
• cd program files ; This brings you into the sub directory "Program Files"
• cd putty ; This brings you into the sub directory "Program Files\putty"

Followings commands copy the data collection from config node into the directory 'data' of Master Console:
for SVC code level 2.1.0.3 and above :
> pscp -unsafe -load <name_of_putty_session> admin@<cluster_ip_address>:/
dumps/snap* c:/<target_dir>
if we request more information use first both commands of below 2.1.0.3
for SVC code level below 2.1.0.3 :
> pscp -unsafe -load <name_of_putty_session> admin@<cluster_ip_address>:/
dumps/* c:/<target_dir>

> pscp -unsafe -load <name_of_putty_session> admin@<cluster_ip_address>:/


dumps/elogs/* c:/<target_dir>

> pscp -unsafe -load <name_of_putty_session> admin@<cluster_ip_address>:/


dumps/configs/* c:/<target_dir>

Note:
80 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

for example:
<name_of_putty_session> -> PFE_SVC
<cluster_ip_address> -> 9.155.86.33
<target_dir> -> data

• zip all files from 'data' directory together with name of PMR_ClusterName.zip
Data must be sent to following as follows:
The following procedure must be done for every SVC problem BEFORE sending it to our EBESAN, 25V queue
because there are several problems for uploading the correct data to the correct servers in Hursley.
With this procedure we do not need our laptop to download and upload the data and will save up to 4 hours when
working from a remote location.
ALL SVC data must be sent to ECuRep.
NEVER use nesting zip, tar or other compression program (e.g. zipping a zipfile) as the decoder has a problem to get
the data readable when nested compression is used.
Procedure for the correct ECuRep procedure is on the following website:
http://itcenter.mainz.de.ibm.com/ecurep/emea.html
There will be an automatic update in the PMR if the data is correctly uploaded to ECuRep.
2 options are available; WEB-Upload or FTP
Following PMR number 00062.239.624 is only an example. Use your specific PMR number instead.
Tasks of the customer:
Do NOT compress the output file as this is already a .tgz file.

Note: It can happen that the .tgz file is changed into .tg or .gz file. This is a windows habit to do this. Please be sure
it is a .tgz file, otherwise rename it.

o Option 1: WEB-Upload

Note:
If you have problems with WEB-Upload, please use alternative the FTP procedure (see below)
http://www.ecurep.ibm.com/app/upload for standard upload
https://www.ecurep.ibm.com/app/upload for secure upload
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 81

Write down the PMR number


Use Hardware in the Upload is for field
Use your OWN email address
ECuRep will add the PMR number at the front on the file so NO need to rename the file as for normal FTP procedure.

o Option 2: FTP procedure


Please rename the file and add the PMR number at the front of the file
e.g. 00062.239.624.snap.106176.080116.125027.tgz (PMR is separated by DOTS "."!)
FTP to 'ftp.emea.ibm.com'
login as 'anonymous'
enter the email ID as password
> bin
> cd toibm
> cd hw

Type to upload the data:


> put 00062.239.624.snap.106176.080116.125027.tgz

SVC Data Collection via SVC GUI


a) PUTTY session ; collect data onto config node
• Go to the Master console
• Open a Putty session to the SVC you want to collect the data
82 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

• login as 'admin'

• Depend on SVC code, perform the following commands to capture the data collection onto the Config node:
• SVC code 4.1 and above use step 3 and 4 (a)
• SVC code less as 4.1 use step 1 to 4 (b or c)
1. IBM_2145:admin>svcinfo lsnode -delim :
Write down the node id's of all Non config node's (1 to 7)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 83

1. IBM_2145:admin>svctask cpdumps -prefix /dumps <node_id_of_non-config_node>


Perform this step for ALL non config node's
1. IBM_2145:admin>svcconfig backup
1. Choose a, b or c depending on code level.
a) for SVC code level 4.1.0.1 and above: *
IBM_2145:admin>svc_snap dumpall ; will take some time
Output format is snap_data collected in
/dumps/snap.FrontPanel_ID.yymmdd.hhmm.tgz
(eg : snap.106176.080116.125027.tgz)
b) for SVC code level 3.1.0.2 - 3.1.0.5:
IBM_2145:admin>svc_snap dumpall ; will take some time
Output format is snap_data collected in
/dumps/snap.yymmdd.hhmm.tgz
(eg : snap.060117.0723.tgz)
c) for SVC code level 2.1.0.3 - 3.1.0.1:
IBM_2145:admin>svc_snap ; will take some time
Output format is snap_data collected in
/dumps/snap.yymmdd.hhmm.tgz
(eg : snap.060117.0723.tgz)
b) Transfer collect data from the config node onto Master Console
open GUI -> Service and Maintenance > List Dumps > Software Dumps
84 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Select the snap file; right mouse click and "save target as" on local server
Data must be sent to following as follows:
The following procedure must be done for every SVC problem BEFORE sending it to our EBESAN, 25V queue
because there are several problems for uploading the correct data to the correct servers in Hursley.
With this procedure we do not need our laptop to download and upload the data and will save up to 4 hours when
working from a remote location.
ALL SVC data must be sent to ECuRep.
NEVER use nesting zip, tar or other compression program (e.g. zipping a zipfile) as the decoder has a problem to get
the data readable when nested compression is used.
Procedure for the correct ECuRep procedure is on the following website:
http://itcenter.mainz.de.ibm.com/ecurep/emea.html
There will be an automatic update in the PMR if the data is correctly uploaded to ECuRep.
2 options are available; WEB-Upload or FTP
Following PMR number 00062.239.624 is only an example. Use your specific PMR number instead.
Tasks of the customer:
Do NOT compress the output file as this is already a .tgz file.

Note: It can happen that the .tgz file is changed into .tg or .gz file. This is a windows habit to do this. Please be sure
it is a .tgz file, otherwise rename it.

o Option 1: WEB-Upload

Note:
If you have problems with WEB-Upload, please use alternative the FTP procedure (see below):
http://www.ecurep.ibm.com/app/upload for standard upload
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 85

https://www.ecurep.ibm.com/app/upload for secure upload

Write down the PMR number


Use Hardware in the Upload is for field
Use your OWN email address
ECuRep will add the PMR number at the front on the file so NO need to rename the file as for normal FTP procedure.

o Option 2: FTP procedure


Please rename the file and add the PMR number at the front of the file
e.g. 00062.239.624.snap.106176.080116.125027.tgz (PMR is separated by DOTS "."!)
FTP to 'ftp.emea.ibm.com'
login as 'anonymous'
enter the email ID as password
> bin
> cd toibm
> cd hw

Type to upload the data :


> put 00062.239.624.snap.106176.080116.125027.tgz

Tasks of IBMer (You!):


After upload is done, you will get a message similar to this in the PMR:

Go to https://ecurep.mainz.de.ibm.com/aex/ and type in you PMR number (separated with commas like in Retain)
86 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Click on "toPMR" .
In the next window you will find the uploaded file. Select the file(s) you want to transfer.

Along with other options like compressing or extracting you have the possibility to transfer the file via FTP. Select
this option and click on execute.

In the next window set the Destination host to SVCSUPPORT.SSD.HURSLEY.IBM.COM and the directory
to /2145/in/00062.239.624 (PMR #; separated by dots instead of commas)
Folder 00062.239.624 will be made if it is not there.
User is anonymous and password is your email-address.
Mode has to be set to binary.

For big files ( 200MB and more) use Parallel.


If Execution was on "Synchronous" you will get following message, when update is ready.
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 87

FINAL STEPS for IBMers


Log in to the Hursley dumpdecoder:
http://dumpdecoder.ssd.hursley.ibm.com/cgi-bin/DumpDecoder/DD_main.pl
Frontend members should have access. Please inform your teamleader or us (Level 2) when not!
Press the "ADD PMR" button on the bottom of the left pane.
88 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Fill in the fields EXACTLY as you did when you FTPed the files.
Click on Add PMR button
This automatically opens a new window like:

Click on "Add File" on the bottom of the right Pane

This opens the folder where you uploaded the files using FTP
Hit "Get" (If you used a different folder in the FTP part you can change it here)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 89

Wait a while (depending on the amount of data between 1 and 5 minutes)


Use "refresh" in your browser and you will get something like:

Now you can have a look in the error logs to see if these are readable.

ICAT, masterconsole (GUI) and CIMOM datacollection


If an SVC customer is experiencing ICAT/GUI errors or command failures. Collect the following logs for additional
PD. Please note that the problem does have to be recreated ones the logging is enabled.
The instructions are listed below for both the GUI (WAS) and CIM Agent.
See also the Wiki page for the latest version on:
http://svcsupport.ssd.hursley.ibm.com/wiki/index.php/SVC_GUI_and_CIMOM_Data_Collection

Note: The instructions for the GUI for SVC Console V4.2.1 ("Pegasus"-based) are different; please find it on: http://
svcsupport.ssd.hursley.ibm.com/wiki/index.php/SVC_Console_Log_Collection_%28V4.2.1%29

Pathnames
In the instructions below <WAS> refers to the base directory of the Websphere Application Server (WAS). In the
instructions below <CIM> refers to the base directory of the CIM Agent (CIMOM+Provider).
90 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

The directory for WAS is:


\Program Files\IBM\svcconsole\console\embeddedWAS
and for cimom:
\Program Files\IBM\svcconsole\cimom
WAS Log Files
The WAS log files are located in the following directory:
<WAS>/logs/server1
For debug/analysis purposes the
<WAS>/logs/server1/trace.log
<WAS>/logs/server1/SystemOut.log
1. Change the default directory to the WAS bin directory.
> cd <WAS>\bin
1. Enable logging, by entering the following on the command line.

wsadmin -connType NONE -c "$AdminControl setAttribute [$AdminControl


completeObjectName type=TraceService,process=server1,*] traceSpecification
ConsoleTrace=all=enabled"
> wsadmin -connType NONE -c "$AdminControl setAttribute [$AdminControl
completeObjectName type=TraceService,process=server1,*] traceSpecification
ConsoleTrace=all=enabled"

1. The following file will now contain the trace information

<WAS>/logs/server1/trace.log

A successfully enabled logging session will be indicated with an entry in the trace.log file similar to
[5/21/03 14:31:13:874 PDT] 2ff3581b ManagerAdmin I TRAS0018I: The trace state has changed. The new trace
state is ConsoleTrace=all=enabled.
At this point, attempt to do the procedure that is not completing successfully in the GUI.
After you have recreated the problem do the following:
1. Change the default directory to the WAS bin directory.
> cd <WAS>\bin
1. Disable logging, by entering the following on the command line.
> wsadmin -connType NONE -c "$AdminControl setAttribute [$AdminControl
completeObjectName type=TraceService,process=server1,*] traceSpecification
ConsoleTrace=event=disabled:ConsoleTrace=debug=disabled"

1. Trace information will no longer be written to the following file:

<WAS>/logs/server1/trace.log
A successfully disabled logging session will be indicated with an entry in the SystemOut.log file similar to

[5/21/03 14:38:57:400 PDT] 2ff3581b ManagerAdmin I TRAS0018I: The trace state


has changed. The new trace state is *=all=disabled.
Enable CIM Provider Logging
1. Stop CIMOM. To stop the CIMOM service:
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 91

• Control Panel->Administrative Tools->Services


• Select the service: "IBM CIM Object Manager - SVC".
• Right Click with the mouse and select 'Stop'.
• This will stop the CIMOM.
1. Enable logng and set level. Logging can be set to one of three different levels depending on required level

DEBUG_MIN

DEBUG_MID

DEBUG_MAX
Use DEBUG_MAX
• Edit the text file <CIM>/logger.properties
• Update the following entries to the desired debug level listed above

message.logger.level=

service.logger.level=

security.logger.level=

trace.logger.level=
• For general logging, trace and message logging should be enabled for logging. This is required for GUI
application debug/analysis.
1. Start CIMOM. To start the CIMOM service:
• Go to Control Panel->Administrative Tools->Services
• Select the service: "IBM CIM Object Manager - SVC".
• Right Click with the mouse and select 'Start'.
This will start the CIMOM.
1. Debug/Trace output will be displayed in <CIM>/providerTrace.log
The latest trace output is written to providerTrace.log, historic trace output is archived in providerTrace[x].log

Perform the function in the GUI that is not completing successfully


After the problem has been recreated, perform the following:
1. Stop CIMOM. To stop the CIMOM service:
• Go to Control Panel->Administrative Tools->Services
• Select the service: "IBM CIM Object Manager - SVC".
• Right Click with the mouse and select 'Stop'.
This will stop the CIMOM.
1. Disable logging and set level.
• Edit the text file <CIM>/logger.properties
• Update the following entries to the following debug level listed above

message.logger.level=DEBUG_MIN

service.logger.level=DEBUG_MIN

security.logger.level=DEBUG_MIN

trace.logger.level=DEBUG_MIN
1. Start CIMOM. To start the CIMOM service:
• Go to Control Panel->Administrative Tools->Services
92 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

• Select the service:"IBM CIM Object Manager - SVC".


• Right Click with the mouse and select 'Start'.
This will start the CIMOM.
Note: Enabled logging in the GUI application and/or the CIM Provider will impact the performance on the GUI.
Logging should be disabled where performance is a concern.
See also the following website: "http://svcsupport.ssd.hursley.ibm.com/wiki/index.php/
SVC_GUI_and_CIMOM_Data_Collection"
The following procedure must be done for every SVC problem BEFORE sending it to our EBESAN, 25V queue
because there are several problems for uploading the correct data to the correct servers in Hursley.
With this procedure we do not need our laptop to download and upload the data and will save up to 4 hours when
working from a remote location.
ALL SVC data must be sent to ECuRep.
NEVER use nesting zip, tar or other compression program (e.g. zipping a zipfile) as the decoder has a problem to get
the data readable when nested compression is used.
Procedure for the correct ECuRep procedure is on the following website:
http://itcenter.mainz.de.ibm.com/ecurep/emea.html
There will be an automatic update in the PMR if the data is correctly uploaded to ECuRep.
2 options are available; WEB-Upload or FTP
Following PMR number 00062.239.624 is only an example. Use your specific PMR number instead.
Tasks of the customer:
o Option 1: WEB-Upload
Do NOT compress the output files, do not rename or change the extention (.log)
e.g. providerTrace[x].log
Repeat this step for all files

Note:
If you have problems with WEB-Upload, please use alternative the FTP procedure (see below):
http://www.ecurep.ibm.com/app/upload for standard upload
https://www.ecurep.ibm.com/app/upload for secure upload

Write down the PMR number


Use Hardware in the Upload is for field
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 93

Use your OWN email address


ECuRep will add the PMR number at the front on the file so NO need to rename the file as for normal FTP procedure.

o Option 2: FTP procedure


Please rename the file and add the PMR number at the front of the file
e.g. 00062.239.624.providerTrace[x].log (PMR is separated by DOTS "."!)
FTP to 'ftp.emea.ibm.com'
login as 'anonymous'
enter the email ID as password
> bin
> cd toibm
> cd hw
Type to upload the data:
> put 00062.239.624.providerTrace[x].log
Repeat this step for all files
Tasks of IBMer (You!):
After upload is done, you will get a message similar to this in the PMR:
-CDDR PMRUPDATE RS4 -SANVCNTL1 -L203/-------P1S2-07/01/28-20:31 -AT
Material received from FTP Server and stored in ECuRep Directory: /ecurep/pmr/5/5/00062.239.624
File: 00062.239.624.providerTrace[x].log 70378492 bytes
Go to https://ecurep.mainz.de.ibm.com/aex/ and type in you PMR number (separated with commas like in Retain)

Note:
Note: Pictures are for a snap but it is the same procedures for the .log files.

Click on execute.
In the next window you will find the uploaded file. Select the file(s) you want to transfer.

Along with other options like compressing or extracting you have the possibility to transfer the file via FTP. Select
this option and click on execute.
94 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

In the next window set the Destination host to SVCSUPPORT.SSD.HURSLEY.IBM.COM and the directory
to /2145/in/PMR (separated by dots instead of commas)
Folder00062.239.624 will be created if it is not there.
User is anonymous and password is your email-address.
Mode has to be set to binary.

For big files ( 200MB and more) use Parallel.


If Execution was on "Synchronous" you will get following message, when update is ready.
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 95

FINAL STEPS for IBMers


Log in to the Hursley dumpdecoder:
http://dumpdecoder.ssd.hursley.ibm.com/cgi-bin/DumpDecoder/DD_main.pl
Frontend members should have access. Please inform us (Level 2) when not!
Press the "ADD PMR" button on the bottom of the left pane.
96 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Fill in the fields EXACTLY as you did when you FTPed the files.
Click on Add PMR button
This automatically opens a new window like:

Click on "Add File" on the bottom of the right Pane

This opens the folder where you uploaded the files using FTP
Hit "Get" (If you used a different folder in the FTP part you can change it here)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 97

Wait a while (depending on the amount of data between 1 and 5 minutes)


Use "refresh" in your browser and you will get something like:

Now you can have a look in the error logs to see if these are readable.

SVC performance data collection by using SVC Stats


There are 2 different methods to generate SVC performance data

- SVC Stats :
Every SVC code 3.1+ include this option.
SVC generates statistics files at a specific interval
for SVC 3.1+ (between 15 and 60 minutes)
for SVC 4.1+ (between 1 and 60 minutes)

- TPC :
TPC allows this interval to be set as low as 5 minutes.
(please goto chapter "SVC performance data collection by using TPC")
Customer needs the TPC standard edition. This edition includes the performance measurement (TPC for Disk.) The
SSPC has the TPC basic Edition only ; SSPC performance bundle is required.
98 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

- more details are available in SVC WIKI under


http://svcsupport.ssd.hursley.ibm.com/wiki/index.php/Performance/Data_Collection_Summary
http://svcsupport.ssd.hursley.ibm.com/wiki/index.php/Performance/Data_Collection_Guidelines#Not_Enough_Data
1.SVC performance data collection with 'SVC Stats'
Each interval, SVC generates 3 files per SVC node, named as follows:
Nv_stats_nnnnnn_dddddd_tttttt
Nm_stats_nnnnnn_dddddd_tttttt
Nn_stats_nnnnnn_dddddd_tttttt
Where nnnnnn is the 6 digit node front panel ID, dddddd is the date on which the file was generated, and tttttt is the
time at which the file was generated. Nv_stats files contain data for Virtual Disks (vdisks), Nm_stats files data for
Managed Disks (mdisks), and Nn_stats files data for Nodes and Node Ports.
Using SVC GUI (See below for CLI procedure)

maximum intervals are 16 in this case


manual start and manual stop is required
Procedure:
- open SVC GUI
- start Statistics Collection
- type in the interval (1 to 60 minutes ; V4.1+)
- stop Statistics Collection ; if 16 intervals are reached
- open CLI for this cluster (PUTTY)
- type command 'svc_snap stats'
output e.g. snap.007270.080520.141145.tgz
- customer send snap file to EcuRep (WEB-Tool)
http://www.ecurep.ibm.com/app/upload
if customer use FTP commands, file must rename in e.g.
00062.239.624. snap.007270.080520.141145.tgz
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 99

-> 00062.239.624 is an example for a PMS number


- 'tgz' file must be unpack and pack as 'zip'
- upload snap.007270.080520.141145.zip in
'Performance Analysis Tool'
link to 'Performance Analysis Tool' is available under
http://svcsupport.ssd.hursley.ibm.com/
Using SVC script (See above for GUI procedure)
maximum intervals are not define but 99 are more as enough
manual start and automatic stop after final interval
Preparation
- download 'stats collection tool' from SVC Support Tools page
http://svcsupport.ssd.hursley.ibm.com/

- send this tool to customer


- customer must unpack and install this tool on SVC Master Console for
example in new dir 'io_stats'
100 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

- after unpack, stats.bat and help files are in dir


C:\io_stats\StatDumpScriptExternal\StatDumpScriptExternal

configuration
- open a dos prompt
and change directory where the 'stats.bat' file is stored
cd\
cd C:\io_stats\StatDumpScriptExternal\StatDumpScriptExternal

- start the statistic scrip with following string as example:


stats.bat /I 1 /L 20 /D 9.155.86.33 "C:\Program Files\IBM\svcconsole\cimom\icat.ppk" collection
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 101

/I 1 1 = every 1 minute
Sets a user defined stats collection interval in minutes
nn Specifies the interval at which statistics should be collected
Must be an integer
defaults to 15
/L 20 20 = 20 measurements
Specifies how many sets of files the script should collect before exiting
e.g./L 4 will normally result in the script running for 45 minutes
/L 10 /I 30 will normally result in the script running for 5 hours
/L 1 /I 20 /D will normally result in the script running for 20 minutes
/D Specifies that stats files should be deleted from the cluster before
9.155.86.33 -> Cluster IP-Address
C:\Program Files\IBM\svcconsole\cimom\icat.ppk -> default directory of icat.ppk
collection -> name of directory, which the statistic data will be stored

- following example with I=3 L=2 show, how does it work

- the data collection is stored in


C:\io_stats\StatDumpScriptExternal\StatDumpScriptExternal\collection\stats

you will find all performance data and SVC configuration files
- zip all files in this directory e.g.
as 00062.239.624. io_stats.080520.zip
• 00062.239.624 is an example for a PMS number
102 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

• 080520 is the actual date (YYMMDD)


- customer send zip file to EcuRep (WEB-Tool)
http://www.ecurep.ibm.com/app/upload
- upload 00062.239.624. io_stats.080520.zip in
'Performance Analysis Tool'
link to 'Performance Analysis Tool' is available under http://svcsupport.ssd.hursley.ibm.com/

SVC performance data collection by using TPC


Customers with TPC installed, can - if they have licensed TPC for Disk - provide TPC .csv performance data ( csv =
comma separated values ) for an SVC performance analysis to SVC
L2 and L3 support teams.
Customer needs to be aware that some pre-requirements need to be fulfilled, before TPC
can successfully create performance reports:
1. TPC for Disk has to be installed
2. CIM Agents have been installed for switches and storage subsystems
3. CIM Agents have been discovered by TPC
4. Performance Monitors for collecting performance metrics have been defined
5. Performance Monitors have successfully put performance data into the TPC database
6. TPC Server needs to be accessible
For further questions on how to setup TPC for performance, please see the
TPC infocenter: http://publib.boulder.ibm.com/infocenter/tivihelp/v4r1/index.jsp
or SVC Redbook: http://www.redbooks.ibm.com/redbooks/pdfs/sg247521.pdf
or contact TPC support!
Important tips:
Tip #1: check if you have performance measurement licensed

Check if you have licensed Performance by going to :


IBM TPC
• Configuration Utility
• Disk Manager
• Subsystem Performance Monitor

If this branch exists, you can make use of the TPC performance measurement functionality !
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 103

Tip #2 : check monitor probe configuration


• SVC cluster monitoring should be setup for all SVC clusters in a customer environment,
but in separate monitor probes, which allows collecting .csv performance data for dedicated
SVC clusters - and not for all SVC clusters in common, check this by going to :
IBM TPC
• Monitoring
• Probes
• administrator.your_SVC_probe_name

Tip #3 : check for successful probing


• After having setup the probes for each separate cluster, probing to these clusters
needs to take place and needs to finish successfully as well !
Run the probe job by selecting the specific probe and then right-mouse-click to execute
the "Run Now" function :

Successful probes finish with a green rectangle , while failing probes have a red circle !
104 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Tip #4 : create and run a subsystem performance monitor and set data sample rate
• Each storage subsystem which should provide csv performance data needs to have an appropriate subsystem
performance monitor defined, go to :
IBM TPC
• Disk Manager
• Monitoring
• Subsystem Performance Monitors
and first create the appropriate performance monitor for your specific subsystem:

After having created the performance monitor, check the "Sampling and Scheduling" tab
and define the sampling rate as needed :
Shortest interval length is 5 minutes ( could be 10, 15, 20, 30 or 60 min ).
Leave it at default ( = 5 min ) unless advised by IBM support !
Also : select "Continue indefinitely"
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 105

Now TPC continuously collects performance data for the defined subsystem(s) and
can be used to create performance data csv files !
Tip #6 : check if subsystem pre-reqs are fulfilled
• Check if each storage subsystems fulfil the pre-reqs for performance monitoring here :
IBM TPC
• Configuration Utility
• Disk Manager tab
• Enabled for Performance = =yes=

1.2.1 Creating TPC .csv performance data files


- Check that the cluster which you would like to collect performance
data for is in "normal" state, go to :
IBM TPC
• Disk Manager
• Storage Subsystems
106 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

- SVC L2 and L3 support need SVC specific performance reports, means that for a single SVC cluster the customer
needs to provide a set of 6 different single .csv files with performance data :
1. By Storage Subsystem
2. By Node
3. By IO Group
4. By Managed Disk Group
5. By Managed Disk
6. By Volume
Find the different performance reports in the TPC navigation tree by going to:
IBM TPC
• Disk Manager
• Reporting
• Storage Subsystem Performance

- Select the performance report you would like to generate .csv performance data for by selecting the appropriate
report group, for example "By Node" :
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 107

- In TPC, a right hand pane opens with a subset of performance metrics listed in the right "Included Columns",
means that ALL performance metrics will be included in the to be generated .csv performance data file.
Unless you have special reasons, please leave all performance metrics there!

- Click on the "Selection" button to restrict the report to a single box

Don't do this for report types "ByMdisk" and "ByVolume" as the selection panel takes a very long time to update the
view if there are many entries ( like a bunch of mdisks or vdisks ) !!!
- Next important choice is the "Display Historic Performance Data" section. It allows to set the timeframe for
which the performance data for the previously choosen sub-system will be collected and reported ( selecting "Display
latest performance Data" will only report performance data of the very last single sample interval )
108 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Hint #1 :
Pulling 24 hours of data usually makes sense. If there are many, many volumes, a 24 hour report may be too large to
handle, then try 12 hour or 8 hour selections.
If it is a re-occurring performance impact, placing the worst phase of this impact into the middle of the collection
phase is preferable.
Hint #2 :
There is one tricky aspect to the Date/Time selection. If you select several days, and if you select "From" 8-AM and
"To" 5-PM, you will get only the data between 8 and 5 for each day. If you expect all the data from several days, then
the "From" and "To" times need to be the same, for example if you want all the data from Nov 9 to Nov 11, you
could use this time range ( starting Nov 8 at 23:59, ending at Nov 11 23:59 ) :

- Now select the "Generate Report" button and receive a multi-line multi-sample performance report within the TPC
application : a new tab opens parallel to the
"Selection" tab, here a tab named "Node" as the "By Node" report had been chosen earlier ( reports like "By
Volume" can be quite large ) :
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 109

- Next this table view needs to be saved, therefore select File > Print and print to a .csv file with "Include
Headers" but without "Include Totals" :

- Press "Continue" and save the csv file with a meaningful name :
110 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

Example :
ppppp.bbb.ccc_SvcCluster_ByX_yymmdd.hhmm1-yymmdd.hhmm2.csv

ppppp.bbb.ccc > your pmr number


SvcCluster > name of your SVC cluster
ByX > "X" = "Node", "IoGrp", "MdiskGrp", "Mdisk", etc
yymmdd.hhmm1 > year-month-day-hour-min of the sampling start time
yymmdd.hhmm2 > year-month-day-hour-min of the sampling finish time
> Do this for all the 6 different csv performance data files !!!
- In an notepad editor, the resulting csv performance file should show up like this :

- zip all csv's for a single SVC cluster into an archive an upload to Ecurep
( http://www.ecurep.ibm.com/app/upload ), again in a meaningful format :
ppppp.bbb.ccc_SvcCluster_CSV_yymmdd.hhmm1-yymmdd.hhmm2.zip
For more details of EcuRep see chapter ECuRep procedure on page 8
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 111

4146 (SAN-FS)

For SAN-FS higher Version2 Release1 (V2R1):


a) Metadata Server (Master & Subordinate)
1. run /usr/tank/server/bin/ obdc on the master metadata server and subordinate servers.
This script generates a directory that contains logs and configuration files used for debugging SAN FS issues. Please
this and send it to Datacase for analysis as described in section >PFE datacase<
b) SAN.FS clients
1. On an UNIX SAN FS client run /usr/tank/client/bin/obdc. This script generates a directory that contains logs and
configuration files used for debugging SAN FS issues. Please zip this directory and send it to Datacase for analysis as
described in section >PFE datacase<
2. On a Win2k SAN FS client do the following
a. Open a command window and change directory to C:\Program Files\IBM\Storage Tank\client\bin
b. Run "obdc.exe". This script generates a directory that contains logs and configuration files used for debugging
SAN FS issues. Please zip this directory and send it to Datacase for analysis as described in section >PFE datacase<
If files that are too big to be sent via email (more than 10 Mb) , please ftp them to ECuRep see ECuRep procedure
on page 8 for detailed description.
In addition, please provide
• A complete SAN layout as described in section Additional information required for data collection on page 11
• detailed history of the problem giving dates and times when known or estimated
• support actions already taken
• time difference between SFSs, hosts, SAN, ...

For SAN-FS lower Version2 Release1 (V2R1)


a) Metadata Server (Master & Subordinate)
1. run /usr/tank/server/bin/pmf.sh on the master metadata server and subordinate servers.
This script generates a directory that contains logs and configuration files used for debugging SAN FS issues. Please
zip this directory and send it to Datacase for analysis as described in section >PFE datacase<
NOTE: Step 2 below is onlyfor customers with a SAN FS version below 1.1.1-4. You can check for the version by
looking at the VERSION file located in /usr/tank/server directory on the Master Metadata server.
2. run the following commands on the master metadata server and the subordinate servers. All of these commands
are run in the command line of the system OS. Copy the output of each command to a file and send it to Datacase for
analysis as described in section >PFE datacase<
tanktool lspool -l
tanktool lsproc -l
tanktool lsserver -l
tanktool lsvol -l
tanktool lsadmuser -l
tanktool lsclient -l
tanktool lscontainer -l
tanktool lsengine -l
112 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components

tanktool lslun -l
tanktool lsdrfile -l
tanktool lsimage -l
tanktool lspolicy -l
tanktool lssnmpmgr -l
tanktool lstrapsetting -l
tanktool lsautorestart -l
3. Capture the RSA configuration through the RSA web gui (http://RSA_ip_address). In the web gui on the left panel
click on "Configuration File" or "Server->ASM Control->Configuration File". Then on the right panel click on "view
the current configuration summary".
If you click on this link, a new browser window opens with the configuration content. Copy the content into a text file
and send it to Datacase for analysis as described in section >PFE datacase<
b) SAN.FS clients
1. On an UNIX SAN FS client run /usr/tank/client/bin/pmf.sh. This script generates a directory that contains logs and
configuration files used for debugging SAN FS issues. Please zip this directory and send it to Datacase for analysis as
described in section >PFE datacase<
2. On a Win2k SAN FS client do the following
a. Open a command window and change directory to C:\ProgramFiles\IBM\StorageTank\Client\bin
b. Run "pmf.bat". This script generates a directory that contains logs and configuration files used for debugging SAN
FS issues. Please zip this directory and send it to Datacase for analysis as described in section >PFE datacase<
If files that are too big to be sent via email (more than 10 Mb) , please ftp them to ECuRep see ECuRep procedure
on page 8for detailed description.
In addition, please provide
• A complete SAN layout as described in section Additional information required for data collection on page 11
• detailed history of the problem giving dates and times when known or estimated
• support actions already taken
• time difference between SFSs, hosts, SAN,
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 113

Data collection for Storage components

2105 and 2107 (DS8x00)


Data collection Document for ESS and follow up products :
http://ssgtech3.tucson.ibm.com/S96A/DS8000%20PE%20Field
%20Tips.nsf/1f3905fe337e825007257275005fde4b/51b8f08e8e088905072574eb0063bd6a?OpenDocument

IBM 2106 Modular Storage Server


Send the following files:
/kernel/drv/sd.conf
/kernel/drv/fca.conf
/kernel/drv/fcaw.conf
/kernel/drv/fc-pci.conf
To DATACASE as described in section >PFE datacase<
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Open the Command Line Interface (CLI) and run the following commands. Capture the output in Notepad:
> SHOW THIS FULL
> SHOW OTHER FULL
> SHOW STORAGESETS FULL
> SHOW UNITS FULL
> SHOW DEVICES FULL
> SHOW CONNECTIONS

Run FMU on each controller and issue the command:


> SHOW LAST ALL FULL
Send all outputs to DATACASE as described in section >PFE datacase<
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

All IBM DS4000 and DS5000 models

Performance problems
In case of a performance problem, please provide the answers to the following questions:
• Provide a detailed description of the performance problem and the business impact!
114 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

• Provide a description of the whole solution architecture (Application - Server - SAN - Storage), application
characteristics.
• Which DS4000 volumes/ hosts show the performance problems? Please provide a description of the logical and
physical volume layout (vol. mapping server to storage)
• When was the first occurrence of the problem?
What changed in the environment at that time?
• What is the typical occurrence of the problem during daily business hours?
• What facts do indicate that the performance problem is related to the storage subsystem?
• Does the customer require high IOPS or throughput (MB/s)?
• How and where does the customer measure performance? Please provide any available data that shows the
performance problem and reference data that shows good performance.
• What is the criteria for the problem to be considered as solved?

Storage manager 8.x and lower.


Storage Manager Version
To find that out:
1- Go to Start >> Programs >> IBM DS4000/FAStT Storage Manager Client
2- The Enterprise Management Window appears.
3- Go to Help >> about, the storage Manager Version will be displayed.

Storage Subsystem Profile File


How to View and Save a Storage Subsystem Profile to a text file:
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 115

To save the Event Log from the Subsystem Management Window, do the following:
1.Select View>Storage Subsystem profile

2.The profile file window appears then:


116 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

3.Then click on Save As and save this file please in text format (PMH#.txt)
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 117

MEL (Major Event Log) file


To save the Event Log from the Subsystem Management Window, do of the following:
1- select View>Storage Subsystem Event Log
118 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

The Event Log file window appears then:

2-Make sure that you "de-select view only Critical Events", and enter 8000 in retrieve most recent events and select
update
3- then click on Select All and then Save As ( please don't forget to de-select "view only critical events")
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 119

Send all files and screen shots to DATACASE as described in section >PFE datacase<
also the complete SAN Layout as described in section Additional information required for data collection on page 11

Storage Manager 9.x and higher


In SM9.1 there is a new feature called collect all support data. With this function a zip file is created which includes
the following files:

driveDiagnosticData.txt
majorEventLog.txt
NVSRAMdata.txt
objectBundle
performanceStatistics.csv
persistentReservations.txt
readLinkStatus.csv
recoveryGuruProcedures.html
recoveryProfile.csv
stateCaptureData.dmp
storageSubsystemProfile.txt
unreadableSectors.txt
To collect the information open the Storage Manager and select Advanced -> Troubleshooting -> Collect All Support
Data
120 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

After that specify the name of the zip file and press the Start button.
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 121

Send all files and screen shots to DATACASE as described in section >PFE datacase<
also the complete SAN Layout as described in section Additional information required for data collection on page 11
It is possible that the following additional actionplan will be asked in the PMH

First additional action plan


1-Use a Server, workstation, or laptop installed with Windows 95, 98, 2000, or XP.
2-Connect it to the DS4000/FAStT controller using a Null-Modem Cable
Hyper Terminal Session Parameters:
1-COM Port = COM1
2-Baud Rate = 9600 or higher
3-Data Bits = 8
4-Stop Bits = 1
5-Parity = None
6-Flow Control = Xon/Xoff

Enter the following commands:


> ld</Debug
> arrayPrintSummary
> cfgPhyList
> cfgUnitList
> fcDevs 2
> fcDevs 4
> fcAll
> vdAll vdShow
> unld Debug

Capture the results and send it to DATACASE as described in section >PFE datacase<

Second additional action plan


Instructions on how to run Read Link Status Diagnostics
In IBM DS4000/FAStT Storage Manager 8 Subsystem Management window select the menu Storage Subsystem
and click on Run Read Link Status Diagnostics
122 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

The Read Link Status Diagnostic window will appear

Click Set Baseline and confirm twice.


Generate workload and run I/O on the DS4000/FAStT Storage Subsystem for at least 2 hours. After that time
open up the Read Link Status Diagnostic window again.
Click Run
Click Save As
This will open another window in which you can save the Read Link Status Diagnostics output.
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 123

Save the Read Link Status Diagnostics output to a file with the extension csv
Capture the results and send it to DATACASE as described in section >PFE datacase<

Third additional action plan


Capture State Information:
Please use this option if you have been asked from Support team:
To save the Event Log from the Subsystem Management Window, do of the following:
124 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

select Advanced> CaptureState Information

Specify the file name where to save the output:


IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 125

Select start, so you will get this warning message, please type yes, this warning is just
To tell you that this option can be performed only via Support recommendation
Capture the results and send it to DATACASE as described in section >PFE datacase<

Fourth additional action plan


Text to add
IBM Remote Support Manager (RSM) for Storage
The RSM does Manage and analyze alert notifications for DS3-5K. In case of problems with RSM, please gather
RSM support data by using one of the following 3 methods:
1. On RSM server command line, enter the command
rsm-edump <email_address>
The program rsm-edump collects information about the RSM system and operation of RSM software and compresses
all of the files into a single ZIP file which is stored as /tmp/rsm/host_name.zip
The program takes an email address as an optional argument.
2. From the RSM GUI Browser interface - admin user on "Statistics and Logs page" - lservice user on
"Support and Debug page"
126 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

3. From the remote dialin service menus (rsm-service) select "Support and Debug page"

DS3000 Data collection with Storage Manager 2.x / 9.x


In SM2.x / SM9.x there is a feature called collect all support data. With this function a zip file is created which
includes the following files:
NVSRAMdata.txt
driveDiagnosticData.bin
featureBundle.txt
majorEventLog.txt.mel.txt
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 127

objectBundle.htm
performanceStatistics.csv
persistentReservations.txt
recoveryGuruProcedures.html
recoveryProfile.csv
sasPhyErrorLogs.csv
socStatistics.csv
stateCaptureData.txt.ss.txt
storageArrayConfiguration.cfg
storageArrayProfile.txt.profile.txt
unreadableSectors.txt
To collect the information open the Storage Manager and select Support -> Gather Support Information (Collect All
Support Data)

After that specify the name of the zip file and press the Start button.
128 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

Send all files and screen shots to DATACASE as described in section >PFE datacase<

DS300 and DS400


DATA COLLECTION FOR ALL DS300 DS400 MODELS
IBM ServeRAID Manager 7.2XX
Storage administration is managed by IBM ServeRAID Manager. The same management software used with xSeries
ServeRAID adapters.
You will now be required to install both a management agent and the client GUI to manage the enclosures. All of the
event logging is done on the local management station.
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 129

Obtain the support archive from IBM ServeRAID Manager.


• From ServeRAID Manager right click on desired enclosure
• Select "Save support archive" this saves:
RaidEvt.log (the event log)
RaidErr.log (the error log)
RaidCfg.log (the Subsystem cofig file)
diagnostics.tgz (copressed file containing binary and text files for engineering)
You can also use the controller CLI.
The CLI is embedded into the controller
• Connect to controller using telnet port 23 or the serial connection in the back of the controller.
• Capture result of "show all" command

DS6000
Datacollection via DSCLI

Note: if the codelevel of the system is below 6.2.2.49, the pepackage may not contain all necessary information for
problem determination on all cases, additional data gathering might be required

From DSCLI window of your SMC issue the DSCLI command:


mkpe -noftp IBM.1750-68XXXXX (XXXXX = your serialnumber)
130 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

force a statesave if required by issuing from DSCLI:


diagsi -action warmstart IBM.1750-68XXXXX
Wait 20 minutes until the statesaves are ready to offload and continue:
From your DSCLI window issue the command:
offloadss -noftp IBM.1750-68XXXXX
Data transfer methods:

Note:
- it is required to transfer pepackages and statesaves to testcase
- it is not sufficient to transfer pepackages to the Austin Decoder only
- do not transfer statesaves to the Austin Decoder - this is useless

Testcase
open a DOS window.
go to the directory you put your windows zip file in
> ftp testcase.software.ibm.com
Login as anonymous
use your email address as password
> bin

> cd/ssd/toibm/sharkdumps
optional: enter prompt off (allows the whole transfer to take place without any prompt for the next file)
> mput *.zip
Create a directory at http://fieldsupport.austin.ibm.com/cgi-bin/pfe/pfe.pl and download the file to it
EcuRep
You can also use EcuRep (See chapter ECuRep procedure on page 8 for details)

NAS General
• What exact symptoms are you experiencing?
• What error or failure indications do you see (messages, beeps, error codes, lights)?
• How often does the problem happen?
• Under what conditions does the problem occur?
• What was happening at the time the problem occurred?
• Has this configuration worked previously? If so, when did it last work and what has changed since then?
• Is this a new install? Upgrade? Configuration change?
• Is the problem re-creatable? What steps do you follow to cause it to occur?
• How did you recover from the problem and what specific actions did you take to recover?
• Do you have a work-around that prevents the problem or reduces its frequency?
• Have you run product diagnostics and what were the results?
• Do the diagnostics indicate which specific component has a problem?
• Check www.ibm.com/storage/support/nas IBM NAS Support home page
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 131

IBM 5190 Total Storage NAS 100 (Obsolete due to EOS 1-1-2007)
I)For data collection use the DSA tool See section DSA (Dynamic System Analysis) for procedure
Connect to the NAS100 over terminal services
1. Install the e-gatherer2 executable in the directory : c:\temp\ egather2.exe
2. Create the following batch file with two lines, called 'egatherer.bat' on the NAS100 desktop (this will call up
egatherer with the option -batch , without the disclaimer screen popping up asking for Y/N input from the keyboard) :

> call c:\temp\egather2.exe -batch


> /wait egather2.exe

3. Set up the windows scheduler to start the batch-file above by performing the following steps :
• To open Task Scheduler, click Start,
• point to Settings, click Control Panel,
• and then double-click Scheduled Tasks.
• The scheduled task wizard will appear.
• Click on 'browse' and locate the batch- file 'egather.bat' (located at : My Computer - Desktop ) , Click on the
Batch-File to select ,
• click on Open
• One time only , Next
• Check the Start-Time : Set to 5 minutes from now.
• Check the start date : Set to today (default)
• Select : Next
• Enter password for the machine
• Click on 'Finish'
4. E-Gatherer Program will be started at the time selected , (files egathbb8.tmp and gather.dll will appear on the
desktop at the time of execution, and then disappear again) and the output file IBM-...-... (should have the correct
machine-type and -id in the file name) will be created in the C:\temp directory.
5. Send the output file to ECuRep or EMEASAN@ DE.IBM.COM
6. Provide also the complete SAN Layout as described in section Additional information required for data collection
on page 11
II) To get the current software version for an IBM NAS Appliance, perform the following steps:
• Open up the IBM NASAdmin MMC Console
• Click on Software Version in the left-hand panel
The Software Version with a Build number as well as some additional information will be displayed in the right-hand
panel
Currently the machine BIOS for an IBM NAS 100 Appliance, can only be access through the serial port. The access
the BIOS information, perform the following steps:
• Power off your appliance.
• Connect the COM port of the PC to the serial port of the appliance using a 9-pin serial cable (RS-232C female
crossover cable). Connect via HyperTerminal and select the following values in the Properties window ( 115200,
8N1 , hardware flow control).
• Power on the appliance.
• BIOS POST messages should appear on the HyperTerminal screen. When the message Press F1 to enter setup
opens, press F1.
• The message Enter current password opens. Type the password and press Enter.
• Note: The default password is 001san (not case-sensitive).
• The BIOS setup window opens.
• Go to the Advance menu tab and press Enter
132 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

• Use the arrow keys to move the cursor to VPD Data Configuration.
• The VPD Data Configuration window opens, displaying the following VPD data:
• BIOS build date
• BIOS ID
• Machine type
• Model name
III) The IBM NAS 100 Appliance has three types of LEDs to help you identify problems with hardware failure, some
software errors and system status. Two types of LEDs are located on the front bezel and the third type is located on
the back of the chassis. Please refer to the Troubleshooting chapter of the Installation and Service Guide for detailed
information on all of these items.
1. System status LEDs are located on the operator information panel near the power button.
2. Hard disk drive status LEDs are located along the bottom of the bazel
3. Ethernet port status LEDs are located on top of the Ethernet ports (LAN 1 and LAN 2)
Perform the following steps to access hardware errors in the Windows event log by means of Terminal Services:
1. From the Start menu in the PC task bar, click Program -> Administrator Tools -> Event Viewer.
2. From the Tree panel on the left, click Application Log.
3. Click View -> Filter.
4. Type NAS100Svc in the Event Source field and click OK. The Windows event log opens.
Hardware status can also be viewed using the WEB GUI:
1. Access the NAS 100 using the following URL : http://x.x.x.x(NAS100 IP address):8099
2. Log in as Administrator and NAS100 password
3. Go to the Status tab to view current? status
4. To view Event Logs go to Maintenance -> Logs -> (system, application, security...)
To view error messages that occurred during POST, you need to connect to the NAS 100 via :
1. A LAPTOP or desk top using a Serial, HyperTerminal (115200, 8N1, hardware flow control)
2. Power on the Machine and press the F1 key to enter Setup
3. The message Enter current password opens. Type the password and press Enter. Note: The default password is
001san (not case-sensitive).
4. The BIOS setup window opens. Go to the Advance Tab
5. Press the down arrow to the Event Log Configuration
6. Go to View Event Log and press enter
IV) Additional questions:
1. Which kind of ethernet port is used (Static IP or DHCP)
2. Does the NAS 100 respond to the ping command initiated on the Terminal services
3. While connected to the RS232 port,? what messages are reported during power on.
4. What are the Qfixes levels of the applications on the NAS 100
5. The software infos reported on IBM NAS admin MMC :
• click on software version in left hand panel and provide the software version with build number in the right
side panel
V) Send the output to DATACASEas described in section >PFE datacase<
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

IBM 5194 Total Storage NAS 200, 201, 225 and 226 (Obsolete due to EOS
1-1-2008)
• Software Version
To get the current software version for a 5194, perform the following steps:
• Open the IBM NAS Admin MMC Console
• Click on Software Version in the left-hand panel
• The Software Version with a Build number will be displayed in the right side panel.
• Machine BIOS Version
To get the current machine BIOS for a 5194, perform the following steps:
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 133

• Open the IBM NAS Admin MMC Console


• Click on Maintenance -> System Information ->System Summary to view BIOS version.
• Serve Raid Manager BIOS
To get the current Server Raid Manager BIOS, perform the following steps:
• Open the IBM NAS Admin MMC Console
• Click on Storage -> Serve Raid Manager. Double click the Serve Raid Manager program icon.
• The Serve Raid Manager window will appear, and then click on Controller 1, the right side panel will present the
BIOS version.
• Ethernet Adapter Driver
To get the current ethernet adapter(s) driver level, perform the following steps:
• On the Windows Task Bar, click START -> Settings -> Control Panel -> Intel (R) PROSet II
• In the left window panel of Intel PROSet II, highlight the ethernet adapter.
• In the right window panel, click 'Network Driver' tab to view the driver name and version.
• There are several ways to communicate with the 5194. If you have no ethernet connectivity to the machine, try
these:
• Console - plug in a keyboard, mouse, and monitor into the 5194.
• ASM service port - plug in a standard serial cable into the management port and you can view some information
about the 5194
If you do have connectivity to the machine, any of these should work:
• IBM Advanced Appliance Configuration Utility
• UM Services (http://machine-name:1411)
• IBM Web-based GUI (http://machine-name:8099)
• Terminal Services Client
For data collection use the DSA tool See section DSA (Dynamic System Analysis) for the procedure
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

IBM 5195 Total Storage NAS 300, 301, 325 and 326 (EOS 1-1-2011)
• Software Version
To get the current software version for a 5195, perform the following steps:
• Open the IBM NAS Admin MMC Console
• Click on Software Version in the left-hand panel
• The Software Version with a Build number will be displayed in the right side panel.
• There are several ways to communicate with the 5194. If you have no ethernet connectivity to the machine, try
these:
• Console - plug in a keyboard, mouse, and monitor into the 5195.
• ASM service port - plug in a standard serial cable into the management port and you can view some information
about the 5195
If you do haveconnectivity to the machine, any of these should work:
• IBM Advanced Appliance Configuration Utility
• UM Services (http://machine-name:1411)
• IBM Web-based GUI (http://machine-name:8099)
• Terminal Services Client
For data collection use the DSA tool See section DSA (Dynamic System Analysis) for the procedure
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

IBM 5196 Total Storage NAS 300G G00 and G25 (EOS 1-1-2011)
• Software Version
To get the current software version for a 5196, perform the following steps:
• Open the IBM NAS Admin MMC Console
134 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

• Click on Software Version in the left-hand panel


• The Software Version with a Build number will be displayed in the right side panel.
• There are several ways to communicate with the 5196. If you have no ethernet connectivity to the machine, try
these:
• Console - plug in a keyboard, mouse, and monitor into the 5196.
• ASM service port - plug in a standard serial cable into the management port and you can view some information
about the 5196
If you do have connectivity to the machine, any of these should work:
• IBM Advanced Applianc Configuration Utility
• UM Services (http://machine-name:1411)
• IBM Web-based GUI (http://machine-name:8099)
• Terminal Services Client
For data collection use the DSA tool See section DSA (Dynamic System Analysis) for the procedure
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

IBM 5198 Total Storage NAS 500 (EOS 1-1-2011)


Make sure all file-names indicate the origin, please add pmrnumber.country,branchoffice.(nodenumber.) , preceding
the file name.
For a clustered system the information listed below is required for both nodes.
Again, please rename all files accordingly, to indicate their origin.

Please collect the following data :

Software level :

To identify the level of installed NAS software, provide the output of the commmands :
> Bldlevel
will show the build level of the software installed
>Emgr
will show all e-fixes(if any) installed on the NAS500 in addition to the base build.

NAS configuration :

Please provide the following files:


/.websm.script
/logs/nascfglog

Core dump :

To find a core dump, if one is present, issue the following command :


> /usr/samples/findcore/corepath
To collect the core dump if found , issue :
> /snapcore -d <with coredump name specified>

General data collection (snap) :


IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 135

For a single node system :


Login as root
Run the following commands :
snap -r
snap -ac
This generates an output file Snap.pax.Z in the default directory: /tmp/ibmsupt/ (size approx. 7 mB)
Please rename to pmhnumber.bo.country.Snap.pax.Z

For a dual node system (HACMP cluster) :


Login as root
Run the following commands on BOTH nodes :
> snap -r
> snap -a
> snap -e
> snap -c
This generates an output file Snap.pax.Z in the default directory :/tmp/ibmsupt/ (size approx. 27 mB)
Please also provide the following file, which is not being picked up by the snap command :
/logs/cluster/nascluster.log

CIFS related problems :


Please provide the following information :
(a) Output of the cifsSnap command :
Login as root, change directory to /tmp/ibmsupt, issue (where pmr, country, and branch are the appropriate numbers):
cifsSnap [-U] . pmr.country.branch

This will create a compressed file "pmr.country.branch.currentdate.currenttime.tar.Z" in the current directory ( the
first '.' is required to address the current directory as the target, or pmrnumber.branchnumber will be taken for a
directory to be created, and a file currentdate.currenttime.tar.Z will be created in that directory).

The "-U" option gathers userdata , this can take some time.

Other Vendor Software Products :


If other vendor products are involved with the issue, need to collect those specific logs as well.

LAN and SAN Layout :


Provide drawings of the LAN and SAN Layout.

How to send the data :


Send the output files to DATACASEas described in section >PFE datacase<
or
FTP : Use ECuRep as described in section ECuRep procedure on page 8
136 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

XIV
x-ray is the tool used by the XIV storage controller to collect system and debug data.
This is similar to the DS6000 "pe_package" collection.
A typical x-ray file name is: system_xray_2810A14MN00043_2008-08-17-1359.tar.bz2

How to run x-ray:


1. via xcli
Run via xcli e.g.: C:\Program Files\XIV\GUI10>xcli -m 9.155.56.101 system_logs_collect
You can use either the machine name or a IP address of one of the three management modules.
2. java utility
Easier way to collect a xray package is to go to the site (IBM intranet password protected):
https://steamboat.boulder.ibm.com/webapp/iwm/int/reg/pick.do?source=IIPxiv
And look for XIV utilities at the bottom,

Tick the "I AGREE"


Hit "I CONFIRM"
Here you will find the x-ray collector.
This tool can be given to customers
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 137

Usage:
Copy xray_collect.exe on a windows machine which can reach the XIV management address.
Open a command-line window and go to the directory where the xray_collect.exe is located and invoke e.g:
xray_collect.exe 9.155.56.101
The xray package will be copied into the directory where you started the utility from.
Upload the x-ray file to PFE datacase as described in >PFE datacase<

IBM 4125 Total Storage IP Storage (all models) (Obsolete due to EOS
1-1-2009)
From the main svc prompt issue the "senddebug" command. This will collect logs and other debug information into a
single tar file that can be sent for analysis.
Target Configuration
• initiator to vlun assignment
• vlun to lun assignment
• lun to disk assignment
• physical disk configuration - expansion boxes used must be specified.
Gigabit Card used (copper or fibre)
Message Log including an oops message if any.
Number of initiators
Initiator Configuration (See below)
Sequence of actions in the initiator(s) that caused the problem.
If an application was used, version number is very important.
138 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

If customer can, he/she should attempt to recreate with debug level 3 and report the logs. I'm assuming here that the
debug level can be set from the GUI, if not serial port.
Target GUI issues
Code Version No (I assume its 1.0 but if a patch is added, then the patched version becomes important)
Enable Java Console, console log
Admin stdout
Admin stderr
Initiator: Windows
1. Description of problem, including how to reproduce
2. Description of machine: make/model/type and number of processors is most important
3. Brand/type of ethernet card used, along with any special settings (jumbo frames?)
4. Description of iSCSI configuration
5. Version number of iSCSI client
6. Any IBM iSCSI messages from the event viewer, or any error messages that appear on the screen
Initiator: Linux
1. Level of the IBM iSCSI Linux Client you are using - check the directory name.
2. PC Type and Model, BIOS version/date
3. IBM only supports Red Hat Linux using kernel 2.2.19-6.2.1 with the current iSCSI Linux initiator. If they do not
have this distribution and kernel, send them to http://www.ibm.com/developerworks/
4. Execute ./iscsidebug.sh to collect machine information
5. Step by step procedure to reproduce the problem. Please include as many details as possible.
6. Re-compile the driver with debug messages turned on.
If you are on a uni-processor machine, you need to create the following directory tree: /tmp/iscsi/trace. (use the
mkdir command)
Please remove the file /var/log/messages or at least use the mv command and give it a different filename.
Reboot your machine.
Edit the Makefile and change debug=n to debug=y.
Type make clean and make.
Copy the new .o file to your 'ibmiscsi-1.1.0' directory
Change to the ibmiscsi-1.1.0 directory.
Enter ./iscsi.sh start to restart the iscsi client and then recreate the problem.
If you are on a uni-processor machine, collect the files which were placed in /tmp/iscsi/trace.
For both uni-processor and smp machines, send us /var/log/messages.
• There are several ways to communicate with the 4125. If you have no ethernet connectivity to the machine, try
these:
Console - plug in a keyboard, mouse, and monitor into the 4125.
ASM service port - plug in a standard serial cable into the management port and you can view some information
about the 4125
If you do have connectivity to the machine, any of these should work:
IBM Advanced Appliance Configuration Utility
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 139

UM Services (http://machine-name:1411)
IBM Web-based GUI (http://machine-name:8099)
Terminal Services Client
Send the output file to DATACASEas described in section >PFE datacase<
Provide also the complete SAN Layout if applicable as described in section Additional information required for data
collection on page 11

N-Series - All models


There are 3 possible ways to collect the Autosupport Data:
1) Datacollection using GUI (FilerView)
• Browse to "Filerview" with your Webbrowser: http://nseries_ip/na_admin
• Check "Filer" - "Configure Autosupport" for correctly configured mailhost and mailadress

• Click on "Filer" - "Test Autosupport", type in the PMR# as subject and press "send"-button.
140 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

• Forward the mail you received to EMEASAN@de.ibm.com

2)Datacollection using CIFS


• Browse the root share ( \\nseries_ip\C$ ) from a client pc (you will need to logon with a user that belongs to the
group BUILTIN/Administrators).
• The Autosupport reports are saved in a subfolder in /etc/log/autosupport/
• Copy folder with the correct timestamp to your local hard drive and zip it.
• Send the zip file to EMEASAN@de.ibm.com or ECuRep

3) Datacollection using NFS


• - Mount the export of the root volume ( e.g. nseries_ip/vol/vol0 ) from the administration host (check option
"root=" with CLI command "rdfile /etc/exports")
• - The Autosupport reports are saved in a subfolder in /etc/log/autosupport/
• - Copy folder with the correct timestamp to your local hard drive and zip it.
• - Send the zip file to EMEASAN@de.ibm.com or ECuRep

N-Series Gateway models


In addition to autosupport, provide:
• - SAN drawing, showing gateway to switche(s) and storage ports connections.
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 141

• - switche(s) datacollection
If the gateway cannot boot ONTAP (installation not yet completed) :?
Boot into Maintenance Mode (on CFE prompt 'autoboot' and when filer ask: 'CTR C' to get (1-5) Menue, select '5'
and boot into Maintenance Mode) and provide output of:
> version
> disk show -v
> disk list
> sysconfig -v
> storage show disk -p
> storage show adapter
> fcadmin link_state
> environment show
> storage show port
> aggr status -v
Send the output to EMEASAN@de.ibm.com or ECuRep
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
In addition a parser is available for data uploaded to PFE Datacase:
http://fieldsupport.austin.ibm.com/cgi-bin/pfe/pfe.pl

N-Series dump analysis due to filer panic


In addition to autosupport, provide :
- Collect the dump in /etc/crash/core.month_day_time.year.nz
- FTP the dump in binary mode to the ecurep server

N-Series Performance Problems


In addition to autosupports, provide:
- Detailed Problem description, complete the following template:

-----PERFORMANCE CASE TEMPLATE -----

1a) What performance-affecting behavior are they seeing? (latency values, etc in secs or ms. if errors, then the exact
errors seen, and where)
1b) What is this performance-affecting behavior causing problems with? (applications, resources)
2) What behavior are they expecting to see? How fast is it on normal days, etc. (What is slow? Are reads slow / writes
slow / browsing of file or directories /file creations/mounts? How are you judging "slow"?)
3) When did it start? (Was performance acceptable at one time? / New install and/or performance was never good /
degradation over time? )
4) When does it happen? All the time, only at certain (production) times, etc.
5) Who is it affecting? All users? Users of a particular cifs-based application? Etc
142 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components

6a) What has changed between when it was working fine and now?
6b) Have any new errors or changes been noticed in the filer's autosupports?
7a) Any errors appearing on the filer?

7b) Any errors appearing on the clients?


8) What kind of client machines are working with this filer? Is it straight cifs, cifs/nfs, cifs/SAN, only SAN (to a cifs/
win machine), etc?

In addition provide perfstat logs from the filer(s):


• - the perfstat scripts can be downloaded from the Netapp Now side with (Now level 1 access)
• - http://now.netapp.com/NOW/download/tools/perfstat/
• - if this access is not available please contact the Nseries PFE team to get the scripts.
• - For most performance issues this perfstat output is sufficient (the command has to be issued from a connected
host to the filers, during the time of performance degradation) perfstat-f <filer_IP>-l root:password-t 4-i 5
>perfstat_filer_name.out

Send the output toEMEASAN@de.ibm.comor ECuRep


Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11

N-Series Snapdrive and Snapmanager Problems


In addition to autosupports, provide:
• Output of the OntapWinDC.exe tool
• This can be find at: http://now.netapp.com/NOW/download/tools/snapmanager_e2k_dct/
• If you have no Now level 1 access, please contact the Nseries PFE team for this tool
• In addition, send a configuration layout of customers setup and a detailed problem description.

N-Series issues with connected Tapes


In addition to autosupports, provide this filer outputs:
• filer> storage show tape supported -v
• filer> sysconfig -m
• filer> sysconfig -t
• filer> storage alias
• If the tape drive is fabric connected , please provide logical diagram of fabric and zoning.
• If the tape drives are not seen, log into the switch and gather the switch information and send it:
BROCADE supportshow
CISCO show tech-support details
MCDATA show all nsshow

Send the output toEMEASAN@de.ibm.comor ECuRep . Provide also the complete SAN Layout as described in
section Additional information required for data collection on page 11

Tape ==> See Tape Cookbook


http://9.154.96.133/epsgssd/bo/RMSS/support_script/LTO_PDG.html
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 143

General
• If intergrated SAN Data Gateway (2108 G07) is installed refer to the 2108 data collection chapter to gather
the data for this component.
• For every tape subsystem we need the adapter device driver level
(AIX: Atape, SUN/HP: Atdd, NT/2k: IBMMag, IBMUltrium, or whatever it is called).
144 | IBM Storage Area Network Data Collection Guide | Appendix A. Special notices

Appendix A. Special notices

This publication is intended to help readers in troubleshooting Storage Area Network components. The information
in this publication is not intended as the specification of any programming interfaces, since this book does not contain
any kinds of programming interface information.
References in this publication to IBM products, programs or services do not imply that IBM intends to make these
available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not
intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent
program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product,
program or service.
Information in this book was developed in conjunction with use of the equipment specified, and is limited in
application to those specific hardware and software products and also their levels.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of
this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM
Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.
Licensees of this program who wish to have information about it for the purpose of enabling: (I) the exchange of
information between independently created programs and other programs (including this one) and (ii) the mutual use
of the information that has been exchanged, should contact IBM Corporation, Dept. 600A, Mail Drop 1329, Somers,
NY 10589 USA.
Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of
a fee.
The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS.
The use of this information or the implementation of any of these techniques is a customer responsibility and depends
on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item
may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar
results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at
there own risk.

Anda mungkin juga menyukai