Contents
Customer Information..........................................................................................................5
Introduction to the SAN data collection Guide..................................................................6
How to use the SAN data collection Guide..............................................................................................6
SAN Components.................................................................................................................. 7
Where to send the gathered data.........................................................................................8
Files to EMEASAN..................................................................................................................................8
ECuRep procedure................................................................................................................................... 8
Additional information required for data collection....................................................... 11
Data collection for Host platforms.................................................................................... 12
RS/6000 / AIX (UNIX).......................................................................................................................... 12
NOVELL................................................................................................................................................ 13
iSeries OS/400........................................................................................................................................14
iSeries Linux.......................................................................................................................................... 15
Management Module in BladeCenter.................................................................................................... 15
BladeCenter Modules............................................................................................................................. 16
BladeServer HSxx/LSxx (Intel/AMD) blades and general JSxx(PPC)................................................. 17
BladeCenter JS20 (type 8842) on PowerPC Architecture..................................................................... 17
XSeries/Netfinity/Intel/W2K3/W2K8....................................................................................................18
X-series Linux (SuSE and RedHat)....................................................................................................... 19
HP / HP-UX........................................................................................................................................... 20
SUN / Solaris..........................................................................................................................................21
Dec / Tru64.............................................................................................................................................23
Vmware.................................................................................................................................................. 24
SDD Enhanced trace capability..............................................................................................................24
SDD - using sddgetdata..........................................................................................................................24
SDDPCM Enhanced trace capability..................................................................................................... 25
SDDPCM - using sddpcmgetdata.......................................................................................................... 25
SDDSM Enhanced trace capability........................................................................................................25
SDDSM - using sddgetdata.................................................................................................................... 26
Special data collection for DS8000 attachments....................................................................................26
Performance problems ...................................................................................................... 28
iSeries OS/400 Performance.................................................................................................................. 28
SUN / Solaris Performance.................................................................................................................... 28
Performance Questionnaire.................................................................................................................... 28
Data collection for SAN connectivity components...........................................................33
IBM 2108-G07 Gateway (Obsolete due to EOS 1-1-2009)................................................................... 33
IBM 2108-R03 Gateway (Obsolete due to EOS 1-1-2009)................................................................... 33
IBM 7139/7140 Vicom Slic Router (SSA-FC Converter) (Obsolete due to EOS 1-1-2008)................ 34
IBM TotalStorage SANxxB-R (2109-Axx)........................................................................................... 36
IBM TotalStorage SAN04M-R (2027-R04) and SAN16M-R (2027-R16)........................................... 37
IBM 2006-L10 switch (Emulex)............................................................................................................ 46
IBM Storage Area Network Data Collection Guide | TOC | 3
Customer Information
Name: ANAF
Contact Person: Nimeni Altu
PMR: 23456,826,826
Date: 10.11.2010
Employee: Zoltan
e-Mail: zoltan_vodinszki@ro.ibm.com
6 | IBM Storage Area Network Data Collection Guide | Introduction to the SAN data collection Guide
The sometimes-complex structure of Storage Area Networks and the various different SAN devices used in such
environments has triggered the development of this document. It provides a step-by-step instruction on how to gather
information needed for troubleshooting SAN devices. It is designed to support service technicians and all other
persons involved in solving problems in the Storage Area Network neighbourhood.
"It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit theories, instead of
theories to suit facts."
Sherlock Holmes
SAN Components
For an overview use http://www-03.ibm.com/servers/storage/ here you will find the latest data of all the components
8 | IBM Storage Area Network Data Collection Guide | Where to send the gathered data
Files to EMEASAN
Emails not containing the PMR number or containing attachments > 5 MB will be deleted unread.
So in nearly all cases you must use ECuRep (see below) as all logs are greater than 5MB
Lotus Notes address is EMEASAN@de.ibm.com
The subject should reflect the PMH number, branch number, country number and customer (e.g.
PMH12345,B678,C724 Customer XYZ)
ECuRep procedure
Files to big for E-Mail
In case the gathered files are too large (more than 5 MB) to be sent via E-mail (i.e. snap from AIX), please ftp the
files to ECuRep (EMEA Centralized Customer Data Repository).
What's the purpose of this ftp site?
Customers are not able to send data through the IBM firewall. Therefore, there is an IBM internet-server that
customers can access directly (ftp.emea.ibm.com).
Data on this server is mirrored every 10 minutes to an IBM intranet server.
How customers upload data?
If you request data from the customer, here's what the customer has to do. All data must be compressed before it is
sent to the FTP.EMEA.IBM.COM server (exceptions as desribed in the specific chapters). MVS customers should use
the TRSMAIN program to compress their file [SPACK parameter] which can be obtained from the IBM site: ftp://
ftp.software.ibm.com/s390/mvs/tools/packlib/ . The installation instructions are in the README file. By compressing
their data, they improve data transfer time and convert their files into a compatible record length and record format.
VM users should compress the file to be ftp'ed using the CMS COPYFILE command with the PACK option.'
Note: The FTP procedure (see below) is the preferred way to upload the files. If the customer can not use this
because the FTP possibilty is blocked then he can use:
• http://www.ecurep.ibm.com/app/upload for standard upload
• https://www.ecurep.ibm.com/app/upload for secure upload
IBM Storage Area Network Data Collection Guide | Where to send the gathered data | 9
If a customer is told to upload any data to the EMEA FTP server, he or she has to perform the following steps:
FTP to 'ftp.emea.ibm.com'
login as 'anonymous'
enter the email ID as password
> bin
> cd toibm
> cd hw
10 | IBM Storage Area Network Data Collection Guide | Where to send the gathered data
Note:
Customers must use the appropriate naming conventions as shown in the examples below:
xxxxx.bbb.ccc.yyy.yyy
where the parts have the meaning:
xxxxx =PMR number
bbb = Branch Office (if known)
ccc = IBM Country Code (e.g. Germany 724)
yyy.yyy = Short description for the filetype, e.g. tar.Z, restore.Z, terse
Because login by the customer was done with an anonymous user ID, no file listing is allowed on the server.
Customers cannot check what files they have already uploaded, which means that this has to be done carefully.
INTRANET users, for example IBM employees, MUST use the 'ftp.ncs.mainz.ibm.com' URL to access the FTP
server.
Note: An automatic update to the PMR will only work with a correct file name.
IBM Storage Area Network Data Collection Guide | Additional information required for data collection | 11
To determine what you need to collect you can ask the customer the following questions:
• What is the customer's platform? (Z-series, P-series, I-series, X-series, Linux, HP,?)
• Which platform has the problem or is it a general problem?
• Is the problem on one or on more servers?
• Is it a solid or an intermittent problem?
• Did the customer “fixed” the problem and if yes how? (reboot a component, enable/disable a path,....)
• What components are in the SAN fabric? (2109, 2005, 2006, 2026, 2027,...., long distance solution, data
gateway,.....)
• Is the problem on one component, on more or do not know? (Data needed from one, more or every SAN
component).
• What kind of I/O has the problem? (DS4000, 2105, NAS, Tape)
• A complete SAN layout as described below.
• Detailed history of the problem giving dates and times when known or estimated
• Support actions already taken
• Time difference between SVCs, hosts, SAN, etc...
Drawing of his SAN layout
Including the following information:
• Cabling diagram including cabling from Host to Switch/Gateway to the ESS/Tape
• Port numbers and Domain-id of the switches
• WWNN or WWPN
• Number of cables connect to each of the components
• Are certain servers used in a clustered environment?
• Volume assignment
• How many volumes should each host see?
• Do any of the hosts share volumes?
• Zoning information (if used)
The drawing of the SAN layout can also be uploaded to PFE Datacase or ECurep
12 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms
> snap -r
> snap -gfikLGc
or
> Go to /tmp/ibmsupt
> rm -r*
> snap -gfikLGc
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser (as stated before: 20001 can be
different)
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• aSubsystem information
• Datapath device information
> datapath query adapter
> datapath query device
> datapath query adaptstats
> datapath query devstats
> adapter performance
> device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace
NOVELL
Always request the following data from all involved NetWare servers:
CONFIG.txt (Server Configuration)
Execute from console screen
> LOAD CONFIG
==> This will create a current CONFIG.TXT which will be in SYS:\SYSTEM directory on the server.
ABEND.LOG
An ABEND.LOG is created in the SYS:\SYSTEM directory on a server whenever it abends.
SYS$LOG.ERR
Location of file: SYS:\System\SYS$LOG.ERR
LOGGER.TXT
The LOGGER.TXT contains the latest history of the logger screen, e.g. modules loaded, etc. + output of SDD
"datapath query" commands.
Hit F2-key to save the logger screen to C:\NWSERVER\LOGGER.TXT
If SDD is installed
Provide the outputs of the following SDD commands (can be included in LOGGER.TXT):
datapath query adapter
datapath query device
When DS3000/4000/5000 is connected to Novell NetWare we need:
14 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms
1. config.txt This is to check the Novell Netware version and Support Pack (SP),? the correct driver settings are
loaded (Startup.ncf), the HBA adapter number and PCI slot, the HBA device driver version and the required
MPIO-driver and -module versions
2. list failover devices This is to check the preferred paths (priority) per HBA and Lun
3. cfg.nlm (this is a QLogic Module which comes with the IBM device driver package. It is a MSJ adequate for the
Novell console command line.) This is to check for HBA device driver version, the HBA WWN and BIOS, the
DS3000/4000/5000 controllers seen by the singel HBA and to check the switch domain and port where the HBA
and the DS4000/FAStT controller is inserted.
4. If you use the NetWare MPIO failover driver attached to the DS3000/4000/5000 run the commands
list partitions
list storage adapter
list failover devices
iSeries OS/400
Copies of the following print outputs...
1. System Configuration List
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 7 "Hardware Service Manager" from the "Start a Service Tool" screen
• Press F6 "Print configuration" from the "Hardware Service Manager" screen
2. System Configuration List -- Logical HW Resources:
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 7 "Hardware Service Manager" from the "Start a Service Tool" screen
• Select option 2 "Logical Hardware resources" from the "Hardware Service Manager" screen
• Press F6 "Print configuration" from the "Logical Hardware Resources" screen
3. World Wide Port Name Information (FiberChannel only)
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 7 "Hardware Service Manager" from the "Start a Service Tool" screen
• Select option 2 "Logical Hardware resources" from the "Hardware Service Manager" screen
• Select option 1 "System bus resources" from the "Logical Hardware Resources screen
• For each iSeries FiberChannel Disk Adapter #2766 or #2787:
• Select option 9 for the corresponding IOP #2843 or #2844 of the IOA #2766 or #2787 whose WWPN is of
interest
• Select option 5 "Display detail" for the IOA #2766 or #2787 and note the "Resource name" together with its
"Port worldwide name" (WWPN)
4. Product Activity Log :
• Access system service tools by using the command STRSST
• Select option 1 "Start a service tool" from the "System Service Tools (SST)" screen
• Select option 1 "Product activity log" from the "Start a Service Tool" screen
• Select option 1 "Analyze log" from the "Product Activity Log" screen
• Enter "3" for Log (3 = Magnetic media log) and timeframe of log in the "Select Subsystem Data" screen
• Enter "3" for Report type (3 = Print options) and "Y" for including optional statistical entries in the "Select
Analysis Report Options" screen
• Enter "4" for Report type (4 = Print full report) and "Y" for including hexadecimal data in the "Select Options for
Printed Report" screen
5. System Operator Message Queue:
• Run the "DSPMSG MSGQ(QSYSOPR) OUTPUT(*PRINT)" command to get a spool file of the system operator
message queue contents
6. List of installed PTFs
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 15
iSeries Linux
Copies of the following print outputs...
1. Linux Partition Allocated I/O Resources
• Access system service tools by using the command STRSST
• Select option 5 "Work with system partitions" from the "System Service Tools (SST)" screen
• Select option 1 "Display partition information" from the "Work with System Partitions" screen
• Select option 3 "Display allocated I/O resources" from the "Display Partition Information" screen
• Enter the corresponding Linux partition number for "System partition(s) to display"
• Press F6 "Print" from the "Display Allocated I/O Resources" screen
2. Linux Partition Reference Code History
• Access system service tools by using the command STRSST
• Select option 5 "Work with system partitions" from the "System Service Tools (SST)" screen
• Select option 1 "Display partition information" from the "Work with System Partitions" screen
• Select option 8 "Display secondary partition reference code history" from the "Display Partition Information"
screen
• Enter the corresponding Linux partition number for "System partition(s) to display"
• Press F6 "Print" from the "Display Secondary Partition Reference Code History" screen
3. Startup Console Log
• Issue command "dmesg > /var/log/dmesg.txt" and obtain the output file
• Alternatively from the managing OS/400 partition:
• Access system service tools by using the command STRSST
• Select option 5 "Work with system partitions" from the "System Service Tools (SST)" screen
• Select option 1 "Display partition information" from the "Work with System Partitions" screen
• Select option 10 "Display guest environment console log" from the "Display Partition Information" screen
4. System Message Log
• Obtain the syslogd system message text file "/var/log/messages"
5. List of installed PTFs on the OS/400 managing partition.
• Issue command "DSPPTF OUTPUT(*PRINT)" and obtain the generated spool-file
BladeCenter Modules
a) DataCollection for BladeCenter Ethernet Switch Module
- Perl is required to run a support script, therefor Perl needs to be installed on the MM Server
- Perl is open source and can be downloaded at http://www.perl.org
- request the 'dlink_dump.zip' package to generate html Ethernet Switch Module data from IBM
- this package can be download at http://bisc.raleigh.ibm.com
- from there go to 'Resources -> Utilities -> Dlink Module Dump Tool' and save this .zip file
- out of this zip archive you need 2 files to generate the output file we need :
- enter the IP address of your switch module to the 'Host Address' field,
( when in Bay3 of the BladeCenter, the IP address defaults to 192.168.70.129,
when in Bay4,? it defaults to 192.168.70.130 )
- enter a '23' to the 'Port Number' field
- press 'OK'
- in the HyperTerminal main window, you now can logon to the switch :
• for Qlogic SanBox2 use : sanbox2 login=USERID and password=PASSW0RD (0=zero!)
• for Brocade 3016 use : user= USERID and password= PASSW0RD (0=zero!)
- start recording of logging data with HyperTerminal menu 'Transfer -> Capture Text -> Start',
define file location and file name for the logging data of the switch and press 'Start'
- in the HyperTerminal main window, send the correct FCSM cmd to generate the data we need
- stop recording
- send the generated log file to ECUREP as described in section ECuRep procedure on page 8
- if you have multiple FCSMs, repeat the above process while using the appropriate IP address.
- collect JS20 data by following the 'Linux Aid Install Instructions for JS20' :
http://www14.software.ibm.com/webapp/set2/sas/f/lopdiags/images/JS20_Install.pdf
• generate lvcfg output : lvcfg > filename.txt
• generate snap data : snap (generates snap.tar.gz )
• send generated filename.txt and snap.tar.gz to ECUREP as described in section ECuRep procedure
on page 8
18 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms
XSeries/Netfinity/Intel/W2K3/W2K8
Customer needs to provide the following information:
• Manufacturer of FC Adapter/SCSI Adapter
• BIOS level of the FC Adapter/SCSI Adapter
For data collection use the DSA tool. See section for procedure.
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If SDD is installed:
Run the following commands
> datapath query adapter
> datapath query device
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
Verifying that the SDD server has started
W2K3 and W2K8
Click Start > Programs > Administrative Tools > Computer Management.
Expand the Services and Applications tree.
Click Services.
Right-click SDD_Service.
Click Start. The status of SDD Service should be Started if the SDD server has automatically started.
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• Subsystem information
• Datapath device information
• Server trace
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 19
Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
Type ps wax |grep sddsrv to verify if the SDD server has automatically started.
command output looks like this:
31616 S 0:00 /opt/IBMsdd/bin/sddsrv
31617 S 0:00 /opt/IBMsdd/bin/sddsrv
31618 S 0:00 /opt/IBMsdd/bin/sddsrv
31619 S 0:10 /opt/IBMsdd/bin/sddsrv
31620 S 0:00 /opt/IBMsdd/bin/sddsrv
31621 S 0:00 /opt/IBMsdd/bin/sddsrv
31622 S 0:00 /opt/IBMsdd/bin/sddsrv
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser
- Enter "Server trace" and provide "get server trace"
20 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms
HP / HP-UX
Question to customer:
Is the EMS (Event Monitoring Service) DISABLED on all HPUX attached to ESS?
Run this commands:
> swlist > output.txt
> swlist -l patch
> ioscan -fnC disk
> dmesg >> output.txt
> diskinfo /dev/rdsk/c#t#d#
> pvdisplay
> uname -m
> uname -r
> lsdev
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If SDD is installed
Run the following commands
> showvpaths output
> datapath query adapter
> datapath query device
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 21
Additional SDD action plan (See also >SDD, SDDPCM and SDDDSM data collection<)
• Type ps -ef |grep sddsrv to verify if the SDD server has automatically started and on what port (sddsrv starts with
20001 as the default port number; the customer may use a different port)
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser (as stated before: 20001 can be
different)
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
Status options:
• Server status
• Device information
• Subsystem information
• Datapath device information
• adapter status (datapath query adapter)
• device status (datapath query device)
• adapter statistics (datapath query adaptstats)
• device statistics (datapath query devstats)
• adapter performance
• device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace
SUN / Solaris
There is a data collection tool available in the internet.
Go to http://sunsolve.sun.com/search/document.do?assetkey=1-9-82329-1
Customer must be registered for this tool
Download the SunTM Explorer Data Collector
Run the collector
Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Collect also the data for SDD or veritas (see next page)
OR
For data collection of Solaris Systems provide the following information:
1. modinfo > modinfo.output
2. showrev -p > showrev.output will show the installed patch level
Update the installed patch level if needed. Patches for Solaris 2.6 can
be downloaded at:
22 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms
http://sunsolve.sun.com/pub-cgi/show.pl?target=patches/patch-access
3. Send the file "/etc/systems"
4. Send the file "/kernel/drv/sd.conf" if connected to disk
Send the file "/kernel/drv/st.conf" if connected to tape
5. If FC adapters are used
Send the file "/kernel/drv/fcaw.conf" for JNI/JAYCOR - FC adapter
or
Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
If SDD is installed:
Run the following commands
> showvpaths output
> datapath query adapter
> datapath query device
Send the output files to ECUREPas described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Additional SDD action plan (see also >SDD, SDDPCM and SDDDSM data collection<)
• Type ps -ef |grep sddsrv to verify if the SDD server has automatically started and on what port (sddsrv starts with
20001 as the default port number; the customer may use a different port)
• collect SDD Server Daemon Trace and SDD Device Driver Trace
enter "http://<URL>:20001" or "http://<IP address>:20001" on any web browser (as stated before: 20001 can be
different)
- Enter "Server trace" and provide "get server trace"
- Provide "Device driver trace"
Remark: The outputs of the 'datapath' commands can also be obtained via "http://<URL>:20001", menu "Datapath
device information"
Output: SDD Server Status Information Available
• Server status
• Device information
• Subsystem information
• Datapath device information
• adapter status (datapath query adapter)
• device status (datapath query device)
• adapter statistics (datapath query adaptstats)
• device statistics (datapath query devstats)
• adapter performance
• device performance
• Server trace
• get server trace
• start detailed server trace
• start normal server trace
• Device driver trace
Dec / Tru64
24 | IBM Storage Area Network Data Collection Guide | Data collection for Host platforms
Vmware
Run the following script on the service console:
> /usr/bin/vm-support
This script collects all relevant ESX Server system and configuration information and ESX Server log files.
Send the requested data and files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
When you report an SDD problem, it is essential to run this script and send this output file for problem determination.
Steps within the sddgetdata script might fail depending on the problem and the system condition. Is this case, you
might have to execute manual commands.
Here is an example output for the AIX platform:
/tmp/sdd_getdata>sddgetdata
/tmp/sdd_getdata>ls
./ ../ sdddata_host1_20050315_122521.tar
Send the requested data and files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
Remarks:
Home Dir is in case of WinXP: C:\Documents and Settings\USERID
Install Dir defaults: WIN = c:\Program Files\IBM\DSCLI\...
UNIX based systems: \opt\ibm\dscli\...
DS8000 Storage manager related problems:
If you have the Storage manger installed on a separate client gather the following logs:
(Paths are only valid, if the default install directories got used during installation):
Windows client:
IBM Storage Area Network Data Collection Guide | Data collection for Host platforms | 27
C:\Program Files\IBM\SMServer\AppServer\logs\*.*
C:\Program Files\IBM\SMServer\AppServer\log\*.*
C:\Program Files\IBM\dsniserver\logs
C:\Program Files\IBM\SMServer\logs
C:\Program Files\IBM\DS8000StorageManager\logs
C:\Program Files\IBM\DS8000StorageManager\SM\logs
Linux client:
/opt/IBM/SMServer/logs/*.*
/opt/IBM/SMServer/AppServer/logs/server1/*.*
/opt/IBM/SMServer/AppServer/logs/*.*
28 | IBM Storage Area Network Data Collection Guide | Performance problems
Performance problems
Performance Questionnaire
SVC L2 Support EMEA’s Performance Questionaire
Note: To get a clear picture of the situation, the involved devices and the impact, fill out the below questions as
precise as possible.
1. General Questions :
2.3.6 In questions of sw levels and device drivers, the affected host(s) is ...
... pretty much uptodate ???
... very outdated ???
.
2.4 Can the performance problem be bound to a single or to
multiple applications ?
2.4.1 What applications ?
2.4.2 What does application(s) logs show ?
3.2 Does SVC report any specific errors which indicates I/O issues ?
5.1 How does the SAN layout look like ? Core-edge ? Fully-meshed ?
5.1.1 Provide a detailed SAN topology with distances, port numbers,
attached devices, etc !
5.1.2 Describe SAN devices and which are affected by the problem !
IBM Storage Area Network Data Collection Guide | Performance problems | 31
5.2 Have error counters for all ports on all boxes been cleared and
monitored and - if needed - have appropriate actions been taken
( replace SFPs / cables / etc ) to solve obvious issues ?
5.4 Are specific SAN switch features in use ( Preferred Path, Trunking,
traffic isolation ... ) ?
.
5.5 What about he SAN details :
5.5.1 Are long distance links in use ?
5.5.2 Are DWDMs / dark fiber / IP Routing / etc ... in use ?
5.5.3 If there is an FCIP WAN network, describe network and Service
Level Agreement SLA (bandwidth, MTU, delay) and TELCO provider ?
5.5.4 If an IP network used, is any other type of traffic mixed with
storage traffic ?
From the Serial Port Service Utility Key Assignments select Display VPD
Capture the window and send it to ECUREPas described in section ECuRep procedure on page 8
2.Provide the settings of DIP Switch 1 and DIP Switch 2
Both switches are located on the side of the unit
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 35
?
Slow Blink: LED is ON 90% and OFF 10%
Steady Blink: LED is ON 50% and OFF 50%
If the Status LED Blinks Code refer to the following section in order to decipher the LED Blinks
The Blink Code of the Status LED presents digital numbers as follows:
36 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
Numerical Digits
Presentation of
0 = short, fast blink
1 = LED blinks once
2 = LED blinks twice with one short duration between blinks
3 = LED blinks three times with one short duration between each blink
And so on
Decimal Numbers
Each digit can be obtained by the number of blinks in series separated by only a short duration of LED Off
Decimal Digits are separated by a medium duration of LED OFF (two seconds)
Status Code is separated by a long duration of LED Off (four seconds) before the code is repeated
Example:
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout including the Fibre Channel Side and the SSA side as described in section
Additional information required for data collection on page 11
http://www-1.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session logging
Login into the switch via the RS232 interface or through the Ethernet port
The default login is set to:
Login: "admin"
Pass: "password"
If the customer has changed the default login settings he needs to provide the current values.
Run the command 'supportShow' and send the log file to ECUREP as described in section ECuRep procedure on
page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
press OK
Verify the TFTP root in Windows Explorer
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 39
Open all items under mSAN pane and select the multiprotocol router (in our case 2640-122)
Select Preferences
40 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
Be sure that c:\TFTP-Root is selected in the TFTP rootpath. If not use browse to get it in.
Select OK
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 41
Select Tools at the top than navigate over reports and select Configuration Archive
Go to Local Disk C
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 43
Go to TFTP-Root
44 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
Name the file (in our case it is the TCP/IP name of the multiprotocol Router)
select open
Check the results in this panel under "Archive location and file name"
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 45
Press Start
It is possible that you get the following message if you did not follow the procedure correctly
(You maybe forgot to open the mSAN configuration window in a previous step)
When you see that all .log files are received than you must go to the file (in our case c:\TFTPRoot
\9.155.69.119.zip) and send this file to ECuRep
Or:
Send the output files to ECUREP as described in section ECuRep procedure on page 8
Switch:admin> supportsave
Switch:admin> cmsh
Switch:cmsh> terminal length 0
Switch:cmsh> show tech-support
Run the command and send the log file to ECUREP as described in section ECuRep procedure on page 8:
DCFM
If client and server are running on the same system you need to perform part 1 only
If client and server are running on different system you need to perform part 1 and 2.
1/ DCFM Server data collection
Starting DCFM Server Console using SMC Batch File
Double click on <install directory>\DCFM 10.0.X\bin\smc.bat
Or From command line
Form DOS
C:\Program Files\DCFM 10.X.X\bin\smc.bat or
From Solaris and Linux
/opt/DCFM10_X_X/bin/smc
The process status should be started (if not, try to start all the process using the start button)
Using the "technical support information tab" choose a output path and click capture
52 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
When DCFM supportSave completes you will see the following message with location of the file
-If Client and Server running same system, the server supportSave will capture both Client and Server supportSave
data (DCFM client supportSave, HTTP capture, Java Console)
If client and server are not running on the same system we need to perform the client data collection as follow
2/ DCFM client data collection
In this part we will collect: client supportSave, HTTP capture and Java console output
Client supportSave
On Microsoft Windows client system you will find "clientsupportSave.bat"file at:
C:\Documents and Settings\<User currently logged in Windows Host>\DCFM\<Server IP>".
-The DCFM Client collection will take a few minutes
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 53
HTTP capture
We also need to capture the HTTP on MS Windows
How to get HTTP Capture on MS Windows
- Only Available on Microsoft Windows, not available on Unix
- Go to the "%DCFM_HOME%\bin"directory
--C:\Program Files\DCFM 10.0.2\bin (default location)
- Double click on the "httpcapture.bat"file.
- The HTTP Capture Utility will capture all required HTML pages ofthe Fabrics and Switches currently monitored in
DCFM.
- The HTTP Capture will be captured under "%DCFM_HOME%\support\HttpCapture-<TIMESTAMP>"folder.
Please zip the corresponding older when sending it
Then we will need to capture the JAVA console
Java Console output
-If there is a problem starting the Client collect the Java Console output
Click on the JAVA icon the windows control panel
54 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
Enable java console output (use control panel, java, advanced, enable logging and show console.)
Debug information will appear in the java window. Just cut and paste this information.
If some data collection are not possible one alternative would be to collect
DCFM_10.X.X_InstallLog.log file under
x:\Program Files\DCFM 10.x.x
=> And in addition the content of conf and logs directory (Under x:\Program Files\DCFM 10.x.x
Send the output files to ECUREP as described in section ECuRep procedure on page 8:
1
a EFCM 8.7 and higher without group manager license
b EFCM 8.7 and higher with group manager license
Note: For older code you can refer to previous version of SAN cookbook as these are not supported anymore.
On EFCM main windows, click on Configure pulldown menu, Select Group Manager
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 57
Use explorer on the EFCM and get the file you just created.
Send the file ECuRep as described in ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
2) Collecting data using web GUI interface (EFCM basic formerly SANpilot)
For McData E/OS firmware 08.00.00 and above go to subsection a
For McDATA E/OS firmware below 08.00.00 collect data according subsection b
a) Data collection for EFCM Basic with Eos 08.00.00 and above (Web GUI)
Choose Logs| all
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 61
> version
> fc interopMode 1
> fc interopMode 1
> fc config 1
Password: admin
In the IN-VSN Client click Fabrics > Fabric Name (here TIC) and double click on the director's picture as show
below
The Enterprise Manager will display the general view of the director
The Export save menu wills pop-up. Specify a directory and file name to store the file. Make sure that the files
extension is set to .txt (Text file)
Click on Export
Click on Audit Trail
68 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
Click on Export
The Export Save Menu wills pop-up. Specify a directory and file name to store the file. Make sure that the files
extension is set to .csv (Comma Separated Values Files)
Click on Export
Click on EventLog
Click on Export
The Export Save Menu wills pop-up. Specify a directory and file name to store the file. Make sure that the files
extension is set to .csv (Comma Separated Values Files)
Click on Export
Send all files and screen shots to ECUREP as described in section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 69
CNT FC/9000, CNT UMD, IBM 2045-N16 and inVSN 9.0 code
For UMD and 2045 you must complete a special form and do the data collection below (1. or 2.). Send it to ECuREP
or EMEASAN@de.ibm.com
Form is available on our web site. http://web.mainz.de.ibm.com/e_dir/esssstosol.nsf/DocumentsSortedTitle/-
Homepage
SAN
FC switches and directors
UMD and 2045 Problem Determination Form
For the IN-VSN 9.0 code you can collect the Debug Backup file in two ways. Server (1) or Client (2)
1. Using the inVSN Enterprise Manager - Server
1) Go to the inVSN Enterprise Manager - Server
2) Select Maintenance
3) Select Debug Backup
4) Give the .zip file a name and navigate in the folder structure to the place where you want to create the xxx.zip file
and click on Create.
70 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
5) Select from which UMD's or FC/9000 you want to create this debug
backup file. Multiple machines are possible. (Default is all machines)
6) Select OK
7) Check if create was successful and click OK
8) Click Close
9) Send all files and screen shots to EMEASAN@de.ibm.comor ECuRep
10) Provide also the complete SAN Layout as described in section Additional information required for data
collection on page 11
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 71
6) Select from which UMD's or FC/9000 you want to create this debug
backup file. Multiple machines are possible. (Default is all machines)
7) Click on Save As and give the .zip file a name and navigate in the folder structure to the place where you want to
create the xxx.zip file.
8) Select OK
9) Select OK
10) Select Yes
IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components | 73
Select View in
the top left corner
74 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
2) Select Events
These commands are very helpful for analyzing any kind of errors in a FICON environment.
On the affected LPAR (logical Partition) start the z/OS Commands:
D M=CHP command displays the status and type of CHPIDs
D M=CHP(cc) where cc is the affected CHPID.
D U,,,dddd,1 where dddd is the affected device address
DS P,dddd where dddd is the affected device address
D M=DEV (dddd) where dddd is the device address
F IO,D D dddd * where dddd is the device address
Connect to 2061 or 2062 using Ethernet or Serial Port (9600 8-N-1, no flow control). Further information on how to
establish a connection to the switch can be obtained in Chapter 3 'Connecting the Cisco MDS 9500' of the Cisco
MDS 9500 Series Hardware Installation Guide.
Start the terminal program "Net Term" which can be downloaded at:
http://www-01.ibm.com/support/docview.wss?uid=ssg1S4000078
Enable session logging
If you use an AIX or UNIX server be sure the buffer size is big enough. To set it use
dtterm -sb -sl 50000 command (50000 = number of lines)
or run: telnet address.of.a.device | tee /path/to/logfile
where address.of.a.device is an address (a resolvable name or an IP address) of the device you want to collect data
from and /path/to/logfile is a path to a file you want to log your output to. This command will save the whole telnet
session to the file named logfile
Login into the switch
The default login is set to:
Login: "admin"
Pass: "admin"
If the customer has changed the default login settings he needs to provide the current values.
Run the command:
> terminal length 0
Run the command:
> show tech-support detail
Run the command:
> terminal length 24
Run the these commands on every single switch of the fabric and send the log file to ECUREP as described in
section ECuRep procedure on page 8
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11 You may print a copy
of the map provided by Cisco Fabric Manager as a SAN Layout.
Other commands that can be done comparing to 2109:
2109---------------------mds9000
cfgshow(zoneshow)---***--sh zoneset active [vsan vsan#]
76 | IBM Storage Area Network Data Collection Guide | Data collection for SAN connectivity components
2145 (SVC)
Minimum requirements
• open a PMS (compid: SANVCNTL1), not a PMH and send it to EBESAN,25V
• SVC software level (CLI command: svcinfo lscluster <cluster_name>)
• Detailed history of the problem including dates and times when known or estimated
• Upload requested logs to DUMPDECODER (See procedure in the datacollection parts)
• Support actions already taken
• Time difference between SVCs, hosts, SAN, ...
• login as 'admin'
• Depend on SVC code, perform the following commands to capture the data collection onto the Config node:
• SVC code 4.1 and above use step 3 and 4 (a)
• SVC code less as 4.1 use step 1 to 4 (b or c)
1. IBM_2145:admin>svcinfo lsnode -delim :
Write down the node id's of all Non config node's (1 to 7)
b) Transfer collect data from the config node onto Master Console
On the SVC Master Console open a DOS window:
Create a new directory in root of master console with commands
• cd\ ; change directory to root
• md data ; make directory
Followings commands copy the data collection from config node into the directory 'data' of Master Console:
for SVC code level 2.1.0.3 and above :
> pscp -unsafe -load <name_of_putty_session> admin@<cluster_ip_address>:/
dumps/snap* c:/<target_dir>
if we request more information use first both commands of below 2.1.0.3
for SVC code level below 2.1.0.3 :
> pscp -unsafe -load <name_of_putty_session> admin@<cluster_ip_address>:/
dumps/* c:/<target_dir>
Note:
80 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
for example:
<name_of_putty_session> -> PFE_SVC
<cluster_ip_address> -> 9.155.86.33
<target_dir> -> data
• zip all files from 'data' directory together with name of PMR_ClusterName.zip
Data must be sent to following as follows:
The following procedure must be done for every SVC problem BEFORE sending it to our EBESAN, 25V queue
because there are several problems for uploading the correct data to the correct servers in Hursley.
With this procedure we do not need our laptop to download and upload the data and will save up to 4 hours when
working from a remote location.
ALL SVC data must be sent to ECuRep.
NEVER use nesting zip, tar or other compression program (e.g. zipping a zipfile) as the decoder has a problem to get
the data readable when nested compression is used.
Procedure for the correct ECuRep procedure is on the following website:
http://itcenter.mainz.de.ibm.com/ecurep/emea.html
There will be an automatic update in the PMR if the data is correctly uploaded to ECuRep.
2 options are available; WEB-Upload or FTP
Following PMR number 00062.239.624 is only an example. Use your specific PMR number instead.
Tasks of the customer:
Do NOT compress the output file as this is already a .tgz file.
Note: It can happen that the .tgz file is changed into .tg or .gz file. This is a windows habit to do this. Please be sure
it is a .tgz file, otherwise rename it.
o Option 1: WEB-Upload
Note:
If you have problems with WEB-Upload, please use alternative the FTP procedure (see below)
http://www.ecurep.ibm.com/app/upload for standard upload
https://www.ecurep.ibm.com/app/upload for secure upload
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 81
• login as 'admin'
• Depend on SVC code, perform the following commands to capture the data collection onto the Config node:
• SVC code 4.1 and above use step 3 and 4 (a)
• SVC code less as 4.1 use step 1 to 4 (b or c)
1. IBM_2145:admin>svcinfo lsnode -delim :
Write down the node id's of all Non config node's (1 to 7)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 83
Select the snap file; right mouse click and "save target as" on local server
Data must be sent to following as follows:
The following procedure must be done for every SVC problem BEFORE sending it to our EBESAN, 25V queue
because there are several problems for uploading the correct data to the correct servers in Hursley.
With this procedure we do not need our laptop to download and upload the data and will save up to 4 hours when
working from a remote location.
ALL SVC data must be sent to ECuRep.
NEVER use nesting zip, tar or other compression program (e.g. zipping a zipfile) as the decoder has a problem to get
the data readable when nested compression is used.
Procedure for the correct ECuRep procedure is on the following website:
http://itcenter.mainz.de.ibm.com/ecurep/emea.html
There will be an automatic update in the PMR if the data is correctly uploaded to ECuRep.
2 options are available; WEB-Upload or FTP
Following PMR number 00062.239.624 is only an example. Use your specific PMR number instead.
Tasks of the customer:
Do NOT compress the output file as this is already a .tgz file.
Note: It can happen that the .tgz file is changed into .tg or .gz file. This is a windows habit to do this. Please be sure
it is a .tgz file, otherwise rename it.
o Option 1: WEB-Upload
Note:
If you have problems with WEB-Upload, please use alternative the FTP procedure (see below):
http://www.ecurep.ibm.com/app/upload for standard upload
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 85
Go to https://ecurep.mainz.de.ibm.com/aex/ and type in you PMR number (separated with commas like in Retain)
86 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
Click on "toPMR" .
In the next window you will find the uploaded file. Select the file(s) you want to transfer.
Along with other options like compressing or extracting you have the possibility to transfer the file via FTP. Select
this option and click on execute.
In the next window set the Destination host to SVCSUPPORT.SSD.HURSLEY.IBM.COM and the directory
to /2145/in/00062.239.624 (PMR #; separated by dots instead of commas)
Folder 00062.239.624 will be made if it is not there.
User is anonymous and password is your email-address.
Mode has to be set to binary.
Fill in the fields EXACTLY as you did when you FTPed the files.
Click on Add PMR button
This automatically opens a new window like:
This opens the folder where you uploaded the files using FTP
Hit "Get" (If you used a different folder in the FTP part you can change it here)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 89
Now you can have a look in the error logs to see if these are readable.
Note: The instructions for the GUI for SVC Console V4.2.1 ("Pegasus"-based) are different; please find it on: http://
svcsupport.ssd.hursley.ibm.com/wiki/index.php/SVC_Console_Log_Collection_%28V4.2.1%29
Pathnames
In the instructions below <WAS> refers to the base directory of the Websphere Application Server (WAS). In the
instructions below <CIM> refers to the base directory of the CIM Agent (CIMOM+Provider).
90 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
<WAS>/logs/server1/trace.log
A successfully enabled logging session will be indicated with an entry in the trace.log file similar to
[5/21/03 14:31:13:874 PDT] 2ff3581b ManagerAdmin I TRAS0018I: The trace state has changed. The new trace
state is ConsoleTrace=all=enabled.
At this point, attempt to do the procedure that is not completing successfully in the GUI.
After you have recreated the problem do the following:
1. Change the default directory to the WAS bin directory.
> cd <WAS>\bin
1. Disable logging, by entering the following on the command line.
> wsadmin -connType NONE -c "$AdminControl setAttribute [$AdminControl
completeObjectName type=TraceService,process=server1,*] traceSpecification
ConsoleTrace=event=disabled:ConsoleTrace=debug=disabled"
<WAS>/logs/server1/trace.log
A successfully disabled logging session will be indicated with an entry in the SystemOut.log file similar to
DEBUG_MIN
DEBUG_MID
DEBUG_MAX
Use DEBUG_MAX
• Edit the text file <CIM>/logger.properties
• Update the following entries to the desired debug level listed above
message.logger.level=
service.logger.level=
security.logger.level=
trace.logger.level=
• For general logging, trace and message logging should be enabled for logging. This is required for GUI
application debug/analysis.
1. Start CIMOM. To start the CIMOM service:
• Go to Control Panel->Administrative Tools->Services
• Select the service: "IBM CIM Object Manager - SVC".
• Right Click with the mouse and select 'Start'.
This will start the CIMOM.
1. Debug/Trace output will be displayed in <CIM>/providerTrace.log
The latest trace output is written to providerTrace.log, historic trace output is archived in providerTrace[x].log
message.logger.level=DEBUG_MIN
service.logger.level=DEBUG_MIN
security.logger.level=DEBUG_MIN
trace.logger.level=DEBUG_MIN
1. Start CIMOM. To start the CIMOM service:
• Go to Control Panel->Administrative Tools->Services
92 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
Note:
If you have problems with WEB-Upload, please use alternative the FTP procedure (see below):
http://www.ecurep.ibm.com/app/upload for standard upload
https://www.ecurep.ibm.com/app/upload for secure upload
Note:
Note: Pictures are for a snap but it is the same procedures for the .log files.
Click on execute.
In the next window you will find the uploaded file. Select the file(s) you want to transfer.
Along with other options like compressing or extracting you have the possibility to transfer the file via FTP. Select
this option and click on execute.
94 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
In the next window set the Destination host to SVCSUPPORT.SSD.HURSLEY.IBM.COM and the directory
to /2145/in/PMR (separated by dots instead of commas)
Folder00062.239.624 will be created if it is not there.
User is anonymous and password is your email-address.
Mode has to be set to binary.
Fill in the fields EXACTLY as you did when you FTPed the files.
Click on Add PMR button
This automatically opens a new window like:
This opens the folder where you uploaded the files using FTP
Hit "Get" (If you used a different folder in the FTP part you can change it here)
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 97
Now you can have a look in the error logs to see if these are readable.
- SVC Stats :
Every SVC code 3.1+ include this option.
SVC generates statistics files at a specific interval
for SVC 3.1+ (between 15 and 60 minutes)
for SVC 4.1+ (between 1 and 60 minutes)
- TPC :
TPC allows this interval to be set as low as 5 minutes.
(please goto chapter "SVC performance data collection by using TPC")
Customer needs the TPC standard edition. This edition includes the performance measurement (TPC for Disk.) The
SSPC has the TPC basic Edition only ; SSPC performance bundle is required.
98 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
configuration
- open a dos prompt
and change directory where the 'stats.bat' file is stored
cd\
cd C:\io_stats\StatDumpScriptExternal\StatDumpScriptExternal
/I 1 1 = every 1 minute
Sets a user defined stats collection interval in minutes
nn Specifies the interval at which statistics should be collected
Must be an integer
defaults to 15
/L 20 20 = 20 measurements
Specifies how many sets of files the script should collect before exiting
e.g./L 4 will normally result in the script running for 45 minutes
/L 10 /I 30 will normally result in the script running for 5 hours
/L 1 /I 20 /D will normally result in the script running for 20 minutes
/D Specifies that stats files should be deleted from the cluster before
9.155.86.33 -> Cluster IP-Address
C:\Program Files\IBM\svcconsole\cimom\icat.ppk -> default directory of icat.ppk
collection -> name of directory, which the statistic data will be stored
you will find all performance data and SVC configuration files
- zip all files in this directory e.g.
as 00062.239.624. io_stats.080520.zip
• 00062.239.624 is an example for a PMS number
102 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
If this branch exists, you can make use of the TPC performance measurement functionality !
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 103
Successful probes finish with a green rectangle , while failing probes have a red circle !
104 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
Tip #4 : create and run a subsystem performance monitor and set data sample rate
• Each storage subsystem which should provide csv performance data needs to have an appropriate subsystem
performance monitor defined, go to :
IBM TPC
• Disk Manager
• Monitoring
• Subsystem Performance Monitors
and first create the appropriate performance monitor for your specific subsystem:
After having created the performance monitor, check the "Sampling and Scheduling" tab
and define the sampling rate as needed :
Shortest interval length is 5 minutes ( could be 10, 15, 20, 30 or 60 min ).
Leave it at default ( = 5 min ) unless advised by IBM support !
Also : select "Continue indefinitely"
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 105
Now TPC continuously collects performance data for the defined subsystem(s) and
can be used to create performance data csv files !
Tip #6 : check if subsystem pre-reqs are fulfilled
• Check if each storage subsystems fulfil the pre-reqs for performance monitoring here :
IBM TPC
• Configuration Utility
• Disk Manager tab
• Enabled for Performance = =yes=
- SVC L2 and L3 support need SVC specific performance reports, means that for a single SVC cluster the customer
needs to provide a set of 6 different single .csv files with performance data :
1. By Storage Subsystem
2. By Node
3. By IO Group
4. By Managed Disk Group
5. By Managed Disk
6. By Volume
Find the different performance reports in the TPC navigation tree by going to:
IBM TPC
• Disk Manager
• Reporting
• Storage Subsystem Performance
- Select the performance report you would like to generate .csv performance data for by selecting the appropriate
report group, for example "By Node" :
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 107
- In TPC, a right hand pane opens with a subset of performance metrics listed in the right "Included Columns",
means that ALL performance metrics will be included in the to be generated .csv performance data file.
Unless you have special reasons, please leave all performance metrics there!
Don't do this for report types "ByMdisk" and "ByVolume" as the selection panel takes a very long time to update the
view if there are many entries ( like a bunch of mdisks or vdisks ) !!!
- Next important choice is the "Display Historic Performance Data" section. It allows to set the timeframe for
which the performance data for the previously choosen sub-system will be collected and reported ( selecting "Display
latest performance Data" will only report performance data of the very last single sample interval )
108 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
Hint #1 :
Pulling 24 hours of data usually makes sense. If there are many, many volumes, a 24 hour report may be too large to
handle, then try 12 hour or 8 hour selections.
If it is a re-occurring performance impact, placing the worst phase of this impact into the middle of the collection
phase is preferable.
Hint #2 :
There is one tricky aspect to the Date/Time selection. If you select several days, and if you select "From" 8-AM and
"To" 5-PM, you will get only the data between 8 and 5 for each day. If you expect all the data from several days, then
the "From" and "To" times need to be the same, for example if you want all the data from Nov 9 to Nov 11, you
could use this time range ( starting Nov 8 at 23:59, ending at Nov 11 23:59 ) :
- Now select the "Generate Report" button and receive a multi-line multi-sample performance report within the TPC
application : a new tab opens parallel to the
"Selection" tab, here a tab named "Node" as the "By Node" report had been chosen earlier ( reports like "By
Volume" can be quite large ) :
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 109
- Next this table view needs to be saved, therefore select File > Print and print to a .csv file with "Include
Headers" but without "Include Totals" :
- Press "Continue" and save the csv file with a meaningful name :
110 | IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components
Example :
ppppp.bbb.ccc_SvcCluster_ByX_yymmdd.hhmm1-yymmdd.hhmm2.csv
- zip all csv's for a single SVC cluster into an archive an upload to Ecurep
( http://www.ecurep.ibm.com/app/upload ), again in a meaningful format :
ppppp.bbb.ccc_SvcCluster_CSV_yymmdd.hhmm1-yymmdd.hhmm2.zip
For more details of EcuRep see chapter ECuRep procedure on page 8
IBM Storage Area Network Data Collection Guide | Data collection for Virtualisation components | 111
4146 (SAN-FS)
tanktool lslun -l
tanktool lsdrfile -l
tanktool lsimage -l
tanktool lspolicy -l
tanktool lssnmpmgr -l
tanktool lstrapsetting -l
tanktool lsautorestart -l
3. Capture the RSA configuration through the RSA web gui (http://RSA_ip_address). In the web gui on the left panel
click on "Configuration File" or "Server->ASM Control->Configuration File". Then on the right panel click on "view
the current configuration summary".
If you click on this link, a new browser window opens with the configuration content. Copy the content into a text file
and send it to Datacase for analysis as described in section >PFE datacase<
b) SAN.FS clients
1. On an UNIX SAN FS client run /usr/tank/client/bin/pmf.sh. This script generates a directory that contains logs and
configuration files used for debugging SAN FS issues. Please zip this directory and send it to Datacase for analysis as
described in section >PFE datacase<
2. On a Win2k SAN FS client do the following
a. Open a command window and change directory to C:\ProgramFiles\IBM\StorageTank\Client\bin
b. Run "pmf.bat". This script generates a directory that contains logs and configuration files used for debugging SAN
FS issues. Please zip this directory and send it to Datacase for analysis as described in section >PFE datacase<
If files that are too big to be sent via email (more than 10 Mb) , please ftp them to ECuRep see ECuRep procedure
on page 8for detailed description.
In addition, please provide
• A complete SAN layout as described in section Additional information required for data collection on page 11
• detailed history of the problem giving dates and times when known or estimated
• support actions already taken
• time difference between SFSs, hosts, SAN,
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 113
Performance problems
In case of a performance problem, please provide the answers to the following questions:
• Provide a detailed description of the performance problem and the business impact!
114 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
• Provide a description of the whole solution architecture (Application - Server - SAN - Storage), application
characteristics.
• Which DS4000 volumes/ hosts show the performance problems? Please provide a description of the logical and
physical volume layout (vol. mapping server to storage)
• When was the first occurrence of the problem?
What changed in the environment at that time?
• What is the typical occurrence of the problem during daily business hours?
• What facts do indicate that the performance problem is related to the storage subsystem?
• Does the customer require high IOPS or throughput (MB/s)?
• How and where does the customer measure performance? Please provide any available data that shows the
performance problem and reference data that shows good performance.
• What is the criteria for the problem to be considered as solved?
To save the Event Log from the Subsystem Management Window, do the following:
1.Select View>Storage Subsystem profile
3.Then click on Save As and save this file please in text format (PMH#.txt)
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 117
2-Make sure that you "de-select view only Critical Events", and enter 8000 in retrieve most recent events and select
update
3- then click on Select All and then Save As ( please don't forget to de-select "view only critical events")
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 119
Send all files and screen shots to DATACASE as described in section >PFE datacase<
also the complete SAN Layout as described in section Additional information required for data collection on page 11
driveDiagnosticData.txt
majorEventLog.txt
NVSRAMdata.txt
objectBundle
performanceStatistics.csv
persistentReservations.txt
readLinkStatus.csv
recoveryGuruProcedures.html
recoveryProfile.csv
stateCaptureData.dmp
storageSubsystemProfile.txt
unreadableSectors.txt
To collect the information open the Storage Manager and select Advanced -> Troubleshooting -> Collect All Support
Data
120 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
After that specify the name of the zip file and press the Start button.
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 121
Send all files and screen shots to DATACASE as described in section >PFE datacase<
also the complete SAN Layout as described in section Additional information required for data collection on page 11
It is possible that the following additional actionplan will be asked in the PMH
Capture the results and send it to DATACASE as described in section >PFE datacase<
Save the Read Link Status Diagnostics output to a file with the extension csv
Capture the results and send it to DATACASE as described in section >PFE datacase<
Select start, so you will get this warning message, please type yes, this warning is just
To tell you that this option can be performed only via Support recommendation
Capture the results and send it to DATACASE as described in section >PFE datacase<
3. From the remote dialin service menus (rsm-service) select "Support and Debug page"
objectBundle.htm
performanceStatistics.csv
persistentReservations.txt
recoveryGuruProcedures.html
recoveryProfile.csv
sasPhyErrorLogs.csv
socStatistics.csv
stateCaptureData.txt.ss.txt
storageArrayConfiguration.cfg
storageArrayProfile.txt.profile.txt
unreadableSectors.txt
To collect the information open the Storage Manager and select Support -> Gather Support Information (Collect All
Support Data)
After that specify the name of the zip file and press the Start button.
128 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
Send all files and screen shots to DATACASE as described in section >PFE datacase<
DS6000
Datacollection via DSCLI
Note: if the codelevel of the system is below 6.2.2.49, the pepackage may not contain all necessary information for
problem determination on all cases, additional data gathering might be required
Note:
- it is required to transfer pepackages and statesaves to testcase
- it is not sufficient to transfer pepackages to the Austin Decoder only
- do not transfer statesaves to the Austin Decoder - this is useless
Testcase
open a DOS window.
go to the directory you put your windows zip file in
> ftp testcase.software.ibm.com
Login as anonymous
use your email address as password
> bin
> cd/ssd/toibm/sharkdumps
optional: enter prompt off (allows the whole transfer to take place without any prompt for the next file)
> mput *.zip
Create a directory at http://fieldsupport.austin.ibm.com/cgi-bin/pfe/pfe.pl and download the file to it
EcuRep
You can also use EcuRep (See chapter ECuRep procedure on page 8 for details)
NAS General
• What exact symptoms are you experiencing?
• What error or failure indications do you see (messages, beeps, error codes, lights)?
• How often does the problem happen?
• Under what conditions does the problem occur?
• What was happening at the time the problem occurred?
• Has this configuration worked previously? If so, when did it last work and what has changed since then?
• Is this a new install? Upgrade? Configuration change?
• Is the problem re-creatable? What steps do you follow to cause it to occur?
• How did you recover from the problem and what specific actions did you take to recover?
• Do you have a work-around that prevents the problem or reduces its frequency?
• Have you run product diagnostics and what were the results?
• Do the diagnostics indicate which specific component has a problem?
• Check www.ibm.com/storage/support/nas IBM NAS Support home page
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 131
IBM 5190 Total Storage NAS 100 (Obsolete due to EOS 1-1-2007)
I)For data collection use the DSA tool See section DSA (Dynamic System Analysis) for procedure
Connect to the NAS100 over terminal services
1. Install the e-gatherer2 executable in the directory : c:\temp\ egather2.exe
2. Create the following batch file with two lines, called 'egatherer.bat' on the NAS100 desktop (this will call up
egatherer with the option -batch , without the disclaimer screen popping up asking for Y/N input from the keyboard) :
3. Set up the windows scheduler to start the batch-file above by performing the following steps :
• To open Task Scheduler, click Start,
• point to Settings, click Control Panel,
• and then double-click Scheduled Tasks.
• The scheduled task wizard will appear.
• Click on 'browse' and locate the batch- file 'egather.bat' (located at : My Computer - Desktop ) , Click on the
Batch-File to select ,
• click on Open
• One time only , Next
• Check the Start-Time : Set to 5 minutes from now.
• Check the start date : Set to today (default)
• Select : Next
• Enter password for the machine
• Click on 'Finish'
4. E-Gatherer Program will be started at the time selected , (files egathbb8.tmp and gather.dll will appear on the
desktop at the time of execution, and then disappear again) and the output file IBM-...-... (should have the correct
machine-type and -id in the file name) will be created in the C:\temp directory.
5. Send the output file to ECuRep or EMEASAN@ DE.IBM.COM
6. Provide also the complete SAN Layout as described in section Additional information required for data collection
on page 11
II) To get the current software version for an IBM NAS Appliance, perform the following steps:
• Open up the IBM NASAdmin MMC Console
• Click on Software Version in the left-hand panel
The Software Version with a Build number as well as some additional information will be displayed in the right-hand
panel
Currently the machine BIOS for an IBM NAS 100 Appliance, can only be access through the serial port. The access
the BIOS information, perform the following steps:
• Power off your appliance.
• Connect the COM port of the PC to the serial port of the appliance using a 9-pin serial cable (RS-232C female
crossover cable). Connect via HyperTerminal and select the following values in the Properties window ( 115200,
8N1 , hardware flow control).
• Power on the appliance.
• BIOS POST messages should appear on the HyperTerminal screen. When the message Press F1 to enter setup
opens, press F1.
• The message Enter current password opens. Type the password and press Enter.
• Note: The default password is 001san (not case-sensitive).
• The BIOS setup window opens.
• Go to the Advance menu tab and press Enter
132 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
• Use the arrow keys to move the cursor to VPD Data Configuration.
• The VPD Data Configuration window opens, displaying the following VPD data:
• BIOS build date
• BIOS ID
• Machine type
• Model name
III) The IBM NAS 100 Appliance has three types of LEDs to help you identify problems with hardware failure, some
software errors and system status. Two types of LEDs are located on the front bezel and the third type is located on
the back of the chassis. Please refer to the Troubleshooting chapter of the Installation and Service Guide for detailed
information on all of these items.
1. System status LEDs are located on the operator information panel near the power button.
2. Hard disk drive status LEDs are located along the bottom of the bazel
3. Ethernet port status LEDs are located on top of the Ethernet ports (LAN 1 and LAN 2)
Perform the following steps to access hardware errors in the Windows event log by means of Terminal Services:
1. From the Start menu in the PC task bar, click Program -> Administrator Tools -> Event Viewer.
2. From the Tree panel on the left, click Application Log.
3. Click View -> Filter.
4. Type NAS100Svc in the Event Source field and click OK. The Windows event log opens.
Hardware status can also be viewed using the WEB GUI:
1. Access the NAS 100 using the following URL : http://x.x.x.x(NAS100 IP address):8099
2. Log in as Administrator and NAS100 password
3. Go to the Status tab to view current? status
4. To view Event Logs go to Maintenance -> Logs -> (system, application, security...)
To view error messages that occurred during POST, you need to connect to the NAS 100 via :
1. A LAPTOP or desk top using a Serial, HyperTerminal (115200, 8N1, hardware flow control)
2. Power on the Machine and press the F1 key to enter Setup
3. The message Enter current password opens. Type the password and press Enter. Note: The default password is
001san (not case-sensitive).
4. The BIOS setup window opens. Go to the Advance Tab
5. Press the down arrow to the Event Log Configuration
6. Go to View Event Log and press enter
IV) Additional questions:
1. Which kind of ethernet port is used (Static IP or DHCP)
2. Does the NAS 100 respond to the ping command initiated on the Terminal services
3. While connected to the RS232 port,? what messages are reported during power on.
4. What are the Qfixes levels of the applications on the NAS 100
5. The software infos reported on IBM NAS admin MMC :
• click on software version in left hand panel and provide the software version with build number in the right
side panel
V) Send the output to DATACASEas described in section >PFE datacase<
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
IBM 5194 Total Storage NAS 200, 201, 225 and 226 (Obsolete due to EOS
1-1-2008)
• Software Version
To get the current software version for a 5194, perform the following steps:
• Open the IBM NAS Admin MMC Console
• Click on Software Version in the left-hand panel
• The Software Version with a Build number will be displayed in the right side panel.
• Machine BIOS Version
To get the current machine BIOS for a 5194, perform the following steps:
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 133
IBM 5195 Total Storage NAS 300, 301, 325 and 326 (EOS 1-1-2011)
• Software Version
To get the current software version for a 5195, perform the following steps:
• Open the IBM NAS Admin MMC Console
• Click on Software Version in the left-hand panel
• The Software Version with a Build number will be displayed in the right side panel.
• There are several ways to communicate with the 5194. If you have no ethernet connectivity to the machine, try
these:
• Console - plug in a keyboard, mouse, and monitor into the 5195.
• ASM service port - plug in a standard serial cable into the management port and you can view some information
about the 5195
If you do haveconnectivity to the machine, any of these should work:
• IBM Advanced Appliance Configuration Utility
• UM Services (http://machine-name:1411)
• IBM Web-based GUI (http://machine-name:8099)
• Terminal Services Client
For data collection use the DSA tool See section DSA (Dynamic System Analysis) for the procedure
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
IBM 5196 Total Storage NAS 300G G00 and G25 (EOS 1-1-2011)
• Software Version
To get the current software version for a 5196, perform the following steps:
• Open the IBM NAS Admin MMC Console
134 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
Software level :
To identify the level of installed NAS software, provide the output of the commmands :
> Bldlevel
will show the build level of the software installed
>Emgr
will show all e-fixes(if any) installed on the NAS500 in addition to the base build.
NAS configuration :
Core dump :
This will create a compressed file "pmr.country.branch.currentdate.currenttime.tar.Z" in the current directory ( the
first '.' is required to address the current directory as the target, or pmrnumber.branchnumber will be taken for a
directory to be created, and a file currentdate.currenttime.tar.Z will be created in that directory).
The "-U" option gathers userdata , this can take some time.
XIV
x-ray is the tool used by the XIV storage controller to collect system and debug data.
This is similar to the DS6000 "pe_package" collection.
A typical x-ray file name is: system_xray_2810A14MN00043_2008-08-17-1359.tar.bz2
Usage:
Copy xray_collect.exe on a windows machine which can reach the XIV management address.
Open a command-line window and go to the directory where the xray_collect.exe is located and invoke e.g:
xray_collect.exe 9.155.56.101
The xray package will be copied into the directory where you started the utility from.
Upload the x-ray file to PFE datacase as described in >PFE datacase<
IBM 4125 Total Storage IP Storage (all models) (Obsolete due to EOS
1-1-2009)
From the main svc prompt issue the "senddebug" command. This will collect logs and other debug information into a
single tar file that can be sent for analysis.
Target Configuration
• initiator to vlun assignment
• vlun to lun assignment
• lun to disk assignment
• physical disk configuration - expansion boxes used must be specified.
Gigabit Card used (copper or fibre)
Message Log including an oops message if any.
Number of initiators
Initiator Configuration (See below)
Sequence of actions in the initiator(s) that caused the problem.
If an application was used, version number is very important.
138 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
If customer can, he/she should attempt to recreate with debug level 3 and report the logs. I'm assuming here that the
debug level can be set from the GUI, if not serial port.
Target GUI issues
Code Version No (I assume its 1.0 but if a patch is added, then the patched version becomes important)
Enable Java Console, console log
Admin stdout
Admin stderr
Initiator: Windows
1. Description of problem, including how to reproduce
2. Description of machine: make/model/type and number of processors is most important
3. Brand/type of ethernet card used, along with any special settings (jumbo frames?)
4. Description of iSCSI configuration
5. Version number of iSCSI client
6. Any IBM iSCSI messages from the event viewer, or any error messages that appear on the screen
Initiator: Linux
1. Level of the IBM iSCSI Linux Client you are using - check the directory name.
2. PC Type and Model, BIOS version/date
3. IBM only supports Red Hat Linux using kernel 2.2.19-6.2.1 with the current iSCSI Linux initiator. If they do not
have this distribution and kernel, send them to http://www.ibm.com/developerworks/
4. Execute ./iscsidebug.sh to collect machine information
5. Step by step procedure to reproduce the problem. Please include as many details as possible.
6. Re-compile the driver with debug messages turned on.
If you are on a uni-processor machine, you need to create the following directory tree: /tmp/iscsi/trace. (use the
mkdir command)
Please remove the file /var/log/messages or at least use the mv command and give it a different filename.
Reboot your machine.
Edit the Makefile and change debug=n to debug=y.
Type make clean and make.
Copy the new .o file to your 'ibmiscsi-1.1.0' directory
Change to the ibmiscsi-1.1.0 directory.
Enter ./iscsi.sh start to restart the iscsi client and then recreate the problem.
If you are on a uni-processor machine, collect the files which were placed in /tmp/iscsi/trace.
For both uni-processor and smp machines, send us /var/log/messages.
• There are several ways to communicate with the 4125. If you have no ethernet connectivity to the machine, try
these:
Console - plug in a keyboard, mouse, and monitor into the 4125.
ASM service port - plug in a standard serial cable into the management port and you can view some information
about the 4125
If you do have connectivity to the machine, any of these should work:
IBM Advanced Appliance Configuration Utility
IBM Storage Area Network Data Collection Guide | Data collection for Storage components | 139
UM Services (http://machine-name:1411)
IBM Web-based GUI (http://machine-name:8099)
Terminal Services Client
Send the output file to DATACASEas described in section >PFE datacase<
Provide also the complete SAN Layout if applicable as described in section Additional information required for data
collection on page 11
• Click on "Filer" - "Test Autosupport", type in the PMR# as subject and press "send"-button.
140 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
• - switche(s) datacollection
If the gateway cannot boot ONTAP (installation not yet completed) :?
Boot into Maintenance Mode (on CFE prompt 'autoboot' and when filer ask: 'CTR C' to get (1-5) Menue, select '5'
and boot into Maintenance Mode) and provide output of:
> version
> disk show -v
> disk list
> sysconfig -v
> storage show disk -p
> storage show adapter
> fcadmin link_state
> environment show
> storage show port
> aggr status -v
Send the output to EMEASAN@de.ibm.com or ECuRep
Provide also the complete SAN Layout as described in section Additional information required for data collection on
page 11
In addition a parser is available for data uploaded to PFE Datacase:
http://fieldsupport.austin.ibm.com/cgi-bin/pfe/pfe.pl
1a) What performance-affecting behavior are they seeing? (latency values, etc in secs or ms. if errors, then the exact
errors seen, and where)
1b) What is this performance-affecting behavior causing problems with? (applications, resources)
2) What behavior are they expecting to see? How fast is it on normal days, etc. (What is slow? Are reads slow / writes
slow / browsing of file or directories /file creations/mounts? How are you judging "slow"?)
3) When did it start? (Was performance acceptable at one time? / New install and/or performance was never good /
degradation over time? )
4) When does it happen? All the time, only at certain (production) times, etc.
5) Who is it affecting? All users? Users of a particular cifs-based application? Etc
142 | IBM Storage Area Network Data Collection Guide | Data collection for Storage components
6a) What has changed between when it was working fine and now?
6b) Have any new errors or changes been noticed in the filer's autosupports?
7a) Any errors appearing on the filer?
Send the output toEMEASAN@de.ibm.comor ECuRep . Provide also the complete SAN Layout as described in
section Additional information required for data collection on page 11
General
• If intergrated SAN Data Gateway (2108 G07) is installed refer to the 2108 data collection chapter to gather
the data for this component.
• For every tape subsystem we need the adapter device driver level
(AIX: Atape, SUN/HP: Atdd, NT/2k: IBMMag, IBMUltrium, or whatever it is called).
144 | IBM Storage Area Network Data Collection Guide | Appendix A. Special notices
This publication is intended to help readers in troubleshooting Storage Area Network components. The information
in this publication is not intended as the specification of any programming interfaces, since this book does not contain
any kinds of programming interface information.
References in this publication to IBM products, programs or services do not imply that IBM intends to make these
available in all countries in which IBM operates. Any reference to an IBM product, program, or service is not
intended to state or imply that only IBM's product, program, or service may be used. Any functionally equivalent
program that does not infringe any of IBM's intellectual property rights may be used instead of the IBM product,
program or service.
Information in this book was developed in conjunction with use of the equipment specified, and is limited in
application to those specific hardware and software products and also their levels.
IBM may have patents or pending patent applications covering subject matter in this document. The furnishing of
this document does not give you any license to these patents. You can send license inquiries, in writing, to the IBM
Director of Licensing, IBM Corporation, North Castle Drive, Armonk, NY 10504-1785.
Licensees of this program who wish to have information about it for the purpose of enabling: (I) the exchange of
information between independently created programs and other programs (including this one) and (ii) the mutual use
of the information that has been exchanged, should contact IBM Corporation, Dept. 600A, Mail Drop 1329, Somers,
NY 10589 USA.
Such information may be available, subject to appropriate terms and conditions, including in some cases, payment of
a fee.
The information contained in this document has not been submitted to any formal IBM test and is distributed AS IS.
The use of this information or the implementation of any of these techniques is a customer responsibility and depends
on the customer's ability to evaluate and integrate them into the customer's operational environment. While each item
may have been reviewed by IBM for accuracy in a specific situation, there is no guarantee that the same or similar
results will be obtained elsewhere. Customers attempting to adapt these techniques to their own environments do so at
there own risk.