SAN Troubleshooting

Module 11

2006 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice

Objectives
Trace SAN problems Gather troubleshooting data Use switch commands to gather information

Rev. 6.21

HP Restricted

About troubleshooting

Begin at the fabric Determine if the problem is on the storage side or the host side
If a host cannot see a storage device, see if the storage device is logically connected to the switch If the storage can be seen from the switch and the host still cannot see the storage device, there is a problem between the host and the switch
Rev. 6.21 HP Restricted

Tracing problems
1.

Determine what has changed

Has anything new been added/removed/reconfigured? When was it working last?

2.

Analyze physical system setup

Devices attached to switch ports

Diagram system configuration

3.

Gather external switch information

Is the power LED on?

List the LED colors by port

Rev. 6.21

HP Restricted

Analysis
Based on analysis of initial information
Hardware or Software?

Hardware
Run diagnostics as necessary

Software
Review log/traces
portLogShow/Dump supportShow Analyzer traces
HP Restricted

Rev. 6.21

Common problem areas

Rev. 6.21

HP Restricted

Troubleshooting tools

Rev. 6.21

HP Restricted

High availability EVA environment recommendations

In high availability environments under heavy I/O loads
I/O timeouts might occur Use pvchange to increase the IO_timeout value to no more than 60 seconds for LUNs on an EVA 2.0 The increased IO_timeout value enables longer I/O completion times and LUN access delays if a controller failover condition occurs

Rev. 6.21

HP Restricted

Troubleshooting switches

Rev. 6.21

HP Restricted

Gathering troubleshooting data

Finisar, Xyratec trace (FC Analyzer) System configuration information
Switch HBA Storage device

Fibre Channel configuration

Which ports are used? Which type of GBICs? Chassis type and ID Diagram of setup Test results Data patterns on traces Document failure mode Is failure easily reproducible?

Switch trace files using supportShow command Routing outputs using uRouteShow command Front panel errors Port log dump analysis

Switch topology
Describe the network configuration Is the switch cascaded? How many switches are in the configuration? Are there parallel fabrics? Dual host, storage connections?
HP Restricted

Rev. 6.21

Helpful commands (1 of 2)
Name
Error Log Switch Offline Memory Test Port Register Test

Command
errDump switchDisable ramTest portRegTest

Function
Displays the error log without page breaks Sets the switch to offline state necessary to run certain diagnostics Checks CPU RAM. Run offline or online. Checks that the registers and static memory in each ASIC can be successfully accessed. Run offline. Checks that the central memory in each ASIC can be successfully accessed. Run offline. Verifies that control messages can be sent from ASIC to ASIC. Run offline. Verifies CAM functionality. Run offline.

Central Memory Test Control Message Interface Conn Test Content Addressable Memory Test

centralMemoryTes t cmiTest camTest

Rev. 6.21

HP Restricted

10

Helpful commands (2 of 2)

Name

Command

Function
Checks all switch main board/hardware. Frames transmitted are looped and received. Run offline.
Checks al switch paths. Run offline or online. Checks all switch paths. Run offline.

Port Loopback Test portLoopback Test

Cross Port Test Spin Silk Test crossPortTest spinSilk

SRAM Data Retention Test

CMem Data Retention Test

sramRetentionTest
CmemRetention Test

Verifies that data written into ASIC memory is retained. Runs offline.
Verifies that data written into ASIC SRAM is retained. Runs offline.

Rev. 6.21

HP Restricted

11

errShow command

Rev. 6.21

HP Restricted

12

switchshow command

Rev. 6.21

HP Restricted

13

portshow command

Rev. 6.21

HP Restricted

14

portstatsshow command

Rev. 6.21

HP Restricted

15

supportshow command
version uptime tempShow psShow licenseShow diagShow errDump switchShow portFlagsShow portErrShow mqShow portSemShow portShow portRegShow portRouteShow FabricShow TopologyShow qlShow nsShow nsAllShow cfgShow configShow faultShow traceShow portLogDump

Rev. 6.21

HP Restricted

16

Hardware diagnostics commands

fanshow

psshow

tempshow

Rev. 6.21

HP Restricted

17

Interpreting LED activity

Rev. 6.21

HP Restricted

18

Rear panel LEDs

Rev. 6.21

HP Restricted

19

Learning check

Rev. 6.21

HP Restricted

20

Rev. 6.21

HP Restricted

21