Configuring A MetroCluster System With SAS Disk

Configuring a MetroCluster system with SAS disk shelves and
FibreBridge 6500N bridges

FibreBridge 6500N bridges are used in MetroCluster systems whenever SAS disk shelves are used. You can install the bridges
as part of a new MetroCluster installation, as a hot-add to an existing MetroCluster system with SAS or FC disk shelves, or as a
hot-swap to replace a failed bridge. You can also hot-add a SAS disk shelf to an existing stack of SAS disk shelves, and hotremove disk shelves.
This document assumes you are installing a stretch or fabric-attached MetroCluster configuration. For installing a shared-switch
configuration, see the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.
The Best Practices for MetroCluster Design and Implementation TR-3548 document has more information about using bridges.
All documents can be found on the NetApp Support Site at support.netapp.com.
For the most current information about supported platforms, disk shelves, cables, and Data ONTAP releases, see the IBM
System Storage N series Introduction and Planning Guide at the N series support website (accessed and navigated as described
in Websites).
MetroCluster configurations with FibreBridge 6500N bridges do not support the following:
SNMP (Simple Network Management Protocol)

ACP (Alternate Control Path)
SAS optical cables
This document contains the following procedures and information:
Installing bridges and SAS disk shelves as part of a new MetroCluster installation
Hot-adding a stack of SAS disk shelves and bridges to an existing MetroCluster system
Hot-swapping a FibreBridge
Hot-adding a SAS disk shelf to a stack of SAS disk shelves
Hot-removing disk shelves in systems running Data ONTAP 8.2.1 or later
Converting an HA pair or stretch MetroCluster system with SAS disk shelves to a fabric-attached MetroCluster system
Replacing SAS copper cables
Other FibreBridge management interfaces
FibreBridge error messages
MetroCluster configuration examples

The following illustrations are examples of fully cabled stretch and fabric-attached MetroCluster systems.
215-05845_E0
Copyright 2014 NetApp, Inc. All rights reserved.

Web: web.netapp.com Feedback: doccomments@netapp.com
Stretch MetroCluster system

Site 1
HA interconnect
Controller 1
FC
FC1
FC
FC
FC2
Network
Connect each
FibreBridge
management 1 port
to the network
M1
FC
FC2
FibreBridge B
M1
SAS A
FC1
IOM A
First
shelf
FC
FC2
FC
FC2
FibreBridge B
SAS A
M1
FC
FC1
FibreBridge A
Stack of SAS shelves
Last
shelf
SAS A
Controller 2
FC
FC1
FibreBridge A
Site 2
M1
SAS A
IOM B
IOM A
IOM B
First
shelf
Last
shelf
Configuring a MetroCluster system with SAS disk shelves and FibreBridge 6500N bridges
Fabric MetroCluster system

Site 1
Site 2
Controller 1
FC
FC
FC
Controller 2
FC
FC
HA Interconnect
(FC-VI adapter
connections)
FC1
FC switch 2
FC2
FC1
FibreBridge A
Connect each
FibreBridge
management 1 port
to the network
FC
FC
Inter-Switch
Links (ISLs)
FC switch 1
Network
FC
M1
SAS A
FC2
FibreBridge B
M1
SAS A
FC switch 3
FC1
FC2
FC1
FibreBridge A
FC2
FibreBridge B
SAS A
M1

IOM A
FC switch 4
M1
SAS A
IOM B
IOM A
First
shelf
First
shelf
Last
shelf
Last
shelf
IOM B
Installing bridges and SAS disk shelves as part of a new MetroCluster

installation
You can install and cable bridges and SAS disk shelves as part of a new MetroCluster installation.
About this task
This procedure is written with the assumption that you are using the recommended bridge management interfaces: the ATTO
ExpressNAV GUI and ATTO QuickNAV utility.
You use the ATTO ExpressNAV GUI to configure and manage a bridge, and to update the bridge firmware. You use the ATTO
QuickNAV utility to configure the bridge Ethernet management 1 port.
You can use other management interfaces instead, if needed, such as a serial port or Telnet to configure and manage a bridge
and to configure the Ethernet management 1 port, and FTP to update the bridge firmware. If you choose any of these
management interfaces, you must meet the applicable requirements in the Other FibreBridge management interfaces section.
Related references
Other FibreBridge management interfaces on page 30
Preparing for the installation

Preparing to install the bridges as part of your new MetroCluster system involves ensuring that your system meets certain
requirements, including meeting setup and configuration requirements for the bridges. Other requirements include downloading
the necessary documents, the ATTO QuickNAV utility, and bridge firmware.
Before you begin
Your system must be a supported configuration and running a supported version of Data ONTAP.
For fabric-attached MetroCluster configurations, each FC switch must have one FC port available for one bridge to connect
to it.
For stretch MetroCluster configurations, each controller must have one QLogic 8-Gb or 4-Gb FC port available for each
bridge to connect to it.
Only QLogic 8-Gb and 4-Gb ports are supported. These ports can be onboard or on PCIe HBAs. For example, in a
configuration with one stack of SAS disk shelves and two bridges at each site, a total of four FC ports is used on each
controller.
Note: For improved performance, you should use 8-Gb ports when possible.
The computer you are using to set up the bridges must be running an ATTO-supported web browser to use the ATTO
ExpressNAV GUI. The ATTO-supported web browsers are Internet Explorer 8 and 9, and Mozilla Firefox 3.
The ATTO Product Release Notes have an up-to-date list of supported web browsers. You can access this document using
the following information
Steps
1. Download the following documents from the NetApp Support Site at support.netapp.com:
Interoperability Matrix Tool (IMT)

Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode
The Fabric-attached MetroCluster Systems Brocade Switch Configuration Guide or Fabric-attached MetroCluster
Systems Cisco Switch Configuration Guide if you are installing a fabric-attached MetroCluster system.
Hardware and Service Guide for your disk shelf model
2. Go to the ATTO FibreBridge Description page on the NetApp Support Site at support.netapp.com and complete the
following substeps:
a) Navigate to the ATTO FibreBridge Description page by clicking Software, scrolling to Protocol Bridge, choosing
ATTO FibreBridge from the drop-down menu, clicking Go!, and then clicking View & Download.
b) Access the ATTO web site using the link provided and download the following:
ATTO FibreBridge 6500N Installation and Operation Manual
ATTO QuickNAV utility (to the computer you are using for setup)
c) Go to the ATTO FibreBridge 6500N Firmware Download page and do the following:
Navigate to the ATTO FibreBridge 6500N Firmware Download page by clicking Continue at the end of the ATTO
FibreBridge Description page.
Download the bridge firmware file using Steps 1 through 3 of that procedure.
You will update the firmware on each bridge later, when instructed in the Hot-adding the stack of shelves section.
Make a copy of the ATTO FibreBridge 6500N Firmware Download page and release notes for reference when you
are instructed to update the firmware on each bridge.
3. Check the Interoperability Matrix Tool to verify that your system meets all configuration requirements for the bridge.
4. Check the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode to verify that your system
meets all applicable MetroCluster requirements as defined in the MetroCluster installation section.
5. Gather the hardware and information needed to use the recommended bridge management interfacesthe ATTO
ExpressNAV GUI and ATTO QuickNAV utilityby completing the following substeps:
a) Acquire a standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network).
b) Determine a non-default user name and password (for accessing the bridges).
It is recommended that you change the default user name and password.
c) Obtain an IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge.
d) Disable VPN clients on the computer you are using for setup.
Active VPN clients cause the QuickNAV scan for bridges to fail.
6. Acquire four screws for each bridge (to flush-mount the bridge "L" brackets securely to the front of the rack).
The openings in the bridge "L" brackets are compliant with rack standard ETA-310-X for 19-inch (482.6 mm) racks.
Installing the new system

After ensuring that the system meets all the requirements in the Preparing for the installation section, you can install your new
system.
About this task
Multipath HA is the only supported configuration for MetroCluster systems that use bridges.
Both controllers must have access through the bridges to the disk shelves in each stack.
You should use an equal number of disk shelves at each site.
The system connectivity requirements for maximum distances for disk shelves, FC switches, and backup tape devices using
50-micron, multimode fiber-optic cables, also apply to FibreBridges.
The IBM System Storage N series Introduction and Planning Guide has detailed information about system connectivity
requirements.
Steps
1. Properly ground yourself.

2. Install the platforms and, if applicable, the FC switches.
For instructions on installing the platforms, see the Installation and Setup Instructions that came with your platform. For
instructions on installing FC switches, see the Fabric-attached MetroCluster Systems Brocade Switch Configuration Guide
or Fabric-attached MetroCluster Systems Cisco Switch Configuration Guide.
3. Install the disk shelves, power them on, and set the shelf IDs.
For instructions, see the Hardware and Service Guide for your disk shelf model.
Note: You must power-cycle each disk shelf.
Note: Shelf IDs must be unique for each SAS disk shelf within the entire storage system.
4. Install each bridge by completing the following substeps:

a) Secure the "L" brackets on the front of the bridge to the front of the rack (flush-mount) with the four screws you acquired
in the Preparing for the installation section.
For more information and an illustration of the installation, see the ATTO FibreBridge 6500N Installation and Operation
Manual.
b) Connect each bridge to a power source that provides a proper ground.
c) Power on each bridge.
Note: For maximum resiliency, ensure that bridges attached to the same stack of disk shelves are connected to
different power sources.
The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on
self test sequence.
For information about power requirements and LEDs, see the ATTO FibreBridge 6500N Installation and Operation
Manual.
5. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable.
Note: The Ethernet management 1 port enables you to quickly download the bridge firmware (using ATTO ExpressNAV
or FTP management interfaces) and to retrieve core files and extract logs.
6. Configure the Ethernet management 1 port for each bridge by following the procedure in the ATTO FibreBridge 6500N
Installation and Operation Manual, section 2.0.
Note: When running QuickNAV to configure an Ethernet management port, only the Ethernet management port that is
connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2
port, you would need to connect the Ethernet cable to port 2 and run QuickNAV.
7. Configure the bridges.

Be sure to make note of the user name and password that you designate.
a) Configure the data rate/speed of the bridge FC ports.
When you set the bridge FC port speed, you must set it to the appropriate speed for the length of cable you are using: up
to 150 meters is 8 Gb, up to 270 meters is 4 Gb, up to 500 meters is 2 Gb.
The most current information on supported distance can be found in the Interoperability Matrix Tool on the NetApp
Support Site at support.netapp.com.
If you have a fabric-attached MetroCluster system, the following applies:
If you are using the Cisco 9222i switch, you must set the FC port on each bridge to the same speed as the Cisco
9222i switch port to which the bridge will connect.
For example, if the switch port is 8 Gb, you need to set the bridge port to 8 Gb. If the switch port is 4 Gb, you need to
set the bridge port to 4 Gb.
If you are using a Brocade switch, the bridge port must be set to 8 Gb.
b) Configure the connection mode that the bridges use to communicate across the FC network.
If you have a fabric-attached MetroCluster system, you must set the bridge connection mode to ptp (point-to-point).
If you have a stretch MetroCluster system, you must set the bridge connection mode to loop.
Note: If you are connecting bridges to an X1143A FC adapter in the controller, you must set the bridge connection
mode to ptp.
For example, if you were to use the command line interface (CLI) to set the bridge FC 1 port speed to 8 Gb and the
connection mode to ptp, you would enter the following three commands; the third command saves the configuration
changes:
Set FCDataRate 1 8Gb
Set FCConnMode 1 ptp
SaveConfiguration
Result: You are prompted to restart the bridge.

The ATTO FibreBridge 6500N Installation and Operation Manual has the most current information on available commands
and how to use them.
8. Update the firmware on each bridge to the latest version by following the instructionsstarting with Step 4on the
FibreBridge 6500N Download page.
9. Cable the disk shelves to the bridges by completing the following substeps:
a) Daisy-chain the disk shelves in each stack.
For information about daisy-chaining disk shelves, see the Installation and Service Guide for your disk shelf model.
b) For each stack of disk shelves, cable IOM A square port of the first shelf to SAS port A on FibreBridge A.
c) For each stack of disk shelves, cable IOM B circle port of the last shelf to SAS port A on FibreBridge B.
Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, and
bridge B connects to the B-side of the stack through the last shelf.
Note: The bridge SAS port B is disabled.
The following illustration shows a set of bridges cabled to a stack of three disk shelves:
FC1
FC2
FibreBridge A
FibreBridge B
SAS A
M1
FC2
FC1
SAS A
M1

IOM A
IOM B
First
shelf
Last
shelf
10. Verify that each bridge can detect all disk drives and disk shelves it is connected to.
If you are using the...
Then...
ATTO ExpressNAV GUI a.
In a supported web browser, enter the IP address of a bridge in the browser box.
You are brought to the ATTO FibreBridge 6500N home page, which has a link.
b.
Click the link and enter your user name and the password that you designated when you configured the
bridge.
The ATTO FibreBridge 6500N status page appears with a menu to the left.
c.
Click Advanced in the menu.
d.
Enter the following command, and then click Submit:

sastargets
Serial port connection
Enter the following command:

sastargets
Example
The output shows the devices (disks and disk shelves) that the bridge is connected to. Output lines are sequentially
numbered so you can quickly count the devices. For example, the following output shows that 10 disks are connected.
Tgt
0
1
2
3
4
5
6
7
8
9
VendorID
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
ProductID
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
Type
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
SerialNumber
3QP1CLE300009940UHJV
3QP1ELF600009940V1BV
3QP1G3EW00009940U2M0
3QP1EWMP00009940U1X5
3QP1FZLE00009940G8YU
3QP1FZLF00009940TZKZ
3QP1CEB400009939MGXL
3QP1G7A900009939FNTT
3QP1FY0T00009940G8PA
3QP1FXW600009940VERQ
Note: If the text response truncated appears at the beginning of the output, you can use Telnet to connect to the
bridge and enter the same command to see all the output.
11. Verify the command output shows the bridge is connected to all disks and disk shelves in the stack that it is supposed to be
connected to.
If the output is...
Then...
Correct
Repeat Step 10 for each remaining bridge.
Not correct
a.
Check for loose SAS cables or correct the SAS cabling by repeating Step 9.
b.
Repeat Step 10.
12. Cable each bridge to the controllers or FC switches, depending on your MetroCluster configuration:
If you have a...
Then...
Stretch MetroCluster a.
configuration
b.
Fabric-attached
MetroCluster
configuration
Cable FC port 1 of bridge A to an 8-Gb or 4-Gb FC port on the controller at Site 1.

Cable FC port 2 of bridge A to the same speed FC port on the controller at Site 2.
c.
Repeat Substeps a and b for bridge B.
a.
Cable FC port 1 of bridge A to FC switch 1 at Site 1.
b.
Cable FC port 1 of bridge B to FC switch 2 at Site 1.
c.
Repeat Substeps a and b for Site 2.

Note: You can connect the bridge to any available port on the FC switch. Whichever port you choose, you
must use the same port consistently on each FC switch that you are connecting at both sites. This is for ease of
manageability. For example, if you choose FC switch port 3, use only that port on each FC switch.
Note: The FC port 2 on each bridge remains unused.
13. Connect and configure the controllers, and if applicable the FC switches, following the procedure for your MetroCluster
configuration in the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode:
a) If applicable, configure the FC switches.
Note: Each FC switch port connected to the bridge must be set to G_Port.
b) Cable the FC-VI and inter-switch link as appropriate for your configuration.
c) Check whether your system has disk autoassignment enabled by entering the following command at the console of either
controller.
options disk.auto_assign
If disk autoassignment is enabled, the output shows disk.auto_assign on.

d) If your system does not have disk autoassignment enabled or if disk drives in the same stack are owned by both
controllers, assign disk drives to the appropriate pools by using the information in the Data ONTAP High Availability
and MetroCluster Configuration Guide for 7-Mode; otherwise, go to the next step.
Note: If you are splitting a single stack of disk shelves between two controllers, you must disable disk autoassignment
before assigning disk ownership; otherwise, when you assign any single disk drive, the remaining disk drives might be
automatically assigned to the same controller and pool.
To disable disk autoassignment, run the options disk.auto_assign off command at the console of both
controllers.
e) Configure the system and enable licenses as needed using the information about configuring an HA pair in the Data
ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.
14. Verify disk shelf connectivity by entering the following command at the system console of either controller:
sysconfig -v
The output shows the following:
Each bridge on a separate line. Additionally, for fabric-attached MetroCluster configurations, each bridge appears under
each FC port to which it is visible. For stretch MetroCluster configurations, each bridge is directly connected.
For example, the output for a fabric-attached MetroCluster configuration would look like the following:
FC-to-SAS Bridge:
Brocade300-1:9.126L0
: ATTO
: ATTO
FibreBridge6500N 1.11
FB6500N100189
FB6500N100162
For example, the output for a stretch MetroCluster configuration would look like the following:
FC-to-SAS Bridge:
125L0
: ATTO
125L0
: ATTO
FB6500N100053
FB6500N100048
Each disk shelf is on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroCluster
configurations).
For example:
Shelf
Shelf
0: IOM6
1: IOM6
Firmware rev. IOM6 A: 0111 IOM6 B: 0111

Each disk drive is on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroCluster
configurations).
: NETAPP
: NETAPP
X421_HCOBD450A10 NA01 418.0GB (879097968 520B/sect)

125L1
125L2
: NETAPP
: NETAPP
X421_HCOBD450A10 NA01
40.0GB (82262416 520B/sect)

40.0GB (82262416 520B/sect)
Related concepts
FibreBridge error messages on page 31
Hot-adding a stack of SAS disk shelves and bridges to an existing

MetroCluster system
You can hot-add a stack of SAS disk shelves and set of bridges to an existing MetroCluster system that has SAS or FC disk
shelves.
About this task
Related references
Preparing for the hot-add

Preparing to hot-add a stack of SAS disk shelves and bridges involves downloading documents as well as the disk drive and
disk shelf firmware.
Before you begin
All disk drives and disk shelves in the system must be running the latest firmware version.
For fabric-attached MetroCluster configurations, each FC switch must have one FC port available for one bridge to connect
to it.
Note: If you are hot-adding to a system that contains FC disk shelves, you might need to upgrade the FC switch
depending on the FC switch compatibility.
For stretch MetroCluster configurations, each controller must have one QLogic 8-Gb or 4-Gb FC port available for each
bridge to connect to it.
Only QLogic 8-Gb and 4-Gb ports are supported. These ports can be onboard or on PCIe HBAs. For example, in a
configuration with one stack of SAS disk shelves and two bridges at each site, a total of four FC ports are used on each
controller.
Note: For improved performance, you should use 8-Gb ports when possible.
The computer you are using to set up the bridges must be running an ATTO supported web browser to use the ATTO
ExpressNAV GUI: Internet Explorer 8 or 9, or Mozilla Firefox 3.
the information in the steps.
Steps
Interoperability Matrix Tool

following substeps:
10
3. Check the Interoperability Matrix Tool to verify that the disk shelf you are hot-adding is supported.
4. Download the latest disk and disk shelf firmware:
Note: In this step you are only completing the download portion of the instructions provided in the links. You also need to
copy the installation portion of the instructions in the links for use later. The installation portion of the instructions is
found in the Hot-adding the stack of shelves section.
a) Download the disk firmware from support.netapp.com/NOW/download/tools/diskfw and make a copy of the disk
firmware instructions for reference later.
b) Download the disk shelf firmware from support.netapp.com/NOW/download/tools/diskshelf and make a copy of the disk
shelf firmware instructions for reference later.
6. Acquire four screws for each bridge (to flush-mount the bridge "L" brackets securely to the front of the rack).
Hot-adding the stack of shelves

After ensuring that the system meets all the requirements in the Preparing for the hot-add section, you can hot-add a stack of
SAS disk shelves and bridges.
About this task
This is a nondisruptive procedure if all bridge requirements in the Interoperability Matrix Tool are met.
Multipath HA is the only supported configuration for MetroCluster systems using bridges.
You should hot-add an equal number of disk shelves at each site.
50-micron, multimode fiber-optic cables also apply to bridges.
requirements.
Attention: Hot-adding a disk shelf requires you to update the disk drive firmware on the hot-added disk shelf by running the
disk_fw_update command. Running this command can be disruptive if the firmware on existing disk drives in your system
is an older version.
Steps

2. Follow Steps 3 through 11 of the Installing bridges and SAS disk shelves as part of a new MetroCluster installation section
of this document.
3. Check whether your system has disk autoassignment enabled by using the options disk command at the console of either
controller.
11
If disk autoassignment is enabled, the output shows disk.auto_assign on.

4. If your system does not have disk autoassignment enabled, assign disk drive ownership by using the information in the Data
ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.
controllers.
Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware has been updated.
5. Update the disk drive firmware to the most current version by entering the following command at the system console:
disk_fw_update
You must run this command on both controllers.

If needed, refer to the instructions you copied when you downloaded the firmware as part of the preparation for this task.
6. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded the
firmware as part of the preparation for this task.
You can run the commands in the procedure from the system console of either controller.
7. If applicable, repeat this procedure for the remote site.
Related concepts
When a bridge fails, you can hot-swap the bridge without disrupting the storage system when the system is cabled correctly for
Multipath HA.
About this task
Related references
Preparing for the hot-swap

Preparing to hot-swap a bridge as part of your new MetroCluster system involves downloading documents and bridge firmware.
Steps
1. Verify that the system is cabled for Multipath HA by entering the following command at the system console of both
controllers:
12
sysconfig
Note: It might take up to a minute for the system to complete discovery.
For example, the following output states that the system is cabled for Multipath HA:
NetApp Release 8.1 7-Mode: Thu Sep 22 02:50:01 PDT 2011
System ID: 1231231231 (stqa-3270-2); partner ID: 4564564564 (stqa-3270-1)
System Serial Number: 700000123123 (stqa-3270-2); partner Serial Number: 700000456456
(stqa-3270-1)
System Rev: B0
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA
2. Verify that the command output shows that the system is cabled for Multipath HA.
If the System Storage
Configuration states that it is...
Then...
Multipath HA
Go to Step 3.
Mixed-path or single-path HA
You must correct the SAS and FC cabling because not all disk drives are accessible through the new
bridge.
Attention: If you must correct cabling on a system that is serving data, you can perform hotcable moves only if one path to the disk drives is maintained.
Attention: When bridges are not cabled as Multipath HA, restarting a bridge might cause loss of
access to the disk drives and result in a multi-disk panic.
following substeps:
4. Complete the following substeps to use the recommended bridge management interfaces: the ATTO ExpressNAV GUI and
ATTO QuickNAV utility:
a. Determine a non-default username and password (for accessing the bridge).
It is recommended that you change the default username and password.
b. Disable VPN clients on the computer you are using for setup.
Note: For the new bridge, use the IP address, subnet mask, and gateway information for the Ethernet management 1 port
from the failed bridge..
13
Hot-swapping the FibreBridge

After ensuring that the system meets all the requirements in the Preparing for the hot-swap section, you can hot-swap a bridge.
Steps

2. Power off the failed bridge and disconnect it from the power source.
3. Disconnect the cables connected to the failed bridge.
You should make note of the port each cable was connected to.
4. Remove the failed bridge from the rack.
5. Install the new bridge into the rack.
6. Reconnect the power cord and Ethernet cable.
Note: Do not reconnect the SAS or FC cables at this time.
7. Connect the bridge to a power source and turn it on.

The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on self
test sequence.
connected by the Ethernet cable is configured. For example, if you also wanted to configure the Ethernet management 2

mode to ptp.
changes:
14

SaveConfiguration

10. Update the firmware on the bridge to the latest version by following the instructionsstarting with Step 4on the
11. Reconnect the SAS and FC cables to the same ports on the new bridge.
12. Verify that each bridge can see all disk drives and disk shelves it is connected to.
Then...
In a supported web browser, enter the IP address of the bridge in the browser box.
b.
bridge.
c.
d.

sastargets

sastargets
Example
numbered so that you can quickly count the devices. For example, the following output shows that 10 disks are connected:
Tgt
0
1
2
3
4
5
6
7
8
9
VendorID
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
ProductID
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
Type
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
SerialNumber
3QP1G3EW00009940U2M0
Note: If the text response truncated appears at the beginning of the output, you can use Telnet to access the bridge
and enter the same command to see all the output.

13. Verify that the command output shows that the bridge is connected to all disks and disk shelves in the stack it is supposed to
be connected to.
If the output is...
Then...
Correct
Repeat Step 12 for each remaining bridge.
15
If the output is...
Then...
Not correct
a.
Check for loose SAS cables or correct the SAS cabling by repeating Step 11.
b.
Repeat Step 12.
14. Verify that both controllers have access through the new bridge to the disk shelves (that the system is cabled for Multipath
HA), by entering the following command at the system console of both controllers:
sysconfig
For example, the following output states that the system is cabled for Multipath HA:
NetApp Release 8.1 7-Mode: Thu Sep 22 02:50:01 PDT 2011
System ID: 1231231231 (stqa-3270-2); partner ID: 4564564564 (stqa-3270-1)
System Serial Number: 700000123123 (stqa-3270-2); partner Serial Number: 700000456456
(stqa-3270-1)
System Rev: B0
System Storage Configuration: Multi-Path HA
System ACP Connectivity: NA
15. Verify that the command output shows that the system is cabled for Multipath HA.
If the System Storage
Configuration states that it is...
Then...
Multipath HA
You are done.
Mixed-path or single-path HA
You must correct the SAS and FC cabling because not all disk drives are accessible through the new
bridge.
Attention: If you must correct cabling on a system that is serving data, you can perform hotcable moves only if one path to the disk drives is maintained.
Attention: When bridges are not cabled as Multipath HA, restarting a bridge might cause loss of
access to the disk drives and result in a multi-disk panic.
16. Return the failed part to NetApp as described in the RMA instructions shipped with the kit. Contact technical support at
support.netapp.com, 888-463-8277 (North America), 00-800-44-638277 (Europe), or +800-800-80-800 (Asia/Pacific) if you
need the RMA number or additional help with the replacement procedure.
Related concepts

You can hot-add one or more SAS disk shelves to a stack of SAS disk shelves.
Preparing for the hot-add

Preparing to hot-add a disk shelf involves downloading documents as well as the disk drive and disk shelf firmware.
Before you begin
16
The disk shelf you are installing must be supported.

All disk drives and disk shelves in the system must be running the latest firmware version.
Steps

2. Download the latest disk and disk shelf firmware:

Note: In this step, you are only completing the download portion of the instructions provided in the links. You also need
to copy the installation portion of the instructions in the links for use later. The installation portion is found in the Hotadding the shelf section.
a) Download the disk firmware from support.netapp.com/NOW/download/tools/diskfw and make a copy of the disk
firmware instructions for reference later.
b) Download the disk shelf firmware from support.netapp.com/NOW/download/tools/diskshelf and make a copy of the disk
shelf firmware instructions for reference later.
3. Check the Interoperability Matrix Tool to verify that the disk shelf you are hot-adding is supported.
Hot-adding the shelf

After ensuring that the system meets all the requirements in the Preparing for the hot-add section, you can hot-add a disk shelf.
About this task
This procedure is for hot-adding a disk shelf to the logical last disk shelf in a stack.
The logical last disk shelf in a stack is connected from IOM B to bridge B.
This is a nondisruptive procedure.
You should hot-add an equal number of disk shelves at each site.
If you are hot-adding more than one disk shelf, you must hot-add one disk shelf at a time.
Attention: Hot-adding a disk shelf requires you to update the disk drive firmware on the hot-added disk shelf by running the
disk_fw_update command. Running this command can be disruptive if the firmware on existing disk drives in your system
is an older version.
Steps

2. Install and cable the disk shelf by completing the following substeps:
a) Install the disk shelf, power it on, and set the shelf ID.
For instructions, see the Hardware and Service Guide for your disk shelf model.
Note: You must power-cycle the disk shelf.
Note: Shelf IDs must be unique for each SAS disk shelf within the entire storage system.
b) Disconnect the SAS cable from the IOM B circle port of the last shelf in the stack and reconnect it to the same port in the
new shelf.
The other end of this cable remains connected to bridge B.
c) Daisy-chain the new disk shelf by cabling the new shelf IOM square ports (of IOM A and IOM B) to the last shelf IOM
circle ports (of IOM A and IOM B).
For an example of a cabled stack of disk shelves, you can reference any of the illustrations in this document.
3. Verify disk shelf connectivity by entering the following command at the system console of either controller:
sysconfig -v
17
The output shows the following:
Each bridge on a separate line and, for fabric-attached MetroCluster configurations, under each FC port to which it is
visible; otherwise, for stretch MetroCluster configurations, directly connected.
FC-to-SAS Bridge:
: ATTO
: ATTO
FB6500N100189
FB6500N100162
FC-to-SAS Bridge:
125L0
: ATTO
125L0
: ATTO
FB6500N100053
FB6500N100048
Each disk shelf on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroCluster
configurations).
For example:
Shelf
Shelf
0: IOM6
1: IOM6

Note: Depending on the version of Data ONTAP your system is running, the output might appear differently.
Each disk drive on a separate line under each FC port to which it is visible (for fabric-attached or stretch MetroCluster
configurations).
: NETAPP
: NETAPP

125L1
125L2
: NETAPP
: NETAPP
40.0GB (82262416 520B/sect)

40.0GB (82262416 520B/sect)
4. Check that your system has disk autoassignment enabled by entering the following command at the console of either
controller:
If disk autoassignment is enabled, the output shows options disk.auto_assign on.

5. If your system does not have disk autoassignment enabled or if disk drives in the same stack are owned by both controllers,
assign disk drives to the appropriate pools by using the information in the Data ONTAP High Availability and MetroCluster
Configuration Guide for 7-Mode; otherwise, go to the next step.
controllers.
Note: Do not add disk drives to aggregates or volumes until after the disk drive and disk shelf firmware has been updated.
6. Update the disk drive firmware to the most current version by entering the following command at the system console:
disk_fw_update
18
You must run this command on both controllers.

If needed, refer to the instructions you copied when you downloaded the firmware as part of the preparation for this task in
the Preparing for the hot-add section.
7. Update the disk shelf firmware to the most current version by using the instructions you copied when you downloaded the
firmware as part of the preparation for this task.
You can run the commands in the procedure from the system console of either controller.
8. If you are hot-adding more than one disk shelf, repeat Steps 1 through 7 for each disk shelf that you are hot-adding.

If your system is running Data ONTAP 8.2.1 or later, you can hot-remove disk shelvesphysically remove disk shelves that
have had the aggregates removed from the disk drivesin a multipath HA stretch or fabric MetroCluster configuration with
FibreBridge 6500N bridges that is up and serving data. You can hot-remove one or more disk shelves from anywhere within a
stack of disk shelves or remove a stack of disk shelves.
Before you begin
Your storage system must be running Data ONTAP 8.2.1 or later.

Your storage system must be a multipath HA system.
You must have already removed all aggregates from the disk drives in the disk shelves you are removing.
Attention: Do not attempt this procedure unless you have already removed all aggregates from the disk shelves you are
removing.
For MetroCluster configurations, removing aggregates involves splitting the mirrored aggregates on the disk shelves you are
removing and then re-creating the mirrored aggregates with another set of disk drives. For procedures, see Splitting a
mirrored aggregate and Rejoining split aggregates in the Data ONTAP Data Protection Online Backup and Recovery
Guide for 7-Mode.
As a best practice, you should remove disk drive ownership after you remove the aggregates from the disk drives in the disk
shelves you are removing.
Note: This procedure follows the best practice of removing disk drive ownership; therefore, steps are written with the
assumption that you have removed disk drive ownership.
For information about removing disk drive ownership, see the Removing ownership from a disk procedure in the Storage
Management Guide for 7-Mode. This document is available on the NetApp Support Site at support.netapp.com.
Note: The procedure for removing ownership from disk drives requires you to disable disk autoassignment. You reenable
disk autoassignment when prompted at the end of this shelf hot-remove procedure.
Multipath HA configurations cannot be in a takeover state.

If you are removing one or more disk shelves from within a stack, you must have factored the distance to bypass the disk
shelves you are removing; therefore, if the current cables are not long enough, you need to have longer cables available.
The IBM System Storage N series Introduction and Planning Guide at the N series support website (accessed and navigated
as described in Websites) contains information about supported SAS cables.
About this task
This procedure follows cabling best practices; therefore, references to IOMs and IOM SAS square and circle ports align with
the best practices. If your storage system is cabled differently from what is prescribed as best practice, the IOM and/or IOM
ports might be different.
Path A refers to the A-side disk shelf I/O module (IOM A) located in the top or left of the disk shelf depending on your disk
shelf model.
Path B refers to the B-side disk shelf I/O module (IOM B) located in the bottom or right of the disk shelf depending on your
disk shelf model.
The first disk shelf in the stack is the disk shelf with a SAS square port directly connected to a bridge.
19
The interim disk shelf in the stack is the disk shelf directly connected to other disk shelves in the stack.
The last disk shelf in the stack is the disk shelf with a SAS circle port directly connected to a bridge.
The next disk shelf is the disk shelf downstream of the disk shelf being removed, in depth order.
The previous disk shelf is the disk shelf upstream of the disk shelf being removed, in depth order.
After completing this procedure for the local site, you are instructed to repeat this procedure for the remote site if you are
also removing disk shelves for the remote site.
Steps
1. Verify that your system configuration is Multi-Path HA by entering the following command from the console of either
controller:
sysconfig
It might take up to a minute for the system to complete discovery.

The configuration is listed in the System Storage Configuration field.
Attention: If your system configuration is shown as something other than Multi-Path HA, you cannot continue with this
procedure. Your system must meet the prerequisites stated in the Before you begin section of this procedure.
2. Verify that no aggregates are on the disk drives in the disk shelves you are removing, by completing the following substeps:
a) Enter the following command at the console of either controller:
aggr status -r
If you have traditional volumes, you enter the vol status -r command.
b) Check the output SHELF column to verify that the disk shelves from which you have removed aggregates do not appear.
Attention: If any disk drives in the disk shelves you are removing are anything other than spares or failed, you cannot
continue with this procedure. Your system must meet the prerequisites stated in the Before you begin section of this
procedure.
Example
The following aggr status -r output is for a stretch MetroCluster system with disk shelf 3 being removed. Because
aggregates have been removed from disk shelf 3, only disk shelves in the stack with aggregates (disk shelves 2 and 4) appear
in the SHELF column:
fas6200> aggr status -r
RAID Disk
Device
blks)
--------------------------...
dparity
2c.125L3
847884/1736466816
parity
2c.125L12
847884/1736466816
data
2c.125L15
423946/868242815
...
HA
SHELF BAY CHAN Pool Type
RPM
Used (MB/blks)
Phys (MB/
------------- ---- ---- ---- ----- -------------2c
FC:A
BSAS
7200 847555/1735794176
2c
11
FC:A
BSAS
7200 847555/1735794176
2c
14
FC:A
BSAS
7200 423111/866531584
3. Verify that ownership is removed from the disk drives on the disk shelves you are removing:
a) Enter the following command at the console of either controller:
disk show -v
b) Check the output and take the appropriate action:
20
If the output shows...
Then...
Not Owned or FAILED
Go to the next step.

The disk drives in the disk shelves you are removing are in the correct state.
Any disk drives in the disk

shelves you are removing
have ownership
You can use the Removing ownership from a disk procedure referenced in the Before
you begin section of this procedure.
Example
The following output for the disk show -v command on a stretch MetroCluster system shows that disk drives on the
disk shelf being removed (disk shelf 3) are inOWNER a correct state for removal. Two disk drives have the ownership
removed; therefore, Not Owned appears in the column. Two disk drives are failed; therefore, FAILED appears in the
POOL column:
fas6200> disk show -v
DISK
OWNER
CHKSUM
---------- ------------------...
2c.125L24
Not Owned
Block
2b.125L27
Not Owned
Zoned
2c.125L43
dg-8020-04(536871573)
Block
2c.125L9
dg-8020-03(536871641)
Block
POOL
SERIAL NUMBER
DR HOME
-----
-------------
-------------
NONE
N11DNLML
dg-8020-03(536871641)
NONE
9QMC2777
dg-8020-04(536871573)
FAILED
9QMCPPWP
dg-8020-04(536871573)
FAILED
9QMCDZQZ
dg-8020-03(536871641)
4. Physically identify and note the disk shelf IDs on the front of the disk shelves you are removing.
You need to be certain of which disk shelves you are removing so that you can correctly recable path A and path B later in
this procedure.
5. If you are removing an entire stack of disk shelves, complete the following substeps; otherwise, go to the next step:
a) Remove all SAS cables on path A and path B.
This includes bridge-to-shelf cables and shelf-to-shelf cables for all disk shelves in the stack you are removing.
b) Go to Step 9.
6. If you are removing one or more disk shelves from a stack (but keeping the stack), recable the applicable path A stack
connections to bypass the disk shelves you are removing by completing the applicable set of substeps:
If you are removing more than one disk shelf, complete the applicable set of substeps one disk shelf at a time.
21
If you are
removing...
Then...
The first disk shelf

in a stack
For path A, do the following:

a.
Remove the cable connecting the IOM A SAS circle port of the first disk shelf and the IOM A SAS square port
of the second disk shelf in the stack and set it aside.
b.
Move the cable connecting the bridge to the IOM A SAS square port of the first disk shelf to the IOM A SAS
square port of the second disk shelf in the stack.
Later, when you are instructed to repeat this procedure for path B, do the following:
a.
Remove the cable connecting the IOM B SAS circle port of the first disk shelf and the IOM B SAS square port
of the second disk shelf in the stack and set it aside.
b.
Go to the next step in this procedure.

You have completed the recabling for path B.
An interim disk
shelf in a stack
a.
Remove the cable connecting the IOM A SAS circle port of the disk shelf being removed and the IOM A SAS
square port of the next disk shelf in the stack and set it aside.
b.
Move the cable connecting the IOM A SAS square port of the disk shelf being removed to the IOM A SAS
square port of the next disk shelf in the stack.
Later, when you are instructed, repeat these substeps for path B.
The last disk shelf
in a stack

a.
Remove the cable connecting the IOM A SAS square port of the last disk shelf and the IOM A SAS circle port
of the previous disk shelf in the stack and set it aside.
b.
Go to the next step in this procedure.

You have completed the recabling for path A.
Later, when you are instructed to repeat this procedure for path B, do the following:
a.
Remove the cable connecting the IOM B SAS square port of the last disk shelf and the IOM B SAS circle port
of the previous disk shelf in the stack and set it aside.
b.
Move the cable connecting the bridge to the IOM B SAS circle port of the last disk shelf to the IOM B SAS
circle port of the previous disk shelf in the stack.
7. Verify that the cabling on path A has successfully bypassed the disk shelves you are removing and all disk drives on the disk
shelves you are removing are still connected through path B, by entering the following command from the console of either
controller:
storage show disk -p
It might take up to a minute for the system to complete discovery.

Example
In this example of how the output should appear for a stretch MetroCluster system, the disk shelf being removed is disk shelf
3. One line item appears for each disk drive connected through path B (now the primary path); therefore, the disk drives are
listed in the PRIMARY column and B appears in the first PORT column. There is no connectivity through path A for any of the
disk drives in the disk shelf being removed; therefore, no information is shown in the SECONDARY or second PORT columns:
fas6200> storage show disk -p
PRIMARY
-------2b.125L27
2b.125L28
2b.125L3
2b.125L30
2b.125L31
22
PORT
---B
B
B
B
B
SECONDARY PORT SHELF BAY

--------- ---- --------3
0
3
1
3
2
3
3
3
4
2b.125L6
2b.125L7
2b.125L34
2b.125L36
2b.125L37
2b.125L38
2b.125L39
2b.125L43
2b.125L44
2b.125L45
...
B
B
B
B
B
B
B
B
B
B
3
3
3
3
3
3
3
3
3
3
5
6
7
8
9
10
11
12
13
14
Attention: If the output shows anything other than all the disk drives connected only through path B, you must correct the
cabling by repeating Step 6.

8. Complete the following substeps:
a) Repeat Step 6 and Step 7 for path B.
b) Repeat Step 1 to confirm that your system configuration is the same as before you began this procedure.
c) Go to the next step.
9. If, when you removed ownership from the disk drives as part of the preparation for this procedure, you disabled disk
autoassignment, then reenable disk autoassignment by entering the following command; otherwise, go to the next step:
options disk.auto_assign on
Enter the applicable command from the console of each controller.

10. Power off the disk shelves you disconnected and unplug the power cords from the disk shelves.
11. Remove the disk shelves from the rack or cabinet.
To make disk shelves lighter and easier to maneuver, remove the power supplies and IOMs. Avoid removing the disk drives
if possible, because excessive handling can lead to internal damage.
It is recommended that you use a mechanical hoist or lift if you are moving a fully populated 4U disk shelf. For example, a
fully populated DS4243 or DS4246 disk shelf can weigh approximately 110 lbs (49.9 kg).
12. If you are removing disk shelves at the remote site, repeat this procedure.
Converting an HA pair or stretch MetroCluster system with SAS disk

shelves to a fabric-attached MetroCluster system
You can convert an HA pair or stretch MetroCluster system that has SAS disk shelves to a fabric-attached MetroCluster system.
About this task
Related references
23
Preparing for the conversion

Preparing to convert an HA pair or stretch MetroCluster system with SAS disk shelves to a fabric-attached MetroCluster system
involves ensuring that your system meets certain requirements. These requirements include meeting setup and configuration
requirements for the bridges, for downloading documents, the ATTO QuickNAV utility, and bridge firmware.
Before you begin
The computer you are using to set up the bridges must be running an ATTO-supported web browser to use the ATTO
ExpressNAV GUI: Internet Explorer 8 or 9, or Mozilla Firefox 3.
the information in the steps.
If you are converting an HA pair, you must have two bridges for each stack of shelves in your configuration.
Steps
TR-3517, MetroCluster Upgrade Planning Guide

Fabric-attached MetroCluster Systems Brocade Switch Configuration Guide or Fabric-attached MetroCluster Systems
Cisco Switch Configuration Guide
following substeps:
3. Check the Interoperability Matrix Tool to verify that your system meets all configuration requirements for the bridge.
4. Check the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode to verify that your system
meets all applicable MetroCluster system requirements as defined in the MetroCluster system section.
24
6. Acquire four screws for each bridge to flush-mount the bridge "L" brackets securely to the front of the rack.
7. Update the following to the latest versions for each system as described in the Data ONTAP Upgrade and Revert/
Downgrade Guide for 7-Mode:
Data ONTAP
Note: Both controllers must be running the same supported Data ONTAP version.
Storage system firmware

Disk shelf firmware
Disk drive firmware
8. If you have a chassis with two controllers, complete the following substeps so that there is only one controller in each
chassis:
a) Move one controller to a new chassis to form the partner node of the MetroCluster system.
b) Obtain and install the FC-VI interconnect card on both systems.
For details about the conversion process, see TR-3517, MetroCluster Upgrade Planning Guide.
Converting the system

After ensuring that the system meets all the requirements in the Preparing for the conversion section, you can convert your HA
pair or stretch MetroCluster system.
About this task
If you have a standard HA pair, you must make it a mirrored HA pair (in Step 19), which requires you to add disk shelves,
disable disk autoassignment, assign the new disk drives to pool 1, and then reboot each controller.
Multipath HA is the only supported configuration for MetroCluster systems using bridges.
You should use an equal number of disk shelves at each site.
You must have two bridges for each stack of disk shelves at each site.
Each FC switch must have one FC port available to allow one bridge to connect to it.
50-micron, multimode fiber-optic cables also apply to bridges.
requirements.
Steps
1. Make note of the shelf IDs of the disk shelves that are connected to the system.
You use this information later when performing the disk autoassignment.
2. Move the NVRAM adapter and FC-VI adapter to the correct slots for your model, as shown in the IBM System Storage N
series Introduction and Planning Guide at the N series support website (accessed and navigated as described in Websites).
3. Determine your switch and general configuration by completing the Planning the fabric-attached MetroCluster installation
worksheet in the Data ONTAP High Availability and MetroCluster Configuration Guide for 7-Mode.
4. If needed, update the firmware on each FC switch to the latest supported version for the Data ONTAP release that your
system is running.
The list of supported FC switch firmware is found in the Interoperability Matrix Tool.
5. Set up and configure the FC switches, and verify your FC switch licenses, as described in the Brocade Switch Configuration
Guide for Fabric-attached MetroClusters or Fabric-attached MetroCluster Systems Cisco Switch Configuration Guide.
25
Make sure of the following:
Always refer to MetroCluster documentation when installing and configuring your FC switches.
If you are using Brocade switches, the switch configuration and firmware requirements for a MetroCluster environment
are different from the requirements for SAN environments.
If you are using switches that were previously used for another purpose, you must restore the switches to the factory
defaults.
Each switch port connected to a bridge must be set to G_Port.
6. If you are converting a stretch MetroCluster system, got to Step 11; otherwise, install the bridges by completing the
following substeps, and then continue to Step 8:
a) Secure the "L" brackets on the front of each bridge to the front of the rack (flush-mount) with the four screws you
acquired in the Preparing for the installation section.
b) Connect each bridge to a power source that provides a proper ground.
c) Power on each bridge.
Note: For maximum resiliency, ensure that bridges attached to the same stack of disk shelves are connected to
different power sources.
The bridge Ready LED might take up to 30 seconds to illuminate, indicating that the bridge has completed its power-on
self test sequence.
For information about power requirements and LEDs, see the ATTO FibreBridge 6500N Installation and Operation
Manual.
7. Connect the Ethernet management 1 port on each bridge to your network using an Ethernet cable.
Note: The Ethernet management 1 port enables you to quickly download the bridge firmware (using ATTO ExpressNAV
or FTP management interfaces) and to retrieve core files and extract logs.
connected by the Ethernet cable is configured. For example, if you also want to configure the Ethernet management 2

26
mode to ptp.
changes:
SaveConfiguration

10. Update the firmware on each bridge to the latest version by following the instructionsstarting with Step 4on the
11. Cable the disk shelves to the bridges by completing the following substeps:
Note: The disk shelves in the stack remain daisy-chained.
a) For each stack of disk shelves, cable IOM A square port on the first shelf to SAS port A on FibreBridge A.
b) For each stack of disk shelves, cable IOM B circle port on the last shelf to SAS port A on FibreBridge B.
Each bridge has one path to its stack of disk shelves; bridge A connects to the A-side of the stack through the first shelf, and
bridge B connects to the B-side of the stack through the last shelf.
Note: The bridge SAS port B is disabled.
The following illustration shows a set of bridges cabled to a stack of three disk shelves:
FC1
FC2
FibreBridge A
M1
SAS A
FC1
FC2
FibreBridge B
M1
SAS A

IOM A
IOM B
First
shelf
Last
shelf
12. Verify that each bridge can see all the disk drives and disk shelves that it is connected to.
27
Then...
In a supported web browser, enter the IP address of the bridge in the browser box.
b.
bridge.
c.
d.

sastargets

sastargets
Example
numbered so you can quickly count the devices. For example, the following output shows that 10 disks are connected:
Tgt
0
1
2
3
4
5
6
7
8
9
VendorID
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
NETAPP
ProductID
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
X410_S15K6288A15
Type
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
DISK
SerialNumber
3QP1G3EW00009940U2M0
Note: If the text response truncated appears at the beginning of the output, you can use Telnet to access the bridge
and enter the same command to see all the output.
13. Cable the bridges to the FC switches by completing the following substeps:
a) Cable FC port 1 on bridge A to FC switch 1.
b) Cable FC port 1 on bridge B to FC switch 2.
Note: You can connect the bridge to any available port on the FC switch. Whichever port you choose, you must use
the same port consistently on each FC switch that you are connecting for both sites. This is for ease of manageability.
For example, if you choose FC switch port 3, only use that port on each FC switch.
Note: The FC port 2 remains unused on each bridge.
14. Cable the controllers to the FC switches using the information in the Data ONTAP High Availability and MetroCluster
Configuration Guide for 7-Mode.
15. Check whether your system has disk autoassignment enabled by entering the following command at the console of either
controller:
If disk autoassignment is enabled, the output shows options disk.auto_assign on.

16. If your system does not have disk autoassignment enabled or the disk drives in the same stack are owned by both controllers,
assign disk drives to the appropriate pools by using the information in the Data ONTAP High Availability and MetroCluster
Configuration Guide for 7-Mode; otherwise, go to the next step.
You can use the disk drive information that you gathered in Step 1.
28
controllers.
17. Install the Data ONTAP licenses on each controller in the following order:
a) syncmirror_local
b) cf_remote
18. Reboot each controller.
19. Configure the controller by using the information in the Data ONTAP High Availability and MetroCluster Configuration
Guide for 7-Mode:
If you are converting a...
Then...
Standard HA pair with SAS storage
Set up mirroring, and then configure the controller.

Note: You need to add disk shelves, disable disk autoassignment, assign the new
disk drives to pool 1, and then boot each controller.
Stretch MetroCluster system with SAS storage Configure the controller.
20. Verify that the aggregate Snapshot copy reserve for mirroring is 5 percent or larger by entering the following command at
the console of each controller:
aggr options aggrname
21. Configure the MetroCluster system by using the information in the Data ONTAP High Availability and MetroCluster
Configuration Guide for 7-Mode.
22. Transport the partner controller, FC switches, disk shelves, and bridge to the remote site.
23. Install and set up the remote site system by repeating this procedure.
Related concepts

You can nondisruptively replace SAS copper cables with other SAS copper cables in a multipath HA configuration when a
cable has failed or you need a different-length cable.
Before you begin
You must have met the requirements given at the beginning of this document.
About this task
Replacing a SAS copper cable means that you are replacing a cable using the same exact ports for bridge-to-shelf or shelfto-shelf connections.
Attention: After your storage system is up and serving data, you cannot move SAS copper cables (change the SAS ports
to which a cable is connected) nondisruptively. If you need to correct system cabling, you can use a maintenance period to
do so.
You cannot change any disks, disk shelves, or components of a controller module as part of these procedures.
29
Steps
1. Verify that your MetroCluster system is Multi-Path HA by running the following command at the console of both
controllers:
sysconfig
The configuration is listed in the System Storage configuration field. It should be the fourth line of output.
Caution: If your system configuration is shown as something other than Multi-Path HA, you cannot continue with this
procedure.
2. Replace SAS cables by completing the following substeps:

Note: When replacing a SAS cable, wait a minimum of 10 seconds before plugging in the new cable so that the system
can detect the cable change.
You can ignore cabling messages that might appear on the console.
a) Replace cables on Side A one cable at a time.
The Side A cables are the cables connected to IOM A of each disk shelf.
b) Verify that you have correctly replaced the SAS cables by entering the following command at the system console:
sysconfig
For HA pairsoperation in 7-Mode, you can run the command from either controller. For clustered systems, you must run
this command from the nodeshell of the target controller.
The output should be the same as Step 1: the system should be Multi-Path HA, and the SAS port and attached disk
shelf information should be the same.
If the output is something other than Multi-Path HA, you must identify the cabling error, correct it, and run the
sysconfig command again.
c) Repeat substeps a and b for Side B.
The Side B cables are the cables connected to IOM B of each disk shelf.
Other FibreBridge management interfaces

You can use a combination of serial port, Telnet, and FTP instead of the recommended management interfaces. Your system
must meet the requirements for the applicable interface before you install the bridges.
You can use a serial port or Telnet to configure the bridge and Ethernet management 1 port, and to manage the bridge. You can
use FTP to update the bridge firmware.
Note: The ATTO FibreBridge 6500N Installation and Operation Manual has more information about management interfaces.
You can access this document on the ATTO web site by using the link provided on the ATTO FibreBridge 6500N
Description page on the NetApp Support Site at support.netapp.com. From this web site, navigate to the ATTO FibreBridge
Description page by clicking Software, scrolling to Protocol Bridge, choosing ATTO FibreBridge from the drop-down
menu, clicking Go!, and then clicking View & Download.
Serial port
When using the serial port to configure and manage a bridge, and configure the Ethernet management 1 port, your system must
meet the following requirements:
30
A serial cable (which connects from the bridge serial port to a serial (COM) port on the computer you are using for setup)
The bridge serial port is RJ-45 and has the same pin-out as the controllers.
A terminal emulation program such as Hyperterminal, Teraterm, or PuTTY to access the console
The terminal program should be capable of logging screen output to a file.
An IP address, subnet mask, and gateway information for the Ethernet management 1 port on each bridge
Telnet
When using Telnet to configure and manage a bridge, your system must meet the following requirements:
A serial cable (which connects from the bridge serial port to a serial (COM) port on the computer you are using for setup)
The bridge serial port is RJ-45 and has the same pin-out as the controllers.
(Recommended) A non-default user name and password (for accessing the bridge)
A terminal emulation program such as Hyperterminal, Teraterm, or PuTTY to access the console
The terminal program should be capable of logging screen output to a file.
FTP
When using FTP to update bridge firmware, your system must meet the following requirements:
A standard Ethernet cable (which connects from the bridge Ethernet management 1 port to your network)
(Recommended) A non-default user name and password (for accessing the bridge)

Bridge error messages inform you of bridge configuration errors that Data ONTAP has detected when cabling bridges or when
downloading disk shelf and disk drive firmware.
Attention: If you must correct cabling on a system that is serving data, you can perform hot-cable changes only if one path to
the disk drives is maintained.
config.bridge.adapter.invalid
Message
config.bridge.adapter.invalid
Severity
ERROR
Description
This message occurs when an FC-to-SAS bridge is attached to an unsupported FC adapter port.
Corrective action Attach the FC-to-SAS bridge to a QLogic 4-Gb/s or 8-Gb/s adapter (which are the only supported
adapters).
FC ports can be onboard and/or on PCIe HBAs.
SNMP trap ID
N/A
config.bridge.extra.port
Message
config.bridge.extra.port
Severity
ERROR
Description
This message occurs when both FC ports of an FC-to-SAS bridge are attached to the same controller.
Corrective action Correct the cabling by following the defined stretch or fabric-attached MetroCluster cabling information
and illustrations in this Configuring a MetroCluster system with SAS disk shelves and FibreBridge 6500N
bridges document.
SNMP trap ID
N/A
31
config.bridge.fw.mismatch
Message
config.bridge.fw.mismatch
Severity
ERROR
Description
This message occurs when two or more attached FC-to-SAS bridges are running different firmware
versions.
All FibreBridges should be running the same firmware version.
Corrective action Upgrade the firmware version to the latest version on the FC-to-SAS bridges by using the instructions in
the ATTO FibreBridge 6500N Firmware Download page on the NetApp Support Site at
now.netapp.com/NOW/cgi-bin/software.
SNMP trap ID
N/A
config.bridge.SAS.miswired
Message
config.bridge.SAS.miswired
Severity
ERROR
Description
This message occurs when the SAS disk shelves behind an FC-to-SAS bridge are not wired in the correct
order.
Corrective action Correct the cabling by following the defined stretch or fabric-attached MetroCluster cabling information
and illustrations in this Configuring a MetroCluster system with SAS disk shelves and FibreBridge 6500N
bridges document.
SNMP trap ID
N/A
config.bridge.SAS2Domain
Message
config.bridge.SAS2Domain
Severity
WARNING
Description
This message occurs when a disk is multipathed to one SAS domain; that is, both ports of the disk are
visible from the same bridge FC port.
Corrective action Recable the disk shelf containing the specified disk, so that the two disk ports are in different SAS
domains.
You can use the cabling information and illustrations in this Configuring a MetroCluster with SAS disk
shelves and FibreBridge 6500N document.
SNMP trap ID
N/A
dfu.badBridgeShelfConfig
Message
dfu.badBridgeShelfConfig
Severity
ERROR
Description
This message occurs when the disk firmware update aborts because it cannot contact the two peer FC-toSAS bridges.
Corrective action Make sure that the storage controller is correctly cabled to the two peer FC-to-SAS bridges and that the
bridges are correctly cabled to the SAS disk shelves, and then run the disk_fw_update command again.
32
SNMP trap ID
N/A
sfu.badBridgeShelfConfig
Message
sfu.badBridgeShelfConfig
Severity
ERROR
Description
This message occurs when the disk shelf firmware update aborts because it cannot contact the two peer
FC-to-SAS bridges.
Corrective action Make sure that the storage controller is correctly cabled to the two peer FC-to-SAS bridges and that the
bridges are correctly cabled to the SAS disk shelves, and then run the storage download shelf
command again.
SNMP trap ID
N/A
Trademark information
NetApp, the NetApp logo, Network Appliance, the Network Appliance logo, Akorri, ApplianceWatch, ASUP, AutoSupport,
BalancePoint, BalancePoint Predictor, Bycast, Campaign Express, ComplianceClock, Cryptainer, CryptoShred, CyberSnap,
Data Center Fitness, Data ONTAP, DataFabric, DataFort, Decru, Decru DataFort, DenseStak, Engenio, Engenio logo, E-Stack,
ExpressPod, FAServer, FastStak, FilerView, Flash Accel, Flash Cache, Flash Pool, FlashRay, FlexCache, FlexClone, FlexPod,
FlexScale, FlexShare, FlexSuite, FlexVol, FPolicy, GetSuccessful, gFiler, Go further, faster, Imagine Virtually Anything,
Lifetime Key Management, LockVault, Manage ONTAP, Mars, MetroCluster, MultiStore, NearStore, NetCache, NOW
(NetApp on the Web), Onaro, OnCommand, ONTAPI, OpenKey, PerformanceStak, RAID-DP, ReplicatorX, SANscreen,
SANshare, SANtricity, SecureAdmin, SecureShare, Select, Service Builder, Shadow Tape, Simplicity, Simulate ONTAP,
SnapCopy, Snap Creator, SnapDirector, SnapDrive, SnapFilter, SnapIntegrator, SnapLock, SnapManager, SnapMigrator,
SnapMirror, SnapMover, SnapProtect, SnapRestore, Snapshot, SnapSuite, SnapValidator, SnapVault, StorageGRID,
StoreVault, the StoreVault logo, SyncMirror, Tech OnTap, The evolution of storage, Topio, VelocityStak, vFiler, VFM, Virtual
File Manager, VPolicy, WAFL, Web Filer, and XBB are trademarks or registered trademarks of NetApp, Inc. in the United
States, other countries, or both.
IBM, the IBM logo, and ibm.com are trademarks or registered trademarks of International Business Machines Corporation in
the United States, other countries, or both. A complete and current list of other IBM trademarks is available on the web at
www.ibm.com/legal/copytrade.shtml.
Apple is a registered trademark and QuickTime is a trademark of Apple, Inc. in the United States and/or other countries.
Microsoft is a registered trademark and Windows Media is a trademark of Microsoft Corporation in the United States and/or
other countries. RealAudio, RealNetworks, RealPlayer, RealSystem, RealText, and RealVideo are registered trademarks and
RealMedia, RealProxy, and SureStream are trademarks of RealNetworks, Inc. in the United States and/or other countries.
All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.
NetApp, Inc. is a licensee of the CompactFlash and CF Logo trademarks.
NetApp, Inc. NetCache is certified RealSystem compatible.
Trademark information
33
How to send your comments

You can help us to improve the quality of our documentation by sending us your feedback.
Your feedback is important in helping us to provide the most accurate and high-quality information. If you have suggestions for
improving this document, send us your comments by email to doccomments@netapp.com. To help us direct your comments to
the correct division, include in the subject line the product name, version, and operating system.
You can also contact us in the following ways:
34
NetApp, Inc., 495 East Java Drive, Sunnyvale, CA 94089 U.S.

Telephone: +1 (408) 822-6000
Fax: +1 (408) 822-4501
Support telephone: +1 (888) 463-8277

Configuring A MetroCluster System With SAS Disk

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Configuring A MetroCluster System With SAS Disk

Diunggah oleh

Hak Cipta:

Format Tersedia

Configuring a MetroCluster system with SAS disk shelves and

FibreBridge 6500N bridges

SNMP (Simple Network Management Protocol)

This document contains the following procedures and information:

MetroCluster configuration examples

Copyright 2014 NetApp, Inc. All rights reserved.

Stretch MetroCluster system

Stack of SAS shelves

Stack of SAS shelves

Fabric MetroCluster system

Stack of SAS shelves

Stack of SAS shelves

Installing bridges and SAS disk shelves as part of a new MetroCluster

Other FibreBridge management interfaces on page 30

Preparing for the installation

Interoperability Matrix Tool (IMT)

Installing the new system

1. Properly ground yourself.

4. Install each bridge by completing the following substeps:

7. Configure the bridges.

Result: You are prompted to restart the bridge.

Stack of SAS shelves

ATTO ExpressNAV GUI a.

Click Advanced in the menu.

Enter the following command, and then click Submit:

Serial port connection

Enter the following command:

Repeat Step 10 for each remaining bridge.

Repeat Step 10.

Cable FC port 1 of bridge A to an 8-Gb or 4-Gb FC port on the controller at Site 1.

Repeat Substeps a and b for bridge B.

Cable FC port 1 of bridge A to FC switch 1 at Site 1.

Cable FC port 1 of bridge B to FC switch 2 at Site 1.

Repeat Substeps a and b for Site 2.

If disk autoassignment is enabled, the output shows disk.auto_assign on.

The output shows the following:

Firmware rev. IOM6 A: 0111 IOM6 B: 0111

X421_HCOBD450A10 NA01 418.0GB (879097968 520B/sect)

40.0GB (82262416 520B/sect)

FibreBridge error messages on page 31

Hot-adding a stack of SAS disk shelves and bridges to an existing

Other FibreBridge management interfaces on page 30

Preparing for the hot-add

Interoperability Matrix Tool

Hot-adding the stack of shelves

1. Properly ground yourself.

If disk autoassignment is enabled, the output shows disk.auto_assign on.

You must run this command on both controllers.

FibreBridge error messages on page 31

Other FibreBridge management interfaces on page 30

Preparing for the hot-swap

from the failed bridge..

Hot-swapping the FibreBridge

1. Properly ground yourself.

7. Connect the bridge to a power source and turn it on.

9. Configure the bridges.

Set FCDataRate 1 8Gb

Result: You are prompted to restart the bridge.

ATTO ExpressNAV GUI a.

Click Advanced in the menu.

Enter the following command, and then click Submit:

Serial port connection

Enter the following command: