UCS Engineering Details for the SAN Administrator

Craig Ashapa

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

First things first: debunking a myth

Today (June 2012 UCS 2.02m) there is no FCoE northbound of UCS
unless you really really really want to

Thats right, there is no FCoE on external Ethernet uplinks by default In other words, FCoE is used only at the access layer (Server to Fabric Interconnect) in a totally transparent manner You dont need to know anything about FCoE to understand how storage works with UCS
Everything works as usual. Servers have HBAs that FLOGI into the SAN fabric, you use zoning and LUN masking just like you do with rackmounts today

UCS release 1.4 introduced support for Direct Connect FCoE targets
Today, this is the only case where youll see FCoE traffic Youll have to go through explicit configuration steps to make this happen

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Topics
Introduction of Converged Network Adapters (CNAs)
CNAs and Port WWNs Considerations

UCS Storage modes of operation and recommendations

End Host Mode - NPV (N_Port Virtualization) FC Switching Mode

Upstream Connectivity
F_Port Trunking / F_Port Channeling

Direct Connectivity of Storage

Port Types What works and what doesnt

IP-Based Storage
Appliance Ports iSCSI

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Introduction of Converged Network Adapters

CNAs and Port WWNs Considerations

Unified Fabric with FCoE

CNA: Converged Network Adapter
CNA presents multiple PCI addresses to the Operating System (OS)

Nexus Edge participates in both distinct FC and IP Core topologies

OS loads two unique sets of drivers and manages two unique application topologies
Server participates in both topologies since it has two stacks and thus two views of the same unified wire Host FC Multi-Pathing driver provides failover between two fabrics (SAN A and SAN B) UCS hardware based Fabric Failover (preferred) or OS NIC Teaming provides Ethernet traffic failover
UCS-4794 2012 Cisco and/or its affiliates. All rights reserved.

FCF

FCF

Unified Wire shared by both FC and IP topologies

Nexus Unified Edge supports both FC and IP topologies

10GbE
Link

10GbE Fibre Channel

Ethernet

Ethernet Driver bound to Ethernet NIC PCI address

FC Driver bound to FC HBA PCI address

PCIe
Ethernet Drivers Fibre Channel Drivers

Operating System

Cisco Public

CNA: Converged Network Adapter

Operating System View
Emulex / Qlogic / Cisco Standard drivers Same management Operating System sees:
N port or Dual port (depending on hardware) 10 Gigabit Ethernet adapter N port or Dual Port (depending on hardware) Fibre Channel HBAs
UCS-4794 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public

vHBAs: WWN assignment

WWN assignment: just like Ethernet MAC addresses Either inherited from burnt-in WWN (not with M81KR!) Or manually set Or borrowed from a pool (recommended) Up to two HBAs (called vHBAs) per server with non-M81KR CNAs Backplane path failover does not exist for HBAs! A vHBA either goes through switch A or B at any given time OS-level multipathing provides path resiliency Same dynamic pinning concept as with 10GE NICs
Manual override allowed by using SAN pin-groups

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

A word on World-Wide Name formats

Avoid interop issues by choosing IEEE Extended Type 2 WW Names inside pools:
Section 1
2

Section 2
0:00

Section 3
00:25:B5

Section 4
XX:XX:XX

Section 1 identifies the WWN as an extended format WWN Section 2 is a vendor specific code and can be used to identify specific ports on a node or to extend the serial number (section 4) of the WWN Section 3 identifies the vendor (00:25:B5 = Cisco) Section 4 is the serial number for the device

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Quick tip how do I determine my port WWNs?

Determine the servers pWWN
Assigned through the service profile

Verify on the host it will match:

Check local FLOGI for that pWWN on UCS and MDS:

On Windows, use free fcinfo.exe utility

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Fabric Interconnects
Modes of Operation

What is NPV?
N-Port Virtualizer (NPV) utilizes NPIV functionality to allow a switch to act like a server performing multiple logins through a single physical link Physical servers connected to the NPV switch login to the upstream NPIV core switch
Physical uplink from NPV switch to FC NPIV core switch does actual FLOGI Subsequent logins are converted (proxy) to FDISC to login to upstream FC switch

No local switching is done on an FC switch in NPV mode

FC edge switch in NPV mode Does not take up a domain ID

UCS Fabric Interconnect
F-Port

FC NPIV Core Switch

NP-Port F-Port

Eth1/1

Server1 N_Port_ID 1 Server2 N_Port_ID 2 Server3 N_Port_ID 3

Eth1/2

F_Port

Eth1/3
N-Port
UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

N-Port Virtualization (NPV) mode

UCS FI works in NPV mode by default Server-facing ports are regular F ports Uplinks toward SAN core fabric are NP ports UCS distributes (relays) FCIDs to attached devices No domain ID to maintain locally Zoning, FSPF, DPVM, etc are not configured on the UCS Fabrics Domain Mgr, FSPF, Zone Server, Fabric Login Server, Name Server They do not run on UCS Fabrics No local switching All FC traffic routed via the core SAN switches

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

UCS Storage Defaults and Recommendations for FC

Default (recommended) - N_Port Virtualization (NPV) Mode
End Host Mode for FC, UCS functions as Node Port (initiator)
Small to Large Scale Deployments of homogeneous or heterogeneous operating systems Extensive interoperability with SAN and array ecosystem

Option - FC Switching Mode

UCS FI has limited FC switching features, No Zoning Configuration Must still have upstream MDS or Nexus FC switch via FC Uplink

Direct Connect from Fabric Interconnect to Storage Array FC target

Designed for POD or Small scale Limited interoperability with Storage ecosystem
UCS-4794 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public

UCS Storage Defaults and Recommendations for FCoE

Default
FCoE Traffic is internal to UCS system (blade to Fabric Interconnect) FCoE packets are terminated at the fabric interconnect, no northbound FCoE packets transmitted from UCS

Options FC Switching Mode

UCS FI has limited FC switching features, no zoning configuration Must still have upstream MDS or Nexus FC switch via FC Uplink Direct Connect from Fabric Interconnect to Storage Array(s) FCoE target Use of FCoE Storage Targets with UCS Limited interoperability with Storage ecosystem

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

UCS Storage Defaults and Recommendations for NAS

Default (recommended) - End Host Mode
Superior traffic engineering - Native L2 multipathing; no spanning-tree

Easier integration into network

1.4 Introduced Appliance Ports which allow direct connect NAS filers

Options - Ethernet Switching Mode

As of 1.4 no storage based reasons to use this mode Previous releases required switching mode for direct connect NAS

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

UCS Connectivity Summary

LAN CLOUD

FC STORAGE

SAN Switch
Upstream Switch in NPIV Mode

NAS STORAGE

LAN Switch
UCS Ethernet in EHM UCS FC in NPV Mode

UCS Fabric Interconnect

Access Layer LAN & SAN Unified Fabric (FCoE)

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Upstream Connectivity
SAN Uplinks, Port Channels and Trunks

FC Uplink VSAN Membership

Default SAN integration is very straightforward:
Connect UCS to external SAN switch (MDS / Brocade) Make FC uplink member of one and only one VSAN No F_Port or NP_Port trunking by default

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

FC Port Trunking (Multiple VSANs per Link)

Provide isolation to SAN traffic over the same physical FC link Help consolidate FC infrastructure

vHBAs can be on different VSANs

Uplink FC ports will be in NPV mode (NPort) by default All VSANs will be trunked on every uplink FC port
vFCs

Selecting a subset of VSANs for individual uplink ports not supported

Scalability: Max of 32 VSANs per UCS system VSAN trunking supported in FC switch mode as well VSAN Trunking is not available for direct connect FC (or FCoE) Storage Port types

VSAN 100 VSAN 200 VSAN 300 VSAN 400

SAN A

SAN B

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

FC Port Channels
Aggregate and maximize available bandwidth while maintaining isolation Increases resiliency and guard against port failures Up to 16 FC ports can be aggregated together for a single port channel Different combination of FC ports from different expansion modules on the FI can be placed on the same port channel

vFCs

In case of port speed mismatch port channel forces port speed to highest commonly supported speed
VSANs can be trunked over the port channel

Port Channel

VSAN 100 VSAN 200 VSAN 300 VSAN 400

VSAN trunking and port channel supported for both NPV and switch mode FI operation
FC Port channeling is not available for direct connect FC Storage Port types
UCS-4794 2012 Cisco and/or its affiliates. All rights reserved.

SAN A

SAN B

Cisco Public

How FC Trunking Is Enabled (Global)

Global Setting for both FIs Default is Not enabled

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Creation and Management of FC Port Channels

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Fabric Interconnects
FC and Direct Storage Connectivity Options

FC SAN Boot
Create Service Profile Configure Storage (simple or expert)

Zone

Create vHBAs (1, 2, or more (M81KR))

Assign Initiator WWPN (manual or pool) Assign VSAN to vHBAs Assign vHBA Placement Select or Create Boot Policy Assign Target WWPN and LUNID

Associate Server Mask Boot to vMedia Install Drivers and OS

UCS-4794 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public

Enable Direct Connection of FC/FCOE Storage

FCoE Storage FC Storage

Customer benefits
Support to directly connect FC/ FCoE storage to 6100 End to end FCoE topologies possible Lower cost point for small deployments (no access layer FC switches)

Feature details
Support for NetApp and EMC direct attached storage Ability to turn On/Off Fibre Channel switching mode Zoning configuration not exposed with UCS Manager
UCS 6100 UCS 6100

Zoning must be inherited from upstream switch

Ethernet and FC switching modes are independent

UCS B-Series

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

UCS Direct Connect Storage

Three UCS port types related to direct storage capability

Storage FC Port direct connect this port to FC port on array

Storage FCoE Port direct connect this port to FCoE port on array Appliance Port - direct connect this port to 10G Ethernet port on array

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

UCS Manager View of New Ports

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Hybrid Topology with Direct-Attach and SAN

SAN Fabric
Storage Arrays Fibre Channel Fabric A Direct Attach Fabric B Ethernet

Core

Unified I/O
FCoE

FC Storage FCoE Storage

SAN Edge A SAN Edge B

Security via Zoneset Merge

UCS 6100 UCS 6100

AND Security via LUN Masking LUN Masking Always Used to Some Extent

UCS B-Series
UCS-4794 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public

How to Place FI into FC Switching Mode

This operation results in both FIs going into reboot cycles which takes 15 min or so, system wide downtime, by design
When FC switch mode is turned on, all the FC ports come up in TE mode

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Configuring an FC Storage Port

To connect an FC storage device directly into one of the 6100s FC port, the user must configure the port as an FC storage port
The VSAN is configured under the Storage Cloud object Internally, the port is configured as follows:
As a F_Port

As an access port
The speed is kept as auto The user is also allowed to select a named VSAN for that port.

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Configuring an FC Storage Port (GUI)

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Configuring a VSAN on an FC Port (GUI)

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Configuring an FCoE Storage Port (GUI)

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Configuring an FCoE Storage Port (Cont.)

Since the Ethernet port is configured as trunk, a native VLAN must be configured on that port.
The FCoE VLAN ID can not be the same as the native VLAN ID User is not requested to provide a native VLAN value. An arbitrary VLAN (4048) was chosen and is always used as the native VLAN for FCoE storage ports unless changed

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

FCoE Port Must Be Assigned to a VSAN

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

IP-based Storage
Appliance Ports and iSCSI

UCS Manager Appliance Ports

What is an Ethernet Appliance
A specialized device for use on a Ethernet network, for example Network Attached Storage , iSCSI, security appliances, Nexus 1010 What qualifies to be an appliance An Appliance is a specialized external host, does not run STP (today only NetApp and EMC IP storage devices qualified) Purpose of Appliance Port Connect Ethernet appliances only

!
UCS-4794

Do Not to use this port type for switch connectivity to avoid traffic loops
2012 Cisco and/or its affiliates. All rights reserved. Cisco Public

Appliance Ports How to Configure

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Appliance Port Exposed Settings

QoS per port settings, normal UCS QoS constructs
Manual (static) pinning using pin groups for border port selection Select which VLANs can traverse this port Optionally specify the destination MAC address of the filer . Some Filers do not broadcast their MAC address

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

VLANs and Appliance Ports

Similar to VSAN concept, there are two scopes Traditional, LAN Cloud

Appliance Cloud with scope restricted to appliance ports and associated VLANs
Use the same VLAN ID in both scopes

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

iSCSI Feature Overview

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

iSCSI Boot Flow

Create iSCSI vNICs Create iSCSI boot policy Provide UCSM with iSCSI boot information
Target ip, iqn

Initiator ip/mask/gw, iqn

vMedia map the OS and drivers Adapter successfully initializes

Install OS and Drivers (if required)

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

43

Storage Vendor Support

UCS 1.4 Direct Connect only support EMC and NetApp for all topologies (FC, FCoE, NAS)
We may add more vendors as a function of time given business case justifications

iSCSI Boot will only support EMC and NetApp Arrays

We may add more vendors as a function of time given business case justifications

Please Consult the these resources

http://www.cisco.com/en/US/products/ps10477/prod_technical_reference_list.html http://www.cisco.com/en/US/docs/unified_computing/ucs/interoperability/matrix/r_hcl_ B_rel2.02.pdf

UCS-4794

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public

Complete Your Online Session Evaluation

Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete.

Complete your session evaluation online now (open a browser through our wireless network to access our Dont forget to activate your Cisco Live Virtual account for access to portal) or visit one of the Internet stations throughout the Convention all session material, communities, and on-demand and live activities throughout Center. the year. Activate your account at the
Cisco booth in the World of Solutions or visit www.ciscolive.com.
UCS-4794 2012 Cisco and/or its affiliates. All rights reserved. Cisco Public 45

Presentation_ID

2012 Cisco and/or its affiliates. All rights reserved.

Cisco Public