DCV Nexus

2011 Cisco FlexPod Training Guide Page 1 of 217
Data Center Virtualization (DCV)

Lab Guide

Peter Phan, Systems Engineer, Cisco
pephan@cisco.com

September 26, 2011

Page 2 of 217 FlexPod Training Guide 2011 Cisco

DATA CENTER VIRTUALIZATION (DCV) 1
LAB GUIDE 1
1 DATA CENTER VIRTUALIZATION TRAINING LAB OVERVIEW 6
1.1 DATA CENTER VIRTUALIZATION ARCHITECTURE ................................................................. 6
1.1 DOCUMENTATION KEY ....................................................................................................... 7
1.2 LAB TOPOLOGY AND ACCESS .............................................................................................. 8
2 DATA CENTER VIRTUALIZATION FOR VDI CONFIGURATION DEPLOYMENT 11
2.1 LAB REFERENCE GUIDE ..................................................................................................... 11
2.2 CABLING INFORMATION ................................................................................................... 13
2.3 REQUIRED SOFTWARE VERSIONS ..................................................................................... 16
2.4 GLOBAL CONFIGURATION VARIABLES ............................................................................... 17
2.5 NETAPP CONFIGURATION VARIABLES ............................................................................... 18
2.6 CISCO CONFIGURATION VARIABLES .................................................................................. 21
2.7 VMWARE CONFIGURATION VARIABLES ............................................................................ 22
3 CISCO NEXUS AND MDS DEPLOYMENT PROCEDURE: PART I 23
3.1 NEXUS INITIAL SETUP .................................................................................................... 27
3.2 ENABLE FEATURES ............................................................................................................ 34
3.3 NEXUS GLOBAL SETTINGS .............................................................................................. 35
3.4 NEXUS CONFIGURE ETHERNET INTERFACES .................................................................... 40
3.5 CONFIGURING PORT CHANNELS ....................................................................................... 43
3.6 CONFIGURING VIRTUAL PORT CHANNELS ......................................................................... 49
3.7 CONFIGURING FEX ON N5K-1 AND N5K-2 ......................................................................... 54
3.8 PERFORM THE INITIAL SETUP OF MDS9124 ....................................................................... 58
4 CISCO UNIFIED COMPUTING SYSTEM DEPLOYMENT PROCEDURE 61
4.1 POWER ON THE ESX HOSTS AND VERIFY THE NEXUS INTERFACES ..................................... 61
5 CISCO NEXUS AND MDS DEPLOYMENT PROCEDURE: PART II 63
5.1 CREATE FIBRE CHANNEL OVER ETHERNET (FCOE) INTERFACES .......................................... 63
5.2 DEVICE ALIASES, ZONES, AND ZONESETS .......................................................................... 71
6 VMWARE ESXI DEPLOYMENT PROCEDURE 77
6.1 ESXI INSTALLATION AND BASIC SETUP .............................................................................. 77
6.2 ESXI NETWORKING ........................................................................................................... 82
6.3 ESXI DATASTORES ............................................................................................................ 89
7 VMWARE VCENTER SERVER DEPLOYMENT PROCEDURE 92
7.1 ADDING HOSTS TO VMWARE VCENTER SERVER ................................................................ 94
7.2 CONFIGURE FIBRE CHANNEL STORAGE ON ESX HOSTS ...................................................... 95
7.3 ADD A VM FROM NFS ATTACHED STORAGE ...................................................................... 97
8 CISCO NEXUS 1000V DEPLOYMENT PROCEDURE 98
8.1 INSTALL VIRTUAL SUPERVISOR MODULE (VSM) AS A VM ON ESXI ..................................... 98
8.2 REGISTERING THE CISCO NEXUS 1000V AS A VCENTER PLUG-IN ...................................... 100
8.3 CONFIGURING NETWORKING ON THE CISCO NEXUS 1000V ............................................. 101
8.4 NEXUS 1000V CREATE VLANS ......................................................................................... 102
8.5 NEXUS 1000V CREATE PORT PROFILES ............................................................................ 104
8.6 INSTALL VIRTUAL ETHERNET MODULES (VEMS) ON ESXI HOSTS ...................................... 107
8.7 MIGRATE ESXI HOSTS TO NEXUS 1000V .......................................................................... 108
8.8 MIGRATE VIRTUAL MACHINE PORTS .............................................................................. 112
9 CONFIGURING OVERLAY TRANSPORT VIRTUALIZATION 113
9.1 LAB TOPOLOGY .............................................................................................................. 114
9.2 JOB AIDS ........................................................................................................................ 117
9.3 BASE CONFIGURATION ................................................................................................... 127
9.4 SPANNING TREE ............................................................................................................. 129
9.5 INTERFACE CONFIGURATION .......................................................................................... 133
9.6 OSPF CONFIGURATION ................................................................................................... 134
9.7 CONFIGURING OTV TO CONNECT EDGE DEVICES TO REMOTE END-SITES ......................... 138
9.8 OTV VERIFICATION AND MONITORING ........................................................................... 144
9.9 VERIFYING THE VMWARE VSPHERE SETUP ...................................................................... 148
10 VMOTION ACROSS DATA CENTERS 151
10.1 MISSING L2 CONNECTIVITY ACROSS SITES WITHOUT OTV ............................................... 152
10.2 SUCCESSFUL CONNECTIVITY WITHIN SAME SITE ............................................................. 153
10.3 SUCCESSFUL VMOTION ACROSS SITES DUE TO L2 CONNECTIVITY WITH OTV ................... 154
11 MIGRATE VM TO SAN ATTACHED STORAGE AND CONFIGURE VM DISKS 158
11.1 CLONE A VM TO SAN ATTACHED STORAGE ..................................................................... 159
11.2 CONFIGURE VIRTUAL MACHINE NETWORKING ............................................................... 160
11.3 MIGRATE A VM TO SAN ATTACHED STORAGE ................................................................. 163
11.4 CONFIGURE VM DISKS (OPTIONAL) ................................................................................. 165
12 SUMMARY 168
12.1 FEEDBACK ...................................................................................................................... 168
13 APPENDIX A: COPYING SWITCH CONFIGURATIONS FROM A TFTP SERVER 169
14 APPENDIX B: RECOVERING FROM THE LOADER PROMPT 170
15 NETAPP FAS2020A DEPLOYMENT PROCEDURE: PART 1 173
15.1 NETAPP ASSIGNING DISKS ........................................................................................... 173
15.2 NETAPP ONTAP INSTALLATION .................................................................................... 174
15.3 NETAPP INITIAL SETUP ................................................................................................. 175
15.4 NETAPP - AGGREGATES AND VOLUMES .......................................................................... 179
15.5 NETAPP NETWORK & SECURITY ................................................................................... 180
15.6 NETAPP - VOLUMES ....................................................................................................... 183
15.7 NETAPP IP SPACE AND MULTISTORE ............................................................................ 187
15.8 NETAPP NFS ................................................................................................................ 190
15.9 NETAPP PERFORMANCE OPTIMIZATION ...................................................................... 190
16 NETAPP FAS2020A DEPLOYMENT PROCEDURE: PART II 192
16.1 FLEXCLONE ..................................................................................................................... 193
16.2 REMOVE CLONED VOLUMES AND LUNS .......................................................................... 197
16.3 REMOVING VFILERS........................................................................................................ 197
16.4 REMOVING VFILER VOLUMES ......................................................................................... 197
17 APPENDIX COMMAND SUMMARY 198
18 REFERENCES 215


Table 1 Device Management Addresses and Accounts ........................................................................................ 11
Table 2 - ESXi Network Parameters ......................................................................................................................... 11
Table 3 Virtual Machines ...................................................................................................................................... 11
Table 4 VLAN Summary ........................................................................................................................................ 12
Table 5 - Ethernet Cabling Information ................................................................................................................... 14
Table 6 - Ethernet Cabling Information - Management Switch............................................................................... 15
Table 7 - Fibre Channel Cabling Information ........................................................................................................... 15
Table 8 - Data Center Virtualization global variables. ............................................................................................. 17
Table 9 - NetApp FAS2020 A variables. ................................................................................................................... 18
Table 10 - NetApp licensing variables. .................................................................................................................... 20
Table 11 - NetApp disk and volume variables ......................................................................................................... 20
Table 12 - Cisco Nexus 5010 variables. .................................................................................................................... 21
Table 13 - Cisco Nexus 1000v variables. .................................................................................................................. 21
Table 14 - VMware variables. .................................................................................................................................. 22
Table 15 - Commands .............................................................................................................................................. 24
Table 16 - Commands .............................................................................................................................................. 25
Table 17 - WWPN Addresses ................................................................................................................................... 71
Table 18 - IP Addresses for Uplinks and Loopbacks .............................................................................................. 116
Table 19 - OTV Edge Access Ports Connectivity to Access Switches ..................................................................... 116
Table 20 OTV Multicast Addresses ..................................................................................................................... 116
Table 21 - Commands used in this exercise .......................................................................................................... 118

Important
Prior to configuration, be sure to obtain the latest version of this document http://db.tt/LI79cwH.
1 DATA CENTER VIRTUALIZATION TRAINING LAB OVERVIEW

Welcome to the Cisco Data Center Virtualization Lab. This lab is intended to provide you with a solid
understanding of what you need to implement a wide range of solution features.

The lab tasks are designed to focus on achieving:

Customer awareness of what the solution can do for them.
Customer understanding of why the Cisco solution is unique and an improvement over the status quo or
competitive solutions.
Customer introduction to the deployment process of the demonstrated solution.

The FlexPod demonstration should go beyond the topics of interest to the technical decision maker (TDM) and
should appeal to the business decision maker (BDM) by focusing on the benefits that this solution provides.

The Quick Reference Guide section provides general positioning and primary marketing messages, as well as a
guide to which demonstrations will work together to show the benefits for a particular person in the workplace.
As always, you will want to tailor your sales presentation to address specific audience needs or issues.
Demonstration Script Style

The demonstration scripts are organized by task; they include important marketing messages as well as product
and feature overviews and demonstration instructions. Using the Quick Reference Guide, you will be able to
quickly tailor demonstrations for different customers, while communicating the benefits of each one to facilitate
product sales.

Industry trends indicate a vast data center transformation toward shared infrastructures. Enterprise customers
are moving away from silos of information and moving toward shared infrastructures to virtualized
environments and eventually to the cloud to increase agility and reduce costs.

The Cisco Data Center Virtualization lab is built on the Cisco Unified Computing System (Cisco UCS), Cisco
Nexus data center switches, NetApp FAS storage components, and a range of software partners. This guide is
based on the design principle of the FlexPod Implementation Guide.

AUDIENCE
This document describes the basic architecture of FlexPod and also prescribes the procedure for deploying a
base Data Center Virtualization configuration. The intended audience of this document includes, but is not
limited to, sales engineers, field consultants, professional services, IT managers, partner engineering, and
customers who want to deploy the core Data Center Virtualization architecture.

1.1 DATA CENTER VIRTUALIZATION ARCHITECTURE
The Data Center Virtualization architecture is highly modular or pod like. While each customers Data Center
Virtualization unit might vary in its exact configuration, once a Data Center Virtualization unit is built, it can
easily be scaled as requirements and demand change. This includes scaling both up (adding additional resources
within a Data Center Virtualization unit) and out (adding additional Data Center Virtualization units).

Data Center Virtualization includes NetApp storage, Cisco networking, Cisco Unified Computing System (Cisco
UCS), and virtualization software in which the computing and storage fit in one data center rack with the
networking residing in the same or separate rack. The networking components can accommodate multiple Data
Center Virtualization configurations. Figure 1 shows our lab components.

Our lab hardware includes:
Two Cisco Nexus 5010 switches
One Cisco MDS 9124 Switch
Two Cisco UCS C200 M1 and One Cisco UCS C250 M1 servers powered by Intel Xeon processors
o Quanities and types might vary for lab
One NetApp 2020 Filer

For server virtualization, the lab includes VMware vSphere Enterprise Plus with vCenter Standard.

1.1 DOCUMENTATION KEY

The following is a description of the conventions, colors and notation used through this document:

Sections with this background color and this icon cover the technical description of the lab task, with
items and talking points of interest to technical audiences.

Sections with this background color and this icon provide a lab tip for the step or task.

Sections with this background color and this icon are for scenario description: Provides background
information for performing a step or task.

Sections with this background color and this icon represent a warning: read this section for special
instructions and considerations.

Sections with this background color and this icon touch on the business benefits of the step or task with
items and talking points highlighting a value proposition of a Solution.
1.2 LAB TOPOLOGY AND ACCESS

This document guides the reader through the low-level steps of deploying the base architecture. In this lab, you
configure all of the hardware in the figure above except for the NetApp filer and Management switch. You will
also configure the software and virtualization (ESXi, vCenter, and Nexus 1000V).

Figure 1 - Topology for Single Pod

Your management tasks will be performed on an RDP server (VC_SERVER or MGMT_PC). You will access the
UCS, Nexus, and etc via SSH and each devices element manager. The Putty SSH client is on the Desktop.

Figure 2 - Lab Tools Interface

Here is a view of how all the Data Center Virtualization Pods are interconnected.

Figure 3 - Full Topology for Three Pods in a VDC Deployment

2011 Cisco Data Center Virtualization Volume 1 Page 10 of 217
The following diagram illustrates how all the different networks/vlans are interconnected. The router in the
center is connected to the Nexus 5000s via a Port-Channel Trunk.
Figure 4 - Logical Topology of Lab

2 DATA CENTER VIRTUALIZATION FOR VDI CONFIGURATION DEPLOYMENT

The following section provides detailed information on configuring all aspects of a base FlexPod environment.
The Data Center Virtualization architecture is flexible; therefore, the exact configuration detailed in this section
might vary for customer implementations depending on specific requirements. Although customer
implementations might deviate from the information that follows, the best practices, features, and
configurations listed in this section should still be used as a reference for building a customized Data Center
Virtualization architecture.

2.1 LAB REFERENCE GUIDE
Table 1 Device Management Addresses and Accounts
Device Management IP Username Password
N5K-1 10.1.111.1 admin 1234Qwer
N5K-2 10.1.111.2 admin 1234Qwer
N7K-1-OTV-XA 10.1.111.3 admin 1234Qwer
N7K-2-OTV-XB 10.1.111.4 admin 1234Qwer
MDS 10.1.111.40 admin 1234Qwer
CIMC-ESX1 10.1.111.161 admin 1234Qwer
Fabric Manager admin 1234Qwer
Device Manager admin 1234Qwer

Table 2 - ESXi Network Parameters
Device Management IP Username Password vMotion NFS
ESX1 10.1.111.21 root 1234Qwer 10.1.151.21 10.1.211.21
ESX2 10.1.111.22 root 1234Qwer 10.1.151.22 10.1.211.22
ESX3 10.1.111.23 root 1234Qwer
10.1.151.23
10.1.211.23

Table 3 Virtual Machines
Device Role Management IP Username Password
VCENTER-1 vCenter, VSC 10.1.111.100 administrator 1234Qwer
vsm-1 N1KV VSM 10.1.111.17 admin 1234Qwer
AD AD,DNS,DHCP 10.1.111.10
Server01 XenDesktop 10.1.111.11
Server02 XenApp 10.1.111.12
Server03 PVS 10.1.111.13
WIN7POC
WIN7STREAM
WIN7MASTER
Server 2003

Table 4 VLAN Summary
Summary of all VLAN
VLAN Description VSAN Description
111 MGMT

131
VMTRAFFIC

151
VMOTION

171
CTRL-PKT

211
NFS

1011
Fabric A FCoE VLAN

1012
Fabric B FCoE VLAN

999
Native VLAN

1005
OTV Site VLAN

11 Fabric A VSAN

12 Fabric B VSAN

2.2 CABLING INFORMATION
The following information is provided as a reference for cabling the physical equipment in a Data Center
Virtualization environment. The tables include both local and remote device and port locations in order to
simplify cabling requirements. Anyone interesting in recreating our labs in their own environment can use the
tables below as a reference.

The tables in this section contain details for the prescribed and supported configuration for the following
devices:
FAS2020 running Data ONTAP 7.3.5.
o This configuration leverages the onboard FC storage target ports, a dual-port 1 Gig adapter, and
the onboard SAS ports for disk shelf connectivity.
5010 - 20 Ten Gig ports and an add on card that provides 4 Ten Gig ports and 4x1/2/4 Gig FC ports.
2148 - 48 Gig ports with 4 10 Gig ports for uplinks.
3560/3750 Management Switch - 24 One Gig ports for device management and routing

Note: The FlexPod Implementation Guide assumes that out-of-band management ports are plugged into an
existing management infrastructure at the deployment site.
Note: Be sure to follow the cable directions in this section. Failure to do so will result in necessary changes
to the deployment procedures that follow because specific port locations are mentioned.

Table 5 - Ethernet Cabling Information
POD # Device Local Ports Device Access Ports
POD X N5K-1 e1/4 MGMT Switch 1/23
POD X N5K-1 e1/7 FEX A port1
POD X N5K-1 e1/8 FEX A port2
POD X N5K-1 e1/9 ESX1 vmnic0
POD X N5K-1 e1/17 N5K-2 e1/17
POD X N5K-1 e1/18 N5K-2 e1/18
POD 1 N5K-1 e1/19 N7K-1 e1/14
POD 1 N5K-1 e1/20 N7K-2 e1/14
POD 2 N5K-1 e1/19 N7K-1 e1/22
POD 2 N5K-1 e1/20 N7K-2 e1/22
POD 3 N5K-1 e1/19 N7K-1 e1/30
POD 3 N5K-1 e1/20 N7K-2 e1/30
POD X N5K-1 m0 MGMT Switch e1/7
POD X N5K-2 e1/4 3750 1/24
POD X N5K-2 e1/7 FEX B port1
POD X N5K-2 e1/8 FEX B port2
POD X N5K-2 e1/17 N5K-1 e1/17
POD X N5K-2 e1/18 N5K-1 e1/18
POD 1 N5K-2 e1/19 N7K-1 e1/16
POD 1 N5K-2 e1/20 N7K-2 e1/16
POD 2 N5K-2 e1/19 N7K-1 e1/24
POD 2 N5K-2 e1/20 N7K-2 e1/24
POD 3 N5K-2 e1/19 N7K-1 e1/32
POD 3 N5K-2 e1/20 N7K-2 e1/32
POD X N5K-2 m0 MGMT Switch e1/8
POD X NetApp-A bmc MGMT Switch e1/12
POD X NetApp-A e0a MGMT Switch e1/13
POD X NetApp-A e0b MGMT Switch e1/14
Cisco UCS Server Cabling Information
POD X ESX1 vmnic0 N5K-1 e1/9
POD X ESX1 vmnic2 FEX A e1/1
POD X ESX1 vmnic3 FEX B e1/1
POD X ESX1 cimc 3750 1/1
Nexus 1010 A&B Ethernet Cabling Information. Note: Require the use of two 1GbE Copper SFP+s (GLC-T=) on the
N5K side.

Table 6 - Ethernet Cabling Information - Management Switch
POD X MGMT Switch 1/0/1 ESX1 CIMC
POD X MGMT Switch 1/0/2 ESX1 vmnic
POD X MGMT Switch 1/0/7 N5K-1 m0
POD X MGMT Switch 1/0/8 N5K-2 m0
POD X MGMT Switch 1/0/9 MDS9124 m0
POD X MGMT Switch 1/0/10 VC Server RDC
POD X MGMT Switch 1/0/11 VC Server
POD X MGMT Switch 1/0/12 NTAP bmc
POD X MGMT Switch 1/0/13 NTAP e0a
POD X MGMT Switch 1/0/14 NTAP e0b
POD 1 MGMT Switch 1/0/15 FlexMGMT 1/37
POD 1 MGMT Switch 1/0/17 N7K-1 3/24
POD X MGMT Switch 1/0/23 N5K-1 e1/4
POD X MGMT Switch 1/0/24 N5K-2 e1/4

Table 7 - Fibre Channel Cabling Information
POD 1 N5K-1 fc2/3 MDS9124 fc1/1
POD 1 N5K-1 fc2/4 MDS9124 fc1/2
POD 1 N5K-2 fc2/3 MDS9124 fc1/3
POD 1 N5K-2 fc2/4 MDS9124 fc1/4
NetApp Controller
POD 1 NetApp-A 0a MDS9124 fc1/5
POD 1 NetApp-A 0b MDS9124 fc1/6
MDS
POD 1 MDS9124 fc1/1 N5K-1 fc2/3
POD 1 MDS9124 fc1/2 N5K-1 fc2/4
POD 1 MDS9124 fc1/3 N5K-2 fc2/3
POD 1 MDS9124 fc1/4 N5K-2 fc2/4
POD 1 MDS9124 fc1/5 NetApp A 0a
POD 1 MDS9124 fc1/6 NetApp A 0b

2.3 REQUIRED SOFTWARE VERSIONS

These are the resources and equipment required to complete the exercise:
NetApp 2020 with 7.3.5
Cisco Nexus 7000 devices with NX-OS 5.2(1)
Cisco Nexus 5000 and 2000 Unified Fabric Switches with NX-OS 5.0(3)N2(1)
Cisco Nexus 1000v with 4.2(1) SV1(4)
Cisco UCS 6120 with 1.4(3m) (Not Applicable)
MDS 9124 Fibre Channel Switch with NX-OS 5.0(1a)
VMware ESXi 4.1 u1
VMware vCenter 4.1 u1
Citrix XenDesktop/XenApp (Not Applicable)
XenDesktop 5
XenApp 6 2008
Provisioning Server 5.6 SP1
Internet access

Variable Name Customized Value Description
NetApp cluster license
code
0 Provide the license code to enable cluster mode
within the FAS2020 A configuration.
NetApp Fibre Channel
license code
0 Provide the license code to enable the Fibre Channel
protocol.
NetApp Flash Cache
license code
0 Provide the license code to enable the installed
Flash Cache adapter.
NetApp NearStore license
code
0 Provide the license code to enable the NearStore

capability, which is required to enable
deduplication.
NetApp deduplication
license code
0 Provide the license code to enable deduplication.
NetApp NFS license code 0 Provide the license code to enable the NFS protocol.
NetApp MultiStore license
code
0 Provide the license code to enable MultiStore
.
NetApp FlexClone license
code
0 Provide the license code to enable FlexClone.

2.4 GLOBAL CONFIGURATION VARIABLES
The variables shown in Table 8 are used throughout the deployment and are considered as global variables.
Table 8 - Data Center Virtualization global variables.
VLAN ID for NFS traffic 211 Provide the appropriate VLAN ID used for NFS traffic
throughout the DCV environment.
Network address for NFS traffic 10.1.211.0/24 Network address for NFS VLAN traffic in CIDR
notation (that is, 10.1.30.0/24).
VLAN ID for management traffic 111 VLAN ID used for management traffic throughout
the DCV environment.
VLAN ID for VMotion traffic 151 VLAN ID used for VMotion traffic throughout the
DCV environment.
Network address for VMotion
traffic
10.1.151.0/24 Network address for VMotion VLAN traffic in CIDR
notation (that is, 10.1.30.0/24).
VLAN ID for the Cisco Nexus
1000v packet and control traffic
171 Provide the appropriate VLAN ID used for the Cisco
Nexus 1000v packet and control traffic.
VLAN ID for native VLAN 999 Provide the appropriate VLAN ID that will be used
for the native VLAN ID throughout the DCV
environment.
VLAN ID for VM traffic 131 Provide the appropriate VLAN ID that will be used
for VM traffic by default.
Default password 1234Qwer Provide the default password that will be used in the
initial configuration of the environment.
DNS server name 10.1.111.10 Provide the IP address of the appropriate
nameserver for the environment.
Domain name suffix dcvlabs.lab Provide the appropriate domain name suffix for the
environment.
VSAN ID for fabric A 11 The VSAN ID that will be associated with fabric A.
This will be associated with both FC and FCoE traffic
for fabric A.
VSAN ID for fabric B 12 The VSAN ID that will be associated with fabric B.
This will be associated with both FC and FCoE traffic
for fabric B.
FCoE VLAN ID for fabric A 1011 Provide the VLAN ID of the VLAN that will be
mapped to the FCoE traffic on fabric A.
FCoE VLAN ID for fabric B 1012 Provide the VLAN ID of the VLAN that will be
mapped to the FCoE traffic on fabric B.
SSL country name code US Provide the appropriate SSL country name code.
SSL state or province name CA Provide the appropriate SSL state or province name.
SSL locality name San Jose Provide the appropriate SSL locality name (city,
town, and so on).
SSL organization name Cisco Provide the appropriate SSL organization name
(company name).
SSL organization unit WWPO Provide the appropriate SSL organization unit
(division).
NTP Server IP Address 80.84.57.23 Provide the NTP server IP address.

2.5 NETAPP CONFIGURATION VARIABLES
Table 9 through Error! Reference source not found. show the variables that are specific to the NetApp portion of
the deployment only.
Table 9 - NetApp FAS2020 A variables.
FAS2020 A hostname NTAP1-A Provide the hostname for NetApp FAS2020 A.
Netboot interface name Incomplete Designate the appropriate interface to use for initial
netboot of each controller. Interface e0M is the
recommended interface.
NetApp FAS2020 A netboot
interface IP address
Incomplete Provide the IP address for the netboot interface on
NetApp FAS2020 A.
interface subnet mask
Incomplete Provide the subnet mask for the netboot interface
on NetApp FAS2020 A.
interface gateway IP address
Incomplete Provide the gateway IP address for the netboot
interface on NetApp FAS2020 A.
NetApp Data ONTAP 7.3.5
netboot kernel location
Incomplete Provide the full TFTP path to the 7.3.5 Data ONTAP

boot image.
NetApp FAS2020 A management
10.1.111.151 Provide the IP address for the management
interface subnet mask
255.255.255.0 Provide the subnet mask for the management
interface gateway IP address
10.1.111.254 Provide the gateway IP address for the management
NetApp FAS2020 A
administration host IP address
10.1.111.100 Provide the IP address of the host that will be used
for administering the NetApp FAS2020 A.
NetApp FAS2020 A location Nevada Provide a description of the physical location where
the NetApp chassis resides.
NetApp FAS2020 A mailhost
name
Incomplete Provide the appropriate mail hostname.
NetApp FAS2020 A mail host IP
address
Incomplete Provide the appropriate mail host IP address.
NetApp Data ONTAP 7.3.5 flash
image location
Incomplete Provide the http or https Web address of the
NetApp Data ONTAP 7.3.5 flash image to install the
image to the onboard flash storage.
NetApp FAS2020 A
administrators e-mail address
pephan@cisco.com Provide the e-mail address for the NetApp
administrator to receive important alerts/messages
by e-mail.
NetApp FAS2020 A
infrastructure vFiler IP address
10.1.211.151 Provide the IP address for the infrastructure vFiler
unit on FAS2020 A.
Note: This interface will be used for the export of
NFS datastores and possibly iSCSI LUNs to
the necessary ESXi hosts.
NetApp FAS2020 A
infrastructure vFiler
administration host IP
10.1.111.10 Provide the IP address of the host that will be used
to administer the infrastructure vFiler unit on
FAS2020 A. This variable might have the same IP
address as the administration host IP address for the
physical controllers as well.


Table 10 - NetApp licensing variables.
NetApp cluster license code 0 Provide the license code to enable cluster mode
within the FAS2020 A configuration.
NetApp Fibre Channel license
code
0 Provide the license code to enable the Fibre Channel
protocol.
NetApp Flash Cache license code 0 Provide the license code to enable the installed
Flash Cache adapter.
NetApp NearStore license code 0 Provide the license code to enable the NearStore

capability, which is required to enable
deduplication.
NetApp deduplication license
code
0 Provide the license code to enable deduplication.
NetApp NFS license code 0 Provide the license code to enable the NFS protocol.
NetApp MultiStore license code 0 Provide the license code to enable MultiStore
.
NetApp FlexClone license code 0 Provide the license code to enable FlexClone.
Table 11 - NetApp disk and volume variables
NetApp FAS2020 A total disks
attached
9 Number of disks assigned to controller A using
software ownership.
Note: Do not include the three disks used for the
root volume in this number.
NetApp FAS2020 A total disks in
aggregate 1
9 Number of disks to be assigned to aggr1 on
controller A.
NetApp FAS2020 A ESXi boot
volume size
20g Each Cisco UCS server boots by using the FC
protocol. Each FC LUN will be stored in a volume on
either controller A or controller B. Choose the
appropriate volume size depending on how many
ESXi hosts will be in the environment.

2.6 CISCO CONFIGURATION VARIABLES
Table 12 and Table 13 show the variables that are specific to the Cisco portion of the deployment.
Table 12 - Cisco Nexus 5010 variables.
Cisco Nexus 5010 A hostname N5K-1 Provide the hostname for the Cisco Nexus 5010 A.
Cisco Nexus 5010 B hostname N5K-2 Provide the hostname for the Cisco Nexus 5010 B.
Cisco Nexus 5010 A
management interface IP
address
10.1.111.1 Provide the IP address for the mgmt0 interface on
the Cisco Nexus 5010 A.
Cisco Nexus 5010 B
address
10.1.111.2 Provide the IP address for the mgmt0 interface on
the Cisco Nexus 5010 B.
Cisco Nexus 5010 A
management interface subnet
mask
255.255.255.0 Provide the subnet mask for the mgmt0 interface on
the Cisco Nexus 5010 A.
Cisco Nexus 5010 B
management interface subnet
mask
255.255.255.0 Provide the subnet mask for the mgmt0 interface on
the Cisco Nexus 5010 B.
Cisco Nexus 5010 A
management interface gateway
IP address
10.1.111.254 Provide the gateway IP Address for the mgmt0
interface on the Cisco Nexus 5010 A.
Cisco Nexus 5010 B
IP address
10.1.111.254 Provide the gateway IP address for the mgmt0
interface on the Cisco Nexus 5010 B.
Cisco Nexus 5010 virtual port
channel (vPC) domain ID
10 Provide a unique vPC domain ID for the
environment.
Table 13 - Cisco Nexus 1000v variables.
Primary Cisco Nexus 1000v
virtual supervisor module host
name
vsm-1 Provide the hostname for the primary VSM.
virtual supervisor module
address
10.1.111.17 Provide the IP address for the management
interface for the primary Cisco Nexus 1000v virtual
supervisor module.
management interface netmask
255.255.255.0 Provide the netmask for the management interface
for the primary Cisco Nexus 1000v virtual supervisor
module.
10.1.111.254 Provide the gateway for the management interface
for the primary Cisco Nexus 1000v virtual supervisor
module.
Cisco Nexus 1000v virtual
supervisor module domain ID
11 Provide a unique domain ID for the Cisco Nexus
1000v VSMs. This domain ID should be different
than the domain ID used for the Cisco Nexus 1010
virtual appliance domain ID.

2.7 VMWARE CONFIGURATION VARIABLES
Table 14 shows the variables that are specific to the VMware portion of the deployment.
Table 14 - VMware variables.
ESXi server 1 hostname ESX1 The hostname for the first ESXI host in the
infrastructure cluster.
ESXi server 1 management
10.1.111.21 The IP address for the management VMkernel port
on the first host in the infrastructure cluster.
interface netmask
255.255.255.0 The netmask for the management VMkernel port on
the first host in the infrastructure cluster.
interface gateway
10.1.111.254 The gateway for the management VMkernel port on
the first host in the infrastructure cluster.
ESXi server 1 NFS VMkernel
10.1.211.21 The IP address for the NFS VMkernel port on the
first host in the cluster.
interface netmask
255.255.255.0 The netmask for the NFS VMkernel port on the first
host in the infrastructure cluster.
ESXi Server 1 VMotion
VMkernel interface IP address
10.1.151.21 The IP address for the VMotion VMkernel port on
the first host in the cluster.
ESXi server 1 VMotion
VMkernel interface netmask
255.255.255.0 The netmask for the VMotion VMkernel port on the
first host in the infrastructure cluster.
ESXi server 2 hostname ESX2 The hostname for the second ESXi host in the
infrastructure cluster.
10.1.111.22 The IP address for the management VMkernel port
on the second host in the infrastructure cluster.
interface netmask
255.255.255.0 The netmask for the management VMkernel port on
the second host in the infrastructure cluster.
interface gateway
10.1.111.254 The gateway for the management VMkernel port on
the second host in the infrastructure cluster.
10.1.211.22 The IP address for the NFS VMkernel port on the
second host in the cluster.
interface netmask
255.255.255.0 The netmask for the NFS VMkernel port on the
second host in the infrastructure cluster.
VMkernel interface IP address
10.1.211.22 The IP address for the VMotion VMkernel port on
the second host in the cluster.
VMkernel interface netmask
255.255.255.0 The netmask for the VMotion VMkernel port on the
second host in the infrastructure cluster.
SQL server VM hostname n/a The hostname of the SQL Server

virtual machine
that runs the vCenter server database.
SQL server VM IP address n/a The IP address of the SQL Server virtual machine
that runs the vCenter server database.
vCenter server VM hostname VCSERVER The hostname of the vCenter server virtual machine.
vCenter server VM IP address 10.1.111.100 The IP address of the vCenter server virtual
machine.
vCenter server license key The vCenter server license key.
vSphere license key The vSphere license key.

3 CISCO NEXUS AND MDS DEPLOYMENT PROCEDURE: PART I

The following section provides a detailed procedure for configuring the Cisco Nexus 5010 switches for use in a
DCV environment. Complete this lab exercise to learn how to configure Virtual Port Channeling (vPC), Fibre
Channel over Ethernet (FCoE), and Fabric Extender (FEX Nexus 2000) using the NX-OS command line
interface.

Note: The Data Center Virtualization labs start up with completed configurations for VPC, FCoE, and FEX.
Sections 3 - 5 provide you with the opportunity to build up these configurations from the ground up.
If you just want to test or demo other features such as OTV or Nexus 1000v then please proceed to
Section 6.

Complete this lab exercise to practice initial switch configuration.

EXERCISE OBJECTIVE
In this exercise you will use the NX-OS CLI to configure vPC and FEX in a Dual Homed Fabric Extender vPC
Topology. After completing these exercises you will be able to meet these objectives:

Clear the current startup configuration and reboot the switch
Recover from the loader prompt
Start the interactive setup process on the Nexus 5000 and MDS 9124 switch
Configure a Nexus 5000 and an MDS 9124 switch for out-of-band management
Navigate through the switch CLI structure on the Nexus 5000 and MDS 9124
Use command completion and help
Save the running configuration
Save the switch configuration to a tFTP/FTP server
Enable the vPC feature
Create a vPC domain and enter vpc-domain mode
Configure the vPC peer keepalive link
Configure vPC role priority
Create the vPC peer link
Move the PortChannel to vPC
Configuring VSANs and Fibre Channel Interfaces
Configure Zones and Zone Sets
Map a VSAN for FCoE traffic onto a VLAN
Create virtual Fibre Channel interfaces to carry the FCoE traffic
Configure an Ethernet Interface

COMMAND LIST
The commands used in this exercise are described in the table below.

Table 15 - Commands
Command Description
write erase boot Erases the switchs startup configuration
boot kickstart
bootflash:filename
Configures the boot variable for the kickstart software image to the file named in
bootflash:
boot system
bootflash:filename
Configures the boot variable for the system software image to the file named in
bootflash:
show boot Displays the boot variable configuration
reload Reboots the switch
setup Enter the basic device setup dialog
show ? Displays all the permissible features for the show command for the current user
show running-config Shows the running configuration
show interface brief Displays an interface status summary
show vlan Displasy VLAN configuration and status
show vsan Displays VSAN configuration and status
show version Displays current code version
show environment Displays environment-related switch information, such as fan, power, and temperature
status
config term Enters configuration mode
ping Packet internet gopher used to determine network connectivity
interface fc1/3 Enters configuration submode for FC port 3 on module 1
show module Displays all the modules associated with the network device
copy tftp://x.x.x.x/filename
bootflash:/filename
Copies a file (filename) from the tftp server with the address x.x.x.x to the bootflash:
load bootflash:/filename Loads the system file (filename) from the bootflash: when booting from the loader
prompt
show file volatile Examine the contents of the configuration file in the volatile file system
del file volatile Deletes the file from the volatile system
dir volatile Display volatile file to confirm action
exit Exits one level in the menu structure. If you are in EXEC mode this command will log
you off the system
end Exits configuration mode to EXEC mode
shut Disables an interface
no shut Enables an interface
copy running-config startup-
config
Saves the running configuration as the startup configuration
copy running-config
tftp://ip_address/path
Saves the running configuration to a TFTP server

copy tftp Copy the system file from the TFPT server to the local bootflash
load bootflash Loads the system file from bootflash
show fcns database Shows the FCNS database
dir [volatile: |
bootflash:]
Displays the contents of the specified memory area

show file name Displays the contents of the specified file
del name Deletes the specified file


Table 16 - Commands
Command Description
show interface Ethernet 1/4
capabilities Ethernet1/4
Verify that interface is Gigabit capable
configure terminal Enter configuration mode
interface Ethernet Enter interface mode
shutdown Shut down an interface
speed 1000 Set the port speed to 1 Gig
description Insert 1 Gb SFP here Adds a description to the Ethernet interface
no shutdown Bring interface out of shutdown mode
end Exit current configuration mode
show interface Ethernet 1/4 Display the configuration, used to confirm changes made
switchport mode trunk Sets interface to trunk mode
switchport mode trunk allowed Allows VSANs to traverse the trunk mode
spanning-tree port type edge
trunk
Enable Portfast
interface vfc 3 Create a virtual Fibre Channel interface
bind interface Ethernet 1/1 Binds the virtual Fibre Channel interface to a physical Ethernet interface
vsan database Enter VSAN configuration mode
vsan 10 interface vfc 3 Add the Virtual Fibre interface to the VSAN
vlan 100 Enter VSAN configuration mode
fcoe vsan 10 Bind the Ethernet VLAN to the FCoE VSAN
show interface vfc 3 View the configuration information of the virtual FC interface
show interface brief | include
vfc
View all of the virtual Fibre Channel interfaces
interface fc 2/1 2 Enter configuration mode for FC interfaces
Selected a range of Used to select the physical FC interfaces
switchport ? Used to examine the available switchport options
switchport mode auto Configures the port mode to auto-negotiation on the FC ports
switchport speed auto Configures the port speed to auto-negotiation on the FC ports
show interface fc2/1 View and verify FC interface configuration
<snip>
interface vfc Enter configuration mod for virtual Fibre Channel interface
description to Virtual Center Add a description to the virtual Fibre Channel interface
show flogi database Verify that devices have completed a fabric login into the Nexus 5000
show fcns database Verify devices have registered in the Fibre Channel server
switchport mode NP Define a port as a N-Port proxy
show npv status Verify NIV port configurations

STARTING THE TFTP SERVER

Log on to the VC_SERVER from the SSL Dashboard (if prompted). The username/password is
administrator/1234Qwer

Double-click on the tftpd32 or tftpd64 icon on the desktop. The default directory is c:\tftp:

JOB AIDS

Nexus 5000 CLI Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli/CLIConfiguratio
nGuide.html

Cisco Nexus 5000 Series Switches - Virtual PortChannel Quick Configuration Guide
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configuration_guide_c07-543563.html

Cisco Nexus 5000 Series NX-OS Software Configuration Guide - Configuring Virtual Interfaes
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/cli_rel_4_0_1a/VirtIntf.html

Cisco Nexus 5000 Series Switch Fabric Manager Software Configuration Guide
http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration/guide/fm/FabricManager.html

Cisco MDS 9000 Family CLI Quick Configuration Guide - Configuring VSANs and Interfaces
http://www.cisco.com/en/US/docs/storage/san_switches/mds9000/sw/san-os/quick/guide/qcg_vin.html

3.1 NEXUS INITIAL SETUP

EXERCISE OBJECTIVE
In this exercise you will configure the Nexus 5000 and MDS 9124 switches for out-of-band management and
explore the Nexus Operating System (NxOS) command line interface (CLI). After completing the exercise you will
be able to meet these objectives:

Clear the current startup configuration and reboot the switch
Recover from the loader prompt
Start the interactive setup process on the Nexus 5000 and MDS 9124 switch
Configure a Nexus 5000 and an MDS 9124 switch for out-of-band management
Navigate through the switch CLI structure on the Nexus 5000 and MDS 9124
Use command completion and help
Save the running configuration locally and to a remote server

Step 1 Perform initial Cisco Nexus 5010 Switch setup
Duration: 60-75 minutes
Cisco Nexus 5010 A - N5K-1
1.1 Access N5K-1 using the console button on the lab interface.
1.2 The prompt should be at the System Admin Account Setup. Run through the setup script.
1.3 If the switch is not at the System Admin Account Setup, log into the switch and issue the following
commands.
switch# write erase
Warning: This command will erase the startup-configuration.
Do you wish to proceed anyway? (y/n) [n] y
switch# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
1.4 Upon initial boot and connection to the serial or console port of the switch, the NX-OS setup
should automatically start.
Do you want to enforce secure password standard (yes/no): yes
Enter the password for "admin": 1234Qwer
Confirm the password for "admin": 1234Qwer
---- Basic System Configuration Dialog ----
<snip>
Would you like to enter the basic configuration dialog(yes/no): yes
Create another login account (yes/no) [n]: n
Configure read-only SNMP community string (yes/no) [n]: n
Configure read-write SNMP community string (yes/no) [n]: n
Enter the switch name : N5K-1
Continue with Out-of-band (mgmt0) management configuration? (yes/no) [y]: y
Mgmt0 IPv4 address : 10.1.111.1
Mgmt0 IPv4 netmask : 255.255.255.0
Configure the default gateway? (yes/no) [y]:y
IPv4 address of the default gateway : 10.1.111.254
Enable the telnet service? (yes/no) [n]: n
Enable the http-server? (yes/no) [y]: y
Enable the ssh service? (yes/no) [y]: y
Type of ssh key you would like to generate (dsa/rsa) : rsa
Number of key bits <768-2048> : 1024
Configure the ntp server? (yes/no) [n]: n
Enter basic FC configurations (yes/no) [n]: n

1.5 Review you configuration.
The following configuration will be applied:
switchname N5K-1
interface mgmt0
ip address 10.1.111.1 255.255.255.0
no shutdown
exit
vrf context management
ip route 0.0.0.0/0 10.1.111.254
<snip>
Would you like to edit the configuration? (yes/no) [n]: n
1.6 Enable and save your configuration.
Use this configuration and save it? (yes/no) [y]: y
[########################################] 100%

Cisco Nexus 5010 B - N5K-2
1.7 Log in to the Nexus 5000 using the console button on the lab interface. The prompt should be at
the System Admin Account Setup. Run through the setup script.
commands.
switch# write erase
switch# reload
WARNING: This command will reboot the system
Do you want to continue? (y/n) [n] y
1.9 Upon initial boot and connection to the serial or console port of the switch, the NX-OS setup
should automatically start.
Do you want to enforce secure password standard (yes/no): yes
<snip>
Would you like to enter the basic configuration dialog(yes/no): yes

Enter the switch name : N5K-2
Configure the default gateway? (yes/no) [y]:y
Type of ssh key you would like to generate (dsa/rsa) : rsa
Number of key bits <768-2048> : 1024
Enter basic FC configurations (yes/no) [n]: n
<snip>
[########################################] 100%

MANAGEMENT VRF

The default gateway is connected through the management interface. The management interface is by default
part of the management VRF. This particular VRF is part of the default configuration and the management
interface mgmt0 is the only interface allowed to be part of this VRF.

The philosophy behind Management VRF is to provide total isolation to the management traffic from the rest
of the traffic flowing through the box by confining the former to its own forwarding table.

These are the steps for the exercise:
Verify that only the mgmt0 interface is part of the management VRF
Verify that the default gateway is reachable only using the management VRF

Step 2 Verify that only the mgmt0 interface is part of the management VRF.
2.1 Log in to N5K-1
N5K-1 login: admin
Password: 1234Qwer
2.2 Show available VRF. Then, show interfaces in management VRF.
N5K-1# show vrf
VRF-Name VRF-ID State Reason
default 1 Up --
management 2 Up --
N5K-1# show vrf management interface
Interface VRF-Name VRF-ID
mgmt0 management 2

Step 3 Verify that the default gateway is reachable only using the management VRF
3.1 Ping the default gateway using the default VRF.
N5K-1# ping 10.1.111.254
PING 10.1.111.254 (10.1.111.254): 56 data bytes
ping: sendto 10.1.111.254 64 chars, No route to host
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

--- 10.1.111.254 ping statistics ---
5 packets transmitted, 0 packets received, 100.00% packet loss

Note: The ping fails because the default gateway is reachable only from the management interface, while
we used the default VRF.

3.2 Lets try again specifying the management VRF.
N5K-1# ping 10.1.111.254 vrf management
PING 10.1.111.254 (10.1.111.254): 56 data bytes
Request 0 timed out
64 bytes from 10.1.111.254: icmp_seq=1 ttl=254 time=2.361 ms

--- 10.1.111.254 ping statistics ---
round-trip min/avg/max = 2.361/3.593/4.07 ms
N5K-1#
3.3 Alternatively, we can set the routing context for the VRF management interface to allow for layer 3
access. This will also allow you to ping and TFTP as needed in the following exercises
N5K-1# routing-context vrf management
3.4 Ping the tFTP server
N5K-1%management# ping 10.1.111.10
PING 10.1.111.10 (10.1.111.10): 56 data bytes
Request 0 timed out

--- 10.1.111.10 ping statistics ---
3.5 Set the routing context back to default:
N5K-1%management# routing-context vrf default
Step 4 Repeat Step 3 to verify connectivity.

EXPLORING THE NEXUS 5000 CLI
In this task you will briefly explore the Nexus 5000 command line interface (CLI).

Cisco Nexus 5010 A or B
Step 5 Type ? to view the current command options.
N5K-1# ?
5.1 Display all commands that begin with s, sh, and show. Press Enter or space to scroll through the
list of commands.
N5K-1# s?
N5K-1# sh?
N5K-1# show ?
5.2 Display the current running configuration.
N5K-1# show running-config
5.3 Display the current installed version of code and environmental information.
N5K-1# show version
N5K-1# show environment
5.4 Display the Ethernet and Fibre Channel modules of the Nexus 5020. This is where youll find the
WWN range for the FC ports and the range of Ethernet addresses for the 10 Gigabit Ethernet
ports. The first address (whether FC or Ethernet) is associated with port 1 of that transport type
and subsequent ascending address numbers are associated with the next ascending port number.
N5K-1# show module
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ---------------------- ------------
1 20 20x10GE/Supervisor N5K-C5010P-BF-SUP active *
2 8 4x10GE + 4x1/2/4G FC Module N5K-M1404 ok

Mod Sw Hw World-Wide-Name(s) (WWN)
--- -------------- ------ --------------------------------------------------
1 5.0(2)N2(1) 1.2 --
2 5.0(2)N2(1) 1.0 2f:6c:69:62:2f:6c:69:62 to 63:6f:72:65:2e:73:6f:00

Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
1 0005.9b7a.03c8 to 0005.9b7a.03ef JAF1413CEGC
2 0005.9b7a.03f0 to 0005.9b7a.03f7 JAF1409ASQD
N5K-1#

Abbreviate the syntax, then hit tab key to complete each word; for example, type sh<tab> ru<tab>.

5.5 Display the status of the switch interfaces. Notice that only Ethernet interfaces are listed.
N5K-1# show interface brief (abbr: sh int bri)

--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth1/1 1 eth access down SFP validation failed 10G(D) --
Eth1/2 1 eth access down SFP not inserted 10G(D) --
Eth1/3 1 eth access down Link not connected 10G(D) --
Eth1/4 1 eth access down Link not connected 10G(D) --
<snip>

--------------------------------------------------------------------------------
Port VRF Status IP Address Speed MTU
--------------------------------------------------------------------------------
mgmt0 -- up 10.1.111.1 1000 1500
5.6 Display VLAN information.
N5K-1# show vlan
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4
Eth1/5, Eth1/6, Eth1/7, Eth1/8
Eth1/9, Eth1/10, Eth1/11
Eth1/12, Eth1/13, Eth1/14
Eth1/15, Eth1/16, Eth1/17
Eth1/18, Eth1/19, Eth1/20
Eth2/1, Eth2/2, Eth2/3, Eth2/4

Remote SPAN VLANs
-------------------------------------------------------------------------------

Primary Secondary Type Ports
------- --------- --------------- -------------------------------------------

5.7 Display VSAN information.
N5K-1# show vsan
^
% Invalid command at '^' marker.

The fcoe feature must be activated to use the fibre channel features.

5.8 Activate the fcoe features:
N5K-1# configure terminal
N5K-1(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
N5K-1# show vsan
vsan 1 information
name:VSAN0001 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:down

vsan 4079:evfp_isolated_vsan

vsan 4094:isolated_vsan

5.10 Log into N5K-2 and activate the fcoe feature on N5K-2.
N5K-2 login: admin
Password: 1234Qwer

N5K-2# conf t
Enter configuration commands, one per line. End with CNTL/Z.
N5K-2(config)# feature fcoe
FC license checked out successfully
fc_plugin extracted successfully
FC plugin loaded successfully
FCoE manager enabled successfully
FC enabled on all modules successfully
N5K-2(config)# exit

SAVING THE NEXUS 5000 SWITCH CONFIGURATION
In this task you will save and restore switch configurations using the command line interface (CLI).

Step 6 Update the startup configuration with the changes made in the running configuration.
6.1 Save the running configuration for N5K-2
N5K-1# copy running-config startup-config
[########################################] 100%
6.2 Access your desktop (Username: Administrator/ password: 1234Qwer) and start your TFTP server
6.3 Save your running configuration on the tFTP server.
N5K-1# copy running-config tftp://10.1.111.100/N5K-1-Lab1-config
Enter vrf (If no input, current vrf 'default' is considered): management
Trying to connect to tftp server......
Connection to Server Established.

TFTP put operation was successful
Note: Be sure you start the tFTP/FTP Server before attempting to save the configuration or your copy will
fail. Please review Lab 0 Lab Services for instructions on how to use the tFTP/FTP server.
Use a tFTP/FTP Server in production networks to keep backup configurations and code releases for
each network device. Be sure to include these servers in your regular Data Center backup plans.
6.4 Save the running configuration for N5K-2
N5K-2# copy run start
[########################################] 100%

3.2 ENABLE FEATURES
Step 7 Enable the appropriate Cisco Nexus features
Duration: 5 minutes
7.1 Enable Virtual Port Channel.
7.2 Enable LACP port channel negotiation.
7.3 Enable FC and Fibre Channel over Ethernet.
7.4 Enable N Port ID Virtualization.
7.5 Enable Fibre Channel port channeling and trunking.
7.6 Enable Fabric Extender.
feature vpc
feature lacp
feature fcoe
feature npiv
feature fport-channel-trunk
feature fex

7.7 Enable Virtual Port Channel.
7.8 Enable LACP port channel negotiation.
7.9 Enable FC and Fibre Channel over Ethernet.
7.10 Enable N Port ID Virtualization.
7.11 Enable Fibre Channel port channeling and trunking.
7.12 Enable Fabric Extender.
feature vpc
feature lacp
feature fcoe
feature npiv
feature fex

Type show feature and verify that the appropriate licenses are enabled.
N5K-1(config)# show feature | i enabled
assoc_mgr 1 enabled
fcoe 1 enabled
fex 1 enabled
fport-channel-trunk 1 enabled
lacp 1 enabled
lldp 1 enabled
npiv 1 enabled
sshServer 1 enabled
vpc 1 enabled

3.3 NEXUS GLOBAL SETTINGS
Step 8 Set global configurations
Duration: 5 minutes
8.1 From the global configuration mode, enable bpduguard on all edge ports by default.
spanning-tree port type edge bpduguard default
8.2 Enable bpdufilter on all edge ports by default.
spanning-tree port type edge bpdufilter default
8.3 Create an access list to match Platinum traffic. The ACL is matching for traffic from NFS vlan.
ip access-list ACL_COS_5
10 permit ip 10.1.211.0/24 any
20 permit ip any 10.1.211.0/24
8.4 Create a Class Map for Platinum Traffic.
class-map type qos CLASS-PLATINUM
match access-group name ACL_COS_5
8.5 Create an access list to match Silver traffic.
10 permit ip 10.1.151.0/24 any
20 permit ip any 10.1.151.0/24
8.6 Create a Class Map for Silver Traffic.
class-map type qos CLASS-SILVER
8.7 Create a policy map that will be used for tagging incoming traffic.
policy-map type qos POL_CLASSIFY
class CLASS-PLATINUM
set qos-group 2
exit
class CLASS-SILVER
set qos-group 4
exit
exit

8.8 Create a network-qos class map for Platinum traffic to be used in a Network QoS policy.
class-map type network-qos CLASS-PLATINUM_NQ
match qos-group 2
8.9 Create a network-qos Class Map for Silver traffic to be used in a Network QoS policy.
class-map type network-qos CLASS-SILVER_NQ
match qos-group 4
8.10 Create a network-qos policy map to be applied to the System QoS policy. Set Platinum class to CoS
value of 5 and to MTU of 9000. Set Silver class to CoS value of 4 and to MTU of 9000. Set Default
class to MTU of 9000.
policy-map type network-qos POL_SETUP_NQ
class type network-qos CLASS-PLATINUM_NQ
set cos 5
mtu 9000
exit
class type network-qos CLASS-SILVER_NQ
set cos 4
mtu 9000
exit
!!! The following section will enable Jumbo Frames for all unclassified traffic.
class type network-qos class-default
mtu 9000
exit

8.11 Associate the policies to the system class policy map using service policies.
system qos
service-policy type qos input POL_CLASSIFY
service-policy type network-qos POL_SETUP_NQ
exit
8.12 Save your configuration.
copy run start

Find out more about Configuring QoS on the Nexus 5000.
http://www.cisco.com/en/US/docs/switches/datacenter/nexus50
00/sw/qos/Cisco_Nexus_5000_Series_NX-
OS_Quality_of_Service_Configuration_Guide_chapter3.html
8.13 Repeat steps 8.1 8.12 for N5K-2.
spanning-tree port type edge bpduguard default
spanning-tree port type edge bpdufilter default
10 permit ip 10.1.211.0/24 any
20 permit ip any 10.1.211.0/24
class-map type qos CLASS-PLATINUM
10 permit ip 10.1.151.0/24 any
20 permit ip any 10.1.151.0/24
class-map type qos CLASS-SILVER
set qos-group 2
exit
class CLASS-SILVER
set qos-group 4
exit
exit
match qos-group 2
match qos-group 4
set cos 5
mtu 9000
exit
set cos 4
mtu 9000
exit
mtu 9000
exit
system qos
exit
copy run start

Use the show run command to view the global spanning-tree configuration
N5K-1(config)# show run ipqos
class-map type qos class-fcoe
class-map type qos match-all CLASS-SILVER
class-map type qos match-all CLASS-PLATINUM
class-map type queuing class-all-flood
match qos-group 2
class-map type queuing class-ip-multicast
match qos-group 2
set qos-group 2
class CLASS-SILVER
set qos-group 4
match qos-group 4
class-map type network-qos class-all-flood
match qos-group 2
match qos-group 2
class-map type network-qos class-ip-multicast
match qos-group 2
set cos 5
mtu 9000
set cos 4
mtu 9000
class type network-qos class-fcoe
pause no-drop
mtu 2158
mtu 9000
system qos

Step 9 Create necessary VLANs
Duration: 5 minutes
9.1 Create VLANs for Management, VM, vMotion, Nexus 1000V Control and Packet, and NFS storage
traffic.
vlan 111
name MGMT-VLAN
vlan 131
name VMTRAFFIC
vlan 151
name VMOTION
vlan 171
name PKT-CTRL
vlan 211
name NFS-VLAN

9.2 Create VLANs for Management, VM, vMotion, Nexus 1000V Control and Packet, and NFS storage
traffic.
vlan 111
name MGMT-VLAN
vlan 131
name VMTRAFFIC
vlan 151
name VMOTION
vlan 171
name PKT-CTRL
vlan 211
name NFS-VLAN

9.3 Use the show vlan command to show the list of VLANs that have been created on the switch.
N5K-1(config-vlan)# show vlan | include "Status|active" | exclude VLAN0
1 default active Eth1/1, Eth1/2, Eth1/3, Eth1/4
10 INFRA-MGMT-VLAN active
110 MGMT active
111 VMTRAFFIC-VLAN active
151 VMOTION-VLAN active
171 PKT-CTRL-VLAN active

3.4 NEXUS CONFIGURE ETHERNET INTERFACES

Step 10 Add individual port descriptions for troubleshooting
Duration: 10 minutes
10.1 Placeholder for new storage array.
interface Eth1/1
description NTAP1-A:e2a
interface Eth1/2
description NTAP1-B:e2a
10.2 Router uplink.
interface Eth1/4
description To 3750:
10.3 FEX ports.
interface Eth1/7
description N2K-1:
interface Eth1/8
description N2K-1:
10.4 Server ports.
interface Eth1/9
description ESX1:vmnic0
interface Eth1/10
interface Eth1/11
10.5 vPC peer link ports.
interface Eth1/17
description N5K-2:Eth1/17
interface Eth1/18
10.6 OTV uplinks.
interface Eth1/19
description N7K-1:
interface Eth1/20
description N7K-2:

10.7 Placeholder for new storage array.
interface Eth1/1
description NTAP1-A:e2b
interface Eth1/2
description NTAP1-B:e2b
10.8 Router uplink.
interface Eth1/4
description To 3750
10.9 FEX ports.
interface Eth1/7
description N2K-2:
interface Eth1/8
description N2K-2:
10.10 Server ports.
interface Eth1/9
interface Eth1/10
interface Eth1/11
10.11 vPC peer link ports.
interface Eth1/17
interface Eth1/18
10.12 OTV uplinks.
interface Eth1/19
description N7K-1:
interface Eth1/20
description N7K-2:


Step 11 Use the show interface status command to print a list of ports and corresponding information
including configured port descriptions
11.1 Output from N5K-1
N5K-1(config-if)# show interface status | include ":"

Eth1/1 NTAP1-A:e2a sfpAbsent 1 full 10G 10g
Eth1/2 NTAP1-B:e2a sfpAbsent 1 full 10G 10g
Eth1/4 To 3750: sfpInvali 1 full 10G 10g
Eth1/7 N2K-1: connected 1 full 10G 10g
Eth1/8 N2K-1: connected 1 full 10G 10g
Eth1/9 ESX1:vmnic0 connected 1 full 10G 10g
Eth1/17 N5K-2:Eth1/17 connected 1 full 10G 10g
Eth1/18 N5K-2:Eth1/18 connected 1 full 10G 10g
Eth1/19 N7K-1: notconnec 1 full 10G 10g
11.2 Output from N5K-2
N5K-2(config-if)# sh interface status | i :
Eth1/1 NTAP1-A:e2b sfpAbsent trunk full 1000 10g
Eth1/2 NTAP1-B:e2b sfpAbsent 1 full 10G 10g
Eth1/4 To 3750: sfpInvali trunk full 10G 10g
Eth1/7 N2K-2: vpcPeerLn 1 full 10G 10g
Eth1/8 N2K-2: vpcPeerLn 1 full 10G 10g
Eth1/17 N5K-1:Eth1/17 connected trunk full 10G 10g
Eth1/18 N5K-1:Eth1/18 connected trunk full 10G 10g

3.5 CONFIGURING PORT CHANNELS
Step 12 Create necessary port channels
12.1 Port channel for Virtual Port Channel Peer Link between Nexus 5000s.
interface Po1
description vPC peer-link
interface Eth1/17-18
channel-group 1 mode active
no shutdown
12.2 Port channel for Netapp NTAP1-A.
interface Po11
description NTAP1-A
interface Eth1/1
no shutdown
12.3 Define port channel for Netapp NTAP1-B.
interface Po12
description NTAP1-B
interface Eth1/2
no shutdown
12.4 Define port channel for servers. Add server host link to port-channel group.

For VPC and FCoE, we recommend setting channel-mode to on versus active (aka LACP). This is
useful for operating systems that dont support port-channel negotiation such as ESXi.
interface Po13
description ESX1
interface Eth1/9
channel-group 13 mode on
no shutdown
interface Po14
description ESX2
interface Eth1/10
no shutdown
interface Po15
description ESX3
interface Eth1/11
no shutdown
12.5 Port channel for L3 Switch.
interface Po20
description 3750
interface Eth1/4
no shutdown
12.6 Port channel for FEX 101.
interface Po101
description FEX1
interface Eth1/7-8
no shutdown
12.7 Save your configuration
copy run start

12.8 From the global configuration mode, type
interface Po1
interface Eth1/17-18
no shutdown
interface Po11
description NTAP1-A
interface Eth1/1
no shutdown
interface Po12
description NTAP1-B
interface Eth1/2
no shutdown
interface Po13
description ESX1
interface Eth1/9
no shutdown
interface Po14
description ESX2
interface Eth1/10
no shutdown
interface Po15
description ESX3
interface Eth1/11
no shutdown
interface Po20
description 3750
interface Eth1/4
no shutdown
interface Po101
description FEX2
interface Eth1/7-8
no shutdown

copy run start

12.10 Verify that the portchannel descriptions have been entered.
N5K-1(config-vlan)# show interface status | inc Po
Port Name Status Vlan Duplex Speed Type
Po1 vPC peer-link connected 1 full 10G --
Po11 NTAP1-A noOperMem 1 auto auto --
Po12 NTAP1-B noOperMem 1 auto auto --
Po13 ESX1 connected 1 full 10G --
Po20 3750 noOperMem 1 auto auto --
Po101 FEX1 noOperMem 1 full auto --

N5K-2(config)# show interface status | inc Po
Port Name Status Vlan Duplex Speed Type
Po1 vPC peer-link connected trunk full 10G --
Po11 NTAP1-A noOperMem 1 auto auto --
Po12 NTAP1-B noOperMem 1 auto auto --
Po20 3750 noOperMem 1 auto auto --
Po101 FEX2 noOperMem 1 full auto --

12.11 Verify that the correct individual ports have been added to the correct port-channel.
N5K-1(config-vlan)# show port-channel summary
<snip>
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/17(P) Eth1/18(P)
11 Po11(SD) Eth LACP Eth1/1(D)
13 Po13(SU) Eth NONE Eth1/9(P)
101 Po101(SD) Eth LACP Eth1/7(I) Eth1/8(I)

N5K-2(config)# show port-channel summary
<snip>
--------------------------------------------------------------------------------
Channel
--------------------------------------------------------------------------------
1 Po1(SU) Eth LACP Eth1/17(P) Eth1/18(P)
101 Po101(SD) Eth LACP Eth1/7(I) Eth1/8(I)


Step 13 Add port channel configurations/
int Po1
switchport mode trunk
switchport trunk native vlan 999
switchport trunk allowed vlan 111,211,171,151,131
spanning-tree port type network
no shut
Note: Do not allow any vlans that carry FCoE traffic on the vPC peer link .
13.2 Configure port-channel for NetApp.
int Po11-12
spanning-tree port type edge trunk
no shut
13.3 Configure port-channel for ESX Servers. They will allow vlans 111,211,171,151,and 131.
int Po13-15
no shut
13.4 Configure port channel for L3 Switch. Our L3 switch is 1GB so we set our speed to 1000.
interface Po20
speed 1000
no shutdown
copy run start


int Po1
no shut
13.7 Configure port-channel for NetApp.
int Po11-12
no shut
13.8 Configure port-channel for ESX Servers.
int Po13-15
no shut
13.9 Configure port channel for L3 Switch. Our L3 switch is 1GB so we set our speed to 1000.
interface Po20
speed 1000
no shutdown
copy run start

Step 14 Use the show run interface <interface name> command to show the configuration for a given
interface or portchannel.
N5K-1(config-if-range)# sh run int po1,po11-15,po20
interface port-channel1

description NTAP1-A

description NTAP1-B

description ESX1

<snip>

description 3750

N5K-2(config-if)# sh run int po1,po11-15,po20
vpc peer-link

description NTAP1-A

description NTAP1-B

description ESX1

description ESX2

description ESX3

description 3750

3.6 CONFIGURING VIRTUAL PORT CHANNELS

To use Virtual Port Channeling, you must activate the feature on the Nexus 5000 switch. It is recommended that
you use LACP for link aggregation to other switches. vPC configuration on the Cisco Nexus 5000 Series includes
these steps:

1. Enable the vPC feature.
2. Create a vPC domain and enter vpc-domain mode.
3. Configure the vPC peer keepalive link.
4. (Optional) Configure system priority.
5. (Optional) Configure vPC role priority.
6. Create the vPC peer link.
7. Move the PortChannel to vPC.

Step 15 Configure virtual portchannels (vPCs)
15.1 Create the vPC domain. The domain ID must match between VPC peers, but must differ from other
VPC pairs.
vpc domain 10
15.2 Configure the vPC role priority (optional): We will make N5K-1 the primary switch.
The switch with the lower priority will be elected as the vPC primary switch.
role priority 10
15.3 Configure the peer keepalive link. The management interface IP address for N5K-2 is 10.1.111.2 :
peer-keepalive destination 10.1.111.2 source 10.1.111.1

The system does not create the vPC peer link until you configure a vPC peer keepalive link.
15.4 Designate the port-channel to be used as the vPC peer link.
interface Po1
vpc peer-link
15.5 Configure VPC on the port-channels going to the storage devices.
interface Po11
vpc 11
interface Po12
vpc 12
15.6 Configure VPC on the port-channels going to the ESX Servers.
interface Po13
vpc 13
interface Po14
vpc 14
interface Po15
vpc 15
15.7 Configure VPC on the port-channels going to the Upstream Router.
interface Po20
vpc 20
copy run start

15.9 Create the vPC domain. The domain ID must match between VPC peers, but must differ from other
VPC pairs.
vpc domain 10
15.10 Configure the vPC role priority (optional): We will make N5K-1 the primary switch.
The switch with the lower priority will be elected as the vPC primary switch.
role priority 20
15.11 Configure the peer keepalive link. The management interface IP address for N5K-1 is 10.1.111.1 :

The system does not create the vPC peer link until you configure a vPC peer keepalive link.
15.12 Designate the port-channel to be used as the vPC peer link.
interface Po1
vpc peer-link
15.13 Configure VPC on the port-channels going to the storage devices.
interface Po11
vpc 11
interface Po12
vpc 12
15.14 Configure VPC on the port-channels going to the ESX Servers.
interface Po13
vpc 13
interface Po14
vpc 14
interface Po15
vpc 15
15.15 Configure VPC on the port-channels going to the Upstream Router.
interface Po20
vpc 20
copy run start

The following show commands are useful for verifying the vPC configuration.

Cisco Nexus 5010 A & B - N5K-1 & N5K-2
Step 16 Check the vPC role of each switch.
16.1 N5K-1 is the primary because we set the role priority number lower :
N5K-1(config)# show vpc role
vPC Role status
----------------------------------------------------
vPC role : primary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:0a
vPC system-priority : 32667
vPC local system-mac : 00:05:9b:7a:03:bc
vPC local role-priority : 10
16.2 N5K-2 is the secondary because we set the role priority number higher :
N5K-2(config)# show vpc role
vPC Role status
----------------------------------------------------
vPC role : secondary
Dual Active Detection Status : 0
vPC system-mac : 00:23:04:ee:be:0a
vPC system-priority : 32667
vPC local system-mac : 00:05:9b:79:b1:fc
vPC local role-priority : 20

Step 17 Verify VPC status on N5K-1 and N5K-2.
17.1 Make sure the domain id and role is correct. Make sure your peer status is ok or alive.
N5K-1(config-if)# sh vpc brief
Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 10
Peer status : peer adjacency formed ok
vPC keep-alive status : peer is alive
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 6
<snip>
vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 111,131,151,171,211

vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
<snip>
13 Po13 up success success 111,131,151
,171,211
,171,211
,171,211
,171,211

17.2 Make sure the domain id and role is correct. Make sure your peer status is ok or alive.
N5K-2(config-if)# show vpc bri
Legend:
(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id : 10
Peer status : peer adjacency formed ok
Configuration consistency status: success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : secondary
Number of vPCs configured : 6
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled

vPC Peer-link status
---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 Po1 up 111,131,151,171,211

vPC status
----------------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
------ ----------- ------ ----------- -------------------------- -----------
11 Po11 down* failed Consistency Check Not -
Performed
12 Po12 down* failed Consistency Check Not -
Performed
,171,211
,171,211
,171,211
,171,211

Cisco Nexus 5010 A & B - N5K-1 & N5K-2
17.3 View information on the peer-keepalive messages :
N5K-1(config)# show vpc peer-keepalive
<snip>

N5K-2(config-if)# show vpc peer-keepalive

--Peer is alive for : (2158) seconds, (636) msec
<snip>

vPC Keep-alive parameters
--Destination : 10.1.111.1
<snip>

17.4 View the running-configuration specific to vpc :
N5K-1(config)# show running-config vpc
feature vpc

vpc domain 10
role priority 10

vpc peer-link

vpc 11

vpc 12

vpc 13

vpc 14

vpc 15

vpc 20

N5K-2(config-if)# sh run vpc
feature vpc

vpc domain 10
role priority 20

vpc peer-link

vpc 11

vpc 12

vpc 13

vpc 14

vpc 15

vpc 20

3.7 CONFIGURING FEX ON N5K-1 AND N5K-2

In this task we activate the Fabric Extender feature and configure the Nexus 2000. We will be deploying a
pair of single-homed fabric extenders. Each ESX server will have a 1 Gbps link going to a different Nexus
2000 (FEX100 and FEX101).

On initial start up, the servers will treat these links as two active trunk links, but not as a port-channel. We do
this because the default load balancing is based on virtual port IDs and not on an IP hash.

Note: We do configure the 10 Gbps to be a port-channel and to load balance via IP hash.

Here are the steps for this section:
Enable feature for FEX. (Completed in an earlier section)
Pre-provision a Fabric Extender identifier (for example, "100").
Configure the fabric EtherChannel links for the Fabric Extender.
Configure each host interface port on the Fabric Extender on both Nexus 5000 Series switch.

Figure 5 - vPC to Dual Fabric Extenders

17.5 Configure the Nexus 2000 Fabric Extender and move the fabric interfaces of N5K-1 to the vPC.
Interfaces Eth1/7-8 connect to the Nexus 2000 uplink ports.
feature fex
N5K-1(config)# show feature | grep fex
fex 1 enabled
17.6 Pre-provision a Fabric Extender (FEX) identifier.
slot 100
provision model N2K-C2148T
17.7 Configure the fabric EtherChannel links for the Fabric Extender 100.
int po100
description single-homed FEX100
int e1/7-8
channel-group 100
int po100
switchport mode fex-fabric
fex associate 100

It may take several minutes for the Nexus 2000 to register with the Nexus 5000 switches. A syslog
notification will announce when the FEX is online.
17.8 Configure the Nexus 2000 (FEX) Ethernet Interfaces on N5K-1. The FEX interfaces will be used as
management ports for the ESXi servers. Ports Eth100/1/1-3 will be configured to trunk. We are not
going to going to put these ports into a channel group, so we commented out those lines. The port
channel configuration is also not necessary, but is included in case we need to port channel them
later.
int po113
description ESX1
vpc 113
int po114
description ESX2
vpc 114
int po115
description ESX3
vpc 115
int ethernet 100/1/1
description ESX1 vmnic2
! channel-group 113 force

The vPC number does not need to
match the PortChannel number, but
it must match the number of the vPC
peer switch for that vPC bundle.
17.9 Configure the Nexus 2000 Fabric Extender and move the fabric interfaces of N5K-2 to the vPC.
Interfaces Eth1/7-8 connect to the Nexus 2000 uplink ports.
feature fex
17.10 Pre-provision a Fabric Extender (FEX) identifier.
slot 101
provision model N2K-C2148T
17.11 Configure the fabric EtherChannel links for the Fabric Extender 101.
int po101
description single-homed FEX101
int e1/7-8
channel-group 101
int po101
fex associate 101
17.12 Configure the Nexus 2000 (FEX) Ethernet Interfaces on N5K-2. The FEX interfaces will be used as
management ports for the ESXi servers. Ports Eth100/1/1-3 will be configured to trunk. We are not
going to going to put these ports into a channel group, so we commented out those lines. The port
channel configuration is also not necessary, but is included in case we need to port channel them
later.
int po113
description ESX1
vpc 113
int po114
description ESX2
vpc 114
int po115
description ESX3
vpc 115

The vPC number does not need to
match the PortChannel number, but
it must match the number of the vPC
peer switch for that vPC bundle.
Step 18 Verify FEX configuration. Check FEX number and state.
N5K-1(config-if)# show fex detail
FEX: 100 Description: FEX0100 state: Online
FEX version: 5.0(3)N2(1) [Switch version: 5.0(3)N2(1)]
FEX Interim version: 5.0(3)N2(1)
Switch Interim version: 5.0(3)N2(1)
Extender Model: N2K-C2148T-1GE, Extender Serial: JAF1414CAKL
Part No: 73-12009-06
Card Id: 70, Mac Addr: 68:ef:bd:68:5b:c2, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/8
Fabric interface state:
Po100 - Interface Up. State: Active
Eth1/7 - Interface Up. State: Active
Fex Port State Fabric Port
Eth100/1/1 Up Po100
Eth100/1/2 Up Po100
Eth100/1/3 Up Po100
<snip>

N5K-2(config-if)# show fex detail
FEX: 101 Description: FEX0101 state: Online
FEX version: 5.0(3)N2(1) [Switch version: 5.0(3)N2(1)]
FEX Interim version: 5.0(3)N2(1)
Switch Interim version: 5.0(3)N2(1)
Extender Model: N2K-C2148T-1GE, Extender Serial: JAF1311APEC
Part No: 73-12009-05
Card Id: 70, Mac Addr: 00:0d:ec:cb:52:42, Num Macs: 64
Module Sw Gen: 12594 [Switch Sw Gen: 21]
post level: complete
pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/7
Fabric interface state:
Po101 - Interface Up. State: Active
Fex Port State Fabric Port
Eth101/1/1 Up Po101
Eth101/1/2 Up Po101
Eth101/1/3 Up Po101

3.8 PERFORM THE INITIAL SETUP OF MDS9124
In this task you will complete the initial configuration of the MDS 9124 switch for out-of-band management.

Step 19 Complete these steps on the MDS 9124:
19.1 Log in to the MDS 9214 using the console button on the lab interface. The prompt should be at
the System Admin Account Setup. Run through the setup script.
commands.
switch# wr erase
switch# reload
This command will reboot the system. (y/n)? [n] y
19.3 Connect to the MDS 9124 using the console button on the lab interface and perform the System
Admin Account Setup:
---- System Admin Account Setup ----
Do you want to enforce secure password standard (yes/no) [y]:y
<snip>
Would you like to enter the basic configuration dialog (yes/no): yes
Enter the switch name : MDS9124
Configure the default gateway? (yes/no) [y]: y
Configure advanced IP options? (yes/no) [n]: n
Type of ssh key you would like to generate (dsa/rsa) [rsa]: rsa
Number of rsa key bits <768-2048> [1024]: 1024
Configure clock? (yes/no) [n]: n
Configure timezone? (yes/no) [n]: n
Configure summertime? (yes/no) [n]: n
Configure default switchport interface state (shut/noshut) [shut]: shut
Configure default switchport trunk mode (on/off/auto) [on]: on
Configure default switchport port mode F (yes/no) [n]: n
Configure default zone policy (permit/deny) [deny]: deny
Enable full zoneset distribution? (yes/no) [n]: y
Configure default zone mode (basic/enhanced) [basic]: basic

The following configuration will be applied:
password strength-check
switchname MDS9124
interface mgmt0
ip address 10.1.111.40 255.255.255.0
no shutdown
ip default-gateway 10.1.111.254
ssh key rsa 1024 force
feature ssh
no feature telnet
feature http-server
system default switchport shutdown
system default switchport trunk mode on
no system default zone default-zone permit
system default zone distribute full
no system default zone mode enhanced

[########################################] 100%
Step 20 Verify basic connectivity.
20.1 Log in to the MDS
MDS login: admin
Password: 1234Qwer
20.2 Verify management connectivity by pinging the default gateway:
MDS# ping 10.1.111.254
PING 10.1.111.254 (10.1.111.254) 56(84) bytes of data.
Note: Press <CTL>-C to escape the continuous ping

EXPLORING THE MDS 9124 CLI
In this task you will briefly explore the MDS 9124 command line interface (CLI).

Cisco MDS 9124
Step 21 Type ? to view the current command options.
MDS9124# ?
21.1 Display all commands that begin with S, sh, and show. Press Enter or space to scroll through the
list of commands.
MDS9124# s?
MDS9124# sh?
MDS9124# show ?
21.2 Display the current running configuration.
MDS9124# show running-config
Abbreviate the syntax, then hit tab key to complete each word; for example, type sh<tab> ru<tab>.

21.3 Display the status of the switch interfaces. Notice that fibre channel interfaces fc 1/1 - fc 1/6 are
down.
MDS9124# sh int brief
-------------------------------------------------------------------------------
Interface Vsan Admin Admin Status SFP Oper Oper Port
Mode Trunk Mode Speed Channel
Mode (Gbps)
-------------------------------------------------------------------------------
fc1/1 1 auto on down swl -- --
fc1/7 1 auto on sfpAbsent -- -- --
<snip>
MDS9124# show vsan
vsan 1 information
name:VSAN0001 state:active
interoperability mode:default
loadbalancing:src-id/dst-id/oxid
operational state:down

vsan 4079:evfp_isolated_vsan

vsan 4094:isolated_vsan
21.5 Display the current installed version of code and environmental information.
MDS9124# show version
MDS9124# show environment

Step 22 Save your configuration locally and to a remote server.
Cisco MDS 9124
22.1 Update the startup configuration with the changes made in the running configuration.
MDS9124# copy running-config startup-config
[########################################] 100%
22.2 Save your running configuration to the tFTP server.
MDS9124# copy running-config tftp://10.1.111.100/MDS9124-Lab1-config
Connection to server Established. Copying Started.....
|
TFTP put operation was successful
Note: Be sure you start the tFTP/FTP Server before attempting to save the configuration or your copy will
fail. Please review Lab 0 Lab Services for instructions on how to use the tFTP/FTP server.
Use a tFTP/FTP Server in production networks to keep backup configurations and code releases for
each network device. Be sure to include these servers in your regular Data Center backup plans.

4 CISCO UNIFIED COMPUTING SYSTEM DEPLOYMENT PROCEDURE
The following section provides a detailed procedure for configuring the Cisco Unified Computing System for use
in a DCV environment. These steps should be followed precisely because a failure to do so could result in an
improper configuration.

4.1 POWER ON THE ESX HOSTS AND VERIFY THE NEXUS INTERFACES

We will use Cisco Unified Computing System C-Series Servers, powered by Intel Xeon processors, providing
industry-leading virtualization performance, to validate our configuration.

The ESX CNA interfaces must be up in order to verify interface connectivity and fabric login. Power up the ESX
hosts, then use show commands on the Nexus 5000 to verify the interfaces.

Step 23 Power up ESXi hosts.
23.1 Connect to the VC_SERVER from the SSL Dashboard.
23.2 Log into the server with credentials: administrator/1234Qwer.
23.3 Double click on the ESX1 CIMC shortcut on the desktop (or http://10.1.111.161/).
23.4 Accept any SSL warnings.
23.5 Authenticate with admin/1234Qwer.

23.6 Select Power on Server under Server Summary :

Step 24 Repeat Step 23 for ESX2 CIMC (http://10.1.111.162) and ESX3 CIMC (http://10.1.111.163).

3
1
2
5 CISCO NEXUS AND MDS DEPLOYMENT PROCEDURE: PART II
This section contains the procedural steps for the second part of the Cisco Nexus 5010 deployment.

5.1 CREATE FIBRE CHANNEL OVER ETHERNET (FCOE) INTERFACES

Cisco Nexus 5000 Series switches support Fibre Channel over Ethernet (FCoE), which allows Fibre Channel and
Ethernet traffic to be carried on the same physical Ethernet connection between the switch and the servers.
The Fibre Channel portion of FCoE is configured as a virtual Fibre Channel (vfc) interface. Logical Fibre Channel
features (such as interface mode) can be configured on virtual Fibre Channel interfaces.

In this task you are going to:
Create a vlan to carry the FCoE traffic and bind it the vlan.
Add the FCoE vlan to the allowed vlan list.
Define a virtual FC interface(vFC) and bind it to an interface.
Configure SAN port-channel uplinks.
Create a vsan database.
Assign the vFC interfaces to their vsan.
Enable FC and vFC interfaces.

Step 25 Create vlan 1011 to carry FCoE enabled vsan 11.
vlan 1011
fcoe vsan 11
25.1 Add vlan 1011 to the server port channels.
interface po13-15
switchport trunk allowed vlan add 1011
25.2 Create virtual Fibre Channel interfaces. Bind them to server port-channel interfaces. Then bring up
the vFC interfaces.

When FCoE hosts are using vPC, vfc interfaces need to bind to the port-channel interface instead of the
physical interface.

interface vfc13
bind interface po13
interface vfc14
bind interface po14
interface vfc15
bind interface po15
int vfc13-15
switchport trunk allowed vsan 11

2011 Jan 14 06:05:37 N5K-1 %$ VDC-1 %$ %PORT-2-IF_DOWN_ERROR_DISABLED: %$VSAN 1%$
Interface vfc3 is down (Error disabled)

You will get error disabled messages, if the servers have not been powered up, yet.

25.3 Define SAN port-channel for uplinks.
interface san-port-channel 111
channel mode active

interface fc2/3-4
channel-group 111

switchport trunk mode auto
25.4 Create vsan 11. On N5K-1, associate vsan 11 with vfc 13-15 and san-port-channel 111.
vsan database
vsan 11 name FABRIC_A
vsan 11 interface vfc 13-15
vsan 11 interface san-port-channel 111
exit
25.5 Enable the interfaces fc2/1-4:
interface fc2/1-4
no shut
int vfc13-15
no shut

FC ports are shut down by default.

Step 26 Perform Steps 1-5 on N5K-2 to configure vfc interfaces bound to port-channel 3-5 and to bind vsan 20
to vlan 120 :
vlan 1012
fcoe vsan 12
exit
26.1 Add vlan 1012 to the server port channels.
int po13-15
switchport trunk allowed vlan add 1012
26.2 Create virtual Fibre Channel interfaces. Bind them to server port-channel interfaces. Then bring up
the vFC interfaces.

When FCoE hosts are using vPC, vfc interfaces need to bind to the port-channel interface instead of the
physical interface.
int vfc13
bind interface port-channel 13
int vfc14
int vfc15
int vfc13-15
exit
channel mode active

interface fc2/3-4
channel-group 112

switchport trunk mode auto
26.4 Create vsan 12. On N5K-1, associate vsan 12 with vfc 13-15 and san-port-channel 111.
vsan database
vsan 12 name FABRIC_B
vsan 12 interface vfc13-15
vsan 12 interface san-port-channel 112
exit
Note: VLAN and VSAN needs to be different from N5K-1. This is so we can create two paths.
interface fc2/1-4
no shut
int vfc13-15
no shut
exit

FC ports are shut down by default.

Cisco MDS9124
Step 27 Create vsan 10 and vsan 20. Assign fc1/3,fc1/5 to vsan 10 and fc 1/4,fc 1/6 to vsan 20.:
Note: FC Port Connectivity: MDS fc1/1 to N5K-1 fc2/1, MDS fc1/2 to N5K-2 fc2/1, MDS fc1/3 to EMC SPA.
27.1 Put descriptions on each fc interface. (optional)
int fc1/1
switchport description Trunk N5K-1:fc2/3
int fc1/2
int fc1/3
int fc1/4
int fc1/5
switchport description Trunk NTAP:e2a
int fc1/6
switchport description Trunk NTAP:e2b
exit
interface port-channel 111
channel mode active
! switchport rate-mode dedicated

interface fc1/1-2
channel-group 111 force
no shutdown

channel mode active

interface fc1/3-4
channel-group 112 force
no shutdown
27.3 Create vsan 11 and vsan 12.
vsan database
vsan 11 name FABRIC_A
vsan 12 name FABRIC_B
27.4 Assign fc1/5 and port-channel 111 to vsan 11. Assign fc 1/6 and port-channel 112to vsan 12.:
vsan 11 interface fc1/5
vsan 11 interface port-channel 111
vsan 12 interface port-channel 112
exit
int fc1/1-6
no shutdown
Note: FC ports are shut down by default.

Step 28 Verify Fibre Channel configuration.
28.1 Verify membership for VSANs.
N5K-1(config)# sh vsan membership
vsan 1 interfaces:
fc2/1 fc2/2

vsan 11 interfaces:
fc2/3 fc2/4 san-port-channel 111 vfc13
vfc14 vfc15
<snip>

N5K-2(config)# show vsan membership
vsan 1 interfaces:
fc2/1 fc2/2

vsan 12 interfaces:
fc2/3 fc2/4 san-port-channel 112 vfc13
vfc14 vfc15

MDS9124(config-vsan-db)# show vsan membership
vsan 1 interfaces:
fc1/7 fc1/8 fc1/9 fc1/10
<snip>

vsan 11 interfaces:
fc1/1 fc1/2 fc1/5 port-channel 111

vsan 12 interfaces:
fc1/3 fc1/4 fc1/6 port-channel 112

28.2 Verify fcoe vlan.
N5K-1(config)# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
1011 11 Operational
Note: If the association state is non-operational, then you did not define vsan 10 in a previous step.

N5K-2(config)# show vlan fcoe
Original VLAN ID Translated VSAN ID Association State
---------------- ------------------ -----------------
1012 12 Operational

28.3 View all of the virtual Fibre Channel interfaces. Make sure all defined vFCs are present and in the
correct VSANs.
N5K-1(config)# sh int brief | include vfc
vfc13 11 F on trunking -- TF auto --
Note: All of the vfc interfaces will up as errDisabled if the servers are turned off.

N5K-2(config)# sh int bri | i vfc

28.4 Confirm the configuration of the virtual Fibre Channel interface. Note the bound Ethernet
interface information. The rest of the information is similar to a standard fibre channel port.
N5K-1(config)# sh int vfc13-15 | grep next 8 vfc
vfc13 is trunking
Bound interface is port-channel13
Hardware is Virtual Fibre Channel
Port WWN is 20:0c:00:05:9b:7a:03:bf
Admin port mode is F, trunk mode is on
Port mode is TF
Port vsan is 11
Trunk vsans (admin allowed and active) (11)
--
vfc14 is trunking
Port WWN is 20:0d:00:05:9b:7a:03:bf
Port mode is TF
Port vsan is 11
--
vfc15 is trunking
Port WWN is 20:0e:00:05:9b:7a:03:bf
snmp link state traps are enabled
Port mode is TF
Port vsan is 11
Note: The interfaces will show down if the connecting servers are powered off.

N5K-2(config-if)# sh int vfc13-15 | grep next 8 vfc
vfc13 is trunking
Port WWN is 20:0c:00:05:9b:79:b1:ff
Port mode is TF
Port vsan is 12
--
vfc14 is trunking
Port WWN is 20:0d:00:05:9b:79:b1:ff
Port mode is TF
Port vsan is 12
--
vfc15 is trunking
Port WWN is 20:0e:00:05:9b:79:b1:ff
Port mode is TF
Port vsan is 12


MDS9124(config-if)# sh int fc1/5-6 | grep next 8 fc1
fc1/5 is up
Port description is Trunk NTAP:e2a
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port WWN is 20:05:00:05:9b:7a:ec:c0
Admin port mode is auto, trunk mode is on
Port mode is F, FCID is 0xc80000
Port vsan is 11
Speed is 4 Gbps
--
fc1/6 is up
Port description is Trunk NTAP:e2b
Hardware is Fibre Channel, SFP is short wave laser w/o OFC (SN)
Port mode is F, FCID is 0x1c0000
Port vsan is 12
Speed is 4 Gbps

28.1 Verify SAN port-channel summary.
N5K-1(config-if)# sh san-port-channel sum
U-Up D-Down B-Hot-standby S-Suspended I-Individual link
summary header
--------------------------------------------------------------------------------
Channel
--------------------------------------------------------------------------------
111 San-po111 FC PCP (U) FC fc2/3(P) fc2/4(P)

N5K-2(config)# show san-port-channel sum
--------------------------------------------------------------------------------
Channel
--------------------------------------------------------------------------------
112 San-po112 FC PCP (U) FC fc2/3(P) fc2/4(P)

MDS9124(config-if)# show port-channel sum
------------------------------------------------------------------------------
Interface Total Ports Oper Ports First Oper Port
------------------------------------------------------------------------------
port-channel 111 2 2 fc1/2
port-channel 112 2 2 fc1/4

28.1 Verify that SAN port channel uplinks configuration and status.
N5K-1(config-if)# sh int san-port-channel 111
san-port-channel 111 is trunking
Hardware is Fibre Channel
Port WWN is 24:6f:00:05:9b:7a:03:80
Admin port mode is auto, trunk mode is auto
Port mode is TE
Port vsan is 11
Speed is 8 Gbps
Trunk vsans (up) (11)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()

N5K-2(config)# sh int san-port-channel 112
san-port-channel 112 is trunking
Port WWN is 24:70:00:05:9b:79:b1:c0
Admin port mode is auto, trunk mode is auto
Port mode is TE
Port vsan is 12
Speed is 8 Gbps
Trunk vsans (up) (12)
Trunk vsans (isolated) ()
Trunk vsans (initializing) ()

MDS9124(config-if)# sh int port-channel 111-112 | grep next 8 channel
port-channel 111 is trunking
Port WWN is 24:6f:00:05:9b:7a:ec:c0
Port mode is TE
Port vsan is 11
Speed is 8 Gbps
--
port-channel 112 is trunking
Port mode is TE
Port vsan is 12
Speed is 8 Gbps

5.2 DEVICE ALIASES, ZONES, AND ZONESETS
Zoning enables you to set up access control between storage devices or user groups. You create zones to
increase network security and to prevent data loss or corruption.
Note: Devices that do not belong to a zone follow the policy of the default zone.

Here are the general steps for creating zones and zone sets:
Create aliases
Create zones
Create zone sets
Activate the zone set.

Note: For the following steps, you will need the information from the table below. On occasion, hardware
needs to replaced or upgraded, and the documentation is not updated at the same time. One way to
verify this is to compare the output from a show flogi database versus the output from show run
zone. In other words, compare the values of the devices registering in versus the values you
manually zoned in.

Table 17 - WWPN Addresses
POD # DEVICE WWPN-A to N5K-1 WWPN-B to N5K-2
POD 1 NTAP1-A Boot Target 50:0a:09:81:88:bc:c3:04 50:0a:09:82:88:bc:c3:04
POD 1 ESX1 21:00:00:c0:dd:12:bc:6d 21:00:00:c0:dd:12:bc:6f
POD 1 ESX2 21:00:00:c0:dd:14:60:31 21:00:00:c0:dd:14:60:33
POD 1 ESX3 21:00:00:c0:dd:11:bc:e9 21:00:00:c0:dd:11:bc:eb
POD 2 NTAP1-A Boot Target 50:06:01:60:4b:a0:66:c7 50:06:01:61:4b:a0:66:c7
POD 2 ESX1 21:00:00:c0:dd:13:ec:19 21:00:00:c0:dd:13:ec:1b
POD 2 ESX2 21:00:00:c0:dd:14:71:8d 21:00:00:c0:dd:14:71:8f
POD 2 ESX3 21:00:00:c0:dd:14:73:c1 21:00:00:c0:dd:14:73:c3
POD 3 NTAP1-A Boot Target 50:06:01:60:4b:a0:6e:75 50:06:01:61:4b:a0:6e:75
POD 3 ESX1 21:00:00:c0:dd:13:eb:bd 21:00:00:c0:dd:13:eb:bf
POD 3 ESX2 21:00:00:c0:dd:13:ed:31 21:00:00:c0:dd:13:ed:33
POD 3 ESX3 21:00:00:c0:dd:14:73:19 21:00:00:c0:dd:14:73:1b
POD 4 NTAP1-A Boot Target 50:0a:09:81:88:ec:c2:a1 50:0a:09:82:88:ec:c2:a1
POD 4 ESX1 21:00:00:c0:dd:12:0e:59 21:00:00:c0:dd:12:0e:59
POD 4 ESX2 21:00:00:c0:dd:12:0d:51 21:00:00:c0:dd:12:0d:53
POD 4 ESX3 21:00:00:c0:dd:14:73:65 21:00:00:c0:dd:14:73:67


Step 29 Create device aliases on each Cisco Nexus 5010 and create zones for each ESXi host
29.1 Aliases for storage (targets).
device-alias database
device-alias name NTAP1-A_0a pwwn <ntap1_a_wwpn>
29.2 Aliases for hosts (initiators)
device-alias name ESX1_NTAP1-A_A pwwn <esx1_a_wwpn>
exit
device-alias commit
Note: Get this information from Error! Reference source not found. and Error! Reference source not
found..

29.3 Create the zones for each service profile. Each zone contains one initiator and one target. We
place port 1 of each CNA in a zone with NTAP1-A 0a for VSAN 11.
zone name ESX1_NTAP1-A_A vsan 11
member device-alias ESX1_NTAP1-A_A
member device-alias NTAP1-A_0a
exit
exit
exit

29.4 Create the zoneset and add the necessary members.
zoneset name FLEXPOD_A vsan 11
member ESX1_NTAP1-A_A
exit

29.5 Distribute and activate the zoneset.
zoneset distribute full vsan 11
zoneset activate name FLEXPOD_A vsan 11
29.6 Type copy run start.

Step 30 Create device aliases on each Cisco Nexus 5010 and create zones for each ESXi host
30.1 From the global configuration mode, type:
device-alias database
device-alias name NTAP1-A_0b pwwn <ntap1_b_wwpn>
device-alias name ESX1_NTAP1-A_B pwwn <esx1_b_wwpn>
exit
device-alias commit
Note: Get this information from Table 17.

30.2 Create the zones for each service profile. Each zone contains one initiator and one target. We
place port 2 of each CNA in a zone with NTAP1-A 0b for VSAN 12.
zone name ESX1_NTAP1-A_B vsan 12
member device-alias ESX1_NTAP1-A_B
member device-alias NTAP1-A_0b
exit
exit
exit

30.3 After all of the zones for the Cisco UCS service profiles have been created, create a zoneset to
organize and manage them.
30.4 Create the zoneset and add the necessary members.
zoneset name FLEXPOD_B vsan 12
member ESX1_NTAP1-A_B
exit

30.5 Distribute and activate the zoneset.
zoneset activate name FLEXPOD_B vsan 12
30.6 Type copy run start.

Cisco MDS9124
Note: When you activate the zone sets on N5K-1 and N5K-2, the switches will propagate the zone info to
the MDS.

30.7 Verify that the entries were successfully entered into the device alias database by entering show
device-alias. Examples below are for Pod1.
N5K-1# show device-alias database
device-alias name NTAP1-A_0a pwwn 50:0a:09:81:88:bc:c3:04
device-alias name NTAP1-A_0b pwwn 50:0a:09:82:88:bc:c3:04
device-alias name ESX1_NTAP1-A_A pwwn 21:00:00:c0:dd:12:bc:6d
device-alias name ESX1_NTAP1-A_B pwwn 21:00:00:c0:dd:12:bc:6f
device-alias name ESX2_NTAP1-A_A pwwn 21:00:00:c0:dd:14:60:31
device-alias name ESX2_NTAP1-A_B pwwn 21:00:00:c0:dd:14:60:33
device-alias name ESX3_NTAP1-A_A pwwn 21:00:00:c0:dd:11:bc:e9
device-alias name ESX3_NTAP1-A_B pwwn 21:00:00:c0:dd:11:bc:eb

Total number of entries = 8

N5K-2(config)# show device-alias database

MDS9124(config)# show device-alias database

30.8 Verify that the ESX hosts have completed a fabric login into N5K-1 and N5K-2. Make sure the VSAN
numbers are correct and that their alias shows up. Port numbers might not match yours.
N5K-1# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc13 11 0xdb0002 21:00:00:c0:dd:12:0e:59 20:00:00:c0:dd:12:0e:59
[ESX1_NTAP1-A_A]
vfc14 11 0xdb0001 21:00:00:c0:dd:12:0d:51 20:00:00:c0:dd:12:0d:51
[ESX2_NTAP1-A_A]
vfc15 11 0xdb0000 21:00:00:c0:dd:14:73:65 20:00:00:c0:dd:14:73:65
[ESX3_NTAP1-A_A]

Total number of flogi = 3.

N5K-2# show flogi database
--------------------------------------------------------------------------------
INTERFACE VSAN FCID PORT NAME NODE NAME
--------------------------------------------------------------------------------
vfc13 12 0xb80002 21:00:00:c0:dd:12:0e:5b 20:00:00:c0:dd:12:0e:5b
[ESX1_NTAP1-A_B]
vfc14 12 0xb80001 21:00:00:c0:dd:12:0d:53 20:00:00:c0:dd:12:0d:53
[ESX2_NTAP1-A_B]
vfc15 12 0xb80000 21:00:00:c0:dd:14:73:67 20:00:00:c0:dd:14:73:67
[ESX3_NTAP1-A_B]

30.9 Verify devices registered in the Fibre Channel Name server. The output fromhere shows you all
the hosts that have registered into the database. Note that you can you see an entry for the
NetApp array in here but not in the show flogi database above.
N5K-1# sh fcns database
VSAN 11:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0x140000 N 50:0a:09:81:88:ec:c2:a1 (NetApp) scsi-fcp:target
[NTAP1-A_0a]
0xdb0000 N 21:00:00:c0:dd:14:73:65 (Qlogic) scsi-fcp:init
[ESX3_NTAP1-A_A]
0xdb0001 N 21:00:00:c0:dd:12:0d:51 (Qlogic) scsi-fcp:init
[ESX2_NTAP1-A_A]
0xdb0002 N 21:00:00:c0:dd:12:0e:59 (Qlogic) scsi-fcp:init
[ESX1_NTAP1-A_A]

N5K-2# show fcns database
VSAN 12:
--------------------------------------------------------------------------
FCID TYPE PWWN (VENDOR) FC4-TYPE:FEATURE
--------------------------------------------------------------------------
0xb80000 N 21:00:00:c0:dd:14:73:67 (Qlogic) scsi-fcp:init
[ESX3_NTAP1-A_B]
0xb80001 N 21:00:00:c0:dd:12:0d:53 (Qlogic) scsi-fcp:init
[ESX2_NTAP1-A_B]
0xb80002 N 21:00:00:c0:dd:12:0e:5b (Qlogic) scsi-fcp:init
[ESX1_NTAP1-A_B]
0xd00000 N 50:0a:09:82:88:ec:c2:a1 (NetApp) scsi-fcp:target
[NTAP1-A_0b]


30.10 Verify that the zones are correct by issuing the command show zoneset active. The output
should show the zoneset and the zones that were added to the zoneset. Examples below are for
Pod1.
N5K-2# show zoneset active
* fcid 0xd40002 [pwwn 21:00:00:c0:dd:12:bc:6d] [ESX1_NTAP1-A_A]
* fcid 0xc80000 [pwwn 50:0a:09:81:88:bc:c3:04] [NTAP1-A_0a]

* fcid 0xd40000 [pwwn 21:00:00:c0:dd:14:60:31] [ESX2_NTAP1-A_A]

* fcid 0xd40001 [pwwn 21:00:00:c0:dd:11:bc:e9] [ESX3_NTAP1-A_A]

N5K-2(config)# show zoneset active
* fcid 0x620001 [pwwn 21:00:00:c0:dd:12:bc:6f] [ESX1_NTAP1-A_B]
* fcid 0x1c0000 [pwwn 50:0a:09:82:88:bc:c3:04] [NTAP1-A_0b]

* fcid 0x620002 [pwwn 21:00:00:c0:dd:14:60:33] [ESX2_NTAP1-A_B]

* fcid 0x620000 [pwwn 21:00:00:c0:dd:11:bc:eb] [ESX3_NTAP1-A_B]

MDS9124(config)# show zoneset active
* fcid 0xd40002 [pwwn 21:00:00:c0:dd:12:bc:6d] [ESX1_NTAP1-A_A]

* fcid 0xd40000 [pwwn 21:00:00:c0:dd:14:60:31] [ESX2_NTAP1-A_A]

* fcid 0xd40001 [pwwn 21:00:00:c0:dd:11:bc:e9] [ESX3_NTAP1-A_A]

* fcid 0x620001 [pwwn 21:00:00:c0:dd:12:bc:6f] [ESX1_NTAP1-A_B]

* fcid 0x620002 [pwwn 21:00:00:c0:dd:14:60:33] [ESX2_NTAP1-A_B]

* fcid 0x620000 [pwwn 21:00:00:c0:dd:11:bc:eb] [ESX3_NTAP1-A_B]

6 VMWARE ESXI DEPLOYMENT PROCEDURE

This section presents a detailed procedure for installing VMware ESXi within a Data Center Virtualization
environment. The deployment procedures that follow are customized to include the specific environment
variables that have been noted in previous sections.

6.1 ESXI INSTALLATION AND BASIC SETUP

This procedure highlights using the built-in KVM console and virtual media features within Cisco UCS
Manager to map remote installation media to each individual blade server.

For your convenience, Section 6.1 ESXi Installation and Basic Setup has been completed for you. This
means that we:
Installed ESXi 4.1 Cisco OEM image
Configured the ESXI root password as 1234Qwer
Configured both of the onboard 1Gbps NICs to be part of the default vSwitch0.
Configured IP address, Subnet mask, Gateway, and DNS information
Enabled CLI support

After, performing Step 31, you can skip to Section 6.2, where you configure the advance network
features and network storage of ESXi.

Step 31 Connect to the VC_SERVER from the SSL Dashboard.
31.1 Log into the server with these credentials: administrator/1234Qwer.
31.2 Double click on the ESX1 CIMC shortcut on the desktop (or http://10.1.111.161/).
31.3 Accept any SSL warnings.
31.4 Authenticate with admin/1234Qwer.
31.5 Check the Power State for the server. If it is Powered off, then click on the Power On Server
link. If it is on, then click on the Power Cycle Server link under Server Summary :

31.6 Under the Actions section, click the Launch KVM Console link. Click Run on any certificate
mismatch warning dialogs that may pop up. You will now have a java KVM Console to the server
31.7 Repeat Steps 31.1 - 31.6 for ESX2 CIMC (http://10.1.111.162), and ESX3 CIMC
(http://10.1.111.163).

Skip the remaining steps and go to Section 6.2.

1

Step 32 Setting up the ESXi install
This step has already been done for you. Skip to the next step.
On both ESXi Hosts ESX1 and ESX2
32.1 Under the VM tab in the KVM window, select Add image.
32.2 Click the Add Image button in the window that displays.
32.3 Browse to the ESXi installer iso image file.
Note: The file is at E:\Lab\Software\VMware-VMvisor-Installer-4.1.0-260247_Cisco.iso
32.4 Click Open to add the image to the list of virtual media.
32.5 Click the checkbox for Mapped next to the entry corresponding to the image you just added.
32.6 The hosts should detect the presence of the virtual media on reboot.

Step 33 Installing ESXi
Note: This step has already been done for you. Skip to the next step.
On both ESXi hosts ESX1 and ESX2
33.1 Reboot the server using the Power Cycle Server button at the top of the KVM window.
a. It doesnt matter whether you use a soft or hard reboot, because the blades do not have an
OS.
33.2 On reboot, the machine detects the presence of the ESXi install media.
33.3 Select ESXi Installer from the menu that displays.
33.4 After the installer is finished loading, press Enter to continue with the install.
33.5 Read through the EULA and press F11 to accept and continue with the install.
33.6 Select the NetApp LUN (2GB in size) that you set up previously as the install disk for ESXi and then
press Enter to continue.
33.7 The installer warns you that existing partitions will be removed on the volume. After you are sure
this is what you want, press F11 to install ESXi.
33.8 After the install is complete, be sure to unmap the ESXi install image by unchecking the Mapped
checkbox in the Virtual Media window.
a. This is so that the server reboots into ESXi and not the installer.
33.9 The Virtual Media window might warn you that it is preferable to eject the media from the guest.
Because we cannot do this (and the media is read-only), click Yes and unmap it anyway.
33.10 Press Enter to reboot the server.
33.11 Each of the hosts should now have a bootable ESXi environment installed from the virtual media.

Step 34 Setting up the ESXi hosts administration password
This step has already been done for you. Skip to the next step.
34.1 After the server is done rebooting, press F2 (the Customize System option).
34.2 Login with root as the login name and an empty password field.
34.3 Select the Configure Password menu option.
34.4 Enter 1234Qwer as the password you want to use for administering the ESXi host.
34.5 Enter the same password to confirm, and press Enter to set the password.


Step 35 Setting up the ESXi hosts management networking.
Duration: 3 minutes
ESXi host 1 - ESX1
35.1 From the System Customization menu, select the Configure Management Network option.
35.2 Select the IP Configuration menu option.
35.3 Select the Set static IP address and network configuration: option to manually setup the
management networking.
35.4 Enter 10.1.111.21 for the IP address for managing the ESXi host.
35.5 Enter 255.255.255.0 as the subnet mask for the ESXi host.
35.6 Enter 10.1.111.254 as the default gateway for the ESXi.
35.7 Press Enter to accept the changes to the management networking.
35.8 Press Esc to exit the Configure Management Network submenu.
35.9 Press y to confirm the changes made and return to the main menu.
ESXi host 2 - ESX2
35.11 Select the IP Configuration menu option.
35.12 Select the Set static IP address and network configuration: option to manually setup the
management networking.
35.13 Enter 10.1.111.22 for the IP address for managing the ESXi host.
35.14 Enter255.255.255.0 as the subnet mask for the ESXi host.
35.15 Enter 10.1.111.254 as the default gateway for the ESXi.
35.16 Press Enter to accept the changes to the management networking.

Step 36 Setting up the management VLAN

36.2 Select the VLAN (optional) menu item.
36.3 Input 111 for the VLAN ID of the management interface.
36.5 Press y to confirm the changes made and to return to the main menu.
36.6 Select Test Management Network to verify that the management network is set up correctly.
Note: DNS test will fail because we have not configured DNS, yet.
36.7 Press Esc to log out of the console interface.
36.8 To verify, in the right panel of the ESXi configuration window, when the VLAN (optional) item is
highlighted, the specified VLAN should be shown.


Step 37 Setting up DNS

ESXi host 1 - ESX1
37.2 Select the DNS Configuration menu option.
37.3 Because we manually specified the IP configuration for the ESXi host, we also must specify the DNS
information manually.
37.4 Enter 10.1.111.10 as the primary DNS servers IP address.
37.5 (Optional) Enter the secondary DNS servers IP address.
37.6 Enter ESX1.dcvlabs.lab as the hostname for the ESXi host.
37.7 Press Enter to accept the changes to the DNS configuration.
37.10 Select Test Management Network on the System Configuration screen.
37.11 On the Test Management Network screen, press the Enter key. You should see OK as the
result from pinging the default gateway, DNS server and test resolution of the ESXi server
hostname. If any of the tests fails, contact your instructor.
ESXi host 2 - ESX2
37.13 Select the DNS Configuration menu option.
37.14 Because we manually specified the IP configuration for the ESXi host, we also must specify the DNS
information manually.
37.15 Enter 10.1.111.10as the primary DNS servers IP address.
37.16 (Optional) Enter the Secondary DNS servers IP address.
37.17 Enter ESX2.dcvlabs.lab as the hostname for the ESXi host.
37.18 Press Enter to accept the changes to the DNS configuration.

37.21 You can verify this step and the two previous steps by selecting the Test Management Network
option from the System Customization menu. Here you can specify up to three addresses to
ping and one hostname to resolve by using the DNS server.


Step 38 Enable CLI Support for ESXi.

38.1 From the System Customization menu, select Troubleshooting Options.

38.2 Select the Enable Local Tech Support menu item.

38.3 Select the Enable Remote Tech Support menu item.

38.4 Press Esc twice to log out of the console interface.
38.5 Press ALT-F1 to access the command-line console interface.
38.6 Login with the root user ID and password.

6.2 ESXI NETWORKING

This task demonstrates how to configure ESXi networking on two 10Gig Converged Network Adapters
and two 1Gig adapters with Jumbo Frames.

Step 39 Create vSwitch1. Enable vSwitch0 and vSwitch1 for Jumbo Frames.
In this step, youll create port groups for Management Traffic (111), CTRL-PKT (171), and Local VLAN (24)
traffic on ESX1 and ESX2.

In VMware vSphere, port groups represent a logical representation of the available network resources.
Therefore, port groups usually map to different VLANs and/or different uplinks.

A summary of the commands in the following steps can be found at the end of Step 42.5. This allows you
to quickly set up your ESXi servers.

39.1 SSH into your ESXi host using root as the username and 1234Qwer as the password.
39.2 Confirm your current network settings.
~ # esxcfg-vswitch -l
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 128 4 128 1500 vmnic2,vmnic3

PortGroup Name VLAN ID Used Ports Uplinks
VM Network 0 0 vmnic2,vmnic3
Management Network 111 1 vmnic2,vmnic3

vmnic2 and vmnic3 are the 1Gbps nics connected to the Cisco Nexus 2248 Fabric Extenders. They are
both active and uses the default ESXi virtual port id load balancing mechanism.

39.3 Enable jumbo frames for default vSwitch0. Type esxcfg-vswitch -m 9000 vSwitch0.
esxcfg-vswitch -m 9000 vSwitch0
39.4 Add a new vSwitch for the 10Gbps CNA ports. Enable jumbo frames for vSwitch1.
esxcfg-vswitch -a vSwitch1
39.5 Add uplinks to vSwitch1.
esxcfg-vswitch -L vmnic0 vSwitch1


Step 40 Create necessary port groups on vSwitch1.
40.1 Add a new port group called MGMT Network to vSwitch0 and assign it to vlan 111.
esxcfg-vswitch -A "MGMT Network" vSwitch1
esxcfg-vswitch -v 111 -p "MGMT Network" vSwitch1

Why am I creating another Management network group? The default Management Network is a
vmkernel management interface. This new port group is for VMs to be on the Management VLAN.

40.2 Add a new port group called NFS to vSwitch1 and assign it to vlan 211.
esxcfg-vswitch -A NFS vSwitch1
esxcfg-vswitch -v 211 -p NFS vSwitch1
40.3 Add a new port group called VMotion to vSwitch1 and assign it to vlan 151.
esxcfg-vswitch -A VMotion vSwitch1
esxcfg-vswitch -v 151 -p VMotion vSwitch1
40.4 Add a new port group called CTRL-PKT to vSwitch0 and assign it to vlan 171.
esxcfg-vswitch -A "CTRL-PKT" vSwitch1
esxcfg-vswitch -v 171 -p "CTRL-PKT" vSwitch1
40.5 Add a new port group called VMTRAFFIC to vSwitch0 and assign it to vlan 131.
esxcfg-vswitch -A "VMTRAFFIC" vSwitch1
esxcfg-vswitch -v 131 -p "VMTRAFFIC" vSwitch1
40.6 Add a new port group called Local LAN to vSwitch0 and assign it to vlan 24.
esxcfg-vswitch -A "Local LAN" vSwitch1
esxcfg-vswitch -v 24 -p "Local LAN" vSwitch1
40.7 Refresh your network settings.
vim-cmd hostsvc/net/refresh

You need to run a refresh of your network settings for the following steps. This is important when
running these commands from a script.

40.8 Verify the MTU 9000 setting and the addition of Port Groups. Type esxcfg-vswitch -l.
~ # esxcfg-vswitch -l

VM Network 0 0 vmnic2,vmnic3
Management Network 111 1 vmnic2,vmnic3


Local LAN 24 0 vmnic0,vmnic1
CTRL-PKT 171 0 vmnic0,vmnic1
MGMT Network 111 0 vmnic0,vmnic1
VMotion 151 0 vmnic0,vmnic1
NFS 211 0 vmnic0,vmnic1

Step 41 Enable load balancing via IP Hash on vSwitch1.
41.1 Set vSwitch1 to load balance based on IP Hash. The Nexus 10Gbps ports have already been
configured for load balancing based on IP Hash.
vim-cmd /hostsvc/net/vswitch_setpolicy --nicteaming-policy='loadbalance_ip' vSwitch1
41.2 Verify your vSwitch load balancing policy. vSwitch0 should be set to lb_srcid and vSwitch1
should be set to lb_ip
~ # grep "vswitch" /etc/vmware/esx.conf | egrep '(teamPolicy\/team|vSwitch)'
/net/vswitch/child[0000]/name = "vSwitch0"
/net/vswitch/child[0000]/teamPolicy/team = "lb_srcid"
/net/vswitch/child[0001]/name = "vSwitch1"
/net/vswitch/child[0001]/teamPolicy/team = "lb_ip"

Step 42 Create vmkernel interfaces for vMotion and NFS storage.
On ESXi host ESX1
42.1 Create vmkernel interface for NFS traffic. Enable it for Jumbo Frames on port group NFS.
esxcfg-vmknic -a -i 10.1.211.21 -n 255.255.255.0 -m 9000 -p NFS
42.2 Create vmkernel interface for VMotion traffic. Enable it for Jumbo Frames on port group VMotion.
esxcfg-vmknic -a -i 10.1.151.21 -n 255.255.255.0 -m 9000 -p VMotion
On ESXi host ESX2
42.3 Create vmkernel interface for NFS traffic. Enable it for Jumbo Frames on port group NFS.
42.4 Create vmkernel interface for VMotion traffic. Enable it for Jumbo Frames on port group VMotion.

Make sure your network is enabled for
Jumbo Frames end-to-end.
42.5 Type esxcfg-vmknic -l and verify that the vmkernel ports were added properly with an MTU of
9000.
On ESXi host ESX1
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask
Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 10.1.111.21 255.255.255.0
10.1.111.255 c4:7d:4f:7c:a7:6a 1500 65535 true STATIC
vmk1 NFS IPv4 10.1.211.21 255.255.255.0
10.1.211.255 00:50:56:7e:60:53 9000 65535 true STATIC
vmk2 VMotion IPv4 10.1.151.21 255.255.255.0
10.1.151.255 00:50:56:7b:ae:78 9000 65535 true STATIC
On ESXi host ESX2
~ # esxcfg-vmknic -l
Interface Port Group/DVPort IP Family IP Address Netmask
Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 10.1.111.22 255.255.255.0
10.1.111.255 68:ef:bd:f6:38:82 1500 65535 true STATIC
vmk1 NFS IPv4 10.1.211.22 255.255.255.0
10.1.211.255 00:50:56:76:bc:47 9000 65535 true STATIC
vmk2 VMotion IPv4 10.1.151.21 255.255.255.0
10.1.151.255 00:50:56:74:b2:7f 9000 65535 true STATIC

Summary of Commands


On ESXi host ESX1
On ESXi host ESX2

Step 43 Logging into VMware ESXi host using VMware vSphere client
Duration: 5 minutes

ESXi host 1 - ESX1
43.1 Open the vSphere client and enter 10.1.111.21 as the host you are trying to connect to.
43.2 Enter root for the username.
43.3 Enter 1234Qwer as the password.
43.4 Click the Login button to connect.

ESXi Host 2 - ESX2
43.5 Open the vSphere client and enter 10.1.111.22 as the host you are trying to connect to.
43.6 Enter root for the username.
43.7 Enter 1234Qwer as the password.
43.8 Click the Login button to connect.
43.9 To verify that the login was successful, the vSphere clients main window should be visible.

Step 44 Setting up the VMotion vKernel port on the virtual switch for individual hosts
Duration: 5 minutes per host

Now we need to enable VMotion on the vmKernel port we created.

ESXi host 1 - ESX1
44.1 Select ESX1 on the left panel.
44.2 Go to the Configuration tab.
44.3 Click the Networking link in the Hardware box.
44.4 Click the Properties link in the right field on vSwitch1.

44.5 Select the VMotion configuration and click the Edit button.

3
1
2
1
44.6 Check the vMotion: Enabled checkbox.

44.7 Click OK to continue.
44.8 Click Close to close the dialog box.

ESXi host 2 - ESX2
44.9 Select ESX2 on the left panel.
44.10 Go to the Configuration tab.
44.11 Click the Networking link in the Hardware box.
44.12 Click the Properties link in the right field on vSwitch1.
44.13 Select the VMotion configuration and click the Edit button.
44.14 Check the vMotion: Enabled checkbox.
44.15 Click OK to continue.
44.16 Click Close to close the dialog box.

44.17 On the right panel, click the Virtual Switch View. Individual VMkernel ports will be displayed for
the various networks defined. Select a VMkernel port and display the VM associated with that
port.

1
Step 45 Change VLAN ID for default VM-traffic port-group called VM Network
Duration: 5 minutes

For each ESXi Host ESX1 and ESX2
45.1 Select the host on the left panel.
45.2 Select the Configuration tab.
45.3 Select the Networking link in the Hardware box.
45.4 Click Properties in the right field for vSwitch0.

45.5 Highlight the VM Network port-group in the listing in the left box.

45.6 Click Edit.
45.7 Type in the VLAN ID for your Pods VM Traffic VLAN (ex 131.)

45.8 Click OK.
45.9 Click OK.

1

1

1
6.3 ESXI DATASTORES

Step 46 Mount the required datastores for individual hosts
Duration: 5 minutes per host.
For each ESXi host ESX1 and ESX2
46.1 Open a ssh session to your ESXi host.
46.2 Add two nas shares from host 10.1.211.151 with label DS and SWAP. -a specifies that we will add
a nas share. -s specifies the nas volume. --host specifies the host.
esxcfg-nas -a --host 10.1.211.151 -s /vol/VDI_VFILER1_DS DS
esxcfg-nas -a --host 10.1.211.151 -s /vol/VDI_SWAP SWAP
46.3 Create a test file.
~ # touch /vmfs/volumes/SWAP/test
46.4 View the contents of the mount to confirm files.
~ # ls /vmfs/volumes/SWAP/
test
46.5 From the vSphere client, view contents of the mount to confirm files. Select your host from the left
panel.
46.6 Select the Configuration tab. Select Storage in the Hardware box.
46.7 Inspect the right panel where the cluster is displayed. You should see all of the datastores
associated with the host.

46.8 Right click on SWAP and select Browse Datastore

46.9 You should see your test file.

Summary of Commands


3

1

2

1

2

1
Step 47 Time configuration for individual hosts - (SKIP for LAB)

47.3 Click the Time Configuration link in the Software box.
47.4 Click the Properties link on the right panel.
47.5 A Time Configuration window displays. Click Options at the bottom.
47.6 An NTP Daemon Options window displays. Select NTP Settings in the left box, then click Add
47.7 Another pop-up window displays. Enter " 192.43.244.18" for the IP address of the NTP server, and
click OK to continue.
47.8 On the original NTP Daemon Options window, check the Restart NTP Service checkbox.
47.9 Click OK at the bottom of the window to continue and close the window.
47.10 On the Time Configuration window, verify that the clock is now set to the correct time. If the time
is correct, click OK to save the configuration and exit.
To verify, the right panel displays the correct time, the NTP client status, and the NTP server IP address.

Step 48 Moving the swap file
Duration: 5 minutes per host.
For ESXi host ESX1, ESX2 and ESX3
48.3 In the Software box, select Virtual Machine Swapfile Location.
48.4 On the right panel, click Edit

48.5 Select the radio button for Store the swapfile in a swap file datastore selected below if it is not
already selected.

3

1

2
48.6 Select SWAP as the datastore you want to store the swapfile on.

48.7 Click OK at the bottom of the page to finish.
48.8 The swapfile location is specified on the right panel.

You are now done with the initial setup of a Base Data Center Virtualization
infrastructure.

The remaining tasks will allow you to configure vCenter, Nexus 1000v, and OTV.

1

1
7 VMWARE VCENTER SERVER DEPLOYMENT PROCEDURE

Step 49 Setting up vCenter datacenter
Duration: 5 minutes
49.1 On the VC_SERVER desktop, double-click the Vmware vSphere Client icon. Make sure that the
settings are for localhost and Using the Windows session credentials (as below) and click Login :

49.2 In the Getting Started tab, click Create a Datacenter.

49.3 Enter FlexPod_DC_1 as the name of the new datacenter.
49.4 On the left panel, the datacenter displays underneath the vCenter name.

2

1

3
1

Step 50 Setting up the management cluster
Duration: 5 minutes per cluster
50.1 Right-click the datacenter and select New Cluster.
50.2 Enter FlexPod_Mgmt as the name for the cluster.
50.3 Check the box for VMware HA. Do not check the box for VMware DRS. Click Next to
continue.
Note: The FlexPod Implementation Guide, recommends you enable and accept the defaults for VMware
DRS.
50.4 Accept the defaults for power management, and click Next to continue.
50.5 Accept the defaults for VMware HA, and click Next to continue.
50.6 Accept the defaults for Virtual Machine Options, and click Next to continue.
50.7 Accept the defaults for VM Monitoring, and click Next to continue.
50.8 Accept the defaults for VMware EVC, and click Next to continue.
50.9 Select Store the Swapfile in the datastore specified by the host in the VM Swapfile Location
section and click Next to continue.

50.10 Review the selections made and click Finish to continue.
50.11 On the left panel, the cluster displays under the datacenter name.

1
7.1 ADDING HOSTS TO VMWARE VCENTER SERVER

Step 51 Adding hosts to a cluster
ESXi host 1 - ESX1
51.1 Right-click the cluster and select Add Host.
51.2 Enter ESX1 for the hostname of the host to be added to vCenter.
51.3 Enter root for the username and 1234Qwer as the credentials for accessing the ESXi host.
51.4 If a security alert is generated that says Unable to verify the authenticity of the specified host,
click Yes to acknowledge that this is fine and continue.
51.5 Review the information on the Host Summary page, and click Next at the bottom to continue
51.6 On the Assign License page, assign vSphere 4 Enterprise Plus license. Click Next at the
bottom to continue.
51.7 On the Lockdown Mode page, disable lockdown if it is not already disabled. Click Next at the
bottom to continue.
51.8 On the Choose Resources Pool page, select Put all of the hosts virtual machines in the
clusters root resource pool. Click Next at the bottom to continue.
51.9 Click Finish to add the host to the cluster.
ESXi host 2 - ESX2
51.10 Right-click the cluster and select Add Host.
51.11 Enter ESX2 for the hostname of the host to be added to vCenter.
51.12 Enter root for the username and 1234Qwer as the credentials for accessing the ESXi host.
51.13 If a security alert is generated that says Unable to verify the authenticity of the specified host,
click Next to acknowledge that this is fine and continue.
51.14 Review the information on the Host Summary page, and click Next at the bottom to continue.
51.15 On the Assign License page, assign vSphere 4 Enterprise Plus license. Click Next at the
bottom to continue.
51.16 On the Lockdown Mode page, disable lockdown if it is not already disabled. Click Next at the
bottom to continue.
51.17 On the Choose Resources Pool page, select Put all of the hosts virtual machines in the
clusters root resource pool. Click Next at the bottom to continue.
51.18 Click Finish to add the host to the cluster.

ESXi host 3 - ESX3
51.19 Repeat previous steps to add ESX3.

51.20 To verify, on the left panel, individual hosts display under the cluster.

7.2 CONFIGURE FIBRE CHANNEL STORAGE ON ESX HOSTS

This task has already been completed for you. You may review for completeness. Please skip ahead to
Section 7.3.

ESX1 vmnic0 is the CNA connected to N5K-1 Eth1/9. ESX2 vmnic0 is the CNA connected to N5K-1 Eth1/4. Add a
datastore to each ESX host presented via FCoE through the fabric.

Step 52 Click on the 10.1.111.21 (ESX1) host under ClusterA cluster. Select the Configuration tab. Click on the
Storage link under Hardware. Click on the Add Storage link:
52.1 Select the Disk/LUN radio button, then click Next :

52.2 Select the 50 GB Fibre Channel disk that is found and click Next.

Note: This LUN is connected via FcoE. ESX1 vmnic0 is the CNA port that is connected to N5K-1 Eth1/9.
52.3 Then, click Next on the Current Disk Layout dialog box that follows.
52.4 Name the datastore NetApp-SAN-1, then click Next

1

1


52.5 Uncheck the Maximize capacity box, and then enter 40.00 GB in the size box. Click Next.

Note: We will not use the full capacity of the LUN

52.6 Click Finish to add the datastore :
52.7 Note that the datastore appears on both ESX1 and ESX2 Storage. This is because the NetApp Array
has this LUN masked for both ESX1 and ESX2 initiators. You might need to click Refresh.

Note: Vmotion requires that VMs reside on shared storage

1

1

7.3 ADD A VM FROM NFS ATTACHED STORAGE

In this section, we are going to add two VMs to vCenter. They will be used as a Server and Client to test
connectivity in later tasks.

Step 53 Add Server VM to ESX1 inventory.
53.1 Select host ESX1 > Configuration > and then Storage under Hardware.
53.2 Right-click on the DS datastore and select Browse Datastore from the pop-up menu.

53.3 Click on the Server-2003R2 to open the folder. Right-click on the Server-2003R2.vmx file and
select Add to Inventory from the pop-up menu.

53.4 Leave the Name as Server-2003R2. Select FlexPod_DC_1. Click Next..
53.5 Specify your cluster and click Next.
53.6 Select ESX1 for the host. Click Next, then click Finish on the Add to Inventory dialog box.
Step 54 Add Client VM to ESX2 inventory.
54.1 Click on the ClientXP to open the folder. Right-click on the ClientXP.vmx file and select Add to
Inventory from the pop-up menu.

54.2 Leave the Name as ClientXP. Select FlexPod_DC_1. Click Next..
54.3 Specify your cluster and click Next.
54.4 Select ESX2 for the host. Click Next, then click Finish on the Add to Inventory dialog box.
54.5 Close the Datastore Browser.

3
1
2

4

2
1
3

2 1
3

8 CISCO NEXUS 1000V DEPLOYMENT PROCEDURE

8.1 INSTALL VIRTUAL SUPERVISOR MODULE (VSM) AS A VM ON ESXI

Step 55 INSTALL VIRTUAL SUPERVISOR MODULE (VSM) as a VM on ESXi
55.1 From the vSphere client, click on host ESX3.
55.2 Then click on File > Deploy OVF Template from the File from the menu bar:

55.3 Specify the following ftp location for the source URL.
ftp://10.1.111.100/Nexus1000v.4.2.1.SV1.4/VSM/Install/nexus-1000v.4.2.1.SV1.4.ova
55.4 Verify the VSM OVF template details such as version number. Click Next.
55.5 Accept the End User License Agreement. Click Next:
55.6 Name it vsm-1. Click Next:
55.7 Select Nexus 1000V Installer for Deployment Configuration. Click Next :
55.8 Select Netapp-SAN-1 for the Datastore.

55.9 Select your Cluster and click Next.
55.10 Select Thick provisioned format storage for Disk Format. Click Next.

1
1
55.11 Map the the Nexus 1000V Control and Packet source networks to CTRL_PKT. Map the
Management source network to "MGMT Network". Click Next.

Note: Cisco supports using the same vlan for Management, Control, and Packet port-groups. We are using
one group for Management traffic and another group for control and packet traffic.
55.12 Fill out the VSM Configuration Properties with information below, and then click Next.
VSM Domain ID: 11
Password: 1234Qwer
Management IP Address: 10.1.111.17
Management IP Subnet Mask: 255.255.255.0
Management IP Gateway: 10.1.111.254
55.13 Click Finish.
55.14 After the template is finished deploying, click Close :
55.15 Power on the VSM by clicking on the Nexus1000v VM and pressing the Power On icon ( ).
55.16 Then, launch the VM Console and verify that the VM boots to the login prompt :

1
8.2 REGISTERING THE CISCO NEXUS 1000V AS A VCENTER PLUG-IN
Step 56 Registering the Cisco Nexus 1000V as a vCenter Plug-in
56.1 Open a Web browser and navigate to the IP address for the Cisco Nexus 1000V. http://10.1.111.17
56.2 Right-click on the link cisco_nexus_1000v_extension.xml.

56.3 Save the XML document to your desktop.
56.4 Select Plug-Ins Manage Plug-ins in the vSphere Client window.

56.5 In the new window, right-click in open area below "Available Plug-ins" and select New Plug-in
(you may have to expand the window to do so).

56.6 Click Browse and navigate to where you saved cisco_nexus_1000v_extension.xml.
56.7 Click Open to open the XML file.
56.8 Click Register Plug-in.
56.9 If you get a security warning, click Ignore.
56.10 Click OK to confirm that the plug-in installed correctly.

1

1
2
1
1
8.3 CONFIGURING NETWORKING ON THE CISCO NEXUS 1000V
Step 57 Configuring networking on the Cisco Nexus 1000V
57.1 Use the putty client to ssh into the VSM 10.1.111.17. Enter admin as the username and
1234Qwer as the password.
57.2 Enter the global configuration mode by typing config t.
57.3 Configure hostname to be vsm-1.
hostname vsm-1
57.4 Configure the system mtu to be 9000. This is on by default.
system jumbomtu 9000
57.5 Configure the Nexus 1000v domain.
svs-domain
domain id 11
control vlan 171
packet vlan 171
svs mode L2
57.6 Configure the Nexus 1000v vCenter Server connections
svs connection vcenter
protocol vmware-vim
remote ip address 10.1.111.100 port 80
vmware dvs datacenter-name FlexPod_DC_1
connect
exit

Step 58 Verify connection to the vCenter and status before adding hosts to the VSM. The command show svs
connections shows VSM connection information to the vCenter. Make sure operational status is
Connected and Sync status is Complete. If the status is good,then proceed to adding hosts.
vsm-1# show svs connections
connection vcenter:
ip address: 10.1.111.100
remote port: 80
protocol: vmware-vim https
certificate: default
datacenter name: FlexPod_DC_1
DVS uuid: 84 52 1a 50 0c aa 52 b2-10 64 47 c3 8d af 46 70
config status: Enabled
operational status: Connected
sync status: Complete
version: VMware vCenter Server 4.1.0 build-345043
58.1 The Cisco Nexus 1000V switch should now be available in the Inventory Networking view.

1
2
8.4 NEXUS 1000V CREATE VLANS
Step 59 Create essential VLANs.
vlan 111
name MGMT-VLAN
vlan 131
name VMTRAFFIC
vlan 151
name VMOTION
vlan 171
name CTRL-PKT
vlan 211
name NFS-VLAN
59.1 Verify that the vlans were created successfully.
vsm-1(config-vlan)# show vlan brief
---- -------------------------------- --------- -------------------------------
1 default active
111 MGMT-VLAN active
131 VMTRAFFIC active
151 VMOTION active
171 CTRL-PKT active
211 NFS-VLAN active

Step 60 Enable lacp and lacp offload.
In our lab, we wont be using LACP to negotiate our port-channel, but we will enable the feature in case we do
later on. LACP offload is a feature that allows the VEM to negotiate the LACP port-channel instead of the VSM.
This is useful in case the VSM becomes unavailable.
60.1 To support LACP port-channels you need to first enable the LACP feature.
feature lacp
60.2 Now we need to enable LACP offload. This WILL require a reboot of the VSM.
lacp offload
copy running startup
reload
60.3 Verify that LACP offload is enabled.
vsm-1# show lacp offload status
Current Status : Enabled
Running Config Status : Enabled
Saved Config Status : Enabled

Summary of Commands
hostname vsm-1
system jumbomtu 9000
svs-domain
domain id 11
control vlan 171
packet vlan 171
svs mode L2
exit
protocol vmware-vim
vmware dvs datacenter-name FlexPod_DC_1
connect
exit
vlan 111
name MGMT-VLAN
vlan 131
name VMTRAFFIC
vlan 151
name VMOTION
vlan 171
name CTRL-PKT
vlan 211
name NFS-VLAN
feature lacp
lacp offload
copy running startup
reload

8.5 NEXUS 1000V CREATE PORT PROFILES

Step 61 Create an uplink port profile for the Virtual Machine Client Network, VMotion and the Nexus 1000V
Control and Packet traffic. Specify VLANs 11X, 13X, 15X, and 17X.
Note: We have a pair of NICs that will be teamed, so we will only need one uplink port profile.
61.1 Type the following commands in the VSM console or terminal session to create the SYSTEM-
UPLINK profile.
port-profile type ethernet SYSTEM-UPLINK
description System profile for blade uplink ports
vmware port-group
mtu 9000
61.2 We are going to turn on port-channel for our uplink.
channel-group auto mode on
Note: For channel-groups, my rule of thumb is:
UCS-B Series use channel-group auto mode on mac-pinning
UCS-C Series to Switch(es) and no port-channel use channel-group auto mode on mac-pinning
UCS-C Series to Switch(es) and port-channel on use channel-group auto mode
UCS-C Series to Switch(es) and port-channel LACP use channel-group auto mode active
61.3 Enable all ports in the profile.
no shutdown
61.4 VLAN 111, 151 , 171, and 211 are used for Management, VMotion, N1K management, and data
store traffic, so they have to be configured as system VLANs to ensure that these VLANs are
available during the boot process.
system vlan 111,211,151,171
61.5 Enable the port profile.
state enabled

Step 62 Create a Management port-profile for your ESXi management VMKernel interface. This port profile
will also be used by the Management interface of the VSM. As VLAN 111 is used for management
traffic, it has to be configured as a system VLAN to ensure that this VLAN is available during the boot
process of the ESXi server.
port-profile type vethernet MGMT-VLAN
vmware port-group
switchport mode access
switchport access vlan 111
no shutdown
system vlan 111
state enabled
Note: If you dont specify a port-profile type it defaults to vethernet.

Step 63 Create a Nexus 1000V Control and Packet port profile for the VSM virtual interfaces.
63.1 As VLAN 171 is used for management traffic it has to be configured as a system VLAN to ensure
that this VLAN is available during the boot process of the ESXi server.
Note: The following section is not used currently, because we are using VLAN 1 for Control, Packet, and
Management.
port-profile type vethernet N1KV-CTRL-PKT
vmware port-group
no shutdown
system vlan 171
state enabled

Step 64 Create a NFS Storage port-profile for NFS VMKernel interface.
64.1 VLAN 211 is used for storage traffic, so it has to be configured as a system VLAN to ensure that this
VLAN is available during the boot process of the ESXi server.
port-profile type vethernet NFS-VLAN
vmware port-group
no shutdown
system vlan 211
state enabled

Step 65 Create a vMotion port-profile for vmotion vmkernel interface.
65.1 Configure the port profile for the Virtual Machine network to which the VSM connects for Control
and Packet traffic. As VLAN 151 is used for management traffic it has to be configured as a system
VLAN to ensure that this VLAN is available during the boot process of the ESXi server.
port-profile type vethernet VMOTION
vmware port-group
no shutdown
system vlan 151
state enabled

Step 66 Create VM Traffic port-profile for VM virtual interfaces. This will be for the non-mangement Virtual
Machines residing on the ESXi hosts.
port-profile type vethernet VMTRAFFIC-VLAN
vmware port-group
no shutdown
! system vlan 131
state enabled
exit
66.1 Save your configuration.
copy run start

66.2 Verification not needed.

Summary of Commands

port-profile type ethernet SYSTEM-UPLINK
description system profile for blade uplink ports
vmware port-group
mtu 9000
no shutdown
system vlan 111,151,171,211
state enabled

port-profile type vethernet MGMT-VLAN
vmware port-group
no shutdown
system vlan 111
state enabled

port-profile type vethernet NFS-VLAN
vmware port-group
no shutdown
system vlan 211
state enabled
exit

vmware port-group
no shutdown
system vlan 151
state enabled
exit

port-profile type vethernet VMTRAFFIC-VLAN
vmware port-group
no shutdown
! system vlan 131
state enabled

port-profile type vethernet N1KV-CTRL-PKT
vmware port-group
no shutdown
system vlan 171
state enabled

8.6 INSTALL VIRTUAL ETHERNET MODULES (VEMS) ON ESXI HOSTS

The Virtual Supervisor Module (VSM) is the control plane of the software switch and is a virtual machine that
runs NX-OS. The Virtual Ethernet Module (VEM) is a virtual line card embedded in each ESX host. The VEM is
software in the kernel of the hypervisor (ESX) and utilizes an agent to communicate with the VSM. The Nexus
1000V distributed virtual switch policies are configured on the VSM and each ESX host VEM is updated via this
agent.

Step 67 Installing the Nexus 1000V VEMs on each ESXi host.
67.1 The Nexus 1000V VEM .vib file has been copied to the VDI_VFILER1_DS datastore for you.
67.2 Log into your ESXi server via SSH. Use the putty client.
67.3 Login with the root user ID and password of 1234Qwer.
67.4 Type cd /vmfs/volumes/DS.
67.5 Type ls and use the cd command to navigate to the directory where the VEM .vib file is stored.
67.6 Execute the VM binary file. esxupdate -b ./<vemname.vib> update.
/vmfs/volumes/e413d232-639669f1 # esxupdate -b ./cross_cisco-vem-v130-4.2.1.1.4.0.0-2.0.1.vib update
Unpacking cross_cisco-vem-v13.. ######################################## [100%]

Installing packages :cross_ci.. ######################################## [100%]

Running [/usr/sbin/vmkmod-install.sh]...
ok.

67.7 Type vem status and confirm that the VEM has been installed properly.
/vmfs/volumes/e413d232-639669f1 # vem status

VEM modules are loaded


VEM Agent (vemdpa) is running

Note: You do not need to install on ESX3.

Summary of Commands
cd /vmfs/volumes/DS
esxupdate -b cross_cisco-vem-v130-4.2.1.1.4.0.0-2.0.1.vib update

8.7 MIGRATE ESXI HOSTS TO NEXUS 1000V
Step 68 Replacing the default virtual switch with the Cisco Nexus 1000V
All ESXi hosts - ESX1 and ESX2
68.1 Select Inventory Networking in the vSphere client.

68.2 Select vsm-1 from the tree on the left. Right-click on it and select Add Host from the menu.

68.3 Select hosts ESX1 and ESX2. Next, select the adapters for each hosts vSwitch1 (vmnic0 and
vmnic1). Dont select vmnic that are used by vSwitch0 (the default virtual switch provided by the
ESXi server).
68.4 Select SYSTEM-UPLINK as the DVUplink port group for all of the vmnics you are adding.

68.5 Click Next to continue.
68.6 For Network Connectivity, do NOT migrate any adapters. Click Next to continue.
68.7 For Virtual Machine Networking, do NOT migrate any virtual machines now. Click Next to
continue.
68.8 Click Finish to apply the changes..

1
2
1

2

3
1
2

1
2

3
Placeholder
Step 69 Verify that the Virtual Ethernet Module(s) are seen by VSM.
vsm-1(config)# show module
--- ----- -------------------------------- ------------------ ------------
1 0 Virtual Supervisor Module Nexus1000V active *
3 248 Virtual Ethernet Module NA ok
4 248 Virtual Ethernet Module NA ok

Mod Sw Hw
--- ---------------- ------------------------------------------------
1 4.2(1)SV1(4) 0.0
3 4.2(1)SV1(4) VMware ESXi 4.1.0 Releasebuild-260247 (2.0)
4 4.2(1)SV1(4) VMware ESXi 4.1.0 Releasebuild-260247 (2.0)

--- -------------------------------------- ----------
1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
3 02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA
4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA

Mod Server-IP Server-UUID Server-Name
--- --------------- ------------------------------------ --------------------
1 10.1.111.17 NA NA
3 10.1.111.21 6da2f331-dfd4-11de-b82d-c47d4f7ca766 esx1
4 10.1.111.22 67ae4b62-debb-11de-b88b-c47d4f7ca604 esx2
* this terminal session
69.1 Verify that uplink is trunking the relevant vlans.
vsm-1(config)# sh int trunk
--------------------------------------------------------------------------------
Port Native Status Port
Vlan Channel
--------------------------------------------------------------------------------
Eth3/1 1 trnk-bndl Po1
Po1 1 trunking --
Po2 1 trunking --

--------------------------------------------------------------------------------
Port Vlans Allowed on Trunk
--------------------------------------------------------------------------------
Eth3/1 111,131,151,171,211
Eth3/2 111,131,151,171,211
Eth4/5 111,131,151,171,211
Eth4/6 111,131,151,171,211
Po1 111,131,151,171,211
Po2 111,131,151,171,211
<snip>

--------------------------------------------------------------------------------
Port STP Forwarding
--------------------------------------------------------------------------------
Eth3/1 none
Eth3/2 none
Eth4/5 none
Eth4/6 none
Po1 111,131,151,171,211
Po2 111,131,151,171,211

Step 70 Migrate the ESXi hosts existing management vmkernel interface on vSwitch0 to the Nexus 1000V.
70.1 From the browser bar, select Hosts and Clusters.

70.2 Select ESX1 (10.1.111.21), select the Configuration tab, select Networking under Hardware,
select the Virtual Distributed Switch tab, click on Manage Virtual Adapters link:

70.3 Click the Add link, select Migrate existing virtual adapters, then click Next:

70.4 Select MGMT-VLAN for any adapter on the Management Network
70.5 Select NFS-VLAN for any adapter on the NFS source port group.
70.6 Select VMOTION for any adapter on the VMotion source port group.
70.7 Click Next to continue. In the figure below, the current switch should say vSwitch1.

70.8 Click Finish.

1
2

1
2

1
2

1 2

70.9 Verify that all the vmkernel ports for ESX1 have migrated to the Nexus 1000V distributed virtual
switch:

Step 71 Repeat Step 70 to move ESX2 to the Nexus 1000V distributed virtual switch.

Step 72 Verify that jumbo frames are enabled correctly for your vmkernel interfaces.
72.1 From VSM run show interface port-channel to verify that the MTU size is 9000.
vsm-1# show interface port-channel 1-2 | grep next 2 port-c
port-channel1 is up
Hardware: Port-Channel, address: 0050.5652.0e5a (bia 0050.5652.0e5a)
MTU 9000 bytes, BW 20000000 Kbit, DLY 10 usec,
--
port-channel2 is up
Hardware: Port-Channel, address: 0050.5652.0d52 (bia 0050.5652.0d52)
72.2 From both ESXi servers, verify that environment is configured for Jumbo frames end-to-end. We
are going to use the -d option to prevent fragmenting the packet.
~ # vmkping -d -s 8000 -I vmk0 10.1.111.151
PING 10.1.111.151 (10.1.111.151): 8000 data bytes

--- 10.1.111.151 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
Note: In our environment, since the NetApp is plugged into our 3750 management switch, we had to also
enable it for jumbo frames using the command system mtu jumbo 9000.

1
2
3
8.8 MIGRATE VIRTUAL MACHINE PORTS
Once the Nexus 1000V Virtual Ethernet Module is active on a host, it is time to migrate virtual ports to the VEM.
This section specifies procedures for migrating these ports.

Step 73 Migrate the Virtual Machine Interfaces to the Nexus 1000V.
73.1 Right-click on the Server 2003R2 VM and select Edit Settings. This brings up a dialog box
where you can select the network adapter that needs to be migrated.

73.2 Select the VMTRAFFIC port-profile from the drop down list and select OK.
73.3 Verify that your VMs virtual interface is showing up in the VSM.
vsm-1(config)# show interface virtual vm
-------------------------------------------------------------------------------
Port Adapter Owner Mod Host
-------------------------------------------------------------------------------
Veth7 Net Adapter 1 Server-2003R2 3 10.1.111.21
73.4 Test to make sure your VM can ping its gateway.
C:\Documents and Settings\Administrator>ping 10.1.131.254

Pinging 10.1.131.254 with 32 bytes of data:

Reply from 10.1.131.254: bytes=32 time<1ms TTL=128
Step 74 Repeat the above steps for any remaining VMs you have except for your VSM. Be sure to select the
appropriate port profile.

1
2
3
9 CONFIGURING OVERLAY TRANSPORT VIRTUALIZATION

Overlay Transport Virtualization (OTV)

OTV is an industry-first solution that significantly simplifies extending Layer 2 applications across distributed
data centers. You can now deploy Data Center Interconnect (DCI) between sites without changing or
reconfiguring your existing network design. With OTV you can deploy virtual computing resources and clusters
across geographically distributed data centers, delivering transparent workload mobility, business resiliency, and
superior computing resource efficiencies. Key OTV features include:

Extends Layer 2 LANs over any network: Uses IP-encapsulated MAC routing, works over any network
that supports IP, designed to scale across multiple data centers
Simplifies configuration and operation: Enables seamless deployment over existing network without
redesign, requires minimal configuration commands (as few as four), provides single-touch site
configuration for adding new data centers
Increases resiliency: Preserves existing Layer 3 failure boundaries, provides automated multihoming,
and includes built-in loop prevention
Maximizes available bandwidth: Uses equal-cost multipathing and optimal multicast replication

Nexus 7000

The Cisco Nexus 7000 Series is a modular data center class series of switching systems designed for highly
scalable end-to-end 10 Gigabit Ethernet networks. The Cisco Nexus 7000 Series is purpose built for the data
center and has many unique features and capabilities designed specifically for such mission critical place in the
network.

Cisco NX-OS

Cisco NX-OS is a state-of-the-art operating system that powers the Cisco Nexus 7000 Platform. Cisco NX-OS is
built with modularity, resiliency, and serviceability at its foundation. Drawing on its Cisco IOS and Cisco SAN-OS
heritage, Cisco NX-OS helps ensure continuous availability and sets the standard for mission-critical data center
environments.

EXERCISE OBJECTIVES

This hands-on lab will introduce participants to the OTV (Overlay Transport Virtualization) solution for the
Nexus 7000. This innovative feature set simplifies Datacenter Interconnect designs, allowing Data Center
communication and transparent Layer 2 extension between geographically distributed Data Centers.

OTV accomplishes this without the overhead introduced by MPLS or VPLS.

By the end of the laboratory session the participant should be able to understand OTV functionality and
configuration with the Nexus 7000. Students will go through the following steps:
1. System Verification.
2. Base configuration.
3. OSPF Configuration.
4. OTV Configuration and Verification.
5. VMotion across Data Centers.

Each lab POD has a pair of Nexus 7000s that are used as edge devices attached to a layer 3 Core cloud. The core
(which you dont configure) consists of a pair of Nexus 7000s that are used to model a simple L3 WAN core
network. A pair of Nexus 5000s with an attached ESX server represent the access layer.

The equipment we are using is the Nexus 7000 10-slot chassis with dual supervisors, one 48-port GE Copper card
(model N7K-M148GT-12) and one 32-port 10GE fiber card (model N7K-M132XP-12) each.

We will convert our single Data Center site environment into two geographically distributed Data Center sites.
Each site will have one ESXi 4.1 server that is part of the same VMWare Host cluster. The sites are connected via
Nexus 7000 edge devices (virtual device contexts) to a Nexus 7000 IP core (virtual device contexts).

We will configure the Nexus 7000s at Site A and B. The goal of the lab is to establish L2 connectivity between
the two sites and then perform a vmotion over a generic IP core leveraging the Nexus 7000 OTV technology.

9.1 LAB TOPOLOGY
Figure 6- Logical Topology for Single OTV Pod

We leverage the Virtual Device Context feature to consolidate multiple nodes and reduce the number of required equipment. The eight Nexus 7000s (N7K)
below are actually two physical boxes.

Figure 7 - Full Topology for Three Pods in a VDC Deployment

Table 18 - IP Addresses for Uplinks and Loopbacks
POD # Device Interface IP on uplink
POD 1 N7K-1 Eth 1/10 10.1.11.3/24
POD 1 N7K-2 Eth 1/12 10.1.14.4/24
POD 1 N7K-1 Lo0 10.1.0.11/32
POD 1 N7K-2 Lo0 10.1.0.12/32
POD 2 N7K-1 Eth 1/18 10.1.21.5/24
POD 2 N7K-2 Eth 1/20 10.1.24.6/24
POD 2 N7K-1 Lo0 10.1.0.21/32
POD 2 N7K-2 Lo0 10.1.0.22/32
POD 3 N7K-1 Eth 1/26 10.1.31.7/24
POD 3 N7K-2 Eth 1/28 10.1.34.8/24
POD 3 N7K-1 Lo0 10.1.0.31/32
POD 3 N7K-2 Lo0 10.1.0.32/32
Table 19 - OTV Edge Access Ports Connectivity to Access Switches
POD # Device Access Ports Device Access Ports
POD 1 N7K-1 e1/14 N5K-1 e1/19
POD 1 N7K-2 e1/16 N5K-2 e1/20
POD 2 N7K-1 e1/22 N5K-1 e1/19
POD 2 N7K-2 e1/24 N5K-2 e1/20
POD 3 N7K-1 e1/30 N5K-1 e1/19
POD 3 N7K-2 e1/32 N5K-2 e1/20
Table 20 OTV Multicast Addresses
POD # Device Access Ports Device Access Ports
POD 1 N7K-1 e1/14 N5K-1 e1/19
POD 2 N7K-1 e1/22 N5K-1 e1/19
POD 3 N7K-1 e1/30 N5K-1 e1/19

Note: If you did not do Sections 3-5, then you can load the configurations from the tftp server. See
Appendix A: Copying Switch Configurations From a tftp Server for instructions. However, you must
do Sections 6 and 7 to prepare the servers and virtual machines.

2011 Cisco Data Center Virtualization Lab 6: Overlay Transport Virtualization Page 117 of 217
9.2 JOB AIDS

Introductory overview on OTV:
http://www.cisco.com/en/US/prod/switches/ps9441/nexus7000_promo.html

Cisco Nexus 7000 Series Switches:
www.cisco.com/en/US/products/ps9402/index.html

Cisco Nexus 7000 Series Switches Configuration Guides
http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html

Cisco Nexus 7000 Series OTV Quick Start Guide
http://www.cisco.com/en/US/docs/switches/datacenter/sw/nx-os/OTV/b_Cisco_Nexus_7000_Series_OTV_Quick_Start_Guide.html

Cisco NX-OS Home Page:
www.cisco.com/go/nxos

OTV Technology Intro and Deployment Considerations
http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/DCI/whitepaper/DCI3_OTV_Intro.html

OTV between 2 DCs connected with Dark Fiber (sent to corporate editing)
"The scope of this document is to provide guidance on configuring and designing a network with Overlay
Transport Virtualization (OTV) to extend Layer 2 between two Data Centers connected via dark fiber links. This is
a very common DCI deployment model and this paper will be very helpful in guiding AS team, partners and
customer in deploying OTV."
http://bock-bock.cisco.com/wiki_file/N7K:tech_resources:otv/OTV_over_DarkFiber-AS_team.docx

Note: If you do not have access to the above document, please contact your local Cisco SE.

Table 21 - Commands used in this exercise
Command Description
show module Display module information (N7K)
show running-config all | section mgmt0 Show the running configuration, including default values.
show vrf Show the VRF on your system.
show vrf interface Show all the interfaces belonging to any VRF context.
show vrf management interface Show the interfaces that belong to the management VRF.
show version Display information about the software version (N7K)
interface Ethernet Enter interface mode
vrf member management Add an interface to a VRF.
show int mgmt0 Show interface information for mgmt0.
ping 10.1.111.254 vrf management Ping a host via a specified VRF context.
sh running-config | grep next 3 mgmt0 Display every match of mgmt0 along with the next 3 lines.
where Display the CLI context that you are in.
Basic Configuration
vlan 20, 23, 1005
no shut
sh vlan br
N7K-1, N7K-2, N5K-1, N5K-2
spanning-tree vlan 20,23,1005 priority 4096 N7K-1
spanning-tree vlan 20,23,1005 priority 8192 N7K-2
int e1/<5k-7k link>
switchport
switchport trunk allowed vlan 20,23,1005
no shutdown
N7K-1, N7K-2, N5K-1, N5K-2
Internal interface.
OSPF Configuration
feature ospf
router ospf 1
log-adjacency-changes
Step 1 Lets configure Layer 3 and OSPF Routing
N7K-1, N7K-2,
interface loopback0
ip address 10.1.0.y/32
ip router ospf 1 area 0.0.0.0
N7K-1, N7K-2, - Refer to Table 18 for loopback info.
interface e1/<uplink_port>
mtu 9042
ip address 10.1.y.z/24
ip ospf network point-to-point
ip igmp version 3
no shutdown
Step 2 Lets now configure the interface towards N7K-1 (Core
Layer). Join Interface
N7K-1, N7K-2, - Refer to Table 18 for uplink info.
show running-config ospf First, lets check our OSPF configuration
show ip ospf neighbors
show ip ospf int brief
check if we were able to establish adjacency.
show ip route ospf-1 verify if we exchanged routes.
Enable OTV - N7K-1, N7K-2
feature otv Enable the OTV feature.
otv site-vlan 1005 Next, we specify the OTV Site VLAN, which is vlan 1005.
interface Overlay 1
otv control-group 239.<X>.1.1
otv data-group 239.<X>.2.0/28
Configure OTV Overlay Interface
Replace X with pod number.
otv join-interface Ethernet1/<uplink> Join The OTV Site to the Core
otv extend-vlan 20,23
no shutdown
Extend a VLAN Across The Overlay
show running-config otv Check the OTV configuration.
show otv overlay 1 Display local OTV status
sh otv vlan check the status of the VLANs extended across the overlay.
sh otv site see how many OTV edge devices are present at the local site.
show otv adjacency Display the status of adjacent sites
show otv arp-nd-cache Display the OTV ARP/ND L3->L2 Address Mapping Cache
show mac address-table Display the MAC addresses of devices learnt on the VLAN.
SYSTEM VERIFICATION (OPTIONAL)

Interfaces throughout the guide refer to Pod 1. If you are on a different Pod, please refer to Figure 7 to identify
correspondent interfaces.

This section is optional. You can skip this section if you are already familiar with the Nexus 7000
hardware and software infrastructure. In this case jump to CLI Familiarization.

Step 75 Verify you current system configuration. (Optional)

75.1 Log into your Nexus 7000s management interface via ssh using username of admin and password
1234Qwer.
75.2 Lets start by checking the system and its configuration.
N7K-1-OTV-1A# show module
--- ----- -------------------------------- ------------------ ------------
1 32 10 Gbps Ethernet Module N7K-M132XP-12 ok
3 48 10/100/1000 Mbps Ethernet Module N7K-M148GT-11 ok
5 0 Supervisor module-1X N7K-SUP1 active *
6 0 Supervisor module-1X N7K-SUP1 ha-standby

Mod Sw Hw
--- -------------- ------
1 5.1(2) 2.0
3 5.1(2) 1.6
5 5.1(2) 1.8
6 5.1(2) 1.8

--- -------------------------------------- ----------
1 1c-df-0f-d2-05-20 to 1c-df-0f-d2-05-44 JAF1438AMAQ
3 1c-df-0f-4a-06-04 to 1c-df-0f-4a-06-38 JAF1443BLRQ
5 b4-14-89-e3-f6-20 to b4-14-89-e3-f6-28 JAF1444BLHB
6 b4-14-89-df-fe-50 to b4-14-89-df-fe-58 JAF1443DDHF

Mod Online Diag Status
--- ------------------
1 Pass
3 Pass
5 Pass
6 Pass

Xbar Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
1 0 Fabric Module 1 N7K-C7010-FAB-1 ok

<snip>

75.3 Next, we will check the currently running software version. Our lab is currently NX-OS 5.1(2).
N7K-1-OTV-1A# show version
Cisco Nexus Operating System (NX-OS) Software
<snip>

Software
BIOS: version 3.22.0
kickstart: version 5.1(2)
system: version 5.1(2)
BIOS compile time: 02/20/10
kickstart image file is: bootflash:///n7000-s1-kickstart.5.1.2.bin
kickstart compile time: 12/25/2020 12:00:00 [12/18/2010 01:55:20]
system image file is: bootflash:///n7000-s1-dk9.5.1.2.bin
system compile time: 11/29/2010 12:00:00 [12/18/2010 03:02:00]

Hardware
cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")
Intel(R) Xeon(R) CPU with 4115776 kB of memory.
Processor Board ID JAF1444BLHB

Device name: N7K-1-OTV-1A
bootflash: 2029608 kB
slot0: 2074214 kB (expansion flash)

Kernel uptime is 9 day(s), 15 hour(s), 50 minute(s), 32 second(s)

Last reset
Reason: Unknown
System version: 5.1(2)
Service:

plugin
Core Plugin, Ethernet Plugin
N7K-1-OTV-1A#

Note: Cisco Overlay Transport Virtualization (OTV) requires NX-OS version 5.0(3) or higher.

NX-OS is composed of two images:
1. a kickstart image that contains the Linux Kernel and
2. a system image that contains the NX-OS software components. They both show up in the configuration.

In future releases, we will be adding other plug-ins, such as the Storage plug-in for FCoE.

Active Plug-in
CPU
Storage Devices
NX-OS Version
Images
Location
75.4 Lets now take a look at the running configuration.

N7K-1-OTV-1A# show running-config
version 5.1(2)

<omitted config>

vlan 1

<omitted interface config>
interface Ethernet1/9




<omitted interface config>

interface mgmt0
ip address 10.1.111.111/24

75.5 This is the configuration for Pod 1. As explained earlier, the Nexus 7000s in each Pod runs within
a Virtual Device Context (VDC). By using the VDC feature, we can segment the physical Nexus
7000 into multiple logical switches, each of which runs in a separate memory space and only has
visibility into the hardware resources that it owns, providing total isolation between the VDCs.

75.6 One of the features of show running-config in NX-OS is the ability to not only look at the
running-config but to also reveal the default values, which do not appear in the base config. The
keyword to use is all.

N7K-1-OTV-1A# show running-config all | section mgmt0
interface mgmt0
no description
speed auto
duplex auto
no shutdown
cdp enable
ip address 10.1.111.111/24

These are the interfaces available to your Pod
(Virtual Device Context)
Management
Interface Config
MANAGEMENT VRF AND BASIC CONNECTIVITY (OPTIONAL)

The management interface is always part of the management VRF. The management interface mgmt0 is the
only interface allowed to be part of this VRF.

The Management VRF provides total isolation of management traffic from the rest of the traffic flowing through
the box.

In this task we will:
Verify that only the mgmt0 interface is part of the management VRF
Verify that no other interface can be part of the management VRF
Verify that the default gateway is reachable only using the management VRF

Step 76 Verify VRF characteristics and behavior.

76.1 Verify that only the mgmt0 interface is part of the management VRF

N7K-1-OTV-1A# show vrf
VRF-Name VRF-ID State Reason
default 1 Up --
management 2 Up --

N7K-1-OTV-1A#show vrf interface
Ethernet1/9 default 1

<omitted output>
mgmt0 management 2

N7K-1-OTV-1A# show vrf management interface
mgmt0 management 2

Note: The management VRF is part of the default configuration and the management interface mgmt0 is
the only interface that can be made member of this VRF. Lets verify it.

76.2 Verify that no other interface can be part of the management VRF.
Note: The following example is for Pod1. Please use e1/17 for Pod 2 or e1/25 for Pod3.
N7K-1-OTV-1A# conf t
N7K-1-OTV-1A(config)# interface ethernet1/9
N7K-1-OTV-1A(config-if)# vrf member management
% VRF management is reserved only for mgmt0

N7K-1-OTV-1A(config-if)# show int mgmt0
mgmt0 is up
Hardware: GigabitEthernet, address: 0022.5577.f8f8 (bia 0022.5577.f8f8)
Internet Address is 10.1.111.17/16
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA
full-duplex, 1000 Mb/s
Auto-Negotiation is turned on
EtherType is 0x0000
1 minute input rate 88 bits/sec, 0 packets/sec
1 minute output rate 24 bits/sec, 0 packets/sec
Rx
9632 input packets 106 unicast packets 5999 multicast packets
3527 broadcast packets 1276448 bytes
<snip>

FastEthernet? GigabitEthernet?... No,
just ethernet interfaces
Lab
Hack!
76.3 Verify that the default gateway is not reachable when using the default VRF. Try reaching the out-
of-band management networks default gateway with a ping.
N7K-1-OTV-1A(config-if)# ping 10.1.111.254
PING 10.1.111.254 (10.1.111.254): 56 data bytes
Request 0 timed out
Request 1 timed out
Request 2 timed out
Request 3 timed out
Request 4 timed out

--- 10.1.111.254 ping statistics ---

N7K-1-OTV-1A(config-if)#
Note: The ping fails because we are trying to reach a system on the out-of-band management network
without specifying the correct VRF.
76.4 Verify that the default gateway is reachable using the management VRF. Try reaching the MGMT
VRFs default gateway with a ping.
Note: In our lab environment, we could not use the mgmt0 interface or management
VRF. Instead, we used the last gigabit port in each as the management interface
and placed into a new VRF called MGMT. To ping other devices in the network
from the Nexus 7000s, you will need to specify this VRF context.

N7K-1-OTV-1A# ping 10.1.111.254 vrf MGMT

PING 10.1.111.254 (10.1.111.254): 56 data bytes

--- 10.1.111.254 ping statistics ---

Linux-like output
CLI FAMILIARIZATION (OPTIONAL)

NX-OS CLI is very IOS-like as you will notice when configuring the system. Also NX-OS implements a hierarchically
independent CLI, so that any command can be issued from any CLI context.

Note: This section is optional. You can skip this section if you are already familiar with the Nexus 7000 CLI
capabilities. In this case, jump to Base Configuration.

In this step we will:
Verify the CLI hierarchy independence by issuing a ping from different CLI contexts
Verify the CLI piping functionality

Step 77 Explore NX-OS CLI capabilities.

77.1 Verify the CLI hierarchy independence by issuing a ping from different CLI contexts
N7K-1-OTV-1A(config)#ping ?
*** No matches in current mode, matching in (exec) mode ***
<CR>
A.B.C.D or Hostname IP address of remote system
WORD Enter Hostname
multicast Multicast ping

N7K-1-OTV-1A(config)#ping 10.1.111.254 vrf management
PING 10.1.111.254 (10.1.111.254): 56 data bytes
<snip>

--- 10.1.111.254 ping statistics ---

N7K-1-OTV-1A(config)#int e1/9
N7K-1-OTV-1A(config-if)# ping ?
*** No matches in current mode, matching in (exec) mode ***
<CR>
A.B.C.D or Hostname IP address of remote system
WORD Enter Hostname
multicast Multicast ping
77.2 Issue ping from within interface configuration context.
N7K-1-OTV-1A(config-if)#ping 10.1.111.254 vrf management
PING 10.1.111.254 (10.1.111.254): 56 data bytes
<snip>

--- 10.1.111.254 ping statistics ---
77.3 You can use the up-arrow and get the command history from the exec mode. Any command can
be issued from anywhere within the configuration.

Hierarchically
Independent CLI
Hierarchically
Independent CLI
77.4 Verify the CLI piping functionality. Multiple piping options are available. Lots of them derived from
the Linux world.
N7K-1-OTV-1A(config-if)#show running-config | ?
cut Print selected parts of lines.
diff Show difference between current and previous invocation
(creates temp files: remove them with 'diff-clean' command
and dont use it on commands with big outputs, like 'show
tech'!)
egrep Egrep - print lines matching a pattern
grep Grep - print lines matching a pattern
head Display first lines
human Output in human format (if permanently set to xml, else it
will turn on xml for next command)
last Display last lines
less Filter for paging
no-more Turn-off pagination for command output
perl Use perl script to filter output
section Show lines that include the pattern as well as the
subsequent lines that are more indented than matching line
sed Stream Editor
sort Stream Sorter
sscp Stream SCP (secure copy)
tr Translate, squeeze, and/or delete characters
uniq Discard all but one of successive identical lines
vsh The shell that understands cli command
wc Count words, lines, characters
xml Output in xml format (according to .xsd definitions)
begin Begin with the line that matches
count Count number of lines
end End with the line that matches
exclude Exclude lines that match
include Include lines that match
77.5 See options for piping to grep.
N7K-1-OTV-1A(config-if)#sh running-config | grep ?
WORD Search for the expression
count Print a total count of matching lines only
ignore-case Ignore case difference when comparing strings
invert-match Print only lines that contain no matches for <expr>
line-exp Print only lines where the match is a whole line
line-number Print each match preceded by its line number
next Print <num> lines of context after every matching line
prev Print <num> lines of context before every matching line
word-exp Print only lines where the match is a complete word
77.6 Display any line that contains mgmt0 and print the next 3 lines after that match.
N7K-1-OTV-1A(config-if)#sh running-config | grep next 3 mgmt0
interface mgmt0
no snmp trap link-status
ip address 10.1.111.17/16
77.7 The [TAB] completes a CLI command and shows the available keywords.
N7K-1-OTV-1A(config-if)# int mgmt 0
N7K-1-OTV-1A(config-if)# [TAB]
cdp exit no shutdown where
description ip pop snmp
end ipv6 push vrf
77.8 If you want to know the CLI context you are in use the where command.
N7K-1-OTV-1A(config-if)# where
conf; interface mgmt0 admin@N7K-1-OTV-1A%default
N7K-1-OTV-1A(config-if)#end

Improved CLI Piping
9.3 BASE CONFIGURATION

In this first step, we will configure the Nexus 5000s to simulate two separate sites. ESX1 and ESX3 will be
on Site A (N5K-1). ESX2 will be on Site B(N5K-2)

Step 78 Split the Data Center into two sites.
78.1 Login to the Nexus 5000s with the following credentials:
Username: admin
Password: 1234Qwer
78.2 Turn off VPC.
no feature vpc
78.3 Shutdown interfaces not needed for Site A on N5K-1.
int port-channel 1
shutdown
78.4 Remove ESX2 from Site A.
int e1/10
interface po14
shutdown
78.5 We need to enable the attached interfaces on N5K-1.
vlan 131,151,171,211,1005
no shut
int e1/19
switchport
no shutdown
78.6 Login to the Nexus 5000s with the following credentials:
Username: admin
Password: 1234Qwer
78.7 Turn off VPC.
no feature vpc
78.8 Shutdown interfaces not needed for Site B on N5K-2.
shutdown
!interface port-channel 101
! shutdown
78.9 Remove ESX 1 & 3 from Site B. We are also shutting down the connection to the 3750 on the B
side.
interface e1/4,e1/9,e1/11
interface po20,po13,po15
shutdown
78.10 We need to enable the attached interfaces on N5K-2.
vlan 131,151,171,211,1005
no shut
int et 1/20
switchport
no shutdown

Summary of Commands
no feature vpc
int port-channel 1
shutdown
int e1/10
interface po14
shutdown
vlan 131,151,171,211,1005
no shut
int e1/19
switchport
no shutdown
no feature vpc
shutdown
! shutdown
!interface e1/4,e1/9,e1/11
shutdown
vlan 131,151,171,211,1005
no shut
int et 1/20
switchport
no shutdown

You have three options at this point. Option 3 is under maintenance, so do NOT use.

1) Go to the next step (Spanning Tree) to manually configure OTV
2) Copy and paste the commands from the Command Summary for OTV on page 212.
3) Restore an OTV config and go to Section 9.8. Perform the following commands on both Nexus 7000s
to load OTV config. SSH into N7K-1 (10.1.111.3) and N7K-2 (10.1.111.4)
rollback running-config checkpoint OTV
copy run start
reload vdc

9.4 SPANNING TREE

It is time to bring up the interfaces and configure the Spanning Tree Protocol.

Rapid-STP aka 802.1w is now incorporated in IEEE 802.1D-2004. Cisco's implementation of RSTP in both
NX-OS and IOS provides a separate spanning tree instance for each active VLAN, which permits greater
flexibility of Layer 2 topologies in conjunction with IEEE 802.1Q trunking. This implementation is also
referred to as Rapid Per-VLAN Spanning Tree (Rapid-PVST). Rapid-PVST is the default spanning tree
mode for NX-OS, so it does not need to be explicitly enabled.

Step 79 Configure the VLANs in each data-center site. Log in to both N7K-1 and N7K-2 via Putty SSH client.
Note: Each site must have two sets of VLANs. One will be local to the site and one set will be extended on
the overlay to the remote data-center site. VLANs are 131, 151 , 171, 211 and 1005. Vlan 131 is the
VM-Client traffic interface. Vlan 151 is used for vmotion traffic. VLAN 1005 is used for intra-site OTV
communication.
N7K-1
Enter configuration commands, one per line. End with CNTL/Z.
79.1 Create necessary VLANs.
vlan 131,151,171,211,1005
no shut
79.2 Verify VLANs.
sh vlan br

---- -------------------------------- --------- -------------------------------
1 default active
20 VLAN0020 active
23 VLAN0023 active
160 VLAN0160 active
1005 VLAN1005 active
N7K-2
79.3 Repeat Step 79.2 for N7K-2-OTV.
Note: Best practices dictate deterministic placement of the spanning tree root in the network. Particularly a
network administrator should ensure that a root switch does not inadvertently end up on a small
switch in the access layer creating a sub-optimal topology more prone to failures.
N7K-1
N7K-1-OTV-1A(config-vlan)#spanning-tree vlan 131,151,171,211,1005 priority 4096
N7K-2
N7K-2-OTV-1B(config-vlan)#spanning-tree vlan 131,151,171,211,1005 priority 8192

Step 80 Now lets bring up the interfaces facing on N5K-1 and N5K-2 in the Access Layer.
N7K-1
80.1 Enable switching for interface connecting to N5K-1.

Refer to Table 19 and Figure 7 for your specific interfaces. (ex. Pod 1:e1/14,Pod2:e1/22,Pod3:e1/30)
int e1/14
switchport
mtu 9216
80.2 Allow VLAN for VM Traffic, VMotion, Control/Packet, and OTV Site.
This will cause VLANS to be overwritten. Continue anyway? [yes] y
no shutdown

N7K-2
80.3 Enable switching for interface connecting to N5K-2.

Refer to Table 19 and Figure 7 for your specific interfaces. (ex. Pod 1:e1/16,Pod2:e1/24,Pod3:e1/32)
int e1/16
switchport
mtu 9216
80.4 Allow VLAN for VM Traffic, VMotion, Control/Packet, and OTV Site.
This will cause VLANS to be overwritten. Continue anyway? [yes] y
no shutdown

Summary of Commands
N7K-1
vlan 131,151,171,211,1005
no shut
spanning-tree vlan 131,151,171,211,1005 priority 4096
int e1/14
switchport
no shutdown
N7K-2
vlan 131,151,171,211,1005
no shut
int e1/16
switchport
no shutdown

Step 81 Check the spanning-tree from both the Nexus 7000 and the Nexus 5000.
N7K-1
N7K-1-OTV-1A#show spanning-tree vlan 1005

VLAN1005
Spanning tree enabled protocol rstp
Root ID Priority 5101
Address 0026.980d.6d42
This bridge is the root
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 5101 (priority 4096 sys-id-ext 1005)

Interface Role Sts Cost Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Eth1/14 Desg FWD 2 128.142 P2p
N7K-2
N7K-1-OTV-1A# show spanning-tree vlan 131

VLAN0020
This bridge is the root


---------------- ---- --- --------- -------- --------------------------------

N5K-1 and N5K-2
N5K-1# show spanning-tree vlan 131

VLAN0020
Cost 2
Port 147 (Ethernet1/19)

Address 0005.9b7a.03bc

---------------- ---- --- --------- -------- --------------------------------
Eth1/19 Root FWD 2 128.147 P2p
Eth100/1/1 Desg FWD 4 128.1025 Edge P2p
Eth100/1/2 Desg FWD 4 128.1026 Edge P2p

Step 82 Verify that you have the correct licenses. OTV requires the LAN Advanced Services license and the
Transport Services license.
N7K-1 and N7K-2
N7K-1-OTV-1A# show license usage
Feature Ins Lic Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
ENHANCED_LAYER2_PKG No - Unused -
SCALABLE_SERVICES_PKG No - Unused -
TRANSPORT_SERVICES_PKG Yes - In use Never -
LAN_ADVANCED_SERVICES_PKG Yes - Unused Never -
LAN_ENTERPRISE_SERVICES_PKG Yes - In use Never -

Note: Be sure to confirm the status of your customers license status and remind them to purchase the
license before the feature grace period expires. Temporary licenses are indicated by the word
Grace in the comments field that reflects the grace period in days and hours left on your temporary
license. In the example below, there is 105 days 15 hours left.
TRANSPORT_SERVICES_PKG No - Unused Grace 105D 15H

Uplink port to N7K OTV is Root
port. N7K is the Root Bridge.
9.5 INTERFACE CONFIGURATION

Identify the interconnection to the core and configure OSPF for L3 connectivity. This interface will be designated
as the Join interface of the OTV Edge device

Step 83 Lets now enable the interface on the Nexus 7000 edge device that connects it to the core. Look at the
topology diagram and based on your POD topologypick one of the 2 interfaces connected to the Core.
First, un-shut both connections to the core:

Refer to Table 18 - IP Addresses for Uplinks and Loopbacks and Figure 7 for your specific interfaces. (ex.
Pod 1:e1/10,Pod2:e1/18,Pod3:e1/26)
N7K-1
N7K-1-OTV-1A(config)# int e 1/<uplink>
N7K-1-OTV-1A(config-if-range)# no shut
83.1 Verify connectivity between your OTV and access switch.
N7K-1-OTV-1A# show cdp neighbors
[Snip]
N7K-1(TBM14364817) Eth1/10 138 R S I s N7K-C7010 Eth1/1
N5K-1(SSI14100CHE) Eth1/14 123 S I s N5K-C5010P-BF Eth1/19[Snip]
N7K-2
N7K-2-OTV-1B(config)# int e 1/<uplink>
N7K-2-OTV-1B(config-if-range)# no shut
83.2 Verify connectivity between your OTV and access switch.
N7K-2-OTV-1B # sh cdp neighbors
[Snip]
N7K-2(TBM14364915) Eth1/12 173 R S I s N7K-C7010 Eth1/2
N5K-2(SSI141004P3) Eth1/16 148 S I s N5K-C5010P-BF Eth1/20
[snip]

Note: CDP may take a while to establish neighborship

Summary of Commands
int e 1/<uplink>
no shut

9.6 OSPF CONFIGURATION

Before Overlay Transport Virtualization (OTV) can be used to extend a Layer 2 domain in Site A to Site B,
you need to setup Layer 3 connectivity between these two sites. This section shows you how to use OSPF
to accomplish this. The Core devices have already been pre-configured.

Step 84 Enable OSPF
N7K-1
84.1 Enable OSPF feature and configure OSPF instance.
N7K-1-OTV-1A(config)# feature ospf
N7K-1-OTV-1A(config)# router ospf 1
N7K-1-OTV-1A(config-router)# log-adjacency-changes

NX-OS is a fully modular operating system. Most software modules dont run unless the correspondent
feature is enabled. We refer to these features that need to be specifically enabled as conditional
services. Once the service is enabled, the CLI becomes visible and the feature can be used and
configured.
84.2 Configure loopback interface for OSPF.

Refer to Table 18 - IP Addresses for Uplinks and Loopbacks and Figure 7 for your specific interfaces.
N7K-1-OTV-1A(config)# interface loopback0
N7K-1-OTV-1A(config-if)# ip address 10.1.0.X1/32
N7K-1-OTV-1A(config-if)# ip router ospf 1 area 0.0.0.0
84.3 Configure each OTV Edges uplink interface that connects to the Nexus WAN(Core Layer).

Refer to Table 18 - IP Addresses for Uplinks and Loopbacks and Figure 7 for your specific interfaces. (ex.
Pod 1:e1/10,Pod2:e1/18,Pod3:e1/26)
N7K-1-OTV-1A(config)# interface e1/<uplink_port>
84.4 Specify a larger MTU to accommodate overhead from OTV header.
N7K-1-OTV-1A(config-if)# mtu 9042

We increased the MTU on the layer 3 links to 9042 bytes. OTV encapsulates the original frame adding 42
bytes to your IP packet, so you will need to increase the MTU on all your WAN links. Since the MTU on
the core has already been adjusted to 9042, you will get an OSPF state of EXSTART until your MTU
matches the core MTU.
N7K-1-OTV-1A(config-if)# ip address 10.1.X1.Y /24

Refer to Table 18 - IP Addresses for Uplinks and Loopbacks and Figure 7 for your specific interfaces.
(ex. Pod 1:10.1.11.3,Pod2:10.1.21.5,Pod3:10.1.31.7)
84.5 Specify OSPF interface network type and OSPF Area.
N7K-1-OTV-1A(config-if)# ip ospf network point-to-point
N7K-1-OTV-1A(config-if)# ip router ospf 1 area 0.0.0.0
84.6 Configure IGMPv3 on join-interface.
N7K-1-OTV-1A(config-if)# ip igmp version 3
84.7 Enable the interface.
N7K-1-OTV-1A(config-if)# no shutdown

The edge devices interface towards the IP core will later be used by OTV as a join interface. Therefore, it
needs to be configured for IGMP version 3.

N7K-2

For the following steps, refer to Table 18 - IP Addresses for Uplinks and Loopbacks and Figure 7 for your
specific interfaces.
84.8 Enable OSPF feature and configure OSPF instance.
N7K-2-OTV-1B(config)# feature ospf
N7K-2-OTV-1B(config)# router ospf 1
N7K-2-OTV-1B(config-router)# log-adjacency-changes
84.9 Configure loopback interface for OSPF.
N7K-2-OTV-1B(config)# interface loopback0
N7K-2-OTV-1B(config-if)# ip address 10.1.0.X2/32
N7K-2-OTV-1B(config-if)# ip router ospf 1 area 0.0.0.0
84.10 Configure each OTV Edges uplink interface that connects to the Nexus WAN(Core Layer).
N7K-2-OTV-1B(config)# interface e1/<uplink>
N7K-2-OTV-1B(config-if)# mtu 9042
N7K-2-OTV-1B(config-if)# ip address 10.1.X4.Y/24
N7K-2-OTV-1B(config-if)# ip ospf network point-to-point
N7K-2-OTV-1B(config-if)# ip router ospf 1 area 0.0.0.0
N7K-2-OTV-1B(config-if)# ip igmp version 3
N7K-2-OTV-1B(config-if)# no shutdown

We increased the MTU on the layer 3 links to 9042 bytes. OTV encapsulates the original frame adding 42
bytes to your IP packet, so you will need to increase the MTU on all your WAN links. Since the MTU on
the core has already been adjusted to 9042, you will get an OSPF state of EXSTART until your MTU
matches the core MTU.

Summary of Commands
N7K-1
feature ospf
router ospf 1

interface loopback0
ip address 10.1.0.X1/32
interface e1/<uplink_port>
mtu 9042
ip address 10.1.X1.Y/24
ip igmp version 3
no shutdown
N7K-2
feature ospf
router ospf 1

interface loopback0
ip address 10.1.0.X2/32
interface e1/<uplink>
mtu 9042
ip address 10.1.X4.Y/24
ip igmp version 3
no shutdown

Step 85 Verify OSPF configuration
85.1 First, lets check our running OSPF configuration. (example from Pod1)
N7K-1-OTV-1A# show running-config ospf
<snip>
feature ospf

router ospf 1

interface loopback0


N7K-2-OTV-1B(config-if)# show running-config ospf
<snip>
feature ospf

router ospf 1

interface loopback0

85.2 Check if the ospf interfaces are up and have neighbors.
N7K-1
N7K-1-OTV-1A# show ip ospf int brief
OSPF Process ID 1 VRF default
Total number of interface: 2
Interface ID Area Cost State Neighbors Status
Lo0 1 0.0.0.0 1 LOOPBACK 0 up
Eth1/10 2 0.0.0.0 4 P2P 1 up
N7K-2
N7K-2-OTV-1B# show ip ospf int bri
Total number of interface: 2
Interface ID Area Cost State Neighbors Status
Lo0 1 0.0.0.0 1 LOOPBACK 0 up
Eth1/12 2 0.0.0.0 4 P2P 1 up
85.3 Next, we will check on our OSPF neighbor adjacency.
N7K-1
N7K-1-OTV-1A# sh ip ospf neighbors
Total number of neighbors: 1
Neighbor ID Pri State Up Time Address Interface
10.1.0.1 1 FULL/ - 02:49:37 10.1.11.1 Eth1/10
N7K-2

N7K-2-OTV-1B# show ip ospf neighbors
Total number of neighbors: 1
Neighbor ID Pri State Up Time Address Interface
10.1.0.2 1 FULL/ - 1w1d 10.1.14.2 Eth1/12

85.4 Lastly, we will verify if we exchanged routes.
N7K-1
N7K-1-OTV-1A(config)# show ip route ospf-1
IP Route Table for VRF "default"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]

10.1.0.1/32, ubest/mbest: 1/0
*via 10.1.11.1, Eth1/10, [110/5], 1w1d, ospf-1, intra
10.1.0.2/32, ubest/mbest: 1/0
10.1.0.12/32, ubest/mbest: 1/0
10.1.7.0/24, ubest/mbest: 1/0
10.1.14.0/24, ubest/mbest: 1/0
N7K-2

Note: Congratulations, youve successfully configured OSPF. Please continue to the next section.

9.7 CONFIGURING OTV TO CONNECT EDGE DEVICES TO REMOTE END-SITES

OTV provides Layer 2 connectivity between remote network sites. OTV uses MAC address-based routing and IP-
encapsulated forwarding across a Layer 3 network to provide support for applications that require Layer 2
adjacency, such as clusters and Vmotion. You deploy OTV on the edge devices in each site. OTV requires no
other changes to the sites or the core network. OTV avoids the addition of multiple routing tables to every
device in the network that other methods, such as Multiprotocol Label Switching (MPLS), require.

Figure 8 - OTV Packet Flow

The following terminology is used for OTV throughout this document:

Site: A Layer 2 network that may be single-homed or multi-homed to the core network and the OTV
overlay network. Layer 2 connectivity between sites is provided by edge devices that operate in an
overlay network. Layer 2 sites are physically separated from each other by the core IP network.
Core Network: The customer backbone network that connects Layer 2 sites over IP. This network can be
customer managed, provided by a service provider, or a mix of both. OTV is transparent to the core
network because OTV flows are treated as regular IP flows.
Edge Device: A Layer 2 switch that performs OTV functions. An edge device performs typical Layer 2
learning and forwarding on the site-facing interfaces (internal interfaces) and performs IP-based
virtualization on the core-facing interfaces. The edge device can be collocated in a device that performs
Layer 3 routing on other ports. OTV functionality only occurs in an edge device.
Internal Interface: The Layer 2 interface on the edge device that connects to site-based switches or site-
based routers. The internal interface is a Layer 2 interface regardless of whether the internal interface
connects to a switch or a router.
Join Interface: The interface facing the core network. The name implies that the edge device joins an
overlay network through this interface. The IP address of this interface is used to advertise reachability
of a MAC address present in this site.


Figure 9 - OTV Terminology (1 of 2)

MAC Routing: MAC routing associates the destination MAC address of the Layer 2 traffic with an edge
device IP address. The MAC to IP association is advertised to the edge devices through an overlay
routing protocol. In MAC routing, MAC addresses are reachable through an IP next hop. Layer 2 traffic
destined to a MAC address will be encapsulated in an IP packet based on the MAC to IP mapping in the
MAC routing table.
Overlay Interface: A logical multi-access multicast-capable interface. The overlay interface encapsulates
Layer 2 frames in IP unicast or multicast headers. The overlay interface is connected to the core via one
or more physical interfaces. You assign IP addresses from the core network address space to the physical
interfaces that are associated with the overlayinterface.
Overlay Network: A logical network that interconnects remote sites for MAC routing of Layer 2 traffic.
The overlay network uses either multicast routing in the core network or an overlay server to build an
OTV routing information base (ORIB). The ORIB associates destination MAC addresses with remote edge
device IP addresses.
Multicast Control-Group: For core networks supporting IP multicast, one multicast address (the control-
group address) is used to encapsulate and exchange OTV control-plane protocol updates. Each edge
device participating in the particular Overlay network shares the same control-group address with all the
other edge devices. As soon as the control-group address and the join interface is configured, the edge
device sends an IGMP report message to join the control group and with that participates in the overlay
network. The edge devices act as hosts in the multicast network and send multicast IGMP report
messages to the assigned multicast group address.
Multicast Data-Group: In order to handle multicast data-traffic one or more ranges of IPv4 multicast
group prefixes can be used. The multicast group address is an IPv4 address in dotted decimal notation. A
subnet mask is used to indicate ranges of addresses. Up to eight data-group ranges can be defined. An
SSM group is used for the multicast data generated by the site.
Authoritative Edge Device: An edge device that forwards Layer 2 frames into and out of a site over the
overlay interface. For the first release of OTV, there is only one authoritative edge device for all MAC
unicast and multicast addresses per VLAN. Each VLAN can be assigned to a different authoritative edge
device.

Figure 10- OTV Terminology (2 of 2)

In this section you will:

Select the Join interface and establish OSPF connectivity with the Core.
Enable OTV
Configure the Overlay interface
Join the Data-Center site to the Core
Extend a VLAN across the overlay

Step 86 Configuring Basic OTV Features
N7K-1
86.1 Enable the OTV feature.
feature otv
86.2 Specify the OTV Site VLAN, which is vlan 1005.
otv site-vlan 1005

The OTV Site VLAN is used to communicate with other OTV edge devices in the local site. If our site had
dual edge devices, it will be used to elect the active forwarder device in the site.

Ensure that the site VLAN is active on at least one of the edge device ports.
86.3 Configure the site identifier. We will use 0x1 for Site A on N7K-1.
otv site-identifier 0x1

OTV uses the site identifier to support dual site adjacency. Dual site adjacency uses both site VLAN and
site identifier to determine if there are other edge devices on the local site and if those edge devices can
forward traffic. Ensure that the site identifier is the same on all neighbor edge devices in the site.

You must configure the site identifier in Cisco NX-OS release 5.2(1) or later releases.
The overlay network will not become operational until you configure the site identifier.

The Site-VLAN and site identifier must be configured before entering the no shutdown command for any
interface overlay and must not be modified while any overlay is up within the site.
86.4 Create an overlay interface.
interface Overlay 1
86.5 Specify the multicast group OTV will use for control plane traffic.
otv control-group 239.X.1.1

The control-group address is used for control plane related operations. Each edge device joins the group
and sends control/protocol related packets to this group. This is used for discovery of other edge-devices.
86.6 Specify the multicast address range OTV will use for multicast data traffic.
otv data-group 239.X.2.0/28

The data-group-range specifies a multicast group range that is used for multi-destination traffic.
86.7 Assign a physical interface to the overlay interface.

Refer to Table 18 for the uplink interface.
N7K-1-OTV-1A(config-if-overlay)# otv join-interface Ethernet1/<uplink>
OTV needs join interfaces to be configured for IGMP version 3

After you enter the join command an informational message reminds you that IGMPv3 is required to be
configured on the join interface. This message can be ignored if IGMPv3 was already configured as
instructed earlier in the guide.

This interface is used for overlay operations such as discovering remote edge-devices, providing the
source address for OTV encapsulated packets and the destination address for unicast traffic sent by
remote edge-devices.
86.8 Specify the VLANs to be extended across the overlay. We will extend VLAN 131,151,171, and 211.
otv extend-vlan 131,151,171,211
no shutdown

OTV only forwards Layer 2 packets for VLANs that are in the specified range for the overlay interface.

Replace X with your POD # (1
for POD 1, 2 for POD 2 and so
on).
N7K-2-OTV-XB
86.9 Enable the OTV feature.
feature otv
86.10 Specify the OTV Site VLAN, which is vlan 1005.
otv site-vlan 1005
86.11 Configure the site identifier. We will use 0x2 for Site B on N7K-2.
86.12 Create an overlay interface.
interface Overlay 1
86.13 Specify the multicast group OTV will use for control plane traffic.
otv control-group 239.X.1.1
86.14 Specify the multicast address range OTV will use for multicast data traffic.
otv data-group 239.X.2.0/28
86.15 Assign a physical interface to the overlay interface.

Refer to Table 18 for the uplink interface.
otv join-interface Ethernet1/<uplink>
86.16 Specify the VLANs to be extended across the overlay. We will extend VLAN 131,151,171, and 211.
no shutdown

Replace X with your POD # (1
for POD 1, 2 for POD 2 and so
on).
N7K-1 and N7K-2
86.17 Now lets check the OTV configuration just completed:
N7K-1-OTV-1A(config-if-overlay)# show running-config otv
<SNIP>
feature otv

otv site-vlan 1005

interface Overlay1
otv join-interface Ethernet1/10
otv control-group 239.1.1.1
otv data-group 239.1.2.0/28
otv extend-vlan 131, 151, 171
no shutdown

N7K-2-OTV-1B(config-if-overlay)# show running-config otv
<snip>
feature otv

otv site-vlan 1005

interface Overlay1
otv extend-vlan 131, 151, 171
no shutdown

Note: You have now completed the OTV configuration in your POD.

Summary of Commands
N7K-1
feature otv
otv site-vlan 1005

interface Overlay 1
no shutdown
N7K-2
feature otv
otv site-vlan 1005

interface Overlay 1
no shutdown

9.8 OTV VERIFICATION AND MONITORING

In this task we will monitor and troubleshoot the Overlay Transport Virtualization (OTV) configuration and verify
connectivity to the Remote Data Center site.

These are the steps for this exercise:

Display local OTV status
Display the status of adjacent sites
Display the OTV ARP/ND L3->L2 Address Mapping Cache

Step 87 First, lets display the OTV overlay status for your sites:
N7K-1-OTV-1A(config-if-overlay)# show otv overlay 1
OTV Overlay Information
Site Identifier 0000.0000.0000

Overlay interface Overlay1

VPN name : Overlay1
VPN state : UP
Extended vlans : 131 151 171 211 (Total:4)
Control group : 239.1.1.1
Data group range(s) : 239.1.2.0/28
Join interface(s) : Eth1/10 (10.1.11.3)
Site vlan : 1005 (up)
AED-Capable : Yes
Capability : Multicast-Reachable

N7K-2-OTV-1B# show otv overlay 1

OTV Overlay Information
Site Identifier 0000.0000.0000

Overlay interface Overlay1

VPN name : Overlay1
VPN state : UP
Extended vlans : 131 151 171 211 (Total:4)
Control group : 239.1.1.1
Data group range(s) : 239.1.2.0/28
Join interface(s) : Eth1/12 (10.1.14.4)
Site vlan : 1005 (up)
AED-Capable : Yes
Capability : Multicast-Reachable
Note: Make sure the state is up, and that the vlans and addresses are correct.

87.1 Next, lets check the status of the VLANs extended across the overlay.
Note: The authoritative device is the OTV node elected to forward traffic to/from the L3 core. For any
given VLAN, only one authoritative edge device (AED) will be elected in a site. The * symbol next to
the VLAN ID indicates that the device is the AED for that vlan.

N7K-1-OTV-1A(config-if-overlay)# sh otv vlan
OTV Extended VLANs and Edge Device State Information (* - AED)

VLAN Auth. Edge Device Vlan State Overlay
---- ----------------------------------- ---------- -------
131* N7K-1-OTV-1A active Overlay1

N7K-2-OTV-1B(config)# show otv vlan
OTV Extended VLANs and Edge Device State Information (* - AED)

VLAN Auth. Edge Device Vlan State Overlay
---- ----------------------------------- ---------- -------
131* N7K-2-OTV-1B active Overlay1

87.2 Next, lets see how many OTV edge devices are present at the local site. The * symbol next to the
hostname indicates that this is the local node.
N7K-1-OTV-1A(config-if-overlay)# sh otv site

Site Adjacency Information (Site-VLAN: 1005) (* - this device)

Overlay1 Site-Local Adjacencies (Count: 2)

Hostname System-ID Up Time Ordinal
-------------------------------- -------------- --------- ----------
* N7K-1-OTV-1A 0026.980d.6d42 00:05:58 0
N7K-2-OTV-1B 0026.980d.92c2 00:05:37 1

Note: If this was a dual-homed site, two nodes would be listed through this command. The other node
would not have a * symbol next to it.

N7K-2-OTV-1B(config-if-overlay)# sh otv site

Site Adjacency Information (Site-VLAN: 1005) (* - this device)

Overlay1 Site-Local Adjacencies (Count: 2)

Hostname System-ID Up Time Ordinal
-------------------------------- -------------- --------- ----------
N7K-1-OTV-1A 0026.980d.6d42 00:10:09 0
* N7K-2-OTV-1B 0026.980d.92c2 00:09:49 1


Step 88 Verify if we connected to the peer edge device at the peer Site.
Note: We should see the remote edge device in our adjacency database.
N7K-1-OTV-1A# show otv adjacency
Overlay Adjacency database

Overlay-Interface Overlay1 :
Hostname System-ID Dest Addr Up Time State
N7K-2-OTV-1B 0026.980d.92c2 10.1.14.4 1w2d UP

N7K-2-OTV-1B# show otv adjacency
Overlay Adjacency database

Overlay-Interface Overlay1 :
Hostname System-ID Dest Addr Up Time State
N7K-1-OTV-1A 0026.980d.6d42 10.1.11.3 07:16:05 UP

88.1 The MAC address table will report MAC addresses of end-hosts and devices learnt on the VLAN. If
no traffic was ever sent across the overlay, then only the local router MAC will be populated in the
table.
N7K-1-OTV-1A# show mac address-table
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports
---------+-----------------+--------+---------+------+----+-----------------
G - 0026.980d.6d42 static - F F sup-eth1(R)

88.2 The MAC address in the table is actually the local router MAC, lets verify this:

Refer to Table 18 - IP Addresses for Uplinks and Loopbacks for the correct uplink interface.
show interface e1/<uplink> mac-address

N7K-1-OTV-1A# show interface e1/10 mac-address

--------------------------------------------------------------------------------
Interface Mac-Address Burn-in Mac-Address
--------------------------------------------------------------------------------
Ethernet1/10 0026.980d.6d42 1cdf.0fd2.0529

Step 89 Display the OTV ARP/ND L3->L2 Address Mapping Cache. In OTV, we also cache ARP resolution for
MAC addresses that are not local to the site and that are learnt via the overlay. If no traffic was ever
sent across the overlay, then no ARP would have been resolved, and so no entries are cached by the
OTV process.
N7K-1-OTV-1A# show otv arp-nd-cache
OTV ARP/ND L3->L2 Address Mapping Cache

9.9 VERIFYING THE VMWARE VSPHERE SETUP

In this lab exercise youll explore the connectivity of the VMware vSphere ESXi hosts to the edge device within
the two sites. You will notice that the ESXi hosts in both sites have access to a Local VLAN, which is not
extended via OTV between the sites, and to an VM-Client VLAN, which has been extended between the two
sites.


Connecting to the VMware vCenter host
Verifying available port groups and interface mappings
Verifying Virtual Machine to port group mappings

Step 90 Connect to vCenter with the vSphere Client
90.1 After a successful login youll see the following vSphere Client application screen.

You can see that a single VMware vSphere logical Data Center with the name FlexPod_DC_1 exists,
which includes a cluster named FlexPod_Mgmt. This cluster consists of three ESXi hosts that will
correspond to your two physical sites. Hosts ESX1 and ESX2 represent Site A and Site B respectively.
Host ESX3 is in Site A and is used for management services.

90.2 Verify interface mappings.

In this step you will verify that the port groups available on the ESX hosts in each site are connected to
the corresponding interfaces on the Nexus 5000 access device. Recall that interconnecting links between
the two Nexus 5000s are either shutdown or not in use, so any interconnections need to go to the Nexus
7000s.

Server Port Group Virtual Switch Uplink
Port
VLAN Connecting
Device
Connecting
Ports
ESX1 VM-Client vSwitch1 vmnic0 131 N5K-1 E1/9
ESX1 Local Lan vSwitch1 vmnic0 24 N5K-1 E1/9
* ESX1 uses physical adapter vmnic 0 (port 1 on 10G CNA) as the physical uplink for vSwitch1 to N5K-1.
ESX2 VM-Client vSwitch1 vmnic1 131 N5K-2 E1/10
ESX2 Local Lan vSwitch1 vmnic1 24 N5K-2 E1/10
* ESX2 uses physical adapter vmnic 1 (port 2 on 10G CNA) as the physical uplink for vSwitch1 to N5K-2.
Note: Remember that only VLANs 131 and 151 have been configured to stretch across the OTV overlay
between the two sites. The VLAN 24 is only local to the two individual sites.


Step 91 VM-Client: Use Cisco Discovery Protocol (CDP) from within the VMware vSphere Client to verify the
physical adapter vmnic2 of the ESX host is connected to the sites 10G access device at port Eth1/9.

91.1 Identify the Virtual Switch vSwitch1. Click on the bubble icon ( ) on the right side of the
corresponding physical adapter vmnic0.
91.2 Verify that the active CNA adapter for ESX1 (vmnic0) is connected to the N5K-1.
Click on the bubble icon ( ) on the right side of the corresponding physical adapter vmnic0.

91.3 Verify that the active CNA adapter for ESX2 (vmnic0) is connected to the N5K-2.
Click on the bubble icon ( ) on the right side of the corresponding physical adapter vmnic1.

3
1
2

3
1
2
CONFIGURE VIRTUAL MACHINE TO PORT GROUP MAPPINGS OF LOCAL LAN

Step 92 Next we change the connectivity of the available virtual machines to the Local Lan port groups.

Your pod should already have two virtual machines titled Server 2003R2 and ClientXP.
92.1 Right click on the virtual machine Server 2003R2 and select Edit Settings from pop up menu.

92.2 Click on Network Adapter. Under Network label, select the Local Lan port group. Click OK.

92.3 Repeat the steps above for the Virtual Machine ClientXP

1
2

1
2
10 VMOTION ACROSS DATA CENTERS

Move running virtual machines from one physical server to another with no impact to end users. VMware
VMotion keeps your IT environment up and running, giving you unprecedented flexibility and availability to
meet the increasing demands of your business and end users.

VMotion relies on the availability of the storage volume from both source and target physical server. It then uses
a process which includes the recursive copying of the VMs memory to migrate the current state of a VM
across the physical hosts.

To do a VMotion, a network connection is used for transferring the state (CPU registers, memory, I/O, )
between the physical servers. A layer 3 connection can be used, however, VMware only supports layer 2
connectivity for this VMotion connection.

VMotion does not perform any changes to the VM, especially its layer 3 network settings. Thus, VM owners are
given the true impression of a virtual NIC with true Layer 2 connectivity. To maintain this impression it is
necessary that both source and target physical host are connected to the same Layer 2 domain. Otherwise,
network connectivity between the VMotioned VM and communication partners would drop.

With this in mind, Cisco Overlay Transport Virtualization (OTV) is the perfect match for enabling VMotion across
geographically dispersed data centers. This step of the lab guide will demonstrate how OTV enables VMotion
across the two sites of your pod.

Missing L2 connectivity across sites without OTV
Successful connectivity within same site
Successful VMotion across sites due to L2 connectivity with OTV

10.1 MISSING L2 CONNECTIVITY ACROSS SITES WITHOUT OTV

As you have verified in the previous lab steps both virtual machines reside in separate sites and are
connected to local site VLANs that do not share Layer 2 connectivity. At the same time, both VMs have
been configured on the same subnet and therefore require Layer 2 connectivity between each other for
successful communication.

In this lab step you will see that without OTV, these VMs are not able to communicate across sites, only
within the same site. A VMotion of a single VM to another site thereby breaks the VMs network
connectivity and has to be considered as a failed VMotion.

Step 93 Verify that the Virtual Machines do not have Layer connectivity across sites over the local-only VLANs:
93.1 Click on the Virtual Machine ClientXP. Click on the Open Console ( ) icon to connect to the
VMs desktop.
93.2 Within the Console of the VM, on the desktop, double-click on the PingServer icon.
93.3 This will start a continuous ping between the local ClientXP VM (10.1.131.33) and the Server
2003R2 VM (10.1.131.31)
93.4 Notice that Server 2003R2 is unreachable due to the lack of Layer 2 connectivity between the VMs

Note: Leave the continuous ping running and the Console window open for further lab steps.

1
2
10.2 SUCCESSFUL CONNECTIVITY WITHIN SAME SITE

To demonstrate successful connectivity between the two VMs when they reside in the same site you will
migrate one of the VMs so that they will reside in the same site. Once this has been accomplished you can
observe through the continuous ping from the VM Server 2003R2-Clone to the VM Server 2003R2 that
Layer 2 connectivity exists.

Step 94 Migrate (VMotion) the VM Server 2003R2 to site Site B:

94.1 Right-click on the Virtual Machine Server 2003R2 to open the Action menu for this VM.
94.2 Choose Migrate within the Action menu to start the VMotion process

94.3 Leave the default setting of Change host and click on Next.
94.4 Pick the host ESX2 as the target of the VMotion and click Next..

94.5 For vMotion Priority, leave the default setting of High Priority and click on Next
94.6 Verify the selected choices and click on Next to start the VMotion process.
94.7 Monitor the Console of the VM Server 2003R2 during the VMotion process.

94.8 When the VMotion process nears completion, network connectivity between the VM ClientXP
(10.1.131.32) and the VM Server 2003R2 (10.1.131.31) is established. Therefore the ping
between them succeeds.

1
2
1
10.3 SUCCESSFUL VMOTION ACROSS SITES DUE TO L2 CONNECTIVITY WITH OTV

In this step of the lab guide, you will connect both VMs to VLAN 20, which has been extended via OTV between
the two sites. You will verify that OTV is used to extend the Layer 2 domain across sites. This enables
connectivity between the two VMs when they reside in the same site, and when they reside in different sites. As
a result, OTV is used to successfully enable VMotion across data center sites.

Note: The vmotion vlan itself is extended over the WAN. However, since vmotion can technically work over
an IP boundary, we will test Layer 2 activities as well to show that there is no trickery here.

Step 95 Configure both Virtual Machines to use the port group VM-Client. As demonstrated in previous lab
steps, this port group uses a VLAN that has been extended between the two sites via OTV:
95.1 Click on the Virtual Machine Server 2003R2 to highlight this VM. Then perform a right-click to
open the Action menu for this VM. Choose Edit Settings within the Action menu to change the
virtual NIC settings of the VM

95.2 Choose Network Adapter 1 under Hardware. In the Network Connection area, change the
Network Label to VMTRAFFIC and confirm the settings with OK.

1
1
2
95.3 Verify that the port group for the VM Server 2003R2 has been changed to VMTraffic.

95.4 Repeat the steps above for the VM ClientXP.

You will lose network connectivity between the two VMs while one VM is connected to the port group
VM-Client and the other VM is still connected to Local LAN. This is due to the two port groups being
mapped to two different Layer 2 domains.
95.5 Verify that the VM Server 2003R2-Clone has Layer 2 network connectivity to the VM Server
2003R2 while both are connected to the port group VM-Client and reside within the same site.
95.6 Migrate (VMotion) the VM Server 2003R2 back to site Site A. During and after this migration
the VM ClientXP will still have connectivity to the VM Server 2003R2:
95.7 Click on the Virtual Machine Server 2003R2 to highlight this VM. Then perform a right-click to
open the Action menu for this VM.
95.8 Choose Migrate within the Action menu to start the VMotion process

95.9 Leave the default setting of Change host and click on Next.
95.10 Pick the host ESX1 as the target of the VMotion and click Next.

95.11 Leave the default setting of High Priority and click on Next.
95.12 Verify the selected choices and click on Next to start the VMotion process.
95.13 Monitor the Console of the VM ClientXP during the VMotion process.

1

1
2
1
Note: You will notice that while the VMotion is progressing, network connectivity between the VM
ClientXP (10.1.131.33) and the VM Server 2003R2 (10.1.131.31) remains active. Therefore the
ping between them succeeds.

95.14 Check on the local Nexus 7000 that MAC addresses of the remote VM servers were learned on the
local site and that ARP Table entries, mapping remote IPs and MACs, were cached successfully.

Your MAC addresses will be different depending on what vSphere assigns your VMs.

N7K-1-OTV-1A# show mac address-table
Legend:
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G - 0026.980d.6d42 static - F F sup-eth1(R)
* 151 0050.5670.e096 dynamic 0 F F Eth1/14
O 151 0050.5674.b27f dynamic 0 F F Overlay1
* 151 0050.567b.cdd7 dynamic 930 F F Eth1/14
* 211 0016.9dad.8447 dynamic 360 F F Eth1/14
O 211 0050.5676.bc47 dynamic 0 F F Overlay1
* 211 0050.567d.6c56 dynamic 420 F F Eth1/14
* 211 0050.567e.d107 dynamic 300 F F Eth1/14
* 211 02a0.9811.5474 dynamic 0 F F Eth1/14

If the Authoritative Edge Device (AED) is the local node, the remote MAC address will be learned
through the Overlay. If the Nexus 7000 is not the Authoritative Edge Device the remote MAC address will
be learned through the interconnection to the AED Node.

N7K-2-OTV-1B# show mac address-table
Legend:
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
G - 0026.980d.92c2 static - F F sup-eth1(R)
O 151 0050.5670.e096 dynamic 0 F F Overlay1
* 151 0050.5674.b27f dynamic 0 F F Eth1/16
O 151 0050.567b.cdd7 dynamic 0 F F Overlay1
O 211 0016.9dad.8447 dynamic 0 F F Overlay1
* 211 0050.5676.bc47 dynamic 0 F F Eth1/16
O 211 0050.567d.6c56 dynamic 0 F F Overlay1
O 211 0050.567e.d107 dynamic 0 F F Overlay1
O 211 02a0.9811.5474 dynamic 0 F F Overlay1

95.15 Lets check the ARP/ND cache

N7K-1-OTV-1A# show otv arp-nd-cache

Overlay Interface Overlay1
VLAN MAC Address Layer-3 Address Age Expires In
20 0050.56b6.0007 10.1.131.32 00:01:55 00:06:04

N7K-2-OTV-1B# show otv arp-nd-cache

Overlay Interface Overlay1
VLAN MAC Address Layer-3 Address Age Expires In
20 0050.56b6.0006 192.168.2.25 00:00:46 00:07:13

95.16 You can check reachability of remote MACs through the OTV route command.

N7K-1-OTV-1A# show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address Metric Uptime Owner Next-hop(s)
---- -------------- ------ -------- --------- -----------
20 0050.56b6.0000 42 03:37:33 overlay N7K-2-OTV-1B
20 0050.56b6.0006 1 00:08:34 site Ethernet1/14
20 0050.56b6.0007 42 00:30:10 overlay N7K-2-OTV-1B
23 0050.5672.b514 1 00:08:41 site Ethernet1/14
23 0050.5678.38a6 42 00:08:41 overlay N7K-2-OTV-1B

N7K-2-OTV-1B# show otv route

OTV Unicast MAC Routing Table For Overlay1

VLAN MAC-Address Metric Uptime Owner Next-hop(s)
---- -------------- ------ -------- --------- -----------
20 0050.56b6.0000 1 03:38:04 site Ethernet1/16
20 0050.56b6.0006 42 00:09:05 overlay N7K-1-OTV-1A
20 0050.56b6.0007 1 00:30:41 site Ethernet1/16
23 0050.5672.b514 42 00:09:11 overlay N7K-1-OTV-1A
23 0050.5678.38a6 1 00:09:12 site Ethernet1/16

Congratulations! You successfully migrate a VM across data center sites, while the VM remains
reachable via Layer 2 thanks to Cisco Overlay Transport Virtualization (OTV).

11 MIGRATE VM TO SAN ATTACHED STORAGE AND CONFIGURE VM DISKS

Complete this lab exercise to learn how to migrate a Virtual Machine from local storage to SAN attached
storage. We run our ESXi hypervisors on Cisco Unified Computing System C-Series Servers, powered by Intel
Xeon processors, providing industry-leading virtualization performance. Once migrated, you will configure
Virtual Machine networking and VM disks.
EXERCISE OBJECTIVE

In this exercise you will use VMware vSphere to migrate a Virtual Machine to SAN attached storage, configure
the Virtual Machine networking, and add VM disks. After completing these exercises you will be able to meet
these objectives:

Migrate a VM to SAN attached storage
Configure VM networking
Configure VM disks
Manage VM disks in the Virtual Machine Windows 2003 operating system

11.1 CLONE A VM TO SAN ATTACHED STORAGE

Step 96 Clone Server 2003 R2 VM to a different server and datastore. (optional)
96.1 Right-click on the Server-2003R2 VM and select Clone from the pop-up menu.

96.2 Name the VM Server 2003R2-Clone. Click on FlexPod_DC_1 datacenter. Then, click Next.
96.3 Select FlexPod_Mgmt for the cluster. Click Next.
96.4 Select ESX1 for the host. Click Next.
96.5 For Datastore, select the Netapp-SAN (FC shared storage). Click Next.
96.6 Click the Same format as source radio button, then click Next
96.7 Use the default settings. Click Next until you get to the final dialog box. Click Finish.
96.8 Wait for the Clone to complete.

1
2
11.2 CONFIGURE VIRTUAL MACHINE NETWORKING

Start the VMs, add My Computer icon to the desktop, change IP address and gateway, change the server name
and allow it to reboot.

Step 97 Configure Virtual Machine Networking
97.1 Click on Server 2003R2. Then, click on Power on icon from the toolbar.

97.2 Click on the Virtual Machine Console button ( ) in the toolbar, then click in the console window.
97.3 You should already be automatically logged on. If needed, press CTL-ALT-INSERT (instead of CTL-
ALT-DEL). Alternatively, select the VM menu > Guest > Send Ctrl+Alt+del to get to the windows
log on window. Authenticate with administrator/1234Qwer.

97.4 Change the Server name and IP address by double-clicking on the MakeMe Server1 shortcut. This
launches a batch file that changes the computer name to server1 and the IP address to
10.1.131.31. Allow the computer to restart.

1
2

3
1
2
97.5 After the server restarts, verify that the hostname is SERVER1 and the IP address is 10.1.131.31.
The background image should reflect this.

Note: To allow mouse focus out of the VM console window, press Crtl+Alt

3
1
2

4
1
Step 98 Repeat Step 97 on Server 2003R2-Clone.
IP address 10.1.131.32/24 GW =10.1.131.254
Computer name = server2

Step 99 Check that both VMs virtual nic settings are in the ESX hosts vSwitch0 and in the proper Port Group.
99.1 Select the ESX host (ESX1 (10.1.111.21) in this example), select Configuration tab, select
Networking under Hardware, select Virtual Switch tab and verify that the VM nic is in the Port
Group.

99.2 If the VM nic is not in the proper Port Group, select the VM (Server 2003R2 in this example), right-
click on it and select Edit Settings from the pop up menu.

99.3 Select the Network adapter, and change the Port Group under the Network Label drop-down.

3
1
2

4

1
2

1
2
11.3 MIGRATE A VM TO SAN ATTACHED STORAGE

Demonstrate the ability to migrate a VM to a different storage location and a different host. Use the vSphere
client to migrate Server 2003R2 VM located on host ESX1 datastore DS to host ESX2 datastore Netapp-SAN-
1.

Step 100 Migrate Server 2003R2 to a different host and datastore.
100.1 Right-click on Server 2003R2 VM and select Migrate from the pop-up menu.

100.2 Select Change both host and datastore radio box and click Next.

100.3 Select host ESX2 as the destination. Click Next.
100.4 Select Netapp-SAN-1 datastore, and then click Next.
100.5 Select Same format as source radio box, then click Next.
100.6 Click Finish.
100.7 Wait for the migration to finish.

1
2
1
Step 101 Verify that the VM is on ESX2.
101.1 Click on VM Server-2003R2. Then, click on Summary Tab. Note that the host is ESX2 and that the
Datastore is Netapp-SAN-1.

3
1
2

11.4 CONFIGURE VM DISKS (OPTIONAL)
Step 102 Add a VM disk (vmdk file) to each VM, format the disk, and verify that the disk is available to the VM.
102.1 Select Server 2003R2 VM. Right click on it and select Edit settings from the pop up menu.
102.2 Click Add.

102.3 Select Hard Disk. Click Next.

102.4 Select the Create a new virtual disk radio button, and then click Next.

102.5 Change the Disk Size to 3 GB, select the Specify a datastore radio button, and then click Browse.

1

1

1

3
1
2
102.6 Select the Netapp-SAN-1 datastore, then click OK. Back at the Create a Disk window, click Next.

102.7 Click Next on Advanced Options to accept the default values.
102.8 Click Finish. Then click OK to close the Add Hardware window.
102.9 Log into the VM.
102.10 Right-click on My Computer, select Manage.

102.11 Select Disk Management and click Next on the pop-up window.

102.12 Click Next to Initialize the new disk.
102.13 Click in the checkbox to select Disk 1 and click Next to Convert the disk to a dynamic disk.
102.14 Click Finish to start the disk initialization.

1

1
2

1
2
102.15 Right-click in the Disk1 Unallocated window and select New Volume from the pop-up menu. Go
through the wizard using the default settings for all of the settings.

102.16 Right-click in the New Volume and select Format. Use the default settings for the pop-up
windows. Close the Computer Management window.

102.17 Double-click on My Computer and verify that the Disk is available.

1
1

2
12 SUMMARY

In this lab you:

Installed and Configured Nexus 5010.
o Virtual Port Channel
o Fibre Channel, SAN Port Channel, FCoE
VSAN Databases and Zone
o FEX
Preprovision FEX
Configured MDS 9124.
o Fibre Channel Port Channel
Configured OTV and learned some of the aspects of OTV and its use case:
o Enables Layer 2 connectivity between data center sites
o Requires a Multicast enabled IP Core network between sites
o Can be used to enable VMware VMotion across sites.
Configured Vmware
o Added hosts to a cluster
o Added NFS SAN
o Performed vMotion and storage VMotion over OTV

12.1 FEEDBACK

We would like to improve this lab to better suit your needs. To do so, we need your feedback. Please take 5
minutes to complete the online feedback for this lab. We carefully read and consider your scores and
comments, and incorporate them into the content program

Just click on the link below and answer the online questionnaire.

Click here to take survey

Thank you!

13 APPENDIX A: COPYING SWITCH CONFIGURATIONS FROM A TFTP
SERVER

In the event that you want to reload the startup configuration to the Nexus 5000 and MDS switches, you may
copy completed configurations from the VC_SERVER using tftp.

Step 103 From the SSL Dashboard, log into the VC_SERVER using credentials administrator/1234Qwer.
103.1 Double-click the Tftpd32/64 icon on the desktop to start the tftp server.

103.2 Using the console from each switch, copy the appropriate file to running-config:
Cisco MDS9124
MDS9124# copy tftp://10.1.111.100/mds-base.cfg running-config
Connection to server Established. Copying Started.....
|
<snip>

N5K-1# copy tftp://10.1.111.100/n5k-1-base.cfg running-config
Enter vrf (If no input, current vrf 'default' is considered): management

TFTP get operation was successful
<snip>

N5K-2# copy tftp://10.1.111.100/n5k-2-Lab2 running-config vrf management

TFTP get operation was successful
<snip>
Note: You will have to run the copy twice due to features not active when the configuration is applied.

14 APPENDIX B: RECOVERING FROM THE LOADER PROMPT

This recover procedure should only be performed if loader issues occur:

In this task you will recover from the loader prompt and restore the switch configuration from the tFTP/FTP
server using the command line interface (CLI). The recovery procedure for the Nexus 5000 is slightly different
than the recovery procedure for the MDS 9124.

NEXUS 5000 RECOVERY WHEN THE KICKSTART AND SYSTEM FILES ARE ON BOOTFLASH
Cisco Nexus 5010 A or B - N5K-1 or N5K-2
Step 104 Use the directory command to determine if the kickstart and system files required for the Nexus 5000
to work are stored locally in bootflash. You will need these file names in the boot variables set for the
Nexus 5000.
loader> dir
bootflash:
lost+found
config.cfg
license_SSI14100CHE_4.lic
n5000-uk9-kickstart.5.0.2.N2.1.bin
n5000-uk9.5.0.2.N2.1.bin
<snip>
104.1 Use the boot command to boot the kickstart image:
loader> boot bootflash:n5000-uk9-kickstart.5.0.2.N2.1.bin
104.2 Use the load command to load the system file:
switch(boot)# load bootflash:n5000-uk9.5.0.2.N2.1.bin
104.3 Log in to the N5K:
N5K-1 login: admin
Password: 1234Qwer
104.4 Set the boot system and kickstart variables:
N5K-1# conf t
N5K-1(config)# boot system bootflash:n5000-uk9.5.0.2.N2.1.bin
N5K-1(config)# boot kickstart bootflash:n5000-uk9-kickstart.5.0.2.N2.1.bin
N5K-1(config)# copy run st
[########################################] 100%

NEXUS 5000 RECOVERY WHEN THE KICKSTART AND SYSTEM FILES ARE NOT ON BOOTFLASH
Cisco Nexus 5010 A or B - N5K-1 or N5K-2
Step 105 Use the set command to assign an IP address to the management interface:
loader> set ip 10.1.111.1 255.255.255.0
105.1 Boot the kickstart image from the tftp server:
loader> boot tftp://10.1.111.100/n5000-uk9-kickstart.5.0.2.N2.1.bin
105.2 Once the kickstart is booted, configure the IP address on the management interface
switch(boot)# conf t
switch(boot)(config)# int mgmt0
switch(boot)(config-if)# ip address 10.1.111.1 255.255.255.0
switch(boot)(config-if)# no shut
switch(boot)(config-if)# end
105.3 Copy the kickstart and system files from the tftp server to bootflash:
switch(boot)# copy tftp://10.1.111.100/n5000-uk9-kickstart.5.0.2.N2.1.bin bootflash:
switch(boot)# copy tftp://10.1.111.100/n5000-uk9.5.0.2.N2.1.bin bootflash:
105.4 Load the system file:
switch(boot)# load bootflash:n5000-uk9.5.0.2.N2.1.bin
105.5 Log into the switch:
Nexus 5000 Switch
N5K-1 login: admin
Password: 1234Qwer
N5K-1# conf t
N5K-1(config)# boot system bootflash:n5000-uk9.5.0.2.N2.1.bin
N5K-1(config)# boot kickstart bootflash:n5000-uk9-kickstart.5.0.2.N2.1.bin
105.7 Save the configuration:
N5K-1(config)# copy run start
[########################################] 100%

MDS9124 RECOVERY WHEN THE KICKSTART AND SYSTEM FILES ARE ON BOOTFLASH
Cisco MDS9124
Step 106 Complete these steps on the MDS9124
106.1 Use the directory command to view the files stored on bootflash.
loader> dir

bootflash:

12288 lost+found/
2296 mts.log
18723840 m9100-s2ek9-kickstart-mz.5.0.1a.bin
56219997 m9100-s2ek9-mz.5.0.1a.bin
2995 config.cfg
106.2 Use the boot command to boot the kickstart image:
loader> boot bootflash:m9100-s2ek9-kickstart-mz.5.0.1a.bin
106.3 Load the system image:
switch(boot)# load bootflash:m9100-s2ek9-mz.5.0.1a.bin
106.4 Log into the switch:
MDS9124 login: admin
Password: 1234Qwer
MDS9124# conf t
MDS9124(config)# boot system bootflash:m9100-s2ek9-mz.5.0.1a.bin
MDS9124(config)# boot kickstart bootflash:m9100-s2ek9-kickstart-mz.5.0.1a.bin
MDS9124(config)# end
MDS9124# copy run st
[########################################] 100%

MDS9124 RECOVERY WHEN THE KICKSTART AND SYSTEM FILES ARE NOT ON BOOTFLASH
Step 107 Complete these steps on the MDS9124
107.1 Use the network command to set the ip address and mask for the management interface:
loader> network --ip=10.1.111.40 --nm=255.255.255.0
107.2 Boot the kickstart image from the tftp server:
loader> boot tftp://10.1.111.100/m9100-s2ek9-kickstart-mz.5.0.1a.bin
107.3 Configure the IP address on the management interface:
switch(boot)# conf t
switch(boot)(config)# int mgmt0
switch(boot)(config-if)# ip address 10.1.111.40 255.255.255.0
switch(boot)(config-if)# no shut
switch(boot)(config-if)# end
107.4 Copy the kickstart and system files from tftp to bootflash:
switch(boot)# copy tftp://10.1.111.100/m9100-s2ek9-kickstart-mz.5.0.1a.bin bootflash:
switch(boot)# copy tftp://10.1.111.100/m9100-s2ek9-mz.5.0.1a.bin bootflash:
107.5 Load the system file from bootflash:
switch(boot)# load bootflash:m9100-s2ek9-mz.5.0.1a.bin
107.6 Log into the MDS9124:
MDS9124 login: admin
Password: 1234Qwer
MDS9124# conf t
MDS9124(config)# boot system bootflash:m9100-s2ek9-mz.5.0.1a.bin
MDS9124(config)# boot kickstart bootflash:m9100-s2ek9-kickstart-mz.5.0.1a.bin
MDS9124(config)# copy run start
[########################################] 100%
MDS9124(config)#

15 NETAPP FAS2020A DEPLOYMENT PROCEDURE: PART 1
The following section provides a detailed procedure for configuring the NetApp FAS2020 A for use in a FlexPod
environment. These steps should be followed precisely. Failure to do so could result in an improper
configuration.

15.1 NETAPP ASSIGNING DISKS

Step 108 Assign controller disk ownership - DONE
Controller A - NTAP1-A
108.1 During controller boot, when prompted to Press CTRL-C for special boot menu, press CTRL-C.
108.2 At the menu prompt, choose option 5 for Maintenance Mode.
108.3 Type Yes when prompted with Continue to boot?
108.4 Type disk show.
108.5 Reference the Local System ID: value for the following disk assignment.
Note: Half the total number of disks in the environment will be assigned to this controller and half to the
other controller. Divide the number of disks in half and use the result in the following command for
the <# of disks>.
108.6 Type disk assign -n <# of disks>.
108.7 Type halt to reboot the controller.

Controller B - NTAP1-B
108.8 During controller boot, when prompted to Press CTRL-C for special boot menu, press CTRL-C.
108.9 At the menu prompt, choose option 5 for Maintenance Mode.
108.10 Type Yes when prompted with Continue to boot?
108.11 Type disk show.
108.12 Reference the Local System ID: value for the following disk assignment.
Note: Half the total number of disks in the environment will be assigned to this controller and half to the
other controller. Divide the number of disks in half and use the result in the following command for
the <# of disks>.
108.13 Type disk assign -n <# of disks>.
108.14 Type halt to reboot the controller.

108.15 Type disk show on the command line for each controller to generate a list of disks owned by
each respective controller.
NTAP1-A> disk show
DISK OWNER POOL SERIAL NUMBER
------------ ------------- ----- -------------
0c.00.3 storage (135053985) Pool0 JLVD3HRC
0c.00.1 storage (135053985) Pool0 JLVD2NBC
0c.00.4 storage (135053985) Pool0 JLVD3KPC
0c.00.5 storage (135053985) Pool0 JLVBZW1C
0c.00.2 storage (135053985) Pool0 JLVD3HTC
0c.00.0 storage (135053985) Pool0 JLVBZ9ZC

15.2 NETAPP ONTAP INSTALLATION

Step 109 Upgrading from Data ONTAP 7.3.1 to 7.3.5 DONE/INSTRUCTOR
Duration: Up to 75 minutes
Note: This step is not necessary if Data ONTAP 7.3.5 is already installed on your storage controllers.
Duration: 60-75 minutes
109.1 From the LOADER> prompt, configure an interface to netboot the controller.
ifconfig Incomplete addr=Incomplete -mask=Incomplete -gw=Incomplete
109.2 After the netboot interface is configured, netboot from the 7.3.5 image.
netboot Incomplete
109.3 When prompted, press Ctrl+C to enter the special boot menu.
109.4 Select option 4a, Same as option 4, but create a flexible root volume.
109.5 The installer asks if you want to zero the disks and install a new file system. Answer y.
109.6 A warning displays that this will erase all of the data on the disks. Answer y if you are sure this is
what you want to do.

Note: The initialization and creation of root volume can take up to 75 minutes or more to complete
depending on the number of disks attached.

109.7 To verify successful booting of the Data ONTAP installer, check to see if you are presented with the
setup wizard for Data ONTAP. It should prompt for a hostname.

15.3 NETAPP INITIAL SETUP

Step 110 Setting up Data ONTAP 7.3.5 - DONE
110.1 After the disk initialization and the creation of the root volume, Data ONTAP setup begins.
110.2 Enter NTAP1-A for the hostname of the storage system.
110.3 Answer n for setting up IPv6.
110.4 Answer y for setting up virtual network interfaces.
110.5 Enter 1 for the number of virtual interfaces to configure.
110.6 Name the interface ifgrp1.
110.7 Enter l to specify the interface as LACP.
110.8 Select i for IP based load balancing.
110.9 Enter 2 for the number of links for ifgrp1.
110.10 Enter e2a for the name of the first link.
110.11 Enter e2b for the name of the second link.
110.12 Press Enter when prompted for an IP address for ifgrp1 to accept the blank IP address.
110.13 Answer y for should virtual interface ifgrp1 take over a partner virtual interface during
failover?
Note: You might receive a message saying that the cluster failover is not yet licensed. That is fine, because
we will license it later.
110.14 Enter ifgrp1 for the partner interface to be taken over by ifgrp1.
110.15 Enter 10.1.111.151 for the IP address of the management interface, e0M.
110.16 Enter 255.255.255.0 as the subnet mask for e0M.
110.17 Enter y for the question Should interface e0M take over a partner IP address during failover?
110.18 Enter e0M for the partner interface to be taken over during failover.
110.19 Press Enter to accept the default flow control of full.
110.20 Press Enter to accept the blank IP address for e0a.
110.21 Answer n to have the interface not takeover a partner IP address during failover.
110.22 Press Enter to accept the blank IP address for e0b.
110.23 Answer n to have the interface not takeover a partner IP address during failover.

110.24 Answer n to continuing setup through the Web interface.

110.25 Enter 10.1.111.254 as the IP address for the default gateway for the storage system.
110.26 Enter 10.1.111.100 as the IP address for the administration host.
110.27 Enter Nevada as the location for the storage system.
110.28 Answer y to enable DNS resolution.
110.29 Enter dcvlabs.lab as the DNS domain name.
110.30 Enter 10.1.111.10 as the IP address for the first nameserver.
110.31 Answer n to finish entering DNS servers, or answer y to add up to two more DNS servers.
110.32 Answer n for running the NIS client.
110.33 Answer y to configuring the SP LAN interface.
110.34 Answer n to setting up DHCP on the SP LAN interface.
110.35 Enter Incomplete as the IP address for the SP LAN interface.
110.36 Enter 255.255.255.0 as the subnet mask for the SP LAN interface.
110.37 Enter Incomplete as the IP address for the default gateway for the SP LAN interface.
110.38 Enter Incomplete Incomplete as the name and IP address for the mail host to receive SP
messages and Auto Support.
110.39 Answer y to configuring the shelf alternate control path management interface.
110.40 Accept the default interface for the ACP management.
110.41 Accept the default domain and subnet mask for the ACP interface.
110.42 After these steps are completed, the controller should be at the command line prompt.
110.43 Type reboot.

Please enter the new hostname []: NTAP1-A
Do you want to enable IPv6? [n]: n
Do you want to configure virtual network interfaces? [n]: y
Number of virtual interfaces to configure? [0] 1
Name of virtual interface #1 []: ifgrp1
Is ifgrp1 a single [s], multi [m] or a lacp [l] virtual interface? [m] l
Is ifgrp1 to use IP based [i], MAC based [m], Round-robin based [r] or Port based [p
] load balancing? [i] i
Number of links for ifgrp1? [0] 2
Name of link #1 for ifgrp1 []: e0a
Name of link #2 for ifgrp1 []: e0b
Please enter the IP address for Network Interface ifgrp1 []:
No IP address specified. Please set an IP address.
Please enter the IP address for Network Interface ifgrp1 []:
No IP address specified. Please set an IP address.
Please enter the IP address for Network Interface ifgrp1 []: 10.1.1.151
Please enter the netmask for Network Interface ifgrp1 [255.255.255.0]: 255.255.255.0
Please enter media type for ifgrp1 {100tx-fd, tp-fd, 100tx, tp, auto (10/100/1000)}
[auto]: auto
Would you like to continue setup through the web interface? [n]: n
Please enter the name or IP address of the IPv4 default gateway: 10.1.1.254
The administration host is given root access to the filer's
/etc files for system administration. To allow /etc root access
to all NFS clients enter RETURN below.
Please enter the name or IP address of the administration host: 10.1.1.10
Where is the filer located? []: Nevada
Do you want to run DNS resolver? [n]: y
Please enter DNS domain name []: dcvlabs.com
You may enter up to 3 nameservers
Please enter the IP address for first nameserver []: 10.1.1.10
Do you want another nameserver? [n]:
Do you want to run NIS client? [n]: n
This system will send event messages and weekly reports to NetApp Technical Suppor
t. To disable this feature, enter "options autosupport.support.enable off" within
24 hours. Enabling Autosupport can significantly speed problem determination and r
esolution should a problem occur on your system. For further information on Autosu
pport, please see: http://now.netapp.com/autosupport/
Press the return key to continue.

The Baseboard Management Controller (BMC) provides remote management capab
ilities
including console redirection, logging and power control.
It also extends autosupport by sending down filer event alerts.

Would you like to configure the BMC [y]: y
Would you like to enable DHCP on the BMC LAN interface [y]: n
Please enter the IP address for the BMC [0.0.0.0]: 10.1.1.152
Please enter the netmask for the BMC [0.0.0.0]: 255.255.255.0
Please enter the IP address for the BMC Gateway [0.0.0.0]: 10.1.1.254
Please enter gratuitous ARP Interval for the BMC [10 sec (max 60)]:

The mail host is required by your system to enable BMC to send
ASUP message when filer is down

Please enter the name or IP address of the mail host [mailhost]:
You may use the autosupport options to configure alert destinations.
The initial aggregate currently contains 3 disks; you may add more
disks to it later using the "aggr add" command.
Now apply the appropriate licenses to the system and install
the system files (supplied on the Data ONTAP CD-ROM or downloaded
from the NOW site) from a UNIX or Windows host. When you are
finished, type "download" to install the boot image and
"reboot" to start using the system.

110.44 To verify the successful setup of Data ONTAP 7.3.5, make sure that the terminal prompt is
available and check the settings that you entered in the setup wizard.

Step 111 Installing Data ONTAP to the onboard flash storage DONE/INSTRUCTOR
Duration: 2 minutes
Note: For this step, you will need a web server to host your ONTAP installation file.

111.1 Install the Data ONTAP image to the onboard flash device.
software update Incomplete
111.2 After this is complete, type download and press Enter to download the software to the flash
device.
Controller B - NTAP1-B
111.3 Install the Data ONTAP image to the onboard flash device
software update Incomplete
111.4 After this is complete, type download and press Enter to download the software to the flash
device.
111.5 Verify that the software was downloaded successfully by entering software list on the command
line and verifying that the Data ONTAP zip file is present.
Netapp1> software list
7351_setup_e.exe

Step 112 Installing required licenses
Duration: 3 minutes
112.1 Install the necessary Data ONTAP licenses.
license add var_ntap_cluster_lic var_ntap_fcp_lic var_ntap_flash_cache_lic
var_ntap_nearstore_option_lic var_ntap_a_sis_lic var_ntap_nfs_lic var_ntap_multistore_lic
var_ntap_flexclone_lic

112.2 To verify that the licenses installed correctly, enter the command license on the command line
and verify that the licenses listed above are active.


Step 113 Start FCP service and make sure of proper FC port configuration. DONE/INSTRUCTOR
Duration: 3 minutes
On both controllers - NTAP1-A and NTAP1-B
113.1 Start fcp and verify status.
NTAP1-A> fcp start
Fri May 14 06:48:57 GMT [fcp.service.startup:info]: FCP service startup
NTAP1-A> fcp status
FCP service is running.
113.2 The fcadmin config command confirms that our adapters are configured as targets
NTAP1-A> fcadmin config

Local
Adapter Type State Status
---------------------------------------------------
0c target CONFIGURED online
0d target CONFIGURED online
113.3 If either FC port 0c and 0d is listed as initiator, use the following command to change its
status to target
fcadmin config t target <port>
113.4 Re-run the fcadmin config: both ports should now either state initiator or (Pending)
initiator.
113.5 Reboot the storage controller to enable the cluster feature and also to enable the FC ports as
target ports as necessary.

15.4 NETAPP - AGGREGATES AND VOLUMES

Step 114 Creating the data aggregate aggr1. DONE/INSTRUCTOR

Best Practice
Use RAID-DP, the NetApp high-performance implementation of RAID 6, for better data protection on all
RAID groups that store virtual disks for the Hyper-V VMs. Data aggregates should have a RAID group size
of no less than 12. A NetApp best practice is to create as large an aggregate as possible.

For the example below, lets assume we have 24 drives. 12 Drives are assigned to each controller. 3 of the 12 is
assigned to aggr0 for the root volume, so that leaves us with 9 drives for aggr1. Note: Since this is a lab setup we
are reserving no disks for spares.

114.1 Create aggr1 on the storage controller.
aggr create aggr1 9

This command usually finishes quickly. Depending on the state of each disk, some or all of the disks might
need to be zeroed to be added to the aggregate. This might take up to 60 minutes to complete.

114.2 Verify that the aggregate was created successfully.
NTAP1-A> aggr status
Aggr State Status Options
aggr1 online raid_dp, aggr
32-bit
aggr0 online raid_dp, aggr root
32-bit

NetApp RAID-DP is an advanced RAID technology that provides the default RAID level on all storage
systems. RAID-DP protects against the simultaneous loss of two drives in a single RAID group. It is very
economical to deploy; the overhead with default RAID groups is a mere 12.5%. This level of resiliency and
storage efficiency makes data residing on RAID-DP safer than data residing on RAID 5 and more cost
effective than RAID 10.
15.5 NETAPP NETWORK & SECURITY
CREATE VLAN INTERFACES

Step 115 Enabling 802.1q VLAN trunking and adding the NFS VLAN. - DONE/INSTRUCTOR
Duration: 5 minutes

Since we grouped our two 10 Gig interfaces into one interface group, we will enable VLAN trunking to create
multiple logical interfaces for our vfilers.

115.1 Type vlan create ifgrp1 211 to enable 802.1q VLAN trunking on ifgrp1 and to add VLAN 211
for nfs traffic.
vlan create ifgrp1 211
## The following are additional VLANs for our LAB
vlan add ifgrp1 111
115.2 Type wrfile a /etc/rc vlan create ifgrp1 211.
wrfile a /etc/rc vlan create ifgrp1 211 212 213
115.3 Type ifconfig ifgrp1-211 mtusize 9000.
ifconfig ifgrp1-111 mtusize 9000
ifconfig ifgrp1-211 mtusize 9000

115.4 Type wrfile -a /etc/rc ifconfig ifgrp1-211 mtusize 9000.
!!! The following are for our LAB
wrfile -a /etc/rc ifconfig ifgrp1-211 mtusize 9000
115.5 Type rdfile /etc/rc and verify that the commands from the previous steps are in the file
correctly.
Netapp1> rdfile /etc/rc
#Regenerated by registry Thu Apr 21 06:36:34 GMT 2011
#Auto-generated by Setup Wizard Mon Oct 18 17:04:15 GMT 2010
vif create multi ifgrp1 -b ip e0b
ifconfig e0a `hostname`-e0a netmask 255.255.255.0 mediatype auto mtusize 1500 wins
flowcontrol none
ifconfig e0b `hostname`-e0b netmask 255.255.255.0 mediatype auto mtusize 1500 wins
flowcontrol none
ifconfig ifgrp1 `hostname`-ifgrp1 netmask 255.255.255.0 mtusize 9000
route add default n 1
routed on

savecore
options dns.enable off
options nis.enable off

115.6 Verify that in the output of the command ifconfig -a the interface ifgrp1-211 shows up.


Step 116 Hardening storage system logins and security. - DONE
Duration: 5 minutes
116.1 Type passwd to change the password for the root user.
116.2 Enter the new root password of 1234Qwer twice as prompted.
116.3 Type secureadmin setup ssh to enable ssh on the storage controller.
116.4 Accept the default values for ssh1.x protocol.
116.5 Enter 1024 for ssh2 protocol.
116.6 Enter yes if the information specified is correct and to create the ssh keys.
NTAP1-A> secureadmin setup ssh
SSH Setup
<snip>

Please enter the size of host key for ssh1.x protocol [768] :768
Please enter the size of server key for ssh1.x protocol [512] :512
Please enter the size of host keys for ssh2.0 protocol [768] :1024

You have specified these parameters:
host key size = 768 bits
server key size = 512 bits
host key size for ssh2.0 protocol = 1024 bits
Is this correct? [yes] yes

After Setup is finished the SSH server will start automatically.
116.7 Disable telnet on the storage controller.
NTAP1-A> options telnet.enable off
116.8 Enable ssl on the storage controller. Type secureadmin setup ssl.
116.9 Enter country name code: US, state or province name: CA, locality name: San Jose,
organization name: Cisco, and organization unit name: WWPO.
116.10 Enter NTAP1-A.dcvlabs.lab as the fully qualified domain name of the storage system.
116.11 Enter pephan@cisco.com as the administrators e-mail address.
116.12 Accept the default for days until the certificate expires.
116.13 Enter 1024 for the ssl key length.
NTAP1-A> secureadmin setup ssl
Country Name (2 letter code) [US]: US
State or Province Name (full name) [California]: CA
Locality Name (city, town, etc.) [Santa Clara]: San Jose
Organization Name (company) [Your Company]: Cisco
Organization Unit Name (division): WWPO
Common Name (fully qualified domain name) [NTAP1-A.dcvlabs.com]: NTAP1-A.dcvlabs.lab
Administrator email: pephan@cisco.com
Days until expires [5475] :5475
Key length (bits) [512] :1024
Thu May 13 22:12:07 GMT [secureadmin.ssl.setup.success:info]: Starting SSL with new certificate.
116.14 Disable http access to the storage system.
NTAP1-A> options httpd.admin.enable off
116.15 Verify that the root password has been setup by trying to log into the controller with the new
credentials. To verify that telnet is disabled, when you try to access the controller by telnet, it
should not connect. To verify that http access has been disabled, you should not be able to access
FilerView through http but rather through https.


Step 117 Create SNMP requests role and assign SNMP login privileges. Duration: 3 minutes
On both controller A and B - NTAP1-A and NTAP1-B
117.1 Execute the following command:
useradmin role add snmpv3role -a login-snmp
117.2 To verify, execute the useradmin role list on each of the storage controllers.

Step 118 Create SNMP management group and assign SNMP request role to it. Duration: 3 minutes
useradmin group add snmpv3group -r snmpv3role

Step 119 Create SNMP user and assign it to SNMP management group. Duration: 3 minutes
useradmin user add Incomplete -g snmpv3group
Note: You will be prompted for a password after creating the user. Use 1234Qwer when prompted

Step 120 Enable SNMP on the storage controllers. Duration: 3 minutes
120.1 Execute the following command: options snmp.enable on.
120.2 To verify, execute the command options snmp.enable on each of the storage controllers.
Netapp1> options snmp.enable
snmp.enable on

Step 121 Delete SNMP v1 communities from the storage controllers.
Duration: 3 minutes
121.1 Execute the following command: snmp community delete all.
Netapp1> snmp community
ro public
Netapp1> snmp community delete all
121.2 To verify, execute the command snmp community on each of the storage controllers.
Netapp1> snmp community
Netapp1>

Step 122 Set SNMP contact, location, and trap destinations for each of the storage controllers
Duration: 6 minutes
122.1 Execute the following commands:
snmp contact pephan@cisco.com
snmp location Nevada
snmp traphost add ntapmgmt.dcvlabs.lab
snmp traphost add snmp_trap_dest??
122.2 To verify, execute the command snmp on each of the storage controllers.
Netapp1> snmp
contact:
pephan@cisco.com
location:
TNI
authtrap:
0
init:
0
traphosts:
10.1.111.10 (10.1.111.10) <10.1.111.10>
community:

Step 123 Reinitialize SNMP on the storage controllers. Duration: 3 minutes
123.1 Execute the following command snmp init 1.
123.2 No verification needed.

15.6 NETAPP - VOLUMES

Step 124 Creating the necessary infrastructure volumes. - DONE/INSTRUCTOR
Duration: 5 minutes

In this step, we create volumes. A volume for VM datastore and a volume for VM swap space. Since this is a lab,
we will turn off automatic snapshots and we will not be reserving any space for snapshots.

124.1 Create a root volume for the infrastructure vFiler unit.
vol create VDI_VFILER211_ROOT -s none aggr1 20m
124.2 Create the volume that will later be exported to the ESXi servers as an NFS datastore.
vol create VDI_VFILER1_DS -s none aggr1 200g
124.3 Set the Snapshot reservation to 0% for this volume. Disable automatic snapshot option for this
volume.
snap reserve VDI_VFILER1_DS 0
vol options VDI_VFILER1_DS nosnap on
124.4 Create the volume that will hold the ESXi boot LUNs for each server.
vol create ESX_BOOT_A -s none aggr1 20g
vol create ESX1_BOOT_A -s none aggr1 20g
124.5 Set the Snapshot reservation to 0% for this volume. Disable automatic snapshot option for this
volume.
snap reserve ESX1_BOOT_A 0
vol options ESX1_BOOT_A nosnap on

Step 125 Creating a virtual swap file volume. - DONE/INSTRUCTOR
Duration: 3 minutes

ESX servers create a VMkernel swap or vswap file for every running VM. The sizes of these files are considerable;
by default, the vswap is equal to the amount of memory configured for each VM. Because this data is transient
in nature and is not required to recover a VM from either a backup copy or by using Site Recovery Manager,
NetApp recommends relocating the VMkernel swap file for every virtual machine from the VM home directory
to a datastore on a separate NetApp volume dedicated to storing VMkernel swap files. For more information,
refer to TR-3749: NetApp and VMware vSphere Storage Best Practices and vSphere Virtual Machine
Administration Guide.

125.1 Create the volume that will later be exported to the ESXi servers as an NFS datastore.
vol create VDI_SWAP -s none aggr1 20g
Note: This volume will be used to store VM swap files. Since swap files are temporary they do not need
snapshots or deduplications.
125.2 Disable the Snapshot schedule and set the Snapshot reservation to 0% for this volume. Disable
automatic snapshot option for this volume.
snap sched VDI_SWAP 0 0 0
snap reserve VDI_SWAP 0
vol options VDI_SWAP nosnap on

Verification
NTAP1-A> snap sched VDI_SWAP
Volume VDI_SWAP: 0 0 0
NTAP1-A> vol options VDI_SWAP
nosnap=on, nosnapdir=off, minra=off, no_atime_update=off, nvfail=off,


Step 126 Setup Deduplication.
Duration: 5 minutes

NetApp deduplication saves space on primary storage by removing redundant copies of blocks within a
volume. This process is transparent to the application and can be enabled and disabled on the fly. In a Citrix
XenDesktop environment, deduplication provides great value when we consider that all users in theenvironment
have their own user data either on the user data disk (for persistent desktops) and/or CIFS home directories
(nonpersistent desktops). In many environments, user data is duplicated multiple times as various identical
copies and versions of documents and files are saved. For more information, refer to NetApp TR-3505: NetApp
Deduplication for FAS, Deployment and Implementation Guide.

126.1 Enable deduplication on the infrastructure and boot volumes and set them to run every day at
12:00 a.m.
sis on /vol/VDI_VFILER1_DS
sis on /vol/ESX1_BOOT_A
sis on /vol/vol1
sis config -s 0@sun-sat /vol/VDI_VFILER1_DS
sis config -s 0@mon,tue,wed,thu,fri,sat,sun /vol/ESX1_BOOT_A
sis config -s 0@sun-sat /vol/vol1
126.2 The following command can be used to start processing existing data.
sis start -s /vol/VFILER1_DS

Step 127 Verification
127.1 Monitor the status of the dedupe operation:
sis status

Path State Status Progress
/vol/ESX1_BOOT_A Enabled Idle Idle for 00:01:53
<snip>
127.2 Verify the scheduling of the dedupe operations:
NTAP1-A> sis config
Path Schedule Minimum Blocks Shared
<snip>
/vol/ESX3_BOOT_A 0@mon,tue,wed,thu,fri,sat,sun
/vol/VFILER1_DS 0@sun-sat
127.3 View the space saving stats from the dedupe operation.
NTAP1-A> df -s
Filesystem used saved %saved
<snip>
/vol/INFRA_DS_1/ 156 0 0%
/vol/VMHOST_BOOT_A/ 136 0 0%

127.4 Verify that the volumes were created correctly.
NTAP1-A> vol status
Volume State Status Options
ESX1_BOOT_A online raid_dp, flex guarantee=none
sis
sis
sis
VFILER1_ROOT online raid_dp, flex guarantee=none
INFRA_SWAP online raid_dp, flex nosnap=on, guarantee=none
VFILER1_DS online raid_dp, flex guarantee=none
sis

Here are the LAB INSTRUCTOR commands for enabling deduplication for all the lab volumes.
sis on /vol/LAB_VFILER1_DS
sis config -s 0@mon,tue,wed,thu,fri,sat,sun /vol/LAB_VFILER1_DS

sis on /vol/INFRA_DS_XEN
sis config -s 0@mon,tue,wed,thu,fri,sat,sun /vol/INFRA_DS_XEN

15.7 NETAPP IP SPACE AND MULTISTORE

Step 128 Create the infrastructure IP space
Duration: 5 minutes

In this step we will create secure IP space (logical routing table specific for each vfiler). Each IP Space provides an
individual IP routing table per vFiler unit. The association between a VLAN interface and a vFiler unit allows all
packets to and from the specific vFiler unit to be tagged with the appropriate VLAN ID specific to that VLAN
interface. IP spaces are similar to the concept of VRFs in the Cisco world.

128.1 Type ipspace create ips-vfiler211 to create the IP space for the vdi_vfiler_211 vFiler unit.
NTAP1-A> ipspace create ips-vfiler111
NTAP1-A> ipspace create ips-vfiler211
128.2 Assign interfaces to our IP spaces using the command ipspace assign vdi_vfiler_211 ifgrp1-211.
NTAP1-A> ipspace assign ips-vfiler111 ifgrp1-111
NTAP1-A> ipspace assign ips-vfiler211 ifgrp1-211

128.3 Verify that the IP space was created and assigned successfully by issuing the command ipspace
list and verifying that the ipspace and interface assigned to it are listed.
NTAP1-A> ipspace list
Number of ipspaces configured: 18
default-ipspace (e0M e0P e0a e0b losk ifgrp1)
vfiler1 (no interfaces)
ips-vfiler2 (ifgrp1-212)


Step 129 Creating the infrastructure vFiler units DONE/INSTRUCTOR
Duration: 5 minutes

129.1 Create a vfiler called vfiler_1. Assign it to IP Space ips-vfiler1 and give it an IP address of
10.1.211.151. Assign /vol/INFRA_ROOT to it.
vfiler create vdi_vfiler_211 -s ips-vfiler211 -i 10.1.211.151 /vol/VDI_VFILER211_ROOT
Note: You can only create one vfiler at a time. The commands below should NOT be copied and pasted all
at once.
129.2 Accept the IP address that you specified on the command line by pressing Enter.
129.3 Type ifgrp1-211 for the interface to assign to the vFiler unit.
129.4 Press Enter to accept the default subnet mask.
129.5 If necessary, type 10.1.111.10 as the IP address of the administration host for the vFiler unit.
129.6 Enter n for running a DNS resolver.
129.7 Enter n for running an NIS client.
129.8 Enter a password for the vFiler unit.
129.9 Enter the same password a second time to confirm.
129.10 Enter y for setting up CIFS.
NTAP1-A> vfiler create vdi_vfiler_211 -s ips-vfiler211 -i 10.1.211.151 /vol/VDI_VFILER211_ROOT
<snip>
Setting up vfiler vdi_vfiler_211
Configure vfiler IP address 10.1.211.151? [y]: y
Interface to assign this address to {ifgrp1-211}: ifgrp1-211
Netmask to use: [255.255.255.0]: 255.255.255.0
Please enter the name or IP address of the administration host: 10.1.111.10
Do you want to run DNS resolver? [n]: n
Do you want to run NIS client? [n]: n
New password: 1234Qwerty
Retype new password: 1234Qwerty
Do you want to setup CIFS? [y]: n

129.11 To verify that the vFiler unit was created successfully, enter the command vfiler status and verify
that the vFiler unit is listed and that its status is running.
NTAP1-A> vfiler status
vfiler0 running
lab-vfiler1 running
lab-vfiler2 running
lab-vfiler3 running


Step 130 Mapping the necessary infrastructure volumes to the infrastructure vFiler unit
DONE/INSTRUCTOR
Duration: 5 minutes

In this step we are going to add a datastore volume and a swap volume to each vfiler. This will provide each lab
pod the required volumes to support a virtualization infrastructure.

130.1 Type vfiler add vdi_vfiler_211 /vol/VDI_VFILER1_DS. The add subcommand adds the
specified paths to an existing vfiler.
NTAP1-A> vfiler add vdi_vfiler_211 /vol/VDI_SWAP /vol/VDI_VFILER1_DS
<snip>
Mon Sep 26 11:00:26 PDT [cmds.vfiler.path.move:notice]: Path /vol/VDI_SWAP was mov
ed to vFiler unit "vdi_vfiler_211".
Mon Sep 26 11:00:26 PDT [cmds.vfiler.path.move:notice]: Path /vol/VDI_VFILER1_DS w
as moved to vFiler unit "vdi_vfiler_211".
130.2 To verify that the volumes were assigned correctly, enter the command vfiler run
infrastructure_vfiler vol status and then check that the two volumes are listed in the output.
NTAP1-A> vfiler run vdi_vfiler_211 vol status

===== vdi_vfiler_211
Volume State Status Options
VDI_VFILER1_DS online raid_dp, flex nosnap=on, fs_size_fixed=on,
guarantee=none
VDI_SWAP online raid_dp, flex nosnap=on, guarantee=none,
fractional_reserve=0
VDI_VFILER211_ROOT online raid_dp, flex guarantee=none,
fractional_reserve=0

15.8 NETAPP NFS

Step 131 Exporting the infrastructure volumes to the ESXi servers over NFS
DONE/INSTRUCTOR
Duration: 5 minutes

131.1 Type vfiler context infrastructure_vfiler_1 to enter the context or command line for the
infrastructure vFiler unit.
NTAP1-A> vfiler context vdi_vfiler_211
vdi_vfiler_211@NTAP1-A> Mon Sep 26 11:04:02 PDT [vdi_vfiler_211@cmds.vfiler.consol
e.switch:notice]: Console context was switched to a vFiler(tm) unit vdi_vfiler_211.

vdi_vfiler_211@NTAP1-A>
131.2 Allow the ESXi servers read and write access to the infrastructure nfs datastore. The following
command exports /vol/VDI_VFILER1_DS and /vol/VDI_SWAP
exportfs -p rw=10.1.111.0/27:10.1.211.0/27,root=10.1.111.0/27:10.1.211.0/27,nosuid /vol/VDI_SWAP
exportfs -p rw=10.1.111.0/27:10.1.211.0/27,root=10.1.111.0/27:10.1.211.0/27,nosuid
/vol/VDI_VFILER1_DS
131.3 To verify that the volumes were exported successfully, enter the command exportfs and make
sure the volumes are listed.
vdi_vfiler_211@NTAP1-A> exportfs
/vol/VDI_VFILER1_DS -sec=sys,rw=10.1.111.0/27:10.1.211.0/27,root=10.1.111.0/27
:10.1.211.0/27,nosuid
/vol/VDI_SWAP -sec=sys,rw=10.1.111.0/27:10.1.211.0/27,root=10.1.111.0/27:10.1.21
1.0/27,nosuid
/vol/VDI_VFILER211_ROOT -sec=sys,rw=10.1.10.100,root=10.1.10.100

15.9 NETAPP PERFORMANCE OPTIMIZATION

Step 132 Setting Priority Levels for the Volumes DONE/INSTRUCTOR
Duration: 5 minutes

The priority family of commands manages resouce policies for the appliance. These policies are especially
applicable on a heavily loaded appliance where resources are limited.

132.1 If needed, switch the vFiler context back to the physical controller by typing vfiler context
vfiler0.
infrastructure_vfiler_1@NTAP1-A> vfiler context vfiler0
NTAP1-A> Sun May 16 03:32:20 GMT [cmds.vfiler.console.switch:notice]: Console cont
ext was switched to a vFiler(tm) unit vfiler0.
NTAP1-A>
132.2 Globally enable priority level management on the appliance.
!!! Before
ntap1-A> priority show
Priority scheduler is stopped.

NTAP1-A> priority on
Priority scheduler starting.

!!! After
ntap1-A> priority show
Priority scheduler is running.
132.3 Set the priority level for operations sent to the volume when compared to other volumes. The
value may be one of VeryHigh, High, Medium, Low or VeryLow. A volume with a higher priority
level will receive more resources than a volume with lower resources. This option sets derived
values of scheduling (CPU), concurrent disk IO limit and NVLOG usage for the volume, based on the
settings of other volumes in the aggregate.
priority set volume INFRA_DS_1 level=VeryHigh
priority set volume ESX1_BOOT_A level=VeryHigh cache=keep
priority set volume VDI_VFILER1_DS level=VeryHigh cache=keep
priority set volume VDI_SWAP level=Medium cache=reuse

132.4 To verify that the priority levels were set correctly, issue the command priority show volume
and verify that the volumes are listed with the correct priority level.
Netapp1> priority show volume
Volume Priority Relative Sys Priority
Service Priority (vs User)
INFRASTRUCTURE_SWAP on VeryHigh Medium
VMHOST_BOOT_A on VeryHigh Medium
INFRA_DS_1 on VeryHigh Medium

ntap1-A> priority show volume
Volume Priority Relative Sys Priority
Service Priority (vs User)
LAB_VFILER10_DS on VeryHigh Medium
VMHOST_BOOT_A on VeryHigh Medium

16 NETAPP FAS2020A DEPLOYMENT PROCEDURE: PART II

This section presents a detailed procedure for configuring the interface groups (or igroups), creating LUNs for
the service profiles on the storage controllers, and mapping those LUNs to the igroups to be accessible to the
service profiles.

Step 133 Creating igroups - DONE
Duration: 2 minutes

In this step, we are going to create igroups for the different hosts. You can create igroups for individual
interfaces, hosts, clusters, host types, etc.

133.1 Create igroups for each host. We will specify the WWPN from both HBA ports.
igroup create -f -t vmware ESX1 20:00:00:25:B5:01:0A:00 20:00:00:25:B5:01:0B:00
133.2 Verify that the igroups were created successfully by entering the command igroup show and
verify that the output matches what was entered.
NTAP1-A> igroup show
VMHOST1 (FCP) (ostype: vmware):
20:00:00:25:b5:01:0a:00 (not logged in)
20:00:00:25:b5:01:0b:00 (not logged in)
20:00:00:25:b5:01:0a:01 (not logged in)
20:00:00:25:b5:01:0b:01 (not logged in)
<snip>
133.3 Verify that the igroups were created successfully by entering the command igroup show and
verify that the output matches what was entered.
Step 134 Creating LUNs for the service profiles - DONE/Instructor
Duration: 5 minutes
134.1 Create a LUN for the service profile booting from NTAP1-A. It will be 10GB in size, type vmware,
and will not have any space reserved.
Note: We are currently only using controller for active connections in our lab.
lun create -s 4g -t vmware -o noreserve /vol/ESX1_BOOT_A/ESX
134.2 Verify that the LUNs were created successfully by entering the command lun show and verify
that the new LUNs show up in the output.
NTAP1-A> lun show
/vol/ESX_BOOT_A/ESX 4g (4294967296) (r/w, online)


Step 135 Mapping LUNs to igroups
Duration: 5 minutes
135.1 For each LUN created, enter the following command to map the created LUNs to the two initiator
groups per service profile:
lun map /vol/ESX1_BOOT_A/ESX ESX1 0

135.2 Verify that the LUNs were mapped successfully by entering the command lun show and verify
that the LUNs report their status as mapped.
NTAP1-A> lun show
/vol/VMHOST_BOOT_A/VMHOST1_NTAP1-A 2g (2147483648) (r/w, online, mapped)

16.1 FLEXCLONE
Step 136 FlexClone the ESX boot volume to create individual boot volume/luns for each ESX server.
136.1 FlexClone a fas3170_vfiler2 volume and add that clone to fas3170_vfiler1
136.2 Take a snapshot of the FlexVol that has the VMFS datastore you want cloned. Name your snapshot
clone_base_snap so that you can identify the purpose of the snapshot. The command below will
create a snapshot of DCV_VFILER9_DS named clone_base_snap.
NTAP1-A> snap create ESX_BOOT_A clone_base_snap
136.3 Create a FlexClone based on the Snapshot that you just created. You will provide the name of the
new volume, the base volume, and the snapshot from the base volume.
NTAP1-A> vol clone create ESX1_BOOT_A_clone -s none -b ESX_BOOT_A clone_base_snap
136.4 show volumes (clone is only in vfiler0)
fas3170> vfiler run * vol status #
ntap1-A> vfiler run * vol status
<snip>
LAB_VFILER3_SWAP online raid_dp, flex create_ucode=on, guarantee=none,
sis fractional_reserve=0
136.5 (optional) You can split your clone off so that it is completely independent.
vol clone split start LAB_VFILER9_XEN
vol clone split status
136.6 Unmap base LUN from ESX1 igroup.
lun unmap /vol/ESX_BOOT_A/ESX ESX1
136.7 Bring cloned luns online. Cloned LUNs are offline when created.
lun online /vol/ESX1_BOOT_A_clone/ESX
136.8 Map cloned luns to igroups
lun map /vol/ESX1_BOOT_A_clone/ESX ESX1 0

136.9 Show volumes in vfilers (but not vfiler0)
ntap1-A> vfiler status -a
lab-vfiler1 running
ipspace: ips-vfiler1
IP address: 10.1.211.151 [ifgrp1-211]
Path: /vol/LAB_VFILER1_ROOT [/etc]
Path: /vol/LAB_VFILER1_DS
Path: /vol/LAB_VFILER1_SWAP
UUID: 5dd244ac-8707-11e0-bb73-00a09816bfba
Protocols allowed: 7
Allowed: proto=rsh
Allowed: proto=ssh
Allowed: proto=nfs
Allowed: proto=cifs
Allowed: proto=iscsi
Allowed: proto=ftp
Allowed: proto=http
Protocols disallowed: 0

lab-vfiler2 running
IP address: 10.1.212.151 [ifgrp1-212]
UUID: b094290c-86f4-11e0-bb73-00a09816bfba
Allowed: proto=rsh
Allowed: proto=ssh
Allowed: proto=nfs
Allowed: proto=cifs
Allowed: proto=ftp
Allowed: proto=http

vfiler run lab-vfiler1 exportfs -p rw=10.1.211.21,root=10.1.211.21 /vol/LAB_VFILER1_DS

136.10 Add cloned volumes into vfiler.
vfiler add lab-vfiler1 /vol/LAB_VFILER1_XEN
Note: Might be useful to add _CLONE suffix to the end for ease of reference.
136.11 # show volumesclone is now in vfiler1
ntap1-A> vfiler status -a
lab-vfiler1 running
IP address: 10.1.211.151 [ifgrp1-211]
Path: /vol/LAB_VFILER1_XEN
UUID: 5dd244ac-8707-11e0-bb73-00a09816bfba
Allowed: proto=rsh
Allowed: proto=ssh
Allowed: proto=nfs
Allowed: proto=cifs
Allowed: proto=ftp
Allowed: proto=http

lab-vfiler2 running
IP address: 10.1.212.151 [ifgrp1-212]
Path: /vol/LAB_VFILER2_XEN
UUID: b094290c-86f4-11e0-bb73-00a09816bfba
Allowed: proto=rsh
Allowed: proto=ssh
Allowed: proto=nfs
Allowed: proto=cifs
Allowed: proto=ftp
Allowed: proto=http

vfiler run lab-vfiler1 exportfs -p rw=10.1.211.21,root=10.1.211.21 /vol/LAB_VFILER1_XEN
vfiler run lab-vfiler1 exportfs -p rw=10.1.211.20:10.1.211.21,root=10.1.211.20:10.1.211.21
/vol/LAB_VFILER1_XEN


Step 137 FlexClone a LUN.
137.1
clone start /vol/VMHOST_BOOT_A/VMHOST9_NTAP1-A /vol/VMHOST_BOOT_A/VMHOST1_clone

137.2 Unmap existing lun map to igroup.
lun unmap /vol/VMHOST_BOOT_A/VMHOST1_NTAP1-A VMHOST1

137.3 Map new LUN to igroups as disk 0.
#lun map /vol/VMHOST_BOOT_A/VMHOST1_clone VMHOST1 0
lun map /vol/VMHOST_BOOT_A/VMHOST2_clone VMHOST2 0
lun map /vol/VMHOST_BOOT_A/VMHOST3_clone VMHOST3 0

16.2 REMOVE CLONED VOLUMES AND LUNS

137.4 Take cloned volumes offline.
vol offline /vol/LAB_VFILER1_XEN

137.5 Destroy cloned volumes.
vol destroy /vol/LAB_VFILER1_XEN -f

16.3 REMOVING VFILERS
In our original lab design we created extra vfilers and volumes that we no longer need. The following steps will
allow us to stop and remove the extra vfilers.

137.6 Removing vFilers
vfiler stop lab-vfiler10
vfiler destroy lab-vfiler10 -f



16.4 REMOVING VFILER VOLUMES
In our original lab design we created extra vfilers and volumes that we no longer need. The following steps will
allow us to remove the extra volumes.
Note: These steps should be performed after the extra vfilers have been destroyed.

137.7 Take volumes offline and then destroy them.
vol offline LAB_VFILER9_ROOT
vol offline LAB_VFILER9_DS
vol offline LAB_VFILER9_SWAP
vol destroy LAB_VFILER9_ROOT -f
vol destroy LAB_VFILER9_DS -f
vol destroy LAB_VFILER9_SWAP -f

17 APPENDIX COMMAND SUMMARY

CISCO MDS FABRIC A
version 5.0(4b)
feature npiv
role name default-role
description This is a system defined role and applies to all users.
rule 5 permit show feature environment
rule 4 permit show feature hardware
rule 3 permit show feature module
rule 2 permit show feature snmp
rule 1 permit show feature system
username admin password 5 $1$KZOMZngh$g4mA5RpwcqQpgDl/EzP8M1 role network-admin
ip domain-lookup
aaa group server radius radius
snmp-server user admin network-admin auth md5 0x81a3ee7fed914a71b2e284fca6491b63 priv
0x81a3ee7fed914a71b2e284fca6491b63 localizedkey
snmp-server host 10.1.111.10 traps version 2c public udp-port 2162
!ntp server x.x.x.x
!! The following rmon statements are new with 5.0(4)
rmon event 1 log trap public description FATAL(1) owner PMON@FATAL
rmon event 2 log trap public description CRITICAL(2) owner PMON@CRITICAL
rmon event 3 log trap public description ERROR(3) owner PMON@ERROR
rmon event 4 log trap public description WARNING(4) owner PMON@WARNING
rmon event 5 log trap public description INFORMATION(5) owner PMON@INFO
vsan database
vsan 10
vsan 20
!! Currently, no vsan port-channels are set up.
! channel mode active
! switchport description To dc3-5k-1
vsan database
ip default-gateway 10.1.111.254
switchname MDS9124
line console
exec-timeout 0
boot kickstart bootflash:/m9100-s2ek9-kickstart-mz.5.0.4b.bin
boot system bootflash:/m9100-s2ek9-mz.5.0.4b.bin
interface fc1/1-24

interface fc1/1
switchport description Trunk To N5K-1
port-license acquire
no shutdown

interface fc1/2
switchport description Trunk To N5K-2
no shutdown

interface fc1/3
no switchport trunk allowed vsan all
switchport description NetApp Storage 0a
switchport trunk mode off
no shutdown

interface fc1/4
no switchport trunk allowed vsan all
switchport description NetApp Storage 0b
switchport trunk mode off
no shutdown

interface fc1/5-8

interface fc1/9-24

interface mgmt0
ip address 10.1.111.40 255.255.255.0
no system default switchport shutdown

CISCO NEXUS 5010 1 - N5K-1

version 5.0(2)N2(1)
feature fcoe
no feature telnet
no telnet server enable
cfs eth distribute
feature private-vlan
feature udld
feature interface-vlan
feature lacp
feature vpc
feature lldp
feature fex
username admin password 5 $1$1D5NtLb8$NoaVz9MCMF97uhD5ljpni0 role network-admin
banner motd #LAB2 SAVED CONFIG

#
ip domain-lookup
ip domain-lookup
switchname N5K-1
logging event link-status default
service unsupported-transceiver
!class-map type queuing class-fcoe
! match qos-group 1
match qos-group 2
match qos-group 2
!class-map type network-qos class-fcoe
! match qos-group 1
match qos-group 2
match qos-group 2
policy-map type network-qos jumbo
pause no-drop
mtu 2158
mtu 9000
system qos
service-policy type network-qos jumbo
fex 100
pinning max-links 1
description "FEX0100"
!!! This is a placeholder for a single-homed FEX.
fex 101
pinning max-links 1
snmp-server user admin network-admin auth md5 0xfa96c2442eb06eea84a35684c9b13850 priv
0xfa96c2442eb06eea84a35684c9b13850 localizedkey
!snmp-server host 10.1.111.10 traps version 2c public udp-port 1163
!snmp-server host 10.1.111.10 traps version 2c public udp-port 2162
snmp-server enable traps entity fru
!ntp server x.x.x.x
!ntp server x.x.x.x use-vrf management
ip route 0.0.0.0/0 10.1.111.254
vlan 1
vlan 20
name VM-Client
vlan 21
name ERSPAN
vlan 22
vlan 23
name vmotion
vlan 24
vlan 25
name PVLAN
vlan 100
fcoe vsan 10
vlan 160
name N1KV_CONTROL_PACKET
vlan 162
name iscsi
!vlan 200
! name 1kv-control
!vlan 201
! name 1kv-packet
vlan 520
name backend-storage
vlan 999
name NATIVE
udld aggressive
port-channel load-balance ethernet source-dest-port
vpc domain 1
role priority 1000
peer-keepalive destination 10.1.111.2
vsan database
vsan 10

interface Vlan1

!!! We currently do not SAN port-channel configured.
!interface san-port-channel 256
! switchport mode NP
! switchport description To p3-mds9148-1
! switchport trunk mode on

vpc peer-link
!!! No fcoe vlans allowed on vpc peerlink.
switchport trunk allowed vlan 1,20-25,160,200-201
speed 10000

description ESX1
vpc 3
switchport trunk allowed vlan 1,20-25,100,160,200-201
speed 10000

description ESX2
vpc 4
speed 10000

description ESX3
vpc 5
speed 10000

description link to core
vpc 60
switchport trunk allowed vlan 1,20-25,160
speed 10000

!!! We currently do not have IP storage plugged directly into our 5Ks.
!!! IP storage comes through core switches.
!interface port-channel70
! description IP Storage Array
! vpc 70
! switchport access vlan 162

description dual-homed 2148 can use as management switch
vpc 100
fex associate 100

description single-homed 2248
fex associate 101

interface vfc3
bind interface port-channel3
no shutdown

interface vfc4
no shutdown

interface vfc5
no shutdown
vsan database
vsan 10 interface vfc3
!!! This is a placeholder in case we want to do NPV and san port-channels.
! vsan 10 interface san-port-channel 256

interface fc2/1
switchport description To MDS9124 1/1
switchport trunk mode on
no shutdown

interface fc2/2-4

!!! This is a placeholder in case we want to go to NPV mode.
!feature npv
!npv enable

description To 3750
switchport trunk allowed vlan 1
speed 1000


description To ESX1 vmnic0
spanning-tree bpduguard enable
channel-group 3

channel-group 4

channel-group 5


!!! Associate interfaces e1/7-8 to fex 101 when moving to single homed FEX.
fex associate 100
channel-group 100

fex associate 100
channel-group 100

interface Ethernet1/9-16



! swtchport trunk native vlan 999

! switchport trunk native vlan 999


interface mgmt0
ip address 10.1.111.1/24

interface Ethernet100/1/1


interface Ethernet100/1/3-48

line console
exec-timeout 0
line vty
exec-timeout 0
boot kickstart bootflash:/n5000-uk9-kickstart.5.0.2.N2.1.bin
boot system bootflash:/n5000-uk9.5.0.2.N2.1.bin
interface fc2/1-4

CISCO NEXUS 5010 2 - N5K-2

version 5.0(2)N2(1)
feature fcoe
no feature telnet
no telnet server enable
cfs eth distribute
feature private-vlan
feature udld
feature interface-vlan
feature lacp
feature vpc
feature lldp
feature fex
username admin password 5 $1$1D5NtLb8$NoaVz9MCMF97uhD5ljpni0 role network-admin
banner motd #LAB2 SAVED CONFIG

#
ip domain-lookup
ip domain-lookup
switchname N5K-2
logging event link-status default
service unsupported-transceiver
!class-map type queuing class-fcoe
! match qos-group 1
match qos-group 2
match qos-group 2
!class-map type network-qos class-fcoe
! match qos-group 1
match qos-group 2
match qos-group 2
policy-map type network-qos jumbo
pause no-drop
mtu 2158
mtu 9000
system qos
service-policy type network-qos jumbo
fex 100
pinning max-links 1
fex 101
pinning max-links 1
snmp-server user admin network-admin auth md5 0xfa96c2442eb06eea84a35684c9b13850 priv
0xfa96c2442eb06eea84a35684c9b13850 localizedkey
snmp-server enable traps entity fru
!ntp server x.x.x.x
!ntp server x.x.x.x use-vrf management
ip route 0.0.0.0/0 10.1.111.254
vlan 1
vlan 20
name VM-Client
vlan 21
name ERSPAN
vlan 22
vlan 23
name vmotion
vlan 24
vlan 25
name PVLAN
vlan 120
fcoe vsan 20
vlan 160
name N1KV_CONTROL_PACKET
vlan 162
name iscsi
vlan 200
name 1kv-control
vlan 201
name 1kv-packet
vlan 520
name backend-storage
vlan 999
name NATIVE
udld aggressive
port-channel load-balance ethernet source-dest-port
vpc domain 1
role priority 2000
peer-keepalive destination 10.1.111.1
vsan database
vsan 20

interface Vlan1

!interface san-port-channel 256
! switchport mode NP
! switchport description To p3-mds9148-1
! switchport trunk mode on

vpc peer-link
speed 10000

description ESX1
vpc 3
speed 10000

description ESX2
vpc 4
speed 10000

description ESX3
vpc 5
speed 10000

vpc 60
speed 10000

!!! We currently do not have IP storage plugged directly into our 5Ks.
!!! IP storage comes through core switches.
!interface port-channel70
! description IP Storage Array
! vpc 70
! switchport access vlan 162

description dual-homed 2148
vpc 100
fex associate 100

description single-homed 2248
fex associate 101

interface vfc3
no shutdown

interface vfc4
no shutdown

interface vfc5
no shutdown
vsan database
!!! This is a placeholder in case we want to do NPV and san port-channels.
! vsan 20 interface san-port-channel 256

interface fc2/1
switchport description To MDS9124 1/2
switchport trunk mode on
no shutdown

interface fc2/2-4

!feature npv
!npv enable

description To 3750
switchport trunk allowed vlan 1
speed 1000


channel-group 3

channel-group 4

channel-group 5


!!! Associate interfaces e1/7-8 to fex 101 when moving to single homed FEX.
fex associate 100
channel-group 100

fex associate 100
channel-group 100







interface mgmt0
ip address 10.1.111.2/24



interface Ethernet100/1/3-48

line console
exec-timeout 0
line vty
exec-timeout 0
boot kickstart bootflash:/n5000-uk9-kickstart.5.0.2.N2.1.bin
boot system bootflash:/n5000-uk9.5.0.2.N2.1.bin
interface fc2/1-4

ESX
ESX1 and ESX2
On ESXi host ESX1
On ESXi host ESX2

ESX1 and ESX2

CISCO NEXUS 1000V - VSM-1
version 4.2(1)SV1(4)
no feature telnet
username admin password 5 $1$THFpg.Mp$p5Rh4aBqgKuUFZlvqAhu30 role network-admin
banner motd #Nexus 1000v Switch#
ssh key rsa 2048
ip domain-lookup
ip domain-lookup
hostname VSM-P
vem 3
host vmware id 00d18e47-54f1-de11-ba89-0022bdd3392e
vem 4
host vmware id 6da2f331-dfd4-11de-b82d-c47d4f7ca766
vem 5
host vmware id 67ae4b62-debb-11de-b88b-c47d4f7ca604
vem 6
host vmware id 30b0cdb3-deaf-11de-b5ac-c47d4f7ca574
vem 7
host vmware id a0565a73-a811-df11-b671-8843e1c2694c
vem 8
host vmware id a5206300-ff60-11de-9bbb-f5803dad1e37
snmp-server user admin network-admin auth md5 0xcac2e012077bc51a340006d3fca7f363 priv
0xcac2e012077bc51a340006d3fca7f363 localizedkey

ip route 0.0.0.0/0 192.168.1.254
vlan 1
vlan 131
name VM-Client
vlan 151
name vmotion
vlan 171
name n1k_control_packet
vlan 211
name NFS-VLAN
port-channel load-balance ethernet source-dest-ip-port-vlan
port-profile default max-ports 32
port-profile type ethernet Unused_Or_Quarantine_Uplink
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type ethernet VM_UPLINK
vmware port-group
switchport trunk allowed vlan 20,23,160,162
mtu 9000
no shutdown
system vlan 23,160,162
state enabled
port-profile type ethernet VM_UPLINK2
vmware port-group
switchport trunk allowed vlan 20,23,160
mtu 9000
no shutdown
system vlan 23,160
state enabled

port-profile type vethernet MGMT
vmware port-group
no shutdown
system vlan 1
state enabled
vmware port-group
no shutdown
system vlan 23
state enabled
port-profile type vethernet STORAGE
vmware port-group
no shutdown
system vlan 162
state enabled
port-profile type vethernet N1KV_CONTROL_PACKET
The VMotion, NFS, and Control/Packet VLANs need to be system
VLANs for availability.
vmware port-group
no shutdown
system vlan 160
state enabled
port-profile type vethernet VM_CLIENT
vmware port-group
no shutdown
state enabled

vdc VSM-P id 1
limit-resource vlan minimum 16 maximum 2049
limit-resource monitor-session minimum 0 maximum 2
limit-resource vrf minimum 16 maximum 8192
limit-resource port-channel minimum 0 maximum 768
limit-resource u4route-mem minimum 32 maximum 32
limit-resource u6route-mem minimum 16 maximum 16
limit-resource m4route-mem minimum 58 maximum 58
limit-resource m6route-mem minimum 8 maximum 8

inherit port-profile VM_UPLINK



interface mgmt0
ip address 192.168.1.200/24

interface Vethernet1
inherit port-profile N1KV_CONTROL_PACKET
description Nexus1000V-P,Network Adapter 1
vmware dvport 164 dvswitch uuid "90 8a 19 50 83 ea 6a 15-c8 2c 13 44 d3 43 06 fe"
vmware vm mac 0050.5699.000B

description Nexus1000V-P,Network Adapter 3
vmware vm mac 0050.5699.000D

inherit port-profile VMOTION
description VMware VMkernel,vmk1
vmware vm mac 0050.567F.90F4

description Nexus1000V-S,Network Adapter 1
vmware vm mac 0050.5699.0011

description Nexus1000V-S,Network Adapter 3
vmware vm mac 0050.5699.0013

vmware vm mac 0050.5671.25BC

inherit port-profile VM_CLIENT
description Server 2003R2-Clone,Network Adapter 1
vmware vm mac 0050.5699.0007

inherit port-profile VM_CLIENT
description Server-2003R2,Network Adapter 1
vmware vm mac 0050.5699.0005

vmware vm mac 0050.567A.956B







interface control0
line console
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-1
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-2
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-2
svs-domain
domain id 10
control vlan 160
packet vlan 160
svs mode L2
protocol vmware-vim
vmware dvs uuid "90 8a 19 50 83 ea 6a 15-c8 2c 13 44 d3 43 06 fe" datacenter-name Lab
connect
vnm-policy-agent
registration-ip 0.0.0.0
shared-secret **********
log-level

Make sure these VLANs are created and designated as SytemsVLANs
in UPLINK Ethernet profiles.
OTV

no feature vpc
int port-channel 1
shutdown
int e1/10
interface po14
shutdown
vlan 131,151,171,211,1005
no shut
int e1/19
switchport
no shutdown

no feature vpc
shutdown
! shutdown
!interface e1/4,e1/9,e1/11
shutdown
vlan 131,151,171,211,1005
no shut
int et 1/20
switchport
no shutdown

N7K-1
vlan 131,151,171,211,1005
no shut
!int e1/14
int e1/22
!int e1/30
switchport
mtu 9216
no shutdown

int e 1/<uplink>
no shut

feature ospf
router ospf 1

interface loopback0
! ip address 10.1.0.11/32
ip address 10.1.0.21/32
! ip address 10.1.0.31/32
!interface e1/10
interface e1/18
!interface e1/26
mtu 9042
! ip address 10.1.11.3/24
ip address 10.1.21.5/24
! ip address 10.1.31.7/24
ip igmp version 3
no shutdown

feature otv
otv site-vlan 1005

interface Overlay 1
! otv control-group 239.1.1.1
! otv data-group 239.1.2.0/28
! otv join-interface Ethernet1/10
no shutdown

N7K-2
vlan 131,151,171,211,1005
no shut
!int e1/16
int e1/24
!int e1/32
switchport
mtu 9216
no shutdown

int e 1/<uplink>
no shut

feature ospf
router ospf 1

interface loopback0
! ip address 10.1.0.12/32
ip address 10.1.0.22/32
! ip address 10.1.0.32/32
!interface e1/12
interface e1/20
!interface e1/28
mtu 9042
! ip address 10.1.14.4/24
ip address 10.1.24.6/24
! ip address 10.1.34.8/24
ip igmp version 3
no shutdown

feature otv
otv site-vlan 1005

interface Overlay 1
no shutdown

18 REFERENCES
VMware Fibre Channel SAN Configuration Guide
http://www.vmware.com/pdf/vsphere4/r41/vsp_41_san_cfg.pdf
Cisco Nexus 1000V Port Profile Configuration Guide, Release 4.2(1) SV1(4)
http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4/port_profile/c
onfiguration/guide/n1000v_portprof_4system.html#wpxref14373

NOW (NetApp on the Web) site
http://.now.netapp.com

NetApp FAS2020 Storage Controller
http://now.netapp.com/NOW/knowledge/docs/hardware/hardware_index.shtml#Storage%20appliances%20an
d%20V-series%20systems/gFilers

Cisco Nexus 5010 Switch

Cisco Unified Computing System
www.cisco.com/en/US/netsol/ns944/index.html

Cisco Nexus 1010 Virtual Services Appliance

VMware vSphere
www.vmware.com/products/vsphere/


VLAN ID for NFS traffic
Network address for NFS traffic
VLAN ID for management traffic
VLAN ID for VMotion traffic
Network address for VMotion traffic
VLAN ID for the Cisco
Nexus 1000v packet and control traffic
VLAN ID for native VLAN
VLAN ID for VM traffic
Default password
DNS server name
Domain name suffix
VSAN ID for fabric A

11

21:00:00:c0:dd:14:73:2f


DCV Nexus

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

DCV Nexus

Diunggah oleh

Hak Cipta:

Format Tersedia

2011 Cisco FlexPod Training Guide Page 1 of 217

Data Center Virtualization (DCV)

Anda mungkin juga menyukai