How to Deploy a Nexus 1000v lab with a single ESX host.

By Robert Burns CCIE Data Center #37856

*Slight variation works with VMware Workstation/Fusion also. *Details of third party OS & application installation is beyond the scope of this guide. Pre-requisites: -Disable Windows firewalls on your client -Java must be installed Example Host List Host IP 10.85.49.215 10.85.49.216 10.85.49.217 10.85.49.218 10.85.49.219 10.85.49.220 VM Name RHEL62-Test-1 vCenter-5 ESX5-Nested-1 ESX5-Nested-2 N1000v Description Bare Metal ESX host Redhat test VM vCenter server Nested ESX 5.1 (VM 1) Nested ESX 5.1 (VM 2) VSM Management IP

1. Infrastructure setup a. Install ESX bare metal host. Ensure you have > 75GB available space on your VMFS if you plan on installing a vCenter VM with Update Manager. b. Configure management network interface. c. Install one WIN2K8/2012 VM (for vCenter) allocated 30GB or more for virtual disk. Next install vCenter 5.1 on Windows VM. Alternately you can use the vCenter appliance. d. On the vCenter server install VMware Update Manager (VUM). e. Install two ESX 5.1 VMs following Nested ESX Instructions here: http://www.vcritical.com/2011/07/vmware-vsphere-can-virtualize-itself/ i. Configured four vNICs each using the default vSwitch VM Network port group for now. ii. Boot Nested ESX VMs and configure Management network interfaces from the VM console within VI Client. f. Test all IP connectivity between vCenter and all three ESX hosts. g. Add Nested ESX hosts to vCenter. See Fig. 1

Tip: Put the Nested ESX hosts in their own Cluster for easier management.
Note: The two ESX5-Nested-X VMs below correspond to 10.85.49.218 & 10.85.49.219 hosts connected to vCenter in the ESXNested cluster.

Fig. 1 Infrastructure setup with Nested ESX VMs installed & added to vCenter.

2. Deploy the 1000v VSM a. Download the latest 1000v SW from CCO. *For ESX 5.1 and later you must use 1000v version Nexus1000v.4.2.1.SV2.1.1 or later. http://software.cisco.com/download/release.html?mdfid=282646785&flowid=3090&softwa reid=282088129&release=4.2%281%29SV2%281.1a%29&relind=AVAILABLE&rellifecycle=&r eltype=latest

b. Unzip the bundle and navigate to the Install_App folder & launch the application.

c. Start with the VSM Complete Installation selecting Custom install.

d. Review the pre-reqs & click Next. e. Enter the appropriate details for your vCenter

f.

Enter the appropriate info for your VSM. Since Im hosting the VSM pair on my single bare metal ESX host, Ive used the same ESX Host IP twice. Click Next when complete. Notes: - Recommend setting your Domain ID to something other than default 1. - Im setting my Management VLAN to what my physical switches use for their native VLAN. - I chose to use L2 mode for simplicity versus L3, but either will work. - Do not migrate hosts at this time.

g. Review configuration and click Next. Be patient, deployment will take up to 15mins.

h. Next step will prompt to add additional Modules.

i.

Select the hosts you wish to have the VEM agent installed. Click Next.
Note: This method requires VUM to be previously installed. If not, youll need to manually install the VEM agent vibs.

j.

Review and then click Finish to proceed.

You can monitor the progress from the VI Client Recent Task log

k. The Install App hopefully completed successfully for all hosts. **If the VEM installation fails, it likely points to a problem with VMware Update Manager (VUM).

l.

From the VI Client go to Home -> Inventory -> Networking and you should see your two new hosts as part of the 1000v DVS. Ensure you click on the 1000v DVS in the left pane.

m. SSH into the VSM and and check the modules

N1000v(config)# show mod Mod --1 2 3 4 Ports ----0 0 248 248 Module-Type -------------------------------Virtual Supervisor Module Virtual Supervisor Module Virtual Ethernet Module Virtual Ethernet Module Model -----------------Nexus1000V Nexus1000V NA NA Status -----------active * ha-standby ok ok

Mod -1 2 3 4

Sw -----------------4.2(1)SV2(1.1) 4.2(1)SV2(1.1) 4.2(1)SV2(1.1) 4.2(1)SV2(1.1)

Hw -----------------------------------------------0.0 0.0 VMware ESXi 5.1.0 Releasebuild-838463 (3.1) VMware ESXi 5.1.0 Releasebuild-838463 (3.1)

Mod --1 2 3 4 Mod --1 2 3 4

MAC-Address(es) -------------------------------------00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 02-00-0c-00-03-00 to 02-00-0c-00-03-80 02-00-0c-00-04-00 to 02-00-0c-00-04-80 Server-IP --------------10.85.49.220 10.85.49.220 10.85.49.218 10.85.49.219 Server-UUID

Serial-Num ---------NA NA NA NA Server-Name -------------------NA NA 10.85.49.218 10.85.49.219

-----------------------------------NA NA 422954ef-1f4d-f096-2a47-4b64cd67b932 42299b1d-8226-47bc-f375-2432c7cbe87e

3.

Migrate Test VM & Test Connectivity a. Cold migrate (powered off) your Test VM to one the Nested ESX hosts. If you get any errors, youve likely done one of the following: Didnt modify your Nested ESX VM to Virtual Machine Version 9 prior to install Didnt enable the Expose NX/XD flag to guest option in the Nested ESX VM.

b.

Before we power it up were going to create a Port Profile for it on the 1000v. N1000v(config)# port-profile type vethernet rhel-pp N1000v(config-port-prof)# switchport mode access N1000v(config-port-prof)# switchport access vlan 711 N1000v(config-port-prof)# state enabled N1000v(config-port-prof)# no shut N1000v(config-port-prof)# vmware port-group

c.

Now change the virtual network binding of your test VM from the vSwitch to the 1000v port profile.

d.

Power up your test VM, and verify the interface on the 1000v.
N1000v(config)# show interface virtual ------------------------------------------------------------------------------Port Adapter Owner Mod Host ------------------------------------------------------------------------------Veth1 Net Adapter 1 RHEL62-Test-1 3 10.85.49.218 N1000v(config)#

Assuming your networking & port profiles are setup correctly you should have connectivity to your Test VM.

4.

Advanced Configuration (optional) a. Now that we have basic connectivity, lets add the remaining uplinks to your Nested ESX VEM hosts. Select the host Configuration Networking vSphere Distributed Switch tab Manage Physical Adapters

b.

Find the uplink port profile and click Add NIC. Add each of the 2 remaining NICs from each host.

c.

Verify the uplinks on the 1000v. Your uplink port profile should be configured for mac pinning in which case you should see two new Port channels automatically created.
N1000v(config)# show int brief -------------------------------------------------------------------------------Port VRF Status IP Address Speed MTU -------------------------------------------------------------------------------mgmt0 -up 10.85.49.220 1000 1500 -------------------------------------------------------------------------------Ethernet VLAN Type Mode Status Reason Speed Port Interface Ch # -------------------------------------------------------------------------------Eth3/2 711 eth trunk up none 1000 1 Eth3/3 711 eth trunk up none 1000 1 Eth3/4 711 eth trunk up none 1000 1 Eth4/2 711 eth trunk up none 1000 2 Eth4/3 711 eth trunk up none 1000 2 Eth4/4 711 eth trunk up none 1000 2 -------------------------------------------------------------------------------Port-channel VLAN Type Mode Status Reason Speed Protocol Interface -------------------------------------------------------------------------------Po1 711 eth trunk up none a-1000(D) none Po2 711 eth trunk up none a-1000(D) none <snip>

5.

Exercise - Determine which uplink your Test VM is utilizing i. Find which module the VM is hosted by. N1000v(config)# show int virtual ------------------------------------------------------------------------------Port Adapter Owner Mod Host ------------------------------------------------------------------------------Veth1 Net Adapter 1 RHEL62-Test-1 3 10.85.49.218 ii. Identify the Sub Group IDs of all uplinks on that host. N1000v(config)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 2 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip> You can see from the output, SGID 1 = vmnic1, 2 = vmnic2 and 3 = vmnic3 iii. Find the VMs pinned Sub Group ID from the same ouput. N1000v(config)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 2 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip> From this we can see that the Test VM is assigned to SGID 2, which will use vmnic2 for external communication.

6.

Exercise Force your Test VM to utilize vmnic1. a. b. First determine what the SGID of vmnic1 is. From our previous output, this would be SGID 1. Configure either the port profile or the individual interface to prefer this Sub Group. N1000v(config)# port-profile rhel-pp N1000v(config-port-prof)# pinning id 1 Verify the change. N1000v(config-port-prof)# module vem 3 execute vemcmd show port LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type 18 Eth3/2 UP UP FWD 305 1 vmnic1 19 Eth3/3 UP UP FWD 305 2 vmnic2 20 Eth3/4 UP UP FWD 305 3 vmnic3 49 Veth1 UP UP FWD 0 1 RHEL62-Test-1.eth0 305 Po1 UP UP FWD 0 <snip>

c.

7.

Explore & play with various features - ACLs, QoS, PVLANs, etc!