Best Practices for IBM Storage with VMware SVC / Storwize V7000 / XIV
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
VMware NMP
VMware NMP is an extensible module that manages sub plug-ins
Storage Array Type Plug-Ins (SATPs) Path Selection Plug-Ins (PSPs)
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
VMware NMP
Storage Array Type Plug-In (SATP)
VMware recognizes SAN and loads
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
Issue/impact
First identified path (preferred path) same for all
LUNs
VMW_PSP_MRU for Most Recently Used mode VMW_PSP_FIXED for Fixed mode VMW_PSP_RR for Round Robin mode
4.0 (then use fix path). Do not use MRU Zoning Balance load across all IO controllers
better LVM extents are supported but not recommended. In vSphere 5.0 with XIV, you can increase the size of a datastore and LUN (up to 64TB) on-line. For all best practices http://www.redbooks.ibm.com/abstracts/sg247904.html
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
10
Set both the queue_depth and the Disk.SchedNumReqOutstanding VMWare kernel parameter to 128 on an ESX host that has exclusive access to its LUNs. Set both the queue_depth and the Disk.SchedNumReqOutstanding VMWare kernel parameter to 64 when a few ESX hosts share access to a common group of LUNs.
Enable use of non-optimal paths with Round-Robin PSP #esxcli nmp roundrobin setconfig --device eui.0017380000691cb1 --useANO=1
Tune amount of I/O to use per path # esxcli nmp roundrobin setconfig --device eui.0017380000691cb1 --iops=10 --type "iops"
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
11
New feature in IBM Storage Management Console for VMware vCenter 3.0
VMware Storage Agnostic Features
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
12
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
13
Place all ports for all hosts in single group Or be diligent with LUN-ID assignment
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
14
Max performance is 12 paths per initiator but this may not be suitable for large scale implementations
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
15
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
16
Port Binding
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
17
Creates 1-to-1 path for VMware NMP Utilize same PSP as for FC connectivity
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
18
iSCSI Sessions Created: Node-1 Vmk-0 (1.1.1.1) to Port-0 (1.1.1.10) Vmk-0 (1.1.1.1) to Port-1 (1.1.1.11) Vmk-1 (1.1.1.2) to Port-0 (1.1.1.10) Vmk-1 (1.1.1.2) to Port-1 (1.1.1.11) Node-2 Vmk-0 (1.1.1.1) to Port-0 (1.1.1.12) Vmk-0 (1.1.1.1) to Port-1 (1.1.1.13) Vmk-1 (1.1.1.2) to Port-0 (1.1.1.12) Vmk-1 (1.1.1.2) to Port-1 (1.1.1.13) Configuration results in 4 sessions per V7000 node
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
19
Guidelines for the Attachment of VMware iSCSI Hosts to SAN Volume Controller and Storwize V7000 Errata (v6.2.x and higher) Feb 8th 2012
Maximum of 1 VMware iSCSI initiator session per V7000 IQN (node) Static Discovery Only, Dynamic Discovery is not supported
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
20
iSCSI Sessions Created: Node-1 Vmk-0 (1.1.1.1) to Port-0 (1.1.1.10) Vmk-0 (1.1.1.1) to Port-1 (1.1.1.11) Vmk-1 (1.1.1.2) to Port-0 (1.1.1.10) Vmk-1 (1.1.1.2) to Port-1 (1.1.1.11) Node-2 Vmk-0 (1.1.1.1) to Port-0 (1.1.1.12) Vmk-0 (1.1.1.1) to Port-1 (1.1.1.13) Vmk-1 (1.1.1.2) to Port-0 (1.1.1.12) Vmk-1 (1.1.1.2) to Port-1 (1.1.1.13 Configuration results in 1 sessions per V7000 node
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
21
iSCSI Sessions Created: Node-1 Vmk-0 (1.1.1.1) to Port-0 (1.1.1.10) Vmk-0 (1.1.1.1) to Port-1 (1.1.1.11) Vmk-1 (1.1.1.2) to Port-0 (1.1.1.10) Vmk-1 (1.1.1.2) to Port-1 (1.1.1.11) Node-2 Vmk-0 (1.1.1.1) to Port-0 (1.1.1.12) Vmk-0 (1.1.1.1) to Port-1 (1.1.1.13) Vmk-1 (1.1.1.2) to Port-0 (1.1.1.12) Vmk-1 (1.1.1.2) to Port-1 (1.1.1.13 Configuration results in 4 sessions per V7000 node
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
22
Enable jumbo frames for throughput intensive workloads (must be done at all
layers)
discovery and manual load balancing of V7000 ports if more than one host is
used
or dynamic discovery can be used and all V7000 ports can be used by all hosts
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
23
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
24
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
25
http://www.vmware.com/pdf/vsphere5/r50/vsphere-50-configuration-maximums.pdf
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
2012 IBM Corporation
26
Keep in mind time required to migrate volumes if needed. Also snapshots are a consideration here Less orphaned space
Avoid using very few LARGE volumes to balance workloads
Take advantage of all ports on system Better utilization of queues Try to balance workload across at least 8 LUNs
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
27
VM Snapshots
Best Practices
VMware Snapshots are not backups, although backup software typically use VMware snapshots Regular monitoring Configure vCenter Snapshot alarms KB 1018029
Improvements
ESX(i) 4.0 U2 Snapshot deletion takes up less space on disk ESXi 5.0 New functionality to monitor snapshots and provide warning if snapshots need consolidation.
Snapshot Best Practices KB 1025279 Understanding Virtual Machine Snapshots in ESX KB 1015180
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
28
FlashCopy Manager
Common solution for VMware: Common solution for VMware: -Shares a common UI Shares a common UI -Can configure two possible targets Can configure two possible targets 1. TSM Storage Pool 1. TSM Storage Pool 2. Hardware snapshot 2. Hardware snapshot
Hardware Snapshot
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
29
29
Storage DRS Manual Mode Recommend running Storage DRS manual mode initially
Recommendations display Space Utilization (before & after) for source & destination datastores as well as the current latency values of the source and destination.
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
30
Storage DRS Manual Mode When SDRS is in manual mode, recommendations are displayed in the Storage DRS tab:
Recommendations display Space Utilization (before & after) for source & destination datastores as well as the current latency values of the source and destination.
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
31
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
32
Performance Advantage
Expected better performance on VAAI enabled storage
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
33
V M D K
V M D K
V M D K
V M D K
V M D K
V M D K
34
Block Zeroing
Block Zeroing
Reduces IOPS and CPU overhead for creation of EZT (Eager Zeroed Thick) VMDKs
Why EZT?
Smallest IO overhead (after creation) Somewhat better performance VMware Fault Tolerance for vSphere 4 requires EZT
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
35
VMFS-5
VMDK VMDK
Extent
With API
SCSI UNMAP releases the blocks backSCSI WRITE to the free pool. SCSI DATA WRITE Is used anytime VMFS deletes (svMotion, Delete VM, Delete Snapshot, Delete)
DATA SCSI WRITE SCSI DATA WRITE DATA SCSI UNMAP Space is immediately reclaimed
Without API only VMFS metadata change Data remains consumed on disk
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
36
Left Graphic - Configuration, Storage in vSpere Client, select the datastore name, properties, and by hovering the mouse over the device name Right Graphic Under the IBM Storage tab in vCenter, select the datastore and view the ID
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
37
SSH to the ESX server and type esxtop (or use resxtop in the powercli) and go the LUN performance view by pressing u. To view VAAI information add new columns and remove others Once in esxtop, press u then f (or letter for VAAI stats. Disable all others so output will fit in screen)
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
38
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
39
IBM Storage Plugin Supports XIV SVC V7000 and SONAS Simply create LUNs or Shares Present LUNs or Shares to ESX servers (NAS) Features
Provisioning mapping FC
and NAS Support the ESX multi-path for XIV storage LUNs in bulk
Manage
Provision
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
40
Select a datastore
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
41
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
42
Tasks in vCenter
ESX servers rescan to discover new LUNS Updates IBM plugin
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
43
LUNs in vCenter
New LUN
Can be used for Datastore or RDM
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en #IBMEDGE
44
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
45
Filesystem (for VMware, need to consider VMFS and/or Guest OS filesystem) Database Application
Format options Defrag (may defeat thin provisioning by touching space) Zero file utilities can enable space reclamation for storage systems with zero detect or scrubbing
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
46
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
47
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
48
More Space Savings than disk over time (assume UNMAP not available)
More aware of data that has moved or deleted
Disadvantages
When a thin provisioned disk grows, the ESX host must make a SCSI reservation
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
49
SAN Volume Controller(SVC) / Storwize V7000 Thin Provisioning Considerations Grain size for a volume set at 256K Cannot be changed after the thin-provisioned volume has been created 256K is Strongly recommended and is default for 6.3.0.3 and above Specify the same grain size for FlashCopy Use autoexpand unless very closely monitored Storage pool reserve can protect over-allocation of volumes Do not thin provision external virtualized disk behind SVC or V7000 Thin provisioning exposes volume space utilization (usage is not displayed for thick volumes) Thin provisioning speeds up copying (only actual data is copied from thin source volume) Volume brought in as image mode cannot be thin provisioned
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
50
For thin pool, user specifies: Pool soft size (aggregate volume sizes presented to hosts) Pool hard size (aggregate physical space available for data)
51
Plan procedures for adding space (decide whether to automate) Use VAAI
Limits impact of SCSI Reservations is Thin Provisioning is used Improves performance Please check with your IBM storage expert for Code required for VAAI
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
52
Databases DB2, Oracle, MS-SQL, etc. Applications based on databases SAP, Oracle Applications, etc. Server Virtualization KVM, VMware, Hyper-V, etc. Other compressible workloads engineering, seismic, collaboration, etc.
Workloads using pre-compressed data types such as video, images, audio, etc. Workloads using encrypted data Heavy sequential write oriented workloads Other workloads using incompressible data or data with low compression rate
4 core systems (V7000, CF8, older CG8) with less than 25% CPU utilization (before enabling compression) 6 core systems (newer CG8) with less than 50% CPU utilization (before enabling compression)
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
53
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
54
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?lang=en
55
55
Host bus adapters (HBAs) include SCSI, iSCSI, RAID, and FCHBA adapters.
latency stats from the device, the kernel, and the guest
Average latency (ms) of the device (LUN) Average latency (ms) in the VMkernel, also known as queuing time Average latency (ms) in the guest. GAVG = DAVG +
Recordings:
https://www.ibm.com/developerworks/mydeveloperworks/blogs/accelerate/?order=desc&maxresults=100&sortby=0&lang=en
2012 IBM Corporation