Questions (FAQ)
November 20, 2017
Table of Contents
1. vSAN FAQ
1.1.Introduction
1.2.Architecture
1.3.Availability
1.4.Security
1.5.Performance
1.6.Operations
1. vSAN FAQ
.
1.1 Introduction
This document provides answers to frequently asked questions (FAQ) regarding VMware vSAN.
vSAN is enterprise-class storage for hyper-converged infrastructure (HCI). Native to the VMware
vSphere hypervisor, vSAN delivers ash-optimized, secure storage. It utilizes commodity x86 server
components to lower costs up to 50% versus traditional server and storage array architectures.
Seamless integration with vSphere and the VMware ecosystem makes it the ideal storage platform for
business-critical applications, disaster recovery sites, remote oce and branch oce (ROBO)
implementations, test and development environments, management clusters, security zones, and
virtual desktop infrastructure (VDI). Today, customers of all industries and sizes trust vSAN to run their
most important applications.
1.2 Architecture
A vSAN cluster must have at least two physical hosts with local storage devices dedicated to vSAN. A
vSAN cluster containing hosts with magnetic drives in the capacity tier is commonly called a hybrid
cluster or conguration. A cluster with hosts containing ash devices in the capacity tier is referred to
as an all-ash cluster or conguration.
Hosts participating in a vSAN cluster must be connected to the network using at least one network
interface card (NIC). Multiple NICs are recommended for redundancy. Hybrid vSAN congurations can
use 1Gb or higher networks although 10Gb or higher is recommended. All-ash vSAN congurations
require 10Gb or higher networks. Multicast network trac is required for vSAN 6.5 and previous
versions. vSAN 6.6 and newer versions do not require multicast network trac.
Cluster Size
A vSAN cluster supports any number of physical hosts from two up to a maximum of 64 hosts in a
cluster. Multiple clusters can be managed by a single VMware vCenter Server instance. vSAN 2-node
congurations have two physical hosts. A stretched cluster can have up to 30 physical hosts (15 at
each site).
Turn-key appliances such as Dell EMC VxRail provide a fully integrated VMware hyper-converged
solution for a variety of applications and workloads. Simple deployment enables customers to be up
and running in as little as 15 minutes. Dell EMC VxRack SDDC powered by VMware provides an easy
path to a VMware software-dened data center supporting a wide variety of enterprise workloads.
Custom congurations using jointly validated components from all the major OEM vendors is also an
option. The vSAN Hardware Quick Reference Guide provides some sample server congurations as
directional guidance. All components should be validated using the VMware Compatibility Guide for
vSAN.
The ash device in the cache tier of a hybrid vSAN conguration is used for read caching and write
buering. 70% of the capacity is allocated for read cache and 30% for buering writes. Data is de-
staged from the cache tier to the capacity tier. The ash device in the cache tier enables very good
performance for a hybrid conguration.
In most cases, vSphere Storage vMotion can be used to migrate VMs between these various datastore
types. This feature makes it easy to migrate existing workloads when there is a need to perform
maintenance or retire and older storage solution. The rst part of this click-through demo shows how
simple it is to migrate VMs to a vSAN datastore using vSphere Storage vMotion: Migrating VMs to
vSAN .
Recommendation : Implement consistent hardware and software congurations across all hosts in a
vSAN cluster. Verify vMotion compatibility across all of the hosts in the cluster - see this VMware
Knowledge Base (KB) Article: Enhanced vMotion Compatibility (EVC) processor support (1003212) .
vSAN includes an iSCSI service to provide access to vSAN storage for non-VM workloads. Certied
solutions for le services are available through the VMware Ready for vSAN program. Organizations
can deploy these solutions with condence to extend HCI environments with proven, industry-leading
solutions. Using these solutions with vSAN provides benets such as simplied setup and
management, documented recommendations, and robust support.
Note : The witness component should not be confused with the witness host virtual appliance
discussed earlier in this document as they are two dierent items.
Multiple concatenated components are created for objects larger than 255GB. For example, a 750GB
virtual disk object consists of a minimum of three components.
If storage devices smaller than 255GB are used in the capacity tier, vSAN breaks the component down
into smaller parts. The image below shows the component distribution for a 250GB virtual disk with a
storage policy assigned that contains these rules:
RAID-1 mirroring
FTT=1
100% object space reservation
The cluster contains three hosts, one disk group per host, and each disk group has two 200GB
capacity drives. Note that the virtual disk is smaller than the maximum component size of 255GB.
Normally, there would be two 250GB components distributed across two hosts (plus a witness
component on a third host). In the scenario shown here, the capacity drives are 200GB each. vSAN
concatenated two smaller componentsone on each capacity drive of a host.
10
vSAN also features a local read cache, which is kept in memory on the host where the VM is running.
This helps avoid reads across the network and further improves performance considering the speed of
reading from memory is exponentially faster than reading from persistent storage devices.
1 http://www.qlogic.com/Resources/Documents/TechnologyBriefs/Adapters/
Tech_Brief_Introduction_to_Ethernet_Latency.pdf
2 http://www.intel.com/content/dam/www/public/us/en/documents/product-specications/ssd-dc-
p3700-spec.pdf
1.3 Availability
vSphere HA is tightly integrated with vSAN. The VMs that were running on a failed host are rebooted
on other healthy hosts in the cluster in a matter of minutes. A click-through demonstration of this
scenario is available here: vSphere HA and vSAN 50 VM Recovery.
As an example, a VM has a virtual disk with a data component on Host1, a second mirrored data
component on Host2, and a witness component on Host 3. Host1 is isolated from Host2 and Host3.
11
Host2 and Host3 are still connected over the network. Since Host2 and Host3 have greater than 50%
of the components (a data component and a witness component), the VMs virtual disk is accessible.
However, if all three hosts in our example above are isolated from each other, none of the hosts have
access to greater than 50% of the components. vSAN makes the object inaccessible until the hosts are
able to communicate over the network. This helps ensure data integrity.
Recommendation: Build your vSAN network with the same level of resiliency as any other storage
fabric.
When a device is degraded and error codes are sensed by vSAN, all of the vSAN components on the
aected drive are marked degraded and the rebuilding process starts immediately to restore
redundancy. If the device fails without warning (no error codes received from the device), vSAN will
wait for 60 minutes by default and then rebuild the aected data on other disks in the cluster. The 60-
minute timer is in place to avoid unnecessary movement of large amounts of data. As an example, a
disk is inadvertently pulled from the server chassis and reseated approximately 10 minutes later. It
would be inecient and resource intensive to begin rebuilding several gigabytes of data when the disk
is oine briey.
When failure of a device is anticipated due to multiple sustained periods of high latency, vSAN
evaluates the data on the device. If there are replicas of the data on other devices in the cluster, vSAN
will mark these components as absent. Absent components are not rebuilt immediately as it is
possible the cause of the issue is temporary. vSAN waits for 60 minutes by default before starting the
rebuilding process. This does not aect the availability of a VM as the data is still accessible using one
or more other replicas in the cluster. If the only replica of data is located on a suspect device, vSAN will
immediately start the evacuation of this data to other healthy storage devices.
Note: The failure of a cache tier device will cause the entire disk group to go oine. Another similar
scenario is a cluster with deduplication and compression enabled. The failure of any disk (cache or
capacity) will cause the entire disk group to go oine due to the way deduplicated data is distributed
across disks.
Recommendation: Consider the number and size of disk groups in your cluster with deduplication and
compression enabled. While larger disk groups might improve deduplication eciency, this also
increases the impact to the cluster when a disk fails. Requirements for each organization are dierent
so there is no set rule for disk group sizing.
12
redundancy in environments aected by the unplanned downtime. When additional resources come
back online, vSAN will continue the repair process to comply with storage policies.
Recommendation: Maintain enough free capacity or "slack space" for rebuild operations and other
activities such as storage policy changes, VM snapshots, and so on. 25-30% of the vSAN datastore
capacity is recommended. Example: If the vSAN datastore capacity is 20TB, slack space should be
approximately 5-6TB.
vSAN has a highly available control plane for health checks using the VMware Host Clienteven if
vCenter Server is oine. Hosts in a vSAN cluster cooperate in a distributed fashion to check the health
of the entire cluster. Any host in the cluster can be used to view vSAN Health. This provides
redundancy for the vSAN Health data to help ensure administrators always have this information
available.
Recommendation: Verify your data protection vendor supports the use of their product with vSAN.
13
vSAN 6.6 also includes local failure protection. RAID-1 mirroring or RAID-5/6 erasure coding can be
implemented within each stretched cluster site to provide local resiliency to disk and host failures. In
addition to providing higher levels of redundancy, this minimizes production and resynchronization
trac across the intersite link.
1.4 Security
A Key Management Server (KMS) is required to enable and use vSAN encryption. Nearly all KMIP-
compliant KMS vendors are compatible, with specic testing completed for vendors such as HyTrust,
Gemalto, Thales e-Security, CloudLink, and Vormetric.
For versions of vSAN prior to 6.6, self-encrypting drives (SEDs) can be used to encrypt data at rest.
vSphere VM encryption is also an option.
Note: VMs encrypted with vSphere VM encryption can be deployed to a vSAN datastore just like other
datastore types such as VMFS and NFS. However, vSAN space eciency features such as
deduplication and compression will provide little benet with these encrypted VMs.
Recommendation: Do not run the VMs that comprise a KMS cluster on the encrypted vSAN datastore.
14
1.5 Performance
15
example, consider a virtual disk that is 600GB. vSAN will stripe this object across three components
that are approximately 200GB each in size. In this case, the components might reside on the same
drive, on separate drives in the same disk group, or across multiple drives in separate disk groups and/
or hosts (unlike the Number of Disk Stripes per Object rule where striped components are always
striped across separate drives). Another reason vSAN might stripe an object across multiple
components is to improve the balance of drive utilization across a cluster. Splitting large components
into smaller components enables more exibility in placing these components across drives with
higher capacity utilization. The gure below shows a basic example of this.
HCIBench simplies and accelerates proof-of-concept (POC) performance testing in a consistent and
controlled way. The tool fully automates the process of deploying test VMs, coordinating workload
runs, aggregating test results, and collecting data for troubleshooting purposes. The output from
HCIBench can be analyzed by the Performance Diagnostics feature in vSAN 6.6.1 and newer versions
of vSAN. See this VMware Knowledge Base article for more information: vSAN Performance
Diagnostics (2148770)
HCIBench can be used to evaluate the performance of vSAN and other HCI storage solutions in a
vSphere environment.
Recommendation : Use HCIBench to run performance tests versus running a workload from a single
VM. HCIBench can be congured to deploy and distribute multiple VMs across the hosts in an HCI
cluster to provide more realistic and accurate test results.
16
1.6 Operations
Hosts containing local storage devices can be added to a vSAN cluster. Disk groups must be
congured for the new hosts after the hosts are added to the cluster. The additional capacity is
available for use after conguration of the disk groups. This scale-out approach is most
common and also adds compute capacity to the cluster.
More storage devices can be added to existing hosts assuming there is room in the servers
chassis to add these devices. After the storage devices are added, additional disk groups can be
created or existing disk groups recongured to use the new devices. This is considered a scale-
up approach.
Existing storage devices can be replaced with new, higher-capacity devices. Data should be
evacuated from the existing storage devices before replacing them. The evacuation of data is
performed using vSphere maintenance mode. This is also considered a scale-up approach.
Unlike traditional storage solutions, vSAN enables a just-in-time provisioning model. Storage and
compute capacity can be quickly provisioned as needed.
Customers enabling the Customer Experience Improvement Program (CEIP) feature with vSAN 6.6
receive additional benets through online health checks. These checks will be dynamically updated
from VMwares online system as new issues are identied, knowledge base articles are created, and
new best practices are discovered.
17
Details on how data is handled are provided in the vSAN documentation. In summary, the default
option, Ensure accessibility," is used when the host will be oine for a shorter period of time. For
example, during maintenance such as a rmware upgrade or adding memory to a host. Full data
migration is typically appropriate for longer periods (hours or days) of planned downtime or the host
is being permanently removed from the cluster. "No data migration" commonly allows the host to
enter maintenance mode in the shortest amount of time. However, any objects with Primary Level of
Failures to Tolerate (PFTT) set to zero with components on the host going into maintenance mode are
inaccessible until the host is back online.
Note: As mentioned above, recent versions of vSAN take into account capacity when reporting cluster
impact. It does not factor in the number of fault domains. Entering host maintenance mode with the
"Full data migration" option will fail if there is an insucient number of fault domains available to
satisfy storage policies (after a host enters maintenance mode).
18
Recommendation: Read the vSphere Upgrade Guide and product release notes prior to performing an
upgrade.
By default, vSAN will ensure data is compliant with storage policies during the operation. If there is not
enough free capacity in other disk groups, the operation will fail. Clicking Allow Reduced
Redundancy when enabling deduplication and compression or encryption allows vSAN to reduce the
number of copies of data temporarily, if needed, to complete the requested operation.
Allow Reduced Redundancy is more commonly required in small vSAN clusters such as three or four
hosts with one disk group each. This option might also be required if the free capacity in the cluster is
low.
19
The upgrade process is automated. Simply use the Remediate option in vSphere Update Manager to
perform a rolling upgrade of the cluster. vSphere Update Manager migrates virtual machines from the
host being upgraded to other hosts in the cluster with no downtime.
20