Anda di halaman 1dari 36

Kubernetes - vSphere Cloud Provider

October 30, 2017

1

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

  • 1. Get Started

1.1.Introduction

  • 2. vSphere Cloud Provider

2.1.Overview

Table of Contents

2.2.Kubernetes Storage Support 2.3.Storage Policy Based Management for dynamic provisioning of volumes 2.4.High Availability of Kubernetes Cluster

  • 3. Deployment

3.1.Prerequisites

3.2.Kubernetes Anywhere 3.3.Congurations on Existing Kubernetes Cluster 3.4.Best Practices

  • 4. Applications & Examples 4.1.Running Stateful application - Guestbook App 4.2.Deploying Sharded MongoDB Cluster 4.3.Deploying S3 Stateful Containers - Minio

  • 5. Miscellaneous

5.1.FAQs

5.2.Known Issues

2

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

1. Get Started

Get Started

3

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

1.1 Introduction

Introduction

Containers have changed the way applications are packaged and deployed. Not only containers are ecient from infrastructure utilization point of view, they also provide strong isolation between process on same host. They are lightweight and once packaged can run anywhere. Docker is the most commonly used container runtime technology and this user guide outlines how vSphere is compatible with Docker ecosystem.

Persistent Storage in Container World

Although it is relatively easy to run stateless Microservices using container technology, stateful applications require slightly dierent treatment. There are multiple factors which need to be considered when you think about handling persistent data using containers such as:

• Containers are ephemeral by nature, so the data that needs to be persisted has to survive through the restart/re-scheduling of a container. • When containers are re-scheduled, they can die on one host and might get scheduled on a dierent host. In such a case the storage should also be shifted and made available on new host for the container to start gracefully. • The application should not have to worry about the volume/data and underlying infrastructure should handle the complexity of unmounting and mounting etc. • Certain applications have a strong sense of identity (For example. Kafka, Elastic etc.) and the disk used by a container with certain identity is tied to it. It is important that if a container with a certain ID gets re-scheduled for some reason then the disk only associated with that ID is re- attached on a new host.

Kubernetes Storage Primitives

Kubernetes provides abstractions ensure that the storage details are separated from allocation and usage of storage. Kubernetes has various API resources to provision, categorize and consume the storage.

API Resources:

• Volumes • Persistent Volumes • Persistent Volumes Claims • Storage Class • Statefulsets

Please refer Kubernetes documentation for details.

4

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

2. vSphere Cloud Provider

vSphere Cloud Provider

5

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

2.1 Overview

Overview

Containers are stateless and ephemeral but applications are stateful and need persistent storage. vSphere adds this persistent storage support to Kubernetes through interface called Cloud Provider. Cloud provider is an interface which helps in extending Kubernetes with cluster of instances managed by virtualization technologies, public/private cloud platforms and required networking for these instances.

Kubernetes - vSphere Cloud Provider 2.1 Overview Overview Containers are stateless and ephemeral but applications are

Kubernetes cloud provider is an interface to integrate various nodes (i.e. hosts), load balancers and networking routes. This interface allows extending Kubernetes to use various cloud and virtualization solutions as base infrastructure to run on.

Cloud provider provides following interfaces to eectively integrate cloud platforms:

Instances - interface for virtual machine management • Load Balancers - interface to integrate with load balancer provided by cloud platform • Routes - interface to add new routing rules of cloud platform • Zones - integrate with zones if implemented by cloud platform

vSphere Storage Concepts

vSphere is one of the cloud providers of Kubernetes and thus allows Kubernetes Pods use enterprise grade storage. vSphere Storage (VMFS, vSAN, NFS) has proven features like policy based management, QoS, high availability and data reliability for containers using Kubernetes.

6

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Datastores is an abstraction which hides storage details and provide uniform interface for storing persistent data. Datastores enables simplied storage management with features like grouping them in folders. Depending upon the backend storage the datastores can be vSAN, VMFS or NFS.

• vSAN is a hyper-converged infrastructure storage which provides excellent performance as well as reliability. vSAN advantage is simplied storage management with features like policy driven administration. • VMFS (VMware FIle System) is a cluster le system that allows virtualization to scale beyond a single node for multiple VMware ESX servers. VMFS increases resource utilization by providing shared access to pool of storage. • NFS (Network File System) is distributed le protocol to access storage over network like local storage. vSphere supports NFS as backend to store virtual machines les.

Kubernetes Storage support

vSphere supports all the the Kubernetes storage primitives which brings enterprise grade storage features to Kubernetes. vSphere uses vSAN and VMFS datastores, these datastores store the volumes as VMDK les.

Kubernetes volumes are dened in Pod specication. They reference VMDK les and these VMDK les are mounted as volumes when the container is running. When the Pod is deleted the Kubernetes volume is unmounted and the data in VMDK les persists

Kubernetes - vSphere Cloud Provider Datastores is an abstraction which hides storage details and provide uniform

vSphere is cloud provider which implements Instances interface and supports following Kubernetes storages primitives:

• Volumes • Persistent Volumes (PV)

7

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

• Persistent Volumes Claims (PVC) • Storage Class • Stateful Sets

2.2 Kubernetes Storage Support

Volumes

A Pod can specify vsphereVolume as Kubernetes Volumes and then vSphere VMDK is mounted as Volume into your Pod. The contents of a volume are preserved when it is unmounted. It supports both VMFS and VSAN datastores.

All the example yamls can be found here unless otherwise specied. Please download these examples.

Here is an example of how to create a VMDK le and how a Pod can use it.

Create VMDK

First ssh into ESX and then use following command to create vmdk on datastore1

vmkfstools -c 2G /vmfs/volumes/datastore1/volumes/myDisk.vmdk

Create Pod which uses vSphere Volume ‘myDisk.vmdk’

#vsphere-volume-pod.yaml

apiVersion: v1 kind: Pod metadata:

name: test-vmdk spec:

containers:

  • - image: gcr.io/google_containers/test-webserver name: test-container volumeMounts:

- mountPath: /test-vmdk name: test-volume volumes:

  • - name: test-volume # This VMDK volume must already exist. vsphereVolume:

volumePath: "[datastore1] volumes/myDisk" fsType: ext4

Create the pod

$ kubectl create -f vsphere-volume-pod.yaml

Verify that pod is running

$ kubectl get pods test-vmdk

NAME

READY

STATUS

RESTARTS

AGE

test-vmdk

1/1

Running

0

48m

Persistent Volumes & Persistent Volumes Claims

In case of Kubernetes Volumes we know that once the Pod is deleted the specication of the volume in the Pod is also lost. Even though VMDK le persists but from Kubernetes perspective the volume is deleted.

8

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Persistent Volumes API resource solves this problem where PVs have lifecycle independent of the Pods and not created when Pod is run. PVs are unit of storage which we provision in advance, they are Kubernetes objects backed by some storage, vSphere in this case. PVs are created, deleted using kubectl commands.

In order to use these PVs user needs to create PersistentVolumeClaims which is nothing but a request for PVs. A claim must specify the access mode and storage capacity, once a claim is created PV is automatically bound to this claim. Kubernetes will bind a PV to PVC based on access mode and storage capacity but claim can also mention volume name, selectors and volume class for a better match. This design of PV-PVCs not only abstract storage provisioning and consumption but also ensures security through access control.

Note:

All the example yamls can be found here unless otherwise specied. Please download these examples.

Here is an example of how to use PV and PVC to add persistent storage to your Pods.

Create VMDK

First ssh into ESX and then use following command to create vmdk,

vmkfstools -c 2G /vmfs/volumes/datastore1/volumes/myDisk.vmdk

Create Persistent Volume

#vsphere-volume-pv.yaml

apiVersion: v1 kind: PersistentVolume metadata:

name: pv0001 spec:

capacity:

storage: 2Gi accessModes:

- ReadWriteOnce persistentVolumeReclaimPolicy: Retain vsphereVolume:

volumePath: "[datastore1] volumes/myDisk" fsType: ext4

In the above example datastore1 is located in the root folder. If datastore is member of Datastore Cluster or located in sub folder, the folder path needs to be provided in the VolumePath as below.

vsphereVolume:

VolumePath:"[DatastoreCluster/datastore1] volumes/myDisk"

Create the persistent volume

$ kubectl create -f vsphere-volume-pv.yaml

Verify persistent volume is created

$ kubectl describe pv pv0001

Name:pv0001

Labels:<none>

Status:Available

Claim:

Reclaim Policy:Retain Access Modes:RWO

Capacity:2Gi

Message:

9

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Source:

Type:vSphereVolume (a Persistent Disk resource in vSphere) VolumePath:[datastore1] volumes/myDisk

FSType:ext4

No events.

Create Persistent Volume Claim

#vsphere-volume-pvc.yaml

kind: PersistentVolumeClaim apiVersion: v1 metadata:

name: pvc0001 spec:

 

accessModes:

 

-

ReadWriteOnce

 

resources:

requests:

storage: 2Gi

Create the persistent volume claim

$ kubectl create -f vsphere-volume-pvc.yaml

Verify persistent volume claim is created

$ kubectl describe pvc pvc0001

Name:pvc0001

 

Namespace:default Status:Bound

 

Volume:pv0001

Labels:<none>

Capacity:2Gi

Access Modes:RWO No events.

Create Pod which uses Persistent Volume Claim

#vpshere-volume-pvcpod.yaml

apiVersion: v1 kind: Pod metadata:

 

name: pvpod spec:

containers:

-

name: test-container image: gcr.io/google_containers/test-webserver

volumeMounts:

-

name: test-volume mountPath: /test-vmdk

volumes:

-

name: test-volume persistentVolumeClaim:

 

claimName: pvc0001

Create the pod

$ kubectl create -f vsphere-volume-pvcpod.yaml

Verify pod is created

10

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

$ kubectl get pod pvpod

NAME

READY

STATUS

RESTARTS

AGE

pvpod

1/1

Running

0

48m

StatefulSets

StatefulSets are valuable for applications which require any stable identiers or stable storage. vSphere Cloud Provider suppoorts StatefulSets and vSphere volumes can be consumed by StatefulSets.

Note:

All the example yamls can be found here unless otherwise specied. Please download these examples.

Create a storage class that will be used by the volumeClaimTemplates of a Stateful Set

#simple-storageclass.yaml

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

name: thin-disk provisioner: Kubernetes.io/vsphere-volume parameters:

diskformat: thin

Create a Stateful set that consumes storage from the Storage Class created

#simple-statefulset.yaml

---

apiVersion: v1 kind: Service metadata:

name: nginx labels:

app: nginx spec:

ports:

- port: 80 name: web clusterIP: None selector:

app: nginx

---

apiVersion: apps/v1beta1 kind: StatefulSet metadata:

name: web spec:

serviceName: "nginx" replicas: 14 template:

metadata:

labels:

app: nginx

spec:

containers:

11

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

- name: nginx image: gcr.io/google_containers/nginx-slim:0.8 ports:

  • - containerPort: 80 name: web volumeMounts:

  • - name: www mountPath: /usr/share/nginx/html volumeClaimTemplates:

- metadata:

name: www annotations:

volume.beta.kubernetes.io/storage-class: thin-disk spec:

accessModes: [ "ReadWriteOnce" ] resources:

requests:

storage: 1Gi

This will create Persistent Volume Claims for each replica and provision a volume for each claim if an existing volume could be bound to the claim.

2.3 Storage Policy Based Management for dynamic provisioning of volumes

Overview

One of the most important features of vSphere for Storage Management is Policy based Management. Storage Policy Based Management (SPBM) is a storage policy framework that provides a single unied control plane across a broad range of data services and storage solutions. SPBM enables vSphere administrators to overcome upfront storage provisioning challenges, such as capacity planning, dierentiated service levels and managing capacity headroom

As we discussed in previously StorageClass species provisioner and parameters. And using these parameters you can dene the policy for that particular PV which will be dynamically provisioned.

You can specify the existing vCenter Storage Policy Based Management (SPBM) policy to congure a persistent volume with SPBM policy. storagePolicyName parameter is used for this.

Note:

• SPBM policy based provisioning of persistent volumes will be available in 1.7.x release.** • All the example yamls can be found here unless otherwise specied. Please download these examples.

Create Storage Class

#vsphere-volume-spbm-policy.yaml

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

name: fast provisioner: kubernetes.io/vsphere-volume parameters:

diskformat: zeroedthick storagePolicyName: gold

12

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

The admin species the SPBM policy - “gold” as part of storage class denition for dynamic volume provisioning. When a PVC is created, the persistent volume will be provisioned on a compatible datastore with maximum free space that satises the “gold” storage policy requirements.

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

name: fast provisioner: kubernetes.io/vsphere-volume parameters:

diskformat: zeroedthick storagePolicyName: gold datastore: VSANDatastore

The admin can also specify a custom datastore where he wants the volume to be provisioned along with the SPBM policy name. When a PVC is created, the vSphere Cloud Provider checks if the user specied datastore satises the “gold” storage policy requirements. If yes, it will provision the persistent volume on user specied datastore. If not, it will error out to the user that the user specied datastore is not compatible with “gold” storage policy requirements.

Virtual SAN policy support

Vsphere Infrastructure(VI) Admins will have the ability to specify custom Virtual SAN Storage Capabilities during dynamic volume provisioning. You can now dene storage requirements, such as performance and availability, in the form of storage capabilities during dynamic volume provisioning. The storage capability requirements are converted into a Virtual SAN policy which are then pushed down to the Virtual SAN layer when a persistent volume (virtual disk) is being created. The virtual disk is distributed across the Virtual SAN datastore to meet the requirements.

The ocial vSAN policy documentation describes in detail about each of the individual storage capabilities that are supported by vSAN. The user can specify these storage capabilities as part of storage class denition based on his application needs.

For vSAN policies few additional parameters in StorageClass can be specied:

cacheReservation: Flash capacity reserved as read cache for the container object. Specied as a percentage of the logical size of the virtual machine disk (vmdk) object. Reserved ash capacity cannot be used by other objects. Unreserved ash is shared fairly among all objects. Use this option only to address specic performance issues.

diskStripes: The minimum number of capacity devices across which each replica of a object is striped. A value higher than 1 might result in better performance, but also results in higher use of system resources. Default value is 1. Maximum value is 12.

forceProvisioning: If the option is set to Yes, the object is provisioned even if theNumber of failures to tolerate, Number of disk stripes per object, and Flash read cache reservation policies specied in the storage policy cannot be satised by the datastore

hostFailuresToTolerate: Denes the number of host and device failures that a virtual machine object can tolerate. For n failures tolerated, each piece of data written is stored in n+1 places, including parity copies if using RAID 5 or RAID 6.

iopsLimit: Denes the IOPS limit for an object, such as a VMDK. IOPS is calculated as the number of I/O operations, using a weighted size. If the system uses the default base size of 32 KB, a 64-KB I/O represents two I/O operations

13

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

objectSpaceReservation: Percentage of the logical size of the virtual machine disk (vmdk) object that must be reserved, or thick provisioned when deploying virtual machines. Default value is 0%. Maximum value is 100%.

Note:

• Here you don’t need to create persistent volume it is created dynamically • vSAN storage capability based provisioning of persistent volumes is available in 1.6.5 release.

Create Storage Class

#vsphere-volume-sc-vsancapabilities.yaml

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

name: fast provisioner: kubernetes.io/vsphere-volume parameters:

diskformat: zeroedthick hostFailuresToTolerate: "2" cachereservation: "20"

Here a persistent volume will be created with the Virtual SAN capabilities - hostFailuresToTolerate to 2 and cachereservation is 20% read cache reserved for storage object. Also the persistent volume will be zeroedthickdisk.

The ocial vSAN policy documentation describes in detail about each of the individual storage capabilities that are supported by vSAN and can be congured on the virtual disk. You can also specify the datastore in the Storageclass as shown in above example. The volume will be created on the datastore specied in the storage class. This eld is optional. If not specied as shown in example 1, the volume will be created on the datastore specied in the vsphere cong le used to initialize the vSphere Cloud Provider.

#vsphere-volume-sc-vsancapabilities.yaml

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

name: fast provisioner: kubernetes.io/vsphere-volume parameters:

diskformat: zeroedthick datastore: VSANDatastore hostFailuresToTolerate: "2" cachereservation: "20"

Note: If you do not apply a storage policy during dynamic provisioning on a vSAN datastore, it will use a default Virtual SAN policy.

Create the storageclass

$ kubectl create -f vsphere-volume-sc-vsancapabilities.yaml

Verify storage class is created

$ kubectl describe storageclass fast Name:fast Annotations:<none> Provisioner:kubernetes.io/vsphere-volume Parameters:diskformat=zeroedthick, hostFailuresToTolerate="2", cachereservation="20" No events.

14

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Create Persistent Volume Claim

kind: PersistentVolumeClaim apiVersion: v1 metadata:

name: pvcsc-vsan annotations:

volume.beta.kubernetes.io/storage-class: fast spec:

accessModes:

- ReadWriteOnce resources:

requests:

storage: 2Gi

Create the persistent volume claim

$ kubectl create -f vsphere-volume-pvcsc.yaml

Verifying persistent volume claim is created

$ kubectl describe pvc pvcsc-vsan Name:pvcsc-vsan Namespace:default Status:Bound

Volume:pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d

Labels:<none>

Capacity:2Gi

Access Modes:RWO No events.

Persistent Volume is automatically created and is bounded to this pvc

Verify if persistent volume claim is created

$ kubectl describe pv pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d

Name:pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d

Labels:<none> Status:Bound Claim:default/pvcsc-vsan Reclaim Policy:Delete Access Modes:RWO

Capacity:2Gi

Message:

Source:

Type:vSphereVolume (a Persistent Disk resource in vSphere) VolumePath:[VSANDatastore] kubevols/kubernetes-dynamic-

pvc-80f7b5c1-94b6-11e6-a24f-005056a79d2d.vmdk

FSType:ext4

No events.

Note: VMDK is created inside kubevols folder in datastore which is mentioned in ‘vsphere’ cloudprovider conguration. The cloudprovider cong is created during setup of Kubernetes cluster on vSphere.

Create Pod which uses Persistent Volume Claim with storage class

#vsphere-volume-pvcscpod.yaml

apiVersion: v1

kind: Pod

metadata:

name: pvpod

15

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

spec:

containers:

  • - name: test-container image: gcr.io/google_containers/test-webserver volumeMounts:

- name: test-volume mountPath: /test volumes:

  • - name: test-volume persistentVolumeClaim:

claimName: pvcsc-vsan

Create the pod

$ kubectl create -f vsphere-volume-pvcscpod.yaml Verifying pod is created:

$ kubectl get pod pvpod

NAME

READY

STATUS

RESTARTS

AGE

pvpod

1/1

Running

0

48m

2.4 High Availability of Kubernetes Cluster

Kubernetes ensures that all the the Pods are restarted in case the node goes down. The persistent storage API objects ensure that same PVs are mounted back to the new Pods on restart or if they are recreated.

But what happens if the node/host is the VM and the physical host fails? vSphere HA leverages multiple ESXi hosts congured as a cluster to provide rapid recovery from outages and cost-eective high availability for applications running in virtual machines. vSphere HA provides a base level of protection for your virtual machines by restarting virtual machines in the event of a host failure

Applications running on Kubernetes on vSphere can take advantage of vSphere Availability ensuring resilient and highly available applications.

Node VM Failure: Node VM failure will cause Kubernetes to recreate a new pod to run the containers. vSphere Cloud Provider will mount the disk to a live node and unmount disk from the dead node automatically. The validation description is as follows:

• Shutdown one Kubernetes node VM. This will cause Kubernetes to remove the node VM from the Kubernetes cluster.

• The Kubernetes cluster will recreate the pod on an idle node in the original cluster after the simulated node failure. Kubernetes vSphere Cloud Provider will:

Mount the disks from the shutdown node VM to the idle node. Unmount the disks from the powered onode VM. • Fix the issue of the node VM (if any) and power it on. Kubernetes will add the node back to the original cluster and it will be available for new pod creation.

Physical Host Failure: Powering oone of the ESXi hosts will cause the vSphere Availability to restart the node on one of the running ESXi servers. The node in Kubernetes cluster will temporarily change to UNKNOWN. After less than two minutes, the node will be available in the cluster. No pod recreation is required.

Failure Components

Recovery

Time

Kubernetes node

shutdown or

< 1 min

corruption

Result and Behavior

The pod is recreated, and Kubernetes vSphere Cloud Provider automatically unmounts/mounts the disk to standby node. The standby node changes to working node

16

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Failure Components

ESXi host failure or powered o

Recovery

Time

< 2 mins

Result and Behavior

vSphere Availability restarted the Kubernetes node on another ESXi host; no node recreation required.

Please note the recovery time is dependent upon the hardware. For additional details please refer this

17

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

3. Deployment

Deployment

18

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

  • 3.1 Prerequisites

Following is the list of prerequisites for running Kubernetes with vSphere Cloud Provider:

• We recommend Kubernetes version 1.6.5 and vSphere version 6.0.x • We support Photon, Ubuntu, CoreOs and Centos. it has been tested on Photon 1.0 GA, Ubuntu 16.04, CoreOS 4.11.2 and Centos 7.3 • vSphere setup to deploy the Kubernetes cluster. • vSphere shared storage. It can be any one of VSAN, sharedVMFS, sharedNFS. Shared storage allows all the nodes in the Kubernetes cluster to access the storage blocks. • vCenter user with required set of privileges. To know privileges required to install Kubernetes Cluster using Kubernetes-Anywhere, please check this link If you are not installing Kubernetes cluster using Kubernetes-Anywhere and just enabling vSphere Cloud Provider on Kubernetes cluster already deployed on vSphere, Please refer this link to know about privileges required for Cloud Provider. • VMware Tools needs to be installed on the guest operating system on each Node VM. Please refer this link for instruction on installing VMware tools. • Node VM name requirements VM names can not begin with numbers. VM names can not have capital letters, any special characters except . and -. VM names can not be shorter than 3 chars and longer than 63 • The disk.EnableUUID parameter must be set to “TRUE” for each Node VM. Please refer this

  • 3.2 Kubernetes Anywhere

Kubernetes Anywhere

There are several deployment mechanisms available to deploy Kubernetes. Kubernetes-Anywhere is one of them. Please refer this link

We have simplied deployment with creating pre-build Kubernetes-Anywhere docker image

User can just pull this image and run a container with this image to get into deployment wizard. User can ll in the questionnaires, Kubernetes-Anywhere will create a terraform deployment script to install Kubernetes cluster on vSphere. After the cluster is deployed, cluster cong le is available at /opt/

Kubernetes-anywhere/phase1/vsphere/.tmp/kubecong.json

Make sure to copy this le, before stopping the deployment container. This le is used to access Kubernetes cluster using kubectl.

Please refer installation steps mentioned in the getting started guide for deploying Kubernetes using

  • 3.3 Congurations on Existing Kubernetes Cluster

Congurations on Existing Kubernetes Cluster

If a Kubernetes cluster has not been deployed using Kubernetes-Anywhere, follow the instructions below to enable the vSphere Cloud Provider. These steps are not needed when using Kubernetes- Anywhere, they will be done as part of the deployment.

19

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Create a VM folder for Node VMs

Create a VM folder. Follow instructions mentioned in this link and move Kubernetes Node VMs to this folder.

Make sure node VM names are compliant

Make sure Node VM names must comply with the regex [a-z](([-0-9a-z]+)?[0-9a-z])?(\.[a- z0-9](([-0-9a-z]+)?[0-9a-z])?)* If Node VMs does not comply with this regex, rename them and make it compliant to this regex.

Node VM names constraints:

• VM names can not begin with numbers. • VM names can not have capital letters, any special characters except . and -. • VM names can not be shorter than 3 chars and longer than 63

Enable disk UUID on Node virtual machines

The disk.EnableUUID parameter must be set to “TRUE” for each Node VM. This step is necessary so that the VMDK always presents a consistent UUID to the VM, thus allowing the disk to be mounted properly.

For each of the virtual machine nodes that will be participating in the cluster, follow the steps below using GOVC tool

• Set up GOVC environment

export GOVC_URL='vCenter IP OR FQDN' export GOVC_USERNAME='vCenter User' export GOVC_PASSWORD='vCenter Password' export GOVC_INSECURE=1

• Find Node VM Paths

govc ls /datacenter/vm/<vm-folder-name>

• Set disk.EnableUUID to true for all VMs

govc vm.change -e="disk.enableUUID=1" -vm='VM Path'

Note: If Kubernetes Node VMs are created from template VM then disk.EnableUUID=1 can be set on the template VM. VMs cloned from this template, will automatically inherit this property.

Create Roles, add Privileges to Roles and assign them to the vSphere Cloud Provider user and vSphere entities

Note: if you want to use Administrator account then this step can be skipped.

vSphere Cloud Provider requires the following minimal set of privileges to interact with vCenter. Please refer vSphere Documentation Center to know about steps for creating a Custom Role, User and Role Assignment.

20

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Roles

Privileges

Resource.AssignVMToPool

System.Anonymous

System.Read

System.View

manage-k8s-

VirtualMachine.Cong.AddExistingDisk

node-vms

VirtualMachine.Cong.AddNewDisk

VirtualMachine.Cong.AddRemoveDevice

VirtualMachine.Cong.RemoveDisk

VirtualMachine.Inventory.Create

VirtualMachine.Inventory.Delete

Datastore.AllocateSpace

manage-k8s-

volumes

Datastore.FileManagement

System.Anonymous

System.Read

 

System.View

StorageProle.View

k8s-system-

read-and-spbm-

prole-view

System.Anonymous

System.Read

System.View

System.Anonymous

ReadOnly

System.Read

System.View

Entities

Propagate to

Children

Cluster,

Hosts,

VM Folder

Yes

Datastore

No

vCenter

No

Datacenter,

Datastore

Cluster,

Datastore

No

Storage Folder

Step-5 Create the vSphere cloud cong le (vsphere.conf). Cloud cong template can be found here

This cong le needs to be placed in the shared directory which should be accessible from kubelet container, controller-manager pod, and API server pod.

vsphere.conf for Master Node:

[Global]

user = "vCenter username for cloud provider" password = "password" server = "IP/FQDN for vCenter" port = "443" #Optional insecure-flag = "1" #set to 1 if the vCenter uses a self-signed cert datacenter = "Datacenter name" datastore = "Datastore name" #Datastore to use for provisioning volumes using storage dynamic provisioning working-dir = "vCenter VM folder path in which node VMs are located" vm-name = "VM name of the Master Node" #Optional vm-uuid = "UUID of the Node VM" # Optional

[Disk] scsicontrollertype = pvscsi

Note: vm-name parameter is introduced in 1.6.4 release. Both vm-uuid and vm-name are optional parameters. If vm-name is specied then vm-uuid is not used. If both are not specied then kubelet will get vm-uuid from /sys/class/dmi/id/product_serial and query vCenter to nd the Node VM’s name.

21

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

vsphere.conf for Worker Nodes: (Only Applicable to 1.6.4 release and above. For older releases this le should have all the parameters specied in Master node’s vSphere.conf le)

[Global]

vm-name = "VM name of the Worker Node"

Below is summary of supported parameters in the vsphere.conf le

user is the vCenter username for vSphere Cloud Provider. • password is the password for vCenter user specied with user. • server is the vCenter Server IP or FQDN • port is the vCenter Server Port. Default is 443 if not specied. • insecure-flag is set to 1 if vCenter used a self-signed certicate. • datacenter is the name of the datacenter on which Node VMs are deployed. • datastore is the default datastore to use for provisioning volumes using storage classes/ dynamic provisioning.

vm-name is recently added conguration parameter. This is optional parameter. When this parameter is present, vsphere.conf le on the worker node does not need vCenter credentials.

Note: vm-name is added in the release 1.6.4. Prior releases does not support this parameter.

working-dir can be set to empty ( working-dir = “”), if Node VMs are located in the root VM folder.

vm-uuid is the VM Instance UUID of virtual machine. vm-uuid can be set to empty (vm-uuid = ""). If set to empty, this will be retrieved from /sys/class/dmi/id/product_serial le on virtual machine (requires root access).

vm-uuid needs to be set in this format - 423D7ADC-F7A9-F629-8454-CE9615C810F1

vm-uuid can be retrieved from Node Virtual machines using following command. This will be dierent on each node VM.

cat /sys/class/dmi/id/product_serial | sed -e 's/^VMware-//' - e 's/-/ /' | awk '{ print toupper($1$2$3$4 "-" $5$6 "-" $7$8 "-" $9$10 "-" $11 $12$13$14$15$16) }'

datastore is the default datastore used for provisioning volumes using storage classes. If datastore is located in storage folder or datastore is member of datastore cluster, make sure to specify full datastore path. Make sure vSphere Cloud Provider user has Read Privilege set on the datastore cluster or storage folder to be able to nd datastore. .

• For datastore located in the datastore cluster, specify datastore as mentioned below

datastore = "DatastoreCluster/datastore1"

• For datastore located in the storage folder, specify datastore as mentioned below

datastore = "DatastoreStorageFolder/datastore1"

Add ags to controller-manager, API server and Kubelet

Add ags to controller-manager, API server and Kubelet to enable vSphere Cloud Provider. * Add following ags to kubelet running on every node and to the controller-manager and API server pods manifest les.

--cloud-provider=vsphere --cloud-config=<Path of the vsphere.conf file>

22

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Manifest les for API server and controller-manager are generally located at /etc/kubernetes

Restart Kubelet on all nodes

• Reload kubelet systemd unit le using systemctl daemon-reload • Restart kubelet service using systemctl restart kubelet.service

Note: After enabling the vSphere Cloud Provider, Node names will be set to the VM names from the vCenter Inventory.

3.4 Best Practices

Best Practices

This section describes the vSphere specic congurations:

vSphere HA

• vSphere Cloud Provider supports vSphere HA. To ensure high availability of node VMs, it is recommended to enable HA on the cluster. Details

Use Resource Pool

• It is recommended to place Kubernetes node VMs in the resource pool to ensure guaranteed performance. Details

23

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

4. Applications & Examples

Applications & Examples

24

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

4.1 Running Stateful application - Guestbook App

Guestbook is PHP application with Redis as backend. In this section, we will demonstrate how to use Kubernetes deployed on vSphere to run Guestbook Application with persistent storage. At the end of this demo, you will have a sample guestbook app running inside Kubernetes where the data is resident inside VMDKs managed by vSphere.

The data in the VMDK is independent of the lifecycle of the pods and persists even if pods are deleted.

Storage Setup

The backing VMDK les are needed when dynamic provisioning is not used. The VMDK les need to exist before creating the service in k8s that uses them.

Login to ESX (if more than one ESX is used make sure to have a common/shared datastore) and create the volumes

# VMFS cd /vmfs/volumes/datastore1/ # VSAN cd /vmfs/volumes/vsanDatastore/

# VMFS mkdir kubevols # Not needed but good hygiene # VSAN /usr/lib/vmware/osfs/bin/osfs-mkdir kubevols # Needed

cd kubevols

vmkfstools -c 2G redis-slave.vmdk vmkfstools -c 2G redis-master.vmdk

Guestbook with statically provisioning

In this example we will provision PVs and Guestbook pods will claim these PVs

Download all les in the PVC PV Demo from here

Create the persistent volume

kubectl create -f redis-master-pv.yaml create -f redis-slave-pv.yaml

Check the persistent volume records

$ kubectl get pv

Create the PVC

$ kubectl create -f redis-slave-claim.yaml $ kubectl create -f redis-master-claim.yaml

Check that the volumes are bound to the claims

$ kubectl get pv

NAME

CAPACITY ACCESSMODES RECLAMIPOLICY STATUS CLAIM REASON AGE

redis-master-pv 2Gi RWO

Retain

Bound

default/redis-master-

claim

34s

redis-slave-pv 2Gi

RWO

Retain

Bound

default/redis-slave-

claim

34s

Create the guestbook

25

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

kubectl create -f guestbook-all-in-one.yaml

Guestbook Application with Dynamic Provisioning

Download the yaml les for storage class based guestbook from here

Create the storage class

$ kubectl create -f redis-sc.yaml

Create the claims that use the storage class

$kubectl create -f redis-master-claim.yaml create -f redis-slave-claim.yaml

This should trigger creation of storage volumes on the datastore congured for the vSphere Cloud Provider.

[root@promc-2n-dhcp41:/vmfs/volumes/57f5768f-856aa050-fde c-e0db55248054/ kubevols] ls *-pvc-* kubernetes-dynamic-pvc-91c9cd60-9a7c-11e6- a431-00505690d6 4d-flat.vmdk kubernetes-dynamic-pvc-a6abf43a-9a7c-11e6- a 431-00505690d64d-flat.vmdk kubernetes-dynamic-pvc-91c9cd60-9a7c-11e6- a431-00505690d6 4d.vmdk kubernetes-dynamic-pvc-a6abf43a-9a7c-11e6- a 431-00505690d64d.vmdk

All the dynamically provisioned volumes will be created in the kubevols directory inside the datastore.

Create the guestbook app consuming the claims

$ kubectl create -f guestbook-all-in-one.yaml

Verify Guestbook Application The following steps verify that the Pods, storage and the application are running correctly

Wait till the pods are ready

$ kubectl get pods NAME

READY STATUS

RESTARTS AGE

frontend-88237173-77e6n

1/1

Running

0

43m

redis-master-1145698066-2hbbk 1/1

Running

0

43m

redis-slave-5845680045-3ffdg 1/1

Running

0

43m

Get the port on which the front end is accessible

$ kubectl describe service frontend | grep NodePort Type: NodePort NodePort: <unset> 31531/TCP

Check the nodes on which the frontend is running

Get the IP address of one of the nodes (This can also be obtained from vCenter) Also possible from vCenter or the console output kube-up

$ kubectl describe node kubernetes-minion-1|grep -i Address Addresses: 10.20.105.59

Combine the IP and Port and head to the URL (Ex: http://10.20.105.59:31531 ). Try out the app and leave a few messages. To check the attach status of VMDKs on ESX head to vCenter (Settings page or Recent Tasks window)

26

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Lifecycle of volume independent Pod

To test the persistence, delete the guestbook and recreate it using the same volumes, the old messages on the guest book should show up.

$ kubectl delete -f guestbook-all-in-one.yaml $ kubectl create -f guestbook-all-in-one.yaml

After create, the port mapping will change.

# Get the new port $ kubectl describe service frontend| grep NodePort

Type:

NodePort

NodePort:

<unset> 32400/TCP

Check the nodes on which the frontend is running

4.2 Deploying Sharded MongoDB Cluster

This section describes the steps to create persistent storage for containers to be consumed by MongoDB services on vSAN. After these steps are completed, Cloud Provider will create the virtual disks (volumes in Kubernetes) and mount them to the Kubernetes nodes automatically. The virtual disks are created with the vSAN default policy.

Dene StorageClass

A StorageClass provides a mechanism for the administrators to describe the “classes” of storage they oer. Dierent classes map to quality-of-service levels, or to backup policies, or to arbitrary policies determined by the cluster administrators. The YAML format denes a “platinum” level StorageClass.

kind: StorageClass apiVersion: storage.k8s.io/v1beta1 Metadata:

name: platinum provisioner: Kubernetes.io/vsphere-volume diskformat: thin

Note: Although all volumes are created on the same vSAN datastore, you can adjust the policy according to actual storage capability requirement by modifying the vSAN policy in vCenter Server. User can also specify VSAN storage capabilities in StorageClass denition based on this application needs. Please refer to VSAN storage capability section mentioned in vSphere CP document

Claim Persistent Volume

A PersistentVolumeClaim (PVC) is a request for storage by a user. Claims can request specic size and access modes (for example, can be mounted once read/write or many times read-only). The YAML format claims a 128GB volume with read and write capability.

apiVersion: v1 kind: PersistentVolumeClaim metadata:

name: pvc128gb annotations:

volume.beta.Kubernetes.io/storage-class:

"platinum"

spec:

accessModes:

- ReadWriteOnce

resources:

requests:

storage: 128Gi

27

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Specify the Volume to be Mounted for the Consumption by the Containers

The YAML format species a MongoDB 3.4 image to use the volume from Step 2 and mount it to path /data/db.

spec:

containers:

- image: mongo:3.4 name: mongo-ps ports:

  • - name: mongo-ps containerPort: 27017 hostPort: 27017 volumeMounts:

    • - name: pvc-128gb mountPath: /data/db

volumes:

  • - name: pvc-128gb persistentVolumeClaim:

claimName: pvc128gb

Storage was created and provisioned from vSAN for containers for the MongoDB service by using dynamic provisioning in YAML les. Storage volumes were claimed as persistent ones to preserve the data on the volumes. All mongo servers are combined into one Kubernetes pod per node.

In Kubernetes, as each pod gets one IP address assigned, each service within a pod must have a distinct port. As the mongos are the services by which you access your shard from other applications, the standard MongoDB port 27017 is assigned to them.

Please refer this Reference Architecture for detailed understanding of how persistent storage for containers is consumed by MongoDB services on vSAN.

Download the yaml les for deploying MondoDB on Kubernetes with vSphere Cloud Provider from here

To understand the conguration mentioned in these YAMLs please refer this link

Execute following commands to deploy Sharded MongoDB Cluster on Kubernetes with vSphere Cloud Provider.

Create StaogeClass

Create Storage Volumes for Shared MondoDB Cluster

Create Mongo DB pods

28

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Create Services

4.3 Deploying S3 Stateful Containers - Minio

This case study describes the process to deploy distributed Minio server on Kubernetes. This example uses the ocial Minio Docker image from Docker Hub.

Create Minio Storage class

#minio-sc.yaml

kind: StorageClass apiVersion: storage.k8s.io/v1 metadata:

name: miniosc provisioner: Kubernetes.io/vsphere-volume parameters:

diskformat: thin

Creating the storage class:

$ kubectl create -f minio-sc.yaml

Create Minio headless Service

Headless Service controls the domain within which StatefulSets are created. The domain managed by this Service takes the form: $(service name).$(namespace).svc.cluster.local (where “cluster.local” is the cluster domain), and the pods in this domain take the form: $(pod-name-{i}).$(service name). $(namespace).svc.cluster.local. This is required to get a DNS resolvable URL for each of the pods created within the Statefulset.

This is the Headless service description.

apiVersion: v1 kind: Service metadata:

name: minio labels:

app: minio spec:

clusterIP: None

ports:

- port: 9000 name: minio selector:

app: minio

Create the headless service

29

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

service "minio" created

Create Minio StatefulSet

A StatefulSet provides a deterministic name and a unique identity to each pod, making it easy to deploy stateful distributed applications. To launch distributed Minio you need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating pods. StatefulSets oer a perfect way to handle this requirement. This is the Statefulset description.

apiVersion: apps/v1beta1 kind: StatefulSet metadata:

name: minio spec:

serviceName: minio replicas: 4 template:

metadata:

annotations:

pod.alpha.Kubernetes.io/initialized: "true" labels:

app: minio

spec:

containers:

- name: minio env:

ports:

  • - containerPort: 9000 hostPort: 9000

# These volume mounts are persistent. Each pod in the PetSet # gets a volume mounted based on this field. volumeMounts:

  • - name: data mountPath: /data # These are converted to volume claims by the controller # and mounted at the paths mentioned above. volumeClaimTemplates:

- metadata:

name: data annotations:

volume.beta.Kubernetes.io/storage-class: miniosc spec:

accessModes:

  • - ReadWriteOnce

resources:

requests:

storage: 5Gi

30

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

Create the Statefulset

statefulset "minio" created

Create service and expose it to external trac using NodePort

Now that you have a Minio statefulset running, you may either want to access it internally (within the cluster) or expose it as a Service onto an external (outside of your cluster, maybe public internet) IP address, depending on your use case. You can achieve this using Services.

There are 3 major service types — default type is ClusterIP, which exposes a service to connection from inside the cluster. NodePort and LoadBalancer are two types that expose services to external trac.

In this example, we expose the Minio Deployment by using NodePort. This is the service description.

#minio_NodePort.yaml

apiVersion: v1 kind: Service metadata:

name: minio-service spec:

type: NodePort ports:

- port: 9000 nodePort: 30000 selector:

app: minio

$ kubectl create -f minio_NodePort.yaml service "minio-service" created

Access Minio

Find the IP addresses of master nodes

$ kubectl describe node master | grep Addresses

Addresses:10.160.132.97,10.160.132.97,master

Find the NodePort

$ kubectl describe service minio-service | grep NodePort Type:NodePort

NodePort:<unset>30000/TCP

Use the following URL to access Mnio http://10.160.132.97:30000/minio/login

31

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

5. Miscellaneous

Miscellaneous

32

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

5.1 FAQs

Is vSphere Cloud Provider ready for production ?

vSphere Cloud Provider is still in beta.

What is the biggest Kubernetes cluster it has been tested for ?

It has been tested on eight node cluster so far.

Where can I nd required Roles and Privileges for the vCenter User for vSphere Cloud Provider ?

Please refer to this section.

Where can I nd the list of vSAN, VMFS and NFS features supported by vSphere Cloud Provider ?

Please refer to this section. Please report in case you nd any features are missing.

33

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

How is running containers on vSphere Integrated Containers dierent from running them on Kubernetes on vSphere ?

VIC is infrastructure platform to run containerized workloads alongside traditional applications whereas vSphere Cloud provider provides an interface to run and take advantage of vSphere storage for workloads running on Kubernetes

Can I run it on a single node cluster on my laptop ?

Yes as long as laptop supports nested virtualization you can try it on your laptop.

Which Kubernetes distribution is supported ?

vSphere Cloud Provider is available in vanilla Kubernetes and all distributions using Kubernetes v1.5 and above should support it. Please refer this section.

Can we deploy multiple Kubernetes Cluster on one vCenter?

Yes. Using resource pool and maintaining the VMs from each Kubernetes cluster in their respective VM folder you can run multiple Kubernetes cluster on vCenter.

Can Kubernetes Cluster access storage from another vCenter?

Yes. vSphere supports shared storage across multiple vCenters. User can use shared storage in multiple Kubernetes Clusters.

Which Operating System are supported ?

We support Photon, Ubuntu, Core OS, please check this section for details

How Kubernetes volumes can be made resilient to failures on vSAN datastore?

Please check the HA section for details.

Can I enable SDRS on VMs hosting kubernetes cluster?

No.

Can we have a setting to ensure all dynamic PVs with have default policy Retain (instead of delete)? Or can we request the desired policy from the moment we request the PV via the PVC?

If the volume was dynamically provisioned, then the default reclaim policy is set to “delete”. This means that, by default, when the PVC is deleted, the underlying PV and storage asset will also be deleted. If you want to retain the data stored on the volume, then you must change the reclaim policy from

34

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

“delete” to “retain” after the PV is provisioned. You cannot directly set to retain PV from PVC request for dynamic volumes. Details

How do we resize the existing dynamic volumes? If we update the PVC with the new desired size, is it enough?

Support for resizing existing dynamic volume is not yet there. Proposal is out for review.

Can we create ReadWriteMany volumes with VSphere storage, with pods on dierent machines?

ReadWriteMany is not supported with Pods on dierent machine. This is supported on the collocated pods on the same node.

Is it mandatory to have all the machines in the same datastore ? If so, it’s a very strong limitation for us.

It is not mandatory to keep all node VMs on the same datastore. But make sure node VM has access to volumes datastores.

Can we have the disk uuid enabled by default, so we don’t need to do it machine by machine? What would be the risks of having it by default.

You can enable disk UUID by default, while creating VM. Just add this parameter at Customize hardware -> VM Options -> Conguration Parameters -> Edit Conguration

Is it mandatory to have all the machines in the same directory?

Current code base requires Master and Node VMs to be present under one VM folder. Each Kubernetes Cluster deployed in vSphere, should be placed in their respective VM folder, else under root folder.

5.2 Known Issues

Release 1.7

• Admin updating the SPBM policy name in vCenter could cause confusions/inconsistencies. Link

• Two or more PVs could show dierent policy names but with the same policy ID. Link

• Node status becomes NodeReady from NodeNotSchedulable after Failover. Link

Release 1.6.5

• Node status becomes NodeReady from NodeNotSchedulable after Failover. Link

Release 1.5.7

• Node status becomes NodeReady from NodeNotSchedulable after Failover. Link

35

Copyright © 2017 VMware, Inc. All rights reserved.

Kubernetes - vSphere Cloud Provider

vCenter Port other than 443

• For Kubernetes 1.6 and 1.7 releases (except Release v1.7.3 and onwards, Release v1.6.8 and onwards) vCenter Port other than 443 is not supported.

Kubernetes-Anywhere

• Destroying a Kubernetes cluster operation using Kubernetes-anywhere is aky Link

36

Copyright © 2017 VMware, Inc. All rights reserved.