DEPLOYMENTS!

W H AT C O U L D P O S S I B LY G O W R O N G ?
A LOT!
DUH.
When I deploy to prod… I do it in production.
When I deploy to prod… I do it in production.
CANARY DEPLOYS
WITH KUBERNETES AND ISTIO
JASON YEE
Te c h n i c a l E v a n g e l i s t
N o m a d & Tr a v e l H a c k e r
Whiskey Hunter
P o k e m o n Tr a i n e r

Tw : @ g i t b i s e c t
Em: jyee@datadoghq.com
D ATA D O G
TW: @datadoghq
SaaS-based monitoring,
tracing & logging

Tr i l l i o n s o f p o i n t s / d a y

We’re hiring:
jobs.datadoghq.com

Note: We’re running some

production services on
Kubernetes & have been
implementing Istio.
BLUE-GREEN DEPLOYMENTS
BLUE-GREEN DEPLOYMENTS

• Pros:

• Zero-downtime deploys

• Easy rollbacks
CTRL+Z
BLUE-GREEN DEPLOYMENTS

• Cons:

• ✌Easy✌ rollbacks… 🤞😬
CANARY DEPLOYMENTS
or
PA U S E . M O N I T O R .
DOUBLE CHECK.
M AY B E O N E M O R E T I M E ,
J U S T T O B E C E R TA I N .
PA R T Y !
CANARY DEPLOYMENTS

• Small scope
CANARY DEPLOYMENTS

• Small scope

• Limited ramifications
CANARY DEPLOYMENTS

• Small scope

• Limited ramifications

• Easier rollbacks
CANARY DEPLOYMENTS

• Small scope

• Limited ramifications

• Easier rollbacks

• Load tolerant
CANARY DEPLOYMENTS

• Small scope

• Limited ramifications

• Easier rollbacks

• Load tolerant

• Concurrency
CANARY
S T R AT E G Y
C A N A R Y S T R AT E G Y
How do you choose your sample set?

• Random
C A N A R Y S T R AT E G Y
How do you choose your sample set?

• Random

• Representative
C A N A R Y S T R AT E G Y
How do you choose your sample set?

• Random

• Representative
• Geography
• Time
• Use patterns
C A N A R Y S T R AT E G Y
How do you choose your sample set?

• Random

• Representative
• Geography
• Time
• Use patterns

• Granularity
C A N A R Y S T R AT E G Y
How do you choose your sample set?

• Random

• Representative
• Geography
• Time
• Use patterns

• Granularity

• Resource mapping
MONITORING
S T R AT E G Y
M O N I T O R I N G S T R AT E G Y
How do you evaluate your deployment?

• Tags! Tags! Tags! Tags! Tags!

M O N I T O R I N G S T R AT E G Y
How do you evaluate your deployment?

• Tags!

• p90, p95, p99

M O N I T O R I N G S T R AT E G Y
How do you evaluate your deployment?

• Tags!

• p90, p95, p99

• Outliers
Outliers: one of these things is not like the others
M O N I T O R I N G S T R AT E G Y
How do you evaluate your deployment?

• Tags!

• p90, p95, p99

• Outliers

• Anomalies
Anomalies: It wasn’t like this before
Anomalies: It wasn’t like this before
 Anomalies: It wasn’t like this before

M T W TH F M T W TH F M T W TH F M T W TH F

NOT A NORMAL TUESDAY

S I G N A L S T O W AT C H

Latency
S I G N A L S T O W AT C H

Latency Errors
S I G N A L S T O W AT C H

Latency Errors

Traffic
S I G N A L S T O W AT C H

Latency Errors

Traffic Saturation
W H AT D O E S K U B E R N E T E S H A V E
TO DO WITH ANY OF THIS?
C O N TA I N E R S E R V I C E
O R C H E S T R AT O R
 C O N TA I N E R S E R V I C E
O R C H E S T R AT O R

p.s. - Maybe a Squirtle orchestrator? Fun fact: Pokemon Go runs on Kubernetes!

So what if we want to deploy one service?
Blue-green doesn’t make any sense!
Kubernetes handles service deployments
Kubernetes handles service deployments. YAY!
WHY DO I NEED A
SERVICE MESH?
Kubernetes does
rolling deploys
really well!
Canary deploys,
not so much.
CANARY DEPLOYING WITH
KUBERNETES
SERVICE
apiVersion: v1
kind: Service
metadata:
name: my-app
labels:
app: my-app
spec:
ports:
- port: 80
name: http
selector:
app: my-app
DEPLOYMENT
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
spec: spec:
replicas: 3 replicas: 3
selector: selector:
matchLabels: matchLabels:
app: my-app app: my-app
template: template:
metadata: metadata:
labels: labels:
app: my-app app: my-app
version: v1 version: v2
spec: spec:
containers: containers:
- name: my-app - name: my-app
image: jyee/my-app:v1 image: jyee/my-app:v2
imagePullPolicy: Always imagePullPolicy: Always
CANARY?
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
spec: spec:
replicas: 9 replicas: 1
selector: selector:
matchLabels: matchLabels:
app: my-app app: my-app
template: template:
metadata: metadata:
labels: labels:
app: my-app app: my-app
version: v1 version: v2
spec: spec:
containers: containers:
- name: my-app - name: my-app
image: jyee/my-app:v1 image: jyee/my-app:v2
imagePullPolicy: Always imagePullPolicy: Always
CANARY?
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
spec: spec:
replicas: 8 replicas: 2
selector: selector:
matchLabels: matchLabels:
app: my-app app: my-app
template: template:
metadata: metadata:
labels: labels:
app: my-app app: my-app
version: v1 version: v2
spec: spec:
containers: containers:
- name: my-app - name: my-app
image: jyee/my-app:v1 image: jyee/my-app:v2
imagePullPolicy: Always imagePullPolicy: Always
CANARY?
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
spec: spec:
replicas: 7 replicas: 3
selector: selector:
matchLabels: matchLabels:
app: my-app app: my-app
template: template:
metadata: metadata:
labels: labels:
app: my-app app: my-app
version: v1 version: v2
spec: spec:
containers: containers:
- name: my-app - name: my-app
image: jyee/my-app:v1 image: jyee/my-app:v2
imagePullPolicy: Always imagePullPolicy: Always
CANARY?
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
spec: spec:
replicas: 99 replicas: 1
selector: selector:

😫
matchLabels: matchLabels:
app: my-app app: my-app
template: template:
metadata: metadata:
labels: labels:
app: my-app app: my-app
version: v1 version: v2
spec: spec:
containers: containers:
- name: my-app - name: my-app
image: jyee/my-app:v1 image: jyee/my-app:v2
imagePullPolicy: Always imagePullPolicy: Always
W H AT D O E S A S E R V I C E M E S H G E T Y O U ?
SERVICE MESHES

• Routing & load balancing

SERVICE MESHES

• Routing & load balancing

• Service discovery
SERVICE MESHES

• Routing & load balancing

• Service discovery

• Timeouts & retries

SERVICE MESHES

• Routing & load balancing

• Service discovery

• Timeouts & retries

• Policy enforcement
SERVICE MESHES

• Routing & load balancing

• Service discovery

• Timeouts & retries

• Policy enforcement

• Monitoring & tracing

How does it work?
1. Add a data plane
2. Add a control plane
 I N S TA L L I N G I S T I O

1. curl -L https://git.io/getLatestIstio | sh -
 I N S TA L L I N G I S T I O

1. curl -L https://git.io/getLatestIstio | sh -

2. cd istio-1.0.2
 I N S TA L L I N G I S T I O

1. curl -L https://git.io/getLatestIstio | sh -

2. cd istio-1.0.2

3. export PATH=$PWD/bin:$PATH
 I N S TA L L I N G I S T I O

1. curl -L https://git.io/getLatestIstio | sh -

2. cd istio-1.0.2

3. export PATH=$PWD/bin:$PATH

4. kubectl apply -f install/kubernetes/istio-demo.yaml

 W H AT D O E S I T I N S TA L L ?

istio-citadel istio-statsd-prom-bridge
istio-egressgateway istio-telemetry
istio-ingressgateway grafana
istio-pilot prometheus
istio-policy servicegraphng
istio-sidecar-injector zipkin
ISTIO SERVICES

• Istio Pilot
ISTIO SERVICES

• Istio Pilot

• Istio Mixer
ISTIO SERVICES

• Istio Pilot

• Istio Mixer

• Istio Ingress/Engress

Istio Pokemon Go is next!

ISTIO SERVICES

• Istio Pilot

• Istio Mixer

• Istio Ingress/Engress

• Istio Citadel

It has 15,782 steps!

CANARY DEPLOYING WITH ISTIO
SERVICE
apiVersion: v1
kind: Service
metadata:
name: my-app
labels:
app: my-app
spec:
ports:
- port: 80
name: http
selector:
app: my-app
SERVICE
apiVersion: v1
kind: Service

!
metadata:

E
name: my-app

AM
labels:

S
app: my-app
spec:
ports:
- port: 80
name: http
selector:
app: my-app
DEPLOYMENT
apiVersion: apps/v1
kind: Deployment
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
version: v1
spec:
containers:
- name: my-app
image: jyee/my-app:v1
imagePullPolicy: Always
DEPLOYMENT
apiVersion: apps/v1
kind: Deployment

!
spec:

E
replicas: 3

M
selector:

A
matchLabels:

S
app: my-app
template:
metadata:

f )
labels:
app: my-app
version: v1
(s o rt o
spec:
containers:
- name: my-app
image: jyee/my-app:v1
imagePullPolicy: Always
istioctl kube-inject -f my.yaml > mod.yaml
kubectl apply -f mod.yaml
TEENAGE
M U TAT I N G W E B H O O K
ADMISSION CONTROLLERS!
AKA AUTO-SIDECAR INJECTION
ISTIO VIRTUALSERVICES
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: my-app-routing
spec:
hosts:
- my-app
http:
- route:
- destination:
host: my-app
subset: v1
I S T I O D E S T I N AT I O N R U L E S
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-app-destination
spec:
  host: my-app
  subsets:
    - name: v1
      labels:
        version: v1
 ISTIO ISTIO K8S
kind: VirtualService kind: DestinationRule kind: Deployment
... ... spec:
spec: spec: replicas: 3
hosts:   host: my-app selector:
- my-app   subsets: matchLabels:
http:     - name: v1 app: my-app
- route:       labels: template:
- destination:         version: v1 metadata:
host: my-app labels:
subset: v1 app: my-app
version: v1
spec:
containers:
...
DEPLOYMENT
apiVersion: apps/v1 apiVersion: apps/v1
kind: Deployment kind: Deployment
spec: spec:
replicas: 3 replicas: 3
selector: selector:
matchLabels: matchLabels:
app: my-app app: my-app
template: template:
metadata: metadata:
labels: labels:
app: my-app app: my-app
version: v1 version: v2
spec: spec:
containers: containers:
- name: my-app - name: my-app
image: jyee/my-app:v1 image: jyee/my-app:v2
imagePullPolicy: Always imagePullPolicy: Always
I S T I O D E S T I N AT I O N R U L E S
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: my-app-destination
spec:
  host: my-app
  subsets:
    - name: v1
      labels:
        version: v1
    - name: v2
      labels:
        version: v2
ISTIO VIRTUALSERVICES
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: my-app—routing
spec:
  hosts:
    - my-app
  http:
  - route:
    - destination:
        host: my-app
        subset: v1
      weight: 80
  - route:
    - destination:
        host: my-app
        subset: v2
      weight: 20
ISTIO VIRTUALSERVICES
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
...
  http:
  - match:
    - headers:
        cookie:
          user: my-logged-in-user
    route:
    - destination:
        host: my-app
        subset: v2
      weight: 20
W H AT E L S E C A N I T D O ?

LOTS!
https://istio.io/docs/reference/config/
RECAP

• Service meshes give you more control

RECAP

• Service meshes give you more control

• Canary deploys: Representative & Granular

RECAP

• Service meshes give you more control

• Canary deploys: Representative & Granular

• Monitoring: Tags, Outliers, Anomalies

RECAP

• Service meshes give you more control

• Canary deploys: Representative & Granular

• Monitoring: Tags, Outliers, Anomalies

• What to watch: Latency, Errors, Traffic, Saturation

RECAP

• Service meshes give you more control

• Canary deploys: Representative & Granular

• Monitoring: Tags, Outliers, Anomalies

• What to watch: Latency, Errors, Traffic, Saturation

• GO PLAY WITH ISTIO 1.0.2!!!

QUESTIONS?
Slack: #2018addo-cloudnative
email: jyee@datadoghq.com
twitter: @gitbisect