KVMLiveMigration

UriLublin,Qumranet AnthonyLiguori,IBM

Agenda

Introduction Algorithm MigrationProtocols Howtoaddmigrationsupportfornewdevices Usingit Summary/Meritscomparedtootherhypervisors FutureWork

KVMForum2007

Introduction

LiveMigration,Avaluablefeatureofanyhypervisor
Almostunnoticeableguestdowntime LoadBalancing,Maintenance,HardwareUpgrades
SoftwareUpgrades

Guestisnotinvolved CapableoftunnelingVMStatethroughanexternal
program ShortandSimple EasytoEnhance Hardware(almost)independence
KVMForum2007

Algorithm
HostA MigratetoB HostB 1.Setup 2.TransferMemory

KVMForum2007

Algorithm
HostA HostB 1.Setup 2.TransferMemory

WholeMemory

1133000
KVMForum2007

Algorithm
HostA HostB 1.Setup 2.TransferMemory DirtyPages

25800
KVMForum2007

Algorithm
HostA STOP HostB 1.Setup 2.TransferMemory 3.StoptheVM

350
KVMForum2007

Algorithm
HostA HostB 1.Setup 2.TransferMemory VMState 3.StoptheVM 4.TransferState

150
KVMForum2007

Algorithm
HostA Continue HostB 1.Setup 2.TransferMemory 3.StoptheVM 4.TransferState 5.ContinueVM

KVMForum2007

Algorithm

1.

MigrationRequestarrives(migrateVMfromAtoB)
Spawnexternalcommand(Ifapplicable) Connect(ifapplicable)andsendheader Allocateresources+Setup

KVMForum2007

10

Algorithm

1. 2.

MigrationRequestarrives TransferMemory

Firsttransferallmemorypages(firstiteration) ForeverynextiterationItransferalldirtypagesof iterationi1 Untilconvergence

KVMForum2007

11

Algorithm

1. 2. 3.

MigrationRequestarrives TransferMemory StoptheVM

KVMForum2007

12

Algorithm

1. 2. 3. 4.

MigrationRequestarrives TransferMemory StoptheVM TransferVMState

Eachdevicetransferitsownstate Dirtypages(fromthelastiteration)included

KVMForum2007

13

Algorithm

1. 2. 3. 4. 5.

MigrationRequestarrives TransferMemory StoptheVM TransferVMState ContinuetheVM

Onremote(B)ifmigrationwassuccessful Send(broadcast)anEthernetpackettoannouncethe newlocation Onlocalhost(A)ifmigrationfailed

KVMForum2007

14

MemoryTransfer

Requiressupportfordirtymemorypage
logging

Homogeneouspageoptimization Rapidlywrittenpages/WriteableWorkingSet DynamicBandwidthLimitation

KVMForum2007 15

DirtyPageLogging

Qemu

Onebyteperpagesupportsupto8differentdirtytypes Devicesthatwritedirectlyintoguestmemorymustupdate
thedirtybytemap

KVM

Onebitperpage PagesaremappedROtointerceptfirstwrite Enabled/Disabledwhenmigrationbegins/ends MergedwithQemusdirtylogbeforeeverymemory

transferiteration

KVMForum2007

16

StateSave/Load

Qemudevicesregistersave/loadfunctions FunctionsarecalleduponVMsave/load/migrate Versioning/BackwardsCompatibility KVMState(registervalues)issynchronizedwith

qemuaspartofcpuandotherdevicesstate functions
KVMForum2007

17

MemoryTransferConvergenceRules

Transitionsthealgorithmfromlivephasetooffline
phase,accordingtothefollowingrules:

Convergence:N1=50dirtypages(orless)left NoProgress:N2=2iterationswherethenumberof
transferredmemorypagesissmallerthanthe numberofpagesthatgotdirty.

Hardlimit:N3=30iterationspassed
KVMForum2007 18

Convergenceexample:fc6runninghttpd
Numberofpagessentperiterationduringfc6httpdlivemigration
1000000

133168 100000

6901 10000 NumberofPages 2345 1185 1000 528 455 2696 1301 574 311 284 258 211 133 39

100

10

8 Iterationnumber

10

11

12

13

14

15

KVMForum2007

19

WWSexample:fc6runninghttpd
Histogram:Numberoftimesapageswassentduringfc6httpdlivemigration
1000000

126084 100000

10000 numberofpages

4141

1000

419

627

674

603

348 91 44 21 15 28 25 26 22

100

10

10

11

12

13

14

15

numberoftimes

KVMForum2007

20

EndOfMigrationProtocol(fortcp://)

Goal:preventacasewherea guestcontinuestorunonboth hosts Algorithm:

WorstCase:Gowaslost

ATransfersstatetoBandwaits forACK Breceivesstate,sendsACKand waitsforGO AreceivesACK,andsendsGO BreceivesGOandcontinues Uponanytimeout(lost messages),migrationfails VMdoesnotrun(onanyhost) Astops,Bexits Thirdparty(management) interventionrequired
KVMForum2007

21

MigrationSupportfornewdevices

Ifthenewdevicewritesdirectlytoguestmemory
updatebytemaplog

Ifsomesyncingneedstobedonebefore/afterstate
transfer,registertogetVMstop/contevents

Registersave/loadstatefunction Dontforgetversioning(supportforbackwards
compatibility)
KVMForum2007 22

MigrationSupportforPVdrivers

PVhostsidemust

Save/Loaditsstate Makesuregueststateisvalidonremotehost

UseNonLockingguesthostsynchronization
mechanism.

Suchasrings. Guestmustneverbestoppedwhileholdingaguesthost
sharedlock.

Keephypercallcallingasinglecommand
KVMForum2007 23

Qemumonitorcommandsandcmdline

(qemu)migrate[d]<migration_protocol:params> (qemu)migrate_set_speed<bytes_per_second> (qemu)migrate_cancel (qemu)infomigration

KVMForum2007

Onremote<kvmcmdline>incoming<protocol:params> /usr/bin/kvmm512hda/images/a.imgincomingstdio

24

MigrationProtocolsUseCases UsingTCPsockets

BuiltInsshsupport Saveimagetofile

Migratetcp://remote:port incomingtcp://0:port Migratessh://remote

Migrateexec:ddof=STATEFILE incomingfile://STATEFILE Migrateexec:gzipc>FILE.gz

Compressusinggzip(bzip2)

Encryptusinggpgintoafileortoremote

Migrateexec:gpgqerKEYoFILE.gpg Migrateexec:gpgqerKEY|ncremoteport Nclport|gpgqdrKEY|<kvmcmd>

KVMForum2007 25

Meritscomparedtootherhypervisors

ShortandSimple Builtinsecurityusingssh Guestisnotinvolved Hardwareindependence Migrationofstoppedguests Tunneling/Flexibility/Extensibility Compression/Encryption BackwardsCompatibility UponFailure,guestcontinuestorunonsourcehost. Open
KVMForum2007 26

FutureWork

Supportformoremigrationprotocols

WWSoptimization Finetuneparameters Dealwithdeviceassignment/directaccess. Migratetoaremotelocation Migratediskastogetherwithstate. Supportfornewfeaturesofkvm(e.g.passthrough)

KVMForum2007 27

LiveCheckpoints,compression,encryption,file Partialsupportforunknownmigrationprotocols

ThankYou
KVMForum2007 28