VIRTUALIZATION

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

CHAPTER GOALS
• Define key virtualization terms
• Summarize the history of virtualization
• Explain virtualization layer architecture and how it interacts with client
software
• Describe CPU architectural features that improve virtualization
performance
• Discuss virtualization networking issues and how they’re addressed
• Describe the costs and benefits of virtualization in common deployment
settings

Previous chapters have briefly discussed some aspects of virtual resources and virtual machines.

This chapter expands on that discussion to provide a comprehensive treatment of virtualization,

touching on all aspects of systems architecture: hardware, software, and networks.

 Virtualization

2
WHAT IS VIRTUALIZATION?
At its core, virtualization is a combination of software, hardware, and networking tech-
nologies that enable a single physical computer to act as though it were multiple indepen-
dent computers. A virtual machine (VM) is a software emulation of hardware resources
that behaves like a physical computer. A user can install an operating system (OS) and
other system and application software, called client software, on a VM and interact with
the VM as though it were a physical computer. To the client software, the VM behaves
exactly like a physical computer. The client software is unaware that it’s running on

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
virtual rather than physical resources.
Virtualization uses host software, generically called a virtualization layer, to control
physical computer hardware resources and configure these resources to support one or
more VMs, as shown in Figure 1. A hypervisor, first described in Chapter 11, is the
most common type of virtualization layer. Hypervisors come in two basic types: bare-
metal hypervisors and virtualization environments. A bare-metal hypervisor is a special-
purpose OS that provides only the minimal set of services required to manage and control
VMs and is commonly used for server consolidation and virtualized desktop computing
environments. A virtualization environment, installed as an application program under
another OS, is generally used for software development and testing and by end users to
run application software designed for one OS on a single-user computer with a different
installed OS.

VM-1 VM-2 VM-3 VM-4

Hypervisor

Hardware resources

I/O
CPUs Memory Disks
devices

FIGURE 1 Interaction between VMs, the virtualization layer, and underlying physical resources
Courtesy of Course Technology/Cengage Learning

VIRTUALIZATION HISTORY
The need to subdivide computer resources into independent subsets and devote each
subset to a specific task isn’t new. In fact, it was a goal of hardware and system software
designers from the earliest days of computing. Over the years, various approaches have
 Virtualization History

been tried, and some are still used. In the era before complex OSs, writing large applica-
3
tion programs that could perform multiple tasks was the only way to make a computer do
multiple things at once. No other way was possible because each application program
controlled all hardware resources, and there was no mechanism for application programs
running concurrently to share resources.
Multitasking OSs changed the nature of how computing resources were shared across
tasks by enabling multiple specialized service and application programs to run concur-
rently. When they were first introduced, they were seen mainly as a way of using expen-
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

sive computing resources more efficiently, especially the CPU. When a program or service
was waiting for an I/O operation to finish, the OS shifted CPU control to another program
or service. This method kept the expensive CPU busier and, therefore, increased the
amount of work accomplished and the economic return on the hardware investment.
As CPUs became faster and cheaper, related techniques, such as I/O buffering, storage
caching, and virtual memory, were developed to expand the number of concurrent
tasks executing on a single hardware platform.
As multitasking OSs matured in the 1950s and 1960s, IBM and others experimented
with ways to support multiple OSs on a single computer system. IBM was especially inter-
ested in separating high-volume batch transaction processing from online processing.
The characteristics of these processing types were different enough that system software
designers thought addressing each type with a customized OS was best. Further, because
few customers were willing to buy two mainframes, one to support each OS type,
virtualization was seen as way of economizing on hardware.
In 1970, IBM released a new generation of mainframe hardware called System/370,
which included a hypervisor called VM/370. VM/370 enabled the System/370 to run mul-
tiple OSs, including OSs for the earlier 360 mainframe and a new OS optimized for
interactive VDT-based applications called Conversational Monitor System. Other hyper-
visors were under development in industry and academia, although none enjoyed the
commercial success of VM/370.
By the late 1970s and throughout the 1980s, a number of trends in computing hard-
ware and software combined to push virtualization into the background. One was the
emergence of minicomputers (now called midrange computers) early in the 1970s and
microcomputers (now called workstations or PCs) by the end of the decade. Organizations
purchased multiple computer systems, each customized to and sized for a particular type
of processing. By the mid- to late 1980s, most organizations used mainframes only for
large-scale batch processing and large-scale services, such as database management. Typi-
cally, smaller and less expensive minicomputers handled interactive processing, and
interactive single-user applications were hosted on microcomputers. Because smaller
computers were usually more cost effective, there was little interest in “software tricks”
that enabled one large mainframe to act like several smaller machines.
Through the 1980s and 1990s, hardware continued to increase in power and decline
in price. Small servers and PCs proliferated. By the end of the 20th century, most large
organizations had multiple mainframes, racks full of midrange servers, at least one work-
station per employee, and a rapidly increasing number of laptops and other mobile
computing devices. For various economic reasons explored later in “Economics of
Virtualization,” organizations were becoming more concerned about the costs of frequent
hardware upgrades and administering so many computers. Hardware cost had declined to
 Virtualization

the point that software and administrative costs were higher, and these costs ballooned
4
upward with the number of computer systems.
Virtualization reappeared in the commercial realm starting in the late 1990s. Early
adopters included software developers who needed to test their products on many differ-
ent system configurations and organizations that managed large collections of midrange
servers. Virtualization use expanded throughout the 2000s, and new use cases were
added, including virtualized desktop computing, remotely accessible computer labs for
education and training, and support for multiple OSs on a single-user computer. By 2010,

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
virtualization was firmly entrenched in the marketplace, with promising growth potential
for the next decade.
VMware is the current market leader in virtualization, with a market share estimated
as high as 90%. VMware’s ESX product line, a bare-metal hypervisor, is widely used for
server consolidation and desktop virtualization. VMware Workstation, a virtualization
environment, is used often in software development, testing, and user support.
Microsoft entered the virtualization marketplace in the mid-2000s with workstation-
and server-oriented virtualization environments named Virtual PC and Virtual Server.
Virtual Server was replaced by Hyper-V, a hypervisor that’s bundled with Windows Server
2008 and later versions. An open-source hypervisor called Xen is used in some Linux
environments. Neither Xen nor any Microsoft virtualization product appears to have much
traction in the marketplace.

SOFTWARE SUPPORT FOR VIRTUALIZATION

Earlier chapters covered a variety of architectural features of application and system soft-
ware. Many of these same concepts are “reused” in the realm of virtualization. So
although you might not yet realize it, you already know a lot about how virtualization
works. The following sections will fill in the blanks to complete your understanding.

Virtualization Layer
The role of a virtualization layer is to support one or more client OSs and give them
access to underlying hardware resources yet prevent them from interfering with one
another. This role is similar to an OS’s role with services and application programs. In
fact, a virtualization layer is a specialized type of multitasking OS, in which the “tasks” are
other OSs rather than application programs. Therefore, many OS architectural features
and design issues covered in Chapter 11 apply to virtualization layers.
The differences between a virtualization layer and an OS are the extent of virtual
resource management and the absence of most service-layer functions. In an OS,
resources such as memory and secondary storage are virtualized on behalf of application
programs. Each program “thinks” it has exclusive access to memory regions, files, and I/O
devices. The OS intervenes as applications access these resources, which enables resource
sharing among applications. Examples of related OS techniques include moving processes
between idle and running states, swapping memory pages in from disk as needed, and
accessing secondary storage and I/O devices on behalf of applications when they make
related service calls. The service layer and an extensive set of interrupt handlers make it
all happen.
 Software Support for Virtualization

A virtualization layer uses similar techniques to share virtual resources among client
5
OSs, although the service layer is much smaller than in a OS. The virtualization layer
defines a collection of virtual CPU, memory, secondary storage, and I/O resources; this
resource collection constitutes a VM for use by a client OS. As with an OS, the sum of
available virtual resources is usually greater than the underlying physical resources.
Therefore, the virtualization layer must reallocate physical resources to VMs as they’re
needed and use sharing mechanisms, such as CPU scheduling and virtual memory
management.
Like an OS, a virtualization layer “traps” accesses to resources that aren’t immedi-
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

ately available to the client OS. Recall from Chapters 6 and 11 that interrupt processing is
the mechanism by which an OS steps in when needed to control access to shared or vir-
tual resources, as shown in Figure 2(A). Application programs can be suspended while
resources are unavailable and then resumed when they become available.

Application CPU OS Interrupt

(A)
program hardware supervisor handler

Hypervisor Client OS
Application CPU Hypervisor Client OS
(B) interrupt interrupt
program hardware supervisor supervisor
handler handler

FIGURE 2 Interrupt processing with (B) and without (A) a hypervisor

Courtesy of Course Technology/Cengage Learning

When a virtualization layer is in use, some interrupts that normally cause control
transfer to a client OS supervisor are overridden; instead, control passes to the virtualiza-
tion layer, as shown in Figure 2(B). The hypervisor then performs the following steps:
1. Intercepts interrupts that require access to shared or unavailable virtual
resources
2. Executes its own interrupt handler to make these resources available
3. Passes control to the client OS supervisor and interrupt handler
Recall from Chapter 6 that although CPUs are now optimized for interrupt handling,
it’s still a time-consuming process, so the extra layer of interrupt handling shown in
Figure 2(B) can have a substantial performance impact. Virtualization layers compensate
for this impact in two ways:
• The virtualization layer supervisor and interrupt handlers are coded for opti-
mal efficiency and use CPU hardware assistance if it exists (see the later
Technology Focus for a description of hardware assistance).
• The virtualization layer intercepts interrupts only when absolutely necessary.
When possible, physical resources are assigned exclusively to a VM to
prevent the need to trap interrupts associated with those resources.
 Virtualization

6 NOTE
Of course, assigning resources exclusively to one VM makes them unavailable to others. Therefore,
hypervisor designers and system administrators must make trade-offs between the performance of each
VM (improved by exclusive resource assignment) and overall performance of a collection of VMs
(usually improved by sharing as many resources as possible).

In most instances, the virtualization layer dedicates disk space and some portion of

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
memory to each VM but enables these resource allocations to increase beyond the dedi-
cated amounts. Accesses to dedicated memory and disk regions can be made without
intercepting interrupts, but access beyond dedicated resources does require virtualization
layer intervention. For example, a hypervisor might allocate 4 GB of memory exclusively
to a server VM but enable the VM to access up to 8 GB. By modifying the virtual memory
page tables and associated interrupt handling, the hypervisor can give client OSs access to
the first 4 GB to run entirely in the OS. Accesses to the upper 4 GB could trigger page
swaps, and the associated interrupts would be handled by the hypervisor first.
As with processes and threads executing in a multitasking OS, the sum of physical
resource demands by all VMs running under a virtualization layer can exceed available
physical resources, a situation sometimes described as “oversubscription.” As long as all
VMs aren’t using all their allocated resources at the same time, the performance impact on
the group of VMs is negligible or zero. As oversubscription increases, however, VMs incur
delays in accessing resources as the hypervisor shifts them from one VM to another.
The virtualization layer uses the same techniques to share virtual resources among
VMs that a client OS does to share virtual resources among processes. It intercepts inter-
rupts normally processed by the client OS when processing these interrupts requires
access to unavailable physical resources that the virtualization layer manages. By balanc-
ing exclusive and nonexclusive resource allocation to VMs, the virtualization layer can
minimize the performance impact of the extra layer of interrupt processing for each VM. It
can also support a large collection of VMs through oversubscription, although performance
degradation increases with the amount of oversubscription.

Client Software
In theory, system and application software running in a hypervisor or virtualization envi-
ronment should be “unaware” that it’s not actually running on physical hardware. As a
practical matter, some modification to client software is usually required to make virtuali-
zation work. These modifications fall into two categories: virtual device drivers and OS
kernel modifications.

Virtual Device Drivers

A virtualization layer presents generic interfaces for (virtual) devices, such as storage
device controllers and I/O devices, to a VM. When possible, the virtual interfaces comply
with well-defined standards, such as SCSI, SATA, and Super VGA. Device drivers for
physical devices are optimized for direct physical access. If a client OS uses a device
driver to access a virtualized device, VM performance usually suffers. To address this
problem, virtualization layers typically provide customized device drivers optimized to
 Hardware Support for Virtualization

work with the virtualization layer and achieve higher performance. Virtual device drivers
7
are installed on the client OS during OS installation or shortly thereafter. To the client
OS, they “look” like physical device drivers, and the client OS is unaware of their
interfaces to and optimizations for the virtualization layer.

Kernel Modifications
Modifying the OS kernel violates the principle that a client OS should be unaware it’s
running under the control of a virtualization layer. Nonetheless, some virtualization
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

approaches use kernel modifications to improve virtualization performance. In fact, this

approach might be a trend in OS design for the next decade or more, particularly for
server OSs increasingly hosted by hypervisors.
Modifying client OS kernels to improve virtualization performance is mainly found in
the Xen hypervisor and Linux client OSs. Because Linux is an open-source OS, it’s possi-
ble to modify the kernel source code and recompile it for use in a hypervisor. It’s unclear
at this point whether this approach will migrate to proprietary OSs, such as Windows.

HARDWARE SUPPORT FOR VIRTUALIZATION

Because virtualization layers and OSs use similar techniques for resource sharing, CPUs
were, in a sense, already optimized for virtualization when it reemerged in the late 1990s.
However, there are unique performance issues to address when multiple VMs share a set
of hardware resources. These performance issues extend to all computer resources,
including CPUs, storage devices, and I/O devices.
Like OS service layers and network protocol stacks, virtualization is an example of
software layering, although in this case, one of the software layers is a emulation of a
hardware layer. Virtualization exhibits the same sort of costs and benefits as other
software-layering applications, including the following:
• Each layer is simple, making it faster to develop and update.
• Alternative implementations of a layer are interchangeable if their interfaces
to layers above and below are well defined (for example, replacing TCP with
UDP in a network protocol stack).
• Each interface between layers introduces a translation that slows operation of
the entire group or stack. Therefore, the more layers used, the less efficient
the group or stack is as a whole.
The first two points underlie economic benefits of virtualization, as described later in
“Economics of Virtualization.” The third point underlies an economic cost—one that
hardware designers have worked to minimize since the early 2000s.
As discussed, a second layer of interrupt processing enables the virtualization layer to
intervene when a client OS attempts to access a virtual resource that’s not currently
available. Most of the recent effort to optimize CPU designs for virtualization environ-
ments has been directed at minimizing the performance penalty of layered interrupt pro-
cessing. The most common technique is defining an additional processor mode or
privilege level for the virtualization layer that’s higher than that used by VM client OSs.
Instructions and interrupts that require virtualization layer intervention are modified to
 Virtualization

transfer control automatically to a hypervisor running in the higher-level processor

8
mode. Other instructions and interrupts are processed normally, with automatic CPU
control passed to the client OS supervisor. In essence, CPU hardware provides an auto-
matic “switch” between hypervisor and client OS control. As with other hardware optimi-
zations, moving control decisions and associated processing into hardware improves
performance dramatically, compared with having the hypervisor intercept and process
every interrupt.
Another area of hardware development to improve virtualization performance is the

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
interface between the CPU, the system bus, and storage and I/O devices. When many VMs
share a limited number of I/O devices, bottlenecks can occur as they access these devices.
Again, interrupts associated with device accesses must be handled by the hypervisor,
which acts like a network switch to connect VMs to devices as needed. Using interrupt
processing for this task degrades performance because every access to a shared I/O or
storage device requires duplicate interrupt handling by the virtualization layer and one
of the client OSs.
Today’s computer architecture generally has a supporting chipset to connect one or
more CPUs to external buses and the storage and I/O devices attached to them. Currently,
chipset manufacturers use a variety of approaches to address virtualization performance
issues. One is to define multiple dedicated communication pathways between peripheral
devices and VMs. Rather than present one device interface to the CPU by using one
small set of interrupts and a single I/O port, a chipset can implement an internal switching
network and define multiple sets of interrupts and I/O ports. Each set can be assigned
exclusively to one VM. The associated interrupts are processed by VM client OS without
hypervisor intervention, and data moves through an I/O port mapped to a dedicated
memory page. In essence, this approach moves the role of “traffic cop” from the CPU
and hypervisor to the supporting chipset.

TECHNOLOGY FOCUS

Intel Virtualization Technology

In the early 2000s, Intel began working on enhancements to its CPU architectures to
address emerging issues associated with virtualization. As in many large companies,
marketing staff develop a handy name for a new technology that they can sell—therefore,
the acronym Intel VT for Intel Virtualization Technology was created. Intel has used this
acronym to describe a series of CPU architectural features and changes to supporting
chipsets and I/O devices.
Intel VT has been expanded several times to improve performance and address new
uses for virtualization. As a result, the acronym has morphed into several variations,
including Intel VT-x for Xeon processor features, Intel VT-i for Itanium processor fea-
tures, Intel Virtualization Technology Directed I/O (VT-d) for supporting chipset features,
and Intel Virtualization Technology for Connectivity (Intel VT-c) for NICs and other
peripheral I/O hardware. The next sections focus on features common to VT-x and VT-i
that improve CPU virtualization performance.
(continued)
 Hardware Support for Virtualization

9
CPU Control
An OS is constructed under the assumption that it has full control over all hardware
resources. If a client OS had full hardware control when running under a hypervisor, it
could inadvertently cause the hypervisor to malfunction. So an important issue in
hypervisor software design, CPU design, or both is enabling the hypervisor to control
some hardware resources exclusively yet making all client OSs “think” they have
complete control.
Intel VT adds a new CPU operating mode (called VMX root mode) and several sup-
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

porting instructions and registers to enable the hypervisor and client OSs to coexist and
improve the efficiency of context switches between them. The hypervisor runs in VMX
root mode with full access to hardware resources. Client OSs run in VMX non-root mode
until they execute privileged instructions that might affect the hypervisor. The CPU
intercepts these instructions, saves the CPU state automatically, and reverts to VMX root
mode, which transfers control to the hypervisor automatically. The hypervisor executes
the instruction on behalf of the client OS in a “safe” way, and then restores the saved
CPU state before returning control to the client OS, which is unaware of the hypervisor’s
role in executing the instruction.
In essence, transitions into and out of VMX root mode add a second level of inter-
rupt handling to the CPU, thus enabling the hypervisor to call interrupt handlers and
service calls automatically when a client OS executes a privileged instruction. The
advantage of this approach is that it maintains the client OS’s illusion that it has com-
plete control of the hardware yet enables the hypervisor to protect itself. The disadvan-
tage is that the extra layer of interrupt handling adds complexity and degrades
performance, especially if the hypervisor intercepts many different instructions.
Interrupts from peripheral devices, such as NICs, can be a major source of perfor-
mance degradation because interrupt processing overhead is effectively doubled. Intel VT
enables the hypervisor to choose which peripheral device interrupts cause an automatic
transition to VMX root mode and which don’t. With this feature, the hypervisor can avoid
VMX mode transitions for devices assigned exclusively to a single client OS (such as a
dedicated NIC or other I/O device).

VM Movement Between CPUs

Intel CPUs support an instruction named CPUID that, when executed, fills four registers
with data representing the processor type and its capabilities. This instruction enables
OSs and other software to vary their behavior to take advantage of advanced CPU fea-
tures when present and avoid them when they’re absent. For example, an OS uses
CPUID at installation time to configure itself, and applications manipulating audio or
video streams use CPUID to determine which streaming media extension (SSE)
instructions are available.
In many virtualization settings, it’s desirable to enable a hypervisor to move a run-
ning VM from one CPU (or set of CPUs) to another. These settings include load balancing
across nodes in a cluster and transferring a VM from one physical computer or cluster to
another. If the source and destination CPUs are identical, VM movement is straightfor-
ward. If the CPU architectures differ, the VM might fail on the destination CPU because
the OS or applications executed CPUID on the source processor, determined that certain
advanced instructions were available, and then attempted to execute these instructions
on a destination CPU lacking the instructions. This issue is most important in large
virtualization clusters that typically include multiple generations of hardware.
(continued)
 Virtualization

10
Intel server CPUs incorporate one of two methods to move running VMs to incom-
patible CPUs. In 32-bit CPUs, a feature called “CPUID masking” enables a hypervisor to
define a mask that zeros out some bit flags reported by CPU. In 64-bit processors,
executing CPUID always triggers a transition to VMX root mode, thus enabling the
hypervisor to step in and control the features reported by the instruction. In either case,
the hypervisor must be designed to control features reported by the CPU. At startup, a
hypervisor executes CPUID on each CPU in a cluster, determines which features are
supported by all CPUs in the cluster, and then adjusts CPUID masking or the behavior of

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
trapped CPUID instructions to report only common features.

NETWORKING AND VIRTUALIZATION

In a nonvirtualized environment, computers running a single OS interact with one
another via physical NICs and one or more physical networks. In a virtualized environ-
ment, multiple VMs reside in the same physical computer and interact with one another
and other machines via virtual NICs. To support VM network communication, a virtuali-
zation layer must implement a virtual network on the host computer. Virtual networks
have additional uses in some deployment and testing settings.
A virtual network is an emulated physical network used by VMs in a single computer
or cluster. Like a physical network, it has different types of interacting hardware and
software components, including virtual
• Communication pathways (emulated transmission lines)
• NICs
• Switches, routers, and firewalls
• DNS, DHCP, and other network services
In a physical network, wired connections using Category 6 or fiber-optic cable and
wireless connections operating under an 802.11 standard carry messages from one com-
puter to another. In a virtualization setting, VMs can be located in the same physical
computing node or in different computing nodes of a cluster. In either case, the virtuali-
zation layer must emulate a physical network (usually Gigabit Ethernet or 10 Gigabit
Ethernet) between VMs. For VMs hosted on the same computing node, inter-VM commu-
nication traverses the system bus in the form of emulated network messages. For VMs
hosted on different nodes of a cluster, inter-VM communication traverses the nodes’
internals bus connections and the physical network interfaces connecting cluster nodes.
For communication between VMs and external computers, messages traverse internal
buses, physical NICs, and internal and external networks to which the virtualization host
is connected.
Recall from Chapter 9 that each device on a physical network has at least one physi-
cal NIC, and each NIC has a physical (MAC) address and an associated IP address. Virtual
NICs used by VMs also have MAC and IP addresses, which must be different from
addresses of physical NICs on the virtualization host. The virtualization layer must gener-
ate unique MAC addresses for the NICs used by all VMs and associate unique IP addresses
with these MAC addresses.
 Networking and Virtualization

Virtual switches, routers, and firewalls implemented in the virtualization layer con-
11
nect virtual NICs to physical NICs and, therefore, to external networks, such as the Inter-
net. As with physical devices, configuration is required to manage the flow of messages
efficiently and control what types of messages can traverse the virtual network. To make
deployment flexible, the virtualization layer must provide virtual devices that system
administrators can connect and configure in a variety of ways.
For example, a system administrator might configure a network to support a database
server VM, multiple back-end application server VMs, and a publicly accessible Web
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

(proxy) server that directs traffic to and from application servers. A typical network to
support this deployment would include the following:
• An Internet-connected firewall that prevents access to the database and
application servers and restricts the type of messages that can flow to and
from the proxy server
• A private switched network to connect all internal servers
• At least two NICs on the proxy server, one connected to the firewall and the
other to the internal switch
• One or more NICs on the database and application servers connected to one
or two internal switches
If the server farm is virtualized, a virtual network is most efficient for communication
between servers, the server switch, and the firewall. The firewall can be implemented as a phys-
ical device connected to the virtualization host (see Figure 3) or as a virtual device on the host.

Virtual Virtual Virtual Virtual

proxy application application database
server server server server
Internet

Router Virtual
Firewall server
switch

Virtualization host

FIGURE 3 A private virtual server network

Courtesy of Course Technology/Cengage Learning

The most common method of supporting IP communication on virtual networks is

through internal network services on a private IPv4 or IPv6 network. Typically, nonrouta-
ble addresses (for example, in the 192.168.0.* subnet) are used for all internal devices. A
virtual DHCP service can be configured to assign addresses in this range to virtual NICs,
 Virtualization

or static addresses can be assigned. If a private DNS namespace is used, a virtual DNS
12
server must also be configured, and VMs and other network devices must be configured to
use it. Networks of this type are often used in testing setups for both server deployment
and network security. When testing network security, a private virtual network enables
system administrators to probe for security weaknesses but limit attacks and malware
propagation to the virtual network.
Some common server deployment settings create unique network deployment pro-
blems. For example, in the hypothetical server farm described previously, assume that

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
normal IP addresses are used for its VMs and network devices. If the owning organization
wants to expand capacity by cloning the server farm, how can the two duplicate farms
functions with duplicate IP addresses? One technique to address this issue is fencing,
which isolates a virtual network behind an IP address–mapping service similar to the Port
Address Translation service embedded in many routers. All virtual networks are assigned
to a single external IP address, and the virtualization layer implements a virtual router
that directs traffic from this address to virtual networks, remapping IP addresses and ports
as needed to segregate the connections. In this way, existing VMs and virtual networks
can be cloned and operated without changing any IP address assignments.

ECONOMICS OF VIRTUALIZATION
As with many systems architecture trends, user adoption of virtualization is largely driven
by economic considerations. The economic advantage of virtualization is based on a
complex combination of costs and benefits, including the following:
• Cost and efficient use of hardware
• Cost and efficient use of supporting physical resources, such as space, power,
and cooling
• Software costs
• System administration and support costs
• Deployment flexibility and scalability
The following sections explore these factors in depth and summarize them by exam-
ining a few common deployment settings.

Efficient Use of Computer Hardware

Two conflicting factors determine comparative hardware efficiency in virtualization
settings:
• Resources consumed by the virtualization layer
• Aggregating hardware resource demand

Virtualization Layer Resource Requirements

As described, virtualization adds at least one extra layer to the software layers forming a
functioning computer. Although there are substantial benefits to using software layers,
each additional layer imposes a performance penalty that isn’t always easy to measure. In
virtualization settings, this penalty is incurred when a VM accesses underlying physical
 Economics of Virtualization

hardware resources, such as memory, disk, and network interfaces. In traditional deploy-
13
ments, an application program accesses hardware resources via OS service calls, which
then access the underlying hardware resources. In a virtualization setting, client OS
accesses to VM hardware are intercepted by the virtualization layer, which redirects these
accesses to underlying physical hardware resources or accesses physical resources on
behalf of the client OS.
If virtual and physical hardware devices are similar, access redirection is efficient. For
example, if the physical CPU is a multicore AMD Opteron, and the VM CPU is a single-core
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

CPU of the same type, CPU instructions from the VM can be passed through the virtuali-
zation layer unchanged. In essence, the process of directing VM CPU instructions to phys-
ical cores is a scheduling process similar to that in any current OS. Similarly, enabling
multiple VMs to access a single physical NIC is a switching or routing task handled in a
single computer rather than across a network of computers and connecting devices.
If the VM and physical hardware have major differences, efficiency is much lower
because the virtualization layer must translate accesses to virtual hardware into accesses
to a different physical hardware type. Changing the previous example, if the VM CPU uses
a different instruction set, such as that of an IBM POWER processor, every POWER
instruction the client VM issues must be translated into an equivalent instruction for an
AMD Opteron. Similarly, differences in physical and virtual disk types, such as SCSI
versus SATA, require protocol conversion for every disk access.
Typically, hypervisors and virtualization environments limit the range of possible differ-
ences between physical and virtual hardware devices. For example, most commercial hyper-
visors require instruction-level compatibility of physical and virtual CPUs. Limited variation in
secondary storage and I/O is supported with protocol conversion and other translation func-
tions performed in the virtualization layer, device drivers on the client VM, or both.
Regardless of the approach taken to bridging differences between physical and virtual
hardware, the mere existence of an extra software layer introduces some inefficiency
because this layer consumes some physical resources to perform functions such as sched-
uling, switching, routing, and interrupt processing. As the differences between physical
and virtual hardware increase, the virtualization layer’s job becomes harder and consumes
more physical hardware resources, which are then unavailable to client VMs.
A key question in any virtualization deployment setting is what’s the performance
penalty? The answer is complex because it varies across virtualization layers, VM resource
demands, and the range of physical/virtual hardware differences that are supported. As a
result, there’s no universal formula for computing the performance penalty. Instead, simu-
lation and live testing are required to estimate the penalty in a variety of situations.
Recent tests in server consolidation with the latest versions of Microsoft and VMware vir-
tualization software generally estimate the performance penalty as between 5% and 10%.
In other words, the hardware demands of a single VM running on a bare-metal hypervisor
are approximately 5% to 10% higher than having the same server OS and application soft-
ware installed on the underlying hardware. Performance penalties are much higher in
desktop virtualization environments, particularly with major differences between physical
and virtual CPU architecture.
The performance penalty can be converted to an estimated monetary cost if the cost
of underlying physical hardware is known. For example, if the server requires a $50,000
computer to achieve adequate performance when installed on the hardware, the cost of
 Virtualization

supporting this server as a VM can be estimated between $52,500 and $55,000. The difference
14
($2500 to $5000) is the cost of the hardware performance penalty associated with virtualizing
the server. Note that this calculation ignores many other costs and benefits described in the
following section, so it’s only one part of a virtualization cost-benefit analysis.

Aggregating Hardware Resource Demand

As discussed, each VM incurs a performance penalty that can be converted into a mone-
tary cost estimate. However, you can’t estimate the total cost of virtualization perfor-

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
mance penalty for a group of servers by adding the performance penalty cost per server.
The reasons that summing these costs per server don’t work are roughly summarized by
the phrase “economies of scale.”
In virtualization settings, economies of scale are largely achieved by aggregating ser-
vers’ resource demand patterns. As an example, say you have a rack of four same-sized
midrange computers, each performing one of these specialized server functions:
• Database management
• E-commerce Web site
• E-mail and messaging exchange
• Production scheduling
The four lines at the bottom of Figure 4 represent the percentage of available hard-
ware resources each server consumes throughout the day. Only three servers ever

300

250
midrange computer
% capacity of one

200

150

100

50

0
0– 0

0– 0

0– 0

0– 0

0– 0

0
0– 0
0– 0

0– 0

0– 0

0– 0
0– 0
:0

:0
0
0

0
0
04 04:

06 06:

08 08:

22 22:
20 20:
10 10:

14 14:

6:

8:
2:
02

24
1

1
1
0–

:0
:0

:0

:0

:0

:0

:0

:0

:0

:0
:0

:0
00

02

12

16

18

Time of day

Web server E-mail server

Production scheduling server Database server
Aggregate demand Aggregate demand with VMs

FIGURE 4 Plotting individual and aggregate capacity utilization for four servers
Courtesy of Course Technology/Cengage Learning
 Economics of Virtualization

consume all available capacity and do so for brief peak periods during the day. At other
15
times, these resources are underutilized or idle.
What would happen if the applications on all four servers were combined on a single
larger server? This server would be busiest in the late morning and early afternoon,
reflecting the aggregate resource demand of all applications. However, as shown by the
thick gray line in the upper part of the graph, aggregate demand never exceeds 250% of a
single smaller server. Aggregate demand is lower because hardware resources are shifted
from one application to another as demand rises and falls.
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

Now return to the issue of virtualization performance penalty, which had been esti-
mated at 5% to 10% for typical server consolidations. The thick black line in the upper part
of the graph shows aggregate demand if the single large server uses a virtualization layer to
host four independent server VMs with a 10% performance penalty. Peak aggregate
demand is approximately 275% of one of the original physical servers. Therefore, the
combined load of all four servers operating as VMs can be accommodated by a cluster of
three of the original physical servers or a single larger server with roughly 75% of the four
servers’ combined capacity. Aggregating the smaller servers in this example as VMs
reduces hardware cost by about 25%, assuming that hardware cost scales linearly with
capacity. Had the original servers been underutilized (a common occurrence), hardware
savings could be much higher. So for many organizations, virtualizing existing servers
provides the same capabilities with a smaller hardware investment.
Although this example illustrates a clear cost savings, actual savings will vary,
depending on the servers being aggregated, the performance penalty for virtualization, and
the size of the original and aggregated hardware platforms. As discussed in Chapter 2, cost
per unit of computing power tends to be lowest for smaller computers compared with
larger ones. Therefore, aggregating many inexpensive servers as VMs in a mainframe might
not yield substantial cost savings because mainframes tend to cost more per unit of com-
puting power than small midrange computers. However, the cost differential between mid-
range and mainframe computers is smaller now than in past decades, and a cluster of
midrange computers can often provide mainframe power at midrange cost.

Supporting Physical Resources

Reducing the number of computers needed to meet aggregated hardware resource demand
saves more than just the cost of computer systems. It also reduces physical resource
requirements, such as server room space, power consumption, and cooling. When consid-
ered over the three- to five-year life of a typical server, these savings can add up to
thousands or tens of thousands of dollars per server rack.

Software Costs
It isn’t clear yet whether software costs are higher or lower in virtualization settings, for
reasons that include the following:
• Widespread virtualization use is a fairly recent development.
• The cost of virtualization layers is changing rapidly.
• Software vendors are struggling to adapt their pricing and usage agreements
to deal with virtualization.
 Virtualization

All other factors being equal, a virtualized deployment should be more expensive than
16
a nonvirtualized deployment by the virtualization layer’s cost. Software costs might
increase because of licensing system and application software on multiple smaller VMs
compared with one larger physical machine. However, software vendors are trying to
adapt their license pricing so that virtualization isn’t a factor. Therefore, they now offer
organization-wide licenses for a fixed fee, or they price software based on a specific
number of physical CPUs.

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
System Administration and Support Costs
The most important benefit of virtualization is a reduction in system administration and
support costs, although reducing these costs isn’t an automatic benefit, and the savings
vary substantially across deployments. The costs savings are a result of several factors,
including the following:
• Consolidating hardware at fewer locations
• Reducing the number of physical computer systems
• Cloning commonly used VMs
System administration and support costs tend to rise with the number of physical
hardware locations, the number of physical hardware devices, and variations in configur-
ing physical hardware devices. Virtualizing servers and workstations on fewer clustered
computers reduces the system administration and support workload by reducing the
number of physical computers and concentrating them in fewer locations. Although this
point might seem fairly obvious, the related cost savings can be substantial because each
full-time system administrator or support technician costs $100,000 or more per year.

NOTE
The cost of a full-time employee is usually 150% to 200% of salary when costs of benefits, training,
office space, and supporting equipment are added.

Cloning common server and desktop configurations can simplify administration and sup-
port. To understand VM cloning, it helps to know how a VM is stored in a hypervisor or virtua-
lization environment. Figure 5 shows the contents of a directory containing a VM controlled by
VMware Workstation. This list displays a handful of file types, such as the following:
• .nvram—BIOS information
• .vmsd—Information about VM snapshots (saved machines states)
• .vmx—Configuration information describing virtual hardware and other
settings
• .vmxf—Configuration information for VMs that run as a group
• .vmdk—Virtual disk file containing all disk content (OS, applications, and
data files)
• .log—Log file describing VM startups, shutdowns, and errors
 Economics of Virtualization

17
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

FIGURE 5 Files making up a virtual machine in VMware Workstation

Courtesy of Course Technology/Cengage Learning

The simplest way to clone a VM is to copy the files shown in Figure 5. If the original
and copied VMs are then started, two identical machines will be in use. Cloning by making
a simple copy can create problems, however, as when two VMs have the same static IP
address or have identical names or ID numbers in a centralized directory. Virtualization
environments have utilities for cloning an existing VM that enable the system administra-
tor to change per-machine settings, such as IP addresses, machines names, and machine
ID numbers. With these utilities, system administrators can create many distinct clones
of a single VM quickly.

NOTE
Providing unique identifiers for cloned VMs is a requirement for some OS licenses.

Cloning VMs simplifies system administration and support by reducing the number of
different machine configurations that must be developed and supported. For example, an
organization might operate a variety of Web servers, and each can be created as a clone of
a standard configuration. All clones will have the same OS and applications as the original,
simplifying administrative and support tasks for the entire group.
Cloning can be supplemented with other techniques, such as snapshots and templates.
A snapshot is a stored machine state captured at a specific point in time. Changes made
past that time are recorded in separate files, and the virtualization environment logs all
changes to disk content and VM configuration information so that the VM can continue
functioning correctly. At a future point, a VM can be rolled back to a snapshot state,
effectively resetting its configuration and disk content to an earlier time. Snapshots are
useful for testing and to simplify redeploying an existing VM.

NOTE
To prevent damage and ensure the consistency of disk and memory content, a VM must be suspended
or powered down during cloning or when taking a snapshot.
 Virtualization

A VM template combines aspects of cloning and snapshots. A template defines a base

18
configuration that can be extended. For example, a system administrator might define a
basic desktop workstation VM with Windows, Microsoft Office, and customizations for the
organization, such as security settings and desktop background. This VM can be stored as
template, and more specialized version of it can be created with different applications and
hardware configurations for different groups of employees. In addition, a customized VM
can be rolled back to the base template, if needed. In some virtualization environments,
updates to the template, such as a security patch, can be applied automatically to all VMs

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
derived from the template.
The combination of reducing the number of platforms, consolidating these platforms in
fewer locations, and reducing the number of supported configurations can substantially reduce
the overall cost and complexity of system administration and support. For many organizations,
these cost reductions are far more important than reductions in hardware costs.

Deployment Flexibility
Virtualization adds flexibility to how organizations deploy their computing resources.
Because describing all the possible ways flexibility can increase is difficult, the following
sections compare a few common deployments with and without virtualization.

Desktop Virtualization
Without virtualization, each employee has a desktop computer that requires installing
application software and other customizations. (Hardware vendors usually install OSs.)
Software updates and patches are installed regularly, and an expensive and complex
deployment software package is used to minimize the visits support personnel must make
to each workstation. Hardware is replaced every two to four years, and each replacement
requires reinstalling applications and customizing each workstation again.
With virtualization, few, if any, applications are installed on desktop computers, and
customizations are minimal. Instead, applications are installed on a template VM. When a
user accesses a workstation VM from his or her Web browser, a clone is created from the
template automatically. (User-specific settings and data files are stored on a server, not on
the VM.) When the user logs off the VM clone, it’s deleted. Updates and patches are
applied to the VM template, and users see these updates automatically at their next logon.

New Branch Office

Without virtualization, new server and workstation hardware are purchased, configured,
and installed at the new location. Configuration involves installing OSs, installing applica-
tions, and performing customizations. Completing this process takes several days with a
team of support personnel.
With virtualization, one possibility is purchasing and installing a server or cluster with
preinstalled virtualization software to serve as a computing infrastructure for the new
location. After the host is installed, server and workstation VMs for the new location can
be cloned from existing templates and copied to the new server. Alternatively, a new node
can be added to an existing virtualization cluster at a central location. Server and work-
station VMs are still cloned but run at the central location. Simple workstations or porta-
ble computers with minimal software are configured and installed at the new branch.
 Economics of Virtualization

Scaling Up an E-commerce Site

19
Without virtualization, rapid expansion in e-commerce traffic requires frequent hardware
upgrades and migrations, which are time consuming, labor intensive, and costly. Virtuali-
zation makes several new options possible, including simpler expansion of internal
resources or migration of part or all of the application to a cloud service provider.
Expanding internal resources requires adding hardware capacity to the virtualization host,
usually by adding nodes to a cluster. Server VM capacity is increased by cloning additional
servers from existing templates or by modifying the configuration of existing VMs to
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

include more computing resources, such as CPUs and memory. Creating additional clones
or adding virtual hardware to existing VMs can be done in minutes.
Instead of adding internal capacity, the organization could contract with a cloud com-
puting vendor that offer infrastructure as a service. With this setup, server VMs are copied
from the organization’s existing computers to the vendor’s computers. As with internal
expansion, capacity can be added by creating more server VM clones or by reconfiguring
existing VMs with expanded virtual hardware. The vendor could handle excess demand,
or the entire server farm could run on the vendor’s cloud.

Virtualization Costs and Risks

Although there’s a strong cost-benefit case for virtualization in many settings, it does entail
some costs and risks that aren’t encountered in traditional settings. First is the cost and
complexity of hardware. In the previous example of virtualizing four physical servers in
one cluster containing three of the original computers, the cluster’s complexity is probably
greater than the original collection of physical machines. Likely sources of additional
complexity (and cost) include a dedicated back-end network to support communication in
the cluster and a dedicated storage area network.
If each original server had its own internal secondary storage, the servers wouldn’t be
suitable for use in a clustered virtualization host because the storage devices couldn’t be
pooled easily. Even if each server had a dedicated external storage array, combining these
arrays into a single array would be difficult, and the array interfaces might not provide
enough data transfer capacity in a pooled environment. Therefore, a higher capacity
(and more expensive) secondary storage solution would be required.
Virtualization also increases risks by consolidating physical points of failure. For
example, say you’re virtualizing all desktop computers in a 200-person office. If the vir-
tualization host is unavailable, all 200 employees are idle. With physical workstations, the
failure of one workstation is unlikely to affect others. Similar risks occur in server consol-
idation. To mitigate these risks, hardware vendors use sophisticated monitoring and error
recovery hardware and software, which increase costs and add administrative complexity.
 Virtualization

20
Summary
• Virtualization is a combination of software, hardware, and networking technologies that
enable a single physical computer to act as though it were multiple independent computers.
A virtual machine (VM) is a software emulation of hardware resources that behaves like a
physical computer. Virtualization uses host software, called a virtualization layer or hypervi-
sor, to control physical hardware resources and configure these resources to support one
or more VMs. Hypervisors come in two basic types: bare-metal hypervisors and virtualiza-

© Cengage Learning. All rights reserved. No distribution allowed without express authorization.
tion environments.
• Virtualization was widely used in mainframe computing in the 1970s, fell out of favor in the
1980s and 1990s, and reemerged in the 2000s. VMware is the market leader in virtualiza-
tion software. Microsoft and the open-source Xen are its major competitors. Virtualization is
used now mainly for server consolidation, virtualization of desktop computers, and in soft-
ware testing and development.
• A virtualization layer manages physical resources and allocates them to one or more VMs.
To prevent VMs from interfering with one another or with themselves, the virtualization layer
intercepts interrupts from VMs that require access to physical resources, makes these
resources available, and then passes control back to the VM’s interrupt handlers. Duplica-
tion of interrupt handling degrades performance, which can be minimized by allocating
some physical resources exclusively to one VM. Some virtualization layers use customized
device drivers installed in VMs to improve performance.
• Hardware in general (and CPUs in particular) has evolved to include features that improve
virtualization performance. These features include extra registers, CPU operating modes for
the virtualization layer, and optimizations to supporting chipsets and I/O devices to stream-
line the interface between virtual and physical I/O devices. Intel CPUs enable a hypervisor
to trap CPUID instructions issued by VMs and alter the features reported to these VMs.
This capability enables computing clusters with multiple CPU types to report consistent
CPU features, which makes it possible to move running VMs to other cluster nodes.
• VMs go hand in hand with virtual networks, which are emulations of physical networks
implemented in a virtualization layer or host. Virtual networks interconnect VMs, emulated
network devices (such as routers and firewalls), and the virtualization host’s physical NICs.
Fully functional virtual networks might also require virtualized network services, such as
DHCP and DNS servers. Fencing virtualized networks enables cloned VMs with identical
MAC and IP addresses to operate concurrently on the same virtualization host.
• The benefits of virtualization outweigh its costs in many deployment settings. One benefit is
that consolidating multiple VMs on a single computer or cluster usually reduces overall
hardware requirements. Consolidation also reduces supporting resource requirements, such
as space, power, and cooling. System administration and support costs are usually reduced
by limiting the number of VM configurations and using clones, templates, and snapshots to
manage these configurations more efficiently. Combining these benefits yields a higher
degree of deployment flexibility and scalability than can be achieved without virtualization.
Additional costs and risks of virtualization include the cost of the virtualization layer, the
need to use higher performance hardware and software in virtualized settings, and the risks
associated with fewer physical points of system failure.
 Research Problems

Key Terms 21
bare-metal hypervisor virtual machine (VM)
fencing virtual network
hypervisor virtualization
snapshot virtualization environment
template virtualization layer
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.

Research Problems
1. Investigate current versions of VMware and Microsoft products to support server consolida-
tion through virtualization. Which architectural features and capabilities are similar, and
which are different? In particular, examine the dependency between Hyper-V and Windows
Server. Is Hyper-V a ”true” hypervisor, a virtualization environment, or something in
between?
2. Investigate the hardware features of AMD and IBM processors and other hardware that
support more efficient virtualization. Compare these features with Intel VT.
© Cengage Learning. All rights reserved. No distribution allowed without express authorization.