Performancemonitoring 100409184324 Phpapp01

Developing Performance Monitoring Tools Using vSphere APIs: Techniques and Best Practices TEXIBP 1005 Balaji Parimi
& Ravi Soundararajan, VMware Inc.

Copyright 2010 VMware, Inc. All rights reserved. This product is protected by U.S. and international copyright and intellectual property laws. VMware products are covered by one or more patents listed at http://www.vmware.com/go/patents. VMware is a registered trademark or trademark of VMware, Inc. in the United States and/or other jurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.
Disclaimer
This session may contain product features that are currently under development. This session/overview of the new technology represents no commitment from VMware to deliver these features in any generally available product. Features are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery. Pricing and packaging for any new technologies or features discussed or presented have not been determined.
These features are representative of feature areas under development. Feature commitments are subject to change, and must not be included in contracts, purchase orders, or sales agreements of any kind. Technical feasibility and market demand will affect final delivery.
Follow Us
Join the conversation Get the latest scoop Be a part of the network
Twitter t
Facebook
Search for VMware Partner Network
Search for VMware_Partners
Linkedin
Subscribe to RSS Search for VMware Partner Network
http://blogs.vmware.com/powerofpartnership/
Follow this event LIVE! #VMwarePEX2010
Help us help you! Please fill out the survey at the end of the course.
Who is our Target Audience?

System Administrators: Monitor the performance of their virtual infrastructure Understand the bottlenecks in the datacenter to help reconfigure their storage
or network topology Plan for future growth in the datacenter
VMware ISV Partners: Collect the relevant/important performance counters and hand them off to a
performance analysis tool Create a wizard/tool that helps system administrators troubleshoot their datacenter
Agenda
Use Cases Building a Performance Monitoring Application Q&A
Use Cases
Use Cases: What do I look for and why? CPU Memory Disk Network
CPU
ESX is designed to run Virtual Machines Schedulable entity = world Virtual Machines are composed of worlds (mks, vCPUs) Service Console (agents like vpxa, hostd) (Classic ESX) Proportional-share scheduler for resource management Limits, Shares, and Reservations World states (simplified view): ready = ready-to-run but no physical CPU free run = currently active and running wait = blocked on I/O
So, How Do I Spot CPU Performance Problems?

One common issue is high CPU ready time High ready time
possible contention for CPU resources among VMs
Many possible reasons

CPU over commitment (high %rdy + high %used) Workload variability Limit set on VM
No fixed threshold, but > 20% for a VCPU
Investigate further
CPU: Useful Metrics

Per-Host
Metric (Client) Usage (%)
Metric Metric (sdk) (esxtop) %USED cpu.usage.average
Description CPU used over the collection interval (%) CPU used over the collection interval (MHz)
Usage (MHz)
n/a
cpu.usagemhz.average
CPU: Useful Metrics

Per-VM
Metric (Client) Usage (%) Metric (esxtop) %USED Metric (SDK) cpu.usage.average Description CPU used over the collection interval CPU used over the collection interval)* CPU time spent in ready state* CPU time spent waiting for hostlevel swap-in
Used (ms)
%USED
cpu.used.summation
Ready (ms) Swap wait time (ms) [ESX4.0 hosts]
%RDY %SWPWT
cpu.ready.summation cpu.swapwait.summation
* Units different between esxtop and vSphere client

10
Spotting CPU Overcommitment in esxtop
2-CPU box, but 3 active VMs (high %used) High %rdy + high %used can imply CPU overcommitment
11
Caveat on Ready Time: Workload Variability
Some caveats on ready time Used time ~ ready time: may signal contention. However, might not be overcommitted due to workload variability In this example, we have periods of activity and idle periods: CPU isnt overcommitted all the time
Used time
(screenshot from VI Client)
Ready time ~ used time
Ready time < used time
12
Further ready time examination
High Ready Time High MLMTD: there is a limit on this VM High ready time not always because of overcommitment When you see high ready time, double-check if limit is set
13
Further Ready Time Investigation: No Fixed ThresholdWhy?
ODBC Sniff traffic vSphere VM Probe VM vSphere DB
ESX 1
ESX 2
vSphere communicates with DB Probe VM monitors vSphere-to-DB traffic The more traffic, the more work done by Probe VM
14
Why Isnt There A Fixed Ready Time Threshold?
Ready time jump from 12.5% (idle DB) to 20% (busy DB) didnt notice until responsiveness suffered!
15
Summary of Possible Reasons for High Ready Time

CPU overcommitment
Possible solution: add more CPUs or VMotion the VM
Workload variability
A bunch of VMs wake up all at once Note: system may be mostly idle: not always overcommitted
Limit set on VM
4x2GHz host, 2 vcpu VM, limit set to 1GHz (VM can consume 1GHz) Without limit, max is 2GHz. With limit, max is 1GHz (50% of 2GHz) CPU all busy: %USED: 50%; %MLMTD & %RDY = 150% [total is 200%, or 2 CPUs]
16
Where do we go from here?

Use Cases: What do look for and why
CPU Memory Disk Network
17
Memory
ESX must balance memory usage for all worlds
Virtual machines, Service Console, and vmkernel consume memory Page sharing to reduce memory footprint of Virtual Machines Ballooning to relieve memory pressure in a graceful way Host swapping to relieve memory pressure when ballooning insufficient
ESX allows overcommitment of memory Sum of configured memory sizes of virtual machines can be greater than
physical memory if working sets fit
18
Ballooning vs. Swapping (1)

Ballooning: Memctl driver grabs pages and gives to ESX
Guest OS choose pages to give to memctl (avoids hot pages if possible): either free pages or pages to swap
- Unused pages are given directly to memctl - Pages to be swapped are first written to swap partition within guest OS and then given to memctl
VM1 F memctl 2. Reclaim 3. Redistribute Swap partition w/in Guest OS 1. Balloon
VM2
ESX
19

Swapping: ESX reclaims pages forcibly
Guest doesnt pick pagesESX may inadvertently pick hot pages ( possible VM performance implications) Pages written to VM swap file
VM1
VM2
Swap Partition VSWP (w/in guest) (external to guest)
ESX
1. Force Swap 2. Reclaim 3. Redistribute
20

Bottom line: Ballooning may occur even when no memory pressure just to keep memory
proportions under control
Ballooning is vastly preferably to swapping

Guest can surrender unused/free pages
- With host swapping, ESX cannot tell which pages are unused or free and may accidentally pick hot pages
Even if balloon driver has to swap to satisfy the balloon request, guest chooses what to swap
- Can avoid swapping hot pages within guest
21
Ok, So Why Do I Care About Memory Usage?

If running VMs consume too much host memory Some VMs do not get enough host memory This forces either ballooning or host swapping to satisfy VM demands Host swapping or excessive ballooning reduced VM performance If I do not size a VM properly (e.g., create Windows VM with 128MB RAM) Within the VM, swapping occurs, resulting in disk traffic VM may slow down Butdont make memory too big! (High overhead memory)
22
Important Memory Metrics (Per VM)

Metric (Client)
Swap in rate (ESX4.0 Hosts) Swap out rate (ESX4.0 Hosts) Swapped Swap in (cumulative) Swap out (cumulative)
Metric Metric (SDK) (esxtop)

SWR/s SWW/s SWCUR n/a n/a mem.swapinRate.average mem.swapoutRate.average
Description
Rate at which mem is swapped in from disk Rate at which mem is swapped out to disk
mem.swapped.average (level 2 ~swap out swap in counter) mem.swapin.average mem.swapout.average Mem swapped in from disk Mem swapped out to disk
One rule of thumb: > 1MB/s swap in or swap out rate may mean memory overcommitment
23
Important Memory Metrics (Per Host, sum of VMs)

Metric (Client)
Swap in rate (ESX4.0 Hosts) Swap out rate (ESX4.0 Hosts) Swap used Swap in (cumulative) Swap out (cumulative)
Metric Metric (SDK) (esxtop)

SWR/s SWW/s SWCUR n/a n/a mem.swapinRate.average mem.swapoutRate.average mem.swapused.average (level 2 counter) mem.swapin.average mem.swapout.average
Description
Rate at which mem is swapped in from disk Rate at which mem is swapped out to disk ~swap out swap in Mem swapped in from disk Mem swapped out to disk
One rule of thumb: > 1MB/s swap in or swap out rate may mean memory overcommitment
24
Example of Swapping
Increased swap activity may be a sign of over-commitment
No swapping Lots of swapping
25
A Stacked Chart (per VM) of Swapping
Lots of swapping
No swapping
26

27
Disk Performance Problems 101

What should I look for to figure out if disk is an issue?
Am I getting the IOPs I expect? Am I getting the bandwidth (read/write) I expect? Are the latencies higher than I expect? Where is time being spent?
What are some things I can do? Make sure devices are configured properly (caches, queue depths) Use multiple adapters and multipathing Check networking settings (for iSCSI/NAS)
28
Useful Disk Metrics
Metric (Client)
Commands
Metric (Esxtop)
CMDS/s
Metric (SDK)
disk.commands.summation
Description
commands issued in the sampling interval KB/s read* KB/s written* Average latency at device Average latency in vmkernel Total latency for command
Read rate Write rate Device latency Kernel latency Command latency
MBREADS/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd
disk.read.average disk.write.average disk.deviceLatency.average disk.kernelLatency.average disk.totalLatency.average

29
Disk Performance Example: vSphere Client
SAN cache enabled: High Write Throughput
SAN cache disabled: Poor throughput
30
Another Disk Example: Slow VM Power On

Trying to Power on a VM Sometimes, powering on VM would take 5 seconds Other times, powering on VM would take 5 minutes! Where to begin? Powering on a VM requires disk activity on host
Check disk metrics for host
31
Lets look at the vSphere client
Max Disk Latencies range from 100ms to 1100msvery high! Why? (counter name: disk.maxTotalLatency.latest)
32
Using Esxtop to Examine Slow VM Power On
Note very large DAVG/cmd and GAVG/cmd Rule of thumb: GAVG/cmd > 50ms = high latency! What does this mean? Latency when command reaches device is high Latency as seen by the guest is high Low KAVG/cmd: command isnt queuing in VMkernel Whats up?
33
High Disk Latency: Mystery Solved
Host events: disk has connectivity issues
high latencies!
Bottom line: monitor disk latencies; issues may not be related to virtualization!
34
A More Complex Disk Example
Ethernet VM1 VM2 VM3 VM4
VM15
NAS
ESX
Disk Traffic
VMs communicate with NAS (network traffic) NAS writes to disk (disk traffic)
35
Combining Metrics: A More Complex Disk Example

Setup: Group of Virtual Machines running on a host Each Virtual Machine talks to a Virtual Machine serving as a NAS device Problem: Suddenly, I cannot log in to any of the Virtual Machines (really slow!) Initial Speculation Virtual Machines are saturated in some resource
36
A More Complicated Disk Example, Part I (CPU)
Predictable CPU usage, Host not saturated
Chaotic CPU usage, Host saturated
37
Complicated Disk Example, Part 2: Disk Usage
Predictable, balanced disk usage Uneven, reduced disk usage
38
Complicated Disk Example, Part 3: Write Rate
Read and write traffic Increased write traffic, zero read traffic
39
Complicated Example: Putting it all Together
Each App Virtual Machine reads to & writes from same NAS Something caused excessive writes from each Virtual Machine Increased CPU usage per Virtual Machine Increased write traffic per Virtual Machine Ton of writes on NAS VM! Issue Bug in application within Virtual Machine caused error condition Error condition caused excessive writes to same NAS
- Network traffic for application VMs, disk traffic on NAS VM
Each Virtual Machine is so busy writing that it never reads
40

41
Network Performance Problems 101

What should I look for to figure out if network is an issue?
Am I getting the packet rate that I expect? Am I getting the bandwidth (read/write) I expect? Is all traffic on one NIC, or spread across many NICs? [more advancednot available through counters]: out-of-order packets?
What are some things I can do? Check host networking settings (full-duplex/half-duplex, 10Gig network vs
100Mb network?, firewall settings)
Check VM settings: all VMs on proper networks?
42
Useful networking metrics
Metric (Client)
Metric (esxtop)
Metric (SDK)
Description Packets transmitted in sampling interval Packets received in the sampling interval Amount of data transmitted per second* Amount of data received per second*
Packets PKTTX/s net.transmitted.average transmitted (in sampling interval) Packets received (in sampling interval) Data transmit rate (KBps) Data receive rate (KBps) PKTRX/s net.received.average
MbTX/s
net.transmitted.average
MbRX/s
net.received.average

43
Network Performance Troubleshooting

Customer complains about slow network
Shes running netperf on a GigE Link She sees only 200Mbps Why? I bet its that VMware stuff!! - Note to reader: Please dont blame VMware first
Where do we start?
44
Where do we begin? Check VM Bandwidth.
Measure VM Bandwidth (net.transmitted.average)

200 Mb/s Screenshot from the vSphere client
45
Check Host Bandwidth

Measure Host Bandwidth (net.transmitted.average)
Host sees around 900Mbpswhy is VM at 200Mbps?
Hmmare we sharing this NIC with multiple VMs?
46
All VMs using same NIC (VM Network)
All VMs using VM Network and sharing 1 physical NIC
47
All Traffic is Going Through One Nic!

Measure per-physical-NIC traffic
All traffic through one NIC on this host
Hmmall VM traffic is going through 1 NIC Lets split the VMs across NICs
48
Split VMs Across Multiple NICs. Bingo!
49
So Complicated! Where do I begin?

So much to track But I have different needs! For example
50
What if I want esxtop for multiple VMs at once?
vTop: A stats display similar to esxtop, but showing multiple metrics per VM in one screen
51
What if I want to see if DRS is working?
Highlight the effectiveness of DRS by overlaying stats and events data

52
What if I Want to Compare Guest/Host Metrics on One Chart?
Overlay stats data from the ESX and the GOS

53
What If I Want To Compare the Throughput of 2 RPs?
Aggregate stats data by resource pools, clusters, group of VMs(Windows, Linux), vApps, etc.
54
Sounds Greatbut how do I do this stuff?

Basic Steps 1. Collect the Stats you Need 2. Interpret the Stats 3. Present the stats Observations 1. Stats collection is similar across customer environments 2. Interpretation/Presentation is specific to customer Stats collection is a major pain point 1. For customers: so many stats, complex API 2. For us: we get the same questions over and over again Soweve created a reference application for Stats Collection
55
Agenda
Use Cases Building a Performance Monitoring Application Q&A
56
Building a Performance Monitoring Application

Monitoring Applications consist of a few standard parts:
Selecting entities to monitor (hosts, VMs, etc.) Selecting metrics to monitor (CPU, Memory, etc.) Choosing collection frequency Keeping abreast of inventory changes Collecting stats Presenting data in a meaningful way
57
StatsFeeder: A Reference Monitoring Application
Present data in a meaningful way
Collect statistics (note: many sources!)
Keep track of inventory changes Select Entities and Metrics to monitor and set collection frequency
58
A Brief Walkthrough of StatsFeeder
1. Select Entities/Metrics to Monitor 2. Keep Track of Inventory Changes 3. Collect Statistics 4. Present Statistics
59
Selecting Entities/Metrics to Monitor
Relevant Modules in StatsFeeder

1. Counter Info Initializer 2. User Input Reader 3. Entity MoRef Retriever 4. Entity Metrics and Performance Provider Summary
60
Counter Info Initializer

Retrieves available counters. Tip: Counter information is static:
retrieve only once.
61
Counter Info Initializer: Code Snippet

The PerformanceManager managed object contains a property with the name perfCounter It is an array of type PerfCounterInfo. Use PropertyCollector to retrieve this property and get the supported counters.
PropertySpec pSpec = new PropertySpec(); pSpec.setType(PerformanceManager); pSpec.setPathSet(new String[]{perfCounter});
ObjectSpec oSpec = new ObjectSpec(); oSpec.setObj(serviceContent.getPerfManager());
62
User Input Reader

Reads the configuration file for: The entities for which the
performance statistics are being retrieved.
Specify individual entities by name Specify the name of the container and the child type to monitor the entities in that container
The performance statistics to

retrieve.
The frequency at which the

performance data should be retrieved.
63
User Input Reader: Sample Configuration File
64
Entity MoRef Retriever

Retrieves MoRefs of entries in configuration file. Tricks: MoRef does not change once an item is
added to the vCenter inventory.
MoRef does change if entity is removed

and re-added.
Therefore: Maintain cache of MoRefs Monitor changes to the inventory Use Views to retrieve MoRefs
65
Entity MoRef Retriever

Retrieve the MoRef of individual entities using PropertyCollector or SearchIndex. Retrieve the MoRefs of entities in a container using ContainerView: Obtain the MoRef of the container Create the ContainerView ContainerView has a property with the name view and it is an array of type
ManagedObjectReference. This contains the MoRefs of all the types specified during the ContainerView creation.
vimPort.createContainerView( serviceContent.getViewMgr, containerMoRef, new String[]{VirtualMachine},//MO type true //recursively look in the container );
66
Entity Metrics and Performance Provider Summary

Retrieve the metric IDs for all the entities. E.g.: For dual CPU VM, metric IDs are 0
and 1 for CPU 0 and CPU 1.
Caveat: Cant get this for powered off

VMs and host systems, disconnected hosts and orphaned VMs
Tricks: Performance Metric IDs for entities do

not change if hardware does not change.
Therefore: Cache Metric IDs for entities. Tip: Use wild card for metric IDs
67
Entity Metrics
vimPort.queryAvailablePerfMetric ( serviceContent.getPerfManager(), entityMoRef, null, null, intervalID //If not specified, defaults to //available metrics for historical stats. );
68
Entity Metrics and Performance Provider Summary

Performance Provider Summary provides information about: The type of statistics it supports (realtime or historical)
The refresh rate for statistics

generation
Performance Provider Summary data is the same for managed objects of the same type. Tricks: You dont have to retrieve this data for
all entities.
Make sure the interval between two

performance data queries is equal to at least the refresh rate to avoid getting the same value.
69
Performance Provider Summary
vimPort.queryPerfProviderSummary( serviceContent.getPerfManager(), entityMoRef, );
70
71
Keeping Track of Inventory Changes
Relevant Module in StatsFeeder:

Inventory/Entity Properties Monitor
72

Tracks changes to the inventory (to keep EntityMoRefCache up to date) Tracks changes to the virtual hardware of the VMs (to keep Entity Metrics cache up to date) Use Views to monitor these changes
73

To monitor for changes to inventory: Create InventoryView Open the inventory view folders to monitor for changes Monitor for changes to view property, as it contains the MoRefs of all the
managed objects in the inventory
vimPort.openInventoryViewFolder( inventoryViewMoRef, new ManagedObjectReference[]{}//MoRefs of //all the containers to expand );
74

To monitor for changes to VM hardware: Create ListView with the MoRefs of all the VMs Create the appropriate TraversalSpec to start from the created ListView Monitor for changes to VM hardware using PropertyCollector vimPort.createListView ( serviceContent.getViewManager(), new ManagedObjectReference[]{}//MoRefs of //all the VMs ); TraversalSpec tSpec = new TraversalSpec(); tSpec.setType(ListView); tSpec.setPath(view);
75

PropertySpec pSpec = new PropertySpec(); pSpec.setType(VirtualMachine); pSpec.setPathSet(propertyArr);//Props String array
ObjectSpec oSpec = new ObjectSpec(); oSpec.setObj(viewObj);//View object oSpec.setSelectSet(new SelectionSpec[]{tSpec});
76
77
Collecting Statistics
Relevant Modules in StatsFeeder

1. PerfQuerySpec Cache 2. Stats Reader 3. Stats Receiver
78
PerfQuerySpec Architecture
To grab counters: QueryPerf(PerfQuerySpec[] querySpec) PerfQuerySpec: Specifies which counters to grab
Entity (host, VM) Format (CSV, normal) MetricId StartTime EndTime IntervalID (20s, 300s)
maxSample
PerfQuerySpec[]: [pQs1, pQs2, pQs3, ] Array of PerfQuerySpec objects pQs1, pQs2, pQs2 Can grab multiple stats using single QueryPerf call
79
Complexities of QueryPerf
How Does vSphere Process QueryPerf(querySpec[])? 1. vCenter receives queryPerf request with querySpec[] 2. vCenter takes each querySpec one at a time 3. vCenter gets data for each querySpec before processing next one Options for querySpec[]: 1. 1 entry

1 stat or set of stats for a single entity (e.g., all CPU)

pQs1 VM1,cpu.* pQs2 VM2,cpu.* pQs3 H3,mem.*
2. Multiple entries. Examples:

Each entry for a different entity
Each entry for a different stat type, same entity VM1,cpu.* VM1,net.* VM1,mem.*
80
Implications of QuerySpec
Format of QuerySpec Allows Multiple Client Options 1. Grab each stat one at a time 2. Grab a group of stats per entity at once 3. Grab all stats for all entities at once 4. Grab stats for a subset of entities at once Some Tradeoffs: 1. Network processing (large result sets vs. small result sets) 2. Client aggregation overhead 3. vCenter processing (Each QueryPerf handled in a single thread)
81
PerfQuerySpec Cache
Reuse PerfQuerySpec, as long as there are no changes to the inventory and changes to the hardware of the corresponding entity. Tricks: Keep the PerfQuerySpec cache up to
date using the monitoring module.
Use wild card for metric IDs to

automatically account for hardware changes.
82
PerfQuerySpec
PerfQuerySpec pqSpec = new PerfQuerySpec(); pqSpec.setIntervalId(intervalId); pqSpec.setMaxSample(maxSamples); perfQuerySpec.setFormat(csv);//Set format to CSV
PerfMetricId pmid = new PerfMetricId(); pmid.setCounterId(counterId); pmid.setInstance("*");//Wild card pqSpec.setMetricId(new PerfMetricId[]{pmid}); pqSpec.setEntity(entityMor);
83
Stats Reader
Retrieves the performance data from the vCenter Server and various guest specific performance tools like xperf, perfmon, or top etc. Tip: Use parallelism
84
Stats Reader - Using Parallelism

class StatsReader implements Runnable { private PerfQuerySpec[] perfQuerySpecArr = null; public StatsReader(PerfQuerySpec[] pqsArr) { perfQuerySpecArr = pqsArr; } public void run() { if (perfQuerySpecArr != null) { vimPort.queryPerf (serviceContent.getPerfManager(), perfQuerySpecArr); } } }
85
Stats Receiver
This is a simple interface to process the output data from the StatsReader. By default CSVFileStatsReceiver and
DefaultStatsReceiver are provided.
CSVFileStatsReceiver Writes the performance data in CSV format to a file DefaultStatsReceiver Writes the performance data to the console.
Users can simply implement this interface to create their own StatsReceivers to: Write the performance data to a
database.
Publish the data to a message queue.
86
Stats Receiver
/* All the receivers need to implement an IStatsReceiver interface that has receiveStats method */ public void receiveStats(String entityName, PerfEntityMetricBase[] pembArr) {
writeStatsToDB(entityName, pembArr); }
87
88
Present Data in Meaningful Way
Relevant modules in StatsFeeder

1.Stats Interpreter 2.Stats Presenter
89
Stats Interpreter
Interprets the performance data to identify potential problems. You can write your own rules engine to interpret the performance data and plug it in easily.
90
Stats Presenter
Interpret the data using your own rules and present it in your own way. E.g.: Present the data in graphical
format using histograms, charts etc.
91
Stats Presenter Sample
92
Summary of Pieces
Present data Sample Rules Engine Store data (CSV, DB, etc.) Collect Data Watch for inventory changes Maintain cache of querySpecs Retrieves metadata for entities to be monitored and stats to be collected Retrieves MoRefs of entities in config file Get entities to monitor, stats to retrieve, and collection interval from config file What counters are supported?
93
Conclusion: Performance API Tips and Tricks

Collect static data once CounterIDs, metricIDs, MoRefs etc. Use Views to keep this data up to date. Reuse PerfQuerySpec as much as possible Use CSV format Reduces serialization cost and the size of metadata Choose metrics and query intervals carefully Query the real-time stats at a slower rate than the refresh rate Query over small time increments Choose correct stats levels Use parallelism (multi-threaded clients)
94
Resources for Developers

StatsFeeder Source Code: http://communities.vmware.com/servlet/JiveServlet/downloadBody/ 11933-102-1-11789/StatsFeeder.zip Developer Community: http://communities.vmware.com/community/developer/forums/mana gementapi Code Samples: http://communities.vmware.com/community/developer/codecentral Blogs: http://blogs.vmware.com/developer
95
Q&A
Questions ??? Please complete the survey.
96
Ready. Set. Go! Take Action

GO! Get Educated
Visit Partner University for information on partner

programs and resources Access to VMware Partner Exchange presentations
GO! Get Competent
Enroll in training and certification that counts

toward earning competencies
GO! Register Your Deals to Earn More Margin
Visit Partner Central to learn about VMwares

registration programs
97
Any Questions?
98

Performancemonitoring 100409184324 Phpapp01

Diunggah oleh

Informasi Dokumen

Deskripsi Asli:

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Performancemonitoring 100409184324 Phpapp01

Diunggah oleh

Hak Cipta:

Format Tersedia

Developing Performance Monitoring Tools Using vSphere APIs: Techniques and Best Practices TEXIBP 1005 Balaji Parimi

& Ravi Soundararajan, VMware Inc.

Search for VMware_Partners

Follow this event LIVE! #VMwarePEX2010

Who is our Target Audience?

So, How Do I Spot CPU Performance Problems?

Many possible reasons

No fixed threshold, but > 20% for a VCPU

CPU: Useful Metrics

Metric (Client) Usage (%)

Metric Metric (sdk) (esxtop) %USED cpu.usage.average

CPU: Useful Metrics

Ready (ms) Swap wait time (ms) [ESX4.0 hosts]

* Units different between esxtop and vSphere client

Spotting CPU Overcommitment in esxtop

Caveat on Ready Time: Workload Variability

(screenshot from VI Client)

Ready time ~ used time

Ready time < used time

Further ready time examination

Further Ready Time Investigation: No Fixed ThresholdWhy?

ODBC Sniff traffic vSphere VM Probe VM vSphere DB

Why Isnt There A Fixed Ready Time Threshold?

Summary of Possible Reasons for High Ready Time

Where do we go from here?

Ballooning vs. Swapping (1)

VM1 F memctl 2. Reclaim 3. Redistribute Swap partition w/in Guest OS 1. Balloon

Ballooning vs. Swapping (2)

Swap Partition VSWP (w/in guest) (external to guest)

1. Force Swap 2. Reclaim 3. Redistribute

Ballooning vs. Swapping (3)

Ballooning is vastly preferably to swapping

Ok, So Why Do I Care About Memory Usage?

Important Memory Metrics (Per VM)

Metric Metric (SDK) (esxtop)

Important Memory Metrics (Per Host, sum of VMs)

Metric Metric (SDK) (esxtop)

No swapping Lots of swapping

A Stacked Chart (per VM) of Swapping

Where do we go from here?

Disk Performance Problems 101

Useful Disk Metrics

MBREADS/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd

disk.read.average disk.write.average disk.deviceLatency.average disk.kernelLatency.average disk.totalLatency.average

* Units different between esxtop and vSphere client

Disk Performance Example: vSphere Client

SAN cache enabled: High Write Throughput

SAN cache disabled: Poor throughput

Another Disk Example: Slow VM Power On

Lets look at the vSphere client

Using Esxtop to Examine Slow VM Power On

High Disk Latency: Mystery Solved

Host events: disk has connectivity issues

A More Complex Disk Example

Ethernet VM1 VM2 VM3 VM4

Combining Metrics: A More Complex Disk Example

A More Complicated Disk Example, Part I (CPU)

Predictable CPU usage, Host not saturated

Chaotic CPU usage, Host saturated

Complicated Disk Example, Part 2: Disk Usage

Predictable, balanced disk usage Uneven, reduced disk usage

Complicated Disk Example, Part 3: Write Rate