Best Practices: VMware Monitoring with HP Operations Manager

Written by: Alec King Senior Product Manager Veeam Software

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations

CONTENTS
INTRODUCTION ...............................................................................................................................2 INTENDED AUDIENCE .....................................................................................................................3 DISCLAIMER ...................................................................................................................................3 WHAT INFORMATION IS PROVIDED?....................................................................................3 QUESTIONS TO ASK ........................................................................................................................3 HOW IS THE DATA GATHERED? ...............................................................................................5 THE WRONG WAY...........................................................................................................................5 THE RIGHT WAY ..............................................................................................................................7 ADDITIONAL CONSIDERATIONS.....................................................................................................8 CERTIFIED VMWARE READY OPTIMIZED ..................................................................................8 HOW IS THE SOLUTION INTEGRATED INTO OPS MANAGER? ...................................8 INTEGRAL VS. CONNECTED ............................................................................................................9 ADDITIONAL CONSIDERATIONS.....................................................................................................9 HOW DOES THE SOLUTION SCALE?.................................................................................... 10 SCALABILITY FOR VIRTUAL ENVIRONMENTS .............................................................................. 10 CONCLUSION ................................................................................................................................. 11 ABOUT THE AUTHOR ................................................................................................................. 12 ABOUT VEEAM SOFTWARE..................................................................................................... 12

INTRODUCTION
When an organization chooses HP Operations Center as its enterprise management system (EMS), it makes substantial capital and operational investments in order to reduce downtime and keep the IT environment running smoothly. Increasingly, that environment includes VMware vSphere (or its predecessor, VMware Infrastructure). The main objective for deploying vSphere typically is to reduce costs through increased operational efficiency and capital utilization, and many organizations have realized these benefits. However, vSphere also adds complexity and, like anything in the environment, must be monitored and managed. Ideally, this takes place in the existing EMS. This kind of integrated approach allows operators to monitor physical and virtual systemsand the applications and services running therefrom the comfort of the HP Operations Manager (Ops Manager) console. They can quickly identify the source of a problem and immediately escalate it to the right team, thereby speeding problem resolution. Monitoring vSphere with Ops Manager also minimizes additional management infrastructure and operator training because it uses the framework thats already in place and that operators are already familiar with. It not only leverages existing investments in Ops Manager, it actually protects those investments by ensuring that Ops Manager is truly the single pane of glass it is intended to be.

Page | 2

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
Organizations have several choices for bringing vSphere to the Ops Manager console. In evaluating these choices, there are four important considerations: What information is provided? How is the data gathered? How is the solution integrated into Ops Manager? How does the solution scale?

This paper examines each of these in greater detail to provide readers with a simple but highly relevant set of criteria for choosing the best possible solution. With more and more mission-critical applications and services running on virtual machines, its an important decision with significant implications for IT and the business.

Intended audience
This document is intended for IT directors, data center managers, and HP Ops Manager administrators in organizations evaluating or using VMware vSphere in their production environment.

Disclaimer
Use this proven practice at your discretion. Veeam Software and the author do not guarantee any results from the use of this proven practice. It is provided on an as-is basis for demonstration purposes only.

WHAT INFORMATION IS PROVIDED?

HP Ops Manager is a sophisticated framework for monitoring the IT infrastructure, with advanced alerting, notification, data management, analytics, dashboard, and other capabilities. But it natively knows very little about the distinct IT elements it monitors. Instead, it relies on additional plug-in components to provide the knowhow to effectively monitor the various aspects of the IT infrastructure. How well Ops Manager monitors vSphere depends on the intelligence of the component thats gathering, analyzing, and delivering information about the virtual environment. Therefore, that component should encompass deep vSphere expertise, including what metrics are available, what metrics are most important, and what thresholds are most appropriate. It should serve as the vSphere expert in a box, capturing and putting into action best practices for vSphere monitoring and management.

Questions to ask
When evaluating options for bringing vSphere to HP Ops Manager, organizations should evaluate the vSphere credentials of the vendor and the depth of vSphere expertise in the solution.

Page | 3

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
Relevant questions include: Is the vendor a virtualization specialist? Or are they a generalist with limited vSphere expertise? Why is the vendor offering a vSphere solution? Is this their core business? Or are they just trying to check the box in terms of the platforms they cover? How long has the vendor been managing vSphere? What is their experience in monitoring large production environments? With vSphere deployments growing in size and importance, organizations cannot afford to be the learning lab or the proving ground for a product development team with little vSphere experience. Is the solution specific to vSphere? Or is it a generalized solution, addressing the lowest common denominator among the various hypervisors? What pre-configured monitoring policies does the solution include? Pre-configured policies help avoid overflowing the Ops Manager console with unimportant or informational events. While it should be possible to customize policies (ideally from the Ops Manager console itself), the solution should achieve at least an eighty percent fit right out-of-the-box. Does the solution monitor events as well as performance? Some solutions only capture performance data, ignoring the myriad of vSphere events that alert administrators to potential security, configuration, and licensing issues. Failed tasks, cluster configuration issues, and virtual machine deployment or migration failures are just a few examples of the events that must be monitored in order to have full visibility of whats happeningand what needs attentionin the virtual environment. What value-add does the solution provide? Does it filter, organize, and present information in the most useful way possible? Does it provide derived metrics in addition to whats available natively from vSphere? What supporting information does the solution provide? Does it simply present metrics and events, or does it explain their meaning, outline next steps, and suggest corrective action? Can it shorten the learning curve for operators who are new to virtualization? Does it elevate front-line support staff by providing readily accessible VMware knowledge?

Page | 4

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations

HOW IS THE DATA GATHERED?

At first glance, various methods exist for collecting data about the virtual infrastructure. However, further examination reveals the shortcomings of many of these methods.

The wrong way

Common technical approaches include the following: Ops Manager agent in the Virtual Machine Guest Operating System (OS): For OS and application health monitoring, Ops Manager uses the Ops Manager agent. The assumption is that through OS monitoring, the Ops Manager agent can determine the utilization of the underlying hardware. However, with virtualization, theres not a one-to-one relationship between the OS and the underlying hardware. So an Ops Manager agent running in the OS cannot determine the performance of the underlying hardware system. Only the hypervisor can provide a true picture of hardware utilization and the resources provided to the VM (virtual machine) and the guest OS. An Ops Manager agent inside the guest OS provides a skewed picture, since the Ops Manager agent treats the VM as a physical machine. In addition, vSphere-specific metrics from the hypervisor layerfor example, balloon memory and CPU ready and wait timesare not visible to the Ops Manager agent. Likewise, important components of vSphere ESX(i) hosts, clusters, and vCenterare not visible to the Ops Manager agent. And important vSphere features such as VMotion, perhaps controlled by DRS (Distributed Resource Scheduler) and/or HA (High Availability), are also invisible. Without visibility of vSphere as a whole, it is impossible to monitor host performance, physical hardware status, VM status and configuration, and other critical aspects of vSphere.

Page | 5

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
Ops Manager agent in the ESX Host Console Operating System (COS): Another method of collecting data from vSphere is to deploy a Linux monitoring agent in the ESX hosts Console Operating System (COS). However, the COS is not truly Linux, and the COS is not truly the ESX hypervisor. Thus, an agent monitoring the COS provides a limited view of hypervisor performance. An agent running in the COS also gives only partial visibility of many vSphere aspects such as clustering, DRS, HA, DPM (Distributed Power Management), and vCenter licensing, configuration and security. COS-based agents also have the potential to impact the hypervisor. Hypervisor instability has far-reaching consequences, affecting every VM on the hostand even other hosts and their VMs in the clusterand should be avoided. Finally, COS-based agents should be avoided because they have no longterm viability. VMwares stated direction is to deprecate the role of the COS, as they have already done in ESXi. Thus, any solution based on installing components in the COS has no future.

Legacy management protocols: This includes SNMP trapping, screenscraping the ESX COS via SSH (to capture the limited metrics the COS can request from the hypervisor), and Syslog monitoring. These approaches suffer from many of the same shortcomings as an agent running in the COS, including limited visibility of the complex vSphere environment and the potential to impact hypervisor stability.

For the above reasons VMware developed a secure remote management API (application programming interface) to communicate directly with the hypervisor and with vCenter to gather accurate and detailed data about the virtual environment.

Page | 6

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations

The right way

The VMware vSphere API (previously known as the VI API) allows for secure, efficient, and effective monitoring of the virtual environment and eliminates the need to deploy agents to hosts or virtual machines. An agentless approach is of special importance since deploying an HP Ops Manager agent in the ESX COS has been known to impact the stability and performance of the hypervisor. Even COS agents from hardware vendors (the only kind approved by VMware to run in the service console) must be re-certified for every VMware patch and upgrade, and add additional management overhead. Using the vSphere API eliminates the additional operational costs and the risks associated with COS-based agents. In addition, agent-based monitoring is impossible for ESXi, which doesnt have a COS in which to deploy an agent. The vSphere API allows for safe, efficient, and accurate monitoring of ESXi hosts in the same manner as ESX hosts. The vSphere API also allows for agentless hardware monitoring. VMware needed to provide such a method when they eliminated the COS in ESXi. vSphere implements the CIM-SMASH hardware model (see www.dmtf.org) to publish data about fans, power supplies, temperature sensors, and the like. Solutions that use the vSphere API have access to CIM-SMASH data for any version of ESXi and for ESX 3.5 Update 2 onwards. In an enterprise environment, scalability and overhead must also be considered. The vSphere API is the most efficient method for gathering data about the virtual environment. It uses highly effective data summarization and data packaging techniques, and minimizes overhead and consumption of network bandwidth. vCenter Server itself is built on the vSphere API. The vSphere API is exposed as a web service on vCenter servers and ESX(i) hosts, and can be accessed using the vSphere Web Services SDK (previously known as the VI SDK). Only the vSphere API presents a comprehensive picture of the health of all vSphere aspects and their dependencies. Thus, any viable monitoring solution for VMware must employ the vSphere API.

Page | 7

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations

Additional considerations
In addition to using the vSphere API to collect data, the monitoring solution should allow administrators to configure the monitoring connections to vSphere. This makes it possible to satisfy the broadest variety of requirements and scenarios now and in the future. It should be possible to: Connect to multiple vCenter servers. Collect data only for certain hosts or virtual machines in a given vCenter. Connect directly to ESX(i) hosts. This eliminates vCenter as a potential single point of failure for the gathering of monitoring data, and also allows monitoring of hosts that arent managed by vCenter (as may be the case in remote branch offices).

Certified VMware Ready Optimized

With the success of VMware, the number of vendors participating in the VMware market has gone from tens to thousands in a few short years, and it is in vogue for software companies to say they support the VMware Infrastructure. But which vendors are actually qualified to make such an assertion? Which have played by the rules set forth by VMware? Which are executing a strategy in line with VMwares? VMware created the VMware Ready Optimized program to address these questions. It is the highest level of VMware certification for software products. Products bearing the VMware Ready logo provide optimized levels of integration, functionality, and performance that strictly adhere to VMwares architectural, supportability, and future-proofing requirements, including use of the vSphere API. To ensure the stability of the production environment, the reliability of the information and alerts provided, and the long-term viability of the solution itself, any monitoring solution for VMware should be certified VMware Ready Optimized.

HOW IS THE SOLUTION INTEGRATED INTO OPS MANAGER?

As previously discussed, monitoring VMware with HP Ops Manager provides a number of compelling benefits: Leverages the management infrastructure thats already in place Reduces the amount of training operators need in order to provide 24x7 front-line monitoring for VMware Shortens problem resolution time by providing operators with full visibility of the entire IT infrastructure Protects the integrity of the Ops Manager single pane of glass

These benefits are most fully realized when VMware monitoring is implemented natively in the core Ops Manager architecture. And HP facilitates such an approach.

Page | 8

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
Wanting Ops Manager to be a true enterprise monitor but knowing it could not itself address everything that might encompass, HP encouraged third parties to develop specialized domain-specific monitoring and provided a mechanism to deliver that in Ops Manager using Smart Plug-ins (SPIs). Thus, the use of third-party SPIs has become an integral part of Ops Manager deployments.

Integral vs. Connected

The alternativea separate monitoring platform specifically for VMware, with its own console, database, and alerting and notification systemis costly and disruptive. Introducing another monitoring framework results in console sprawl and requires further investments in staff training and redesigned management processes. Even if the additional monitor includes a connector to Ops Manager, it often provides only basic information or alerts. Only a truly integrated solution allows all the features of Ops Manager to be employed and all the benefits of monitoring VMware with Ops Manager to be realized.

Additional considerations
In addition to the fundamental question of SPI versus connected monitor, the degree of integration with HP Ops Manager should be evaluated. The best solution: Enables all standard EMS functions including performance, event, and state monitoring; reporting and auditing; notifications; and so on. It enables all the management functionality that operations staff expect to see and use when something is monitored in Ops Manager. Integrates directly with the Ops Manager Service Map. It builds a detailed topology of the virtual environment that shows component connections and dependencies, allows for targeted monitoring, and assists with root-cause analysis. Includes pre-defined remedial actions such as tools and tasks to allow direct interaction with virtual machines and hosts (if operators have the necessary permissions, of course).

Page | 9

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
Can be configured from the Ops Manager console to lower management costs and allow administrators to work in the environment they are familiar with. Integrates with HP Operations Center reporting. This includes HP Reporter, HP Performance Manager, and HP Performance Insight. These reporting components are part of the standard Operations Center toolkit for performance and configuration analysis, historical reporting, and capacity planning. The solution should enable this reporting functionality to realize all the benefits of the HP platform.

A solution that offers these capabilitiesand includes built-in VMware expertise, uses the vSphere API, and is configurable and scalable will deliver solid value now and in the future.

HOW DOES THE SOLUTION SCALE?

Scalability may not be the most exciting aspect of a software evaluation, but it is key to the success of the solution ultimately selected and deployed to production. Even the best functionality is of little value if the solution fundamentally cant support the environment. And no one wants to invest time and effort to deploy a solution they quickly outgrowthat is simply too expensive and too disruptive for todays cost-conscious, service-oriented IT department. Scalability may not be the most exciting aspect of software development, either. Its a thankless task, garnering attention only when theres a problem, with little acknowledgementfrom users or, in the case of a commercial software product, from the marketof a job well done. Its also a difficult task, requiring deep knowledge of system internals, solid architecture design skills, and patience to develop, execute, and analyze repeated load tests. And the cost of failure is high: theres typically no quick fix when something goes wrong, and a significant system redesignwhich is disruptive to users, administrators, and project schedulesis often the only solution. Scalability is especially a concern with new software. What works well on paper doesnt necessarily work well in practice. It takes experience, production run-time, and typically several product releases to get it right.

Scalability for Virtual Environments

The virtual infrastructure is complex, with many moving parts and dependencies. So the amount of information about the virtual infrastructure is immense. Monitoring will suffer in the near-term if the solution cant handle all the data. And a solution that works today might not work tomorrow if the virtual environment continues to grow. For VMware monitoring with HP Ops Manager, scalability must be addressed on two fronts: Collecting the data from vSphere Feeding the data to Ops Manager

Page | 10

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
The vSphere API is the most efficient method to gather the data, and use of the vSphere API helps to ensure the scalability of data collection. In terms of delivering the data to Ops Manager, experience shows that the Ops Manager API or other data injection methods arent the best approach. The Ops Manager API is not designed to handle the volume of data coming from the virtual environment, which may have hundreds or even thousands of objects, with many different metrics and events. And if Ops Manager gets bogged down, all monitoring is affectednot just monitoring of the virtual environment. A better approach pairs the collector with an Ops Manager agent that feeds the data to Ops Manager. Multiple collector/agent pairs can be deployed as necessary to handle the data load. This kind of distributed architecture provides virtually unlimited scalability and leverages Ops Managers Management Server back-end, which is designed to support multiple agents. Of course, once you have multiple collectors, you need to a way to manage them. Although the collectors ultimately feed data to a single central point (Ops Manager), the collectors themselves must be configured and managed. To do this one collector at a time would add significant administrative overhead and opportunity for errors. In addition, it should be possible to coordinate the work of multiple collectors. For example, it should be possible to balance the load among collectors, and have the collectors serve as backups for each other (high availability monitoring). The result is a centrally managed, distributed architecture that can easily grow as the monitored environment grows. While scalability may not be the most exciting aspect of a software evaluation, it is absolutely critical to success. Often, the most scalable solution is the most mature one: its gone through several iterations, it includes sophisticated configuration options, and its been proven in real production environments. Product maturity and vendor experience not only affect the depth of vSphere expertise in the solution (as previously discussed), but they also affect scalability and are important considerations for product selection.

CONCLUSION
For organizations that have standardized on HP Operations Center, the benefits of monitoring VMware with HP Ops Manager are clear. What may not be so clear is how to bring VMware to the Ops Manager console. This white paper suggests four criteria for evaluating the alternatives. In doing so, the white paper addresses three rather distinct audiences with three rather distinct sets of needs: The Ops Manager team, which is responsible for the integrity of the EMS and the delivery of monitoring data for all business-critical infrastructure The virtualization team, which is responsible for the performance and availability of the virtual environment The business as a whole, which is responsible for costs and for service to customers, employees, and other users

Page | 11

5 Ways VMware vSphere Improves Backup and Recovery Manager Best Practices: VMware Monitoring with HP Operations
Veeam nworks Smart Plug-in for VMware is one alternative worthy of consideration. Developed in close collaboration with VMware and certified VMware Ready Optimized, the nworks SPI is proven in more than 300 real customer installations and more than 5 years of production use. It fully embraces the vSphere API from VMware as well as the Ops Manager framework from HP, to provide best-in-class monitoring for VMware. And because its from Veeam, a leading virtualization specialist, it encompasses deep knowledge and extensive experience with VMware to provide the most effective monitoring of your virtual environment and the most comprehensive support for your operations staff. For more information about the nworks SPI, visit www.veeam.com/spi.

ABOUT THE AUTHOR

Alec King is a Senior Product Manager at Veeam Software. He has many years experience in enterprise systems management with a variety of companies, including Siemens and the British Broadcasting Corporation. He can be reached at alec.king@veeam.com.

ABOUT VEEAM SOFTWARE

Veeam Software, a premier-level VMware Technology Alliance Partner and member of the VMware Ready Management program, provides innovative software for managing VMware vSphere 4 and Virtual Infrastructure 3. Veeam offers an awardwinning suite of tools to assist the VMware administrator, including: Veeam Backup & Replication: #1 for Virtualization Data Protection Veeam Reporter Enterprise: for VMware performance, storage, and capacity reporting and chargeback Veeam Monitor: for vSphere performance monitoring and alerting across multiple vCenter Servers Veeam Configurator: for complete host configuration management Veeam Business View: for business-aligned management of virtual machines, independent of the virtual infrastructure

With its acquisition of nworks in June 2008, Veeam's products also include the nworks Smart Plug-in and the nworks Management Pack, which incorporate VMware data into enterprise management consoles from HP and Microsoft. Learn more about Veeam Software by visiting www.veeam.com.

Page | 12