Achieving Visible Ops with Intelligent System Automation

Achieving Visible Ops

Introduction
Modern IT is in crisis. Compounding growth in system scale, accelerated change, and the expectation for zero-latency responsiveness create the perfect storm particularly with the added backdrop of contracting budgets and lean, do-morewith-less mandates. This calls for improved IT processes and reliable automation, where the goal is massively leveraged human capital and fast, predicable IT operations. Most IT organizations have begun the journey to process improvement and IT automation, and many have achieved meaningful resultsbut processes remain flawed, and the inherent limitations in automation technology have prevented a real transformation. Many approaches to automation cause the wrong things to happenfaster. Process frameworks like ITIL and CobiT provide best practices for defining IT processes, but they lack prescriptive guidance for how to implement them. Manifestos such as The Visible Ops Handbook go a step further to offer practical guidance and a prescriptive roadmap for implementing release management and change processes, but stop short of discussing automation techniques. This whitepaper focuses on the last lap of the process improvement journey intelligently automating release and change processes to transform and accelerate IT, reduce cost and mitigate risk. More specifically, this whitepaper focuses on intelligently automating Visible Ops. It briefly summarizes the Visible Ops methodology, and explains how rPath can automate key elements of the Visible Ops approach.

The Visible Ops Handbook answers a fundamental questionhow do I implement

ITIL and where do I start? rPath answers an equally important and fundamental questionhow do I implement ITIL principles without the bloat and bureaucracy often associated with this approach? How do I automate for speed and control? For many, ITIL brings to mind slow and ponderous review boards and layers upon layers of process and approval. That doesnt need to be the case.

Achieving Visible Ops

The IT process maturity model:

A w a r e ITIL-aware, beginning to define repeatable and verifiable IT processes O p t i m i z e d Leveraging Visible Ops to implement and optimize ITIL release, control and resolution processes A u t o m a t e d Intelligently automating key elements of Visible Ops for fast, predictable and scalable IT operations
F i g u r e 1 : The IT process maturity model.

Introducing Visible Ops

The IT Infrastructure Library (ITIL) is a framework for implementing processes and best practices into IT operations; Visible Ops is a methodology for adopting ITIL. The Visible Ops team met with hundreds of IT organizations and identified eight high-performing IT operations groups with the highest service levels, best security and best efficiencies. These organizations shared a culture of change management, a culture of causality and a culture that valued effective and auditable controls, promoting fact-based management. Published by the IT Process Institute, The Visible Ops Handbook summarizes how these organizations operate and provides step-by-step guidance and a roadmap for implementing ITIL practices. It describes a control-based on-ramp to ITIL that others can leverage to springboard their own process improvement efforts. The findings are distilled into four practical and auditable steps: 1. S t a b i l i z e P a t i e n t , M o d i f y F i r s t R e s p o n s e Since almost 80% of outages are self-inflicted, the first step is to control risky changes and reduce mean time to repair (MTTR) by addressing how changes are managed and how problems are resolved. 2. C a t c h a n d R e l e a s e , F i n d F r a g i l e A r t i f a c t s Infrastructure is highly diverse and cannot be repeatedly replicated. This step focuses on creating an inventory of assets, configurations and services, to identify those with the lowest change success rates, highest MTTR and highest business downtime costs.

Achieving Visible Ops

3. E s t a b l i s h R e p e a t a b l e B u i l d L i b r a r y The highest return on investment is implementing effective release management processes. This step creates repeatable builds for the most critical assets and services, to make it cheaper to rebuild than to repair. 4. E n a b l e C o n t i n u o u s I m p r o v e m e n t The previous steps have progressively built a closed-loop between the Release, Control and Resolution processes. This step implements metrics to allow continuous improvement of all of these process areas, to best ensure that business objectives are met. For more information on Visible Ops, visit: http://www.itpi.org/home/visibleops.php

Intelligent System AutomationMaking Visible Ops a Reality

Intelligent system automation can be used to codify these principles as automated steps in your release, change and resolution processes. Today, provisioning and updating software systems is more challenging than ever. Software is increasingly diverse, change is accelerating, and the volume of systems is skyrocketing. This means that provisioning a new system takes weeks or longer, change is avoided because updates and patches cause outages, and troubleshooting systems and reversing changes is manual and time-consuming. Visible Ops is a way to start taking the time, cost and risk out of system provisioning and change, but the linchpin is automationits the only way to make Visible Ops a reality in the age of massive scale and low-latency IT, where adding resources is no longer an option. Recognized as a leader and innovator in next-generation system automation, rPath automates the packaging, provisioning and maintenance of software systems across physical, virtual and cloud-based environments. This helps IT organizations to: Accelerate deployment cycles from months or weeks to minutes Reduce costs and improve system-to-administrator ratios by 6-10X Eliminate the pain of change with conflict-free and fully reversible updates

Achieving Visible Ops

There are two key aspects of rPath that make it unique. Deep System Modeling The system model is about creating a blueprint for how systems should look and using that as the basis for constructing and maintaining systems over time. rPath automatically analyzes and deeply models entire software stacksfrom the application through the OSand every layer in between. The result is a deeply modeled system inventory, which describes the desired state of every file, binary, application component, and software stack on every production system, including information about policies that must be adhered to, the entire dependency chainincluding OS components, middleware and librariesand the impact of change. These models are stored as version-controlled system manifeststhey provide deep system transparency, and serve as a basis for conflict-free deployments, updates and managing the complete lifecycle of deployed systems.

F i g u r e 2 : The version-controlled system model provides deep system transparency and serves as the basis for managing change.

Achieving Visible Ops

System Version Control System version control is about describing systems over timeeasily roll the system forward, back or reproduce itexactlyacross release lifecycle stages. rPath is not a source code management systemits an operational management platform that applies the principles and disciplines of source code control to the management of deployable software systemssystem manifests, packages, binaries, policies and configurations. System version control is the technological foundation that allows a complex system to be defined by a single version number, ensuring systems can be quickly reproduced, patched and updated, rolled back and reported on.

F i g u r e 3 : rPath provides a platform for modeling and versioning entire software stacks, including policies, dependencies, configurations, etc.

rPath provides the perfect foundation for automating Visible Ops. What follows is a brief synopsis of each of the four Visible Ops steps, and a description of how rPath can enable and automate those steps.

Achieving Visible Ops

Phase One: Stabilize Patient, Modify First Response
For enterprise IT, change is the only constant. And today, change is more disruptive than ever. There are more sources of change, more systems to maintain, and faster rates of change. According to industry analysts, the average IT enterprise suffers multiple major system outages per year. 80% of the time, the outage is selfinflicted 1a direct result of change gone awry. System outages are crippling and expensive. They lead to lost business, missed service-level agreement (SLA) and compliance risks, as well as excessive labor costs due to all-nighters and expensive overtime. When systems are down, 80% of the effort is spent detecting what changed, and only 20% of the recovery time is spent actually repairing the infrastructure. Blame is readily assigned, fingers are freely pointed, and tension escalates, because no one takes accountability for changes. When changes are finally detected, the source of the change and the reason for the change are not always known. When system failures happen during critical, peak and high-demand periods, the consequences can be catastrophic. Many retailers have suffered major outages just days before the year-end holiday, or during a peak month. One notable example of the cost of IT change is Canada Revenue Agencythey applied a certified patch that resulted in an eleven-day outage. All employees were sent home, it took ten days to manually troubleshoot and restore the system, and the associated cost was $2.4 million! Sometimes changes wreak havoc later, long after the change was made. And sometimes, a change can undo a previous change or even a whole series of changes. For many IT organizations, only 70% of the changes made work the first time without generating a firefighting episode. All of this points to one overarching symptom: Todays IT organizations are grappling with an inability to deal with change at scale. rPath provides a next-generation approach to dealing with IT change. Based on a version-controlled system model as the basis for managing change, rPath allows IT to consume more rapid and frequent change in a way that is scalable and nondisruptive.

Source: Stephen Elliot, Senior Analyst Network and Service Management, IDC, 2004.

Achieving Visible Ops

The goal of the first Visible Ops phase is two-fold: 1) reduce the number of outages by freezing change outside of scheduled maintenance windows, and 2) modify the first response process by ensuring that problem managers have all change information at hand about what could have caused the outage. rPath can enable and automate the specific steps outlined in Phase One: E l i m i n a t e u n a u t h o r i z e d c h a n g e The first phase is devoted to building and reinforcing a functional change management and authorization process. This calls for change automation. rPath automates and controls change, helping to identify, isolate, and ultimately eliminate unauthorized change. E s t a b l i s h a c u l t u r e o f c a u s a l i t y When service outages occur, Visible Ops recommends examining all approved and detected changes first before making a diagnosis. In addition to tracking changes to software artifacts managed in the rPath repository, rPath also tracks all change on a systemby-system basis. This allows for a complete change timeline for any individual system, making it very easy to identify recent changes that could have caused the outage. E n s u r e b a c k o u t p l a n s f o r c h a n g e s System version control makes it very easy to roll back changes. Reversing a change is more than simply reverting to the previous version of the system. rPath maintains a record of all overwritten files, so when a change is reversed, previous files are restored, even if those files were not modeled in rPath (e.g., locally-edited configuration files). S t r i v e f o r c h a n g e s u c c e s s r a t e s h i g h e r t h a n 9 8 % A versioncontrolled system model makes change predictable and scalable. All changes are pre-validatedrPath analyzes imported software artifacts and understands the complete dependency chain. This ensures guaranteed, conflict-free change, and enables 100% change success rates. C r e a t e a C h a n g e A d v i s o r y B o a r d ( C A B ) rPath can help answer questions before and during the CAB meeting, including What questions, such as, what are the expected results of the change? and, What If questions, such as, what is the rollback plan should the change fail? Integrations can auto-pull information from rPath into CAB tickets.

Achieving Visible Ops

R e s o l v e l a t e n t p r o b l e m s rPaths complete and versioned change history is ideal for troubleshooting latent problems. System version control can precisely reproduce any previous version of an entire system to help isolate the exact change that caused the problem. P r e v e n t p r e v i o u s c h a n g e s f r o m b e i n g u n d o n e A change can often cause problems by inadvertently erasing previous changes. rPaths modeldriven change management prevents this. All changes are pre-validated, and the impact of change is deeply understood.

Phase Two: Catch & Release and Find Fragile Artifacts

Uncontrolled changes on systems cause them to deviate from known and trusted states. Instead of parity, variance creeps in. A thousand servers may be identical at deployment, but when their configurations drift over time, they become like snowflakeseach unique and impossible to reproduce. The goal of the second Visible Ops phase is to identify and analyze the critical elements of production infrastructure and the interdependencies between components that make up that infrastructure. This insight will be used to reduce configuration variance. rPath automates this process by creating a system model. Software artifacts are imported into the rPath repository and deeply modeled. Dependencies are automatically discovered and resolved. rPath is essentially a modeled system inventory that describes all of the operational software assets within a running stack. This includes files, binaries, OS, middleware and application components, configurations, policies and the entire dependency chain. All of this content is managed under strong version control. rPath can enable and automate the specific steps outlined in Phase Two: I n v e n t o r y a l l m a n a g e d a s s e t s rPath automates the inventory construction process by examining existing software and constructing initial system models. This is the first step in moving an existing environment along the path to standardization. Minimize configuration variance and normalize the i n f r a s t r u c t u r e rPaths whole-system differencing helps to normalize unique, snowflake systems. Sometimes the differences between systems are arbitrary and incorrectin this case, rPath can be used to

Achieving Visible Ops

automatically correct them. Sometimes the differences are correct, but theyre not modeled and therefore not reproduciblein this case, rPath can be used to explicitly describe the desired differences between a base system model and particular system instances.

Phase Three: Establish Repeatable Build Library

Modern software systems are complex, interdependent and dynamic, comprising tens of thousands of files, thousands of configuration options, and countless file versions. These systems are often hand-assembled in a manual, haphazard manner, without a system model or blueprint. This lack of transparency makes it very difficult to ensure that all system artifacts are consistently synchronized, and that all stakeholders across the release lifecycle are drawing from the same definitive source, and working with the most current software. A common scenario may look something like this: Developers build an application on an IT-issued platform. They hand it off to QA, who recreates the system based on a different version of the platformthey resolve dependencies, and tweak and tune the system until it works. They then pass the certified system to the production teams who recreate the system, again, based on the latest version of the platform. During each of these handoffs, software versions and system configurations drift. Dependencies are missed, found, resolved, poorly documented and, consequently, missed again. Deployments fail, and the system goes back to development and test for hot fixes and recertification. Once the system is finally in production again, the software usedsources, versions and dependenciesremains unclear. Updating and patching the system becomes an ordeal. This problem is so acute that it has inspired a movement called DevOps. Its mission is to bridge the gap between application development and IT operations, and to create an environment in which development and IT are collaborative, agile and high functioning. The goal of Phase Three of Visible Ops is to create and maintain a versioned definitive software library (DSL) for all software and patches, and to ensure perfect configuration synchronization between pre-production and production systems.

Achieving Visible Ops

rPath combines deep system modeling and version control as a basis for managing a definitive software librarya canonical source of record for operational software assets. This enables a very differentand improvedscenario: Development builds a new application using the latest platform. They check in a complete, dependency resolved systemapplication, OS and middleware componentswhich is identified by its own version number. QA is alerted and uses the same version to run their test suite. Production then takes the certified version and successfully provisions the system. Development, test and production are all accessing the same exact version, eliminating drift and confusion. And once a system is deployed, a version manifest becomes the basis for seamless change. Updates can be quickly matched to system inventories, change impact is clearly document, and patches and updates can be implemented incrementally. rPath can enable and automate the specific steps outlined in Phase Three: R e b u i l d i n f r a s t r u c t u r e r a t h e r t h a n r e p a i r Repeated break/fix cycles tend to add variance, whereas rebuilding infrastructure minimizes configuration drift. rPath uses the same version-controlled system model to drive provisioning, image creation, and incremental updates. This allows for flexibility in deciding whether to build or updatein the case of build, the result is immediately up-to-date with every change across the stack. Traditionally provisioned and imaged systems are instantly out-ofdate because change is constant. And since rPath rebuilds a system from its constituent components, a build can be retargeted to a different physical, virtual, or cloud environment without re-developing the system. C r e a t e a n d m a i n t a i n t h e d e f i n i t i v e s o f t w a r e l i b r a r y ( D S L ) rPath provides a solid foundation for managing a versioned software library for IT operations. rPath extends the DSL concept by modeling and versioning entire stacks, including OS components and configuration files. M o v e f r o m p r o d u c t i o n a c c e p t a n c e t o d e p l o y m e n t rPath enables rapid, on-demand, conflict-free provisioning of complete, dependencyresolved systems for physical, virtual and cloud targets. Deployment is fast and conflict-free.

Achieving Visible Ops

Store non-repeatable snowflake builds in the DSL and slowly m i g r a t e Standardization is a gradual process of adoption, and both Visible Ops and rPath accommodate this reality. rPath supports the process by automatically converting existing systems into initial system models. The systems can then be standardized incrementally in successive versions. C l o s e t h e l o o p b e t w e e n p r o d u c t i o n a n d p r e - p r o d u c t i o n The rPath approach ensures a controlled software release lifecycle with clean separation of duties and consistent, access-controlled handoffs between lifecycle phases. This includes explicit functionality for promoting content from one stage to another, and enforcing access or lack of access to different user populations at different stages.

F i g u r e 4 : rPath enables consistent and controlled handoff between lifecycle stages, including explicit functionality for promotion and access-control.

Achieving Visible Ops

Phase Four: Enable Continuous Improvement
The previous Visible Ops steps focused on building a closed loop between the release, control and resolution processes. This step implements metrics to allow continuous improvement of all of these process areas. ReleaseHow efficiently and effectively do we generate and provision infrastructure? ControlsHow effectively do we make good change decisions that keep production infrastructure available, predictable and secure? ResolutionWhen things go wrong, how effectively do we diagnose and resolve issues? rPaths unique approach to system modeling creates the level of transparency and insight needed to answer these fundamental questions, to enable ongoing process improvement, and to satisfy audit and reporting requirements. rPath can inform and automate the specific metrics and improvement points outlined in Phase Four: R e l e a s e m e t r i c s rPath provides deep system transparency and control, and can inform key release metrics, including time to provision known good builds, number of turns to a known good build, shelf life of builds, percent of systems that match known good builds, percent of builds that have a security sign-off, and number of fast-tracked builds. C o n t r o l m e t r i c s rPath can also inform key control metrics such as number of changes authorized per week, number of actual changes made per week, number of unauthorized changes, change success rate, number of emergency changes, number of special changes, and number of business as usual changes. R e l e a s e i m p r o v e m e n t p o i n t s rPath can automate suggested release process improvements, including tracking all configurations in use across the release lifecycle, segregating development/test/production systems, creating a library of automated build systems, and confirming that all deployed system images are under version management. R e s o l u t i o n i m p r o v e m e n t p o i n t s rPaths complete visibility into change enables ongoing improvement of diagnosis processes. rPath also

Achieving Visible Ops

helps define bulletproof back-out processes to recover from failed or unauthorized changes. A v o i d a u t o m a t i n g b r o k e n p r o c e s s e s Visible Ops cautions against automating a flawed process. This only makes bad things happen faster. Rather than simply using custom scripts to automate manual tasks, rPath uses intelligent automation to make release management and change processes predictable and scalable.

Preparing for Audits with rPath

Visible Ops provides a number of helpful tips when preparing for an audit. rPath provides an automated model for creating and deploying software systems that are transparent, easily controlled and easily audited. rPaths reporting capabilities help answer key questions about the state of deployed systems, helping to take the time, cost and risk out of audit preparations. For example, rPath can help document and report on systems and processes used to detect changes. rPath can also help with documenting and proving that an accurate system inventory is being maintained. Finally, rPath can help generate reports on change success rates, deployed systems and whether or not they match golden builds, and integration with change management processes. But compliance is more than something rPath checks after the factcompliance is inherent in the rPath model. The rPath approach to compliance is simple, powerful and uniquea system is compliant if it matches its system model. The model is powerful enough to capture all automatable aspects of audit compliance. This is a much deeper compliance model than usual. It puts an end to the endless drift/audit/repair cycle, since compliance rules are applied for every incremental change. And because no change is allowed to get the system out of compliance, theres the assurance that an audit will pass on the first try.

Achieving Visible Ops

Conclusion
Growth in system scale, accelerated change, and the expectation for zero-latency IT are forcing a transformation in enterprise IT. This creates the need for improved release management and change processes, and next-generation automation technologies. The ITIL release management process is fundamentally about controlling and monitoring the flow of change into an IT infrastructure. The Visible Ops Handbook offers practical guidance and a prescriptive roadmap for implementing the release management, control and resolution processes. But in todays cost-conscious and constrained IT climate, the only way to make Visible Ops a reality is with reliable, intelligent automation. rPath delivers an automated, next-generation model for creating, deploying and changing software systems. The result is dramatic improvements in responsiveness and a way to deal with exploding scale without adding cost. This makes ITIL and Visible Ops achievablewhile remaining responsive, agile and lean.

About rPath
rPath automates system provisioning and maintenance across physical, virtual and cloud environments. rPaths innovative release automation platform is based on the industrys only commercial version control repository for managing deployed software systems. The result is an easy-to-deploy and cost-effective automation solution for rapid, low-risk and low-overhead deployment and maintenance of complex software systems. rPath dramatically improves responsiveness to business lines, reduces compliance risks, and allows resource-constrained IT organizations to significantly reduce operating costs and do more with less. Headquartered in Raleigh, NC, rPath has 80+ customers including some of the worlds largest enterprises and ISVs. Visit www.rpath.com.

Corporate Headquarters: 701 Corporate Center Drive, Suite 450 Raleigh, NC 27607

+1 919.851.3984 Main +1 866.508.6200 Sales +1 919.851.3985 Fax

info@rpath.com www.rpath.com

Copyright 2010 rPath, Inc. All rights reserved. rPath, rBuilder, rPath Lifecycle Management Platform and the rPath logo are registered trademarks or trademarks of rPath, Inc. All other brands and product names are trademarks or registered trademarks of their respective owners. Information supplied by rPath is believed to be accurate and reliable. rPath assumes no responsibility for any errors in this document. rPath reserves the right, without notice, to makes changes in product design or specifications.