Agile Versus CMMI - Process Template Selection and Integration with Microsoft Team Foundation Server

Robert Leithiser
Auburn University 3101 Shelby Center Auburn, AL 36849-5347 1-(334) 844-4330

Drew Hamilton
Auburn University 3101D Shelby Center Auburn, AL 36849-5347 1-(334) 844-6360

leithrm@auburn.edu ABSTRACT
Increasingly many software development organizations must pursue at least a level 3 CMMI certification to compete in the marketplace. Not all software organizations are under this constraint, but the CMMI model offers benefit beyond just a marketing advantage. Agile methods have taken hold in many of the same organizations that stand to gain benefit from CMMI. CMMI adds some stress to the Agile approach by nature of more detailed planning and tracking. Microsoft and other software development organizations have endeavored to automate the more tedious aspects of CMMI, thus making it easier to support agile approaches within more rigid frameworks. The Microsoft Solutions Framework (MSF) initially founded on Microsoft software best practices observed internally and externally has expanded to integrate automated software configuration for team projects. The Team Foundation Server (TFS) product leverages MSF and supports both CMMI and Agile as process templates and provides tools around these templates to automate the use of either approach. In this paper, we explore how Microsoft has integrated these approaches into TFS, the rationale and implications for selecting one approach over another and how agile practices can be integrated into a CMMI framework.

Hamilton@auburn.edu 1. INTRODUCTION
Software development firms face new challenges competing in the global market place. On the one hand, there is pressure to achieve quick return on investment (ROI) through rapidly deployed software solutions. On the other hand, there is pressure to conform to increasingly complex functional requirements while retaining acceptable quality. Increasingly, Capability Maturity Model Integration (CMMI) is becoming an important model for organizations to utilize in order to compete in the market place. Not all firms need to achieve a CMMI compliance level to compete effectively. Agile development methods have proven especially suited for situations where a rapid solution is needed while CMMI is typically utilized for longer/more complex projects that require more rigors and can afford the extra overhead of CMMI. The CMMI/Agile tradeoff begs several questions At what point is a project too big or too critical for Agile or too small for CMMI? What are the key deciding points, particularly as it pertains to the decision facing Information Technology (IT) professionals using the Microsoft Team Foundation Server Product (TFS), which requires the selection of one of these templates in order to start a new project? Does the selection of one approach preclude the use of the other? An important aspect in studying each approach lies with understanding that much of the value of the processes depends on the effectiveness of tools to automate more cumbersome aspects. The Microsoft Solutions Framework (MSF) prescribes templates to help with both approaches. TFS automates the use of these templates in the context of the development process. The purpose of this paper is not to extol Microsofts toolset but to review MSF and TFS in sufficient depth to understand how it assists in both processes. From the TFS view, Agile and CMMI are defined as process templates. For our discussion, we consider CMMI and Agile selection scenarios, implications for selecting one process template over another and integration approaches.

Categories and Subject Descriptors

D.2.9 [Management]: Software configuration management, Software process models.

General Terms
Management, Measurement, Documentation,

Keywords
Team development, Microsoft Solutions Framework, Agile, Capability Maturity Model Integration, Agile, CMMI, Process Templates, Development Methodologies.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. ACM-SE 08, March 2829, 2008, Auburn, AL, USA. Copyright 2008 ACM ISBN 978-1-60558-105-7/08/03$5.00.

2. LITERATURE REVIEW 2.1 Agile

Agile represents a paradigm shift away from formal processdriven software development [11]. One of the formative drivers that compelled the manifesto founders were studies that demonstrated high failure rates associated with large disciplined software development efforts. It emphasizes rapid return on investment and builds trust in the development process. The Agile Manifesto web site [3] outlines the following four

186

principles that highlight its stand against traditional software development discipline: 1. 2. 3. 4. Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan

essential elements of effective processes. It can be used to guide process improvement across a project, a division, or an entire organization. CMMI helps integrate traditionally separate organizational functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes [8]. CMMI evokes fear in some because of the processes involved in certification. However, as is also the case with Agile, CMMI at its core is not a methodology but rather a set of principles. In the case of CMMI, the set of principles focuses on maturation of a software development process. This is a different emphasis than Agile. CMMI is concerned with defining metrics and practices to ensure continuous improvement [9]. The goal of CMMI is not just to support a minimum set of standards to achieve to a particular level, but to enable increasing improvement in an organizations processes. CMMIs approach is based on maturity models. It supports both a staged approach and a continuous model for improvement. It provides several key process areas (KPA) at different levels. The KPAs are intended to help gauge where an organization is at in the maturation process as well as provide guidance for how to achieve the desired level. CMMI associates skill in different process areas with higher maturity levels. The below summarize the current maturity levels associated with CMMI: Level 0: No Process Level 1: Initial ad-hoc processes frequently exceeding budget and schedule and unable to repeat past successes. Level 2: Repeatable some project management to track schedule/cost with some repeatability but with high risk for exceeding cost and time estimates. This level includes software configuration management, software quality assurance, and subcontract management. Level 3: Defined - Standards, process description, and procedures are in place including peer reviews. This level includes integrated software management, software product engineering, inter-group coordination, training programs, and an organizational process focus and definition. Level 4: Managed - Quantitative process management and software quality management are in place. Level 5: Optimizing - Defect prevention, technology change management, and process change management are in place. CMMI is more focused on metrics than Agile. To be CMMI compliant at any level requires rigorous inspection. This involves the use of the Standard CMMI Appraisal Method for Process Improvement (SCAMPI) and often involves over 150 hours for typical organizations. Achieving CMMI compliance is greatly facilitated when there are tools in place to assist in the processes. The use of tools in and of itself is one of the requirements for achieving higher levels of CMMI. Because of CMMIs emphasis on disciplined planning and organizational structure and because of the sheer size and scope of CMMI, many misunderstand CMMI as a traditional waterfall software approach that requires large hierarchical bureaucratic organizational structure. However, CMMI does not prescribe specific software development approaches or specific organizational structures. CMMI is compatible with a variety of

While the manifesto recognizes value in the non-bold items on the right, it recognizes the bolded values on the left greater. Agile stands in contrast to CMMI and other more disciplined software processes in that it rejects many traditional viewpoints regarding detailed planning and requirements definition. This approach is motivated by the reality that requirements change and customers do not always know what they need until they see it being produced. Agile recognizes the reality that requirements for software solutions, particularly those that involve custom-developed solutions are not easy to envision. Agile insists that software development processes must be flexible or else the processes will break resulting in dissatisfied customers. Agile more so represents a set of principles for software development than a specific methodology. The hallmarks of an Agile approach include incremental development consisting of small software releases with rapid cycles, collaborative arrangements with customers and developers working closely together, straightforward, easy to learn methods, and adaptive able to make last moment changes [1]. Agile principles are evident in many different methodologies, including Extreme Programming (XP) [6], Scrum [20], Feature Driven Development [19], and Adaptive Software Development [14]. These methodologies differ in technique, but encapsulate many common recommendations that contribute to higher software quality [21]. These include Iterative software development, continuous verification, customer-oriented requirements, architecture driven design and focus on teams Agile critics lament the lack of project management organization and point out how the process can break down, particularly on larger projects and during the maintenance phases. Abrahamsson [2] identifies shortcomings from the following perspectives: Software development life cycle: Life cycle coverage needs to be explained and interfaces with phases not covered need to be clarified. Project management: Harmonization is needed to support project management activities. Abstract versus concrete guidance: Emphasis is needed to provide guidance to utilize suggestions rather than abstract principles only. Universal versus situational solutions: More work on how to adopt agile methods in different development situations is needed. Empirical support: More empirical, situation-specific experimental work is needed; results need to be published.

2.2 CMMI
CMMI is summarized on the Carnegie Mellon website as follows: Capability Maturity Model Integration (CMMI) is a process improvement approach that provides organizations with the

187

software development approaches in any type of organization. These and other myths of CMMI are espoused upon at the Carnegie Mellon website [8].

2.3 MSF
Microsoft Solutions Framework (MSF) encompasses Microsofts software development principles and experience. MSF defines itself not as a methodology, but rather as a framework that can support different methodologies. It is based on Microsofts experience both inside the corporation and from the field through consulting work. MSF is a deliberate and disciplined approach to technology projects based on a defined set of principles, models, disciplines, concepts, guidelines, and proven practices [16]. It has evolved through various stages; MSF 3.0 mainly focused on the stages involved in software development and emphasized a spiral waterfall that was intended to be agile and ensured customer trust through rapid delivery and verification with agility. MSF historically has been very closely aligned with agile practices. However, the latest version of MSF V4.0 adds additional levels of traceability and feedback for further improvement. In this way, MSF now supports many of the concepts embedded in CMMI.

manager select a template if the organization is not mandated to follow CMMI? Even if an organization formally does not need to embrace the CMMI, there are advantages of CMMI not included in Agile. At what point, do the benefits of CMMI outweigh the costs, specifically in the context of a TFS project. The TFS product provides some guidance in the help: Choose the MSF for Agile Software Development process for projects with short lifecycles and delivery-oriented teams who can work with lots of intermediate documentation. MSF for Agile Software Development is an iterative, scenario-driven development process Choose MSF for CMMI Process Improvement over MSF for Agile Software Development, if your organization is undertaking a broad quality assurance and process improvement initiative or your team needs the assistance of explicit process guidance rather than relying on tacit knowledge and experience. Choose the MSF for CMMI Process Improvement process for projects with longer life cycles and that require a record of decisions made Reading this, one might think that choosing the CMMI model forfeits the capability to utilize agile methods. However, this is not the intent for the two different models. The fundamental construct behind MSF 4.0, utilized by both process templates, is the support of agile methods while providing a framework for CMMI. The CMMI template is thus an addition to the Agile framework inherent in MSF, rather than a replacement. Anderson explains this in detail in his account of the how the CMMI Process Improvement process template was developed at Microsoft [5]. The CMMI process template is thus a super-set rather than a different set of features as illustrated in figure 1.

2.4 Team Foundation Server

Team Foundation Server (TFS) is part of the Microsoft Visual Studio 2005 Team System [17]. The Team Suite development includes the Visual Studio Team Suite Interactive Development Environment (IDE), which contains tools focused mainly on the application developer, but embodies other roles including architects, testers, and database administrators. TFS provides configuration management of both code and documentation artifacts. It provides testing automation services, quality control work flow and serves as a central repository and collaboration portal for a software development team. TFS leverages several different Microsoft products including SQL Server 2005 database, data analysis, and reporting services as well as Microsoft SharePoint Services for collaboration. Although it is possible to implement TFS on a single server, a scalable enterprise TFS implementation requires different server roles including web as well as database and application servers. The TFS infrastructure is pivotal in Microsoft Team development approach and is heavily leveraged by an increasing number of customers and partners as well as internally at Microsoft. TFS provides the core infrastructure for software configuration management, centralized project management, and collaborative and disciplined development for Microsoft team projects. It integrates software development process with quality assurance through rules-based configuration management. It also provides an infrastructure for supporting CMMI goals by providing templates for deliverables, guidance documentation, as well as automated metric collection and reporting features available from the TFS portal.

Functional/Exploratory Testing Iteration Planning Test Driven Development Context-Driven Test

Shadow Architecture

MSF for Agile Software Development Change Requests Process Improvement Activities Formal Reviews Approvals

MSF for CMMI Process Improvement Requirements Analysis

Figure 1- Agile vs. CMMI process template (Source: Randy Miller, Microsoft Tech Ed 2006 Presentation DEV439) Andersons explanation relies heavily on the Deming feedback model where organizational improvement is driven by continuous statistical feedback [12]. Much of the basis for CMMI stems from Demings work. Of Demings 14 points for management, there are 7 points that particularly highlight the benefits of an agile approach within a continuous improvement framework: 3) Cease dependence on quality control to achieve quality. Instead focus on quality assurance throughout the lifecycle. 4) Build trust and loyalty with suppliers. 6) Training on the job 7) Leadership 8) Drive out fear.

3. DISCUSSION 3.1 Differences between CMMI and Agile TFS Process Templates
One of the first decisions the project leader must make when creating a new TFS project is what process template to use. Given the background on CMMI and Agile, how should the project

188

9) Break down barriers between departments. 12) Remove barriers to pride of workmanship. management on quality rather than production numbers. Focus

through the issues or are more issues materializing as we develop? Issues List: What are the issues and how much time have they used and how much time are the expected to continue to consume? Utilizing the CMMI process in TFS pre-creates numerous documents and work item types associated with several KPAs. Not all of the CMMI levels are automated within the CMMI process template. Microsoft chose to target a CMMI level 3 for the 2005 version, but includes aspects from other levels. TFS supports extending the process template to add additional items as well as supplementing sites built from the template with additional document or site templates. 17 of the 21 Level 3 areas are targeted including project planning, monitoring, control, measurement, requirements definition, configuration management, risk management, and requirements validation. Through reporting features, quantitative project management and causal analysis and resolution from levels 4 and 5 are partially addressed. However, some level 4 and level 5 items including supplier agreement management, organizational environment for integration, organizational process focus, organizational training, and integrated supplier management are not provided in the 2005 version.

Given that agile methods can be incorporated into a CMMI approach, this simplifies the selection basis, at least in the context of the TFS scenario. The decision becomes one mainly of determining if the extra items associated with the CMMI template are beneficial for the organization or the software project. In addition, the use of CMMI would not be recommended if the organization already has a process template in place that has proven successful. Utilizing CMMI within the MSF TFS context is mainly a stretch-to-fit choice, a term coined by Alistair Cockburn [10]. Although an agile mindset is accommodated within the TFS CMMI process template, it is important to recognize that some aspects of the CMMI requirements are not agile. The main areas of the CMMI that require more rigidity are [14]: Project Planning (PP) 2.1 asks to identify task dependencies and expects a Critical Path Method (CPM) or Program Evaluation and Review Technique (PERT) chart as a result. PP 1.1 sub-practice 2 asks to Identify the work packages in sufficient detail as to specify estimates of project, tasks, responsibilities and schedule and goes on with The amount of detail in the WBS at this more detailed level helps in developing realistic schedules, thereby minimizing the need for management reserve. Project Monitoring and Control (PMC) 1.1 asks to compare actual completion of activities and milestones against the schedule documented in the project plan, identifying significant deviations from the schedule estimates in the project plan. Technical Solution (TS) 2.2 asks to establish a technical data package and suggest artifacts like product architecture description, allocated requirements, product component descriptions, product characteristics, interface requirements, conditions of use. Understanding the costs of using a CMMI versus an agile approach in a TFS scenario requires understanding TFS product capabilities. Much of the overhead associated with a CMMI process is automated through the product. For example, the number of bugs per work item and iteration are tracked through the automated test feature. An adapter framework is used to gather the data for the reports as well as store in a data warehouse for historical reporting. The metrics gathered by TFS and available for both ad-hoc and standard reporting regardless of template selection include the following [15]. An example report is illustrated in figure 2. Bug Rates: Finding, fixing, and closing bugs as well as bug trends answers questions such as Is the number of new defects increasing slower than the rate at which the defects are being corrected? How long is it taking to resolve defects? Builds: Is code coverage being achieved? Are the validation tests passing? Is code churn [18] occurring wherein modules are continually being modified without any new tests being defined? Project Velocity: The rate at which the team is processing and closing work items answers the question Are we progressing

Figure 2- Sample TFS report The TFS infrastructure provides work item tracking. The Microsoft team development model revolves around work items. Capabilities to enforce automated system builds, regression testing verification, document review, and bug resolution are couched in the linkage of work items to related tasks. The work items are centrally manageable and become part of the data warehouse used to report project status. The state of the project is a reflection of the states of the work items. An important measure derived from monitoring work item state is project velocity, the rate at which the team is processing and closing work items. Guckenheimer and Perez [13] describe this metric as a key indicator for estimating the completion date for the next iteration.

189

by the MSF for CMMI process template, but also for higher CMMI levels. Taking into consideration the automation provided by TFS and that the CMMI process template incorporates agile benefits, we have defined a flowchart to assist in formulating the decision to utilize the CMMI template (figure 4). Our decision-making process emphasizes the role of the business environment over technology issues. In addition, we provide a matrix (table 1) that outlines the key issues involved in the decision along with a prioritization column to auto-calculate the best decision path. Table 1 - Weighting matrix for CMMI vs. Agile decision Factor Traceability Quick ROI Auditing Resource shortage Time shortage Requirements known in advance CMMI + + + Agile + + + Weight

Figure 3 - TFS Work Item State Transitions (Source: Brian Harry, Microsoft Tech Ed 2006 Presentation DEV327) Guckenheimer and Perez also discuss the benefit of a value-up paradigm shift. In a value-up paradigm, the value of the product is realized through iterations and improvement rather than being dictated by a strict specification up-front. The focus is on delivering a solution that meets key customer needs and building upon success rather than trying to produce intermediate deliverables aimed at a long-term specification that may or may not prove accurate given the rate of technology and business changes which frequently occur even with short life cycles.

Many organizations that moved to an agile approach now find themselves trying to conform to a CMMI approach. As the number of organizations performing this transition increases, the literature should improve in quantity and quality. One organization that made the transition and published their results and lessons learned is DTE Energy [7]. DTE Energy describes their viewpoint that CMMI is a framework of what while their agile methods provide an empowering philosophy of how.
No CMMI Mandated

Yes Existing Agile Approach? No Estimated Effort above W $ No Yes Detailed auditing/ traceability No

3.2 TFS Decision Methodology

Utilizing a tools-based approach such as TFS for automating a software development process, whether choosing an agile or CMMI focused approach provides numerous benefits and improves the efficiency of development. One of the underlying assumptions with MSF is that metrics should be descriptive rather than prescriptive. This is based on Robert Austins work Measuring and Managing Performance in Organizations [20]. The theory is that prescriptive metrics become inaccurate when an organizations places an undue emphasis on metrics. Although the TFS tooling helps in many areas, choosing to use the CMMI model will put more stress on the project team due to the sheer number of deliverables that are required. In the CMMI process template, there are 59 work product artifacts while Agile has 25. There are 9 metric charts with Agile but 12 with CMMI. Anderson [5] estimates that 85 % of the overhead is resolved by automation and that the overall amount of effort required for a CMMI template is only 150% beyond that of Agile. In the TFS environment, through the data warehouse capabilities and the ongoing capture of information during the check-in and build processes, metrics are gathered that describe what is occurring at the code development and testing levels. Many of these metrics are required for not only the CMMI level 3 targeted

Yes

Use Agile Process Template as Base

Estimate more than X weeks Customize Agile Framework to match existing approach

Use CMMI Process Template

No

More than Y Function Points

No More than Z Team Members No

Use Agile Process Template

Figure 4 - Process template decision-making flowchart

190

4. CONCLUSION
A comparison of Agile and CMMI is not truly an Apples-toapples comparison. CMMI is a maturation model, while Agile is a development philosophy. The distinction is subtle but significant because it means that the approaches can potentially complement each other rather than being mired in incompatibility, Therefore, the choice of one or another as a process template need not be a sacrificial proposition. It is possible to leverage agile practices in a CMMI context. TFS provides both Agile and CMMI process templates that integrate with Microsoft Team Suite core capabilities. We have provided a flowchart and matrix customizable for an organization to assist in the template selection. Customization is achieved by modifying the variables and weighting used in the trade-off decisions. Using a tool such as TFS that provides configuration management, automated testing, Q/A checking, and metrics gathering facilitates a CMMI approach that still supports agility. This approach can be used to help automate continuous improvement regardless of the CMMI level targeted. A tools-based approach ensures objective metric collection and reduces cost by focusing resources on actual development and project activities, rather than on enforcement, record-keeping and other tasks better done by automated tools.

[8] Carnegie Mellon Software Engineering Institute. 2006. What is CMMI. Retrieved from http://www.sei.cmu.edu/cmmi/general/general.html on December 26, 2006. [9] Carnegie Mellon Software Engineering Institute. 2006. Capability Maturity Model Integration (CMMI) Version 1.2 Overview. Retrieved from http://www.sei.cmu.edu/cmmi/adoption/pdf/cmmioverview06.pdf on December 26, 2006. [10] Cockburn , A. 2006. Crystal Main Foyer. Retrieved from http://alistair.cockburn.us/index.php/Crystal_methodologies_ main_foyer on December 26, 2006. [11] Conboy, K. and Fitzgerald, B. 2004. Toward a conceptual framework of agile methods: a study of agility in different disciplines. In Proceedings of the 2004 ACM Workshop on interdisciplinary Software Engineering Research (Newport Beach, CA, USA, November 05 - 05, 2004). WISER '04. ACM Press, New York, NY, 37-44. DOI= http://doi.acm.org/10.1145/1029997.1030005 [12] Deming, W. 1982. Out of Crisis, The MIT Press, Cambridge, MA. [13] Guckenheimer, S. Perez, J. 2006. Software Engineering with Microsoft Visual Studio Team System. AddisonWesley. Upper Saddle River, NJ. [14] Highsmith. 2000. Adaptive Software Development: A Collaborative Approach to Managing Complex Systems. New York: Dorset House [15] Holland, D. 2006. Microsoft Visual Studio 2005 Team Foundation Server. Retrieved from http://dotnet.syscon.com/read/231944.htm on December 26, 2006. [16] Microsoft. 2006. Microsoft Solutions Framework. Retrieved from http://www.microsoft.com/technet/solutionaccelerators/msf/d efault.mspx on December 26, 2006. [17] Microsoft. 2006. Visual Studio 2005 Team Foundation Server. Retrieved from http://msdn2.microsoft.com/enus/teamsystem/aa718825.aspx on December 26, 2006. [18] Nagappan, N. and Ball, T. 2005. Use of relative code churn measures to predict system defect density. In Proceedings of the 27th international Conference on Software Engineering (St. Louis, MO, USA, May 15 - 21, 2005). ICSE '05., 284292. DOI= http://doi.acm.org/10.1145/1062455.1062514 [19] Palmer, S. and Felsing, J. 2002. A Practical Guide to Feature Driven Development. Upper Saddle River, NJ, Ptentice-Hall. [20] Schwaber. 1995. Scrum Development Process. OOPSLA95 Workshop on Business Object Design and Implementation. Springer-Verlag. [21] Zuser, W., Heil, S., and Grechenig, T. 2005. Software quality development and assurance in RUP, MSF and XP: a comparative study. In Proceedings of the Third Workshop on Software Quality (St. Louis, Missouri, May 17 - 17, 2005). 3-WoSQ. ACM Press, New York, NY, 1-6. DOI= http://doi.acm.org/10.1145/1083292.1083300

5. AKNOWLEDGEMENTS
Thanks to Dr. David Umphress, Auburn University for insights on improving software process efficiency and encouragement with publication of this paper and Dr. Kai Chang, Auburn University for elaborations of CMMI and software quality.

6. REFERENCES
[1] Abrahamsson, Salo, Ronkainen, Wastra. 2002. Agile Software Development Methods: Review and Analysis. VTT Publications 478. Retrieved from http://www.inf.vtt.fi/pdf/publications/2002/P478.pdf on December 26, 2006. [2] Abrahamsson, P., Warsta, J., Siponen, M. T., and Ronkainen, J. 2003. New directions on agile methods: a comparative analysis. In Proceedings of the 25th international Conference on Software Engineering (Portland, Oregon, May 03 - 10, 2003). International Conference on Software Engineering. IEEE Computer Society, Washington, DC, 244-254. [3] The Agile Alliance. 2001. Manifesto for Agile Software Development retrieved from http://agilemanifesto.org/ on December 26, 2006. [4] Austin, R. 1996. Measuring and Managing Performance in Organizations. Dorset House Publishing. New York. [5] Anderson, D. J. 2005. Stretching Agile to fit CMMI Level 3 - the story of creating MSF for CMMI Process Improvement at Microsoft Corporation. In Proceedings of the Agile Development Conference (July 24 - 29, 2005). ADC. IEEE Computer Society, Washington, DC, 193-201. DOI= http://dx.doi.org/10.1109/ADC.2005.42 [6] Beck. 1999. Extreme Programming Explained: Embrace Change. 2000. Addison-Wesley. [7] Baker, S. 2005. Formalizing agility: an agile organization's journey toward CMMI accreditation. Agile Conference, 2005. Proceedings.

191