CM Crossroads Webcast Series

Challenges and Characteristics of

Enterprise Continuous Integration
Moderator
Patrick Egan
Publisher – CM Crossroads and Agile Journal

Speakers
Paul Julius – Willowbark Consulting
Anders Wallgren – Chief Technical Officer Electric Cloud

© CMC Media 2008

 Challenges and
Characteristics of Enterprise
Continuous Integration

www.electric-cloud.com
 Introduction
In the past…
Teams spent weeks or longer verifying that applications
worked
Time spent „integrating‟ was tortuous, lead to integration
„storms‟ of broken builds
Developers would have to track down bugs introduced
months earlier
The answer to this painful process is Continuous Integration

October 9, 2008 Slide 3

 CI: The Beginning
Continuous Integration (CI) dictates that, as soon
as changes are made, they are integrated and run
through a rigorous build and test cycle
Build engineers began using various CI tools,
sometimes without management approval
Once these tools became known, they were
embraced by developers who wanted and needed
the instant feedback on the health of their builds
As teams and applications grow over time, many
enterprises find themselves with CI servers
spread across the company

October 9, 2008 Slide 4

 Challenges
With the proliferation of multiple standalone
systems, next step is to investigate the idea of
consolidation
Consolidated approach will help:
Teams to reuse procedures and tools
Better utilize hardware
Free personnel from manual, low value tasks

October 9, 2008 Slide 5

 Challenges in Real World Enterprise CI

No centralized build-test-deploy infrastructure

Difficulty scaling the environment
Testing and resource administration increase in
complexity
Hardware Utilization difficulties
Barriers to sharing and reusing scripts
Difficulty tracking builds and cross-project
reporting

October 9, 2008 Slide 6

 S.T.A.R
The challenges of Enterprise CI require a CI
solution that is:
Scalable
Turnkey
Automatic
Rapid

October 9, 2008 Slide 7

 Scalable
Has many aspects
Must scale by:
Number of projects
Size of an individual project
Number of users of the system
Number of machines in the build cluster
Complexity of workflow

Enterprise CI is fundamentally about many projects and

many users

October 9, 2008 Slide 8

 Turnkey
Enterprise CI has to be turnkey. The number of
builds, number of users, and the reality of 24/7
operations make this imperative
Turnkey implies that from the very beginning,
Enterprise CI must be useable. It must be trivial
to add scripts to the system, without requiring a
rewrite.

October 9, 2008 Slide 9

 Automatic
CI system must be automatic, builds must be
launch-able with a single click, but it must also be
compatible with existing scripts
Three forms of automatic: On demand, On
stimulus and On the clock
On Demand: Builds should be a single click away
On Stimulus: Typically caused by changes in the
SCM system
On the Clock: Replacement for ‘cron’

October 9, 2008 Slide 10

 Rapid
As the system scales to mulitple projects, and the
number of builds increase, builds have to be
rapid.
Hours long builds are a deal breaker
Reducing build times from hours to minutes
typically is done using parallelism
Enterprise CI systems should be able to work in
tandem with build parallelism software

October 9, 2008 Slide 11

 Other Properties
Three other properties are important to
Enterprise CI:

Auditing

Virtualization

Reporting

October 9, 2008 Slide 12

 Customer Example
Problem
Medium sized Product company
10‟s of disparate existing builds
Varied build infrastructure
Nearly impossible to identify a “good” build

Solution
Unified Dashboard for all builds
Consistent build infrastructure
Templated, resuable build infrastructure

October 9, 2008 Slide 13

 Next Steps
It’s easy to get started with an open source CI
system
But as your needs change and your company
grows you have to evaluate the effectiveness of
this solution.

October 9, 2008 Slide 14

 Build vs. Buy
Key question then, should we use an open source
tool, build our own, or buy a commercial tool?

October 9, 2008 Slide 15

 Four Key Areas
As we mentioned four key areas to review
Fast Builds
Software keeps growing = longer builds

On the Clock Builds

Scheduled Builds
The classic „nightly build‟

On-demand Builds
Ad hoc, as needed builds

On-Stimuli Builds
Driven by changes in SCM

October 9, 2008 Slide 16

 Build Your Own?
Seems pretty straightforward…
Investment

Automated build
and test for one
build tool, one
machine

Scheduled jobs

Features
Trigger builds
after check-ins

October 9, 2008 Slide 17

 Build Your Own?
Just a few finishing touches
Investment

Automated build
and test for one
build tool, one
machine

Scheduled jobs
Web-based CruiseControl
access to results
gives you this
for Java/Ant
Features
Trigger builds Notifications via e-mail/RSS
after check-ins
Database of
October 9, 2008 build results Slide 18
 Build Your Own?
Multiple servers, concurrency
Investment

Resource
pooling, load
Notifications via e-mail/RSS
balancing

Automated build Run multiple

and test for one builds
build tool, one simultaneously
machine

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
access to
Trigger builds simultaneously
results
after check-ins Run job steps
in parallel
Multiple servers,
Database of
remote invocation
build results
Single job can use
October 9, 2008 multiple resources Slide 19
 Build Your Own?
Better monitoring and control
Investment

Monitor system View partial

status results of
builds in
Resource progress
pooling, load
Notifications via e-mail/RSS
balancing

Automated build Run multiple

and test for one builds Cancel jobs
build tool, one simultaneously
machine On-demand job
invocation via
Web

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
access to
Trigger builds simultaneously
results
after check-ins Run job steps
in parallel
Multiple servers,
Database of
remote invocation
build results
Single job can use
October 9, 2008 multiple resources Slide 20
 Build Your Own?
Better reporting
Investment

Monitor system View partial

status results of
builds in
Resource progress
pooling, load
Notifications via e-mail/RSS
balancing

Automated build Run multiple

and test for one builds Cancel jobs
build tool, one simultaneously
Customizable
machine On-demand job
reports
invocation via
Web
Trend reports

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
Cross-product
reports
access to
Trigger builds simultaneously
results
after check-ins Run job steps Tools for Resource
in parallel extracting utilization
Multiple servers,
Database of data from logs reports
remote invocation
build results
Single job can use
Annotate builds
October 9, 2008 multiple resources Slide 21
after completion
 Build Your Own?
Better error handling
Investment

Monitor system View partial

status results of
Retry steps after
builds in
errors/ failures
Resource progress
pooling, load
Notifications via e-mail/RSS
balancing
Detect resource Logging and
Automated build Run multiple failures error reporting
and test for one builds Cancel jobs
build tool, one simultaneously
Customizable Timeouts for
machine On-demand job
reports runaway job
invocation via
steps
Web
Trend reports

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
Cross-product
reports
access to
Trigger builds simultaneously
results
after check-ins Run job steps Tools for Resource
in parallel extracting utilization
Multiple servers,
Database of data from logs reports
remote invocation
build results
Single job can use
Annotate builds
October 9, 2008 multiple resources Slide 22
after completion
 Build Your Own?
Handle multiple
Developer access
to production
builds
User independent groups
impersonation,
password
Shared access by management
multiple teams
Investment

LDAP Priorities
authentication
Monitor system View partial
Role-based status results of
Retry steps after
access control builds in
errors/ failures
Resource progress
pooling, load
Notifications via e-mail/RSS
balancing
Detect resource Logging and
Automated build Run multiple failures error reporting
and test for one builds Cancel jobs
build tool, one simultaneously
Customizable Timeouts for
machine On-demand job
reports runaway job
invocation via
steps
Web
Trend reports

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
Cross-product
reports
access to
Trigger builds simultaneously
results
after check-ins Run job steps Tools for Resource
in parallel extracting utilization
Multiple servers,
Database of data from logs reports
remote invocation
build results
Single job can use
Annotate builds
October 9, 2008 multiple resources Slide 23
after completion
 Build Your Own?
More programming
Developer access
to production
builds
User features
impersonation,
password Modular,
Shared access by management Extensibility composable
multiple teams process steps
Investment

Parameters Control
LDAP Priorities
environment
authentication
Conditional steps for jobs
Monitor system View partial
Role-based status results of
Retry steps after
access control builds in
errors/ failures
Resource progress
pooling, load
Notifications via e-mail/RSS
balancing
Detect resource Logging and
Automated build Run multiple failures error reporting
and test for one builds Cancel jobs
build tool, one simultaneously
Customizable Timeouts for
machine On-demand job
reports runaway job
invocation via
steps
Web
Trend reports

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
Cross-product
reports
access to
Trigger builds simultaneously
results
after check-ins Run job steps Tools for Resource
in parallel extracting utilization
Multiple servers,
Database of data from logs reports
remote invocation
build results
Single job can use
Annotate builds
October 9, 2008 multiple resources Slide 24
after completion
 Build Your Own? It Never Ends
Support multiple SCM integration,
Web interface for
languages and bill of materials
editing processes
build tools
Developer access
to production Command-line
User
builds interface
impersonation,
Portable across
password Modular,
hardware/OS
Shared access by management Extensibility composable
platforms
multiple teams process steps
Investment

Parameters Control
LDAP Priorities
environment
authentication
Conditional steps for jobs
Monitor system View partial
Role-based status results of
Retry steps after
access control builds in
errors/ failures
Resource progress
pooling, load
Notifications via e-mail/RSS
balancing
Detect resource Logging and
Automated build Run multiple failures error reporting
and test for one builds Cancel jobs
build tool, one simultaneously
Customizable Timeouts for
machine On-demand job
reports runaway job
invocation via
steps
Web
Trend reports

Scheduled jobs
Web-based Features
Multiple jobs run
on one resource
Resource
selection criteria
Cross-product
reports
access to
Trigger builds simultaneously
results
after check-ins Run job steps Tools for Resource
in parallel extracting utilization
Multiple servers,
Database of data from logs reports
remote invocation
build results
Single job can use
Annotate builds
October 9, 2008 multiple resources Slide 25
after completion
 Build Your Own
You could try that
Before you do think about…
How much time do you have to write all that code?
How you are going to manage 200+ builds per day?
How are you going to get the build down from 4 hours to 15
minutes?
How are you going to manage hardware failure in your build
system?

Build Managers now control the heartbeat of

software engineering
They need something more than a creaky Perl script

October 9, 2008 Slide 26

 Build Your Own
Fast builds
Buy lots of SMP hardware and try out GNU Make parallelism
or manually parallelize the build.

Scheduled builds
Use cron for that

On demand builds
Build an intranet page, integrate it yourself with the current
build and source code management system

Stimuli builds
Build ad-hoc script attached to source code management
system

October 9, 2008 Slide 27

 Open Source
Fast builds
distcc, ccache, GNU Make -j

Scheduled Builds
cron, CruiseControl

On-demand Builds
Apache Continuum

Stimuli Builds
CruiseControl has some support
SourceControl plug-in

October 9, 2008 Slide 28

 Commercial
Build speed tools
Like ElectricAccelerator

Build visualization and troubleshooting tools

Like ElectricInsight

Build management tools

Like ElectricCommander or CruiseControl

Together these tools enable build managers to

build, package, test, and deploy software
And they enable Agile development

October 9, 2008 Slide 29

 Stepping Stones
Builds up 18 months little by little
Four stepping stones

October 9, 2008 Slide 30

 Stepping Stones (1)
Step 1: Put full builds in the hands of developers
Set up an internal web page that allows a
developer to fire off a build
The developer should be able to select:
Which branch to build…
… or the location of his personal source tree
Whether to run unit tests or not
The build runs and the engineer gets an email
with the result
Start a new rule: developers run on demand
builds before they commit

October 9, 2008 Slide 31

 Stepping Stones (2)
Step 2: Buy those lava lamps
Set up the full build system to run more than
once per day
Set up a system the feedback to the entire team
whether the full build succeeded
red/green lava lamps or
a flat panel screen mounted high enough for engineering to
see and
a web page showing live build status

Scheduled builds start to be the heartbeat of

engineering

October 9, 2008 Slide 32

 Stepping Stones (3)
Step 3: Speed up the build
Look into tools that can speed up your build
Set an under-60 minute goal for a full build and
smoke test
Announce to the team that multiple integrations
can now happen per day, during the day
These fast builds begin to enable agile
development

October 9, 2008 Slide 33

 Stepping Stones (4)
Step 4: Fully automate
Integrate your fast builds with your source code
management system
When the source code changes run a stimuli
build and test it
Now the lava lamps really matter
Now the build is the heartbeat of engineering
Start referring to yourself as the Build Automator
not the Build Manager

October 9, 2008 Slide 34

 When to buy
Look at $$$ build tools when
Existing builds are too slow; developers complain
Existing builds are frequently broken
Existing builds are blocking implementation of new
techniques (e.g. Agile/XP)
Development teams are spread across the country or the
globe

Other drivers are

Company mergers
Implementation of new target platforms
A rapidly growing code base

October 9, 2008 Slide 35

 Summary
No one builds their own:
Editor
Compiler
Debugger
SCM

Most people don’t build their own:

Issue tracker
Test system

Why DIY the build system?

October 9, 2008 Slide 36

 For More Information
Electric Cloud
408.419.4300
info@electric-cloud.com
www.electric-cloud.com

Willowbark Consulting
312.560.5351
PJ@WillowbarkConsulting.com
www.WillowbarkConsulting.com

October 9, 2008 Slide 37

Questions and Answers

Please post your questions now using

the “Ask a Question” box
on left side of the screen

View other Webcasts in the Series at www.cmcrossroads.com/wc

© CMC Media Inc. 2008