Ricardo Vargas Ante Chelson Ballintine

UNCLASSIFIED
Know the EarthShow the Way
Using EVM Reporting to Develop

Cost Estimating Models
The Value and Pit Falls
Alexander Ante Heather Chelson Carly Ballintine

alexander.j.ante@nga.mil heather.f.chelson@nga.mil carolyn.a.ballintine@nga.mil
November 2007
PMI Conference, Cost Estimating Track
UNCLASSIFIED
NATIONAL GEOSPATIAL-INTELLIGENCE AGENCY
NATIONAL GEOSPATIAL-INTELLIGENCE AGENCY UNCLASSIFIED
Abstract
Cost and technical data are the two essential inputs for developing
cost models. One input cannot be analyzed exclusive of the other.
However, we have found that many program offices assume that
cost estimators can simply take Earned Value Management (EVM)
data from the wInsight data and from the Cost Performance
Reports (CPRs) and use them for cost modeling efforts. As a
result, much of the data collection process becomes repairing the
relationship between the technical and cost actuals, in an
extensive effort to normalize the data.
This presentation will address pitfalls and recommend approaches
during the data collection and normalization process to ensure that
the data we receive is complete, auditable, usable, and relevant.
We have made some achievements but have also stumbled
across difficulties.
2
UNCLASSIFIED
What We Are Building

Cost Estimating Relationships (CERs) these
are our cost equations
Analogy Its like one of these
Parametric This pattern holds
Engineering build-up Its made up of these
Cost models
Estimate single cost element
Estimate program costs
Sensitivity analysis
Examining the cost drivers
Building Cost As an Independent Variable (CAIV)
modeling
3
UNCLASSIFIED
Background - Cost Estimating Process
Cost and technical data are essential inputs for developing cost models.
4
UNCLASSIFIED
Getting the Data

leave no source unexplored
CPRs cost inputs Cost data must be normalized and
CFSRs mapped to the Cost Estimating
Structure (CES)
etc.
schedule inputs Schedule data must be structured to

baseline schedule
clearly show initial plans versus actual,
final schedule and the deltas between them
system description technical inputs Technical data must show an

code counts understanding of program mission,
arch / system views capability, functionality, risks, etc.
etc.
Programmatic data must be analyzed
programmatic inputs for decisions, policies, strategies
contract summary
acquisition strategy impacting program life and, in turn, cost
etc.
A complete understanding of schedule, technical, and programmatic

aspects is key to properly using the collected cost data and separating
the anomalies from common experience.
5
UNCLASSIFIED
Understanding the Data
Products and services to be procured

Relationship between the Product Work Breakdown
Structure (PWBS) & Cost Estimating Structure (CES)
Layout of the contract structure
Contract, CLIN, WBS, Control Accounts
Cost Structure (i.e. direct, indirect, ODCs, etc.)
Size of the WBS and control accounts
Cost data tie-ins to schedule, programmatic, & technical
data
Additional data or reports that need to be generated to
complete data collection
6
UNCLASSIFIED
Evaluating the Data
Cost Schedule
Technical Programmatic
Is the data usable?
Integrity, Completeness, Accuracy, Timeliness, Consistency, Constraints

7
UNCLASSIFIED
Evaluating Data Quality

Integrity
Does the contractor have a controlled process to systematically
collect the data?
Are there time lags between the EOM subcontractor data & the
prime contractors reporting dates?
Are there patterns of mischarges & accounting errors?
How often is the EOM data (schedule, EVM, and technical reports)
updated and reconciled?
Completeness
Does the PMB capture the scope of the program as contracted?
Does the WBS structure capture the scope of the program as
contracted? Integrated Master Schedule (IMS)?
Does the IMS reflect the program as planned?
Accuracy
Do all the elements roll up to higher levels?
Can the data (EVM, Schedule, TPMs) be traced month to month?
Does one see a lot of corrections?
Does the EVM methodology used give a good measure of
progress?
8
UNCLASSIFIED
Evaluating Data Quality

(continued)
Timeliness
Are the BCWP / Schedules / TPMs posted and updated in a timely
manner?
Are the LRE and schedule-to-complete updated in a timely manner?
Are changes to the baseline timely and controlled?
Consistency
Are the WBS definitions & structures stable?
Has the EVM data been reliable in the past?
Has the EVM data been consistent with the programs TPMs,
schedules, and other PM tools?
How stable are the EVM indicators? (CPI, SPI, LREs)
Constraints
Is the data proprietary?
How can subcontractor data that is not available to the prime
contractor be used?
Are the contractor/subcontractor rates available?
9
UNCLASSIFIED
Normalizing the Data

Got the Data
Assessed the Data
Normalize the Data
10
UNCLASSIFIED
Normalizing the Data

Normalization begins with the contractor raw data for the
entire program and reflects all involved contracts
Normalization Basic Steps
Obtain raw cost data (both dollars and hours)
Begin with wInsight pulls, but may need to supplement with specific
cost pulls (i.e. month to month, current vs. cumulative, etc.)
Profile data: is it labor, materials, etc.?
Collecting the hours can help in finding imbedded material charges
Be consistent with application of G&A across all contracts; is it
above or below the line?
Remove inflation from the cost data
May need to allocate dollars and hours of summary level accounts
to each program deliverable (i.e. software releases, aircraft
engines, etc.)
Methods to allocate: requirements, hours, WBS, $, schedule
Define each milestone: Is it PSR, ORR, etc?
11
UNCLASSIFIED
Allocation Example
To Complete Allocation for Release 2,
Allocation of a Portion of Release 1
and Release 3 is Unavoidable
Account Requiring Allocation
Release 1 POP
FOCUS RELEASE - Release 2 POP
Allocation Release 3 POP

between 1 & 2
1, 2, 3
2&3
Range of Allocation Activity
12
UNCLASSIFIED
Common Normalization Issue for

AIS Systems
Correct the problem of data inconsistency (common
among programs and over time)
Key component of
One of the major consistency challenges technical data for
is associated with software and AIS programs
software code counts
Inconsistent code counts from release to release
Different code counters, definition of LOC and ELOC equations
between contractors
Possible solution:
Standardize using USC Code CounterTM (Barry Boehm) method
13
UNCLASSIFIED
Consolidating the Normalized Data

Overall end product for data collect is a database
Build database from normalized data
Each step in the normalization process creates data points
What constitutes a data point?
A program, release, or a contract?
Program associated with contract(s)
Scope thru last completed release
Data point must tie out to contract(s) cumulative totals
Release associated with a delivered capability
Releases addressed individually
Combination of Releases may be required in some cases
Sum total of Releases must tie out to contract cumulative totals
Contract associated with the effort defined in a SOW
Includes all Releases (and ECPs) under a single contract number
Data point must tie out to contract cumulative total Pitfall
What % complete is required prior to normalization?
Early processing of data yields misleading and unstable results
14
UNCLASSIFIED
Documenting Points (Creating a Data Book)

The Data Book describes, in detail, every aspect of the data point
and serves as the official documentation of that data point and the
associated normalization process
The Data Book contains (using an AIS example)
Executive Summary
Detailed descriptions of programmatics
Contracts, acquisition strategy, security, training, maintenance concept, etc.
Detailed descriptions of technical information
Capabilities / functionality provided by Release
Hardware lists / software code counts (items and configurations)
Performance characteristics, interfaces, technical issues and impacts, etc.
Documentation of normalization process
Basis for allocation
Rationale supporting mapping decisions
Results of normalized data analysis
Quality assessment of data
15
UNCLASSIFIED
Value-Added At the End of the Process

Results can be used with confidence
Quality reviewed, Government approved data
Results facilitating valid cross-effort comparisons
Consolidated data easily accessible
Analysts can spend time analyzing vice hunting
Full traceability to contractor raw data
Each normalization and allocation step demonstrated
Allows analysts to see what was done and tailor for special
circumstances, if necessary
Clear documentation of normalization rationale
Critical to defense of decisions made during the estimating process
Credible foundation for method/tool improvement and model
calibration in support of improved estimate quality
Methods tailored to the realities of our specific environment
16
UNCLASSIFIED
Active Data Collects:

Common Challenges Revealed
17
UNCLASSIFIED
Common Frustrations and Challenges

Cost estimators and EVM analysts face similar challenges
Data availability (lost data, insufficient detail,
incomplete data) => severely compromises the quality
of the data point for each release where data is
missing.
Different WBS structures, within programs and across
programs (without bridges between old and new)
Inconsistent data across programs
Lower level data how low can we go?
G&A rates, COM/Fee rates
Discrepancies in code counting
Contracts only provide data in totals => leads to difficulty in
finding a basis for allocation and mapping. Also degrades the
data by forcing analysis of cost and schedule characteristics to
higher levels.
18
UNCLASSIFIED
Active Collects Have Taught Us

Lower level data is required to properly allocate costs
Higher level (summary level) accounts are not adequate
Normalization of data cannot be accomplished resulting in
inconsistencies between Releases and among programs
Validated models cannot be constructed (built up)
Monthly profile data is required to properly allocate cost
Final cumulative costs are not adequate, even when lower level
detail is available
Many allocations are based on Release period of performance, which
is masked when only final cumulative cost is available
Losing (or not being able to obtain) insight into historical
program cost detail brings the process to a halt
The ability to allocate and map costs is compromised
19
UNCLASSIFIED
Why Lower-Level Reporting Matters

Original Reporting Structure New Reporting Structure
Do the $/Hr roll up to the higher EXAMPLE 1:

1 Program Management & Sustaining Eng
EXAMPLE 1:
1 Program Management & Sustaining Eng
1.1 Program Management
WBS levels? Are their months 1.1.1
1.1.2
Program Manager Staff
Quality Assurance
1.1.3 Infrastructure Support
missing? 1.1.4
1.1.5
Program Administration
Process Engineering
1.2 Business Management
Structures must allow us to 1.2.1

1.2.2
Cost and Performance Management
Contract Administration
1.2.3 Procurement/Property Administration
segregate 1.2.4
1.2.5
Security Management
Data
1.2.6 Business Process Engineering
Program management activities 1.2.7

1.3
ECP Preparation
Sustaining Engineering
1.3.1 System Engineering
from hardware purchases 1.3.2
1.3.3
System Architecture
Software Configuration Management
1.4 Procurement
Software development from 1.4.1

1.4.2
Hardware
Software
1.4.3 Hardware Maintenance (SDF)
operations and maintenance 1.4.4
1.4.5
Software Maintenance
Site
1.4.6 ITF Hardware
COTS software purchases from 1.5
1.5.1
Operations and Maintenance (O&M)
Site Software Maintenance
1.5.2 SDF Support for Software Maintenance
system engineering 1.5.3
1.5.4
Software Maintenance CFE
ITF Testing Support
1.5.5 Subcontract Support at Site
Data normalization process 1.6
EXAMPLE 2:
Special Studies
EXAMPLE 2:
2 Important Capability 1 2 Important Capability 1
requires segregation of different 2.1
2.1.1
CFE/Procurement
Hardware
2.1.2 Software
activities and account types 2.1.3
2.2
Hardware Maintenance
Systems Engineering
2.3 Software Engineering
Only way to support estimating 2.4
2.4.1
Software Development
Software Management and Planning
2.4.2 Design
activities/methods development 2.4.3
2.4.4
Code and Unit Test
Software Support
2.5 Transition Planning
2.6 OT&E
20
UNCLASSIFIED
Advice and Lessons Learned

Have an active program cost collection regimen
Collect, evaluate, and normalize data at significant milestones
(PSR, ORR, OTR)
Keep the developing contractor(s) involved with the data
collection, normalization, and analysis
Resolve data quality and validity issues as soon as practical
Important: Traceability to contractor raw data
Clear documentation of normalization rationale => critical to
defense of decisions
It is critical to evaluate the quality and validity of the
Earned Value, schedule, technical, and programmatic
data before using them to estimate or develop cost
estimating equations
Consistency, Consistency, Consistency

21
UNCLASSIFIED
Achievements
Standardized data collection and normalization
process
Better understanding of cost, technical, programmatic,
and schedule drivers
Synchronization of information derived from Earned
Value, technical, schedule, and programmatic
documents with the Cost Analysis Requirements
Description (CARD) / Intelligence Capability Baseline
Description (ICBD)
Draft Product Work Breakdown Structure (PWBS) with
Dictionary for future contracts
Implementation of SLOC counting methodology in
tandem with development contractors methodology
22
UNCLASSIFIED
Additional Training Available

Defense Acquisition University (DAU) DAWIA
Business, Cost Estimating, and Financial Management (BCF) 101
Introduction to Cost Estimating (2 weeks)
Business, Cost Estimating, and Financial Management (BCF) 204
Intermediate Cost Estimating (3 weeks)
Society for Cost Estimating and Analysis
www.sceaonline.org
CostPROFTM computer-based training
SCEA Annual Conference (June 24-27, 2008 Ontario, California)
23
UNCLASSIFIED
QUESTIONS?
COMMENTS?
24
UNCLASSIFIED
Bios
Alexander Ante, National Geospatial-Intelligence Agency
Voice: (703) 755-5424 E-Mail: alexander.j.ante@nga.mil
Alexander Ante has over twenty five years experience as Financial Management,
both in Acquisition and Base Operations. In his career as and United States Air
Force Officer, contractor, and government civilian, he has extensive experience
cost estimating, budgeting, & earned value management in support of
procurement of addition, aircraft, space, and information systems. He is currently
a government civilian at the National Geospatial-Intelligence Agency (NGA) where
he provides technical expertise in cost estimating and decision analysis support
for the procurement of intelligence related information systems.
Heather Chelson, National Geospatial-Intelligence Agency

Voice: (703) 755-5108 E-Mail: heather.f.chelson@nga.mil
Heather Chelsons background in cost estimating and financial management has
been primarily in support of the intelligence community through programs at TRW
and TASC (both now Northrop Grumman) and as a civilian at the National
Geospatial-Intelligence Agency. She has been key to developing the cost
capabilities at NGA and is author of several papers including topics such as
implementation of cost estimating risk, schedule achievability for software
development, CAIV and target costing, and predicting CPI.
She has been a member of several intelligence community awarded teams and
has received the NIMA/NGA Meritorious Unit Citation Award (twice), Director of
Central Intelligence Meritorious Unit Citation Award, and NRO Director's Team
Award for her work in the field of cost estimating and financial management.
25
UNCLASSIFIED
Bios (continued)
Carly Ballintine, National Geospatial-Intelligence Agency
Voice: (703) 755-5782 E-Mail: carolyn.a.ballintine@nga.mil
Carly Ballintine has a BS-Economics and a BA-Mathematics from the
University of Minnesota. She is a DAWIA certified cost analyst and has been
with the National Geospatial-Intelligence Agency for two years. Her key
assignments include data collection, CARD/ICBD development, and life cycle
cost estimates for major AIS programs.
26
UNCLASSIFIED
NATIONAL GEOSPATIAL-INTELLIGENCE AGENCY
27
UNCLASSIFIED

Ricardo Vargas Ante Chelson Ballintine

Diunggah oleh

Informasi Dokumen

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Ricardo Vargas Ante Chelson Ballintine

Diunggah oleh

Hak Cipta:

Format Tersedia

UNCLASSIFIED

Know the EarthShow the Way

Using EVM Reporting to Develop

Alexander Ante Heather Chelson Carly Ballintine

What We Are Building

Background - Cost Estimating Process

Getting the Data

schedule inputs Schedule data must be structured to

system description technical inputs Technical data must show an

A complete understanding of schedule, technical, and programmatic

Understanding the Data

Products and services to be procured

Evaluating the Data

Is the data usable?

Integrity, Completeness, Accuracy, Timeliness, Consistency, Constraints

Evaluating Data Quality

Evaluating Data Quality

Normalizing the Data

Assessed the Data

Normalize the Data

Normalizing the Data

Account Requiring Allocation

FOCUS RELEASE - Release 2 POP

Allocation Release 3 POP

Range of Allocation Activity

Common Normalization Issue for

Consolidating the Normalized Data

Documenting Points (Creating a Data Book)

Value-Added At the End of the Process

Active Data Collects:

Common Frustrations and Challenges

Active Collects Have Taught Us

Why Lower-Level Reporting Matters

Do the $/Hr roll up to the higher EXAMPLE 1:

Structures must allow us to 1.2.1

Program management activities 1.2.7

Software development from 1.4.1

Advice and Lessons Learned

Consistency, Consistency, Consistency

Additional Training Available

Heather Chelson, National Geospatial-Intelligence Agency

NATIONAL GEOSPATIAL-INTELLIGENCE AGENCY

Know the EarthShow the Way

Anda mungkin juga menyukai