AIR FORCE

AUDIT AGENCY

AIR FORCE NUCLEAR

ROADMAP ASSESSMENT

AUDIT REPORT
F2009-0005-FD3000
4 May 2009
 Executive Summary

INTRODUCTION Two incidents along with other impairments caused the

Air Force to put high level management emphasis on the
nuclear mission. In 2006, critical nuclear related inter-
continental ballistic missile parts, labeled as helicopter
batteries, were mistakenly sent to Taiwan. In 2007, a
B-52 crew mistakenly flew six nuclear weapons from
Minot AFB, North Dakota to Barksdale AFB, Louisiana.
As a result of these incidents and subsequent investigations,
the Air Force established a Nuclear Task Force to develop a
nuclear enterprise roadmap. The Task Force published the
roadmap titled “Reinvigorating the Air Force Nuclear
Enterprise” (hereafter referred to as the Roadmap) on
24 October 2008. The Roadmap requires an assessment
method capable of measuring progress made to improve
the nuclear enterprise through action plans. This method
uses measures of performance and effectiveness (hereafter
referred to as measures) to determine how well the
Air Force is implementing the Roadmap’s objectives and to
determine if the Air Force is accomplishing a safe, secure,
effective, and reliable nuclear mission.

OBJECTIVES We performed this audit because the Secretary of the

Air Force made improving the nuclear enterprise the
Service’s highest priority and to provide immediate input
and recommendations for management actions during the
Roadmap development process. Our objective was to
evaluate whether the Air Force established an assessment
method to adequately evaluate the nuclear enterprise.
Specifically, we determined whether Air Force personnel
established (a) adequate measures to assess Roadmap
implementation status and facilitate accurate reporting to
nuclear oversight bodies, and (b) adequate data collection
internal controls. We plan to perform a follow-on audit to
validate the accuracy of data collected and reported
through the established Roadmap measures.

CONCLUSIONS Air Force nuclear enterprise assessment methodology,

while generally adequate, required strengthening.
Specifically, Air Force nuclear personnel:

Developed measures adequately addressing root

causes identified in the Roadmap and the measures
should facilitate reporting to responsible oversight
bodies. However, 11 (14 percent) of 79 measures
did not fully address Roadmap objectives.

i
Executive Summary

Incomplete measures could lead to inaccurate

nuclear program assessments. (Tab A, page 1)

Did not develop adequate internal controls over

data collection to guide Major Commands
(MAJCOMs) and wings in their data collection
efforts. Without these controls, nuclear
assessment data is more susceptible to errors and
inconsistencies. (Tab B, page 5)

RECOMMENDATIONS From November 2008 through January 2009, Air Force

personnel corrected the deficient evaluation measures
identified by audit. Further, in February 2009, Assistant
Chief of Staff, Strategic Deterrence & Nuclear Integration
(AF/A10) officials established a data collection internal
control plan in response to audit concerns. These com-
pleted management actions properly address identified
deficiencies; therefore, this report contains no
recommendations requiring corrective action.

MANAGEMENT’S Management officials agreed with the audit results and

RESPONSE corrective actions taken during audit were responsive to the
issues and addressed identified deficiencies.

CHARLES E. ATKINSON, JR. TONY M. AMES

Associate Director Deputy Assistant Auditor General
Air and Space Operations Division Support and Personnel Audits

ii
 Table of Contents

Page

EXECUTIVE SUMMARY i

TAB
A Nuclear Roadmap Measures 1

B Internal Control Plan 5

APPENDIX
I Audit Scope and Prior Audit Coverage 9

II Locations Audited/Reports Issued 11

III Points of Contact 13

IV Final Report Distribution 15

 Tab A
Nuclear Roadmap Measures

BACKGROUND
The nuclear roadmap assessment process was designed to identify and measure progress
made toward reinvigorating the Air Force nuclear enterprise and meet strategic objectives
described in the Roadmap. Subject matter experts (SMEs) evaluate and validate
measures such as those created to assess how well the Air Force performs critical self-
assessments and root cause analysis (RCA). Then, Air Force senior leadership assign a
weight based on importance for each level of command hierarchy. The weighted
measures produce a score for each sub-objective and an overall performance score for
each unit which indicate progress achieved on Roadmap objectives. When implemented,
it should provide the Air Force the ability to communicate its nuclear enterprise status
and trends to DoD, national leaders, and nuclear oversight bodies.

Following validation and leadership approval of the measures, appropriate offices will
collect the required data and construct an initial or baseline assessment. Many planned
measures have never been collected and evaluated across the Air Force nuclear enter-
prise. The first assessment will serve as the baseline comparison for future assessments
to evaluate the progress or regression of any particular objective. Air Force leadership
will direct periodic assessments to maintain visibility on current performance and to track
trends.

AUDIT RESULTS 1 – NUCLEAR ROADMAP MEASURES

Condition. Overall, AF/A10 officials developed measures adequately addressing root
causes identified in the Roadmap and the measures should facilitate reporting to respon-
sible oversight bodies. However, we determined 11 (14 percent) of 79 measures did not
fully address Roadmap objectives (Exhibit 1). For example:

Inspection Findings. The inspection findings Roadmap measure did not identify
who was authorized to certify that RCA had been accomplished. Although SMEs
developed a measure requiring units to report the number of MAJCOM inspection
findings for which RCA was accomplished compared to the number of MAJCOM
inspection findings, the measure did not clearly state who was authorized to
certify that RCA had been accomplished. This created the potential for nuclear
units to respond differently to this measure because authority to certify RCA
completion was not assigned to a specific individual involved in the inspection
process. During the audit, SMEs adjusted this measure to specify commanders as
the sole authority to certify RCA completion. This change should provide senior
leaders with consistent measurement across all units and provide better assurance
that units are implementing Roadmap objectives.

Nuclear Infrastructure. Officials did not effectively plan to collect data needed to
compare the dollar value of Program Objective Memorandum (POM) disconnects

1
Tab A
Nuclear Roadmap Measures

against the total Air Force budget. Although Air Force SMEs developed a
measure requiring units to report the “Number of POM disconnects (unfunded
requirements) in support of nuclear infrastructure,” they did not fully consider the
detailed data needed to resolve ambiguities and to give senior leaders a better
assessment that the projects programmed and planned for a given execution year.
During the audit, management changed this measure to “Number of projects
awarded in the execution year/Number of projects programmed in the execution
year.” With this added context within the measure, senior leadership will have
information needed to make necessary budget adjustments.

Cause. This condition occurred because AF/A10 personnel did not establish a process
to systemically analyze measures and associated data requirements against Roadmap
objectives to determine if the measures were adequate. Instead, AF/A10 officials invited
broad participation by SMEs from Air Staff and MAJCOM organizations with nuclear
missions to develop Roadmap measures and relied upon the expertise of personnel
participating in the process. A systemic process to evaluate the measures helps reduce
the risk that developed measures do not adequately address Roadmap objectives.

Impact. Evaluating measures and data used to report Roadmap progress is critical to
providing an assessment of the health of the nuclear enterprise to the Secretary of the
Air Force, Chief of Staff of the Air Force, and MAJCOM commanders who make up the
Nuclear Oversight Board. These assessments will be the basis for senior leadership
decisions on additional changes to the Air Force nuclear program.

Management Corrective Action. From November 2008 to January 2009, management

corrected the deficiencies noted for all 11 measures identified by audit. Further, AF/A10
officials developed a process for SMEs to analyze data requirements as the measures
were developed. Therefore, this report contains no recommendations addressing these
issues.

Management Comments. AF/A10 concurred with the findings in the report and stated
action taken during the course of the audit should correct the issues identified in the
report.

Evaluation of Management Comments. Management comments addressed the issues

raised in the finding and corrective actions taken during audit were responsive to the
issues and addressed identified deficiencies.

2
 Original Measure Deficiency Identified Corrective Action*
Exhibit 1. Measures Requiring Adjustment. Strategic Objective: The Air Force will implement effective processes for uncovering, analyzing, addressing and reviewing systemic weaknesses.
Number of MAJCOM inspection findings for which RCA Measure did not identify who was Wing commanders were designated
was accomplished/Number of MAJCOM inspection authorized to report completed RCA. as sole authority to certify RCA
findings. completion.
Number of unit-identified discrepancies that include Measure did not specify who was Wing commanders were specifically
RCA/Number of unit-identified discrepancies. authorized to report RCA inclusion. identified as reporting authority.
Number root causes mitigated/Number root causes Measure did not specify who was Same corrective action as above.
identified. authorized to report mitigated root cause.
Number corrective actions closed/Number corrective Measure did not specify who was Same corrective action as above.
actions identified. authorized to close corrective action.
Number nuclear corrective actions and issues viewed in Measure did not evaluate whether units Intent of measure was clarified to
shared database such as Joint Lessons Learned Integration were addressing systemic weaknesses. users; to begin process of uncovering
System (JLLIS)/Number nuclear corrective actions and systemic weaknesses.
issues loaded into shared database such as JLLIS.
Strategic Objective: The Air Force will sufficiently invest in the nuclear deterrence mission arena.
Number of POM disconnects (unfunded requirements) in Measure did not provide a clear assessment Changed measure to compare dollar
support of nuclear infrastructure. of investment in the nuclear mission arena. value of POM disconnects to
Air Force budget.
3

Number of execution year nuclear systems and equipment Measure did not provide context/dollar Changed measure to report dollar
funding adjustments. amount for adjustments. value of adjustments.
Strategic Objective: The Air Force will implement a comprehensive process for ensuring sustained advocacy, focus, and commitment.
Number of strategic communications affirming importance Measure did not identify what could be Clarified measure to only include
of the nuclear mission delivered by Air Force General reported as a strategic communication. communication focused on nuclear

Nuclear Roadmap Measures

Officers. mission approved through public
affairs.
Number of Air Force General Officers who conduct visits Measure did not specify the value of visits Clarified intent measure will
to nuclear capable units. or the office responsible for reporting the establish baseline for Senior Leader
visits. focus on units with nuclear missions;
reported by Protocol offices.
Strategic Objective: The Air Force will implement an end-to-end inter-related, systems, life-cycle nuclear enterprise methodology and discipline.
Number of Nuclear Weapon Related Material^ the Individual units did not have the visibility Air Force Materiel Command
Air Force properly packed/Number of Nuclear Weapon of both ends of a transaction to accurately developed processes to report data
Related Material the Air Force packed. report data. through Air Force-wide systems.
Number of Nuclear Weapon Related Material shipments Same deficiency as above. Same corrective action as above.
correctly delivered/Number of Nuclear Weapon Related

Tab A
Material shipments.
*AF/A10 officials are staffing the revised measures for approval because the coordination process identified areas of additional concern.
^This includes Nuclear Weapon Related Material as defined by the Office of the Secretary of Defense and the Air Force. The Air Force definition is broader and
includes more items.
This Page Intentionally Left Blank
4
 Tab B
Internal Control Plan

BACKGROUND
Internal controls provide reasonable assurance that program objectives will be met. An
internal control plan (ICP) should provide systematic guidance on organization and
individual responsibilities, documentation requirements, data validity certification, key
word definitions, and standard data sources.

In an effort to streamline data collection, review, and reporting processes, the Air Force
planned to use a web-based continuous process improvement management tool called
the Nuclear Enterprise Management Tool to gather required data. A Roadmap ICP is
necessary to provide a consistent set of management controls designed to provide an
unbroken chain of accountability for the web-based nuclear enterprise’s assessment
Roadmap process. The goal is to ensure accuracy, completeness, and integrity of all
information capable of measuring progress made to improve the Air Force nuclear
enterprise through Roadmap action plans. Because Air Force nuclear operations are a
high visibility area, data accuracy, completeness, and integrity are imperative to meet
Roadmap objectives.

AUDIT RESULTS 2 – INTERNAL CONTROL PLAN

Condition. AF/A10 personnel did not develop internal controls necessary to guide
MAJCOMs and wings in consistently collecting data to support the Roadmap assessment
process. Specifically, AF/A10 did not develop an ICP outlining responsibilities for data
validity certification, key word definitions, standard data sources, and documentation
requirements. For example:

Data Validity Certification. The Air Force’s planned use of a web-based data
collection tool for data input posed a risk that unauthorized users could access
and manipulate nuclear measures data.1 An ICP would counter this risk by
limiting access to the Nuclear Enterprise Management Tool to only appoint
MAJCOM points of contact (or their alternate).2 Further, an ICP would require
these points of contact to certify that data collection is complete and was reviewed
for accuracy and completeness. This control provides a reasonable level of
confidence in the accuracy of the data collection process that would not have
existed prior to the ICP development.

1 Although the Nuclear Enterprise Management Tool is hosted on a common access card-enabled server,
the ICP provides guidance on who should be allowed a user account for this tool.

2 This control would also limit data entry capability to appointed points of contact.

5
Tab B
Internal Control Plan

Key Word Definitions. Although AF/A10 personnel attempted to use the most
specific wording possible in developing the Roadmap measures, they had not
defined key terminology. An ICP provides Air Force management a tool to
provide in-depth definitions to units to facilitate consistent data gathering efforts.
To illustrate, one Roadmap measure required units to identify the number of
squadron, group, or wing commanders with nuclear experience. The SMEs
formulating the measures intended nuclear experience to be an individual who
had worked a minimum of 2 of the past 8 years in a nuclear unit. Without an ICP,
each nuclear unit could define the experience requirement differently. This would
create major inconsistencies in data reported by units and could misinform
Air Force senior leadership on the experience level of nuclear squadron, group,
and wing commanders. This control facilitates fact-based decisions related to
future training required to strengthen the nuclear enterprise.

Cause. This condition occurred because AF/A10 personnel had not considered following
proven data collection processes, like those followed for Base Realignment and Closure
data calls.3 Further, they did not consider that an ICP would be a useful management tool
during data collection.

Impact. Sufficiently detailing measures, required data, and the measures’ intent faci-
litates a consistent and accurate picture of the health of each Air Force nuclear unit
reporting to the Secretary of the Air Force, Chief of Staff of the Air Force, and MAJCOM
commanders who make up the Nuclear Oversight Board. Without an ICP, senior leader-
ship could direct unnecessary changes to the nuclear enterprise or, conversely, inaccu-
rately conclude the nuclear enterprise or segments of the nuclear enterprise are effective
based on inaccuracies and inconsistencies in the underlying data.

Management Corrective Action. During January 2009, AF/A10 developed and

coordinated a data collection ICP, based on the Air Force 2005 Base Realignment and
Closure ICP; outlining responsibilities, standard data sources, documentation require-
ments, data validity certification, and key word definitions. The ICP is currently in two-
digit coordination. Therefore, this report contains no recommendations addressing these
issues.

Management Comments. AF/A10 concurred with the findings in the report and stated
action taken during the course of the audit should correct the issues identified in the
report.

3 During the Base Realignment and Closure Process, Air Force managers developed evaluation criteria
and made data calls to obtain information necessary to evaluate which bases should be closed. A critical
element of this process was developing an internal control plan providing guidance to Air Force units on
issues such as key word definitions, standard data sources, and documentation requirements.

6
 Tab B
Internal Control Plan

Evaluation of Management Comments. Management comments addressed the issues

raised in the finding and corrective actions taken during audit were responsive to the
issues and addressed identified deficiencies.

7
This Page Intentionally Left Blank

8
 Audit Scope and
Prior Audit Coverage

AUDIT SCOPE
Audit Coverage. We performed this audit at HQ Air Force, AF/A10 (Appendix II). We
performed the review from October 2008 to January 2009 and issued a draft report to
management in March 2009.

Nuclear Roadmap Measures. To determine whether Air Force officials estab-

lished adequate measures to assess Roadmap implementation status and facilitate
accurate reporting to nuclear oversight bodies, we participated in the Roadmap
measures development meetings in November 2008 and January 2009, held
discussions with AF/A10 officials, analyzed the measures against Roadmap
objectives, and provided input on development of nuclear measures.

Internal Control Plan. To determine whether Air Force officials established

adequate controls over data collection, we reviewed control plans and audit
reports from the Air Force Base Realignment and Closure 2005 data collection
process and applied relevant lessons learned.

Sampling Methodology. We did not use sampling methods or Computer Assisted

Auditing Tools and Techniques to accomplish the objectives of this audit.

Data Reliability. We did not rely on computer-generated data to support conclusions in

this audit.

Auditing Standards. We accomplished this audit work in accordance with generally

accepted government auditing standards and, accordingly, included tests of key internal
controls as considered necessary. Specifically, we reviewed the internal controls
established for the data collection process used to assess Roadmap implementation.

PRIOR AUDIT COVERAGE

We did not identify any Air Force Audit Agency, Inspector General, DoD, or
Government Accountability Office reports issued within the past 5 years that addressed
the same or similar objectives as this audit.

9 Appendix I
This Page Intentionally Left Blank

10
 Locations Audited/
Reports Issued

Installation-Level
Organization/Location Reports Issued

HQ United States Air Force

Assistant Chief of Staff, Strategic Deterrence and Nuclear Integration NONE

(AF/A10)
Pentagon Washington DC

11 Appendix II
This Page Intentionally Left Blank

12
 Points of Contact

Air and Space Operations Division (AFAA/SPO)

Support and Personnel Audits
2509 Kennedy Circle
Brooks City-Base TX 78235-5116

Charles E. Atkinson, Jr., Associate Director

DSN 240-5080
Commercial (210) 536-5080

Terri L. Dilly, Program Manager

Bryce L. Barton, Audit Manager

We accomplished this audit under project number F2009-FD3000-0038.000.

13 Appendix III
This Page Intentionally Left Blank

14
 Final Report Distribution

SAF/OS ACC
SAF/US AETC
SAF/FM AFISRA
SAF/IG AFMA
SAF/LL AFMC
SAF/PA AFOSI
SAF/XC, AF/A6 AFRC
AF/CC AFSOC
AF/CV AFSPC
AF/CVA AMC
AF/A8 ANG
AF/A10 PACAF
AF/RE USAFA
USAFE
Units/Orgs Audited
AU Library
DoD Comptroller
OMB

FREEDOM OF INFORMATION ACT

The disclosure/denial authority prescribed in AFPD 65-3 will make all decisions relative
to the release of this report to the public.

15 Appendix IV
This Page Intentionally Left Blank

16
To request copies of this report or to suggest audit topics

for future audits, contact the Operations Directorate at

(703) 696-7913 (DSN 426-7913) or E-mail to

reports@pentagon.af.mil. Certain government users may

download copies of audit reports from our home page at

www.afaa.hq.af.mil/. Finally, you may mail requests to:

Air Force Audit Agency

Operations Directorate
1126 Air Force Pentagon
Washington DC 20330-1126