AIAA 2010-8860
AIAA SPACE 2010 Conference & Exposition
30 August - 2 September 2010, Anaheim, California
The ABCs of Concept Evolution: A Better-Informed
Materiel Development Decision for USAF Programs
J. R. Loren
SAF/AQRE (Alion Science & Technology), Arlington, VA 22202
The Functional Solutions Analysis (FSA) and the Analysis of Materiel Alternatives (AMA)
no longer exist in either the Joint Capabilities Integration and Development System (JCIDS)
or the Defense Acquisition System governed by Department of Defense Instruction (DoDI)
5000.02. While the intellectual integrity of these analyses is still required in support of preacquisition decisions, namely Materiel Development Decision (MDD) and Milestone (MS) A,
neither the requirements nor the acquisition community has explicit responsibility to either
perform or resource these activities. To address this “content gap,” the Air Force is utilizing
the Concept Characterization and Technical Description (CCTD) to collaboratively
examine, document, and communicate information associated with prospective materiel
solutions developed to address both validated and emerging operational capability needs.
I. Background
Numerous studies and reports by the Government Accountability Office (GAO) (and its predecessor General
Accounting Office), National Research Council (NRC), Center for Naval Analyses, Defense Science Board, and
others, as well as the Weapon System Acquisition Reform Act of 2009 (WSARA), have identified Systems
Engineering (SE) weaknesses as root causes of dramatic cost and schedule growth of acquisition programs.
Programs are often formally initiated without sufficient understanding of the technologies, feasibility, and risks of
the selected solution: technology and technical integration/implementation risks have not been identified, and these
and other risks have not been adequately or objectively addressed before committing to a development approach.
This generally means that programs begin in critical condition, with little if any hope of meeting cost and schedule
expectations for actually delivering the promised capability.
Most recently, “How to fix it” efforts have focused on better linking the JCIDS Capabilities-Based Assessment
(CBA) process to acquisition. Major objectives include:
• Define necessary development planning actions (coupling capability needs, enabling technologies,
and system solutions)
• Develop informed, feasible operational requirements
• Leverage and inform technology planning and investments
• Develop consistent mission-based system-of-systems approaches
• Produce high confidence strategies for timely, effective solutions
In the December 2008 revision of DoDI 5000.02, the Office of the Under Secretary of Defense, Acquisition,
Technology, and Logistics (OUSD(AT&L)) made numerous changes to the early stages of the acquisition system.
Key points included: (1) renaming the entry point from “Concept Decision” to MDD, (2) declaring MDD
mandatory for all programs, (3) renaming the pre-MS A phase from “Concept Refinement” to “Materiel Solution
Analysis (MSA)”, (4) moving the Preliminary Design Review (PDR) prior to MS B, (5) elimination of the AMA,
and (6) directing use of competition and prototyping during the Technology Development (TD) phase. 1
Chairman of the Joint Chiefs Instruction (CJCSI) 3170.01 and the associated JCIDS Manual were revised in
March 2009. As part of this update, the three F-series (Functional Area Analysis, Functional Needs Analysis, and
FSA) were collapsed into the CBA. The scope of the Initial Capabilities Document (ICD) was reduced to only
identify the need for a materiel solution in one of the following categories: (1) an improvement over existing
capability, (2) an IT solution, or (3) a new transformational capability.2,3
*
- Principal Analyst, Life Cycle Systems Engineering Integration; Engineering & Technical Management Branch
(SAF/AQRE); 1550 Crystal Dr., Suite 300, Arlington VA 22202. Senior Member, AIAA
1
American Institute of Aeronautics and Astronautics
Copyright © 2010 by the American Institute of Aeronautics and Astronautics, Inc. The U.S. Government has a royalty-free license to exercise all rights under the copyright claimed herein for Go
WSARA, signed in May 2009, directed reinvigoration of Development Planning (DP) and SE across DoD. It
further identified the Director, Cost Assessment and Program Evaluation (DCAPE) as lead organization for
development of the Analysis of Alternatives (AoA) Study Guidance approved as part of the MDD.
The 2010 Quadrennial Defense Review again identifies the imperative:4
... many systems ... require continuous changes and upgrades -- a challenge that will become only more pressing over
time. The Department will improve how it matches requirements with mature technologies, maintains disciplined systems
engineering approaches, institutionalizes rapid acquisition capabilities, and implements more comprehensive testing. We
must avoid sacrificing cost and schedule for promises of improved performance.
This paper uses the term “concept” to mean “a prospective materiel solution developed to address an identified
capability need.”
II. MDD, the AoA, and MS A: What and Who
A. MDD. The “What” of MDD includes definition of the problem, identification of potential solutions, the
approach to determining the solution set, and an outline of the notional acquisition strategy. The concept
descriptions must clearly be shown to address the stated need, and the approach for the next steps of analysis (e.g.,
the AoA) must align with an understanding of the following factors:
1. What is the problem (defining the need for a material solution)?
2. What are potential solutions (concept characterizations/descriptions)?
3. How will the solution set be determined (AoA planning)?
4. What is the acquisition strategy (executing the MSA phase)?
Air Force Instruction 63-101, Acquisition and Sustainment Life Cycle Management, identifies the purpose of the
MDD review as “ ... substantiating the need for a materiel solution based on a validated capability gap. ... The MDD
review shall ensure that a complete and rigorous analysis/assessment of potential materiel approaches, and their nonmateriel implications, has been or will be conducted.5” Decision support information summarizes pre-MDD
analysis to determine the next steps to address the need. This information includes identification of viable,
affordable solutions to be further examined in the MSA phase.
The MDD principals constitute the “Who”; these key decision-makers -- the Milestone Decision Authority
(MDA), DCAPE, the Service Acquisition Executive (SAE) – along with the requirements community (sponsor;
generally an operational Major Command [MAJCOM]) and the acquisition community (implementing MAJCOM;
provides subject matter expertise in the materiel domain) must consider multiple stakeholder perspectives. These
can include, among others, the Science & Technology (S&T) community, the organization that will manage the
future program, industry, et al.
Sponsor commitment to the next phase, including conduct of the AoA and planning and preparation for
subsequent phases, is an essential component of MDD. In simple terms, MDD is the business case for a material
decision: permission to go forward with deciding on what to acquire in response to the ICD. One of the options can
be to continue investments in relevant S&T efforts to mature critical technologies, and then revisit the capability
need at a later time; another option can be to acquire more of various assets currently in inventory.
A formal Business Case Analysis (BCA) identifies alternatives and presents business, economic, risk, and
technical arguments for selection and implementation. It should help identify likely financial and other business
(non-financial) consequences of executing an investment decision or action. Additionally, a BCA focuses on other
aspects of a proposed investment decision such as the plan for implementing the decision, identifying key
stakeholders, and specifying performance indicators so that the impact of the decision can be tracked over time.
The MDD business case should clearly identify the desired outcome/objective of the review (i.e., MDA
concurrence that a materiel solution is necessary), and key members of the stakeholder community. It should
present the DOTMLPF analysis results, identify relevant non-materiel and S&T recommendations, and provide the
proposed trade space (scope) for the AoA. Each concept being presented should be at a commensurate level of
maturity: a comparable analytical basis covering risk drivers and order-of-magnitude cost information to support the
decision. Initial risk assessments should be made from both the operational (“what happens to the mission without
this capability?”) as well as the more traditional programmatic (cost/schedule/performance) perspectives.
The pressing question is really “How much more analysis must the sponsor present to justify their proposed
investment in a new or upgraded capability?” This represents a major paradigm shift for many stakeholder
communities that have long regarded MDD as the opportunity to get an answer to the question “What new platform
or system are we going to acquire?” And despite DoDI 5000.02 using the term “Materiel Solution Analysis” for the
post-MDD phase, many Air Force stakeholder organizations are coming to regard “Materiel Solutions Analysis” as
a more accurate description of the activities leading up to a MS A decision.
2
American Institute of Aeronautics and Astronautics
B. AoA. In this new context, the AoA report presents a follow-up question as “What could we realistically and
affordably think about acquiring?” This is expressed in terms of the capability need, concepts (prospective materiel
solutions) covering the trade space, and a set of Courses of Action (COA) for the sponsor to consider.
The significance of the AoA is mainly in terms of increased emphasis on Early SE: as many of the GAO and
NRC reports have pointed out, pre-MS A analysis is likely to have the biggest impact. The AoA now effectively
includes the AMA, and in doing so facilitates prioritization of requirements to allow for trade-offs among various
aspects of various technical and technological schemes.
The AoA process starts with a number of options, and for any of a number of reasons often rules out some as
clearly non-responsive to the need. Sometimes the AoA report identifies a “preferred concept”; most frequently,
though, it provides a “+/-” assessment of the remaining ones. In many cases several near-100% answers remain on
the table. These often have different inherent risks, and while the AoA should objectively identify these, it should
not make any outright recommendations. In many cases all the candidate solutions are likely to be viewed as too
expensive, so a proactive AoA should also present some “less filling” (i.e., lower cost, partial capability) options
since the MDA usually wants to understand the incremental cost for that “last 10%” of capability.
AFI 63-101 outlines key COA components; these include military utility assessments, cost-effectiveness
analyses, initial technical baselines, identification of critical technologies, and budget estimates with life cycle
considerations. COAs may have different contracting strategies, incremental development schedules, product
support considerations, or deployment methodologies. A COA cannot be created without an understanding of the
technology and maturity levels needed to provide the new capabilities.6
C. MS A. Ultimately, the sponsor will make the decision on which if any COA to pursue toward a MS A. The
question at this point becomes “What are we going to start down the path to acquiring to address the capability
need?” Decision support information should have specific focus on sponsor commitment to the selected COA for
the TD phase, including planning and preparation for subsequent phases; this information will necessarily feed the
Technology Development Strategy (TDS) and the Systems Engineering Plan (SEP) submittals in support of the
ensuing MS A decision. The TDS, essentially the acquisition strategy for the TD phase, must address the
competition and prototyping approach for critical technologies, including alternatives or “off-ramps” that permit
incremental delivery of the desired capability. The 2008 NRC report makes a key point:7
While today‟s systems are not necessarily more complex internally than those of 30 years ago, their “external complexity”
often is greater, because today‟s systems are more likely to try to meet many diverse and sometimes contradictory
requirements from multiple users. This kind of complexity can often lead to requirements being changed (after)
Milestone B … relying on immature technology.
III. Bridging the Gap: DP and Early SE Processes and Products
A. DP Relationship to CBP. JCIDS CBP addresses the entire DOTMLPF (Doctrine, Organization, Training,
Materiel, Leadership and Education, Personnel, Facilities) continuum of prospective solutions for the entire life
cycle. Air Force Instruction (AFI) 10-601, Operational Capability Requirements Development, identifies the
fundamental purpose: “JCIDS … validates warfighting capability needs while considering the full range of materiel
and non-materiel solutions. Operational capabilities must be defined within the „art of the possible‟ and grounded
within real world constraints of time, technology, and affordability. 8” Representatives of the materiel (acquisition)
community at the table earlier in the requirements development process will assist in tradeoffs by informing the
sponsor and decision-makers on feasibility, achievability, and affordability aspects that might otherwise receive little
if any consideration. Further, they can advise on incremental acquisition strategies to achieve partial early
operational capability: 100% solutions are often too costly, take too long to deliver, or entail excessive risk to attain
desired performance levels. AFI 10-601 details aspects of the CBA.9
The CBA forms the analytic basis for operational capability requirements development … defines the capability required
and any capability gaps/shortfalls identified during the conduct of the assessment. The CBA consists of the following
activities: analyzing what is required for the warfighter across all functional areas to accomplish the mission … ,
comparing the capability needs to the capabilities provided by any existing or planned systems (gap analysis), and
identifying associated gaps/shortfalls and/or redundancies. The final step of the CBA is to offer recommendations on
whether the gaps/shortfalls can be addressed by non-materiel means, materiel means, or both.
In the emerging AF construct, DP is regarded as the materiel contribution to CBP – in other words, when the
focus of the CBA begins to narrow to a materiel solution, the acquisition community begins to participate more
actively. DP should help focus the discussion on this question in terms of technical and technological risks
associated with closing an operational capability gap. It is essential that this collaborative engagement be in place
well before the ICD is validated.
3
American Institute of Aeronautics and Astronautics
The greatest “leverage” in terms of setting a future program on an executable path occurs before MS A, and the
preponderance of that leverage occurs prior to MDD. Well-executed DP provides for a more completely and
comprehensively informed MDD; the up-front preparation ensures robust options are available to support preacquisition investment decisions that address operational capability needs.
B. Early SE Relationship to DP. In the context that DP is much more than engineering and technical analyses, SE
can be viewed as the principal enabler for the technical aspects of DP. However, there are things that make
application of SE processes and practices unique in this environment. The traditional “core” SE processes -- in the
context of acquisition programs -- do not address the sort of JCIDS analyses that support identification of a
capability need. The program office manages, and the contractor executes, SE to transform product and system
specification requirements into hardware/software solutions, but neither entity has much if any impact on the
technical content of the requirements that were generally established well before program launch. The resultant
rework during the acquisition life cycle is often denounced in the same breath as “requirements creep”; it is of
particular concern in the high-visibility period that encompasses source selection and the early years of new
programs.10,11
Application of early SE supports realistic program formulation through robust and disciplined up-front technical
planning. It develops a solid technical foundation for the future program, and ideally should reduce the chances of
poorly planned concepts emerging from AoA with relatively high rankings. Early SE produces concepts that track to
operational and functional (sustainability, reliability, producibility, etc.) needs.
C. Concept Characterization and Technical Description (CCTD). The Air Force has established the CCTD as
the primary artifact of Early SE. It captures data and information associated with a concept – a prospective materiel
solution – or family of related concepts analyzed during the DOTMLPF analysis. This enables the analysis to
produce increasing levels of detail regarding the concept(s) under consideration, and should provide high-quality
decision support information about the feasibility of prospective solutions prior to initiating an acquisition program.
Principal elements of the CCTD, and associated key content, are described below.12
1. Mission/Capability Need Statement/CONOPS. This section may be taken directly from the ICD or validated
warfighter gap/shortfall that the concept is addressing; as such, it is not concept-specific but gap/shortfall-specific.
Engagement between concept developers and sponsors during the CBA is valuable in terms of ensuring mutual
understanding of expectations. Information from a Concept of Operations (CONOPS), whether pre-existing or
developed as part of the CBA, should be referenced or included here. This section can include the mission tasks
under consideration and the capability needs statement, and should describe in general terms how the concept
addresses the capability need.
2. Concept Overview (OV-1). The overview should identify systems that the concept is expected to operate
and/or interface with, information flows (i.e., the OV-2), the physical operating environment, and threats the
concept will encounter. It should take into consideration what is known about the concept‟s relationship with
other systems, Families of Systems (FoS), or Systems of Systems (SoS). Additional text can describe how the
OV-1 “frames” the operational concept.
3. Trade Space Characterization. Trade space considerations can include initial Measures of Military Utility
(MOMU), operational measures, environment, ownership costs, user limitations, manpower/personnel/training
impacts on manning and skill levels of anticipated system operators and maintainers, etc.
4. Evaluation (Studies, Analyses, Experiments). This section contains the analytical data and decision history that
represent a concept‟s technical “pedigree,” including study findings and results of any prior analyses or
experiments accomplished. It should also identify shortfalls of these efforts that will need to be addressed in
future evaluations.
5. Concept Characterization/Design. The studies, analyses, and experiments identified in Section 4 can continue
to provide additional details of the concept design. For concepts (or a family of related concepts) based on
existing systems, this information is fairly well known at the early stages of development. On the other hand, for
prospective materiel solutions that represent entirely new approaches to a capability, it will generally be at a highlevel view.
6. Program Characterization. This section describes the efforts envisioned to develop, test and evaluate,
manufacture, and sustain the materiel concept should it ultimately be selected to become an acquisition program.
7. Risk Assessment and Decision-Certain Consequences. In the earliest phases, the focus is on top-level
operational risks (e.g., threat analysis, mission completion, etc.) associated with not having the desired capability
at the desired time; the next evaluations generally examine the impact of technology shortfalls, i.e., only partially
achieving a MOE. The sponsor must have a thorough understanding of these risks in order to make an informed
recommendation to decision makers.
4
American Institute of Aeronautics and Astronautics
8. DOT_LPF Implications and other Interdependencies. Beyond the basic DOT_LPF (i.e., non-materiel) factors,
there can be other significant interdependencies such as training; security; infrastructure; data protection, transfer,
and storage; and the like.
9. Conclusions (Capability Description/Traceability to Need Statement). This section provides a description of
the capability and conclusions made about the overall concept, and should make clear what is currently known
about the concept, what is not known (known unknowns), and what additional work is recommended to provide
necessary information at future reviews and decisions.
D. Visualizations. Figure 1 depicts the Early SE “V” diagram, and shows how the Trade Space Characterization
phase occurs in concert with inputs from the operational community, the S&T community, and other sources. It also
shows the outputs as decision support information for pre-acquisition reviews.
Figure 1. Early SE “V” Diagram Showing Linkages to Pre-Acquisition Processes and Events.
Concept developers must not regard the “V” as a monolithic end-to-end process. As they start looking at the
entire trade space (the blue oval at the top), they need to ask questions about critical considerations such as
affordability, technology maturation timelines, adequacy of the industrial base, changes in engineering parameters
that push today‟s state of the art, and the like. This is a pre-emptive “check” to think about issues that may arise
during Implementation Analysis phase, particularly the Verification Assessment and Programmatic Analyses subphases; it should help narrow the field of thousands of brainstorming and unconstrained ideas to a point where the
design teams can begin to flesh out the most promising. These are represented by the green oval in the middle; for
discussion purposes, consider the “survival rate” to be 10%. Again, critical questions associated with actualizing
these efforts are likely to result in a decision to focus further development on only the most viable 10%. So the red
oval at the bottom – real concept DESIGN work – represents only about the top 1% of the prospects that emerged
from the early discussions of the capability need.
5
American Institute of Aeronautics and Astronautics
By turning the figure on its left side, the three ovals can be seen as a venturi with its narrow end in the red oval;
this represents narrowing the field of possibilities from thousands to hundreds to tens. Envision a mirror-image
venturi extending to the right edge of the figure; this represents growing the knowledge base associated with a select
few evolving concepts as information is developed in support of the early acquisition decision points. Figure 2 is an
update of the “venturi” depiction in the author‟s 2009 paper; the “filtering” questions and issues could include, but
are certainly not limited to, the following that were advanced in the same paper.13
Figure 2. “Venturi” View of Concept Evolution: Narrowing the Field, Expanding the Knowledge Base.
•
•
•
•
•
•
•
•
•
•
•
•
•
Military Worth – The AoA study team, led by the operational MAJCOM sponsor, conducts
Military Utility Analyses on each alternative.
Timeliness – Can the concept deliver envisioned capability by the user‟s desired IOC date?
Affordability – Life cycle cost must be addressed in order-of-magnitude terms in initial
discussions; if all prospects have the same number of zeroes, cost should not be a determining
factor at this point.
SoS/Architecture Implications – If all approaches rely on the same interfacing systems in the
same way to achieve the same objectives, they can be evaluated on an “apples-to-apples” basis.
Doctrine – Will updates and/or changes to current doctrine help resolve issues?
Vulnerability – Have known issues (i.e., problems emerging late on other programs) surfaced?
Politics – Congressional or international interests can be unique.
Policy – Have other Department or Agency stakeholder (e.g., State, Commerce, FAA) interests
been considered?
Technology – Have the creative thinkers appeared to rely too much on things humorously called
“unobtanium” or “unaffordium” or “impossibilium” or “miraculum” in concept exploration?
Environmental – Have known issues (i.e., problems emerging late on other programs) surfaced?
Data Density & Throughput – Do things like sensor data fusion and user training scenarios take
bandwidth and processing into account?
Legacy Systems Transition – How will current capability be managed as new products and
systems come into inventory?
Other Enablers – Difficulties have arisen from neglect of issues in areas such as intelligence,
industrial base, infrastructure, training, communications, etc.
6
American Institute of Aeronautics and Astronautics
In an attempt to simplify the thinking behind this new paradigm, some AF advocates are beginning to describe a
principal activity during the Trade Space Exploration and Trade Space Refinement/Concept Exploration sub-phases
as the identification and documentation of key “ABCDE” elements associated with the concepts: Assumptions,
Boundaries, Constraints, Dependencies, Enablers.
Assumptions shape the early discussions – in simplest terms, these are the “big rock” questions about the
overall solution space (e.g., Hardware or software? Aircraft or space-based?). Top-down direction is often the source
of answers that shape many if not all of the later decisions.
Boundaries and Constraints are sometimes regarded as the same thing – external ground rules that keep the
discussion in a particular vein or lane. They can be physical, stemming from an assumption (e.g., a concept for a
manned aircraft cannot propose exceeding 9G in any part of the flight regime, and all flight crew station equipment
must meet crashworthiness criteria); they can also be operational, with a basis in any of a number of other domains
(e.g., a directed energy [DE] concept must acknowledge that the Laws of Armed Conflict prohibit use of lethal DE
against personnel). Again, top-down direction can establish these ground rules for discussions among the operational
users, the concept developers, and other key stakeholders.
Dependencies and Enablers often emerge late, and often in an “oh-by-the-way” manner – questions about the
industrial base, the basing and support infrastructure, the timing of software cycles, etc. as alluded to in the above
list of “filtering” questions. Additionally, impacts on interfacing processes and/or systems, such as configuration
management at the SoS level or intelligence data can be better understood if the issues are articulated early.
E. Objectives. Comprehensive CCTDs will frame AoA study plans focused on integrated risk assessments. The
intent is to ensure that pre-acquisition risks associated with concepts are at a commensurate level of detail going
forward. The initial discussion occurs at the Air Force Review Board (AFRB) which occurs prior to MDD for major
programs (where AT&L is the MDA); for lower-level programs with an AF MDA, the AFRB serves as the MDD.
Assessment of the relative technical merit of concepts will remain in the AoA.
Updated technical and technical management processes will enhance requirements analysis and maturation by
use of the CCTD. Concepts must be assessed in terms of how well the “SE Homework” is progressing in terms of
addressing stated customer/user needs; CCTDs become the foundation for individual system and program
Development Plans, another key component of buying down risk.
A forthcoming change to AFI 63-1201, Life Cycle Systems Engineering, identifies some of the interrelationships
among the processes and products:14
When tailored for the technical aspects of DP, the SE and SoSE processes ... are referred to as “Early SE,” and are key
mechanisms for accomplishing these activities prior to MDD and MS A. Concept developers (organizations and/or
teams) and sponsors will collaborate in developing and populating the ... CCTD to capture decision-quality Early SE
information about concepts (prospective materiel solutions) prior to the MDD and the AoA.
Concept developers‟ preliminary assessments support discussions with the requirements community sponsor/
customer during the CBA. These discussions can help determine whether further work is desired on a particular
approach, or whether it might be worth combining elements of several different approaches: Have we covered the
right part of the trade space? Are we really going down a path with a reasonable chance of success?
Continuous analyses during the capabilities/requirements/acquisition life cycle, beginning with requirements
determination and persisting throughout the life of the development effort, feeds into the emerging program. It is
essential that concept developers use consistent definitions, structures, ground rules, assumptions, and levels of
detail for each instance of the capability-based analytic processes. As stated earlier, pre-MDD and pre-MS A are
where quality analyses are likely to have the biggest impact: uncertainties of the scenarios considered; risks of
operational failure; complexity of the assessment; resources at stake in the solution; fielding uncertainty in terms of
cost, schedule, and performance of potential solutions; etc. A more robust and considered approach to technology
transition, that addresses evaluation of technologies and acceleration of their transition to acquisition programs (and
thence to actual operational use), should help address the aforementioned concern in the NRC study about post-MS
B requirements changes due to immature and high-risk technologies being part of the initial program baseline.
7
American Institute of Aeronautics and Astronautics
IV. Beyond the Air Force
OSD is considering adopting aspects of the AF Early SE methodology DoD-wide as part of the response to
WSARA direction to implement DP.
DoDI 5000.02 discusses MDD:15 “The MDA, working with appropriate stakeholders, shall determine whether
there is sufficient information to proceed with a Materiel Development Decision. … MDD is the formal entry point
into the acquisition process and shall be mandatory for all programs.” The Instruction is silent on the definition of
“sufficient”; however, forthcoming DDR&E policy will identify information needs associated with decision support
for MDD.16
The Milestone Decision Authority (MDA) must not only decide whether an investment shall be made to fill the capability
gap, but also determine the fundamental path the materiel development will follow. Decisions must be based on effective
development planning and a strong technical foundation. To support those decisions, the DoD Components shall provide
evidence at the MDD Review that will facilitate the determination that:
a. The candidate materiel solution approaches have the potential to successfully address the capability gap(s)
and associated dependencies, and desired operational attributes.
b. There exists a range of technically feasible solutions generated from across the entire solution space, as
demonstrated through early prototypes, models, or data.
c. Consideration has been given to near-term opportunities to provide a more rapid interim response to the
capability need.
d. The plan to staff and fund analytic, engineering, and programmatic activities supports the proposed
milestone entry requirements.
… DDR&E … shall serve as a standing participant and technical advisor in the development of AoA Study Guidance and
on the AoA Study Advisory Group ... to facilitate the consideration of technology and engineering risks for the
alternatives under consideration.
Again, the February 2010 QDR Report outlines the DoD vision of the future of the acquisition system:17
... requirements for new systems are too often set at the far limit of current technological boundaries. Such ambition can
sometimes help produce breakthrough developments ... But far too often the result is disappointing initial performance
followed by chronic cost and schedule overruns. ... our system of defining requirements and developing capability too
often encourages reliance on overly optimistic cost estimates. In order for the Pentagon to produce weapons systems
efficiently, it is critical to have budget stability -- but it is impossible to attain such stability in DoD‟s modernization
budgets if we continue to underestimate the cost of such systems from the start. We must demand cost, schedule, and
performance realism in our acquisition process ... We cannot afford everything we might desire; therefore ... must balance
capability portfolios to better align with budget constraints and operational needs, based on priorities assigned to
warfighter capabilities.
V. The Road Ahead
The AF has begun to institutionalize DP resources for pre-systems acquisition planning and early SE. The
process hinges on the MAJCOMs as advocates for operational capability needs to provide reinforced acquirer-user
collaboration. Structured processes are in place to ensure that DP investments are leveraged into numerous preMDD activities such as requirements refinement, market research, budget estimates, concept definition development
and trade studies, modeling and simulation, and technology assessments.
The first AF program to go through MDD – T-X Advanced Pilot Training System of Systems – used a
comprehensive CCTD to inform sponsor discussions with DCAPE and other OSD offices. The product had strong
credibility within the MAJCOM; concept development team members provided a coherent approach that addressed
myriad stakeholder issues associated with operational requirements, proposed contractual requirements, budget
inputs, resource requirements, and numerous other issues.
CCTD review is now identified as an AFRB criterion in AFI 63-101; the need for this concept technical
sufficiency assessment should further emphasize the need to drive Early SE discipline into the collaborative
CBP/CBA processes. It should also ensure that the prospective materiel solutions coming forward to MDD have
considered the “realm of the possible” – i.e., appropriately scoped the trade space.
The key to robust pre-acquisition planning in concept development organizations is the use of rigorous SE-like
processes in translating the sponsor‟s capability needs into materiel concepts. Early collaboration can set mutual
expectations about manageable and executable requirements: the operational and technical communities can better
define what future assets are envisioned to do, and in the process reduce or eliminate disconnects during the
transition from “gleams in the eye” to future products and systems. This should yield concepts that track to
operational and functional (sustainability, reliability, producibility, etc.) needs. Successive iterations should reduce
risk and structure programs that are inherently less “inefficient” in EMD and later phases.
8
American Institute of Aeronautics and Astronautics
Conclusion
Rigorously executed DP and Early SE, and the robust products -- principally the CCTD -- of these efforts, will
provide better and more complete decision support information at the MDD and the AoA for prospective Air Force
programs. These new AF processes are also under consideration for DoD-wide instantiation. Expectations are that
immature technologies and unrealistic performance expectations will influence fewer pre-acquisition decisions.
While fiscal realities continue to shrink the number of new acquisition program starts, more of those actually
initiated will begin with higher likelihood of delivering the needed capability within cost and schedule expectations.
Both the Air Force and DoD will have better decision-quality technical information prior to committing resources to
new acquisition programs, which will help address the GAO issue of a balanced investment approach:18
… encouragement within the acquisition environment of overly ambitious and lengthy product developments that embody
too many technical unknowns and not enough knowledge about the performance and production risks they entail. The
knowledge gaps are largely the result of a lack of early and disciplined (SE) analysis … prior to beginning system
development. (SE) translates customer needs into specific product requirements for which requisite technological,
software, engineering, and production capabilities can be identified through requirements analysis, design, and testing.
Early (SE) provides the knowledge … to identify and resolve performance and resource gaps before product development
begins by either reducing requirements, deferring them to the future, or increasing the estimated cost … Because the
government often does not perform the proper up-front requirements analysis to determine whether the program will meet
its needs, significant contract cost increases … occur as the scope of the requirements changes or becomes better
understood by the government and contractor.
The future state will be characterized by better-informed decisions, better-crafted solicitations and requirements,
and lower-risk programs for the next generations of weapon systems.
References
1
Department of Defense (DoD) Instruction 5000.02, “Operation of The Defense Acquisition System,”
URL: http://www.dtic.mil/whs/directives/corres/pdf/500002p.pdf [cited 22 July 2010].
2
Chairman of the Joint Chiefs Instruction CJCSI 3170.01G, “Joint Capabilities Integration and Development
System” 1 March 2009, URL:
https://acc.dau.mil/GetAttachment.aspx?id=267681&pname=file&aid=43839&lang=en-US [cited 22 July 2010]
3
Joint Chiefs of Staff JCIDS Manual, February 2009, URL: https://www.intelink.gov/wiki/JCIDS_Manual.
4
Quadrennial Defense Review Report, Office of the Secretary of Defense, Washington, D.C., 1 February 2010,
pp.xiv-xv.
5
AFI 63-101, “Acquisition and Sustainment Life Cycle Management,” incorporating through Change 2, 16 June
2010, URL: http://www.af.mil/shared/media/epubs/AFI63-1201.pdf [cited 25 July 2010].
6
Ibid.
7
National Research Council of the National Academies, “Pre-Milestone A Systems Engineering: A Retrospective
Review and Benefits for Future Air Force Systems Acquisition,” The National Academies Press, Washington, D.C.,
December 2007, URL: http://books.nap.edu/catalog.php?record_id=12065 [cited 22 July 2010].
8
Air Force Instruction (AFI) 10-601, “Operational Capability Requirements Development,” 12 July 2010,
URL: http://www.af.mil/shared/media/epubs/AFI10-601.pdf [cited 25 July 2010].
9
Ibid.
10
Loren, J. and Bullard, R., “Do It Right, Do It Early; Do It Early, Do It Right: Reshaping the DoD SE Investment
Paradigm,” INCOSE 2008-168, INCOSE International Symposium, June 2008.
11
Bullard, R. and Loren, J., “Systems Thinking and Technical Planning: The Roadmap for Air Force Systems
Engineering in the 21st Century,” INCOSE 2008-326, INCOSE International Symposium, June 2008.
12
SAF/AQR Concept Characterization and Technical Description (CCTD) Guide, August 2010.
13
Loren, J. “Early Systems Thinking and Technical Analysis: Linking JCIDS and Acquisition,” AIAA-2009-6686,
AIAA “Space 2009,” September 2009.
14
AFI 63-1201, “Life Cycle Systems Engineering,” 23 July 2007, URL:
http://www.af.mil/shared/media/epubs/AFI63-1201.pdf [cited 25 July 2010].
15
DoDI 5000.02, op. cit.
16
Directive-Type Memorandum 10-017, Development Planning to Inform Materiel Development Decision (MDD)
Reviews and Support Analyses of Alternatives (AoA), DRAFT, Aug 2010
17
QDR Report, op. cit., p. 76.
18
“DEFENSE ACQUISITIONS: DOD Must Prioritize Its Weapon System Acquisitions and Balance Them with
Available Resources.” GAO-09-501T, Government Accountability Office, Washington, D.C., March 18, 2009.
9
American Institute of Aeronautics and Astronautics