A strategic framework for program managers to improve
Command and Control system interoperability
by
Steven E. Frey
B.S. Mechanical Engineering
Renesselaer Polytechnic Institute, 1991
Submitted to the System Design and Management Program
in Partial Fulfillment of the Requirements for the Degree of
Master of Science in Engineering and Business Management
at the
Massachusetts Institute of Technology
May 2002
The author hereby grants MIT permission to reproduce and distribute publicly paper and electronic
copies of this thesis document in whole or in part.
Signature of Author
Steven FIry
System Design and Management Program
Certified by
Daniel Whitney
Thesis Supervisor
Center for Technology, Policy & Development
Accepted by
Steven D. Eppinger
Co-Director, LFM/SDM
,-
Co-Director, CIPD
M4FM Professor of Managemeat Science and Engineering Systems
Accepted by
Paul A. Lagace
Co-Director, LFM/SDM
Professor of Aeronautics & Astronautics and Engineering System
MASSACHUSETTS INSTITUTE
OF TECHNOLOGY
AUG 0 1 2002
LIBRARIES
BARKER
MITLibraries
Document
Services
Room 14-0551
77 Massachusetts Avenue
Cambridge, MA 02139
Ph: 617.253.2800
Email: docs@mit.edu
http://Iibraries.mit.edu/docs
DISCLAIMER OF QUALITY
Due to the condition of the original material, there are unavoidable
flaws in this reproduction. We have made every effort possible to
provide you with the best copy available. If you are dissatisfied with
this product and find it unusable, please contact Document Services as
soon as possible.
Thank you.
* The Archives copy contains poor quality image reproductions.
The images contained in this document are of
the best quality available.
A strategic framework for program managers to improve
Command and Control system interoperability
by
Steven E. Frey
Submitted to the System Design and Management Program
in Partial Fulfillment of Requirements for the Degree of Masters of Science
in Engineering and Management
Abstract
An emphasis on fielding interoperable Command and Control systems .to support military
operations increases the burden on system program mangers to define and coordinate interfaces
with external partners. This thesis studies barriers to increasing the interoperability of C2
systems presented by system complexity and acquisition culture and makes recommendations to
assist program and enterprise managers in reducing and overcoming these barriers.
Architectural analyses of C2 systems using Design Structure Matrices (DSM) to model the
internal and external information flows were developed. The number of interfaces defined by
the system specification and represented in the DSM was found to be an excellent indicator of
the risk for successful integration based on C2 system development case studies. The DSM was
then used to evaluate the level of control a program office has over interfaces which were part of
the architecture. This analysis revealed only 34% of interfaces were under the direct control of a
case study C2 system and provides a map of interface risks which need to be actively managed.
Additionally the DSM suggests the type of mechanisms/relationships/agreements that should be
put in place to manage the interfaces based on differing levels of management influence and
control. Further analysis using the DSM allows the program office to develop isolation layers
and information hiding strategies to reduce the complexity of system development activities and
reduce the disruption caused by technology refresh of highly inter-dependent sub-systems.
I
Over 73 interviews were conducted with program managers, system developers and users to
develop the DSM, confirm analysis findings, and identify perceived barriers to C2 systems
development. Additionally, a questionnaire was developed and circulated to investigate the
beliefs and mental models held throughout the acquisition community. The thesis goes on to
explore the acquisition culture and mental models held by personnel involved in C2 systems
acquisition and how the culture creates barriers to improving interoperability. Aggressive
decision making on program requirements, technical specifications, and high risk schedules,
coupled with a lack of accountability and a poor reputation/relationship with the user community
are significant problems. Building a culture which recognizes and rewards excellence in
delivering interoperable systems to military warfighters should be a goal for the C2 acquisition
community. Creating awards for achievement in interoperability excellence based on the model
of Malcom Baldridge Quality Award is a first step. Increasing accountability and reducing
program manager turn-over during system development cycles can be accomplished through
pursuit of 'micro-development projects' which are small focused developments using less than 6
developers and delivering finished products in less than 6 months. The limited scope of these
projects has the added benefit of reducing complexity. Finally, greater involvement and
interaction with the user community is recommended to identify and leverage lead user
innovations and deliver desired capabilities back to the them as part of the formal systems.
Using the recommendations and tools developed in this thesis as a template, C2 programs can
generate common simple DSM representations of system interfaces and leverage them for risk
management, interoperability metrics and reporting, system design decisions and management of
technology transition efforts. Furthermore, the incentives and micro project approach can help
reduce the cycle time for delivering tailored capabilities to end users while minimizing the
learning curve of the developers. Overall the thesis presents recommendations which are
designed to stimulate the shift in thinking and practice required to meet demands of improved
interoperability.
Thesis Supervisor:
Daniel Whitney
Senior Research Scientist
Center for Technology, Policy & Development
2
Acknowledgements
As I reflect on what it took to complete this thesis I realized the number of people who
directly or indirectly helped me was tremendous. There are too many to list individually, but I
feel compelled to mention some of the folks who made it possible for me to do this work. The
experience was definitely one of the most challenging during my time at MIT.
At the top of the list is Dan Whitney whose wisdom, encouragement and constant challenges
to explore ideas and dive deeper into the subject made the thesis a better product. Dan's
guidance really helped bring me through the process. Thanks Dan.
To the LAI office at MIT, and Hugh McManus in particular, for their support and guidance
during my first two semesters at school and for making the transition back to school possible. To
the U.S. Air Force and the outstanding military members who are enduring hardships to protect
and defend our nation, a tremendous amount of what I learned came from the community of C2
systems developers and active duty military members who are directly involved with the
systems. To everyone who took the time to explain the issues and problems they faced and share
their thoughts and perspectives on the systems, Thank you. And a special thanks to Col Al
Baker, who is the finest individual I've had the privilege of working for in my career. He more
than anyone, helped to put me in a position to embark on the 2 year journey through the System
Design and Management Program. Thank You Sir!
Many of my friends and co-workers were involved and in brainstorming and acting as a
sounding board to develop and refine ideas. Rich Hubbard, Pet Robson, Deb Schuh, Russ
Graves, Marco Serra, Art Faint, Mike Ripley, Dave Humphrey, and many others let me try out
ideas and gave me sanity checks as I went along. They also picked up the slack at work and
pushed me along as the thesis progressed. Thanks to all and the many others there just isn't
enough room to recognize by name.
To my Mom, Dad, brothers and sister who are a constant source of encouragement and
support. You helped make me the person I am and the courage and ambition to strive to become
a better person every day. Thank you for giving me such a strong start in life, and the security of
knowing there's always someone there for me.
And last, but definitely not least, to my wife Lynnette, daughter Brittney, and son Kelly, for
their understanding and support throughout the process. I owe you a deep and heartfelt thank
you. You shared in the sacrifice required to complete the degree and now we can look forward
to sharing in the rewards. I was often pre-occupied with this thesis and didn't have the time I
wanted to devote to you and what's really important in my life. Thanks for your love, patient,
understanding, and support.
3
Table of Contents
A bstract..............................................................................................................................
I
A cknow ledgem ents ......................................................................................................
3
Table of Contents ........................................................................................................
4
List of Figures....................................................................................................................
6
List of Tables .....................................................................................................................
6
Introduction.......................................................................................................................
7
Problem Statement ......................................................................................................................................
Research M ethodology ................................................................................................................................
Background .....................................................................................................................
7
8
10
. .. . .. .. .. . .. .. .. . .. .. . .. . .. .. .. .. .. .. .. . . .
What are Command and Control systems?.................................................
10
Operational Aspects of C2.........................................................................................................................12
What is interoperability and why is it important?........................................... .. .. .. . .. .. .. . .. .. . .. .. . .. .. .. .. .. .. . . 16
How C2 Systems are developed and fielded .........................................................................................
19
Research & Case Studies................................................................................................
27
Case Study of an AOC System ..................................................................................................................
27
Defense Information Infrastructure Common Operating Environment (DII COE)...........................38
M odernized Intelligence Database (M IDB).....................................................................................
40
United States M essage Text Format (USM TF).................................................................................
42
Joint M apping Toolkit (JM TK).............................................................................................................44
Targeting and W eaponeering (TW )...................................................................................................
Time Critical Targeting (TCT)..............................................................................................................48
Situational Awareness and Assessmnet (SAA)................................................................................
Execution M anagement (EM ) ...............................................................................................................
Case Study Summary ................................................................................................................................
Barriers to Enhancing Interoperability....................................................................
Complexity ................................................................................................................................................
Architectural representations.................................................................................................................54
Stability & technology turnover .......................................................................................................
Emergence.............................................................................................................................................62
The testing treadmill ..............................................................................................................................
Effects in a closed system......................................................................................................................68
Summary ...............................................................................................................................................
Culture .............................................................................................
.........................................................
Incentives reward aggressiveness.....................................................................................................
Turn-over eliminates accountability.................................................................................................
45
50
51
52
54
54
58
65
69
70
70
74
4
View from the user/customer's perspective ......................................................................................
Summary ...............................................................................................................................................
81
84
Recom m endations...........................................................................................................85
DSM s and Enterprise Architectures ......................................................................................................
Introducing enterprise interoperability metrics.....................................................................................
Collaboration for enterprise decisions..................................................................................................
Interfaces, isolation layers, and information hiding .............................................................................
Incentives...................................................................................................................................................93
Creating accountability..............................................................................................................................94
85
85
88
90
Conclusions......................................................................................................................
97
Future W ork....................................................................................................................
99
A ppendix A - Q uestionnaire........................................................................................
101
Bibliography ..................................................................................................................
106
5
List of Figures
Figure 1 - Hierarchy of C2 Enterprise ..............................................................................
Figure 2 - C2 Nodes......................................................................................................
Figure 3 - Current DMPI Process.................................................................................
Figure 4 - Air Force Task List 7 for Command and Control ........................................
Figure 5 - DoD System Acquisition Process....................................................................
Figure 6 - C2 Enterprise Integration Management Model.............................................
Figure 7 - CAOC Node Operational View....................................................................
Figure 8 - New DoD Systems Acquisition Process.......................................................
Figure 9 - AOC System Code Growth .............................................................................
Figure 10 - AOC Process and Organizational Divisions...............................................
Figure 11 - Basic AOC Process, Divisions, and Data Flows ........................................
Figure 12 - Joint Technical Architecture......................................................................
Figure 13 - Interfaces between Mission Applications..................................................
Figure 14 - AOC System Data Flow Diagram .................................................................
Figure 15 - Simple DSM Example...................................................................................
Figure 16 - DSM of System Database Flow ....................................................................
Figure 17 - History of Targeting Application Development........................................
Figure 18 - Results from survey question .....................................................................
Figure 19 - AOC System Data Flow Diagram.................................................................
Figure 20 - Results from survey question ........................................................................
Figure 21 - DSM of System Database Flow .....................................................................
Figure 22 - Problem Resolution Rates .............................................................................
Figure 23 - Cost Reporting Under PM 1......................................................................
Figure 24 - Cost Reporting Under PM 2..........................................................................
Figure 25 - Cost Reporting Under PM 3..........................................................................
Figure 26 - Cost Reporting Under PM 4..........................................................................
Figure 27 - Comparison with and without rebaselining...............................................
11
15
17
20
21
23
24
26
29
30
31
33
34
36
37
37
47
55
56
59
61
67
76
77
78
79
80
Figure 28 - Evolution of the programming system product [4] ..................
82
Figure 29 - Operational Feedback Delay ......................................................................
Figure 30 - DSM for Interoperability Metrics..................................................................
83
87
Figure 31 - DMPI Flow .................................................................................................
88
Figure
..........................................................................
Figure 33 - DSM Interface Trace......................................................................................
Figure 34 - Introduction of TW-MIDB Design Rule ....................................................
89
92
92
List of Tables
Table 1 - DII COE Software Lines of Code..................................................................
Table 2 - Cost Reporting Definitions ...........................................................................
40
75
6
Introduction
Problem Statement
Program managers at the Electronic Systems Center (ESC) are responsible for
acquiring Command and Control systems for the Air Force and Joint military services.
Individually these Command and Control (C2) systems are complex, software intensive
collections of functions, equipment and processes used by military commanders to
maintain awareness of the battlefield situation, arrive at decisions, direct actions, and see
their orders are carried out. The collective set of C2 systems interoperating forms the
nervous system of our nation's military muscle coordinating the actions of troops,
weaponry and intelligence sensors. One of the main goals for C2 systems is to improve
interoperability by allowing them to provide dynamic interactive information and data
exchange for planning, coordination, integration, and execution of a commander's
mission.
Historically, the acquisition of C2 systems has treated them as 'stovepipes', where
individual systems are developed by separate program offices managing their own budget
and maximizing performance for their user's environment. By themselves, these systems
have a track record of cost over-runs, schedule slips, and trouble meeting performance
goals. Increasing the emphasis on interoperability increases the inter-dependencies
among multiple systems and the overall complexity of the systems themselves. Program
managers will have to coordinate between independently managed programs on issues of
schedule, interface conventions, specifications, configuration management, integrated
testing, etc... Program managers and system program offices will lose a degree of
autonomy and must work together to negotiate trade-offs and make group decisions based
on technical input from multiple domains. Already complex system development efforts
will have the additional challenge of interoperating with other programs and systems in
the C2 domain. Interoperability increases the likelihood that changes in individual
systems will cause cascades or ripples throughout the system of systems. This presents a
new acquisition management challenge which the current culture and processes have had
limited success handling.
7
A shift in thinking and practice will be required by the acquisition community to meet
the demands of improved interoperability. Better tools and methods are needed for
defining interdependencies between C2 systems, for creating shared boundary objects,
representations and nomenclatures, for anticipating and minimizing downstream impact
of changes or upgrades, for negotiating trade-offs and implementation decisions between
programs, and for allocating tasks and costs to support interoperability requirements
among individual systems. Achieving improved interoperability in a C2 System of
Systems will require an acquisition approach which takes these issues and others into
account.
This thesis attempts to answer the following question:
Given the system of systems paradigm, what are the barriers to improving
interoperability and what methods, tools and technologies can be used to
improve the efficiency and outcomes of individual program deliveries?
Research Methodology
"We have a habit in writing articlespublished in scientificjournalsto
make the work asfinished as possible, to cover up all the tracks, to not
worry about the blind alleys or describe how you had the wrong ideafirst,
and so on. So there isn't any place to publish, in a dignified manner, what
you actually did in order to get to do the work."
Richard Feynman, American physicist, Nobel Lecture, 1966.
Research supporting this thesis was conducted over a one year period and relies
heavily on direct interaction with command and control program office staff,
development contractors, and operational users. The thesis started with an interest in
collaborative decision making. A research question was proposed, and investigation into
how collaborative decision making could be applied in the C2 domain began. As the
research progressed, we found that the cost, schedule and performance of C2 systems
under development were often affected by decisions made by external systems
developers. The focus of the thesis then shifted to understanding decision making in a
system of systems development environment and what methods, tools and technologies
could be utilized to improve the outcomes of decisions. As the research continued, the
goal of increasing interoperability between C2 systems and the inter-dependencies it
creates in the system of systems emerged as an issue program managers need to
8
understand and manage more effectively. Ultimately, the thesis settled on the topic of
identifying barriers to improving interoperability and methods, tools and technologies
needed by program managers to improve the performance of individual programs.
The thesis research included; 1 - literature reviews of C2 program documents,
acquisition regulations and directives, software development practices, complexity
management, and collaborative decision making, 2 - case studies of C2 programs to
include analysis of architectures using Design Structure Matrices, and review of cost
performance reporting, 3 -73 personal interviews with program office personnel,
development contractors, and operational users, 4 - analysis of data collected through a
questionnaire on the beliefs and mental models held throughout the acquisition
community, and 5 - prior personal experience as a military officer and current experience
supporting defense acquisitions. The research findings identified barriers to improving
interoperability in the areas of acquisition culture, system complexity, and program
management practices. Finally, approaches to overcoming the identified barriers and sets
of recommendations were developed.
9
Background
What are Command and Control systems?
Command and Control (C2) systems are complex, software intensive collections of
functions, equipment and processes used by military commanders to maintain awareness
of the battlefield situation, arrive at decisions, direct actions, and see that their orders are
carried out. The collective set of C2 systems interoperating forms the nervous system of
our nation's military muscle coordinating the actions of troops, weaponry and
intelligence sensors. In essence these are decision support systems which provide data
and automation services to commanders and their staffs. They include computers,
communications networks, sensors, input/output, displays, storage databases, and
business logic designed to provide the "right information, at the right time, in the right
place, in the right format" [I] to support a quick decision.
The hierarchy for Command and Control systems starts with the C2 Enterprise Figure
1. The Enterprise is made up of C2 nodes which are collections of C2 systems. Each of
the C2 systems is an integrated set of mission applications, The mission applications
themselves are made up of software modules or components. Operational users employ
mission applications to perform tasks. The applications retrieve, manipulate, display,
generate and store data using a common infrastructure. Interactions and dependencies
between mission applications and software modules are not depicted in the hierarchy for
clarity, but are an important part of C2 systems.
10
Figure 1 - Hierarchy of C2 Enterprise
The goal of an integrated and interoperable C2 Enterprise Integration is to allow
people and machines to seamlessly exchange information and transact business across the
enterprise. One of the most important aspects of the C2 system is the ability to transmit
tasking orders and control military forces. Air Force Instruction 10-207 provides the
following definitions:
Command and Control-The exercise of authority and direction by a
properly designated commander over assigned forces in the
accomplishment of the mission. Command and control functions are
performed through an arrangement of personnel, equipment,
communications, facilities, and procedures employed by a commander in
planning, directing, coordinating, and controlling forces and operations in
the accomplishment of the mission.
Command and Control System-The facilities, equipment,
communications, procedures, and personnel essential to a commander for
planning, directing, and controlling operations of assigned forces pursuant
to the mission assigned.
The concentrations of systems which make up a C2 node are generally located in
facilities like command posts and operation centers. Few people have an appreciation for
the diversity of systems which make up a modem command post, the functions they
perform and the information they deliver unless they have been exposed to the systems in
11
use during operations. A partial list of C2 systems similar to what would have been used
to control the Air War for operations in Kosovo and Afghanistan is instructive.
" Space Battle Management Core System - provides satellite track data and
analysis tools (what's orbiting in space, and what it can do)
" Tactical Intelligence Broadcast System - provides track data to generate a theater
air picture (what's flying in the air, if its friendly or hostile, status and intentions)
" Worldwide Origin Threat System - provides launch warning for Theater Missile
Defense
- Theater Battle Management Core System - used for Air Tasking Order
production and execution (plans for, monitors and directs air operations)
" Theater Weather Server - provides theater weather information to support
operations
" Time Critical Target Functionality - provides capabilities required to find and
direct attacks on time critical targets (what targets are emerging on the battlefield,
how important are they, and what's available to engage them)
Each of these systems provides unique data and functional capability to allow an
operational commander to direct and control the war effort.
Operational Aspects of C2
Central to the Air Force's ability to effectively employ combat power is the ability to
react to changing circumstances and direct a wide variety of activities, equipment, and
personnel across geographic boundaries. Prior to the explosion in computer and network
technology C2 systems were far less complex. Up through the 1980s wall mounted
grease boards, file cabinets, and standard forms were the primary methods used to store
and track essential information with decisions and orders being transmitted via
messenger, radio, field phone, or formal message. Advances in computers,
communications and networking throughout the 1990s allow commanders to store,
retrieve and operate on data faster and in more complex ways. This gave rise to an
explosion of C2 systems which support individual commanders needs. These systems are
often referred to as 'stovepipes' because they did not communicate to other systems or
inject information into other data channels.
12
The stovepipes created and distributed data and information more rapidly, but the
interfaces between systems were more or less non- existent. Human beings would print
out and re-enter data to move it from one system to the next. As the systems evolved,
individuals began to refine their data needs and define standards for how information
should be produced so that it could be re-used. An early example of data standards
emerging is United States Message Text Format (USMTF) which is a set of highly
formatted messages. These messages capture and transmit essential information to
commanders over secure channels. The highly formatted nature of the messages allows
machines to 'read' the massage and automatically transfer or parse the message
information into mission applications or databases. Efforts to parse the USMTF
messages into 'stovepiped' systems had mixed success because these messages are
difficult to work with, and small human errors in formatting often corrupt the import and
users are forced to re-enter the data by hand. The overall trend among C2 systems has
been the need/desire for more automated interchanges of data (a.k.a. increased
interoperability).
From an operational perspective, modem command posts and operation centers are
where the action is, and commanders are under constant time pressure to make decisions
and direct activities. Decision deadlines are generally dictated by the operational
situation. Consequently, decisions are made as early as possible to keep things moving,
or based on the best information available when a deadline is reached. For instance,
suppose an intelligence tip off comes in that a meeting between key leaders of the
adversary forces is taking place and will disband in the next 2 hours. The meeting
subject is reported to be planning for a weapons of mass destruction (Chemical,
Biological, or Nuclear) attack. This scenario would characterize the meeting location as
a Time Sensitive Target and a fleeting opportunity. To capitalize on the opportunity, the
operational commander needs to know if the tip off is reliable, if the facility where the
meeting is being held is vulnerable to attack, if a suitable military assets is available and
can reach the target in time, if there are collateral damage risks in the target area, what
the risks to friendly forces are, what the opportunity costs of diverting forces are. Then
13
the commander must be able to issue orders. All questions must be resolved, a decision
made, orders issues and received, and forces positioned within the 2 hour timeline.
This stressing and very specialized type of military operation draws on C2 systems to
orchestrate activities and has its own specialized set of terms and language. How the
commander and his battle staff think about and resolve the problem is to draw on
information from battlefield sensors, intelligence databases, and operational databases.
They determine which weapons have a capability against the target, check the availability
and status of the weapons, and check the availability of support assets required. They
review the current plans, determine if retasking the assets is advantageous, and if so they
provide authentication for dynamic retasking. For this scenario an air component
commander might use the Space Battle Management Core System to check the status of
space based sensors and the Global Positioning System constellation needed to support
the engagement, the Tactical Intelligence Broadcast System to maintain situational
awareness of real time air tracks and threats, Worldwide Origin Threat System to monitor
for missile launches, the Theater Battle Management Core System to analyze threats and
task assets to form a strike package, the Theater Weather Server to provide climatological
data for mission planning and predictions of sensors ability to acquire targets, and the
Time Critical Target Functionality to task intelligence sensor, positively identify the
target, determine the target priority in relation to other pre-planned targets, and determine
an optimal weapon target pairing.
Without integrated C2 systems the information needed to make a decision can not be
assimilated in time and the commander is forced to make a judgment call under greater
uncertainty. The opportunity may be lost. With an appropriately integrated C2 systems
providing critical information in a timely, useable format the commander has an option
and can capitalize on the opportunity.
The set of C2 nodes, each with a unique configuration of C2 systems, that a
commander may want to draw on is illustrated in Figure 2. The air component
14
U
I
I I
-
nrr
commander is located at the node labeled CAOC (Combined Air Operations Center) and
draws on information from other nodes and the ring of assets through the
communications infrastructure available to the CAOC.
Figure 2 - C2 Nodes
These C2 systems must support a commander's information and control needs across a
broad spectrum of conflict from relatively benign peacetime situations, to the highly
dynamic unstructured situations encountered during war. The ultimate objective of the
C2 system is toproduce unity of effort by allowing input of multiple commanders and
key warfighters to be brought to bear on a given task, exploit totalforce capabilities by
allowing individuals to be effective during high-tempo operations while under great
stress, properly position critical information and quickly present requested information
where it is needed, and achieve information fusion by reducing information to the
minimum essentials and putting it in a form that people can act on.
The available C2 systems have the capability to completely inundate a commander
with data. Commander's don't need mountains of data, they need specific relevant pieces
15
of information to support decisions. In a tactical situation it would be far more valuable
to know the time and place of a leadership meeting planning the use of weapons of mass
destruction, than to have a mountain of data about the leadership personalities, weapons
development programs, facilities, technical capabilities and researchers who built the
weapons. The point is that increasing system interoperability and facilitating the rapid
exchange and transmission of data does not necessarily create value for the commander.
Decisions about what information not to provide and under what conditions to actively
suppress information may be even more important to an effective C2 system than the
capability to generate a comprehensive data set.
What is interoperability and why is it important?
Interoperability of C2 systems is a key enabler needed to unify the efforts of military
forces, achieve higher military effectiveness, exploit and coordinate the capabilities
available to a commander. When interoperability works it is transparent. When
interoperability is broken there are often multiple systems performing the same or similar
functions to ensure the data is available within the system and information is often
manually entered and processes multiple times. Users are forced to find work-arounds to
gather and input the data they need to do their jobs. They accept voice data, hand written
notes, faxes, emails, hardcopy documents... and must rekey information into their
system. Not only is this inefficient, but it is error prone.
For example, many Precision Guided Munitions (PGMs) require accurate target
coordinates to guide the bomb to the target. The coordinates are needed in
Latitude/Longitude/Elevation are Degrees, Minutees, Seconds format down to
milliseconds (DD MM SS.SSS) and appear as:
022 44 37.456 N 122 33 27.123 W, 126.4 ft MSL
This represents a precise point in space that the weapon should fly to and is known as
a Desired Mean Point of Impact (DMPI). The current process to get the DMPI from its
16
point of origin to weapon's guidance seeker is a good example of poor interoperability.
Multiple systems, multiple steps, and human intervention are involved in moving the
DMPI through the system. The process starts with a precision point being derived from
imagery at an external intelligence organization. The information is annotated onto a
reference image and the annotated image is sent to an Air Operations Center. Within the
Air Operations Center the precise point is manually entered into a database for use in
creating an Air Tasking Order. At another position within the Air Operations Center the
DMPI record is retrieved from the database and transferred to an Excel spreadsheet to
send it to an operational unit to initiate mission planning. At the operational unit the
DMPI is manually entered into an imagery exploitation system to verify that it is correct,
and visually compared with the original reference image. The DMPI is then manually
entered into a Mission Planning System where it is exported to a data transfer device to
program the aircraft and weapon guidance seeker. The steps in the process are illustrated
in the Figure 3 below.
Figre Annotated Image
AOC
Product
Unit Mission
Planning Cell
y-
-
MPS
-
WW I
Figure 3 - Current DMPI Process
17
The process involves four manual entries of the same 22 character lat/long string and
must be done exactly right each time. A typing or transposition error in the seconds digit
(DD MM S .SSS) represents 100ft on the ground. Imagine the tedium and potential for
error while generating 80 individual DMPIs for a single B2 mission. If we assume every
DMPI takes 60 seconds to type and verify during a manual process, then there exists the
potential to save 5 hrs and 20 minutes of effort for every B2 mission. Enhancing system
interoperability can have significant operational payoffs in accuracy, efficiency and
timeliness.
For interoperability to work in this example, every system along the chain must agree
on data formats, geodetic coordinate conventions, units of measure, communications
protocols... When approached from a systems perspective, interoperability is primarily a
technical question. Are there physical or electronic connections between the systems?
Are the communications protocols compatible? Are there shared data conventions or
conversion utilities? These are technical interoperability issues, but interoperability
doesn't stop with a technical implementation. Operational commanders must have the
authority and capability to task or request support from other nodes and systems. They
must have processes and coordination measures in place to take advantage of the
information and services being provided. Hence there is an aspect of operational
interoperability which is defined as:
The ability of systems, units, orforces to provide services to and accept
servicesfrom other systems, units, orforces and to use the services so
exchanged to enable them to operate effectively together.
[21
True interoperability requires both technical and operational aspects to be addressed
and achieving interoperability is more than a systems design and engineering effort. It
requires an interaction between operational processes and technology designed to assist
or automate those processes. Many times there is a trade-off which must be negotiated to
modify an operational process to take advantage of available technology in addition to
the systems engineering and design work to apply the technology to the operational
domain.
18
How C2 Systems are developed and fielded
Historically, the acquisition of C2 systems has treated them as 'stovepipes', were
individual systems are developed by separate program offices to maximize performance
for a particular user's environment. Little consideration was given to interoperability (as
highlighted by the PGM example). Individual system program offices pursued capability
upgrades and migration to the latest technologies based on their cost, schedule and
performance goals. Interdependencies and inter-relationships between programs in the
C2 domain exist, but are not under the direct control of the individual program managers.
Part of the reason for the 'stovepiped' nature of the development had to do with the
requirements, budgeting, and program definition processes. Requirements for a system
start with operational user needs and shortfalls of the current systems and process used to
perform an assigned mission. The missions are based on universal set of high level Air
Force Tasks. Commanders develop Mission Essential Tasks Lists which they are
responsible for, and develop procedures to accomplish those mission essential tasks. An
example of Air Force Tasks for Command and Control is given in Figure 4.
19
[3]
Figure 4 - Air Force Task List 7 for Command and Control
These needs and shortfalls for performing Mission Essential Tasks are documented in
a Mission Needs Statement. New concepts for performing the mission and applying
technology to improve the mission are explored and result in recommendations for either
material solutions (systems), or non-material solutions (new processes or additional
manpower). The need for a material solution spawns an Operational Requirements
Document (ORD) which is intended to state user requirements in performance-based
operational terms (what the system must do) rather than specifications or engineering
design goals (how the capability is achieved). The ORDs are critical document because
they are used to establish programs and budgets in the System Program Offices (SPO).
The ORD is the reason for building a system.
Once an ORD has been reviewed, coordinated, and validated, a SPO is assigned the
responsibility to develop a system which meets the operational requirement. The SPO
develops a Program Objective Memorandum (POM) which is a 5 year projection of what
20
it should cost build, field, and support a system. The POM submission goes into the DoD
Planning, Programming, and Budgeting System and ultimately budget authority is
delegated down to the SPO. Then the SPO has a 5 year spending plan to build a system
which meets the user requirements set out in the ORD. At this point there is a huge
knowledge gap between user needs and potential systems solutions, but the direction and
commitment to build a system is made. The System Program Office generates Requests
for Proposals, commercial firms submit proposals, a source selection board reviews and
chooses the most promising proposal and a program is launched.
New ORDs and program starts are rare occurrences and are long term commitments to
meeting a continuing user need. Therefore cancellation of a program would require
another program to be stood up in its place. Additionally, the process which delegates
budget authority down to programs places restrictions on how the money can be spent.
The program offices are legally obliged to spend budget dollars on the ORD requirements
which justified the budget. When changes to another system are required to improve
interoperability, the other system has little incentive to spend its budget, and transferring
funds between programs to pay for work is a gray area which is likely to incur scrutiny
from auditors. The process for system acquisition as it was defined in the Federal
Acquisition Regulation DoD 5000 series is shown in Figure 5 below. Milestones (MS 0,
I, II, III) are points at which decisions to continue, redefine, or kill a program are made.
The actual program launch starts at MS I, a full system design based on risk reduction
prototyping in ready at MS II, and by MS III a fully tested and certified system is ready
for fielding. Long term sustainment, operations and maintenance of the system follow.
MS 0
MS I
MS II
1& 14 4+
MS III
Figure 5 - DoD System Acquisition Process
21
C2 systems are significantly different than most weapon system acquisitions. Like
many product development projects, users do not always know what they want until they
see it, therefore the requirements are incomplete and prone to change as the system
matures. To complicate matters, the technology life-cycle for IT is much shorter than
most technology and as users are exposed to the latest advances in commercial IT, they
expect to see similar advances in fielded C2 systems. This leads to further requirements
instability. C2 systems require the integration of many independently developed software
modules, many on independently managed delivery schedules, where changes and
upgrades to versions are almost continuous, and module to module version compatibility
must be established and tested. The challenge of getting all the module versions to align
for an integrated delivery is a significant challenge. The likelihood that they will all work
together the first time is slim. Add the probability of a late delivery or deferred piece of
functionality in a module version, and the likelihood of successful system integration and
delivery starts to look vanishingly small.
To help cope with the rapidly changing environment the Air Force established a new
approach to acquisitions for C2 systems in April 2000. Air Force Instruction 63-123,
Evolutionary Acquisition For C2 Systems, treats C2 systems as one integrated weapon
system, and creates a more flexible acquisition strategy based on spiral development to
adapt to evolving requirements and shorter technology life-cycles. Under the new C2
systems acquisition strategy incremental deliveries of capability are packaged in spirals
and requirements and system design evolve based on user feedback and testing results.
The goal is to deliver spiral increments to the field in less than 18 months realizing that
they will be an "80% solutions" and will be enhanced in subsequent deliveries.
The April 2000 policy change at the Air Force level filtered down to the Electronic
Systems Center and direction for organizational changes to support the Evolutionary
Acquisition approach began approximately 1 year later. In Jun 2001 the Designated
Acquisition Commander (DAC) for the C2 Enterprise released Directive 001 which
introduced the concept of managing C2 Nodes. In Oct 2001 DAC Enterprise Directive
22
003 established C2 Node Responsibilities and created an initial set of 72 C2 nodes
grouped into 10 functional areas. The new multi tiered acquisition management structure
defined by the DAC Directives mirrors the operational employment of the systems. The
tiers represent a logical decomposition of responsibilities starting with the establishment
of a C2 Enterprise Integration commander. Below the enterprise level are a series of C2
nodes and node managers, and below the nodes are individual C2 programs and program
managers. At the lowest level C2 programs are composed of individual software
segments and projects which are either contracted for development or purchased as
Commercial Off-The-Shelf (COTS) software. An overview graphic of the C2 enterprise
management structure is presented in Figure 6 below.
Command-level
- Policy
- Direction
Tier-O
EIM-level
* Plan, Budget
* Architect
- Assess, Analyze
}..Tier-1
Execution-level
- Plan, Budget, Architect
- Integrate, Test, Certify
- Deliver, Sustain
}..Tier-2
Execution-level
* Plan, Budget
- Design
- Acquire
Figure 6 - C2 Enterprise Integration Management Model
A year after the organizational changes, individual system program offices are starting
to overcome the resistance to change and grappling with what it means to do spiral
development. To realize the benefits of C2 enterprise management and improve
23
interoperability, cooperation and collaboration will have to occur between managers at
the C2 node and program levels and these relationships are still forming.
As an example, lets consider the scope of the C2 node known as the Combined Air
Operations Center (CAOC). The node manager has responsibility for planning, delivery,
testing, fielding and sustainment of a certified 'weapon system' designated the AN/USO163 Falconer (a.k.a. the CAOC), which is itself a system-of-systems. The 142 systems
which make up the CAOC node are either commercially purchased and integrated, or
developed and delivered by individual program managers. The CAOC node manager has
primary responsibility for ensuring intra-nodal interoperability between the component
systems. Additionally, inter-nodal interoperability issues must be resolved by mutual
agreement between interfacing node managers. Figure 7 shows the inter-nodal interfaces
which must be resolved by mutual agreement.
Figure 7 - CAOC Node Operational View
Program Managers (PM) are responsible for acquiring the systems, subsystems,
components, and/or services utilized by the C2 Nodes. These systems are themselves
often large and complex with many mission applications developed by sub-contractors
24
and delivered to a prime or integrating contractor to assemble and test. The integrating
contractor is often tasked with the responsibility of supply chain management for 3 rd
party deliveries, schedule coordination, integration and testing which adds to the overall
management complexity of delivering a certified program to a node manager. Cost
growth, schedule slips, discrepancies and rework on third 3 rd party deliveries can directly
affect the critical path of the larger program. In some cases the
3rd
party developers have
more than one customer, and the integrating contractors and program managers have
limited leverage over the sub-contractors to meet their schedule or performance
requirements.
Accommodating shorter technology life-cycles and evolving requirements in this
process is an ongoing challenge for program managers. Greater flexibility in the
acquisition strategy for funding, and incremental deliveries using a spiral development
process are enablers for program managers to meet these challenges. The DoD revised
the Federal Acquisition Regulation DoD 5000, to include a new acquisition process
model. See Figure 8 below. Based on this model a commercial or experimental
technology can be inserted at Milestone A, B, C, or directly into operations depending on
the maturity and suitability to the mission area. Additionally, future block releases are
shown to the lower left which indicates evolutionary development. Ultimately, the
development and fielding of C2 systems follows a relatively simple path from mission
needs, to operational requirements definition, to system development and testing, to
fielding and operational use.
25
1C
C
AB
Sustainment
Systems Acquisition
Pre-Systems
(Engineering and Manufacturing
Development Demonstration, LRIP
Production)
=
&
Acquisition
FOC
ORD
MNS
1 bM
lhoy
A Weureet
LCK3
Relationship to Requirements Proces
Figure 8 - New DoD Systems Acquisition Process
The strategic and organizational changes for C2 systems acquisition are well
underway. The acquisitions community is now entering the period where new tools, and
methodologies to deal with the additional complexity required by interoperability must be
introduced. New relationships and processes must be forged to negotiate changes
between inter-dependent systems. And finally, incentives should be developed to
encourage cultural changes better suited to the new paradigms of interoperability and
spiral development.
26
Research & Case Studies
The research and case study analysis for this thesis centered on C2 systems developed
for use in an Air Operations Center (AOC). The case study description draws on a
review of program documentation, cost performance reporting, acquisition regulations
and directives, software development practices, and personal interviews with program
office personnel, development contractors, and operational users. Analysis of system
architectures using Design Structure Matrices pointed to a set of 'high risk' sub-systems
and mission application which are dealt with in greater detail to provide background on
how issues with development and system integration can create barriers to
interoperability.
Case Study of an AOC System
The case study of an AOC System and its relationship to a number of sub-systems is
not meant to be exhaustive. Instead the case focuses on program management and
interoperability aspects from an internal and external standpoint. The analysis starts with
an overview of chronological progression of the program development and then moves
into AOC process and systems details. The AOC system being studied was not a new or
radical concept, but was a migration and merging of functionality contained in three
legacy systems. The legacy systems were 'stove-pipes' that did not interchange
information, and the effort to combine them into one common system was an attempt to
break down the walls of the existing stove-pipes. The requirements and budgets of the
legacy systems were merged with some additions, and analysis of the AOC processes
were conducted.
The program was approved in Oct 1995 with software development starting in Dec 95.
The initial program schedule was 36 months with a $95M dollar budget to deliver a
version 1.0 in Oct 98. The estimates for code development costs were based on the total
System Lines of Code (SLOC) from the legacy system with minor adjustments. The
original baseline was estimated at 181,422 SLOC. Total SLOC divided by a productivity
factor of 69 Lines of Code/Staff Month (LOC/SM) for difficult software was used to
determine 2,629 staff months of development effort would be required. Assuming a
27
$12k/staff month rate the software development estimate was for -$31.5M with the
remaining $63.5M covering hardware, software licenses, overhead, configuration
management, training, documentation, overhead... The underlying assumption was that
the majority of the code from the legacy systems could be re-written and integrated into a
single system. The staffing profile was to ramp up to 100 software developers and ramp
down or shift staff to a version 2.0 project as version 1.0 approached test and delivery.
By Dec 99 the total SLOC had grown to 1,142,000 over 6 times the original estimate,
the staffing level had grown to 350 developers, and version 1.0 still had not delivered
(see Figure 9). Interviews and written correspondence from program office staff
attributed the code growth to an underestimation of the complexity to develop both the
infrastructure and mission applications software required for the system. This initial
underestimate of the complexity resulted in the need for increased staffing levels, and
extended the development schedule to complete the system. A rough projection based on
the 1,142,000 SLOC in Dec 99 would require 16,550 staff months of development effort
and $198M in software development costs to complete. This is consistent with the overall
staffing profile and costs actually incurred to complete the system. In the end it took 65
months to deliver version 1.0 to the field and cost at completion was -$260.1 M. It
should also be noted that many of the original requirements were omitted to save time,
money, and actually deliver a functioning system.
28
AOC System Code Growth
1,200,000
1,000,000
800,000
0
600,000
400,000
200,000
0
May-96
Dec-96
Jun-97
Jul-98
Jan-98
Feb-99
Aug-99
Mar-00
Date
Applications
---
Infrastructure
-A7- Total
Figure 9 - AOC System Code Growth
The AOC system was developed to provide automated decision support tools to
improve planning, preparation, and execution of joint air combat operations and was
designed to operate in a distributed fashion linking together a number of C2 nodes. The
process which the AOC system was designed to automate is shown in Figure 10. The
personnel who use the system are organized into four functional divisions: Intelligence,
Strategy, Combat Plans, and Combat Operations. The AOC process is largely cyclical
starting with the Strategy division establishing the major objectives for an air war and
the
selecting targets for attack to meet those objectives. The Strategy division documents
objectives and targets in an Air Operations Directive (AOD) and hands it to the Combat
Plans division were resources (fighter, bomber, tanker aircraft... ) are assigned to attack
the targets. The Combat Plans division documents their resource assignments in an Air
Battle Plan (ABP) and hands it to the Combat Operations division for execution. The
Combat Operations division creates an Air Tasking Order (ATO) which it distributes to
air forces and this becomes the basis for pilots to plan their combat missions. The
29
Combat Operations division then monitors the mission execution, manages changes in a
dynamic manner as execution progresses, and provides reporting of the results and Battle
Damage Assessment for the day's operations. Reporting and analysis from operations
are fed back into the Strategy division and the process continues until the objectives are
met and the war is ended. While the basic process appears almost serial, the reality is
that activities are running in parallel with Strategy working on the war two days from
now, Plans working on the war for tomorrow, and operations working on today's war. In
short, 3 plans are in parallel development at different stages of maturity at any given time
within the AOC. Intelligence is considered a supporting function and is integrated into
all three divisions simultaneously.
Strategy
Division
Combat
Plans
Intelligence
Combat
Ops
Figure 10 - AOC Process and Organizational Divisions
The information flows between divisions are better represented by Figure 11 which
shows information products being exchanged between divisions internal to the AOC and
external organizations tasked to fly combat missions or to collect and deliver intelligence
30
9
information. The more complete process starts with Step 1 when raw intelligence
information is delivered to the AOC. Step 2 has the intelligence division generating a
requests for information (RFIs) to determine the status of adversary forces. The RFI is
converted by an external organization where a collection manager tasks battlefield
sensors and other sources to collect the required information for step 3. Step 1 completes
the I" loop with information to answer to the RFI delivered to the intelligence division
for exploitation. In Step 4 a tailored intelligence product describing the current
battlefield situation is passed to the strategy division to initiate strategy development.
This often results in more detailed intelligence questions, so Step 5 is another RFI
generated, which runs through the loop to collect, analyze, and build a tailored product
for the strategy division. The process continues with strategy feeding plans, plans
feeding operations, operations tasking external combat assests at the flying units, and
intelligence supporting the information needs of all three divisions simultaneously.
4*
7*
AOC
Non-AOC
Figure 11 - Basic AOC Process, Divisions, and Data Flows
31
1.
Intelligence & Sensor information
2.
3.
Intel RFIs
Sensor Tasking (Collection Requirements)
Tailored Intel support to Strat
4.
7.
Strat RFIs
Finished Strategy Information (AOD)
Tailored Intel support to Plans
8.
Plans RFIs
9.
Finished Plans Information (ABP)
10.
11.
Operations RFIs
Tailored Intel support to Operations
12.
Sensor Operations Coordination
Operations Execution Coordination
Execution Tasking (ATO)
5.
6.
13.
14.
Creating a software system to automate and support the information flows between
organizational divisions within the AOC and external organizations like the intelligence
community and flying units was the objective of the AOC system development. The
system was based on a client-server configuration with multiple client workstations
accessing shared database servers. The architecture for the system software was to be
based on the Joint Technical Architecture which is a layered software architecture (see
Figure 12) consisting of Mission Applications, Application Services, and Infrastructure
Services. Interfaces between mission applications and databases were to be provided by
the application services layer. This was meant to provide isolation and decouple
databases and mission applications thereby allowing greater flexibility in modifying the
system. The AOC system was decomposed into 13 mission application areas which are;
" Air Campaign Planning (ACP)
" Airspace Deconfliction (AD)
" Defensive Planning (DP)
" Execution Management (EM)
"
Intelligence Data Management (IDM)
"
Spectrum Management (SM)
32
0
Resource Management (RM)
" Situational Awareness and Assessment (SAA)
" Targeting and Weaponeering (TW)
=
Theater Air Planning (TAP)
" Threat Evaluation (TE)
" Time Critical Targeting (TCT)
-
Weather (Wx)
Mission Applications
User Interffice. Application Logic
Application Services
Data Access Agents, Alerts, Message Services, etc.
Infrastructure Services
Operating System, D11 COE Kemel, Databases
Figure 12 - Joint Technical Architecture
For reasons which are not entirely clear, the Joint Technical Architecture was violated
early in the system design. Interviewees attribute this to a desire to start development
quickly and show progress. Since many of the specifics of the data interfaces had not
been defined, it was easier to establish direct interfaces between mission applications and
other components of the system and work out the specifics as they went. In this way, one
to one interfaces could be built and modified without the need to redesign the application
services layer in addition to the application and infrastructure layers. An example of an
interface between two mission applications as described in System/Segment Specification
text is
33
"EM - RM Changes to Mission Tasking Data. There shall be an internal
interfacefrom EM to RM to provide mission re-tasking, mission tasking
changes and support data related to that tasking."
No diagrams were prepared showing the interfaces between mission applications and
system components in the specification. The diagram in Figure 13 was generated for this
thesis from the details in the system specification. The diagram maps the relationship of
the software segments to one another and shows where they would be used within the
organizational structure of the AOC. The mission application interfaces bear little
resemblance to the AOC process shown in Figure 11. Looking at the information flows
acoss the division boundaries it becomes apparent the Intelligence systems are highly
interfaced, with all the other divisions.
Strateg y
Combat Operations
Combat Plans
AD
Cs,
Dl
?R
EM
TAP
AC P
Wx
Intelligence
TEB
TW
TCT
IDM
SAA
-
Specified Internal Interfaces
Figure 13 - Interfaces between Mission Applications
As system development began, a number of pictures were created to help staff
describe and understand the system. The early architecture diagrams lacked detail
needed to understand the system interfaces and reason about it to make decisions. The
System Version Requirements Document (SVRD) for version 1.0 had a total of
34
approximately 1012 documented requirements of which 472 (or 47%) are related to
mission application functionality that provide users with features important to completing
C2 tasks. The remainder of requirements were common support functions like disk
access, system administration, networking and communications, etc...needed to build a
working system. Of the 472, approximately 193 (or 41%) dealt with sharing information
between C2 nodes or other mission applications. Examples of mission application
functional requirements which supported interoperability are:
"The ATO/ACO tool shall provide the capability to receive the ATO and
the ACO messages at the unit and remote sites via an external interface"
and
"TW shall provide the capability to transferpost strike reconnaissance
requirements to collection managers."
To assist in understanding the complex relationships which were specified, the
program management office created their own architecture representations. The products
used by the program managers to guide the development of C2 system were primarily
power point diagrams. Figure 14 below was used extensively by program managers over
a three year period. Elements of the Joint Technical Architecture are present in the
diagrams, but the layered nature of the architecture is not evident. In the diagram,
cylinders represent databases, rectangles represent mission applications, ovals represent
application services, the areas with dashed lines are external interfaces, and hexagons are
externally developed hardware systems.
35
--- I>Ii--
Joint
r
AOWSANOWN
FrOB
UNIT
CRC/ICRE
A
ACO
Feed-
Aispace.
Scheduling.
TOC
TNL
eed
];.a F;Wd
Scramble
&
MAGTFC4]
or secwe
isso1
WA
stara
c4J
ATO
P,
TNL
CTL
CAM
ABP
(GTACS)
TargeA
Data
I.
IJNI'TAFORMS:
B&
Remote Termina
AC-
ATO.
AC
DataC
* Secue
WAN
ABP. FrO
Ai space
>
>
ATO
ACO
AMTWINt
ARP
GCCS-A
Secure WAN
:Note ASAS
AIP. FrOB, Airspace
C5
FrO
Weallher
rck
I(UBo
UB)
:USM
AC
Track
V
ABP
Slatus
BCtI
Daa
2
mage(all
XD
:El~ecuxe-WN
---
---
TADIL A, B, J
NATO LJnk I
TDDS
TOPS
-
- ----
HUAV Weather Feed
(AWN, AWDS,
-
Track
--- -
o Secure WAN) C2P- GCCS--A
--- - -- ----- - ----- ---- -- -- - ----- -- -- - --- ---- -I--
-
-STARS
ATDSI
Sie
LEGEND
M
Database
AFATDS
Applicalion
oDataMessage
CST
AFATDS:
-- :
- -- --- ---
Service
Exlenal Interface
Externally Developed
Figure 14- AOC System Data Flow Diagram
A Design Structure Matrix (DSM) was generated for the thesis to perform additional
analysis. Further details on DSMs can be found on line at http://web.mit.edu/dsm. To
build a DSM, the process steps or system components are used to populate the row and
column headings of a square matrix. In this adaptation of a DSM, a square matrix is
populated with system components, one per row/column and ones marked in the matrix
denote an interface or data flow between two components. Reading across the rows
allows a developer to see all the inputs to the component. Reading down the column
allows the developer see all the outputs from a component. Figure 15 is an example of a
simple DSM with three components A, B, and C. A Has an output to B and an input
from C; B has an input from A and an output to C; and C has an input from B and an
output to A. The example system has a feedback loop, A to B to C to A.
36
I
System
Components
A
C
B
4
B
to
:data
Provides
Receives data from
A "1" in the matrix shows that:
Component A provides data to component B
or
ComponentB receives datafrom component A
Figure 15 - Simple DSM Example
The DSM in Figure 16 below is an alternative representation of the system dataflow
diagram in Figure 14 (with minor modifications and additional interfaces that were
omitted from the original diagram for clarity).
dMnfB&*S)
flivi
-'
-1-
1
e
.MaqippTook)
n
II
-
aaab.
A 11
S1
1
-
}1-- -
11
1
-
1
1
1
-
a
icmes
*s
n FU
i i
~
-
-
1
1A
1
- --
i
-
-
-
I
-
-
--
i t
C1
1
1
1
1
+ 1 U-
-.
t
1
-
j
I
~~T
_
Figure 16 - DSM of System Database Flow
37
This DSM representation of the system organizes the components by type, which is
more consistent with the layered nature the architecture software is intended to have. The
software layers are color coded showing the Defense Information Infrastructure Common
Operating Environment (DII COE) in top (row 1), databases next (rows 2-5), application
services (rows 6-11), mission applications (rows 12-32), externally developed systems
(rows 33-35) and external systems (rows 36-42). The DSM quickly reveals that many
applications are directly accessing the database for both reads and writes and shows the
interdependence of mission applications. The DSM also allows one to quickly identify
external interfaces and the mission applications with the greatest number of dependencies
(both internal and external). The benefits of using a DSM representation of C2 software
architectures will be explored in greater detail later.
By quickly summing the rows and columns of the DSM we can see which system
components have more than 10 data input or output interfaces. The list of components
with greater than 10 interfaces could be considered high risk and includes: DII COE,
AODB, MIDB, Message Processing (USMTF), Joint Mapping Tool Kit (JMTK), TW,
TCT, SAA, TAP, and EM. This list of components identified by interface count using
the DSM corresponds almost exactly to components which presented the greatest
challenge during development. Each of the high risk components will be discussed in
greater detail. TAP and AODB were not included in the detailed discussions for brevity.
Both of theses components were under the direct control of the prime contractor and
while difficult to implement were successful in the fielded system.
Defense Information Infrastructure Common Operating Environment (DII COE)
Compliance with the Defense Information Infrastructure Common Operating
Environment (DII COE) was a downward directed requirement and mandated for all C2
systems. This meant that every system component had to be built to DII COE
specifications, and every time DII COE was changed it affected every component in the
system. The rework associated with compliance was significant.
38
What is DII COE? DII COE is a software infrastructure for supporting mission
applications, and a set of guidelines and standards. It includes a kernel, segment installer,
and a library of common segments for functions like alert services, mapping, common
operational picture and overlays. The intent was to provide an approach for building
interoperable systems by creating a collection of reusable software components. Under
the DII COE concept system functionality can be added to or removed in small
manageable units called segments. These segments are the building blocks of mission
applications and are essentially software modules that provide system capability.
Structuring mission applications into segments is meant to provide flexibility in
configuring the system because segments are meant to be individually replaceable. In
fact the DII COE installer can add or remove individual segments quite easily allowing
rapid upgrades. The guidelines and standards specify how to reuse existing software, and
how to properly build new software so that integration is seamess. DII COE is a "plug
and play" open architecture and a foundation for building an open system. The shortfall
is that segment dependencies are not addressed by DII COE. While DII COE makes it
easier to remove and install segments by supplying its version of an install wizard, it does
not explicitly consider the potential conflicts, or incompatibilities in new configurations.
Experience with DII COE in the AOC system was not entirely positive. DII COE was
the major infrastructure component that all mission applications were built to work on.
New releases of the DII COE kernel and common segments are available every six
months and an array of C2 program offices are constantly developing DII COE compliant
segments for reuse. For the AOC system, every mission application that was developed
had to be segmented for DII COE and reuse of DII COE common segments was required.
One of the DII COE common segments that was problematic was the Joint Mapping
Toolkit (JMTK) which is discussed in more detail later. The expense involved in
achieving DII COE compliance was high and the payoff was questionable. The table
below provides a breakout of coding effort associated with building the initial release of
the AOC system software. DII COE compliance affected 48% of total developed SLOC
and segments, and 42% of total software engineering effort. DII COE was associated
with $43M - $51 M of a $106M total development effort.
39
Developed SLOC
Segments
Infrastructure
Services
238,000
173
Application
Services
287,000
66
DII COE Total
525,000
239
Release Total
1,100,0002
500
Table 1 - DII COE Software Lines of Code
From a development standpoint DII COE suffers from many of the same problems that
are encountered with other 3 rd party software deliveries. Overall the integration took
longer than planned, costs associated with upgrading were extremely high because all
segments must be retested following an upgrade, and fixes in new versions came with
new problems & bugs. Future systems may benefit from DII COE segmentation work
done as part of this development, but the opinion of most program managers and
technical staff is that DII COE standards are already outdated and commercial standards
are more useful for improving interoperability. The requirement to adhere to DII COE
standards is viewed as a burden which increases cost and schedule without providing
direct benefits to the program.
While DII COE is an attempt to create a more flexible architecture, the need for an
architecture which plans for and anticipates variation is not addressed. The dilemma
faced by program managers is managing and synchronizing upgrade cycles under
conditions of cost and schedule uncertainty. DII COE segmentation allows program
managers greater flexibility in recombining segments to create a system, but simply
determining what combination of components or upgrades to include in a version release
to meet the required functionality can be a challenge given the range of choices available.
Tools for understanding interdependencies between software segments and the
functionality achieved in differing combinations are sorely needed. The DSM is an
excellent toll for documenting and tracing component dependencies or interfaces.
Modernized Intelligence Database (MIDB)
The MIDB is relational database populated with worldwide intelligence information.
MIDB is managed and developed by an external organization with their own set of
contractors and sub-contractors. As the DSM shows, 13 different mission applications
40
are making calls on the database. Individually the database can respond to the database
calls and provide data and development work proceeded to build and test each interface
individually. The MIDB development cycle releases updates to tables, elements and
rules every 6 months. These periodic updates force rework and retesting of the interfaces
to multiple mission applications. When put in conjunction with DII COE changes, the
rework cycles were excessive and development progress was slowed.
The database itself uses a highly complex set of tie tables to reduce the amount of data
stored, but this has adverse effects on performance because changes and queries must go
through a large number of tie tables to retrieve or update a record. When a query or store
action is initiated on a record in the database, all the tie tables associated with that record
are effectively locked until the query or store action is completed. In a database where
thousands of records are associated with a single tie table these locks are required to
insure updates are completed properly and the relationships are preserved. When the
database is shared by multiple users, contention for access to the database tables becomes
a problem. Within the AOC environment where as many as 450 users are on the system
simultaneously and 40 may be running complex queries or store procedures against the
database, contention within the database became a significant problem when database
query responses of 1,000 records took times on the order of 20 minutes. This
performance issue was not adequately addressed during design.
The highly complex nature of the database required a significant learning curve on the
part of the prime contractor to integrate as part of the system. A number of specialized
stored procedures and shadow tables were created to improve performance. With the
stored procedures and shadow tables, similar query response times were reduced to
approximately 2 minutes. This was a laudable achievement on the part of the
development contractor and absolutely essential to system usability, but the database still
had not been placed under operational loads. The focus was on the unique
implementation necessary to meet system performance and usability needs. This unique
implementation carried with it an impact on the ability to make changes and upgrade to
new database releases available every six months. The rework required to accommodate
MIDB upgrades was extremely difficult and costly. Since MIDB was on a 6 month
update cycle, the integration was consuming a large amount of staff effort and causing
41
cost growth. As a result a decision was made to freeze the MIDB baseline for a two year
period. The MIDB configuration freeze allowed development of the AOC system to
progress more smoothly.
When the system went through component testing performance was acceptable. When
integration testing began with limited load on the database the performance continued to
be stable but poor. When the system went to full scale testing under operationally
representative loads database contention caused performance to drop. As the operational
load on the database increased row and table level locking caused delays in database
responses. When the delays exceeded a preset limit, the database generated an timeout
message which the mission applications had not been designed to handle. This caused
the AOC system client workstations to freeze, but provided no outward indication of the
error. The errors were only detectable by the fact that the client workstation remained
frozen or hung indefinitely. Ultimately, this single MIDB performance problem caused
the system to fail test. Database specialists were brought in to 'tune' the database server
and optimize performance. The originally specified dual processor server was upgraded
to a faster quad processor, and a number of AOC procedural changes were made to
deconflict high load activities on the database. Ultimately the changes to address this one
issue resulted in $1.4M of unanticipated costs and a 4 month schedule delay. Other cost
increases due to MIDB problems were not included in this case study.
United States Message Text Format (USMTF)
USMTF is a highly structured set of text message formats. The structured nature of
the messages allows machines to 'read' the data contained within them and pull out the
relevant data fields for automated processing. The ability to receive a structured or
delimited text message and use it to populate fields in databases or mission applications is
extremely valuable because it eliminates the manpower intensive and error prone task of
retyping data into an application. The value of USMTF made it an integral part of the
system as a whole and there were many mission application, database, and external
dependencies which shared USMFT formats.
42
There are two drawbacks to USMTF. First, the message formats are changed and
updated on an annual basis. This requires the entire set of message formats used to be
reviewed and retested in the system to ensure the data automatically transfers as
designed. Small changes of even one misplaced or changed character can cause the data
to be transposed in the databases or mission applications and result in completely
corrupted data. Validation scripts were developed by the AOC system to protect against
this problem. USMTF format updates impact the validation scripts, mission applications,
and databases behind them (MIDB and AODB). The second drawback is centered on the
humans who input data into the systems. In some cases humans prepare the USMTF
messages directly. Small formatting errors can cause the messages to fail validation, so
users receiving the messages are forced print the messages and retype the data into the
systems on the distant end. In other cases, the systems generate the USMTF messages
automatically, and system operators inputting data must follow strict data formatting
conventions. The difference between "F15" and "F-15" or "F-15E" and "F-15e" are
subtle, but critical to the mission applications. For the systems to interoperate properly
all the mission applications and databases must agree on common formats. Every change
to a USMTF format must be traced through the system to ensure consistency with the
change. When human operators fail to follow the established conventions, subtle
problems and bugs emerge which can be difficult to trace and fix. This requires the
mission applications to enforce consistency by using data-masks or pick lists and
generally make the systems less user friendly. For example, when a mission application
requires multiple data fields to be input by an operator and then processed, if one of the
fields is incorrect the application must handle the error gracefully. There is nothing more
frustrating than to have the system reset all the fields without an explanation of what was
wrong. While these implementation details are not technically challenging, they are
absolutely essential to system interoperability and usability.
The annual upgrades to USMTF were too costly to maintain within the AOC system.
A bi-annual upgrade strategy was adopted for even year USMTF formats (i.e. USMTF
'98, USMTF '00, USMTF '02). The cost to make the USMTF upgrades was high and
delivered no discernable functional improvements to the users. Additionally, external
systems which implement odd year USMTF formats have the potential to be
43
incompatible with the AOC system. Overall, new strategies and approaches using XML
translators may provide isolation and improve interoperability, but they have not been
implemented yet.
Joint Mapping Toolkit (JMTK)
JMTK is a DII COE common segment and was required to meet compliance
mandates. While the mapping tool did not cause system conflicts it did cause usability
problems. During recent operations in Afghanistan where the AOC system was
deployed, the JMTK was deemed unsuitable for operational employment and emergency
patches and work arounds were required to complete the mission.
"The graphicalnature of the targeting mission, intended to be instituted
on the provided JMTK failed. This specific requirementwas not
recognizedfrom targetsearly in the development of the AOC system, but
had been levied from otherfunctional areas... JMTK had been identified
in testing since 1997 as being unable to support warfighter
requirements... It is clear that the acquisition rules levied via thirdparties
had influence on the suitability of the system for operationaluse and that
the testing environment was not of high enough fidelity to ensure
operationalviability of the applications."
Interim Report - AOC Intelligence Capability,
ENDURING FREEDOM, 1 Oct 01- 1 Dec 01
The fact that JMTK was unsuitable for operational use was primarily because the
graphics, map data, and geospacial information products available on JMTK were
inferior. It is reasoned by a number of interviewees that the high level of interfaces (25
identified in the DSM in Figure 16) and required compatibility with other mission
applications limited the choices and development effort devoted to visualization and
mapping data options. Since the real value of the mapping tool is in the visualization
capability it provides, this trade-off made it an inferior choice. Commercially available
mapping products have far surpassed JMTK in terms of performance and provide 3D
visualization and navigation capabilities. JMTK is a 2D tool with limited data sets
available. DII COE has not yet migrated to the next generation of mapping products due
to the expense involved with segmentation. This may seem contradictory since
segmentation is intended to be beneficial to interoperability, but there is a high initial cost
44
to properly segment a software application for DII COE. The requirement to segment
therefore means additional cost, but allows for better code reuse, sharing and
commonality later. The fact that the commercial products have standard Application
Program Interfaces (API) which makes them easy to integrate with other systems, leads
some architects interviewed to believe the DII COE is becoming obsolete as a standard
for interoperability.
Targeting and Weaponeering (TW)
TW is a critical area of the AOC system and has been the nexus of a great deal of
controversy and competition. Over the last decade a number of targeting applications
have emerged on the scene. The first software system focused on the targeting function
was RAAP (Rapid Application of Air Power). It was developed and received a limited
fielding in conjunction with a legacy intelligence system known as CIS (Combat
Intelligence System). When the AOC system was approved, the program migrated a
great deal of the CIS functionality into the new system, but chose to re-engineer the
targeting capability. The new TW application implemented all the previous RAAP
functional requirements in the new architecture, but provided no real enhancements. The
original RAAP developer initiated a new program JTT (Joint Targeting Toolbox) in
parallel with the AOC system development which promised vastly improved capabilities.
The JTT developers leveraged their previous experience, requirements, and contacts
throughout the targeting community to gain early buy in and commitment to the new
concept. JTT aggressively marketed their plans and solicited feedback on requirements
for an enhanced set of capabilities through a series of JTT User Group meetings. These
meetings provided overview briefings and story boards of the Graphical User Interfaces.
This effectively educated the user community on what JTT planned to do and gave JTT
insight into how the users wanted to see the application implemented.
By mid 1998 JTT had broad buy in and backing from the targeting community as a
whole. The fact that the legacy system (RAAP) was no longer being supported and that
the new AOC system with TW was having problems delivering due to schedule slips
increased the emphasis and desire to field JTT quickly. The AOC system made a
commitment in mid 1999 to freeze further development on its own TW application in
45
favor of working with JTT to integrate the improved capabilities. A commitment to fund
$1M a year to support integration of JTT into the AOC was made in addition to the $9M
per year JTT development effort already underway.
As the AOC development schedule continued to slip the targeting community was
growing impatient for a solution and a series of user initiated solutions began to spring
up. The first delivery of the AOC system version 1.0 in Apr 00, with an already outdated
TW application failed to meet user expectations. This was due to the aggressive JTT
marketing and freeze on TW development in mid 1999. The user community expected to
see functionality promised by JTT developers, but received the equivalent to RAAP
functionality they had 5 years earlier.
JTT relied on the Modernized Intelligence Database (MIDB) for targeting data and ran
into performance problems similar to the AOC system. The promise of JTT
enhancements was broken when JTT development activities slipped and the JTT delivery
of version 1.0 was delayed into mid 2000. Initial attempts to integrate JTT 1.0 in the
AOC system encountered further difficulties due to differing approaches to interfacing
with the MIDB. JTT had opted to develop a proprietary set of SQL queries which were
incompatible with the AOC stored procedures and shadow tables. Despite the $1 M per
year integration funding, the AOC system did not have sufficient influence over the JTT
program to alter their implementation. Incompatibilities have made every integration
attempt to date unsuccessful.
Figure 17 shows a history of targeting application development over the last 5 years.
The current fielded system in the AOC is based on the original RAAP with no upgrades.
Progress in fielding enhanced targeting applications has been effectively stagnant for 5
years with the notable exception of user developed initiatives. JTT integration efforts
continue, and the first successful use of JTT with the AOC system was implemented as
an emergency patch for operation Enduring Freedom at a cost of over $3M. It is a unique
implantation which is tailored for one operational command.
46
-U
___________________________________
2001
2000
1999
1998
1997
1996
Intelligence
Combat
AOCSystem
System
1.0
(CIS)
RAAP (CIS)
F uture
ne
TCTF
TCT (beta)
2003
2002
AOC Sys
AOC Sys
AOC Sys
1.0.1/1.0.2
1.1
1.2
,i.W
Firsttargeting tool developed by
automate target system analysis.
JTT 2.0
JTT 1.0
Vulture
*AFDele
-isM~kstutto
User Developed
JTT 3.0
FATE
APEX
.*V&AF
'
I
)TT 2.1.2
Deelopei
-Teto
J
Quiver
.1~o~zmw- F,.l,.
MS ExcelI MS Access DB
ITE
A
magtre
limif
-A
ard copy Target Folders - Collection of imagery, Basic Target Graphics, and intel reporting. Histor & details of the target
Figure 17 - History of Targeting Application Development
The user developed initiatives by contrast have been highly successful and continue to
deliver enhanced capabilities building on previous successes. The best of breed of the
tools are adopted and absorbed into the next generation of tools. These user initiatives
are.based on commercial technology and use web based html front end interfaces and
simplified databases in the back-end. These applications opted to take a small sub-set of
the larger MIDB database and included only relevant data fields, table, record, and
relationships. The user applications thereby avoided the majority of the complexity and
difficulty of implementing an MIDB based solution and stuck to short development
cycles done in conjunction with teams of operational targeteers. Many of the
development cycles are done in preparation for an operational exercise where the
application is used and lessons learned provide the basis for the next set of enhancements.
As a result, the tools are well suited to the users needs and the constant prototyping,
feedback, and design changes increase the capabilities faster than standard acquisition
processes can react to and plan for changes.
47
ITS (Interim Targeting Solution) was developed, tested and fielded in approximately 5
months with a small team of 5 developers and 5 targeting users. The total cost for
development, testing, travel, hardware, and software licenses was less than $1.2M and the
system delivered more functional capability than is planned for JTT 3.0. The 'interim'
designation is viewed as part political necessity and part face saving. The politics of JTT
have to do with the Joint designation and fact that there are multiple customers/investors
paying for the development. Senior decision makers in the Air Force have committed to
building JTT for Army, Navy, and Marine customers in addition to internal Air Force
customers. Killing the JTT program with over $36M invested over 4 years is viewed as a
failure. The fact that ITS was able to deliver more capability in a rapid cycle is viewed
by some as a new development model where software can be considered a disposable
commodity developed for a purpose, and used until it is overcome by the next generation
of technology and enhancements. Others view ITS as a prototype to base larger, long
term developments on and therefore characterize it as 'interim' and unsupportable. No
matter which group individual opinions fall into, there seems to be a universal agreement
on the value of user involvement in the development process, and the fact that small
projects are free of many of the political and bureaucratic burdens which slow progress
on larger projects.
Time Critical Targeting (TCT)
Time Critical Targeting started with a need identified during operation Desert Storm
and the problem of SCUD hunting. Most people remember the SCUD missiles that were
used as terror weapons, launched by Iraq into Saudi Arabia and Israel. The record of
engagement against SCUD Transporter Erector Launchers (TELs) is 84 misses and 1
success according to some sources. The only successful engagement occurred when
special operations forces happened on the SCUD by chance. Early concepts for how to
identify SCUD operating locations focused on terrain delimitation, a process by which
areas too rough for the SCUD TELs to traverse are eliminated from the search areas.
Better intelligence analysis of the battle space to identify support facilities and probable
operating locations was added. The need to task intelligence collection sensors to
positively identify emergent targets, and the capability to identify and task the right strike
48
asset to engage the target were all identified as needs which had to be met in a 30 minute
timeline for target engagement. The result was a basic set of functional requirements and
the initiation of a system acquisition.
The process for identifying and engaging time critical targets was very immature.
When the program was launched in the 1994 SCUD hunting was the focus of the effort
and justification for the funding. Delivery of an operational capability was expected in
1998. The time critical targeting system, which is still in development and has not been
operationally tested or fielded, now consists of 4 primary components. A terrain
delimitation tool, a Intelligence Preparation of the Battlespace (IPB) tool, a WeaponTarget pairing tool, and an Intelligence Surveillance and Reconnaissance Management
tool. Issues stemming from the fact that the time critical process was immature when the
program was launched include the fact that the tool does not deal well with targets other
than SCUDs. There was a recognition that the target set was much broader, but without a
clear concept or process for how to solve the SCUD problem there was little effort
devoted to other classes of targets.
Problems with implementing the TCT system can be tied to the fact that it is a
relatively new concept and the supporting infrastructure and procedures are not in place.
For example, the terrain delimitation tool requires detailed information on the target
classes' (i.e. SCUD, tank, truck, artillery, SUV, radars...) capability to traverse or operate
on different types of terrain. The tool primarily delimits terrain based on the grade or
slope of the terrain. For example if a SCUD can not drive over an incline of greater than
10 degrees or a radar can not be set up on an area with greater than 5 degrees of slope
then those areas would be eliminated from the search area. The terrain delimitation tool
produces marginally valuable results but does not limit the search areas to a great extent.
The IPB tool requires extensive background data to populate its graphical models and
displays. Current estimates would require two staff years of effort to build a model for a
small theater of operations and the intelligence community does not have production
requirements or formats to produce products to support the IPB. Ties into the databases
to support weapons target pairings have not been established and the operations databases
49
have latency issues which means the data does not support the timeliness requirements of
the tool. And finally, the intelligence collection management community is able to retask
sensors, but the exploitation systems tend to treat incoming information as a first come
first served queue, so time critical responses are significantly delayed. The overall
picture is that the time critical targeting system must interoperate with a variety of other
systems, but due to immature process definition the development has been hindered. The
TCT process falls into the class of "and then a miracle happens" and we have this
information to make a time critical decision.
The time critical targeting system example is a valid requirement to pursue and
funding will continue to be added as the system overcomes it's integration and
interoperability issues. Some unrealistic assumptions early on have arguably made the
system less capable, less supportable, and less cost effective than it could have been.
Major changes in production and exploitation of intelligence will have to occur and
current intelligence architectures do not support the new requirements well.
Situational Awareness and Assessmnet (SAA)
SAA focuses on the real time fusion and correlation of multiple intelligence sources.
The system receives feeds from ground based radars, airborne radar, and Identify Friend
or Foe (IFF/SIF) aircraft transpoders to create a comprehensive picture of what is flying
in the airspace. The airspace data is then merged with multiple other intelligence sources
which provide the locations of ground threats based on electronic signatures,
communications intercepts and alerts, thermal signatures from missile launches or aircraft
afterburners... By plotting the composite set of information sources on a common map
display a Common Operational Picture (COP) is born. One of the limiting factor is that
many sensors are seeing and reporting the same events, so the display becomes a
cluttered mess where 5 tracks may be plotted simultaneously for the same object. This is
where a correlation engine is needed to eliminate duplicate reporting of the same tracks.
The complexity of handling multiple inputs from a wide variety of sensor systems and
displaying them on the same map was the initial step and required a fair amount of
engineering work. This work started in the 1950's and 60's with ground radar units
50
linking their radar pictures together to extend the coverage of the airspace. As network
technology came into being within the intelligence community multiple feeds were
integrated into a tactical intelligence broadcast system that could provide the picture as
seen by sensors anywhere in the world. Depending on the sensors available, fields of
view, and filtering established by the user, a picture could be tailored for a particular
geographic area or purpose. The task of actually correlating data from multiple sensors,
in different reporting formats and accommodating time delays was one of the goals of the
SAA mission application.
The first deliveries of SAA in an integrated environment did not work. The feeds
from multiple sources overwhelmed the workstation and the system crashed when too
many tracks were displayed. Additionally, the monitor would become so cluttered that
the display would become meaningless. Human operators or COP managers in the AOC
intervened to set up filter criteria and eliminate multiple tracks manually. The picture
they create is shared across the AOC and can be further refined at individual workstations
to answer specific questions. Improvements in correlator capability continue to be made,
but the initial set of requirements were de-scoped or reduced to simply providing the
capability to simultaneously plot 3,000 tracks on a map display within 15 seconds.
Execution Management (EM)
The original EM mission application was developed by a third party subcontractor and
delivered to the prime contractor for integration. It was highly interfaced with the Air
Operations Database (AODB), ACP, TW, TCT, AD, TAP, external C2 nodes to allow for
dynamic monitoring and retasking of air assets during execution of an air campaign. The
highly coupled nature of the original EM application made it extremely difficult to
integrate and there were numerous problems throughout development. It was so
problematic that an acquisition officer was assigned to monitor its development full time.
Despite the extra-ordinary effort to make this highly coupled application work, the fact
that it was being developed outside the control of prime contractor meant that there were
fewer opportunities for coordination and discussion between mission application
developers that had interfaces and dependencies with the EM application. Many required
51
changes and discrepancies were not identified until integration of EM with other mission
applications and the determination of the cause of the problems became politically
charged since neither contractor wanted to take responsibility for the problem or shoulder
the cost for fixes. Negotiations over how to implement fixes generally met with wide
differences of opinion as to the causes and best approach for solution with the program
manager caught in the middle of a dispute. Subsequent changes led to discovery of the
next level of incompatibilities and another round of tough negotiations. Progress toward
a working solution was very slow.
After a number of in plant integration tests, failures, and fixes, the cost growth and
disruption caused by the EM application was becoming excessive. A dual path approach
was initiated, and second application was introduced which duplicated key functions of
EM, but with drastically reduced functionality and interfaces. The second application
development was launched within the prime contractors development facility. After a
three month development cycle the two applications were placed side by side in a
demonstration to users and the scaled down version of EM was selected as the path
ahead. The first EM application development contract was killed and $25M in
development effort were scrapped in favor of a $2M minimalist version that was less
disruptive to the baseline, had fewer features and as a result was easier to understand and
learn.
Case Study Summary
The AOC system and sub-systems present a significant software engineering
challenge and this system is one of the most complex ever attempted by the C2
acquisition community. The individual components of the system are complex
undertakings and many are driven by requirements outside of the direct control or
influence of the AOC system program manager. The fact that the program was launched
based on crude estimates of the scope and complexity of the integration task and a
minimal understanding of the interfaces and architecture is surprising. As with any large
project, detailed design can not be completed up front, and many of the design challenges
can only be solved after building critical portions of the overall project. In the case of the
AOC system, some of the underlying components like DII COE and MIDB which were
52
critical to the design and equivalent to the load bearing walls of a structure, were not
sufficiently understood by the system architects and engineers at the outset. Additionally,
these load bearing walls were undergoing constant changes and updates, but there was no
way to replace them without causing major rework and re-engineering of the entire
system. It is roughly equivalent to replacing the foundation of a building periodically as
construction progresses.
Hopefully the case study description has provided a basis for understanding the scope
of the development and integration effort faced by program managers at the C2 node and
system level. An exhaustive description is not possible in the context of this thesis and
the purpose is to highlight the major themes and challenges. Increased emphasis on
interoperability between systems only serves to increase the difficulty of planning for and
synchronizing the multiple changes which are ongoing in the system of systems context.
To be successful in delivering such large complex undertakings on time, on budget, and
meeting all the functional performance requirements set at the beginning of a project
requires a degree of collaboration and sophistication which does not appear to be present
in the current practice. The rest of this thesis will go on to explore barriers to increasing
systems interoperability which underlie current program management practice and
propose approaches to minimize their effects.
53
Barriers to Enhancing Interoperability
Complexity
No discussion of software intensive projects would be complete without some
discussion of complexity. Many of the recognized leaders in the software field point to
complexity as a major stumbling block for developers and integrator. Fredrick Brooks
stated "software entities are more complex for their size than perhaps any other any other
human construct..." [4]. Ubiquity Magazine interviewed Paul Gross, Vice President in
charge of the Developer Tools Division at Microsoft, in a September 1998. When asked
"What's holding back most developers today? What are their biggest problems?" Gross
answered, "I think the biggest problems developers face today center around complexity,
the speed of change, and the ability to keep their skills up to date." The following
sections will explore some of the dimensions of complexity which present barriers to
improving C2 systems interoperability.
Architectural representations
For C2 systems and other large software development project, dealing with system
complexity is a major challenge. Architectures define the arrangements and relationships
between components in the system design, and ideally hide system complexity inside
components while presenting simple, understandable interfaces to external components.
A good architecture therefore relieves component developers from the burden of
understanding the entire system and allows them to focus on their piece of the
development task. In a C2 system where there are many hundreds of software modules
built by teams of developers and interconnected by a tangle of interfaces, the full system
complexity is beyond human cognitive capacity. Architectures allow developers to take
part in a complex undertaking, but limit the amount of complexity they must deal to their
component and its immediate interfaces.
54
-U
The importance of good software architectures and their usefulness in managing and
reducing system complexity is a generally accepted principle. However, in the survey
conducted as part of this thesis, 73% of respondents felt that architectures were only
rarely or occasionally well documented and widely understood. See Figure 18 below.
Are architectures well defined and
widely understood?
20
18
161412108642
0
Q3. Please provide your opinion on the following statements as they relate to acquisition programs.
System architectures are well documented
and widely understood
N/A
Rarely
Occasionally
Usually
Often
Always
[]
[]
[]
[]
[]
[I
Figure 18 - Results from survey question
This is a startling finding, since no complex development project should begin without
a clear architecture in hand. You might build a dog house without the benefits of plans or
a formal architecture, but only a fool would start construction of a house without a set of
blueprints to organize the work of masons, carpenters, electricians, plumbers... C2
systems are the software equivalent of sky-scrapers and without formal architectures to
guide the construction and engineering work the project would be doomed to fail. Follow
on interviews revealed that formal architecture products as defined by the DoD
Architecture Framework are largely an afterthought. The framework includes 23
different architecture views divided into three categories (operational views, system
views, and technical views). While the set of architecture views is fairly comprehensive
and potentially useful, they are also time consuming to generate and maintain. These
architecture views are still considered more of a resource drain than a tool to improve
interoperability. Furthermore, the framework is silent on graphical representation and
55
.__.
_
-
Il
naming conventions and there is no formal validation or coordination process between
interdependent programs.
The type of architecture products used to by the program managers to guide the
development of C2 systems are primarily power point diagrams. Recalling the case study
example of the AOC system we can gain a sense of the complexity of the system under
development in Figure 19.
t
UNIT
AT Ow
J ecuAeC
UNIT
2'S
CAMS
Weathr
Data
CRCcRE
FrOB
ABP. Airspace.
(GTACS)
Scheduling.
Scranble
& rission
status
TOC
L
ACO
AO
TNL
CTL
-
-
-
*TrackJd
I urSecure
WAN
BP.
&
BB
Remote Terivital
bo
EMIR
Am
AC
Daia
Weather
sWa1us
Data
I1WAN
cx~~~
.ission
ACM
ABP, FrOB. Airspace
Secure WAN) 2SOCAP. FrOB.
at
>
ACO
ATO
ACO
AFADS,
AC
Tr--k
~
Secure w
Noe
AC
l
Track
SSUB 1.
-----
-A
,MFrOB'
Stti
Data
--TDDS
TOPS
P
--
TADIL A, B. j
NA'TO Link I
-s
STR
- -- - - HUAV
Weather Feed
AN
cir Sece
LEGEN
-
GCCS-
AFA'IDS
WS
WAN)
CCS
-------....-----.--------..--..-----....- t
CT
AN
.. -- --.-....
--
Database
Appliation
Da/MessageService
External huafxe
Externally Developed
Figure 19 - AOC System Data Flow Diagram
Managing the development of such a system is a complex undertaking with many
components and interfaces to develop, integrate and test. The diagram above captures the
high level relationships, but it does not provide a good way of thinking about how to
56
organize tasks or manage changes. Simply finding all the instances of the AAT
application on the diagram is difficult.
... real software systems have many such components, and there is no
repetitive structure to simplify the analysis. Even in highly structured
systems, surprisesand unreliabilityoccur because the human mind is not
able to fully comprehend the many conditions that can arise because of the
interactionof these components. Moreover,finding the right structure has
proved to be very difficult. Well-structured real software systems are still
rare." [5]
The inability to manage complexity and support change decisions within the C2
manifests itself in the form of frustration and decisions which reduce the overall
complexity of the system by eliminating desired features. In the case study of the AOC
system, program managers had to contend with requirements to meet 914 operational
functions implemented in 42 mission applications made up of 387 software segments,
providing 669 external information exchanges. Ignoring for a moment the implications
of the larger enterprise, let's focus on the reaction to a level of system complexity that
was unachievable in given the budget and schedule constraints. When the AOC program
was in imminent danger of being killed due to an inability to deliver the system with the
full set of capabilities, the program managers in conjunction with the operational
requirements managers established a smaller set of 'key performance parameters.' The
set of key performance parameters allow them program to ignore problems with functions
that were not deemed critical. A large number of problem reports that previously
required fixes before the system could field were now deferred and would be revisited in
future versions. Some capabilities were removed from the baseline altogether. For
example in Feb 99 $3.5M of planned capability was deferred, and in Aug 99 another
$7.4M in planned capability was deferred to cover other costs. A $25M dollar
investment in the Execution Management (EM) mission application development was
scraped in favor of a greatly simplified EM alternative solution prototyped and
demonstrated in just 3 months.
In principle, the architecture products should have identified the 'load bearing walls'
of the system and limited changes in those areas to ensure the overall system was stable.
57
The design structure matrix developed for the case study (see Figure 16) and the joint
technical architecture (see Figure 12) pointed to these load bearing walls, but they were
not available or leveraged during the early phases of the program. Efforts to build
comprehensive sets of architecture views as prescribed by the DoD Architecture
Framework are potentially beneficial, but have yet to be fully embraced by program
offices because of the expense and detail required to build and maintain them. In many
ways the gap between known best practices and actual program management practice are
increasing as more ambitious projects are conceived and launched.
Architecture representations are a great tool for enhancing interoperability. In practice
they are not effectively leveraged by the program offices for decision making. Part of the
reason program staff do not take the time to really dig into the architectures and
understand them because the current architecture products are too detailed. This
contributes to the cost to build and maintain them and reduces the likelihood anyone will
actually use them. Rechtin contends that, "It is the responsibility of the architect to know
and concentrate on the critical few details and interfaces that really matter and not to
become overloaded with the rest." [6].
Stability & technology turnover
The C2 systems being used by today's military are subject to the same forces driving
technology turnover in the commercial sector. Upgrades to operating systems,
commercial software applications, mission applications, middleware, services, database
schema modifications, improvements in processors, specialty cards and drivers, better
displays, faster communications, new processes and methods... To take advantage of
advances in technology or to simply fix existing bugs and defects in the current system
requires turnovers and updates to components. Modifying or re-engineering processes
and procedures can drive component redesign and turnover as well. Depending on the
level of dependency in the system, upgrades can be as simple as a single component
swap, or for highly integrated designs, may require a complete system redesign. The
complexity and the cost of changes can be reduced with appropriate up front planning.
58
The goal should be to prepare the system for the inevitable changes that will take place
[7]. During case studies of C2 systems architectures there were attempts to plan for
changes in the systems, and these were documented in Roadmaps and way ahead plans as
well as architecture system views (SV-8s). These plans were focused on how to migrate
the current architecture to some future state. There was little evidence of attempts to
create an architecture that accommodates modifications. There was no focus on
providing hooks, stubs, or other architecture constructs to allow for future variation. An
architecture that easily accommodates unplanned for changes is referred to as flexible.
Rigid architectures on the other hand require more work and effort to accommodate
changes. C2 systems architectures tend to be rigid. This perception is supported by the
survey where 58% of respondents felt that rigid or inflexible architectures were a
significant or primary cause contributing to cost and schedule growth as shown in Figure
20.
Rigid architectures contribute to
cost/schedule growth
14
10
8
0
Q4. What factors contribute to cost and schedule growth?
N/A
Small
Moderate
Significant
Heavy
Primary
Contributor
Contributor
Contributor
Contributor
Cause
Rigid/inflexible architectures
Figure 20 - Results from survey question
59
The case study discussion on Defense Information Infrastructure Common Operating
Environment (DII COE) points to the impact and cost growth that can occur under a set
of architecture mandates which require additional work. DII COE was a significant
contributor to the sense of inflexible architecture felt by staff working on the AOC
systems development. The ability of the architecture to accommodate changes without
requiring a great deal of rework is highly desirable, but should not be considered a
panacea.
Lets consider the simple case of a system that has been logically decomposed into
three functions each of which will be performed by an independent component. If there
are three options available for each component (i.e. different versions of software, or
different vendors or products to choose from), then how many possible ways can the
system be built? Applying the Product Rule from the field of combinotronics, we can
consider each function as a Task (Ti) and each options for a different component to do the
,
task (ni). The product rule states - if task Ti can be done in ni ways after tasks TI, T2 , ...
Ti-I have been done, then there are n1 x n2 x ... nm ways to carry out the procedure. In the
case of the simple system we have T1 , T2, and T3 where nI=3, n 2=3, and n 3=3, therefore
there are 3 x 3 x 3 = 27 ways available to build the system.
Since the components are independent we can assume all the possible configurations
are valid and would work. The process of deciding which combination of upgrades to
implement in the next version would be a trade off between the benefits of an upgrade
and the cost/work to upgrade to a particular option. A fixed budget would potentially
eliminate certain options. In reality, subtle dependencies between options are likely to
exist and must be evaluated to make an informed decision. If component 1 were an
operating system, component 2 were office automation applications and component 3
was personal organizer software we can get a sense of the possible incompatibilities
where windows 2000 might be required to run the latest MS Office suite which included
interfaces to the handheld being purchased in bulk for a corporate workforce. Choosing
Linux and Lotus Notes might meet the functional requirements of the system, but could
force the selection of a different handheld or require special modifications.
60
The example above is only meant to make the point that a small set of components and
options can result in a large set of considerations and trade-offs. This example is trivial
when compared to the real world system represented in the DSM in Figure 21 from the
case study where there are 42 system components. If in version planning there is a binary
choice fro each component to either leave it unchanged or upgrade it, then all ni's=2. The
space of combinations of components to build the system would be 242 = 4.398 x 1012. It
is far beyond human capacity to evaluate all the possible combinations. Even a computer
evaluating 1 million configurations per sec would require 51 days to exhaustively
evaluate all the possible combinations. Simply increasing the number of system
components to 49 yields 249 = 5.6295 x 1014 possible combinations and the same
computer would take an amazing 17.85 years to evaluate all the configurations.
.1
....... .........
I
V,
1..
. 1......
.....
1 1-
rDiWnes
1
4M Mappg TOC&R)
iaa~h~eNdcn
-7
-0 1-
:i
A - ...
-----
Ii
...
-
~
-
-.
-.
~
*7Mg
-r
'slTDDScacticjkWNgffeB.
ices Target AcqiskionRadar (J
.
~
-
---...
-
-
s
-------
Figure 21 - DSM of System Database Flow
Clearly there are no attempts to make exhaustive evaluations of potential upgrade
spaces. Budget limitations, deficiencies and user priorities narrow the range of options
quite effectively. A more realistically set of upgrade decisions may involve 7 system
components each with 3 upgrade options. This yields a more modest 37 = 2,187 possible
61
new configurations assuming the other 35 are left unchanged. This space is still too big
and the real nuances of component interactions have not been addressed. The reality of
version planning comes down to a number of upgrades or changes being dictated by
external forces (i.e. a component version is no longer supported) and recommendations
for upgrades made on the remaining system components in play. The cost for the
upgrades are totaled and then the budget line is drawn. Any initiatives falling below the
line are deferred until the next version. Some re-evaluation and 'horse-trading' can be
done at this point, but the basic decision is in place and the impact of system interactions
and dependencies associated with the changes are tenuously understood at this point. As
development proceeds on the new version, the impact of dependencies becomes apparent
and modifications/fixes are made. Given the difficulty of making upgrades and
understanding dependencies between components, tight configuration management and
code freezes must be enforced or the system would go into a perpetual change cycle and
never complete a test or fielding.
Architectural representations like the DSM Figure 21 are excellent tools for
understanding and evaluating component dependencies. The relationships between
components are instantly identifiable and investigation of how a change will effect the
rest of the system components can proceed quickly. With some early consideration given
to which components are most likely to change, the DSM can be used to evaluate where
isolation layers, services, middleware layers or new system decompositions can be added
to isolate those components and facilitate variability. These concepts will be explored in
greater detail in the recommendations section.
Emergence
Emergence deals with the idea of collective behavior arising from the interaction of
component parts. The emergent behavior is not a function of any one component, but
instead is a large scale property of the system. The complete specifications for all the
components do not necessarily capture or fully express the behavior of the system.
Another way of thinking about emergence is the lock and key metaphor. A complete
62
specification of a key, its geometry, materials, production process... does not convey the
fact that it unlocks a door. The properties of the key when combined with the properties
of a lock give rise to the functions of locking and unlocking. Interestingly, properties of
the systems which can not be assigned to individual components are often only achieved
through the correct relationships or combinations of components. Now in a simple
system like a lock and key emergence is not a very interesting topic, but in complex
systems made up of many independent yet interconnected parts the emergent properties
can be quite unexpected and counter intuitive. Swarm theory and artificial life are
exploring collective the behavior of unsophisticated agents interacting locally. Ant
colonies, bee hives and bird flocks are examples. In a complex system of heterogeneous
components (or agents) following complex rules (or programs) the emergent behaviors
can be highly unpredictable. The basic precursors for emergent behavior are that there is
no centralized control, the subsystems are autonomous, there is a high level of
connectivity or communication, and causality in the system is nonlinear.
How does this relate to C2 systems? Well to start C2 systems possess the precursor
properties which allow emergent behavior to occur. There is no central control residing
over the C2 systems domain, the subsystems are autonomous and perform a variety of
functions independently, they systems are highly interconnected and interoperability is
driving greater interconnectivity, the only question is whether or not there exists nonlinear causality across the system. Early investigation points to the fact that there is a
level of non-linear causality and emergent behavior does exist within the C2 system.
During early integration testing on the AOC system from the case studies experienced
a phenomenon characterized as a 'network storm'. The network storm caused the system
to fail test. Prior to the integration test, individual components had passed lab tests and
the system had passed an integration test in the lab environment. AODB, TAP, and
USMTF, which were all characterized as 'high risk' in the case study, are believed to
have been major contributors to the network storm problems. When the system was
integrated for the first time in a more operationally realistic environment with distributed
63
communications and a full complement of users, the network performance became an
issue. When multiple users began generating data on the system and interacting with
other systems, updates in the form of information flows began to occur. These included
mission application write backs to databases, data base exchanges, database change
notifications, and message traffic from live feeds which updated the databases. The
distributed nature of the system allowed updates to occur at multiple locations and on
multiple systems simultaneously (i.e. no centralized control). The locally generated
updates were propagated through the system to re-synchronize the system with slight
delays. Each update required a comparison of stored data with the update received and
notification of any changes made were propagated. As the level of user and
automatically generated changes on the system increased the system went unstable
because updates began cascading through the system similar to feedback over a public
address system, and hence the network storm. The network storms were intermittent and
often resolved themselves in a matter of minutes, but during the network storms latency
in the communications network and increased the CPU usage effectively stopped all work
on individual client workstations. The overall effect was the system could not meet the
mission requirements.
Initially, development contractors pointed to problems with the underlying
communications and networking infrastructure used for the test. The problem did not lie
in the network, but in the combination of multiple uncoordinated updates occurring
simultaneously with delays. Experimentation with database update and refresh rates
during the test reduced the network storms and the system remained stable. The system
went back into development to fix problems identified during the testing and
experimentation and steps were taken to eliminate multiple updates and centralize update
transmission. While all the component systems and business logic behind them
functioned as designed and worked individually, the combination and interaction of the
components resulted in emergent behavior that was not anticipated by the system
engineers. The real challenge with emergent behavior is that it is not predictable and our
limited understanding and ability to model the dynamic interactions of a highly complex,
heterogeneous system are very limited. Simulation based acquisition is one of the more
64
promising approaches which has a chance of identifying risks of undesirable emergent
early.
The testing treadmill
Closely related to the emergence is the problem of repeated integration testing of
complex arrangements of sub-systems and components where multiple changes are being
made simultaneously. This makes it more or less impossible to isolate variables and
perform trouble shooting. Decisions making about what to fix and how the fixes will
effect one another becomes an exercise of judgment and experience where the next large
test will provide feedback on the changes. Additionally, there tend to be layers of hidden
software bugs and problems. As thels' level problems are resolved, like the network
storms encountered during the early integration test, deeper problems like the MIDB
performance load problems are encountered. The general trend is a reduction in the
number and severity of problem reports, with intermittent surprise serious emergent
problems. There also tends to be an over-optimism about the ability of the system to
'pass' the next test. This over-optimism is due to a tendency by program managers to
focus and track on work to fix known problems and a reluctance to make estimates about
new problems that will emerge during the next round of tests. The program managers
receive closure reports for problems as they are fixed and tend to base their projections
for the project completion on those closure rates. The tendency is to ignore the fact that
additional problems and bugs will emerge as other problems are fixed.
During testing of the AOC system the program manager was under extreme pressure
to deliver a system without Priority 1 or 2 problems. Priority 1 problems are
characterized as extremely serious and will result in mission failure or loss of life.
Priority 2 problems are characterized as causing serious mission impact and no
acceptable workaround exists. An operational system cannot be fielded with Priority 1 or
2 problems. The AOC system began it's testing with government partners in the
development plant in Oct 98 and identified
-
364 priority 1 &2 problems. Based on the
number of problems identified and the ability of the contractor to fix them, the next test
expected to start on a system with few bugs and no serious problems. See Figure 22.
The high level of problems (312 priority I &2's) at the end of the next test was somewhat
surprising to the program management staff. Negotiations with the contractor to fix new
65
problems were quickly completed, and direction was issued to prepare the system for a
third test. The third test was intended to provide the basis for a fielding decision. Prior to
the third test an independent team of engineers working directly for the program manager
and without ties to the development contractor, compiled data on problem open and
closure rates and projected
150 priorities 1 & 2 problems would surface during the third
-
test and as a result the system would not receive a favorable fielding decision. When the
engineers presented the data to the program managers he stated that the development
contractor assured him the problems were known, were the highest priority, would be
fixed, and the system would pass test. The 3rd test resulted in 250 priority 1 &2 problems
being opened! This was higher than the independent engineer's projections and
completely out of line with the contractor assurances. Part of the reason the engineer's
projections were off was due to an assumption that the problems would become easier to
fix as the system matured and the contractor would get better at fixing the problems as
their processes became established. These hoped for improvements never materialized.
Problem Resolution Rates Drive
Over-optimistic finish projections
Problem Reports
0
CL
CE
E
A
o
0.
0
z=1
-+-
400
350
300
250
200
150
100
50
0 1 1
1
Priority 1 Problems
1
-i-
1 1
1
Date
2
Problems
-+-Cumm 1 & 2 Problems
Priority
66
Figure 22 - Problem Resolution Rates
The pressure to deliver a working system prior to Y2K roll-over continued, but now
drastic cuts in functionality and a strict redefinition of what would be counted as a
Priority 1 or 2 problem was implemented. The redefinition of the system delivery
removed many of the priority 1 &2 problems from the system by recasting them as future
enhancements and deferred functionality. The AOC system did not field until Apr 2001
after having gone through a Government In-plant test, a Functional Development Test
(FDT&E), a second functional development test (FDT&E 2), a joint factory acceptance
test (JFAT), a Development/Operational Test and Evaluation (DT/OT), a second
development/operational test (DT/OT 2), a Multi-service Operational Test and Evaluation
(MOT&E), and a second multi-service test and evaluation (MOT&E 2). In total seven
major test events involving 250+ users at each event, plus engineering and contractor
support were required to field the system. The original acquisition plan called for only
three formal tests.
While most of the system components worked well in the labs, the integration of the
system gave rise to a new set of emergent problems that was not planned for or managed
effectively. The inter-dependence of the components makes fixes and changes risky
since they are likely to have effects on other components, and the need to make multiple
changes simultaneously makes isolation and trouble shooting very difficult. The process
of successive fixes, integration and retesting becomes a very expensive approach to
system development and fielding. The testing treadmill has significant effects on the
operational community as well, since they have a military mission to perform in addition
to supporting test events. Deployment requirements to support operational C2 centers
and real world missions have already exceeded the Air Forces manpower capacity. The
additional burden of testing C2 systems in an almost hap-hazard manner is viewed with
extreme disfavor by operational commanders. The Commander of Air Combat
Command was quoted as saying "We will make improvements were we can at the
expense of long term modernization."
Perhaps the only way to identify the emergent system properties and apply fixes is to
do full scale testing. Improvements in this area of acquisition management practice are
67
needed and architecture which do a better job of reducing system complexity and interdependence can help limit the emergent system properties and improve test/fix cycles. A
more regimented approach to building and integrating in small increments and adding
capability to the baseline AOC system is the likely future path. Arriving at a baseline
system to build from is a major hurdle and still requires improvements in methods to
support future systems development.
Effects in a closed system
As costs on a program grow there are limited options for how to deal with the
problem. A program manager can kill the project as being unfeasible, but this is
extremely rare since it is a black mark on the program manager's career, and the
requirement would be left unanswered. Additionally, the program incurs close out costs
on the contract which are often close to projected costs to complete the project.
Additionally, the program managers feel a sense of responsibility for the development
team who will have to look for other work if the contract is killed. This makes killing
even a bad program very unlikely. An alternative is to request additional funding for the
program to cover cost growth. The DoD planning, programming, and budgeting process
allocates budget authority in 5 yrs blocks as described in the background section. This
budgeting process allocates all the available funds to projects and there is not a ready
source of additional funds to draw from for cost growth. Most programs have a
contingency pool to allow some internal flexibility and occasionally a program manager
can get funds from other program's contingency funds, but it is another undesirable
practice and viewed as poor planning on the program manager in need. The reality in
practice is program managers reduce functionality for a given build. This is within the
program managers discretion, and simply moves functionality to future versions of the
system. It also provides a means to increase requests for future budget allocations to
cover the increased costs, so in some regards it grows the program and can be viewed
positively.
In the realm of inter-operable systems, programs with external dependencies must
have some level of insight into and influence over changes in external programs content.
68
Ideally, the program managers have established relationships with external partners and
are actively managing the risks due to inter-dependencies. Cascading effects due to
deferred functionality are not generally catastrophic, but they can be very disruptive and
costly. As an example, an imagery exploitation application was being developed for the
government by a 3rd party commercial contractor. It was planned for delivery to a
number of C2 systems which had a functional requirement for the capability. One of the
receiving C2 systems fully committed to the application and assisted in the development
over a two year period, a second C2 system chose to use commercially available product
in the interim and planned to evaluate the new tool when it was ready. When the
application developer failed to deliver the imagery tool, the
1 st
C2 system was forced to
initiate a short notice acquisition strategy to replace the tool. The cost of the new plan
was approximately $4M and there were $2+M in sunk development costs lost. In the end
the C2 system program manager had incurred $4M in additional costs which displace
other planned for functionality and delayed fielding of capabilities to the users. The
impact of deferring other capabilities has not been traced, but can be expected to affect
functionality related to interoperability with external systems since these are generally
'lower' priority requirements to a program. The second C2 system on the other hand
invested -$750K in fielding the commercial application, provided limited functionality 2
yrs early and avoided the major disruption altogether.
Maintaining cognizance of the development activities of external partners is an
additional burden on program managers, but is absolutely essential to managing risks and
improving interoperability.
Summary
Complexity is a major issue which needs to be addressed and managed as part of C2
systems acquisitions. The complexity of the development task faced by program
managers is significant, and external dependencies play a growing role as interoperability
increases. When the complexity of a system design is too great it undermines confidence
and responsibility, and systems thinking is an antidote [8]. Better, more useable
architectural representation using DSMs is one way to facilitate systems thinking and
69
reduce complexity. Unfortunately, architectural representations are developed
independently, have inconsistent architecture and interface views between interdependent programs exist, and creating shared understanding of dependencies is still
problematic. Stability or changes of sub-systems and components are not sufficiently
isolated, and have downstream effects on external programs thereby increasing the need
for tight configuration management and schedule coordination while decreasing system
flexibility. Attempts to automate immature operational processes require greater than
planned degrees of discovery and iteration and can be expected to cause architecture
instability. Emergent behavior during integrated testing results in new problems which
are difficult or impossible to predict and must be resolved before the systems can be
certified and fielded.
Culture
"Talent and genius operate outside the rules, and theory conflicts with
practice."
Major General Carl von Clausewitz, On War
Incentives reward aggressiveness
The Air Force officer core is a highly competitive group and the importance.of
contributing to the mission and making an impact are strongly held values. This is
reflected in the motivation and dedication of the service members and it is reflected in the
rating and evaluation system. The Air Force Pamphlet (AFPAM 36-2404) on how to
write an officer job description for an evaluation is an enlightening example. The
pamphlet provides weak and strong examples of how to write a job description. The job
descriptions are very important because they are the opening paragraph of the officer's
annual evaluation which are compiled and reviewed by promotion boards. [9]
" Weak Job Description:
o Duty Title: Chief, Resources and Requirements
o Key Duties, Tasks, Responsibilities: Responsible for numerous aspects of
civil engineering operations, including construction and repairs of base
facilities and grounds maintenance. Supervises three personnel and
oversees work force. Manages large supply account for unit. Responsible
for unit vehicles.
" Strong Job Description:
70
o Duty Title: Chief, Resources and Requirements
o Key Duties, Tasks, Responsibilities: Plans, requisitions material, and
schedules civil engineering operations, maintenance, and repairs for base
facilities, including housing and over 5,000 acres of grounds. Directly
supervises three personnel and oversees a 37-person work force. Manages
multi-million dollar account for supplies and equipment. Also responsible
for civil engineering vehicles.
Strongest Job Description:
o Duty Title: Chief, Resources and Requirements
o Key Duties, Tasks, Responsibilities: Responsible for receiving, planning,
programming, material requisition, and scheduling of all civil engineering
in-service operations, maintenance, repair, and minor construction work
on 279 base facilities, 790 family housing units, and 5,100 acres of
grounds valued at $72 million. Oversees a work force of 37 people and
directly supervises 3 section chiefs. Manages the expenditure of $2.5
million for supplies and equipment to accomplish work. Also responsible
for all civil engineering vehicles.
When promotion boards review evaluations, the size of budgets and span of control
become easy metrics to use in deciding who gets promoted and who doesn't. The
officers who are rated by this system know the underlying criteria and mechanics of the
evaluation system. The message sent to officers is that bigger programs are better from a
career standpoint. Occasionally competition can encourage dysfunctional behavior. One
anecdote which illustrates the point comes from an instructor at the Defense Acquisitions
University.
"...when I was an Air Force major at Hanscom AFB, Mass. One of my
friends came back afterfinishing the ProgramManagement Course.
When I asked about the course, he said it was great (the course always
had a top reputationfrom the overwhelming majority of its graduates), but
that it was really competitive. He indicatedthat there was a lot of pressure
and competitionfor grades. He said students were expected to help their
work group and to work together on cases, so one had to be very clever to
providejust enough good help to get by, but keep others a bit confused on
the nuances. By giving or allowingjust enough misinformation in his
area of expertise, he could do better on the exams and have a better shot
at "A's"and top graduatedesignation. I was disappointedto hear the
system discouragedcooperationand encourageddysfunctional behavior,
which sounded like "cheating" other classmatesfrom optimal learning."
[10]
71
This should not be taken as a blanket condemnation of the system, but it is important
to recognize that incentives like 'top graduate designation' can result in undesirable
behavior. Aggressive officers will manage their careers according to their best interests
and seek opportunities to extend their span of responsibility and control. As a result more
aggressive officers are rewarded by receiving stronger written evaluations and ultimately
have greater promotion potential. In some circles 'aggressive' has a negative connotation
but in the military culture aggressiveness is a positive, and desirable trait. Aggressive
officers are go getters, they are leaders, they create opportunities and make things
happen, they make tough decisions, they are not afraid, timid or overly analytical, they
don't wring their hands and worry, they take action and live with the outcome. In a
speech by Anthony G Bohannan, a former senior British Army officer, he stated
"Military history tells us time and again that it is the commander who has the courage to
defy conventional logic and precedence and who reacts to his instincts contrary to the
available information and consensus, who more often surprises his opponent and wins."
While these are positive attributes for a battlefield environment and well matched to a
military culture, they are not necessarily the same set of attributes which are desirable in
military program managers who are part of that culture. Program managers should be
striving to adopt and implement acquisition best practices which bring order,
predictability, and higher levels of productivity and quality to their programs.
The implications for program managers who are part of both the military culture and
acquisition community are potentially counter-productive and have negative effects on
the DoD's capability to acquire systems for the best price. If managing a large program
that delivers more functionality and has a larger budget than a small focused program on
a short schedule is perceived as career enhancing, then officers will naturally compete for
them. Smaller, focused programs are less likely to be proposed or pursued within the
culture. This runs directly counter to findings of the Standish Group who found that the
probability of success for a software program is inversely proportional to its size and
approaches zero for projects costing $10 million or more. [11] Small projects tend to
succeed because they reduce confusion, complexity and cost. The incentive to have a
greater mission impact and control large budgets can be a driver in the opposite direction
72
encouraging project managers to take on additional functional requirements, pursue
advanced technologies, compress schedules for fast deliveries and adopt high risk
acquisition strategies. A General Accounting Office report on Major Management
Challenges and Program Risks reported that;
"We continue to find that the desire of programsponsors to keep cost
estimates as low as possible and to present attractive milestone schedules
encourages the use of unreasonableassumptions about the pace and
magnitude of the technical effort, materialcosts, production rates, savings
from competition, and otherfactors. For example... the originalschedule
for developing the Joint Air-to-Surface Standoff Missile was ambitiously
set at about half of what previous missile programs required. The schedule
was laterdelayed by 22 months, and total program costs increasedby
$500 million." [12]
Aggressive program management and competition among program managers within
the AF acquisitions career field where huge budgets are concerned and decisions early in
a program's lifecycle have long term implications for affordability, maintainability,
scalability, supportability... is entirely undesirable. It is generally accepted that changes
early in the product development process are much less expensive to deal with than
changes late in the process. Following a standard waterfall development model, changes
during requirements definition have almost no cost associated. Changes during the design
phase are relatively small, but as you move into implementation and unit test changes
cost approximately 10 times what they would have if caught during design phase and
increase to 100 times the cost if made during system test or following fielding.
Unfortunately, the incentives structure suppresses early identification of potential
downstream problems and encourages aggressive and ambitious planning to launch large
programs.
"The competition forfunding when a program is launched encourages
aspiringDOD program managers to include performancefeatures and
design characteristicsthat rely on immature technologies. In this
environment, risks in the form of ambitious technology advancements and
tight cost and schedule estimates are accepted as necessaryfor a
successful launch. Problems or indicationsthat the estimates are
decaying do not help sustainprograms in lateryears, and thus admission
73
of them is implicitly discouraged. There arefew rewardsfor discovering
and recognizing potentialproblems early in program development." [131
Ultimately, the officer evaluation system emphasizes "Impact on Mission" as a
primary rating criteria and should self correct if aggressive program management leads to
failed programs with negative impacts on the mission. Which leads into the next topic,
where program manager turn-over virtually guarantees that the program managers who
launch the programs are not held accountable for the outcome.
Turn-over eliminates accountability
Interviews with program managers and questionnaire responses showed that military
program managers on average are assigned to a program for 2.4 years. When compared
to the 5 year spending plans generated by program managers as part of the DoD
programming, planning and budgeting process we can immediately see that the sitting
program manager sets budgetary the stage for next 1 to 2 program managers to follow. A
lack of management continuity needed to oversee large-scale, long-term software
developments has adverse affects on program focus and progress. It also reduces or
eliminates accountability for decisions made during tenure as program manager.
Following the cost performance reporting data and program management changes over a
5 year period for a major C2 system acquisition is instructive.
The cost and schedule performance was based on discrete tasks defined in a Work
Breakdown Structure. The contractor provided the following data in a monthly Cost
Performance Report (CPR):
BCWS
Budgeted Cost of Work
How much effort was scheduled for
Scheduled
period
BCWP
Budgeted Cost of Work Performed How much effort was accomplished
ACWP
Actual Cost of Work Performed
for period
How much the BCWP eventually
74
cost
Table 2 - Cost Reporting Definitions
By comparing BCWS and BCWP the contractor's schedule performance can be
established. For example, if BCWP < BCWS, then contractor is behind schedule.
Similarly, BCWP and ACWP are compared to identify cost performance. If ACWP >
BCWP, then contractor is over planned cost. Cost and Schedule Performance Indices
(SPI and CPI) are developed from this data.
SPI= BCWP/ BCWS
If SPI = 1, contractor is on schedule
If SPI < 1, contractor is behind schedule
If SPI > 1, contractor is ahead of schedule
CPI= BCWP/ACWP
If CPI = 1, contractor is on cost
If CPI < 1, contractor is over cost/overrun
If CPI > I, contractor is below cost/underrun
This cost and schedule performance data is only a trend indicator. Unrealistic
contractor frontloading of planned effort, inaccurate reporting and program rebaselining
can adversely impact the value of this tool. If the contractor has performed well to date,
it does not guarantee future efforts, such as system testing, will not encounter
cost/schedule performance problems. While this is only a trend indicator, findings from a
study of DoD programs behind schedule and over cost reported that no programs
recovered.
The cost and schedule data from program launch through the tenure of the first
program manager for the C2 initiative studied is shown in the graphs in Figure 23 below.
The original program plan was for a $95M dollar development effort starting in Dec 96
and delivering its first version by Oct 98. Numbers in the CPR are in $K dollars and the
timeline is in months.
75
Cumulative Cost Performance Data
under PM 1
40000
3000e
BCWS
---
10000
-h-- ACWP
1
2
3 4
5 6
7 8 9 1011 121314
Cummulative Program
Schedule & Cost Variance under PM1
r-
0)
0
-+-1000
--
-1500
00--w-
Cost
Variance
Schedule
Variance
1.000
0.950
0.900
0.850
-
Ifl
Cummulative Program SPI & CPI
under PM1
-
co
BCWP
20000
0.8W
40
0.750
1 2 3 4 5 6 7 8 9 10 1112 13 14
-2000
Program
Manager 2
takes over
Figure 23 - Cost Reporting Under PM 1
It is worth noting that in month 11 'end of year money' was added to the contract to
put it back on track. The incoming program manager takes over a program that appears
to be recovering after some initial instability standing up the project. Only five months
after the PM2 took over the program was announces as an "Acquisition Reform Success
Story" by the Assistant Secretary of the Air Force (Acquisition), projecting a $6M
savings.
"The xxx (name deleted) program consolidatedthe effort of three separate
program offices and over 10 development organizationsinto a single
integratedprogram office with a single integrationand development
contractorand companion hardwarecontract (doing it better). This
consolidation reduced the manpower necessary to manage the program
(doing it cheaper) and increasedthe ability of the contractorto
accomplish development and integration--resultingin a fully integrated
system provided to Air Force combat units in FY 98 (doing itfaster)."
Air ForceAcquisition homepage, http.iwww.safaq.hq.afmil
Program manager 2 is in place for 20 months and toward the end of PM2's tenure the
program is 'rebaselined.' The practice of rebaselining adjusts for errors in the original
estimates for budgeted and actual work by increasing the BCWP. This has an instant
76
impact on cost and schedule variance. See graphs below. As program manager 3 takes
over the program is now in its
3 3 rd
of 36 months overall and was expected to deliver
version 1 at a total cost of $95M. The program has already expended $119M which is a
25.3% over-run, yet this is not apparent from the cost performance reporting graphs in
Figure 24 below. The cost over-runs are partly due to contractor performance and partly
due to requirements growth and promises for additional functionality.
Cummulative Cost Perfomance Data
under PM 2
140000
120000
100000
80000
-_-BCWS
BCWP
-4-
---
ACWP
20000
Cummulative Cost & Sched Variance
under PM2
Component
--
O- Integration test begins 0
'
i
-
Cumm ulative Program SPI & CPI
test begins
M
0.980
-1
2000
-3000
-4000
-5000
-6000
-u-
Sched Variance
Cost Variance
-7000
under PM 2
0.960
0.940
0.920
Original
Acquisition Success
story announcement
--SPI
+CP1
1.0 delivery
Program
Rebaselined
Program
Manager 3
takes over
Figure 24 - Cost Reporting Under PM 2
As Program Manager 3 takes over the program it is behind schedule and under
incredible pressure to enter operational testing. The series of full scale operational tests
reveal multiple problems which make the software unacceptable until fixed. To bring the
program back on track and actually deliver a product PM 3 was forced to 'descope' the
functionality delivered. Descoping essentially takes tasks off the contract and adjusts
BCWS. This helps to reduce the workload on the contractor, and focuses the team on
critical functionality. Descoping also reduces the functionality that was promised to
77
operational users by previous program managers and used to justify increased budgets.
Another effect of descoping is that less work has been completed for the actual costs
incurred (ACWP) and cost variance grows. This in turn forced two rebaselinings to
adjust BCWS and improve the cost variance. See the graphs in Figure 25 below.
Cummulative Cost Perfornance Data under
PM3
250000
200000
BCWS
150000--
-a-BCWP
100000
P
-i-ACWP
50000
0
M
LWtO
t-
O)
-
M~
Cummulative Cost & Sched Variance under
PM3
.
.
..1-
crQ
r-
Cummulative Program SPI & CPI
Under PM3
(D
1.050
1.000
-
0
-
LO
0
--- Sched
-5000
Variance
-10000
-'-
-15000
Cost Variance
0.950
0.900
0.850
0.800
-20000
C0
to
C.
) r
Program
Rebaselined
Program
Manager 4
takes over
Figure 25 - Cost Reporting Under PM 3
In the end PM 3, who was considered a rising star in the acquisition career field, was
forced to make some tough decisions, break promises with the operational users for
additional functionality; and had to request additional funding to cover additional testing
and rework. These actions put the program in a position to deliver a greatly reduced
product at a higher than expected cost. The program manager was able to avoid
cancellation on a program that had expended $196M against an original $95M dollar
budget and was now 16 months away from operational fielding and would be 36 months
late of the original estimate.
78
-u
Program manager 4 took over the program and the pattern of growing cost variance,
and rebaselining before a new program manager takes over continued. The product was
delivered to the user community seven months after program manager 5 took over. See
the graph in Figure 26 below.
Cummulative Cost Performance Data under
PM4
240000
220000
-+-
200000
BCWS
--- BCWP
-
ACWP
180000
16000026
1
2
3
4
5
6
7
8
9
Cummulative Program SPI & CPI
Cummulative Cost & Sched Variance under
PM4
1.000
1
2
3
4
5
6
7
8
0.995
0.990
9
0
-1000
CPI
Sched Variance
-2-3000
-4000
-a- Cost Variance
0.975
0.970
-5000
Rebaselined/
Program
Manager 5
takes over
Figure 26 - Cost Reporting Under PM 4
Overall, the pattern of over-optimistic cost and schedule estimates, growth in promised
functionality, increasing budgets, and slipping schedules are masked by the practice of
rebaselining. Taking the same cost reporting data and removing the rebaselining
adjustments we get a very different picture of the program's performance. See the graph
in Figure 27 below.
79
Cummulative Cost & Sched Variance (with
Rebaselining)
0
-10000
-20000
$
m COST VAR
-30000
$
n SCH VAR
-40000
-50000
-60000
_
_
_
_
I
_
_
_
_
_
I
_
_
I_
_
_
_
Cun mulative Cc st & Sche4 I Van ance
(Rebaselinir Adjustei I Out
0
-10000
-20000
$
m COST VAR
-30000
$
n SCH VAR
-40000
-500CO
-60000
PMl
PM2
PM3
PM4
PM5
Figure 27 - Comparison with and without rebaselining
Looking at the graph with rebaselining adjusted out we can see that cost growth really
begins to occur as the program moves into testing. The most likely explanation is that the
impact of poor design choices, over-optimistic estimates, coding defects and bugs are not
revealed to the government program managers until testing begins. The true scope of the
problem can only be understood when 380 plus software segments which make up the
overall program are integrated and tested as a whole. Fixing first order problems gives
rise to new problems and reveals still undiscovered second order problems. The
implementation decisions and acquisition approach established by PMs 1 & 2 have
tremendous implications for the success of the program. Early in the program planning
stages faulty assumptions and emergent problems are difficult to anticipate until testing
80
begins to reduce uncertainty and demonstrate the true capabilities of the developed
product. By the time problems are revealed the original program managers have usually
moved on and are no longer accountable. Not only are they protected from
accountability; they often miss the opportunity to learn from the experience of addressing
problems they may have created. The incoming program managers are thrown into a
heroic struggle to save the program.
View from the user/customer's perspective
The user community has been frustrated by the lack of responsiveness of ESC and its
ability to deliver systems. The systems generally take too long to reach the field, are too
costly, are difficult to support, and fail to meet user expectations. What is the cause of
this dissatisfaction? To start, technology holds the promise of improving operational
efficiency and program managers pursue opportunities to build those systems
aggressively. When program managers over promise and then have to slip schedules,
increase costs, and deliver less capability than originally promised, the organization's
credibility suffers.
In today's fast clockspeed development timelines the user community has come to
expect the latest technology delivered now, and for a competitive price. Many times
users are able to implement software solutions to local problems using the 'a couple of
guys in the garage' approach. User developed solutions are often a source of key
innovations which drive larger more formal programs. Harvesting the innovative
potential of lead users has been the subject of reseach by Prof Eric von Hippel.
"contrary to conventional wisdom, successful innovations are often first
developed and tested by the users themselves, "lead users," ratherthan by
the firms that arefirst to bring those innovations to market. "[14]
Leveraging user solutions can be a double-edged sword for program managers. First,
the lead users tend to create solutions tailored to their way of thinking about and
visualizing important information. They know the domain and challenges far better than
81
any team of software developers can hope to learn in a short development cycle. As a
result, the user developed solutions are often an excellent match to their particular
problem. Unfortunately, these solutions do not always scale well or translate into a
generalizable solution. Second, the lead users can implement a solution for extremely
modest costs. When 'professional' developers adopt the solution and deliver to a wider
user audience there are large overhead costs incurred. These costs are easily an order of
magnitude or more. Fredrick Brooks' explanation for this is enlightening. The user
developed 'garage solution' is generally a software program without documentation and
without formal system interfaces and specifications. Generalizing the program so that
other users can read, learn about and use the program increases the cost by a factor of 3.
Integrating the program and building interfaces can increase the cost by another factor of
3. See Figure 28.
A
A
3X
Program
Programming
System
(Interfaces,
System integration)
13x
A
Programming
Product
(Generalization, testing,
documentation, maintenance)
A
Programming
Systems
product
Figure 28 - Evolution of the programming system product [4]
In the end a solution developed and implemented by a user site costing $200,000, can
be expected to require over $2M to re-develop and is likely to deliver a less capable
(albeit supportable) solution. A program office taking on such a development effort runs
the risk of criticism for inefficient and ineffective management of development dollars
and the inability to even copy what was provided to them. The pressure to produce
similar results reusing a known solution can lead to unrealistic cost and schedule
estimates.
82
One final aspect worth considering is the immediacy of the users experience.
Operational commander's are presented with highly dynamic situations and are required
to make decisions and implement solutions on the fly. They see the world in very short
decision cycles and are used to near immediate feedback. When you connect the
operational community's fast feedback cycles to the acquisition community's product
definition and development process there is an immediate tension. Great ideas, concepts,
and innovations rising from the operational user community seem to take an inordinate
amount of time to find their way into operational systems. Causal loop diagrams are a
tool for system thinking and understanding the dynamics of feedback [15]. Figure 29
below is a causal loop diagram that illustrates the point.
Plannin
Acquisition
Community
J 4%
Strategy
Implementation
Operational
Community
6 months
to Years
Months
Pro c ed u re
Lessons Learned
Hoc
%
4Fielding
Operational
3
4
Operational
Solution
System
Warfight
Probem
Days or
Months
Immediate
Figure 29 - Operational Feedback Delay
83
Summary
The cultural tension between operational user community and the military acquisition
community is not conducive to building program management best practices. The culture
creates incentives and competition for increased span of control and influence,
recognition, rewards and promotions which can lead to adoption of high risk acquisition
strategies. The culture further shelters program managers from accountability for bad
decisions and reduces the opportunity to learn from experience. Finally the operational
culture tends to hold a negative image of acquisition professionals (a.k.a. acquisition
weenies) who do not understand the warfighters' needs, are inefficient managers of large
budgets, and unable to deliver the latest technical innovations in a timely manner. The
reality of the situation is that there are many dedicated and conscientious program
managers who are aggressively pursuing enhancements to the nation's warfighting
capabilities. Efforts to improve incentive structures, create accountability and manage
user perceptions could have lasting positive effects on military acquisition acquisitions
and help facilitate the adoption of strategies more in line with commercial best practices.
84
Recommendations
DSMs and Enterprise Architectures
Software projects are more difficult to manage than construction projects [161 and C2
programs are among the most complex and difficult software projects in the world today.
The human mind has a limited ability to deal with complex, probabilistic or dynamic
problems intuitively. C2 Program management decisions tend to fall into the complex
category with many variables and relationships to consider. The sheer number of subsystems and components in a C2 system make them extremely complex. The example of
a C2 system with 49 components and 2 options or choices for each component yields 249
= 5.6295 x 10" possible system configurations. Layer over that the unique functions and
potential incompatibilities inherent in certain combinations and the level of complexity is
beyond a program manager's ability to manage exhaustively. C2 program managers are
also forced to deal with the dynamics of schedules and trying to get the releases of
multiple components to deliver on time so they can be integrated into a unified system.
And there is always the probability that one or more of the systems will deliver late or
create unexpected conflicts within the system. Clearly, C2 program managers need tools
and strategies for managing and reducing the complexity of their task. This section will
provide some suggestions and recommendations to apply to help enhance the ability to
develop and deliver interoperable C2 systems.
Introducing enterprise interoperability metrics
If fielding interoperable systems is a goal for the military then being able to measure
progress on improving interoperability is necessary. There has been no systematic effort
to measure interoperability for the AOC system. One of the primary reasons is that
interoperability is an inherently multi-dimensional concept and there is no single metric
that will give a clear picture of the system's interoperability. Defining and applying a set
of quantitative metrics would be extremely difficult, with the increasing reliance on C2
systems and the need for improved interoperability, demand a more explicit attempt to
85
measure it. Rather than attempting a precise measure which would create controversy,
the thesis recommends a subjective interoperability measure based on practical
assessment and judgement. The basic measures are Green/Yellow/Red for a
Pass/Marginal/Fail assessment of the systems interoperability. This framework can be
applied in three ways. First, as a way to measure interoperability or compliance
assessment for issues like DII COE or the Joint Technical Architecture. Second to
measure information flows between applications or external systems as defined by the
system architecture. And third, to trace a mission thread and determine if the systems
support the end to end mission performance.
The first step in measuring interoperability is to determine where systems, subsystems and components are required to interoperate. Rather than 're-invent the wheel'
program managers will want to use information that is readily available. Adapting the
Design Structure Matrix (DSM) used in earlier in the architecture representations and
interface design would be a logical choice. The DSM in Figure 30 has been shaded to
highlight where all the interfaces exist. Color coding the DSM squares where an
interface exists as Green/Yellow/Red for Pass/Marginal/Fail is a quick way to visualize
the status of the system. In this case the DSM includes DII COE and external systems
and interfaces as well as internal system interfaces. Using the DSWM as an
interoperability metric tool is a great way to track progress and manage the development
activities on the system.
86
-
-
'~1
- -
Figure 30 - DSM for Interoperability Metrics
The Green/Yellow/Red assessments can be used for mission threads as well. The
overall assessment of the interoperability for a thread is equivalent to the worst
interoperability rating for an information path along the thread. In other words, a chain is
only as string as its weakest link. The mission thread for a mensurated point or Desired
Mean Point of Impact (DMPI) was discussed in the background section and provides an
excellent example (Figure 3). A simplified data flow diagram is provided here in Figure
31. From a program management standpoint, five different program managers and six
different systems are involved in providing systems to support this operational thread.
Each program manager and system has a different set of requirements and priorities, so
building consensus on the thread is difficult. It is also noteworthy to point out that some
of the interfaces are not under the control of any single program manager. The use of this
kind of simplified diagram becomes an interoperability tool and is extremely valuable for
building common understanding and determining what negotiations on interfaces must
take place.
87
PM2
PM 1
System
RD
1
Annotated
Image
...................................
Spreadsheet
~.......
............................
PM 4P
PM 3
System 3
Excel
System 2
AOC
ysm4System
5Arra/
MPS
System 6
Aircraft/
Weapon
Figure 31 - DMPI Flow
The use of DSMs and thread diagrams as metrics tools is suggested here to evaluate
systems interoperability. These are simple yet powerful tools for program managers.
The fact that the interfaces for a system architecture or mission thread can be formally
identified and characterized allows for reporting on progress. Error! Reference source
not found. show 246 total interfaces with 179 green, 40 yellow, and 27 red. The system
has achieved 73% of its interoperability requirements and can formally report that metric.
More importantly it can focus on areas where interoperability has not been achieved and
search for trends. Sub-systems and components which are having problems integrating
are highlighted and can be added to a risk management watch list. Figure 31 tells a story
of interfaces between systems that are not being managed. It also highlights the need for
coordination and collaboration between programs across the enterprise. For the 7
interfaces along the mission thread only one has been satisfactorily implemented. Here a
C2 node manager or enterprise manager can assign responsibility for un-managed
interfaces and provide direction to individual program managers to work together on
solutions. Important organizational and teaming relationships can be formed aligned with
system interoperability requirements.
Collaboration for enterprise decisions
Simply establishing interoperability requirements and documenting them in systems
architectures does not insure interoperability will occur. A deeper level of cooperation
88
between sub-system and component developers may be required to achieve
interoperability goals. It is particularly important when a program office does not have
direct control over the interface either because the boundary between sub-systems lies in
a gray area where no explicit control or authority has been established, or where the
interface is between two sub-systems which are outside of the developers control.
Using an abbreviated DSM to assess the AOC system interfaces as documented in the
system specification provides very important insights. The AOC architecture has
requirements for interfaces among sub-systems and therefore has a vested interest in their
interoperability, but a number of the sub-systems and components which are part of the
architecture are built and delivered by external program offices. In the DSM below,
names of sub-systems being delivered by external program offices are highlighted in light
blue. The interface requirements are marked by ones in the DSM cells and are based on
the AOC System Specification. The cells where interfaces are required can then be color
coded based on the level of control the AOC program office has over the interface.
Green indicates the program office has complete control over the development of both
sub-systems which are interfacing. Yellow indicates the program office is in control of
one of the systems, and can negotiate with the other sub-system developer to build to a
common interface. Red indicates the program office is in control of neither sub-system
development and therefore must rely on the delivered systems to work together or must
build a specialized interface after delivery. The red interfaces are extremely undesireable
and can create problems.
DIO
AODB MIDB
USMT
JMTK SM
TE
Wx
AD
DP
TW
ACP IDM
TAP
EM
RM
SAA
TCT
Dli COE
AODB
MIDB
l
I
USWF
.t
S ctrum Ma nent SM
7Te& EvaDa
MTe
t
Airspce Deconffiction (AD)E
-1
-
Theater Ax
Targein
andlniT
Weaom
(I
Air Can ' n Phonng ACP)%
(IDM
Manaent
Data
Intelftence
Theater Air Plannin (TAP)
ment
Execution Ma
enent (R
Resource Ma
S9lhaionatAwareness and Assessment SAA
(rCr
Tre Cnr Twpeft
Figure 32- - Interface Level of Control
89
The DSM above is a good representation of the AOC system and shows 18 subsystems with a focus on the mission applications and critical infrastructure of the
originally planned system. Of the 18 sub-systems 10 were under AOC program office
control. The breakdown of "Interface Level of Control" for all 59 interfaces is 20 are
green (34%), 26 are yellow (44%), and 13 are red (22%). The program office had direct
control of only one third of the interfaces required to build the system! This metric is a
potential indicator for program success and likelihood of cost growth.
This DSM provides a blueprint for collaboration on system interfaces. Green
interfaces can be handled through internal working groups. Yellow and Red interfaces
require a different level of collaboration and coordination. At a minimum, the program
office should have memorandums of agreement with the programs for all interfaces in the
yellow category, and requirements or funding lines established with programs for
interfaces in the red category. Logical sets of interfaces based on a partitioning of the
DSM could be used to establish focused interface control working groups and funding
vehicles to meet these interoperability requirements.
The DSM for interface level of control Figure 32 and mission thread diagram Figure
31 highlight the decision space where collaboration and negotiation must occur. The
DSM and thread diagram aide the process of generating common understanding and
building consensus to resolve interoperability issues. There is ample literature available
on processes for organizing and running effective meetings. The real value to a program
manager is knowing which meetings are important and where to invest limited time and
resources. The DSM for "Interface Level of Control" provides an important tool for
identifying and managing interface collaboration.
Interfaces, isolation layers, and information hiding
As early as 1972 David Parnas introduced the idea of information hiding information
as a way to modularize software designs and improve adaptability [17]. The concept was
based on anticipating where changes are most likely to occur in the software architecture
and modularizing around software components that would undergo change. Papers
produced as recently as 2001 by the Software Engineering Institute support the concept
90
of incorporating mechanisms in software architecture to accommodate change and allow
for future upgrade options. What is not addressed by these approaches is the fact that
modularization can have significant cost implications for the software development, can
delay the initiation of coding until the standardized interfaces are designed, and can
become constraining and costly to change. The approach recommended in this thesis is
to leverage the DSM of a software system to evaluate where modularization mitigates
risks due to change, and provide a tool for selecting and implementing information hiding
strategies.
The DSM describing Interface Level of Control (Figure 32) can be rearranged to
place externally developed software components at the top of the matrix. This provides
a better organization of the interfaces in terms of green, yellow or red levels of control.
Knowing what percentage of external interfaces are not under your direct control is a
critical measure and should be assessed and monitored as a part of any software project
[18]. Following the rearrangement an analysis of the interface between the Targeting and
Weaponeering (TW) and Modernized Integrated Database (MIDB) is traced to determine
how decisions on that implementation can affect other interfaces within the AOC system.
Figure 33 shows the trace and inspection shows that TW must negotiate three interfaces
for data output to AOC components for TAP, EM, and ACP, and on input interface from
ACP. Additionally, there are common data needs where ACP and TAP are building
direct interfaces with MIDB as an additional path to receive targeting data. The overall
result is six specialized or integrated interfaces within the AOC system for targeting data
exchange.
JMTK MIDB USMTIDP
D1
DIICOE
JMTK
MIDB
USMTF
Defensive Planning (DP)
Situational Awareness and Assessment (SAA)
Targeting and Weaponeering (TW)
Threat Evaluation (TE)
Time Critical Targeting (TCT)
Weather (Wx)
Spectrum Management (SM)
Airspace Deconfliction (AD)
AODB
Theater Air Planning (TAP)
Intelligence Data Management (IDM)
Execution Management (EM)
Resource Management (RM)
Air Campaign Planning (ACP)
SAA
TW
TE
TCT
Wx
SM
AD
AODB TAP
IDM
EM
RM
ACP
1
'1
1
1
91
Figure 33 - DSM Interface Trace
By introducing a set of design rules for target data exchange in the form of a
standardized data structure, the set of six unique interfaces can be combined into a single
common interface for input/output of target data. The design rule does not eliminate the
requirement for data to flow, however it does introduce a level of isolation between the
independently managed programs. See Figure 34. With the design rule in place,
changes on either side of isolation layer can take place without the need for formal
coordination. Upgrades to components are simplified and require only an abbreviated
regression testing to ensure the design rules have not been violated. Furthermore the
design rule effectively hides the detail and complexity of components on the other side of
the isolation layer and allows developers to build to an interface which should remain
stable over time. The DSM can be used to evaluate how the risks associated with
interfaces which are not under the direct control of the program office can be mitigated
through the implementation of new design rules. Provided the design rules are
implemented correctly and adhered to, they provide isolation between component
changes [19].
DII
DI1 COE
JMTK
MIDB
USMTF
Defensive Planning (DP)
Situational Awareness and Assessment (SAA)
Targeting and Weaponeering (TW)
Threat Evaluation (TE)
Time Critical Targeting (TCT)
Weather
x
MIDB-TW Interface Rule Set
Spectrum Management (SM)
Airspace Deconfliction (AD)
AODB
Theater Air Planning (TAP)
Intelligence Data Management (IDM)
Execution Management (EM)
Resource Management (RM)
Air Campaign Planning (ACP)
JMTK
MAD
LISMI DP
SAA
TW
TE
TCT
Wx
SM
AD
AODB TAP
IDM
EM
RM
1
Figure 34 - Introduction of TW-MIDB Design Rule
The design rules require significant input and coordination from multiple subsystem
developers, but once established become stable and provide a standard for new external
systems to build to as part of a growing enterprise. The design structure matrix provides
92
ACP
a means to identify and evaluate where introduction of design rules are likely to have the
greatest impact to reduce risk due to changes and improve the system flexibility.
Incentives
Producing a cultural shift and establishing an organizational mindset that values
interoperability and provides individuals with strong incentives to pursue interoperability
is an important undertaking. The Total Quality Management movement was extremely
strong in the 1980s and 1990s largely due to then Secretary of Commerce Malcolm
Baldrige's support and championing of quality initiatives. His commitment to quality
was formalized with the creation of the Malcolm Baldrige Quality Award. The idea was
to promote total quality management practice and to encourage American businesses to
practice effective quality control in the provision of their goods and services. The Award
was enacted by congress and signed by President Regan on Aug 20,1987 [20]. The
Malcolm Baldrige National Quality Improvement Act of 1987 helped to lead a shift in
U.S. businesses practice and nurtured the quality movement. The name of the award and
formal act of congress demonstrated buy in and support for quality at the highest levels of
the government.
There is no reason why a similar effort could not be pursued at ESC or the DoD level
to lead a shift for C2 program development in the pursuit of interoperability excellence.
C2 systems are a critical component of our nation's military capability and the goal of
increased interoperability is a key to the military transformation being pursued by
Secretary of Defense Rumsfeld. In a speech on the State of ESC by Lt Gen Leslie Kenne
she stated,
"Integrated C2 is one of Secretary Rumsfeld's top priorities. He
recognizes transformationwon't be successful without integrated
command, control, intelligence, surveillance and reconnaissance. We're
going to help him achieve that here at ESC...."
The State of ESC address 2001
Lt. Gen. Leslie F. Kenne
Electronic Systems Center Commander
93
By establishing the 'Rumsfeld C2 Interoperability Excellence Award' a strong
message would be sent to the acquisition community on the importance of
interoperability. The creation of the award would help improve C2 system
interoperability by:
" stimulating program managers to improve the interoperability of their systems
for the pride of recognition while increasing the capability they deliver to the
warfighter
* recognizing the achievements of programs which find ways to improve
interoperability for their military customers and provide an example to other
programs
" establishing guidelines and criteria that can be used to evaluate the
interoperability of C2 systems and the processes used to achieve results
* providing specific guidance to program offices to learn how to manage C2
programs to achieve interoperability excellence and making available details of
how winning programs were able to change their cultures and processes
The award process would require C2 programs to submit a formal write up and
nomination package to provides an overview of the program, and describe the
management practices, processes, teaming relationships... used and how they contributed
to the successful outcome. Categories of C2 systems based on size of the program, and
the nature of the system (i.e. communications, decision support, visualization, analysis
and correlation, sensor systems...) would be established, and an annual review board
would review packages and select winners.
Possibly the most valuable aspect of such an awards program is that it becomes a self
populating collection of interoperability challenges, alternative approaches and source of
best practices for future programs to learn from.
Creating accountability
One of the barriers to delivering programs on time, on budget, and with the
functionality promised is the fact that most C2 system development programs are
94
launched by one program manager, developed over a number of years under a series of
managers rotating onto and off of the program. The case study program had an average
program manger tenure of 14 /2 months. When a program finally delivers a system to the
field, it has undergone a number of changes and quite probably significant technology
upgrades. The program managers who sign up to the functional requirements, select the
contractors, establish the budgets, and schedules and review the initial designs are
virtually absolved of responsibility to see the project through to completion.
The idea of keeping a program manager assigned to a program is not part of the
current thinking, and runs counter to many of the evaluation criteria used in promotion
boards. This would be potentially career damaging to officers in the program
management field. An alternative is to make program life cycles shorter. This is actually
very positive move and consistent with fast clockspeed development cycles discussed by
Charles Fine [21]. The Standish Group has a long history of investigating the causes of
software project failures and developed a framework for project success and the trend is
improving.
"The reason most of these projectsfailed was notfor lack of money or
technology; mostfailedfor lack of skilled project management and
executive support." [22]
1994
1996
1998
2000
Succeeded
16%
27%
26%
2%
Failed
31%
40%
28%
23%/
Challenge4
53%
33%
40%
49%
While the overall number of projects being launched is increasing, the average size of
software projects is declining and success rates are much higher on small projects.
According to the Standish group's findings, a 1998 project that had a labor rate under
$750K had a 55% chance of delivering on time, on budget, with all the features and
functions originally specified. As the labor rate rose above $750K the success rate went
down to 33% and as the cost approached $10M the success rate went to zero. This does
95
not mean that a large project can not be built and delivered, it simply means it is virtually
impossible to build it to the original specifications on the budget and timeline planned.
The idea of incorporating 'micro-projects' into the C2 system acquisition process
helps to solve a number of problems. Let's define a micro-project in the C2 domain as a
project that must be completed with six or less people and delivered in six or less months.
The small size of the projects has an immediately impact on reducing complexity of the
programs and solves the accountability problem for program managers, since they can
potentially propose, develop and deliver multiple projects in the course of a single
assignment. The cost, schedule, and performance metrics for a micro project can be
established by a single program manager and feedback from users can be garnered as part
of a program managers evaluation. Incorporating a 'micro-project' approach as part of a
larger system program office has many interesting implications. The six person, six
month development cycle at a $20K/staff month sets a standard project at $720K. A mid
sized system program office with 10 program managers could plan for 20 projects a year
and a $14.4M annual budget. Foreknowledge of user needs and requirements is not
necessary to submit a budget request, and the system program office has the flexibility to
respond to emergent and changing needs of the user community rapidly. Forming a close
relationships with the user community could be used to generate concepts for and select
projects and provide tailored solutions to meet specialized mission needs while educating
the program managers on the mission areas and problems faced by end users.
Additionally, the small scope of the projects allows them to be treated as virtually
disposable commodities. Since there is not a massive capital investment in the microproject development, the need for long term sustainment becomes obsolete in favor of
redesign and migration to newer technology. As a track record of small programs builds
performance trends for individual program managers, program development teams,
mission areas, and technical approaches forms the basis for evaluations, promotions, and
risk management.
To support a micro-project acquisition approach requires a consistent infrastructure in
place to deploy the micro-projects on. A solid architecture and set of development and
interface standards would be needed to support the micro-project development. This is
itself a large complex system that cannot be effectively broken into micro-projects.
96
Experienced program managers and long term civilian and contractor support should be
employed to develop and maintain the infrastructure. The micro-projects become a
training ground for new program managers, giving them time to gain experience building
micro-projects on the infrastructure with the most successful program managers moving
up to manage the infra-structure migration and sustainment. While micro-projects are not
a panacea, they do provide an interesting alternative to the massive multiyear program
developments with integrated sub-systems and components as part of the delivery.
Conclusions
The track record of C2 systems deliveries has been plagued by cost over-runs,
schedule slips, and reductions in performance delivered. While there are a number of
commercial best practices that when applied have been demonstrated to improve results,
the gap between industry best practice which are usually demonstrated on small projects
and what is actually practiced on large complex projects is tremendous. The introduction
of a focused set of tools and methods for use in large complex C2 systems developments
to improve the success rates is a worthwhile effort. A relatively simple set of program
management tools that can be used to help manage the complex nature of interdependencies between systems and build cross program understanding is needed to
improve the cost/schedule/performance metrics and overall interoperability of the C2
systems under development.
The research results point to Design Structure Matrices as an effective tool for
modeling information flows within Command and Control systems as a way to represent
system architectures in a more useful format. The DSMs then become a tool for
analyzing and identifying the high risk components in the system, measuring progress on
interoperability requirements, evaluating program office level of control over interfaces,
and an approach for determining where to introduce design rules to decouple external
interfaces and mitigate effects of changes on the system. When coupled with the
recommendations to provide incentives for cooperation in delivering interoperable
systems and introduction of the concept of micro-projects as part of an evolutionary
acquisition management approach, the potential to improve the success rate of delivering
interoperable C2 systems components appears promising.
97
98
Future Work
There is a great deal of research work left in the area of Command and Control system
interoperability. This thesis focused on the system complexity and acquisition culture
barriers to interoperability, and is far from an exhaustive analysis of the subject. There
are clearly a number of other barriers and approaches that should be explored. The
logical next step to derive value from this work lies in applying the recommendations on
real C2 programs and testing the validity and utility of the recommendations. Developing
and using Design Structure Matrices for current programs as part of an ongoing dynamic
program will be extremely instructive. Changes in C2 programs are frequent and difficult
to keep up with. The thesis was based on historic data and took a snap-shot in time of the
systems architecture to use for analysis. Building DSMs under the often chaotic
changing circumstances of a C2 program may prove difficult. The lessons learned from
practical application of DSMs to represent information flows and define interfaces needs
to be researched and documented. Demonstrating the utility of DSMs in supporting
architecture decisions, determining where and when the DSMs are most useful in the
development process, how understandable and presentable DSMs are to large groups, and
how the benefits of improved understanding and management of systems interfaces
translates into improved interoperability needs to be determined.
A more detailed quantitative analysis of the survey results would be a useful to look
for correlations and derive value from the survey information collected. Additionally,
some of the preliminary research into emergence leads in an interesting direction for C2
systems themselves. The application of theories and technologies in the areas of Self
Organizing Networks, Complex Adaptive Systems, Swarm theory and Swarm
intelligence could profoundly change the definition of C2 systems interoperability. One
of the tenets of airpower is Centralized control and decentralized execution [3]. The
commonly held mental model of command and control treats the military as a large
organism where C2 systems form the nervous system. Swarm theory and self organizing
networks provide an alternative paradigm where weapons platforms and sensors would
be treated more like ants in an ant colony or bees in a bee hive. The platforms would act
as autonomous agents which follow simple rules and exchange information directly. The
99
system then becomes a swarm of independent agents vs. a large complex organism. The
swarms then seeks and arrives at locally optimal solutions through a process of mutation,
evolution, and survival of the fittest instead of centralizing and processing a massive
amount of information, making decisions and providing instructions. This concept leads
to a radically different view of interoperability and is an interesting area for future
research.
The primary goal of the thesis is to lead a shift in thinking and practice of the
acquisition community to meet the demands of improved interoperability. Applying the
recommendations in a measured fashion to create that shift and monitoring the results is
the next phase of work planned by the author.
100
Appendix A - Questionnaire
Note: Compiled datafrom the survey are availablefrom the thesis author, however
personal datafrom respondents and contact information will be maskedfor
confidentialityreasons. Send data requests to sfrey@mitre.org.
C2 Program Acquisition Questionnaire
The following questions are designed to explore the values, beliefs and mental models held by individuals
within C2 acquisition community. Personal contact information is for follow up purposes only. All your
answers will be held in strictest confidence and personal data will be removed from the compiled results.
Your participation is voluntary and deeply appreciated.
Instructions:
Use the Tab and Shift+Tab keys to move between fields. You may also use the mouse and click on the
shaded area to select it. When you have completed the survey, please save the file by selecting Save As on
the File menu. Please name the file using the convention of Lastnamesurvey.doc {e.g. Frey-survey.doc}.
Please return the saved file as an email attachment to sfrey@mit.edu.
Personal Background Data (will be held confidential)
B1. Name
Rank/Grade
Work Phone
e-mail address
B2. Who do you work for?
EJ
Civilian US Government
Military US Government
Commercial Contractor
E]I Other (please specify):
Organization name:
B3. Which best describes your current position?
Senior Manager
Program Manager
__
Team Leader
Program Management Staff
101
[:1
Contractor Support Staff
E_
Technical Staff
Other (please specify):
E]_
B4. On what activities do you currently work? (Please mark all that a ply)
El Program Management
El Requirements Management
El System Architecture
El System Engineering
El Software Design
El Software Coding
El Testing and Integration
El Quality Assurance
E]
Fielding, Training, and Support
El
Configuration Management
Security
Budgeting & Finances
Administrative Tasks
Other (please specify):
El
El
El
El
-
B5. What is your total experience in program management and systems acquisition?
El Less than 1 year
El Between I and 3 years
El Between 3 and 5 years
El Between 5 and 10 years
El Between 10 and 15 years
El Between 15 and 20 years
El Over 20 years
Questions
QI. What aspects of a C2 program make it preferable/career enhancing to work on?
Strong
Pref
No
Prefqrence
El LI El El El []
Strong
* Pref
Large Budget
Small Budget
1:1 El
Short program lifecycle
El Eli E] ElEl
Uses proven & commercially
available technology
Based on established/mature
operational concepts & methods
Incremental enhancements to common
El El El El El El El El El
Employs Leading Edge Technology
El E] El El El El l E El0
Defining revolutionary operational
El El El El E El El El El
concepts & methods
First attempt to solve a unique
El El ElE
processes
Limited number of external interfaces
Long program lifecycle
problem
E El E E
El
El
Complex engineering and system
challenges
Loosely defined system performance
____E_____E______E________integration
Clearly defined system performance
metrics
El El E E E E E
metrics
102
Tight accountability and schedule for
E
1 E E E E 1
Minimal accountability and flexible
I delivery schedule
product delivery
Q2. On average, how many years do you spend working on a single acquisition program before moving to
a new program?
Enter number:
Q3. Please provide your opinion on the following statements as they relate to ac uisition programs.
Often
Occasionally
Usually
N/A
Rarely
Cost deviations are avoidable
Schedule deviations are avoidable
Performance deviations are avoidable
SW development metrics are valuable
Risk management is used effectively
Requirements are stable
System architectures are well documented
and widely understood
System complexity is manageable
It is important to maintain awareness of the
El
11
El
El
El
El
EEl
El
E
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
El
E
El
El
I
L
El
El
E]
El
E]
R
El
E]
El
E
l
[]
E]
El
El
El
El
El
El
El
Always
E]
El
E]
L
E]
El
El
[1
[1
E]
E]
]
[i
]
EF
status of other programs within the enterprise
It is important to coordinate program
decisions with other programs in the
enterprise
Programs share common enterprise
integration goals & objectives
Program managers focus resources and effort
on their program's goals
Program managers have the flexibility and
authority to support C2 enterprise integration
goals
Individual programs cooperate to achieve
enterprise goals
Acquisition policy provides incentives for
enterprise cooperation
Q4. What factors contribute to cost and schedule growth?
Overly optimistic estimates
Low developer productivity
Requirements instability
Discrepancies, bugs & rework
Feature escalation
Low technical maturity/high technical risk
of components
Changes to incorporate new/better
technology
Downward directed changes
Rigid/inflexible architectures
Deficiencies/problems with third party
deliveries
Re-planning/re-work to accommodate third
party variations
N/A
Small
Contributor
Moderate
Contributor
Significant
Contributor
Heavy
Contributor
Primary
Cause
iT
El
El
fl
El
El
El
El
El
E
El
El
E
E
E
El
El
El
E
El
El
El
El
El
El
El0
El
El
F
El
El
El
El
El
El
E
E
l
E
E
l
E
El
El
El
El
E
El
___0_
El
El
IIIa
0F
El
_
1El
I
103
F
El
Eli
U
El
E
LI
l
El
El
El
E
El
E1
U
E]
Eli
U
El
[l
E
U
El
[]
E]
U
Slow/Bureaucratic decision making
El
processes
Poor decision based on inadequate
information
Management/staff turn-over and lack of
E]
continuity
Communication barriers / lack common
understanding of issues
Other (please specify):
Other (please specify):
El
El
El
El
Q7. What factors contribute to reductions in delivered functionality?
Small
Moderate
Significant
Heavy
Primary
Contributor
Contributor
Contributor
Contributor
Cause
U
U
U
U
U
N/A
Cost over-runs
Schedule slips
Requirements instability and changes
Unanticipated technical problems
High risk components failed to deliver
Architecture limitations encountered
Budgets cuts
Program re-baselining
External partner program budget cuts or re-
U
U
U
U
U
U
U
U
1U
U
UU
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
U
baselining
Other (please specify):
Other (please specify):
Conclusion
C1. Please use this space for any additional comments.
Thank you for taking the time to complete this survey. If you are interested in receiving a copy of the
survey results please indicate so below.
IU IYes, please send a copy to my e-mail address provided above.
U No thank you.
Steven E. Frey
MIT System Design & Management Fellow
104
105
Bibliography
1.
Coakley, T., Command and Controlfor War and Peace. 1991, Washington DC:
National Defense University Press.
2.
JCS, D.J.S., DoD Dirctionaryof Military and Associated Terms, in JP 1-02, Z.A.
S A Fry, Editor. 12 April 2001.
3.
SECAF, S.O.T.A.F., Air Force DoctrineDocument 1, in AFDD 1-1, M.E. RYAN,
Editor. 12 Aug 1998.
4.
Brooks, F., The Mythical Man-Month. 1995 ed. 1995: Addison Wesley.
5.
Parnas, D., Software Aspects of the Strategic Defense Initiative. Communications
of the ACM, 1985. 28(12): p. 1326.
6.
Rechtin, E., The Art of System Architecting. 1997: CRC Press.
7.
Bachman, F. Managing variabilityin software architectures.in Symposium on
software reusability. 2001. Toronto Canada.
8.
Senge, P., The Fifth Discipline. 1990, New York: Currency & Doubleday.
9.
HQ AFPCIDPP, S.o.t.A.F., Guide to USAF Officer Evaluation System, in AFPAM
36-2404, H.A.D.C.S.S. Lerum), Editor. 1 December 1996.
10. Beck, D.A.W., Probing the "It Depends" Variables, in ProgramManagement.
2001. p. 22.
11. Johnson, J., Turning Chaos into success,. 1999, The Standish Group.
12. Walker, D.M., MajorManagement Challengesand ProgramRisks, Department
of Defense,. 2001, General Accounting Office.
13. Walker, D.M., Major Challengesand ProgramManagement Risks, Department
of Defense,. 1999, US General Accounting Office.
14. Hippel, E.v., Innovation by User Communities: Learning From Open-Source
Software, in MIT Sloan Management Review. 2001. p. pp. 82-86.
15. Sterman, J., Business Dynamics. 2000: McGraw Hill.
106
16. Roetzheim, W., Turning Around Troubled Software Projects, . 1999.
17. Parnas, D., On Criteriato be used in decomposing systems into modules. ACm,
1972.15(12): p. 1053-1058.
18. Brown, N., The ProgramManager's Guide to Software Acquisition Best
Practices. 1995: DoD.
19. Sullivan, K. The Structure and Value of Modularity in Software Design. in Joint
InternationalConference on Software Engineering. 2001. Vienna.
20. Congress, U., The Malcolm BaldrigeNational Quality Improvement Act of 1987,
in HR 812, O.H.C.o.t.U.S.o. America, Editor. Aug 1987.
21. Fine, C., Clock Speed. 1998, New York, NY: Perseus Books.
22. Johnson, J.H., Micro Projects Cause Constant Change,. 2001, The Standish
Group: West Yarmoth Mass.
107