American Institute of
Aeronautics and Astronautics
CASE 2012
Complex Aerospace Systems Exchange
Pasadena, California
September 11-13, 2012
Summary of Proceedings
1
Executive Summary
This document summarizes the program, presentations, and discussions that comprised the AIAA CASE
(Complex Aerospace Systems Exchange) 2012 conference that was held at the Sheraton Pasadena, in
Pasadena CA, 11-13 September 2012. AIAA’s first CASE Conference was created to provide a forum for
aerospace professionals who are directly engaged in leading, developing, producing, and supporting the
industry’s most advanced vehicles and systems. With the emphasis on information and idea exchange
rather than technical paper presentations, the sessions were comprised largely of informal
presentations from engineering leaders to share their perspectives, complemented with panel
discussions to explore technical and programmatic issues. The conference program was organized into
three tracks:
Track 1: Complex System Development
Track 2: Integration, Test, and Verification of Complex Systems
Track 3: Program Management to Achieve Robust and Resilient Systems
There were a number of themes that arose from the discussions. Today’s complex systems
development is challenged by several factors. We continue to push the limits of technology to deliver
greater performance in all segments of aerospace, in an environment that is increasingly risk averse.
New technical disciplines (e.g. cyber) are being integrated into legacy and emerging systems, while
subsystems are gaining in both complexity and interdependency. Aircraft guidance, navigation, and
control (GNC) systems interact directly, and sometimes autonomously, with global command, control,
communications, computers, and intelligence (C4I) systems. High tech sensors and feedback loops are
embedded in all levels of components to enhance performance but their effects are not always fully
understood or characterized by the product designers.
Traditional systems engineering approaches provide a general methodology, but often break down
when applied to systems so complex that it is not possible to fully characterize all of the subsystems
under all potential conditions. Contributing factors include the remoteness and extremity of the actual
operating environment, the inability to integrate with critical aerospace infrastructure during
development due to considerations for safety and security, the unpredictability of human interfaces, or
just the inexhaustible number of circuit and software logic paths of the system. Even as we bring on
more high power computing and continue to advance simulation capabilities, there is no substitute for
testing, early and often, but ‘test as you fly’ is often not fully feasible.
When failures occur or out of phase technical issues are discovered, along with the subsequent root
cause analyses and corrective actions there are investigations to determine where the process went
2
wrong. Inevitably, breakdowns in the execution of the traditional systems engineering process discipline
are identified, to which we respond by embedding more checklists, reviews, approvals, and analysis
tools, as well as organizational changes, better communications, etc., all wrapped up with the promise
to do better next time. The findings are real and there are always opportunities to improve our
engineering rigor. That said, we should be open to new techniques to supplement our conventional
systems engineering methods to address the challenges of complex systems.
Not all of the opportunities will lie in traditional engineering disciplines. Research to replicate the
evolution of biological systems has led to real applications for technical design. Embracing the power of
social media through crowdsourcing is also proving to be a viable model for design development in
other industries. These are just a couple of examples that demonstrate the innovations that can come
from opening up our engineering aperture and looking to other fields of study for potential solutions.
CASE 2012 also explored the numerous issues associated with the test and verification of complex
systems. Discussions highlighted growing concerns with the domestic test infrastructure and the need
to modernize aerodynamic test facilities and ranges, especially to meet the needs of advanced netted
systems and hypersonic vehicles. Distributed testing offers flexibility, but requires additional planning,
validation, and technical coordination. Challenges also remain in determining the optimal balance of
simulation and testing, testing at the appropriate levels of system maturity, and the need to advance the
integration of design and test. Early engagement with test leads and facilities offers opportunities to
optimize testing and avoid problems. The AIAA Aerospace System Integration Working Group (ASI-WG),
is developing a standards document that will provide comprehensive guidance for all aspects of systems
integration, and a number of useful references have been identified (See the results for Track 2 Session
3 on Verification and Validation Issues).
Track 3 focused on the challenges of Program Management, but the associated issues arose in many of
the program sessions. Many comparisons and contrasts were made between the management of
commercial vs. military systems, but all are facing the difficulties of integrating work across
geographically and culturally remote enterprises. Numerous discussions delved into the balancing of
risk with cost and schedule pressures, while developing the most technologically advanced systems.
Even though there is intrinsic uncertainty, there is no tolerance for the associated risk. Strong
communications to maintain the situational awareness of the entire program provides is a key success
factor.
Overall, the CASE 2012 conference provided an open forum to raise and address the challenges
associated with complex aerospace systems development. The discussions may have identified more
questions than answers, but served to share the experiences, lessons learned, and best practices from
across our industry, and offered future opportunities to focus our efforts for continued advancement in
solving our toughest systems engineering challenges.
3
Table of Contents
Executive Summary:...................................................................................................................................... 2
Background: The Case for CASE ................................................................................................................... 5
Program Overview ........................................................................................................................................ 6
The Results: CASE 2012 Session Summaries ................................................................................................ 9
Track 1: Complex Systems Development Session Summaries ................................................................... 10
Track 2: Integration, Test, and Verification of Complex Systems Session Summaries .............................. 12
Track 3: Program Management to Achieve Robust and Resilient Systems Session Summaries ............... 41
CASE Closing Session Summary .................................................................................................................. 53
Final Remarks on CASE 2012 ....................................................................................................................... 58
Acknowledgements: The CASE 2012 Planning Committee ........................................................................ 59
4
Background: The Case for CASE
AIAA’s first CASE (Complex Aerospace Systems Exchange) Conference was created to provide a forum
for aerospace professionals who are directly engaged in leading, developing, producing, and supporting
the most advanced vehicles and systems. It specifically targeted engineering leaders who work every
day to realize the promise of technology by developing state of the art flying systems. The current
missions being pursued by the aerospace industry continue to stretch the limits of performance, and
although numerous symposia are available for specialists to delve into the details of the myriad of
technical disciplines that are required to develop advanced aerospace vehicles, few events provide for
the needs and interests of the many aerospace engineers that work as systems engineers to understand
user requirements, conceptualize new flight vehicles and systems to address those requirements, lead
development activities, and integrate complicated interdependent subsystems to produce the resulting
technical marvels. Furthermore, the complex nature of today’s systems requires a wider proliferation of
systems thinking across all disciplines. Exposing engineers during the early stages of their careers to
higher level program and technical challenges will heighten curiosity to develop adjacent engineering
skills, stimulate cross-discipline collaboration and understanding of system interactions, and help
accelerate their professional development to become the aerospace leaders of tomorrow. To address
these needs, the inaugural CASE event was held at the Sheraton Pasadena, in Pasadena CA, 11-13
September 2012.
With a focus on the development, integration, test, verification, and program management of robust
and resilient aerospace systems, AIAA’s first CASE event was designed to address system-level execution
issues head on. The sessions were structured to tackle important system development subjects facing
aerospace chief engineers, program managers, and system engineering professionals, such as
increasingly demanding performance requirements, aggressive cost and schedule constraints, and a lack
of tolerance for test failures. With a greater emphasis on information and idea exchange than on
technical paper presentations, experienced leaders in systems engineering discussed their own
challenges and lessons learned so as to benefit the attendees and future aerospace projects. Session
attendees were able to participate in direct discussion and leave the event with practical knowledge and
ideas that were directly applicable to their daily work.
5
Program Overview
The CASE 2012 Conference was organized around an opening plenary session, three tracks, and a wrapup session to conclude the event.
Opening Plenary Session
At a conference-wide plenary session, participants were introduced to the objectives of CASE and the
topics that would be embedded throughout the event, by CASE 2012 Executive Chair, Dr. Michael
Griffin, Schafer Corporation and CASE 2012 General Chair, Laura McGill, Raytheon Missile Systems.
Opening remarks identified the need to provide a forum that addresses systems engineering and
integration challenges and explores hard execution issues that face a broad cross-section of aerospace
professionals working to manage and develop complex aerospace systems.
Track 1: Complex System Development
Society is challenged by the need to successfully develop increasingly complex and critical aerospace
systems that meet stakeholders’ needs within planned budgets and schedules. Track 1 focused on large
system development activities from the establishment of requirements through the conceptual,
preliminary, and detailed design phases, to address challenges faced by chief engineers and systems
engineers working to develop.
This track was designed to appeal to chief engineers, lead systems engineers, technology directors,
component and subsystem design engineers looking to better understand how their work fits in a
broader system context, academic researchers interested in state of the art system engineering
principles and requirements, mid-career professionals moving from technology into Integrated Project
Team lead positions, and early career professionals with an interest in system engineering disciplines
and principles. Track 1 included discussions on:
o
o
o
o
Forensic investigations dealing with problems rooted in design
Elegant design and complex systems development
Forensic investigations dealing with problems rooted in verification and validation
New paradigms for complex systems development
Track 1 was organized as follows:
Session 1-1:
Session 1-2:
Session 1-3:
Session 1-4:
Forensic Investigations I: Problems Rooted in Design
Elegant Design and Complex Systems Development
Forensic Investigations II: Problems Rooted in Program Issues
New Paradigms for Complex Systems Development
6
Session 1-5: Collecting Our Thoughts: Complex Systems Development
Track 2: Integration, Test, and Verification of Complex Systems
Pressure to conduct affordable development programs requires that the integration of complex systems
must be planned to significant detail well in advance of detailed design. This includes strategies to
design and mature models, databases, simulations, and test equipment to support program needs that
extend from bench testing of prototypes through flight testing on multiple ranges.
This track was designed to accommodate the interests of chief engineers and lead systems engineers
focused on the integration and verification phase of development, test engineers, specialists in modeling
and simulation, and engineers engaged in system verification. Track 2 included discussions on:
o
o
o
o
o
Development and execution of integrated test and verification programs
Facility user issues
NASA Aeronautics Test Program (ATP)
Integration of modeling and simulation, ground, and flight testing
Verification and validation
The Track 2 sessions were organized as follows:
Session 2-1:
Session 2-2:
Session 2-3:
Session 2-4:
Planning and Executing an Integration Test Strategy for a Complex Aerospace System
Integration of Modeling and Simulation, Ground Test, and Flight Test
Verification and Validation Issues
Lessons Learned in Integration, Test, and Verification
Track 3: Program Management to Achieve Robust and Resilient Systems
Complex aerospace systems require program management strategies and integrated planning tools to
coordinate the activities of technical teams from multiple organizations that are physically separated but
closely networked to work collaboratively and share common databases. Programs to develop and
support commercial and military aircraft, spacecraft, energy systems, and UAVs are each subject to
unique policy and regulatory issues. Managing the technical and direct support aspects in the
development and operation of complex aerospace systems will be the focus of this track. Other
dimensions of managing complex systems in Track 3 will include the management and integration of
business operations, logistics issues, and workforce infrastructure.
This track was designed to address the issues of concern for program managers, business managers,
system operators, government personnel engaged in acquisition, procurement, and oversight, as well as
supplier and sub-contract project managers. Track 3 will include discussions on:
7
o
o
o
o
o
Policy and regulatory issues
Execution of successful programs
New acquisition approaches
Business operations and logistics support
Workforce issues
The Track 3 sessions were organized as follows:
Session 3-1:
Session 3-2:
Session 3-3:
Session 3-4:
Execution of Successful Programs
New Acquisition and Regulatory Approaches
Business Operations and Logistics
Workforce and Education Issues
Wrap-Up Plenary Session
All of the CASE 2012 attendees were invited to participate in a discussion on key findings that resulted
from the program sessions, and to identify the challenges that should be addressed in future CASE
conferences. These points are captured in the session summaries that follow.
8
The Results: CASE 2012 Session Summaries
The following narrative details the CASE 2012 sessions, as provided in the official conference program
and supplemented with notes captured by rapporteurs to summarize discussions, identify themes, and
collect key points. Notes from specific presentations reflect the best understanding of the rapporteurs,
but have not been reviewed for accuracy by the speakers. Items captured during discussions are not
attributable to specific speakers, as they represent the free flow discussion that was a feature of the
CASE conference.
CASE Opening Plenary Session (Tuesday, 11 Sep, 1030 – 1100 hrs)
Opening remarks identified the need to provide a forum that addresses systems engineering and
integration challenges and explores hard execution issues that face a broad cross-section of aerospace
professionals working to manage and develop complex aerospace systems.
Speakers:
Michael Griffin, Chairman and CEO, Schafer Corporation, AIAA President, Executive Chair of CASE 2012
Laura McGill, Deputy Vice President of Engineering, Raytheon Missile Systems
Summary:
The complexity of the systems being designed today continues to grow. We enter into systems
integration and flight test with more risk than ever before, in an environment that is increasingly risk
averse. We lack a well-defined systems engineering theory to provide a basis for error-proof
methodologies to avoid failures or predict divergent behaviors. Although we can identify the great
practitioners, we are not sure about why they are great. Projects succeed or fail by the ability of
systems engineers to foresee failures and determine mitigations.
This conference is intended to provide a forum for experts to share knowledge and benefit various
players in systems integration: Chief engineers, Test professionals, and Program managers. The event
should also be of interest to mid-career professionals moving from specializations to Integrated Product
Team (IPT) positions.
9
Track 1: Complex Systems Development Session Summaries
Session 1-1: Forensic Investigations I: Problems Rooted in Design (Tuesday, 11 Sep, 1100 – 1230 hrs)
This panel of some of the nation’s most renowned systems engineers will discuss cases where design
issues threatened the success of complex aerospace systems, focusing on what went wrong, what was
done to address the challenge, and what was learned from the experience.
Moderator: Michael Griffin, Schafer Corporation
Panelists:
Steve Rottler, Vice President, Science & Technology, Sandia National Laboratories
Bob Pearce, Director, Strategy, Architecture and Analysis, Aeronautics Mission Directorate, NASA
Headquarters
Brian Muirhead, Chief Engineer, NASA Jet Propulsion Laboratory
Session 1-2: Elegant Design and Complex Systems Development (Tuesday, 11 Sep, 1430 – 1830 hrs)
Leading thinkers in systems engineering arestriving to make the development of complex systems
radically easier by looking at design in a completely different way. The DARPA META and iFAB programs
will be discussed, as well as experience with an implementation of crowdsourcing for vehicle design. The
centerpiece of this session will be open discussion with the attendees in an unconference format.
Session Chair and Moderator:
Laurie Provin, University of Alabama in Huntsville
Panelists:
Robert Neches, Director, Advanced Engineering Initiatives, Office of the Deputy Assistant Secretary of
Defense for Systems Engineering, Department of Defense
Anna-Maria McGowan, Senior Aerospace Engineer, Aeronautics Research Directorate, NASA Langley
Research Center
Session 1-3: Forensic Investigations II: Problems Rooted in Program Issues (Wednesday, 12 Sep,
0800 – 1200 hrs)
No matter how carefully chief engineers plan, requirements and program structures are always liable to
change. This session will discuss actual challenges that have disrupted major system developments, and
how the engineering staff confronted the challenge, including what lessons were learned.
Session Chair and Moderator:
Wilson Felder, Federal Aviation Administration
10
Speakers:
Tom Hancock, Senior Systems Engineer, SAIC
Steve Messervy, Director, Research Institute, University of Alabama in Huntsville
Laura McGill, Deputy Vice President of Engineering, Raytheon Missile Systems
Frank Serna, Chief Systems Engineer, Draper Laboratories
Session 1-4: New Paradigms for Complex Systems Development (Wednesday, 12 Sep,
1400 – 1800 hrs)
Presentations and discussions will focus on the design techniques specifically developed to cope with
complexity. New paradigms will be examined from industry, government, and academia. Much of this
session will be open discussion with the event attendees, in an “unconference” format.
Session Chair: Paul Collopy, University of Alabama in Huntsville
Speakers:
Jimmie McEver, Senior Scientist, Johns Hopkins University/Applied Physics Laboratory
Brian Muirhead, Chief Engineer, NASA Jet Propulsion Laboratory
Russ Althof, Engineering Fellow, Raytheon Missile Systems
Mahmoud Efatmaneshnik, Postdoctoral Researcher, Stevens Institute
Debra Facktor Lepore, Industry Professor, Systems Engineering Research Center, Stevens Institute
Session 1-5: Collecting Our Thoughts: Complex Systems Development (Thursday, 13 Sep,
0800 – 1000 hrs)
Session leaders will gather for a roundtable discussion, including the event attendees, about the insights
and emergent cross-session themes from the three days of interaction.
Session Chair: Paul Collopy, University of Alabama in Huntsville
Speakers:
Michael Griffin, Schafer Corporation
Laurie Provin, University of Alabama in Huntsville
Wilson Felder, Federal Aviation Administration
11
Track 2: Integration, Test, and Verification of Complex Systems Session
Summaries
Session 2-1: Planning and Executing an Integration Test Strategy for a Complex Aerospace System
(Tuesday, 11 Sep, 1430 – 1830 hrs)
This session will focus on attributes of an effective integrated test and evaluation development process
for aerospace systems. Discussion will cover planning through execution for both commercial and
military systems, but will be primarily focused on upfront planning.
Session Chair: Phil Stitch, Arnold Engineering Development Complex
Moderator: Ed Kraft, Arnold Engineering Development Complex
Panelists:
Barry King, Director, Space Test & Operations, Dynetics, Inc.
Thomas Irvine, Deputy Associate Administrator, NASA Aeronautics Research Mission Directorate
Derrick Hinton, Principal Deputy Director, Test Resource Management Center, Office of the Secretary of
Defense
Ward Johnson, General Manager, Jacobs Technology Group
Eileen Bjorkman, Chief Technologist, 412 Test Wing, U.S. Air Force
Ming Chang, Lockheed Martin Fellow, Lockheed Martin
Summary:
The opening discussion set the stage to further explore the session topic by attempting to establish
some definitions and boundary conditions. Aerospace industry development cycle times were
contrasted against those of the automotive and integrated circuit industries, where rapid fielding is
paramount to market success.
System complexity is not inherent in rampant, runaway acquisition time cycles. Reasons may be found
in:
 Poor architecture choices
o Aerospace systems start with a clean sheet in each program
o Auto and IC industries use common platforms over time
 Processes and process ownership
o Many are outdated and need improvement
12
o


Auto and IC industries own their own processes
Accountability
o Problem in aerospace industry
o Program reviews are not sufficiently demanding
Capacity
o Aerospace capacity is periodically diminished by cycles of cutting back after major wars
and conflicts
o Diminished capacity drives development costs up
The complexity of processes and environments are often more severe than the complexity of the
complex system under development - This is an organizational problem.
Definitions: Complex vs. Complicated
 Complex
o Parts are interdependent, often changing, and outcomes are less predictable
o Hard to ”see” the whole, but only what’s available from your vantage point
o Complex systems are not constructed, they are grown
o Complex systems are dynamic—that is, they’re constantly changing and adapting to
current conditions
 Complicated
o Not simple, but knowable
o Clear links between causes and effects
o Tend to give predictable results, as long as the same thing is done the same way, each
time
o Focus needs to be on diligence and accuracy when following particular steps in a
process
 Do not confuse the two, and integrate as early as possible
o Have complicated systems interact with complex environments as soon as possible.
o Modeling and Simulation (M&S) allows early integration of the system and its
environment
 Can use physics-based codes to generate solutions that can be plugged directly
into war games of flight simulators
Key Take-Aways:
 Do not blame expanding development cycles entirely on system complexity
o Don’t exacerbate the impacts of system complexity on development cycles by the use of
bad architecture, processes, poor accountability, or inadequate capacity
 Understand the difference between complicated and complex, and know how to treat each one
13
Ming Chang, Lockheed Martin Fellow, Lockheed Martin Corporation, ‘Complexity from the Defense
Original Equipment Manufacturer (OEM) Perspective’:
This presentation outlined the most essential elements of successful Verification and Validation (V&V) of
complex systems from the perspective of an original equipment manufacturer of military aerospace
systems.







Successful development is dependent upon
o Understanding requirements
o Feasible design
o Early Identification of risks
Removal of defects early is essential to maintaining lower life-cycle costs
Perform V&V in all phases of the development, at all Technology Readiness Levels (TRLs)
V&V of various systems and disciplines in complex systems were contrasted
Flight safety is foremost, a cradle-to-grave culture
Validation is a continuing process using wind tunnels (WTs), structural models, etc.
Verification is a real-time process with processor-In-the-loop (PIL) / human-in-the-loop (HIL)
/system-in-the-loop (SIL) simulation
Key Take-Aways:
 Processes matter
o Processes differ among types of systems or phases of development
 Early V&V is essential, continue through all phases
 Early dependence on legacy systems can lead to complexity creep as system matures
Barry King, Director, Space Test & Operations, Dynetics, Inc. ‘Integration & Test of Complex Programs
(Commercial Perspective)’:
This presentation outlined the most essential elements of successful V&V of complex systems from the
perspective of a manufacturer of commercial aerospace systems.



Program integration requires good situational awareness of the who-what-when-where-howwhy of the program life cycle
o Frequently don’t plan well enough for later stages.
Program integration roadmap is a combination of those of the Department of Defense (DoD)
and NASA
o Some documentation eliminated
Stakeholder needs, goals, and objectives (NGOs) determine what needs to be done
o Key starting point for program and Level 1 requirements
14

System engineers and test engineers are linked from the onset
o Relationships mapped early for systematic execution
o Early test integration averts systems being “thrown over the wall” to test
Key Take-Aways:
 In order to have a successful program, all levels of requirements must be linked and verifiable
along the way
o Analysis, model or test
 Most common mistakes with greatest impacts:
o No base line and/or fluid requirements;
o Integration and test personnel not on design team;
o Allowing design engineers to be involved too long;
o Too little time allowed for testing:
 35% of the schedule for integrated test operations ITO following 65% for design
is inadequate;
o Processes or procedures, rather than experience, are relied on as the basis of discipline
and completeness
 Include a recent graduate on project teams to question “dumb” rules and point to ways to “fix”
procedures
Tom Irvine, Deputy Associate Administrator, National Aeronautics and Space Administration (NASA)
Headquarters, Aeronautics Research Mission Directorate, ‘Complexity and the Research Community’:
This presentation outlined the most essential elements of successful V&V of complex systems from the
perspective of the research community.


Observation of natural complex systems is instructive in understanding engineered complex
systems.
o Used the kelp tank at the Monterey Aquarium as an example of a natural complex
system.
There are similarities and differences in natural and engineered complex systems identified in
the presentation.
o Both natural and engineered complex systems:
 Have dynamic environments that are constantly changing and adapting to
current conditions
 Have interdependent parts and outcomes which are not always predictable
 Require a holistic, collaborative approach
 It is difficult to see the whole but rather only what’s available from your
vantage point
15



Are complex systems that were grown and evolved over time and were not
constructed all at one time
o Natural and complex systems differ in that:
 A natural system is a chaotic environment where life and death are certainties
 Chaos is normally unwelcome in engineered systems or systems of systems
 The natural system is a continuous experiment during all phases of its life cycle
 The engineered system or systems of systems must effect purposeful
integration, and test and evaluation
Understanding of the interdependencies and outcomes by external observers is not required in
the natural complex system; it simply evolves according to a natural order
o In the engineered complex system, we must decompose it into parts we can understand
in order to positively affect design, development, test, evaluation and operations
 This decomposition often leads to sub-optimization
Referring to the SE “V” diagram, the decomposition and definition process usually works well.
o Re-composition and integration is the difficult part and is where we usually fall short.
 Research and Development (R&D) (both foundational and focused) is greatly
affected by this
 R&D is symbiotic with integration, test, evaluation, verification, and validation
 R&D may be either the developer or the user of test and verification
capabilities
 V&V costs will swamp program costs unless new tools come out of R&D
Key Take-Aways:
 Emulate, whenever possible, the incremental growth and development of natural complex
systems when engineering such systems and systems of systems
 Recognize the risk of sub-optimization when decomposing complex systems for understanding
 Use R&D to develop and use new test tools to enable affordable and effective V&V of complex
systems and systems of systems
Ward Johnson, General Manager, Jacobs Technology Group, ‘Planning and Executing an Integrated Test
Strategy for Complex Aerospace Systems (from a Test and Evaluation [T&E] Infrastructure Perspective)’
This presentation outlined the most essential elements of successful V&V of complex systems from the
perspective of the research community.


T&E facilities are very expensive and highly technical
o It takes as long to develop some T&E facilities as it takes to develop the systems they
support
Ensuring successful T&E program delivery requires:
16
o
o

Early and effective up-front understanding with stakeholders
Team with equipment suppliers and/or other design/build suppliers to mitigate risk and
minimize schedule
T&E infrastructure challenges in the United States:
o Maintaining and improving essential core test capabilities
o Preserving specialty test capabilities through investment in the T&E workforce
o Ensuring enhanced collaboration and cooperation across the T&E enterprise
o Capitalizing on simulation and technology in support of evolving integrated test
processes.
Key Take-Aways:
 The U. S. test infrastructure is old
o We have sufficient capability and capacity
o It is not being properly maintained or modernized
 Need more collaboration during lean budget periods
 Must capitalize on simulation and technology in support of evolving integrated test processes
 Two questions were posed to industry leaders:
o Are programs more focused on long-term technological advancement than near-term
client-defined safety, reliability, and sustainment?
o Are we properly balancing risk with responsiveness and cost?
 Program teams must learn from the past…do not reinvent
 T&E and Facility Teams must:
o Question requirements that come as solutions
o Define the end before you start
Eileen Bjorkman, Chief Technologist, 412 Test Wing, U. S. Air Force, ‘Flight Testing of Complex Systems:
Approaches and Lessons Learned’
This presentation outlined the most essential elements of successful V&V of complex systems from the
perspective of the flight test community.


Complex systems context
o Security requirements cause some difficulties
o Complexity exists because of people, processes, policies, technology, engineering,
management, and social sciences
 Solutions require interdisciplinary approaches
Complex systems test challenges
o Reduced budgets
o Pressures to close facilities
17
o
o




Do more with less; accelerated acquisition timelines
Range encroachment (geographic, competing technologies (wind turbines) and Radio
Frequency spectrum encroachment
o Integrated government /contractor facilities
o Aging workforce
Flight test approaches: Distributed testing
o The B-1 strategic bomber used distributed testing
 Full-up B-1 testing would have required prohibitive resources and assets
o Short time-line drove Global Hawk to distributed testing
o Ensure environments are “accredited” when using distributed approach
Fusion Lessons Learned
o Use building block approach
 Understand how individual systems work
o Plan for right assets and ensure they are fully functional, anticipate problems, build in
flexibility, identify key objectives and focus efforts to ensure success in those areas
We are drowning in data; eliminate unnecessary data
Simpler may be better
o Exercise critical functions in less complex scenarios when able
o “Exercising everything” early may create more questions than answers
o Develop confidence in the system before engaging complex scenarios
Key Take-Aways:
 Use what we have; repurpose, recycle, reuse, rely on others
 Complex systems may require new skill sets
 Complex systems may require multiple program offices
 Need to understand distributed M&S
o A “discipline” engineer with a Master’s in SE is the best combination
George Rumford, Deputy Director, Project Development, Department of Defense Test Resource
Management Center, ‘Planning and Executing an Integration Test Strategy for a Complex Aerospace
System’
This presentation described the DoD’s Test Resource Management Center’s (TRMC’s) structure, its
Congressionally-mandated mission for DoD test infrastructure oversight, including its relatively-recently
assigned responsibility for the Joint Mission Environment Test Capability (JMETC) Program and the
National Cyber Range (NCR).

The briefing contains organizational charts and mission statements for TRMC
18



The traditional TRMC mission of range oversight is established under Section 231 of the “Bob
Stump” NDAA for FY03 (PL107-314)
Responsibilities for JMETC and NCR were established in 2005 and 2012, respectively
Cyberspace vulnerabilities exist in DoD weapons systems
o The DoD has inadequacies in its cyber range capability
 Inadequate environments
 Costly and time-consuming to build a federated cyberspace environment
 Participant stovepipes delay integration of cyberspace capabilities
o Realistic cyber environments are needed
 Must determine whether systems can fight through a denial-of-service attack
and still be responsive to the warfighter
 Cyber has recently become more complex
o The NCR will represent :
 A persistent, operationally realistic cyberspace environment with:
 Interoperability Standards that break down “stove pipes”
 Rapidly integrated Live, Virtual, & Constructive capabilities
 Common test & experimentation tools and applications
o In terms of the SE “V” diagram, the NCR will:
 Identify cyberspace system and sub-system dependencies (decomposition side)
 Test systems of systems mitigation of cyberspace threats (recomposition/V&V
side)
Key Take-Aways:
 Tenets of “good” government and industry systems integration testing are:
o Derives from mission-focused requirements & test plans
o Combines a tiered management structure & accountability
o Uses agile development concepts
o Leverages existing government and industry test capabilities
o Employs interoperability standards that enable rapidly integrated live, virtual, &
constructive capabilities
o Shares common test & experimentation tools and applications
o Unifies approach in conducting both cyberspace and “traditional” testing
o Applies effective knowledge management techniques & structures
Phil Stich, Deputy General Manager, ATA, Arnold Engineering Development Complex, Session 1 Chair,
‘Threads from Track 2, Session 1’
The following summarizes the take-aways presented by session chair, Phil Stich, to the Track 2 chairs at
the evening summary meeting following Session 1:
19



Requirements fidelity and stability
o Spend more time up front with analysis of alternatives (AOA) before proceeding
o Beware of requirements that come as solutions
Acquisition process
o Must agree that our current process is cumbersome, especially for military advanced
development projects
o Commercial aircraft uses more streamlined process
Integration of complex systems is very difficult
o Subsystem test methods can give wrong results – or incomplete picture
o Fused data – flight test example
Session 2-2: Integration of Modeling and Simulation, Ground Test, and Flight Test (Wednesday, 12
Sep, 0800 – 1200 hrs)
A panel of service providers and a panel of testing and modeling customers will debate various issues
pertaining to the planning and implementation of Modeling and Simulation (M&S), ground and flight
test during the development of an aerospace system.
Session Chairs:
George Sydnor, NASA Langley Research Center
Terrence Trepal, Adjunct Research Staff Member, Institute for Defense Analysis
Moderator: Mark Melanson, Lockheed Martin
Panelists:
Tim Marshall, Director, Aeronautics Test Program, NASA Aeronautics Research Mission Directorate
Tom Fetterhoff, Technical Director, Test Division, Department of Defense
George Eitleberg, Director, DNW (German-Dutch) Wind Tunnels
Andy Garrell, Technical Director, Aerospace Testing Manager - Transonic Wind Tunnel, Calspan
Bryan Herdick, Senior Systems Engineer and Project Manager, Johns Hopkins University/Applied Physics
Laboratory
Jay Dryer, Director, Fundamental Aeronautics Program, NASA Aeronautics Research Mission Directorate
Barry Lakinsmith, Deputy Director, Army Aeroflightdynamics Directorate, U.S. Army Aviation and Missile
Research Development and Engineering Center
Ken Kilmurray, Strategic Planner, Northrop Grumman Corporation
John Curry, Director, Spacecraft Advanced Development, Sierra Nevada Corporation
20
Summary:
Tim Marshall, Director, Aeronautics Test Program, NASA Aeronautics Mission Research Directorate,
‘Integration of Modeling and Simulation, Ground Test, and Flight Test’
This presentation provided an overview of the mission and strategies of NASA’s Aeronautics Test
Program (ATP) and the Aeronautics Research Mission Directorate (ARMD) of which it is an important
part. Included were observations stemming from NASA’s experience in integrating complex systems.





ARMD’s programs have many interfaces
o Most errors and innovations in complex aerospace systems development occur at
program interfaces
ATP is the ground test and flight test element of ARMD
o ATP provides the underpinning for the four research programs of ARMD
 Provides testing for three of the four
ATP’s mission is to support NASA in every way possible:
o Provides strategic management guidance and recommendation to the NASA ARMD and
Centers with respect to the ATP portfolio
 Basis for ARMD decisions to close or maintain facilities.
o Represents NASA and the nation’s interests for aeronautics testing
 40% to 50% of ATP testing is for NASA; 15% to 20% is for DoD; remainder is
other external customers through Space Act agreements
o Provides strategic direction to capability managers across the portfolio
 Sometimes recommends reliance on non-ATP assets
o Provides financial support to sustain and enhance test facility capabilities and related
workforce
 Must balance sustainment with new capability
ATP’s investment strategies are to enhance capability (higher reliability, relevance, etc.,
therefore higher value) at reduced operational cost (lower cost to users)
o “Value sweet spot” graph was shown: A region on the Cost vs. Capability map that is
considered the target for any given capability (See briefing)
o Capabilities operating outside the boundaries of the “sweet spot” are sought to become
compliant by:
 Improving capability at constant operating cost
 Lowering cost for a given capability
 Or some combination of the two
Challenges:
o Lack of rigor in planning a test; poor estimating by both the developer and the provider
o Lack of engagement between service providers and developers
21


Best way to ensure success is early and frequent communication with the test
providers
Can/should the providers provide integration services for M&S, ground test and flight test, or is
it the responsibility of the developer to determine the tools to use?
o Perhaps the provider can have the tools available and offer the capabilities
Key Take-Aways:
 Improve the value of capabilities to customers
o Stay engaged with customer needs
 Leverage investments for maximum benefit
 “Teach” customers about capabilities
o Early interaction is critical
o Helps streamline the integration of Ground Test (GT), Flight Test (FT), and M&S
o Incomplete early planning often leads to trouble
Tom Fetterhoff, Technical Director, Test Division, Arnold Engineering Development Complex: Arnold
Engineering Development Complex (AEDC)
This presentation discussed the use of integrated test and evaluation at AEDC and how AEDC seeks to
better integrate with FT.




The new Air Force Materiel Command (AFMC) “Test Center” organization combined GT and FT
complexes
o Seeks better integration of testing tools
o Goal is lower test cost and higher quality solutions
AEDC has used integrated test and evaluation (IT&E) for many years
o M&S enables better insight into data obtained in GT and FT
o The use of M&S integrated with GT in several programs was discussed
 Examples included F-15, F-22, B-1, F-101 engine
 Disciplines in examples included store separation, inlet performance, engine
repair
 Store separation has moved toward more M&S, less GT and FT
AEDC works to expand contribution of M&S to early program planning (during the Analysis of
Alternatives [AoA] phase)
o Today WTs provide high-fidelity data faster than computational fluid dynamics (CFD);
this could change in the future
AEDC seeks to provide customers with information, or knowledge, not just data.
o Audience comment: Customer is still responsible for the design and development, and
might not want all the help that is being offered
22




AEDC’s space chamber work that supports the Missile Defense Agency is very robust and
growing in the sensor testing area where IT&E is beginning to be utilized more
AEDC integrates M&S, GT data, and FT data to develop a plume signature data base for all
rockets
Hypersonics requires the use of Test and Evaluation (T&E) and Science and Technology (S&T) to
develop tools and techniques to support future systems
Recent changes to DoD 5000 series regulations requires early communications
o This will drive testers and users to work more closely together
o Will help IT&E methodology work better
Key Take-Aways:
 Purely experimental V&V is no longer viable
 M&S has matured to the point that it is capable of offloading testing to varying degrees
 Early interactions between customers and test professionals enable better, faster, cheaper
solutions
 Effective IT&E early can facilitate/enable streamlining FT
 Stakeholder feedback enhances AEDC capability
George Eitelberg, Director, German Dutch Wind Tunnels (DNW), ‘Integration of services by an
independent service provider’
The presentation discusses the culture and challenges of the independent test provider and how the
German-Dutch Wind Tunnels (DNW) support the integration of T&E services with its customers.


DNW is an independent provider of T&E services, supporting research (15%), military test (40%,
of which half is transport), and civil (45%).
o DNW’s product is reliable data
Workload is uneven/highly volatile
o Heavy usage in some years and light usage in others can lead to looking for other areas
of expertise
 A jack-of-all-trades approach can lead to poor support
 Specialists are needed in all areas; one person cannot cover multiple disciplines
well
 Horizontal integration can alleviate pressure, but may lead to scheduling
conflicts
 If you redeploy staff into other disciplines, you will usually not get them back.
 DNW knows where to find specialists when needed
 Audience: AEDC does a good job of bringing all tools together;
Lockheed-Martin (L-M) used their services on the F-35.
23




DNW integrates with the research community
o The capabilities and limitations of numeric and wind tunnel simulations need to be
understood
o Universities assist with methodologies for IT&E on development programs
o GT is not properly taught; should be thought of as “simulation of flight.”
Network with customers to understand their processes and future needs
o Example: Two problems remain in civil aeronautics: Noise (aero-acoustics) and
propulsion integration (fuel consumption)
Maintenance is managed by internal DNW staff, but it is outsourced as much as possible.
o Provides flexibility
Key Take-Aways:
o Independent providers don’t have to be politically correct…just correct
o A jack-of-all-trades is a master of none
 DNW does not provide one-stop shopping
o If operational cost is a concern, expertise is endangered
o IT&E methodology needs stability to be effective
 This is difficult because it is a project-driven process
 Support from research organizations can help solve the problem
o Independent service providers must depend to some extent on institutionally-funded
sources for development
o Retain flexibility to enable optimum contribution in each area of expertise
Andy Garrell, Technical Director, Aerospace Testing Manager – Transonic Wind Tunnel, Calspan, ‘Calspan
– A Provider’s Perspective’
This presentation discussed Calspan as an independent test service provider with one wind tunnel,
striving to incorporate new technologies to improve results and make usage more efficient.



Technological advancements have affected Calspan’s service area contributions
o Improved efficiencies, data quality and operational approaches
o Resultant reduced usage has made sustainable utilization more of a challenge
 Lower usage leads to higher unit costs because fixed costs remain
o Calspan’s continual improvement program has led to new technology advancements
 Primarily in computer automation and CFD applications
Integration of M&S/GT/FT is not happening quickly at Calspan
o Attributed to Calspan’s small, independent status
o They never see the user’s M&S people…only GT people
o Integration with M&S or FT is considered the customer’s responsibility
For maximum success, developers should emphasize:
24
o
o



Quality physical models brought to the WT
Interaction between M&S and GT providers
 Regularly include CFD person(s) in on-site ground tests
Shortfalls that exist in M&S and GT can be offset be each discipline’s counterpart
o Mesh M&S and GT early and often; integrate the program teams
o Eliminate competition between M&S and GT communities
Areas for improvement:
o Quality of the test article
o Involve CFD with GT
o M&S and GT can each be strengthened through cooperation
User issues:
o Lack of planning
o Timeliness of planning
o Lack of coordination with the test article development and the test facility
Key Take-Aways:
 Small size and independent status inhibit major integration of new technologies and T&E
disciplines
 Major efficiency improvements create a challenge and will drive the unit cost of testing up
 Innovation can be difficult to implement. (Based on a question from the audience)
o Experienced people rely on what has been successful for them in the past
Bryan Herdlick, Senior Systems Engineer and Project Manager, Johns Hopkins University / Applied Physics
Laboratory, ‘SoS-based Capability Development Characterizing Performance Through Integrated M&S
and T&E… ‘
In the area of integrated testing, the Applied Physics Laboratory (APL) sometimes serves as a “trusted
agent” for the government, much like a Federally Funded Research and Development Center. This
presentation discusses APL’s integration of M&S with WT and FT for the verification, validation and
accreditation (VV&A) of systems of systems (SoS).



APL’s mission is to solve complex research, engineering and analytical problems that present
critical challenges to our nation
SoS testing is outside the comfort zone of conventional IT&E
o Multiple platforms, weapons, networks require operators in the loop
o M&S is integrated with GT and FT to augment live T&E
o Existing DoD IT&E capabilities don’t handle SoS well
“Ideal” organization for SoS testing would address:
o Use of live, virtual, and constructive (LVC) environments, chronologically:
 Constituent system testing
 Integration and functional-level “testing”
25


 Full integration and capability-level “testing” composite capability
o Integration of M&S is essential due to expense, scheduling, and operator-in-the-loop
(OITL) considerations of SoS testing
 FT of SoS is often untenable
Must characterize SoS capability vs. testing to a specification
SE approach using “federations” of models is used to address SoS capability development
o Some models not as rigorous as they should be
o Models from various programs must work together
 Some elements of models not tested by individual programs
o Models found lacking must be challenged up front
o Operator-in-the-loop is the ultimate proof
Key Take-Aways:
 Establishing trust in SoS testing is difficult.
 Must apply sound SE methodologies to SoS V&V to mitigate risk.
 Enhancing M&S to support T&E in SoS is a challenge.
 Prefer more V&V in high level architecture (HLA) federation development guidance.
 Virtual and constructive are not well-integrated
 Live testing is often difficult for SoS
 OITL in SoS testing is a “must-have,” as early as possible.
Jay Dryer, Director, Fundamental Aeronautics Program, NASA Aeronautics Research Mission Directorate,
NASA, ‘Integration of Modeling and Simulation, Ground Test, and Flight Test’
This presentation described NASA’s approach to the integration of testing services and steps they are
taking to overcome perceived shortfalls.



NASA’s Fundamental Aeronautics Program (FAP) exists to conduct fundamental research that
will generate innovative concepts, tools, technologies and knowledge to enable revolutionary
advances for a wide range of air vehicles
o The FAP develops and provides knowledge, tools and technologies to enable others to
develop advanced vehicles
o FAP integrates testing into its research by working closely with providers
FAP is undergoing or facilitating changes:
o Increased emphasis on high-fidelity simulators
o Growth in computational capabilities
 Continued growth may require new architectures
o More FT in fundamental research
o Merging Research & Technology (R&T) and T&E capabilities is expected
FAP is improving its performance in two key areas:
26
o

Risk mitigation and requirements development
 Develop requirements early, avoid creep
 Communicate!
 Take risks early on to avoid bigger risk down the road
 Ensure that the “integration” aspect amongst simulation, ground and flight test
is a part of the requirements phase
 Be creative with requirements
 Example: It may be better to invoke FT earlier to help inform the
development of other capabilities
Shortfalls in M&S, GT, and FT services were discussed and are being dealt with
o One recurring theme of note was that impact from improved coordination of M&S with
ground and flight experiments has resulted in more efficient testing.
 This fact, along with reductions in the number of programs, creates a financial
strain on facilities that also affects user costs
Key Take-Aways:
 Need to better integrate fundamental research into programs and test capabilities
 Sustainable resource models for test facilities and ranges must be implemented
Barry Lakinsmith, Deputy Director, Army Aeroflightdynamics Laboratory, U. S. Army Aviation and Missile
Research, Development, and Engineering Center, ‘Army / Rotary Wing Vehicle Perspective’
This presentation focused on the unique challenges of rotary wing T&E.





The rotary wing community uses M&S, GT, and FT
o Cross-specialization of Army personnel through partnerships with NASA and the Air
Force is needed
Many rotary wing programs are not successful
o Better IT&E is needed to assure success
Traditional testing (small-scale and full-scale WT and FT) has not predicted helicopter
aeromechanics and flight control needs well
o In a CV22 fuel dump example, CFD was a better predictor of the actual aerodynamic
phenomena than was a wind tunnel
CV22 is manageable; helicopter modeling is more difficult, but improving.
o Coupling rotor Computational Structural Dynamics (CSD) with CFD gets within 2% of
flight
o Rotor systems are very complicated, might be years before we can properly model
o Gathering small-scale WT, full-scale WT, and flight data to develop codes to predict
rotary wing performance
Improved tools are needed to support Future Vertical Lift (FVL) initiative
o Existing tools might be made to work with time and resources
27

Audience: Rotary wing safety record would not be tolerated in civil aviation
o Rotorcraft fly close to the ground
o Many accidents are caused by human error
o Briefer: Doesn’t think physics is the problem
Key Take-Aways:
 Improved codes are needed to eliminate need to over-design
 Must have multiple tools and good specialist engineers in each discipline
o Integration specialists are also needed
 Computational power is limiting
o We can compute one rotor revolution per day; we need to compute one revolution per
hour
 Systems are coming that will demand capabilities to model even more complexity
John Curry, Director, Spacecraft Advanced Development, Sierra Nevada Corporation, ‘Integration of
Modeling and Simulation, Ground Test, and Flight Test in the Dream Chaser Program’
This presentation described the integration of M&S, GT, and FT from Sierra Nevada Corporation’s user
perspective, using their Dream Chaser program as the example.



Dream Chaser is a complex system with a specific objective: Build and fly a crew space
transportation system using lifting body design that can achieve Low Earth Orbit cost-effectively
o Reverse –engineered Langley Research Center HL-20 design
 Enabled improved fidelity where needed
o M&S is being used extensively to build data base
 Cheaper, but adds risk
 Some areas of the flight regime must be done only with M&S and accept FT risk
o Shroud-less design was used to eliminate complexity in abort phase
 WT used to demonstrate that design without shroud is feasible
 Low budget requires WT to be correct
o A big challenge in GT is whether the model is exactly what flies
 If not, what problems are introduced by the differences?
o Analysis is now complete; now doing helicopter drop tests
 Moving to more complex parts
o Cannot accept low TRLs; all parts must have flown before
Key Take-Aways:
Requirements creep hurts the program.
Not all great ideas are requirements.
28






A simple fly-off would be best
Use experienced people
o You don’t know what you don’t know
Often spend too much time on contingencies and not enough on nominal situations
o Accept some risk and/or limited understanding and move on
Use actual or historical data to estimate mass in launch vehicles. Mass is critical!
When faced with less funding, do less and accept risk
o Explore how things can be done rather than “can’t do.”
Collaborate with NASA, but don’t accept too much oversight
Mark Melanson, Lockheed Martin Corporation, Session 2 Moderator, ‘Closing Comments for Track 2,
Session 2’



Integration of the various tools for V&V is the required future state
o Specialization must be maintained
Need computational tools to be more synergistic in matching WT data.
Should AIAA host a workshop for the three communities to map out a strategy?
o Audience comment: DoD has ongoing Computational Research and Engineering Acquisition
Tools and Environments (CREATE) program to develop and deploy computational
engineering tool sets for acquisition engineers
Session 2-3: Verification and Validation Issues (Wednesday, 12 Sep, 1400 – 1800 hrs)
This session will identify challenges that exist in today’s Verification and Validation (V&V) efforts. It will
begin with a generic perspective and then focus on a specific complex systems effort − the Next
Generation Air Traffic Control System (NextGen). With NextGen as an example of a complex
system/project, various solutions and applicable R&D will be discussed, by a variety of government and
industry speakers. In addition, this session will create awareness of some new and emerging V&V
standards activities (including the new AIAA Integration and Test (I&T) standard and the recent DO 178C
updates).
Session Chairs:
Ron Kohl, R. J. Kohl & Associates
Misty Davies, NASA Ames Research Center
Moderator: Ron Kohl, R. J. Kohl & Associates
Speakers:
Wilson Felder, Director, William J. Hughes Technical Center, Federal Aviation Administration
Mat French, Electrical Systems Engineer, Rolls-Royce
Don Firesmith, Senior Technical Staff, Software Engineering Institute, Carnegie Mellon University
29
Panelists:
Maureen Molz, Manager, Concept Development and Validation Branch, Federal Aviation Administration
Misty Davies, Research Computer Engineer, Intelligent Systems Division, NASA Ames Research Center
Bill Shane, Chief Engineer, Test and Evaluation, The Boeing Company
Eric Neiderman, Human Factors Branch Manager, Federal Aviation Administration
Summary:
Wilson Felder, Director, William J. Hughes Technical Center, Federal Aviatrion Administration (FAA),
‘Challenges in Testing NextGen’
NextGen is a comprehensive initiative that integrates new and existing technologies, procedures and
policies for the transformation of our National Airspace System (NAS), making it flexible and sustainable.
This presentation provided insights as to the challenges of testing such a complex system.





NextGen is a comprehensive system with new and existing technologies, procedures, and
policies
o It involves many people and seven U.S. Government agencies
Navigational performance/area navigation, dependent surveillance, digital communications, and
4-D trajectories are all new paradigms
It is being built in pieces and layers
There are dependencies on avionics, communication systems, and other elements in the aircraft
that cannot be brought into a laboratory
NextGen is a complex system that must be thoroughly tested before implementation, and must
be shown to be safe for future systems
Key Take-Aways:
 LABNET created that links all relevant systems as an assessment platform for complex systemsof-systems
 Using “come-as-you-are” strategy to test interfaces with whatever enters the system
 Removes stovepiped laboratories and allows one integrated toolset
Mat French, Electrical Systems Engineer, Rolls Royce, Co-Chair, AIAA Communication and Membership:
Aerospace System Integration Working Group (ASI-WG)
AIAA is creating a working group to promote and contribute to aerospace system integration (ASI) across
the spectrum of applications ranging from traditional systems to complex systems and systems of
systems.
30





The Working Group (WG) will be implemented in a “crawl-walk-run” process beginning in 2013
The topic is very broad and cuts across almost all AIAA Technical Committees (TCs), WGs, and
Program Committees (PCs)
o Integration from a system design aspect
o It will deal with growing simple systems to systems of systems
Publications and conferences will be the main products
o Possibly a joint journal with the Systems Engineering Technical Committee
o Working on an Aerospace Systems Integration (ASI) Guide
Significant details on the WG’s charter, activities, relationship with other committees, etc., are
presented in the briefing
Audience comments/questions:
o Why is SE increasing in importance? How does SE verification differ from testing?
 Look at risks specifically associated with system integration
o System integration tasks are universally applicable, regardless of scale
 Scale only makes it more complicated
Key Take-Aways:
 Systems Integration is a foundational element of SE
 This WG should help to link committees that have integration within their charter
 Formational activities are ongoing. Looking for members and contributors
o Contact information is given in the briefing
Donald Firesmith, Senior Technical Staff, Software Engineering Institute, Carnegie Mellon University,
‘Common Testing Problems: Pitfalls to Prevent and Mitigate’
The briefing was structured to capture and organize a high-level summary of commonly-occurring
testing problems, make general recommendations as to how to avoid/mitigate these problems, and
provide the information needed to generate testing oversight and assessment checklists.

Problems are characterized in the following categories:
o General testing problems
o Test planning problems
o Modern life-cycle problems
o Requirements and testing problems
o Unit testing problems
o Integration testing problems
o Specialty engineering testing
o System testing problems
o Systems-of-systems testing problems
31

o Regression testing problems
o Maintenance testing problems
An extensive list of recommendations is provided in the presentation slides
Key Take-Aways:
o Root causes of most problems relate to :
 Lack of adequate planning, preparations, or definitions
 Resource limitations
 Turf / rice bowl disputes
 Missing information from poorly-thought-out programs
Panel Session:
Maureen Molz, Manager, Concept Development and Validation Branch, FAA, ‘System of System
Verification & Validation for NextGen’
NextGen represents a fundamental paradigm shift from stove-piped, analog, point-to-point systems to
an integrated, digital, net-centric architecture. This briefing seeks to describe a cost-effective approach
to testing the highly complex NAS system that is constantly in use and can’t be turned off.

Traditionally, 70% of faults in a complex system are introduced in the design phases (downward
leg of the SE “V” where NextGen is decomposed) and only 3.5 percent are found and fixed
 The remainder are dealt with after recomposition at significantly higher cost
Key Take-Aways:
 There is value in using simulation to test NextGen.
o Using live, virtual and constructive models.
o Low-, medium-, and high-fidelity
 Test entire NAS as a system-of-systems
o Influence research, development and test
 Evaluate in realistic/intended environment
Kevin Knudsen, Boeing Test & Evaluation, ‘System-of-Systems and Live, Virtual, Constructive Test
Capability - Global Capabilities’
This presentation described Boeing’s application of LVC test methodology early in system design to
mitigate problems that occur downstream.

Historically Boeing has stovepiped SE, T&E, and M&S
32


o Realized that these must come together to facilitate early V&V
Boeing has initiated a gated approach for its V&V enterprise to identify, examine, and mitigate
risks early and throughout the process
o Cross functionally ensure that the concepts, requirements, architectures, designs, and
operations are affordable, feasible, valid, producible, and testable across the life cycle
o Perform verification and validation as soon as possible to identify, mitigate, and retire
program risks early
Boeing has created a global LVC-test infrastructure for LVC experimentation and test
methodology
o Boeing sites worldwide are on LABNET
o A single T&E organization supports military and commercial programs
Key Take-Aways:
 Industry must revisit the approach to V&V of complex systems/systems of systems
 Early and persistent, cross-functional engagement across a SOS life cycle is required
 Leveraging opportunities for injecting downstream issues of a prototype system into a
controlled, complex SOS Test environment earlier in the life cycle helps identify and mitigate
program risks throughout the life cycle
Misty Davies, Research Computer Engineer, Intelligent Systems Division, NASA Ames Research Center,
‘Assurance of Software-Intensive Flight Critical Systems – A Plan for Enabling V&V in NextGen’
This presentation discussed the plan for applying V&V in NextGen.


80% of a pilot’s functionality is software-based
o 75% of the cost of a new aircraft is software
o Traditionally we underestimate that cost
For software-intensive systems…
o Lines of code have doubled almost every four years since 1968
o Software defects increased 15% from ’97-’98 to 2000
o The cost to fix software deficiencies in the maintenance stage is about 110 times what it
costs in the requirements stage
 64% of deficiencies are introduced in the requirements or design stage, but few
are corrected there
o We have no good way to look at distributed systems that are software intensive
Key Take-Away:
 The key to maintaining safety in NextGen while reducing the cost of V&V is to push V&V earlier
in the lifecycle and use advanced complex system V&V tools and processes
33
Eric Neiderman, Human Factors Branch Manager, FAA, ‘Human Factors of Integration, Test, and
Verification of Complex Systems’
This presentation explored the issue of ensuring that human factors issues are identified and resolved in
complex systems.




Human factors considerations are usability, trainability, and maintainability
Desirable to test human factors in unusual, off-nominal events
o Understand the role and function of the person in rare, abnormal, unusual, emerging
situations
Human factors issues can best be understood through the use of:
o Tools
 Measurement, cognitive agents, and simulation capabilities
o Opportunities
 Human factors long-term R&D focus areas
 System resiliency
 Complexity
 Information
 Human-systems integration
 Safety
o Processes
 Evaluate the system in less than ideal situations and circumstances.
o People
 Involve all disciplines
o Strategy
 Develop both short and long-term strategies in all areas of tools, opportunities,
processes, and people.
Audience interaction:
o CASE has been used during the week as a place to rant about decision-makers who don’t
want to hear about SoS LVC testing in distributed networks
 Institute a mechanism for constructive action
o Went for International Organization for Standardization (ISO) certification on NextGen
test procedures
o How do you V&V systems early enough to make changes soon enough?
 Use a business case to convince decision-makers that there is value in testing
earlier
 Need data to make business case. Must document better
o Organizational structures are based on old models. Not agile enough
34
Key Take-Away:
 Human factors can be systematically identified and resolved in complex systems
Group discussion in closing Session 2-3:
Statements came from both panelists and members of the audience:



There has been a lot of complaining that decision-makers don’t want to consider the use of SOS
testing in a distributed environment
o Perhaps CASE will provide a forum for more constructive action
We need to collect the data to document benefits of early V&V when we do it
o Make the business case
For agile systems, the technical problems can be solved. Organizational structures, processes,
policies, etc., need to be changed.
o E.g., eliminate functional stove-piping
o Our organizational structures are antiquated
o Not agile enough
References (standards, best practices, guidebooks, etc.) related to V&V or I&T as referenced in the
introduction to this section.









AIAA 117 Space Systems V&V
1012-2012, IEEE Standard for System and Software Verification and Validation;
2012: http://standards.ieee.org/findstds/standard/1012-2012.html
NASA IV&V Policy NPG 8730 DRAFT
FAA AMS Policy for
V&V: http://fasteditapp.faa.gov/ams/do_action?do_action=LinkSection&contentUID=4&section
Number=2.1.7
FAA V&V
Toolkit: https://employees.faa.gov/org/linebusiness/ato/operations/technical_operations/best_
practices/discipline/evaluation/
FAA V&V Guidelines, October 2011: http://fast.faa.gov/docs/vandvguidelines.doc
FAA T&E Process Guidelines, March 2012: http://fast.faa.gov/docs/teguidelines.doc
FAA T&E Handbook, August
2010: http://www.faa.gov/about/office_org/headquarters_offices/ang/offices/tc/initiatives/vn
v/documents/publications/VVSPT-A2-PDD-013_TnE_Handbook_v2.0.pdf
AIR FORCE INSTRUCTION 99-103, CAPABILITIES-BASED TEST AND EVALUATION
35


DAU’s TEST AND EVALUATION MANAGEMENT GUIDE, JANUARY 2005
Documents that contain V&V/I&T ‘best practices’ information
o ISO 15288
o ISO/IEEE 12207
o NASA NPG 2820
o NASA Systems Engineering Handbook (6105)
o CMMI
o Mil Std 499B
o DAU Systems Engineering Fundamentals Guidebook
o DAU Defense Acquisition Guide
Session 2-4: Lessons Learned in Integration, Test, and Verification (Thursday, 13 Sep, 0800 – 1000 hrs)
This session will link the three previous Track 2 sessions together by looking at how the planning,
integration, and execution of ground and flight testing is woven throughout the development of a
complex aerospace system. Project managers would like to test as much as possible, but there are
limitations due to cost and schedule. It is a constant challenge and balancing act for these managers.
How do you balance testing needs in a budget-constrained environment? How to determine when a
ground test is sufficient? Flight-testing appears to be the panacea, but is that always the case? What are
the critical aspects of environment modeling? What are the keys to using modeling and simulation
versus actual hardware testing? Panelists will discuss these and other topics as we look at lessons
learned from a wide range of projects/programs across industry, DOD, and NASA. Both aircraft and
space projects will be discussed.
Session Chairs:
David Dress, NASA Langley Research Center
Steve Chan, NASA Johnson Space Center
Moderator: David Dress, NASA Langley Research Center
Panelists:
Marshall Smith, Mission/Flight Test Strategy Lead for Exploration Systems Development Division, NASA
Headquarters (Ares I-X)
Frank Rasor, Director, Flight Test Operations, The Boeing Company (787 and 747)
John Muratore, Director, Mission Assurance, SpaceX (Falcon 9 and Dragon)
Brendan Rhatigan, Director, Engineering & Test Operations, F-35 Integration Test Force, Lockheed
Martin Corporation (F-35)
David Gilman, Senior Technical Advisor, Analytical Mechanics Associates, Inc. (IRVE-3)
36
Summary:
Marshall Smith, Mission/Flight Test Strategy Lead for Exploration Systems Development Division, NASA
Headquarters, ‘Ares I-X Lessons Learned - Integration, Test and Verification
ARES I-X was a fast-paced program to provide early source data for NASA’s ARES I vehicle, using
resources from across the agency, to discover integration, assembly, and test issues for ARES; provide a
realistic simulation of the Ares environments and flight control strategy; and provide flight validation of
integrated aero, structures, and performance models.

Lessons-learned were discussed in four separate theme categories:
o Mission management
 Concentrated leadership/authority
 Schedule is king
 Minimize levels and boards – high decision velocity
 Make decisions quickly and live with them
 Documents and process are not the focus
o Test mindset – Enforce “good-enough” rationale
 Question standards, policies, procedures
 Developed simple alternatives (e.g., modal tests)
o Systems Engineering and Integration (SE&I)
 Objective and goal focused - question requirements and standards
 Small competent teams composed of elements and led by SE&I
 Tiger teams formed and dissolved as necessary
o Heritage hardware and facilities
 Don’t assume heritage will work with less effort
 Atlas Systems Integration Laboratory worked well for evaluating heritage
hardware in new environments.
Key Take-Aways:
o Make decisions when you have sufficient data; don’t wait for all of the data
o Strive for cheaper, faster, good enough
o Make requirements subordinate to goals
o Heritage doesn’t always work; it is a place to start
Frank Rasor, Director, Flight Test Operations, Boeing (787 & 747), ‘Product Validation and Recent Flight
Test Experiences’
Lessons-learned were presented from ongoing Boeing programs. This presentation explored some of the
challenges.
37

Key Take-Aways:
o In spite of schedule pressures, maintain the discipline to finish component tests before
advancing to flight test
o More ground testing can result in less flight testing
o Referring to the Verification and Validation “V,” the way you arrive at and manage the low
point of the “V” influences how efficiently you move up the right-hand-side with test
John Muratore, Director, Mission Assurance, Space X (Falcon 9 and Dragon), ‘System Engineering: A
Traditional Discipline in a Non-traditional Organization’
Space X is a small company of 1800 people, most located in Hawthorne, CA under one roof. They are
developing several versions of Falcon launch vehicles and the Dragon spacecraft. This presentation
described the unique approach to SE in Falcon V&V.

Space X uses “iterative design”
o Maintain continuous design heritage in development with rapid spiral methodology
o Huge development risk on the way down the “V,” rigorous SE on the way back up
o Rapid design-build-test cycles to inform design by experience
o “Test like you fly, test what you fly”
Key Take-Aways:
o Space X’s innovative success is facilitated by its small size and relatively small cost of
systems.
o Rapid spiral development depends on applying lessons-learned
 Immediate redesign when tests reveal problems enable program to advance rapidly
o People are better at solving problems than predicting them
 Up front SE is balanced with learning through rapid spiral development
o The process depends upon rapid, constant communications among team members; e-mails
are answered immediately
o Designing to satisfy the needs of multiple customers
 Flexible design parameters to satisfy multiple mission needs
o Maintain total control over what is built in-house
 Multi-element optimization down to the last part.
o Space X uses a traditional test hierarchy, including in-space testing
 Facilities were low-cost and were built in a very short time
o It is difficult to build a creative high performance engineering culture
o Too much organization, and too many rules and processes, can easily ruin creativity and
performance
o SpaceX is achieving a good balance of creativity and systems engineering for agility and
affordability
Brandon Rhatigan, Director, Engineering & Test Operations, F-35 Integration test Force, Lockheed
Martin, ‘F-35 Lessons Learned in Integration and Test’
38
The F-35 has 24.3 million lines of code (LOC) compared to other aircraft averaging 9.4 million. This leads
to increased complexity in integration and support. This paper presented some of the lessons learned in
testing the F-35.


Test robustly, early and often
o Data are presented for multiple programs that show increased lower-cost ground
testing early actually reduced higher-cost flight testing later
Employed “Fusion Testing”
o F-35 has an overwhelming amount of sensor data; too much for the pilot to process
individually
o F-35 invested heavily in labs and test beds
o Fusion combines sensor data into usable information for the pilot
o Fusion testing means testing systems of sensors rather than single sensors individually,
but enabling the test engineer to look at individual sensor performance as needed.
 Focus is on whether you can complete the mission, not on the performance of a
single sensor
Key Take-Away:
 Reliance on early ground testing and Fusion enabled the F-35 to experience much better
stability and maturity than legacy systems.
David Gilman, Senior Technical Advisor, Analytical Mechanics Associates, Inc., ‘Lessons Learned Inflatable Reentry Vehicle Experiment – 3’
NASA has led the design of the Inflatable Reentry Vehicle and used a three-stage sounding rocket from
NASA’s Wallops Flight Facility to attain flight relevant heating on the vehicle. This presentation discussed
lessons learned.


The program encountered concerns over people becoming overstressed and took numerous
steps (presented in the briefing charts) to relieve stress on the test team
Some of the paths taken were not traditional (i.e., spending $250,000 on using one facility when
another facility would have been cheaper, in order to prevent overloading the team)
Key Take-Aways:
 It is possible to keep people in high-stress situations if there is “light at the end of the tunnel”
 If there is no immediate end in sight, they tend to want to pull back on work hours
 It is prudent to give a day off occasionally to allow team members to “recharge”
Group discussion in closing Session 2-4:
Statements below came from both panelists and members of the audience. No personal attribution was
recorded.
39











Is test-fix-test better than model-test-model?
o Space X uses the test-fix-test approach, constantly recalling boxes and making fixes
Many PMs see early test failures (discoveries of system problems) as program failures
o (Editorial note: Finding problems in a system being tested is not a test failure, although
that is the way it is often described and viewed by PMs and external enablers.)
o We have gotten away from using testing to provide learning opportunities
 We have become more conservative and avoid the risk of testing in highly
visible environments
 Trading technical risk for political risk
Critical anomalies have been missed in test programs, due to lack of adequate instrumentation
As performance requirements and regulations increase, system characteristics that were
previously tolerable have to be eliminated in later design spins
Sometimes test are skipped or delayed to maintain schedule, resulting in late discovery of
problems
Shutting down qualification facilities for black boxes is false economy
o Maintain the capability to learn from flights, modify the box, re-qualify it, fly it
“Good processes don’t ensure good performance, but bad processes can bring a program to its
knees”
o Having the right people is more important than having the right processes
Use tiger teams to solve specific problems and disband them
o Common leadership to stress problem-solving
o Use good people, meet schedule, document results, disband
Regarding stressed team members:
o A stress-performance curve was mentioned. As you pass a point, things begin to break
down easily
o Average age at Space X is 28; work week is 50 hours
Space X spends a lot of time documenting what pass and fail conditions are on a test
o “Nobody on the company succeeds if we are waiting on the ground to solve an
intractable problem”
Getting testers involved early is the best way to go
o Especially important when program starts are separated by long periods of time
o Both Boeing and L-M are trying to institutionalize this
o Lead tester must be someone who is experienced in the entire development process on
another system
 Difficult to find when program starts are infrequent
40
Track 3: Program Management to Achieve Robust and Resilient Systems
Session Summaries
Session 3-1: Execution of Successful Programs (Tuesday, 11 Sep, 1430 – 1830 hrs)
What makes a program successful? Is it luck, or are there management techniques that help keep it on
course? Leaders from varying levels of program management will discuss how they worked through
issues, customer expectations, and Verification, Validation, and Accreditation (VV&A), with the intention
of providing a set of positive lessons learned that audience members can use in their daily work life.
Audience participation is encouraged and will be fostered through question and answer sessions at the
end of each topic.
Session Chairs:
Jim Murphy, NASA Ames Research Center
William West, SURVICE Engineering Company
Panelists:
Ed Waggoner, Director, Integrated Systems Research Program (ISRP) Office, NASA
Ruben Delrosario, Project Manager, Subsonic Fixed Wing Project, NASA
Chris Olson, Program Manager, F-35 Joint Safe Escape Analysis Solutions, U.S. Air Force
Tama Curlee, Program Manager, Combat Weapon Delivery System, U.S. Air Force
Ray Howland, Manager Systems Planning, System Operations Control, American Airlines
Summary:
Overview of the various program areas from the panelists:





For ISRP there is an interim step between fundamental research and full systems integration.
Proven level of maturity at ISRP level to help ensure successful transfer into a full program.
Subsonic fixed wing project
o Early stage project vs. Research & Technology Project (fixed funding)
o Deliverables might still be undetermined
Questions to ask ourselves: What are you trying to do? How is it different than what has been
done? How do we organize to accomplish project determined objectives?
Combat Weapon Delivery System
o Multiple customers
o Legacy architecture -> rearchitecture process (extensible)
o User interface improvements
41
o

Requirements management in legacy environment is a rack and stack of priority
requirements
o Use Agile development process
Horizon Program with American Airlines
o No disaster recovery program on legacy systems
o New distributed architecture (physically & virtually)
Key Questions for the panel members:


What kind of meeting schedule do you have? What kind of battle rhythm do you maintain?
o NASA Perspective
 Weekly meetings – some travel to meeting locations
 Requires a lot of energy to maintain the communication but it is worth the
effort
 Use as much available technology like sharepoints, etc.
 Key is to remain flexible to accommodate the types of teams and individuals you
might have on your teams
o Combat Weapon Delivery Systems
 Agile requires continuous communication at management level generally on a
daily schedule
 There were stove-piping issues and lack of integration early on; worked to bring
the groups together
 Collaborative meetings with stakeholders involved to ensure crosscommunication
 Meetings are not necessarily structured to ensure more open communication
o American Airlines
 Visit vendor sites frequently to ensure consistent communication and vision
What kind of requirements management process did you have?
o Agile process – scope management is key (how to differentiate between nice to haves
and needs)
o Research side
 Need to have good goals; what do we want to accomplish; there is a strong
need for flexibility
 More and more project management rigor in the research community
 At the execution or requirements validation is where the research may be
compromised
 ISRP level strong entry/exit criteria for requirements
 Helps to define how you transition projects into acquisition
 ISRP will start to take into account the “ilities”
42





NASA actually has a tailored process for research projects because original
process is too structured
How do you set customer expectations?
o Organizational change management program that allows us to work with different
personalities
o Knowing the customer is the ultimate goal
What are the key customer insights?
o Change is hard – giving them more than what they expected (exceed expectations)
o Filtering the information until it is appropriate or targeting/scaling down the customer
base is sometimes necessary
o Better alignment of customer needs
o Need to figure out how to really usher in change
o Need concerted effort to meet with stakeholders to understand needs/requirements
o Sometimes knowing who your customer is; making sure that you are relevant
o One voice
Should we do joint programs? Is the complexity that they bring worth it?
o May have cost more, but the answer is dependent on the stakeholder
 Tax Payer: yes
 Program Manager: No
 Operator: Yes
o Need to be able to deal with uncertainty versus complexity
 Uncertainty – just need to try to be clearer
 Complexity – just need to recognize that there is growing number of
stakeholders
Question from Panel: Where do you (audience) see the problems?
o Integration across organizations
o Customer expectations – customers can be the boss so that makes things very difficult
to manage
o Managing the unknown unknowns
 How do you manage risk?
 Really important to have a risk methodology defined

Key Session take-aways:



Knowing when to say no – avoiding scope creep
Meeting with stakeholders frequently; allowances for flexibility
Choosing who to listen to; what SMEs; managing to the right set of customers
43
Session 3-2: New Acquisition and Regulatory Approaches (Wednesday, 12 Sep, 0800 – 1200 hrs)
This two-part session will identify and examine the challenges of regulation and acquisition strategy with
respect to the complex system domains of National Security, “NextGen” Air Traffic Control, (Civil) Space
Exploration Systems, and the emerging Commercial Space Systems. In the first portion of the panel,
speakers will provide in-depth overviews of their systems, highlighting the breadth and scope of their
system domains. The second portion of the panel will identify the top challenges and issues their
systems present and engage the audience in detailed discussions. A special emphasis will be given to
taking a top-level look at the challenges and issues, in an attempt to identify commonalities across the
complex system domains represented.
Session Chairs:
Ken Davidian, FAA Office of Commercial Space Transportation
Bruce Pittman, NASA Ames Research Center
Panelists:
D. Scott Lucero, Deputy Director, Strategic Initiatives, Office of the Deputy Assistant Secretary of Defense
Mike Hritz, Acting Chief Architect of National Airspace Systems, FAA NEXTGEN
Mike Ryschkewitsch, Chief Engineer, Space Exploration Systems, NASA Headquarters
Jim Van Laak, Deputy Associate Administrator, Commercial Space Systems, FAA Office of Commercial
Space Transportation
Stan Rosen, Professor Acquisition Management, Defense Acquisition University
Summary:
Stan Rosen, Professor Acquisition Management, Defense Acquisition University, ‘National Security Space
System Acquisition’:






Significant overruns; how can we do what we need to do with the dollars that we have
There are significant issues regarding new threats and new technologies; how can we be
responsive to these items
Policies continually changing
Acquisition processes changing
Personnel changing – training issues; expertise maintenance
Management approaches and methods
Lessons learned – trying to do a better job of building the SE expertise (acquisition reform); TSPR
(turning a lot of responsibility back to the contractors); WGS FFP lessons learned.
Overall acquisition flow chart – the process is somewhat overwhelming
44
Operational Responsive Space – acquisition innovation


Set up to help meet the needs to be more responsive; experimental approach to meet joint
command needs (initiation, assessment (non-material or material – non-material might be more
of a paradigm change), program approval, build decisions)
Four years – Four launches in order to meet the immediate needs to be much more responsive
Commercially acquiring military satellite technology



Architecture challenges
Looking at how we can effectively leverage the commercial community
2-3 yrs acquisition timeframe as opposed to the 4 year timeframe
Issues and Challenges



Timeline compression
Mission Assurance
Contractor-Government Roles and Relationships
Mike Hritz, Acting Chief Architect of National Airspace Systems, FAA NEXTGEN, ‘Next Gen FAA National
Aerospace System’:
Effective exchange medium for enterprise architecture – intersection between program management
and systems engineering
Complexities:




Integrated Flight Planning / Trajectory Based Operations
Extreme HW and SW and Human Interaction (while we try to eliminate that emergent behavior
we cannot ignore that there is significant human interaction)
Over 65K NAS operational facilities in varying states of capability – challenge will be to
incorporate some of the NextGen capabilities
Cannot just stop operations to upgrade capabilities (how do we perform maintenance without
sacrificing operations) – trying to change the tire on the car while we are still driving down the
road; significant planning activity; not just trying to build a complex system to address issues
now, but must also consider future interactions with systems moving forward
Tools:


NAS Integrated SE Framework (NAS ISEF) – use DODAF
NextGen Segment Implementation Plan
45


IMS
I2I Process
Challenges:



Establishing a complete set of views
Prevent the overlap in new sustained systems from being disruptive
Increasing overlapping dollars
Mike Ryschkewitsch, Chief Engineer, Space Exploration Systems, NASA Headquarters, ‘Human Space
Flight Crew and Cargo’:
In the business of trying to support two distinct human spaceflight tracks


Commercial support for LEO
Extended transportation for outside LEO (“un-Goldilocks”)
Four acquisition activities:




Cargo capability demonstration for ISS (milestone payments; requirements limited to
rendezvous)
Cargo delivery for ISS (normal FAR process)
Commercial Crew (high level rqmts; but there are human rating requirements; FAR based)
Beyond LEO (ground based systems extensible)
Flying is not inherently dangerous…it is just unforgiving of mistakes with man or machine…




Challenge associated with testing for the crew vehicles – don’t have the same advantage of the
experience of Soyuz
Must operate to a nearly full performance limit the first time
Loss of crew probability is still high
Difficult to map the full capability of the vehicle by stressing it…we don’t have the opportunities
Different concept of rqmts level – not as detailed on the Government side; going to rely on the industry
to help decompose those requirements
How do you design a system that have multiple use type – how do you minimize the risks? Depending
on the mission requirements the priority of the risks change…
How lean is too lean…how will we properly evaluate the appropriate level of robustness in the
architecture, design and processes? (Launch abort method; loss of pressurization issues (will need
suits))
46
How do we deal with new and rapidly evolving development methodologies in a high consequence
environment (specifically SW with respect to emergent behavior issues)
Complex vs Complicated

How do we consider the secondary, tertiary failures…
McMurray, Boeing, ‘Commercial Space Transportation’:
Build an efficient but not dangerous model….
Need to consider the interactions with International Partners and NASA
Capsule is being designed to be launch vehicle agnostic since launch vehicles drive the program costs
Leveraging off of existing capabilities in order to get some synergies (both the tangible and intangible
capabilities – meaning HW technologies at the launch facility or operations know-how on the part of
NASA, etc.); really looking at lessons learned with capsule recovery as well….
Multi-purpose usage…can carry cargo other than just the crew
There is a belief that we will eventually have a commercial LEO market



Not just space hotels…but also providing the ability to “lease” space in space
NASA is helping to establishing a commercial space environment (consistent with their space act
charter)
The more involvement of other parties in the commercial LEO market the cheaper it will be in
the long run (“waste of tax dollars – I don’t think so.”)
We think we can leverage the commercial aviation lessons learned in the commercial space
environment
Need to come to a point where the shared overhead is lower across the players…
NASA embedded personnel have not provided contractual advisement – this helps to ensure
collaboration at a level that is actually practical and successful
Challenges



No extreme technical challenges, but how the transactional and regulation types of challenges
impose risk to the technical challenges
Need to set up the correct framework now in order to help the future endeavors
Liability issues (no established risk management regimes)
47



o We are a 16-yr old driver in a new insurance environment
FAA regulates….NASA does not…but the domain knowledge is in NASA
ITAR issues – what is considered to be a defense service?
Business Case – need to be able to understand what’s in it for us
Session 3-3: Business Operations and Logistics (Wednesday, 12 Sep, 1400 – 1800 hrs)
Complex aerospace system requirements for operations and logistics are examined in this session.
Panelists will focus on operations and logistics challenges they have encountered in their programs or
organizations. Topics to be discussed will include the difficulties associated with the logistics and
operations of a complex aerospace program or organization, when dealing with the unknown
unknowns. Attendees will engage directly in the discussion to identify and address challenges faced
within this segment of the aerospace industry.
Session Chairs:
Sophia Bright, The Boeing Company
Anna-Maria McGowan, NASA Langley Research Center
Moderators:
Sophia Bright, The Boeing Company
Anna-Maria McGowan, NASA Langley Research Center
Panelists:
Jeffrey Mendenhall, MIT Lincoln Laboratory
Shannon Flumerfelt, University of Cape Town
Franz-Josef Kahlen, University of Cape Town
Ray Lytle, Director, Engineering Product Support, Raytheon Missile Systems
Summary:
Notes from Flipcharts populated during discussions:



Communication in Engineering
Transfer of knowledge
o R&D to Industry to Fielding the Product
o Using mental models to help understand stakeholders
o Consider providing funding to go out to user community
o Life-cycle follow through (Gulfstream example of core to production)
o Need to find ways to understand the user
Volume of highly detailed information vs. simplified high-level summary that tells a story
o A mental model that describes the system
48













Using examples from other industries to help consider other integration strategies
Training a manager to tell a high-level story or summary to help the team “see” the system
o Can’t drive improvement without understanding the current state
Importance of mentoring – no longer a luxury?
Mental Model
o Collinear lenses: Project Management, Engineering, Decision Making
Know the customer and think like the customer
o Focus beyond the spec
Requirements process early on needs to include reliability or trade spaces in the development or
definition
o This will help to possibly keep the sustainment costs down
Need to think of sustainment improvements as cost savings (not as a cost to production)
Assume that the customer will use the system differently than originally specified
o Consider this in R&D and in development
o Need flexibility and adaptability (turn this into a research thesis)
Lack of reality in operational cost estimating
o Ops Time Span increasing
o Ops Cost increasing
o University involvement to understand operational cost estimating
o Different business models to record operational costs?
o Public awareness to help build the business models (example Hubble Space Telescope,
weather satellites, etc)
Education Stakeholders (PM, SE, etc) to be able to evaluate or inquire about the
manufacturing/production/life-cycle costs
Treat Ops Cost like a complex system?
o Consider the stakeholders, external operations, new starts, universities, ROI, national
infrastructure, businesses, etc.
Are we overdesigning our systems for too long of a life-span?
Can we re-frame the Ops Cost Model to have a transition in them? Can we assume that they
will be operated by another user?
Session 3-4: Workforce and Education Issues (Thursday, 13 Sep, 0800 – 1000 hrs)
This session challenges assumptions and learning on these four issues: (1) Training the current
workforce for complex systems; (2) Educating / preparing the future workforce for complex systems; (3)
Organizing / structuring the workforce for complex systems; and (4) Managing and leading the
workforce for complex aerospace systems.
49
Session Chairs:
Brett Hoffstadt, The Boeing Company
Reece Lumsden, The Boeing Company
Panelists:
Brett Hoffstadt, Technical Program Manager, Vertical Lift Consortium & Project Engineer, CV-22 Osprey,
The Boeing Company
Reece Lumsden, Manager, Systems Integration, KC-46 Tanker, The Boeing Company
Steven D’Urso, Program Coordinator, Aerospace Systems Engineering, University of Illinois/UrbanaChampaign
Anna-Maria McGowan, Systems Analysis and Concepts Branch, NASA Langley Research Center
Steve Skotte, Professor Program Management and Space Acquisition Performance Learning Director,
Defense Acquisition University
Summary:
Key take-aways:


How can we look at the organization structure to avoid siloed organizations?
o Reference to Robert Quinn’s Competing Values Framework; You need both bureaucracy
and adhocracy in an organization
Training in current workforce is based on products – not in services
50
Overall Summary for Track 3 – Program Management:
Themes:




Situational Awareness is key and the way to maintain this awareness is to communicate
effectively. However, engineers inherently don’t communicate well. There are some
fundamental aspects of effective communication:
o Listening
o Knowing when and how to communicate (for example sometimes it is okay to say no to
a customer, but knowing when that is appropriate can be the trick)
Acquisition is a dynamic force and there are still some regulatory challenges that need to be
overcome before we can make some headway in this area.
In order to understand the challenge for program management we really need to understand
complex systems versus complex processes. Additionally, the difference between uncertainties
versus complexities. Knowing the nuances of these things will help programs understand what
decisions need to be made to move forward.
Additionally, evaluating issues in an organization...are they systemic or systematic? Are we
asking the right questions? And then how do we organize ourselves properly to compensate for
them?
Take-Aways:





Need to develop and organize to be adaptable
Need to understand how to engage the “right” stakeholders
Need to develop effective communication strategies (using Mental Models and using proper
taxonomy)
Need to develop a top-level framework through which acquisition and regulatory items can be
addressed
Need to have a better understanding of the issues regarding the transfer of technologies
through the life-cycle of a product
o Need to consider the “ilities” early on in the research and development phases in
anticipation of future applications
o This includes treating operation costs as a complex system
o Need to change the paradigm associated with costing sustainment
Challenges:



Requirements and scope creep is a constant challenge
Uncertainty versus Complexity
o How do we then manage the risk in the context of these two issues?
Consideration of Program Size and/or Type in terms of dealing with complexity issues
51
o
o

R&D is at the low end of complexity versus Production & Operations at the high end
Dealing with the transition and acceleration of products from R&D to Production and its
associated issues
There are two disparate themes in terms of access to space that need to be considered: 1)
DOD: Agile, small scope, and 2) NASA: Leveraging, synergy, and larger in scope projects
o How do businesses put together a long-term business model to allow the industry to
commit, particularly when considering the commercialization of space?
Outrageous Comments:




“We just lie” – from the context of cost of operations for sustainment overruns
“Safe to Share” – common statement when discussing the failures during some of the sessions
“Build taller walls around fewer items…”
“C2I – Chaos, Confusion, and Insubordination”
52
CASE Closing Session Summary
CASE Wrap-Up Session (Thursday, 13 Sep, 1030 – 1200 hrs)
Each track chair summarized key discussions and findings from their sessions.
Report Outs from Tracks
Moderator: Laura McGill, Raytheon Missile Systems
Speakers:
Track 1: Paul Collopy, University of Alabama in Huntsville
Track 2: Allen Arrington, Sierra Lobo/NASA Glenn Research Center
Track 3: Abdi Khodadoust, The Boeing Company
Summary:
Note that track speakers’ comments as presented below include their responses to other track speakers’
reports and are not presented in chronological order of the session.
Track 1 Report – Paul Collopy, University of Alabama, Huntsville:



Three kinds of complexity
o Interdependencies within the system
 Integration is messy; all pieces interact with all other pieces
 Decomposition is not possible
o Very large numbers of states and transitions
 NextGen is an example
 New validation approach needed
o Very large amount of design information
 Trying to fix every small problem that arises is a problem
Can do “build and break” without decomposing system
The form of the organization constrains development of systems
o This came up every day, in 50% of talks
o Structure matters
o The procedure for making design decisions affects the design
o Too formal a system of communications inhibits design
53

o The people side of a Work Breakdown Structure is hugely influential
Ambiguity is essential in developing complex systems
o Good chief engineers are people who can work with ambiguity
o Reliance on computers has caused loss of judgment and understanding of how systems
work
o Comment from audience: Need both precision and ambiguity
o Comment from audience: Need the right questions to enable the right answers
(example: Why X significant digits?)
Track 1 Key Takeaways:
 Learned how to be a better designer
o How to keep a system from becoming more complex than it needs to be
 Bad architecture drives this
o Too many requirements drive us away from design elegance
 Look at test culture as a way to learn rather than a way to find someone to blame
 Make designs more durable
 Define the “goodness” of a complex system
o It should not be simply meeting requirements
 Recycling, re-using, or multi-functioning things do not always save cost
o Can increase complexity and drive cost up
 We can’t possibly develop NextGen the way we have been taught to develop systems
 Cost overruns and delays are symptoms of over-complexity
 Audience comment : Complexity must buy its way into a platform
Track 2 Report – Allen Arrington, Sierra Lobo, NASA Glenn Research Center:



Engage the test community earlier, BUT…only if the test community understands what’s going on in
the design process
We have tools. Are we using them properly?
Are we training the next generation of engineers?
o We have an aging workforce; are we bridging the transition?
 Boeing initiates a new project every two-to-three years to keep trained people in
the pipeline
o Audience comment: Do we always apply the logical progression of analysis, ground test,
flight test?
 “Audacious” comment: Will advances in computing allow skipping one part – maybe
ground test – in the future?
 Could AIAA construct a roadmap to get us to that point?
54
o
The Track 2 Verification and Validation session focused on large numbers of lines of code
and simulation.
 Must ensure early test involvement for effective V&V
Track 2 Key Takeaways:
 Many good ideas, some require cultural transformation
 Starting Monday:
o Better communication
o Attention to requirements fidelity and stability
 More up-front attention to analysis of alternatives
 Perhaps the AIAA Standards committee could take on the challenge of standardizing definitions and
nomenclature for these tracks
Track 3 Report – Sophia Bright, Boeing:





Situational awareness is very important for the systems engineer
o Full capabilities of a system emerge over time
o Many questions need to be asked early in order to better handle complex systems
o Engineers must understand supportability and maintainability
Differentiate systemic issues from systematic issues
o Incorporate into how we train our engineers
Communication is important
o Engineers don’t do it well
 Tools: Mental model; pictorial
 Look for better/more effective ways
 Listening is important for program managers, engineers, etc.
o Be able to know when and how to say no
 Audience comment: Innovation can cause this to be an issue
Overall, acquisition issues are very complex
o Issues can make our lives harder
Track 3 Key Takeaways:
 We make systems more complex than they need to be.
o Focus on good architecture.
 Taxonomy – make sure we define terms before moving forward
o Better define complex vs. complicated.
o What is uncertainty?
 Training is important
55



Need to develop a path forward for commercial use of space and air traffic control
Develop a process for transferring technology/knowledge throughout a program, from design
through operations
o Production needs to understand design
There is a changing paradigm of costing sustainment
o What is real vs. perceived value?
 Seems to be left out of costing models
The Next CASE Event – Laura McGill, Raytheon Missile Systems:
The audience offered several recommendations for future CASE conferences:




No more than three tracks
Complexity is bigger than aerospace
o Bring in other disciplines
o Learn from their experiences
Facilitate more intermingling of tracks
o Encourage cross-attendance of PMs, testers, designers, budgeteers, etc.
o Step back and look at different ways to do things
 Cross-track
Budget is important
o Relate to complexity
o Keep at high-level
Closing Remarks
Speakers:
Michael Griffin, Schafer Corporation
The systems engineering (SE) community should not forget that good SE is about creating an elegant
design. Elegance consists of a design that
 Has efficacy (works);
 Is robust;
 Is efficient;
 Is as free as possible of unintended consequences
Additional Considerations:

SE is about sociological considerations, but not only these
56







SE is the link between analysis and design
o Schools teach engineering science
 How to analyze someone else’s design
 This is only half the story
o SE includes creating that man-made world
o Designers and analysts don’t think the same way
 Analysts pick apart designs and find flaws
o Engineers link designers and analysts
Look for “intelligent decomposition” of systems
o Reduce unintended consequences
Engineers exist to turn ambiguity into knowledge
o Good engineers have a tolerance for ambiguity
 Can stand it longer than most
o Don’t turn ambiguity into soup too soon!
Human cognition must be factored in
o SE is a team sport
o We have learned a lot about how humans decide things and the effects of consequences on
our cognition
Confirmation biases: We look for information that supports our conclusions and ignore other
information.
Distinguish between precision and correctness in the SE process
Sometimes we must “rise above principle”
o Must we always be precise?
 It is better to know some things to four significant digits than none to six
57
Final Remarks on CASE 2012
Through its three tracks, the CASE 2012 conference explored the challenges facing the large number of
industry technical professionals who are directly engaged in the development, deployment, and support
of complex aerospace systems. The discussions highlighted numerous lessons learned and best
practices, and also identified areas of study that hold potential for new systems engineering methods.
Opportunities may be leveraged from the formal sciences of mathematics and logic, as well as the
natural and social sciences, and research is underway. In the meantime, we must maintain our focus on
technical and program discipline in the execution of our projects, to minimize the instances and effects
of engineering failures.
Although we don’t have all the answers, there is great value in bringing these issues forward and
providing a forum for open discussion and the exchange of ideas. Future CASE conferences will build on
this foundation, with the goal of further advancing systems engineering to accommodate the challenges
of tomorrow’s complex aerospace systems.
58
Acknowledgements: The CASE 2012 Planning Committee
The first AIAA Complex Aerospace Systems Exchange Conference was the result of the outstanding
efforts of the following AIAA members that each contributed untold hours to conceive of, develop, and
conduct the CASE 2012 conference:
Dr. Michael Griffin, Schafer Corporation
Executive Program Chair
Laura McGill, Raytheon Missile Systems
Paul Collopy, University of Alabama in Huntsville
Allen Arrington, Sierra Lobo, Inc. NASA Glenn Research Center
Scott Boller, Pratt & Whitney – Rocketdyne
Abdi Khodadoust, The Boeing Company
Sophia Bright, The Boeing Company
Program Chair
Track 1 Chair
Track 2 Chair
Track 2 Deputy Chair
Track 3 Chair
Track 3 Deputy Chair
Nancy Andersen, Lockheed Martin Space Systems
David Dress, NASA Langley Research Center
Mat French, Rolls-Royce Corporation
Brett Hoffstadt, The Boeing Company
Ron Kohl, Ronald H. Kohl and Associates
Paul Lambertson, The Boeing Company
Jeff Laube, Aerospace Corporation
Reece Lumsden, The Boeing Company
Chi Mai, Texas A&M University
Doug Marshall, New Mexico State University
Jim Murphy, NASA Ames Research Center
Planning Committee
Planning Committee and Session Chair
Planning Committee
Planning Committee and Session Chair
Planning Committee and Session Chair
Planning Committee
Space 2012 Program Chair
Planning Committee and Session Chair
Planning Committee
Planning Committee
Planning Committee and Session Chair
59