American Institute of Aeronautics and Astronautics
(AIAA)
Complex Aerospace Systems Exchange (CASE)
Pasadena, California
September 11-13, 2012
Report of the Sessions of Track 2
CONTENTS
Topic
Page
DAY 1
CASE Opening Plenary Session
2
CASE Track 2, Session 1: Planning and Executing an Integration Test
Strategy for a Complex Aerospace System
3
DAY 2
CASE Track 2, Session 2: Integration of Modeling and Simulation,
Ground Test, and Flight Test
11
CASE Track 2, Session 3: Verification and Validation Issues
20
DAY 3
CASE Track 2, Session 4: Lessons Learned in Integration, Test, and
Verification
26
CASE Closing Session
30
Recurring Topics
36
1
Complex Aerospace Systems Exchange (CASE)
American Institute of Aeronautics and Astronautics (AIAA)
Pasadena, California
September 11-13, 2012
Report of the Sessions of Track 2
Day 1
CASE Opening Plenary Session
Laura McGill, Deputy Vice President of Engineering, Raytheon Missile Systems,
AIAA Vice President, Standards, General Chair (CASE): Opening statement
 Why CASE?
o Systems integration becoming more complex.
o Opportunity for experts to share knowledge.
o Program is focused to benefit various players in systems integration:
– Track 1: Chief engineers
– Track 2: Test professionals
– Track 3: Program managers (PMs)
o Should be of interest also to mid-career professionals moving from
specializations to Integrated Product Team (IPT) positions.
 Next CASE scheduled August 12-13, 2013, Los Angeles Hyatt-Regency.
o Will address hard-hitting issues around program execution:
– Budgets
– Schedules
Michael Griffin, Chairman and CEO, Schafer Corporation, AIAA President,
Executive Chair (CASE): Opening statement
Theme: Systems Engineering (SE)
 SE does not yet have well-defined theory.
o We can identify the great practitioners, but we are not sure why they
are great.
 Projects succeed or fail by the ability of systems engineers to foresee failures
and determine mitigations.
2
Day 1 (continued)
CASE Track 2, Session 1: Planning and Executing an Integration Test Strategy for
a Complex Aerospace System
This session was chaired by Phil Stich, Deputy General Manager, ATA, Arnold
Engineering Development Complex, and moderated Ed Kraft, Chief Technologist,
Arnold Engineering Development Complex. Panelists are identified below with their
individual presentations.
Moderator’s Introduction
Ed Kraft, Chief Technologist, Arnold Engineering Development Complex:
Planning and Executing an Integration Test Strategy for a Complex Aerospace
System
Theme: This presentation was an introduction to the session topic and sets the stage
for further discussion by attempting to establish some definitions and boundary
conditions. The moderator contrasted aerospace industry development cycle times to
auto and Integrated Circuit IC industries where rapid fielding is paramount to market
success.
 System complexity is not inherent in rampant, runaway acquisition time cycles.
Reasons may be found in:
o Poor architecture choices.
 Aerospace systems start with a clean sheet in each program.
 Auto and IC industries use common platforms over time.
o Processes and process ownership.
 Many are outdated and need improvement.
 Auto and IC industries own their own processes.
o Accountability
 Problem in aerospace industry.
 Program reviews are not sufficiently demanding.
o Capacity
 Aerospace capacity is periodically diminished by cycles of cutting
back after major wars and conflicts.
 Diminished capacity drives development costs up.
 The complexity of processes and environments are often more severe than the
complexity of the complex system under development.
o This is an organizational problem.
 Definitions: Complex vs. Complicated
o Complex
 Parts are interdependent, often changing, and outcomes are less
predictable.
3

 Hard to ”see” the whole, but only what’s available from your
vantage point.
 Complex systems are not constructed, they are grown.
 Complex systems are dynamic—that is, they’re constantly changing
and adapting to current conditions.
o Complicated
 Not simple, but knowable.
 Clear links between causes and effects.
 Tend to give predictable results, as long as the same thing is done
the same way, each time.
 Focus needs to be on diligence and accuracy when following
particular steps in a process.
o Do not confuse the two, and integrate as early as possible.
 Have complicated systems interact with complex environments as
soon as possible.
 Modeling and Simulation (M&S) allows early integration of the
system and its environment.
 Can use physics-based codes to generate solutions that can
be plugged directly into war games of flight simulators.
Key Take-Aways:
o Do not blame expanding development cycles entirely on system
complexity.
 Don’t exacerbate the impacts of system complexity on development
cycles by the use of bad architecture, processes, poor
accountability, or inadequate capacity.
o Understand the difference between complicated and complex, and know
how to treat each one.
Ming Chang, Lockheed Martin Fellow, Lockheed Martin Corporation: Complexity
from the Defense Original Equipment Manufacturer (OEM) Perspective
Theme: This presentation outlines the most essential elements of successful
Verification and Validation (V&V) of complex systems from the perspective of an original
equipment manufacturer of military aerospace systems.
 Successful development is dependent upon
o Understanding requirements
o Feasible design
o Early Identification of risks
 Removal of defects early is essential to maintaining lower life-cycle costs.
 Perform V&V in all phases of the development, at all Technology Readiness
Levels (TRLs).
4





V&V of various systems and disciplines in complex systems are contrasted
Flight safety is foremost, a cradle-to-grave culture.
Validation is a continuing process using wind tunnels (WTs), structural models,
etc.
Verification is a real-time process with processor-In-the-loop (PIL) / human-inthe-loop (HIL) /system-in-the-loop (SIL) simulation.
Key Take-Aways:
o Processes matter
 Processes differ among types of systems or phases of
development.
o Early V&V is essential, continue through all phases.
o Early dependence on legacy systems can lead to complexity creep as
system matures.
Barry King, Director, Space Test & Operations, Dynetics, Inc.: Integration & Test
of Complex Programs (Commercial Perspective)
Theme: This presentation outlines the most essential elements of successful V&V of
complex systems from the perspective of a manufacturer of commercial aerospace
systems.
 Program integration requires good situational awareness of the who-what-whenwhere-how-why of the program life cycle.
o Frequently don’t plan well enough for later stages.
 Program integration roadmap is a combination of those of the Department of
Defense (DoD) and NASA
o Some documentation eliminated.
 Stakeholder needs, goals, and objectives (NGOs) determine what needs to be
done.
o Key starting point for program and Level 1 requirements.
 System engineers and test engineers are link from the outset.
o Relationships mapped early for systematic execution.
o Early test integration averts systems being “thrown over the wall” to test.
 Key Take-Aways:
o In order to have a successful program, all levels of requirements must be
linked and verifiable along the way
 Analysis, model or test
o Most common mistakes with greatest impacts:
 No base line and/or fluid requirements;
 Integration and test personnel not on design team;
 Allowing design engineers to be involved too long;
 Too little time allowed for testing:
5

35% of the schedule for integrated test operations ITO
following 65% for design is inadequate;
 Processes or procedures, rather than experience, are relied on as
the basis of discipline and completeness.
o Include a recent graduate on project teams to question “dumb” rules and
point to ways to “fix” procedures.
Tom Irvine, Deputy Associate Administrator, National Aeronautics and Space
Administration (NASA) Headquarters, Aeronautics Research Mission Directorate:
Complexity and the Research Community
Theme: This presentation outlines the most essential elements of successful V&V of
complex systems from the perspective of the research community.
 Observation of natural complex systems is instructive in understanding
engineered complex systems.
o Used the kelp tank at the Monterrey Aquarium as an example of a natural
complex system.
 There are similarities and differences in natural and engineered complex systems
identified in the presentation.
o Both natural and engineered complex systems:
 Have dynamic environments that are constantly changing and
adapting to current conditions
 Have interdependent parts and outcomes which are not always
predictable
 Require a holistic, collaborative approach
 It is difficult to see the whole but rather only what’s available
from your vantage point
 Are complex systems that were grown and evolved over time and
were not constructed all at one time.
o Natural and complex systems differ in that:
 A natural system is a chaotic environment where life and death are
certainties.
 Chaos is normally unwelcome in engineered systems or systems of
systems.
 The natural system is a continuous experiment during all phases of
its life cycle.
 The engineered system or systems of systems must effect
purposeful integration, and test and evaluation.
 Understanding of the interdependencies and outcomes by external observers is
not required in the natural complex system; it simply evolves according to a
natural order.
6


o In the engineered complex system, we must decompose it into parts we
can understand in order to positively affect design, development, test,
evaluation and operations.
 This decomposition often leads to sub-optimization.
Referring to the SE “V” diagram, the decomposition and definition process
usually works well.
o Re-composition and integration is the difficult part and is where we usually
fall short.
 Research and Development (R&D) (both foundational and focused)
is greatly affected by this.
 R&D is symbiotic with integration, test, evaluation, verification, and
validation.
 R&D may be either the developer or the user of test and
verification capabilities.
 V&V costs will swamp program costs unless new tools come out of
R&D
Key Take-Aways:
o Emulate, whenever possible, the incremental growth and development of
natural complex systems when engineering such systems and systems of
systems.
o Recognize the risk of sub-optimization when decomposing complex
systems for understanding.
o Use R&D to develop and use new test tools to enable affordable and
effective V&V of complex systems and systems of systems.
Ward Johnson, General Manager, Jacobs Technology Group: Planning and
Executing an Integrated Test Strategy for Complex Aerospace Systems (from a
Test and Evaluation [T&E] Infrastructure Perspective)
Theme: This presentation outlines the most essential elements of successful V&V of
complex systems from the perspective of the research community.
 Jacobs Engineering has more than 8,000 people involved in the design,
operation, and stewardship of test facilities worldwide.
o T&E facilities are very expensive and highly technical.
o It takes as long to develop some T&E facilities as it takes to develop the
systems they support.
 Ensuring successful T&E program delivery requires:
o Early and effective up-front understanding with stakeholders
o Team with equipment suppliers and/or other design/build suppliers to
mitigate risk and minimize schedule
 T&E infrastructure challenges in the United States:
7







o Maintaining and improving essential core test capabilities.
o Preserving specialty test capabilities through investment in the T&E
workforce.
o Ensuring enhanced collaboration and cooperation across the T&E
enterprise.
o Capitalizing on simulation and technology in support of evolving integrated
test processes.
Key Take-Aways:
The U. S. test infrastructure is old.
o We have sufficient capability and capacity.
o It is not being properly maintained or modernized.
Need more collaboration during lean budget periods.
Must capitalize on simulation and technology in support of evolving integrated
test processes.
Two questions were posed to industry leaders:
o Are programs more focused on long-term technological advancement than
near-term client-defined safety, reliability, and sustainment?
o Are we properly balancing risk with responsiveness and cost?
Program teams must learn from the past…do not reinvent.
T&E and Facility Teams must:
o Question requirements that come as solutions
o Define the end before you start
Eileen Bjorkman, Chief Technologist, 412 Test Wing, U. S. Air Force: Flight
Testing of Complex Systems: Approaches and Lessons Learned
Theme: This presentation outlines the most essential elements of successful V&V of
complex systems from the perspective of the flight test community.
 Complex systems context
o Security requirements cause some difficulties.
o Complexity exists because of people, processes, policies, technology,
engineering, management, and social sciences.
 Solutions require interdisciplinary approaches.
 Complex systems test challenges
o Reduced budgets
o Pressures to close facilities
o Do more with less; accelerated acquisition timelines.
o Range encroachment (geographic, competing technologies (wind
turbines) and Radio Frequency spectrum encroachment .
o Integrated government /contractor facilities
o Aging workforce.
8





Flight test approaches: Distributed testing
o The B-1 strategic bomber used distributed testing
 Full-up B-1 testing would have required prohibitive resources and
assets
o Short time-line drove Global Hawk to distributed testing.
o Ensure environments are “accredited” when using distributed approach.
Fusion Lessons Learned
o Use building block approach
 Understand how individual systems work
o Plan for right assets and ensure they are fully functional, anticipate
problems, build in flexibility, identify key objectives and focus efforts to
ensure success in those areas
We are drowning in data; eliminate unnecessary data.
Simpler may be better.
o Exercise critical functions in less complex scenarios when able.
o “Exercising everything” early may create more questions than answers.
o Develop confidence in the system before engaging complex scenarios.
Key Take-Aways:
o Use what we have; repurpose, recycle, reuse, rely on others.
o Complex systems may require new skill sets.
o Complex systems may require multiple program offices.
o Need to understand distributed M&S.
 A “discipline” engineer with a Master’s in SE is the best combination.
George Rumford, Deputy Director, Project Development, Department of Defense
Test Resource Management Center: Planning and Executing an Integration Test
Strategy for a Complex Aerospace System
Theme: This presentation described the DoD’s Test Resource Management Center’s
(TRMC’s) structure, its Congressionally-mandated mission for DoD test infrastructure
oversight, including its relatively-recently assigned responsibility for the Joint Mission
Environment Test Capability (JMETC) Program and the National Cyber Range (NCR).
 The briefing contains organizational charts and mission statements for TRMC.
 The traditional TRMC mission of range oversight is established under Section
231 of the “Bob Stump” NDAA for FY03 (PL107-314).
 Responsibilities for JMETC and NCR were established in 2005 and 2012,
respectively.
 Cyberspace vulnerabilities exist in DoD weapons systems
o The DoD has inadequacies in its cyber range capability
 Inadequate environments
9

 Costly and time-consuming to build a federated cyberspace
environment
 Participant stovepipes delay integration of cyberspace capabilities
o Realistic cyber environments are needed.
 Must determine whether systems can fight through a denial-ofservice attack and still be responsive to the warfighter.
 Cyber has recently become more complex.
o The NCR will represent :
 A persistent, operationally realistic cyberspace environment with:
 Interoperability Standards that break down “stove pipes”
 Rapidly integrated Live, Virtual, & Constructive capabilities
 Common test & experimentation tools and applications
o In terms of the SE “V” diagram, the NCR will:
 Identify cyberspace system and sub-system dependencies
(decomposition side)
 Test systems of systems mitigation of cyberspace threats
(recomposition/V&V side)
Key Take-Aways:
o Tenets of “good” government and industry systems integration testing are:
 Derives from mission-focused requirements & test plans
 Combines a tiered management structure & accountability
 Uses agile development concepts
 Leverages existing government and industry test capabilities
 Employs interoperability standards that enable rapidly integrated
live, virtual, & constructive capabilities
 Shares common test & experimentation tools and applications
 Unifies approach in conducting both cyberspace and “traditional”
testing
 Applies effective knowledge management techniques & structures
Phil Stich, Deputy General Manager, ATA, Arnold Engineering Development
Complex, Session 1 Chair: Threads from Track 2, Session 1
The following is a summary of take-aways presented by session chair, Phil Stich, to the
Track 2 chairs at the evening summary meeting following Session 1:
 Requirements fidelity and stability
o Spend more time up front with analysis of alternatives (AOA) before
proceeding
o Beware of requirements that come as solutions
 Acquisition process
10

o Must agree that our current process is cumbersome, especially for military
advanced development projects.
o Commercial aircraft uses more streamlined process
Integration of complex systems is very difficult
o Subsystem test methods can give wrong results – or incomplete picture
o Fused data – flight test example
11
Day 2
CASE Track 2, Session 2: Integration of Modeling and Simulation, Ground Test,
and Flight Test
This session was chaired by George Sydnor, NASA Langley Research Center, and
Terrence Trepal, Institute for Defense Analyses (IDA). The session moderator was Mark
Melanson, Lockheed Martin Corporation. Panelists and speakers are identified below
with their individual presentations.
Tim Marshall, Director, Aeronautics Test Program, NASA Aeronautics Mission
Research Directorate: Integration of Modeling and Simulation, Ground Test, and
Flight Test
Theme: This presentation provided an overview of the mission and strategies of
NASA’s Aeronautics Test Program (ATP) and the Aeronautics Research Mission
Directorate (ARMD) of which it is an important part. Included were observations
stemming from NASA’s experience in integrating complex systems.
 ARMD’s programs have many interfaces.
o Most errors and innovations in complex aerospace systems development
occur at program interfaces.
 ATP is the ground test and flight test element of ARMD.
o ATP provides the underpinning for the four research programs of ARMD.
 Provides testing for three of the four.
 ATP’s mission is to support NASA in every way possible:
o Provides strategic management guidance and recommendation to the
NASA ARMD and Centers with respect to the ATP portfolio.
 Basis for ARMD decisions to close or maintain facilities.
o Represents NASA and the nation’s interests for aeronautics testing.
 40% to 50% of ATP testing is for NASA; 15% to 20% is for DoD;
remainder is other external customers through Space Act
agreements.
o Provides strategic direction to capability managers across the portfolio.
 Sometimes recommends reliance on non-ATP assets.
o Provides financial support to sustain and enhance test facility capabilities
and related workforce.
 Must balance sustainment with new capability.
 ATP’s investment strategies are to enhance capability (higher reliability,
relevance, etc., therefore higher value) at reduced operational cost (lower cost to
users).
o “Value sweet spot” graph was shown: A region on the Cost vs. Capability
map that is considered the target for any given capability (See briefing).
12



o Capabilities operating outside the boundaries of the “sweet spot” are
sought to become compliant by:
 Improving capability at constant operating cost.
 Lowering cost for a given capability.
 Or some combination of the two.
Challenges:
o Lack of rigor in planning a test; poor estimating by both the developer and
the provider.
o Lack of engagement between service providers and developers.
 Best way to ensure success is early and frequent communication
with the test providers.
Can/should the providers provide integration services for M&S, ground test and
flight test, or is it the responsibility of the developer to determine the tools to use?
o Perhaps the provider can have the tools available and offer the
capabilities.
Key Take-Aways:
o Improve the value of capabilities to customers.
 Stay engaged with customer needs.
o Leverage investments for maximum benefit.
o “Teach” customers about capabilities
 Early interaction is critical.
 Helps streamline the integration of Ground Test (GT), Flight Test
(FT), and M&S.
 Incomplete early planning often leads to trouble.
Tom Fetterhoff, Technical Director, Test Division, Arnold Engineering
Development Complex: Arnold Engineering Development Complex (AEDC)
Theme: This presentation discusses the use of integrated test and evaluation at AEDC
and how AEDC seeks to better integrate with FT.
 The new Air Force Materiel Command (AFMC) “Test Center” organization
combined GT and FT complexes.
o Seeks better integration of testing tools.
o Goal is lower test cost and higher quality solutions.
 AEDC has used integrated test and evaluation (IT&E) for many years.
o M&S enables better insight into data obtained in GT and FT
o The use of M&S integrated with GT in several programs was discussed.
 Examples included F-15, F-22, B-1, F-101 engine.
 Disciplines in examples included store separation, inlet
performance, engine repair.
13








Store separation has moved toward more M&S, less GT and
FT.
AEDC works to expand contribution of M&S to early program planning (during
the Analysis of Alternatives [AoA] phase)
o Today WTs provide high-fidelity data faster than computational fluid
dynamics (CFD); this could change in the future.
AEDC seeks to provide customers with information, or knowledge, not just data.
o Audience comment: Customer is still responsible for the design and
development, and might not want all the help that is being offered.
AEDC’s space chamber work that supports the Missile Defense Agency is very
robust and growing in the sensor testing area where IT&E is beginning to be
utilized more.
AEDC integrates M&S, GT data, and FT data to develop a plume signature data
base for all rockets..
Hypersonics requires the use of Test and Evaluation (T&E) and Science and
Technology (S&T) to develop tools and techniques to support future systems.
Recent changes to DoD 5000 series regulations requires early communications
o This will drive testers and users to work more closely together.
o Will help IT&E methodology work better.
Key Take-Aways:
o Purely experimental V&V is no longer viable.
o M&S has matured to the point that it is capable of offloading testing to
varying degrees.
o Early interactions between customers and test professionals enable
better, faster, cheaper solutions.
o Effective IT&E early can facilitate/enable streamlining FT.
o Stakeholder feedback enhances AEDC capability.
George Eitelberg, Director, German Dutch Wind Tunnels (DNW): Integration of
services by an independent service provider
Theme: The presentation discusses the culture and challenges of the independent test
provider and how the German-Dutch Wind Tunnels (DNW) support the integration of
T&E services with its customers.
 DNW is an independent provider of T&E services, supporting research (15%),
military test (40%, of which half is transport), and civil (45%).
o DNW’s product is reliable data.
 Workload is uneven/highly volatile.
o Heavy usage in some years and light usage in others can lead to looking
for other areas of expertise.
 A jack-of-all-trades approach can lead to poor support
14




 Specialists are needed in all areas; one person cannot cover
multiple disciplines well
 Horizontal integration can alleviate pressure, but may lead to
scheduling conflicts
 If you redeploy staff into other disciplines, you will usually not get
them back.
 DNW knows where to find specialists when needed.
 Audience: AEDC does a good job of bringing all tools
together; Lockheed-Martin (L-M) used their services on the
F-35.
DNW integrates with the research community
o The capabilities and limitations of numeric and wind tunnel simulations
need to be understood
o Universities assist with methodologies for IT&E on development programs
o GT is not properly taught; should be thought of as “simulation of flight.”
Network with customers to understand their processes and future needs
o Example: Two problems remain in civil aeronautics: Noise (aeroacoustics) and propulsion integration (fuel consumption).
Maintenance is managed by internal DNW staff, but it is outsourced as much as
possible.
o Provides flexibility.
Key Take-Aways:
o Independent providers don’t have to be politically correct…just correct.
o A jack-of-all-trades is a master of none.
 DNW does not provide one-stop shopping.
o If operational cost is a concern, expertise is endangered.
o IT&E methodology needs stability to be effective
 This is difficult because it is a project-driven process.
 Support from research organizations can help solve the problem.
o Independent service providers must depend to some extent on
institutionally-funded sources for development.
o Retain flexibility to enable optimum contribution in each area of expertise.
Andy Garrell, Technical Director, Aerospace Testing Manager – Transonic Wind
Tunnel, Calspan: Calspan – A Provider’s Perspective
Theme: This presentation discussed Calspan as an independent test service provider
with one wind tunnel, striving to incorporate new technologies to improve results and
make usage more efficient.
 Technological advancements have affected Calspan’s service area contributions
o Improved efficiencies, data quality and operational approaches
15






o Resultant reduced usage has made sustainable utilization more of a
challenge.
 Lower usage leads to higher unit costs because fixed costs remain.
o Calspan’s continual improvement program has led to new technology
advancements.
 Primarily in computer automation and CFD applications
Integration of M&S/GT/FT is not happening quickly at Calspan
o Attributed to Calspan’s small, independent status
o They never see the user’s M&S people…only GT people.
o Integration with M&S or FT is considered the customer’s responsibility.
For maximum success, developers should emphasize:
o Quality physical models brought to the WT.
o Interaction between M&S and GT providers
 Regularly include CFD person(s) in on-site ground tests
Shortfalls that exist in M&S and GT can be offset be each discipline’s
counterpart.
o Mesh M&S and GT early and often; integrate the program teams.
o Eliminate competition between M&S and GT communities.
Areas for improvement:
o Quality of the test article.
o Involve CFD with GT.
o M&S and GT can each be strengthened through cooperation.
User issues:
o Lack of planning
o Timeliness of planning
o Lack of coordination with the test article development and the test facility.
Key Take-Aways:
o Small size and independent status inhibit major integration of new
technologies and T&E disciplines
o Major efficiency improvements create a challenge and will drive the unit
cost of testing up.
o Innovation is difficult to implement with older people. (Based on a question
from the audience.)
 Older people rely on what has been successful for them in the past.
Bryan Herdlick, Senior Systems Engineer and Project Manager, Johns
Hopkins University / Applied Physics Laboratory: SoS-based Capability
Development Characterizing Performance Through Integrated M&S and
T&E…
Theme: In the area of integrated testing, the Applied Physics Laboratory (APL)
sometimes serves as a “trusted agent” for the government, much like a Federally
16
Funded Research and Development Center. This presentation discusses APL’s
integration of M&S with WT and FT for the verification, validation and accreditation
(VV&A) of systems of systems (SoS).
 APL’s mission is to solve complex research, engineering and analytical problems
that present critical challenges to our nation.
 SoS testing is outside the comfort zone of conventional IT&E
o Multiple platforms, weapons, networks require operators in the loop
o M&S is integrated with GT and FT to augment live T&E
o Existing DoD IT&E capabilities don’t handle SoS well
 “Ideal” organization for SoS testing would address:
o Use of live, virtual, and constructive (LVC) environments, chronologically:
 Constituent system testing
 Integration and functional-level “testing”
 Full integration and capability-level “testing” composite capability
o Integration of M&S is essential due to expense, scheduling, and operatorin-the-loop (OITL) considerations of SoS testing
 FT of SoS is often untenable.
 Must characterize SoS capability vs. testing to a specification.
 SE approach using “federations” of models is used to address SoS capability
development.
o Some models not as rigorous as they should be
o Models from various programs must work together
 Some elements of models not tested by individual programs
o Models found lacking must be challenged up front
o Operator-in-the-loop is the ultimate proof
 Key Take-Aways:
o Establishing trust in SoS testing is difficult.
o Must apply sound SE methodologies to SoS V&V to mitigate risk.
o Enhancing M&S to support T&E in SoS is a challenge.
o Prefer more V&V in high level architecture (HLA) federation development
guidance.
o Virtual and constructive are not well-integrated
o Live testing is often difficult for SoS
o OITL in SoS testing is a “must-have,” as early as possible.
Jay Dryer, Director, Fundamental Aeronautics Program, NASA Aeronautics
Research Mission Directorate: NASA – Integration of Modeling and Simulation,
Ground Test, and Flight Test
Theme: This presentation describes NASA’s approach to the integration of testing
services and steps they are taking to overcome perceived shortfalls.
 NASA’s Fundamental Aeronautics Program (FAP) exists to conduct fundamental
research that will generate innovative concepts, tools, technologies and
knowledge to enable revolutionary advances for a wide range of air vehicles.
17




o The FAP develops and provides knowledge, tools and technologies to
enable others to develop advanced vehicles.
o FAP integrates testing into its research by working closely with providers
FAP is undergoing or facilitating changes:
o Increased emphasis on high-fidelity simulators
o Growth in computational capabilities
 Continued growth may require new architectures
o More FT in fundamental research
o Merging Research & Technology (R&T) and T&E capabilities is expected
FAP is improving its performance in two key areas:
o Risk mitigation and requirements development
 Develop requirements early, avoid creep
 Communicate!
 Take risks early on to avoid bigger risk down the road
 Ensure that the “integration” aspect amongst simulation, ground
and flight test is a part of the requirements phase
 Be creative with requirements
 Example: It may be better to invoke FT earlier to help inform
the development of other capabilities
Shortfalls in M&S, GT, and FT services were discussed and are being dealt with
o One recurring theme of note was that impact from improved coordination
of M&S with ground and flight experiments has resulted in more efficient
testing.
 This fact, along with reductions in the number of programs, creates
a financial strain on facilities that also affects user costs.
Key Take-Aways:
o Need to better integrate fundamental research into programs and test
capabilities
o Sustainable resource models for test facilities and ranges must be
implemented
Barry Lakinsmith, Deputy Director, Army Aeroflightdynamics Laboratory, U. S.
Army Aviation and Missile Research, Development, and Engineering Center:
Army / Rotary Wing Vehicle Perspective
Theme: This presentation focused on the unique challenges of rotary wing T&E.
 The rotary wing community uses M&S, GT, and FT
o Cross-specialization of Army personnel through partnerships with NASA
and the Air Force is needed
 Many rotary wing programs are not successful.
o Better IT&E is needed to assure success.
 Traditional testing (small-scale and full-scale WT and FT) has not predicted
helicopter aeromechanics and flight control needs well.
18




o In a CV22 fuel dump example, CFD was a better predictor of the actual
aerodynamic phenomena than was a wind tunnel.
CV22 is manageable; helicopter modeling is more difficult, but improving.
o Coupling rotor Computational Structural Dynamics (CSD) with CFD gets
within 2% of flight.
o Rotor systems are very complicated, might be years before we can
properly model
o Gathering small-scale WT, full-scale WT, and flight data to develop codes
to predict rotary wing performance.
Improved tools are needed to support Future Vertical Lift (FVL) initiative
o Existing tools might be made to work with time and resources
Audience: Rotary wing safety record would not be tolerated in civil aviation.
o Rotorcraft fly close to the ground
o Many accidents are caused by human error.
o Briefer: Doesn’t think physics is the problem.
Key Take-Aways:
o Improved codes are needed to eliminate need to over-design.
o Must have multiple tools and good specialist engineers in each discipline.
 Integration specialists are also needed.
o Computational power is limiting.
 We can compute one rotor revolution per day; we need to compute
one revolution per hour.
o Systems are coming that will demand capabilities to model even more
complexity.
John Curry, Director, Spacecraft Advanced Development, Sierra Nevada
Corporation: Integration of Modeling and Simulation, Ground Test, and Flight
Test in the Dream Chaser Program
Theme: This presentation described the integration of M&S, GT, and FT from Sierra
Nevada Corporation’s user perspective, using their Dream Chaser program as the
example.
 Dream Chaser is a complex system with a specific objective: Build and fly a crew
space transportation system using lifting body design that can achieve Low Earth
Orbit cost-effectively.
o Reverse –engineered Langley Research Center HL-20 design
 Enabled improved fidelity where needed.
o M&S is being used extensively to build data base.
 Cheaper, but adds risk
 Some areas of the flight regime must be done only with M&S and
accept FT risk.
o Shroud-less design was used to eliminate complexity in abort phase.
19

 WT used to demonstrate that design without shroud is feasible.
 Low budget requires WT to be correct
o A big challenge in GT is whether the model is exactly what flies.
 If not, what problems are introduced by the differences?
o Analysis is now complete; now doing helicopter drop tests.
 Moving to more complex parts
o Cannot accept low TRLs; all parts must have flown before.
Key Take-Aways:
o Requirements creep hurts the program.
o Not all great ideas are requirements.
o A simple fly-off would be best.
o Use experienced people.
 You don’t know what you don’t know.
o Often spend too much time on contingencies and not enough on nominal
situations.
 Accept some risk and/or limited understanding and move on.
o Use actual or historical data to estimate mass in launch vehicles. Mass is
critical!
o When faced with less funding, do less and accept risk.
 Explore how things can be done rather than “can’t do.”
o Collaborate with NASA, but don’t accept too much oversight.
Mark Melanson, Lockheed Martin Corporation, Session 2 Moderator: Closing
Comments for Track 2, Session 2
 Integration of the various tools for V&V is the required future state
o Specialization must be maintained
 Need computational tools to be more synergistic in matching WT data.
 Should AIAA host a workshop for the three communities to map out a strategy?
o Audience comment: DoD has ongoing Computational Research and
Engineering Acquisition Tools and Environments (CREATE) program to
develop and deploy computational engineering tool sets for acquisition
engineers.
20
Day 2 (Continued)
CASE Track 2, Session 3: Verification and Validation Issues
This session was chaired by Ron Kohl, R. J. Kohl & Associates, (Moderator) and Misty
Davies, NASA Ames Research Center. Panelists and speakers are identified below with
their individual presentations. Kohl indicated that a session report separate from this
one will be provided.
 A listing of references (standards, best practices, guidebooks, etc.) related to
V&V or Integration and Test (I&T) was distributed. It is provided at the end of this
section of this report.
Invited Speakers
Wilson Felder, Director, William J. Hughes Technical Center, Federal Aviatrion
Administration (FAA): Challenges in Testing NextGen
Theme: NextGen is a comprehensive initiative that integrates new and existing
technologies, procedures and policies for the transformation of our National Airspace
System (NAS), making it flexible and sustainable. This presentation provided insights as
to the challenges of testing such a complex system.
 NextGen is a comprehensive system with new and existing technologies,
procedures, and policies.
o It involves many people and seven U.S. Government agencies.
 Navigational performance/area navigation, dependent surveillance, digital
communications, and 4-D trajectories are all new paradigms.
 It is being built in pieces and layers.
 There are dependencies on avionics, communication systems, and other
elements in the aircraft that cannot be brought into a laboratory.
 NextGen is a complex system that must be thoroughly tested before
implementation, and must be shown to be safe for future systems.
 Key Take-Aways:
o LABNET created that links all relevant systems as an assessment
platform for complex systems-of-systems.
o Using “come-as-you-are” strategy to test interfaces with whatever enters
the system.
o Removes stovepiped laboratories and allows one integrated toolset.
Mat French, Electrical Systems Engineer, Rolls Royce, Co-Chair, AIAA
Communication and Membership: Aerospace System Integration Working Group
(ASI-WG)
21
Theme: AIAA is creating a working group to promote and contribute to aerospace
system integration (ASI) across the spectrum of applications ranging from traditional
systems to complex systems and systems of systems.
 The Working Group (WG) will be implemented in a “crawl-walk-run” process
beginning in 2013.
 The topic is very broad and cuts across almost all AIAA Technical Committees
(TCs), WGs, and Program Committees (PCs).
o Integration from a system design aspect
o It will deal with growing simple systems to systems of systems
 Publications and conferences will be the main products
o Possibly a joint journal with the Systems Engineering Technical
Committee
o Working on an Aerospace Systems Integration (ASI) Guide.
 Significant details on the WG’s charter, activities, relationship with other
committees, etc., are presented in the briefing.
 Audience comments/questions:
o Why is SE increasing in importance? How does SE verification differ from
testing?
 Look at risks specifically associated with system integration.
o System integration tasks are universally applicable, regardless of scale.
 Scale only makes it more complicated.
 Key Take-Aways:
o Systems Integration is a foundational element of SE
o This WG should help to link committees that have integration within their
charter.
o Formational activities are ongoing. Looking for members and contributors.
 Contact information is given in the briefing.
Donald Firesmith, Senior Tecfhnical Staff, Software Engineeroing Institute,
Carnegie Mellon University: Common Testing Problems: Pitfalls to Prevent and
Mitigate
Theme: The briefing attempts to capture and organize a high-level summary of
commonly-occurring testing problems, make general recommendations as to how to
avoid/mitigate these problems, and provide the information needed to generate testing
oversight and assessment checklists.
 Problems are characterized in the following categories:
o General testing problems
o Test planning problems
o Modern life-cycle problems
o Requirements and testing problems
22


o Unit testing problems
o Integration testing problems
o Specialty engineering testing
o System testing problems
o Systems-of-systems testing problems
o Regression testing problems
o Maintenance testing problems
An extensive list of recommendations is provided in the presentation slides.
Key Take-Aways:
o Root causes of most problems relate to :
 Lack of adequate planning, preparations, or definitions
 Resource limitations
 Turf / rice bowl disputes
 Missing information from poorly-thought-out programs
Panel Session
Maureen Molz, Manager, Concept Development and Validation Branch, FAA:
System of System Verification & Validation for NextGen
Theme: NextGen represents a fundamental paradigm shift from stove-piped, analog,
point-to-point systems to an integrated, digital, net-centric architecture. This briefing
seeks to describe a cost-effective approach to testing the highly complex NAS system
that is constantly in use and can’t be turned off.
 Traditionally, 70% of faults in a complex system are introduced in the design
phases (downward leg of the SE “V” where NextGen is decomposed) and only
3.5 percent are found and fixed.
o The remainder are dealt with after recomposition at significantly higher
cost.
 Key Take-Aways:
o There is value in using simulation to test NextGen.
 Using live, virtual and constructive models.
 Low-, medium-, and high-fidelity
o Test entire NAS as a system-of-systems
 Influence research, development and test
o Evaluate in realistic/intended environment
Kevin Knudsen, Boeing Test & Evaluation: System-of-Systems and Live, Virtual,
Constructive Test Capability - Global Capabilities
Theme: This presentation describes Boeing’s application of LVC test methodology early
in system design to mitigate problems that occur downstream.
23




Historically Boeing has stovepiped SE, T&E, and M&S
o Realized that these must come together to facilitate early V&V
Boeing has initiated a gated approach for its V&V enterprise to identify, examine,
and mitigate risks early and throughout the process.
o Cross functionally ensure that the concepts, requirements, architectures,
designs, and operations are affordable, feasible, valid, producible, and
testable across the life cycle
o Perform verification and validation as soon as possible to identify,
mitigate, and retire program risks early
Boeing has created a global LVC-test infrastructure for LVC experimentation and
test methodology.
o Boeing sites worldwide are on LABNET.
o A single T&E organization supports military and commercial programs.
Key Take-Aways:
o Industry must revisit the approach to V&V of complex systems/systems of
systems.
o Early and persistent, cross-functional engagement across a SOS life cycle
is required.
o Leveraging opportunities for injecting downstream issues of a prototype
system into a controlled, complex SOS Test environment earlier in the life
cycle helps identify and mitigate program risks throughout the life cycle.
Misty Davies, Research Computer Engineer, Intelligent Systems Division, NASA
Ames Research Center: Assurance of Software-Intensive Flight Critical Systems –
A Plan for Enabling V&V in NextGen
Theme: This presentation discusses the plan for applying V&V in NextGen.
 80% of a pilot’s functionality is software-based.
o 75% of the cost of a new aircraft is software.
o Traditionally we underestimate that cost.
 For software-intensive systems…
o Lines of code have doubled almost every four years since 1968.
o Software defects increased 15% from ’97-’98 to 2000.
o The cost to fix software deficiencies in the maintenance stage is about 110
times what it costs in the requirements stage.
 64% of deficiencies are introduced in the requirements or design
stage, but few are corrected there.
o We have no good way to look at distributed systems that are software
intensive.
 Key Take-Away:
24
o The key to maintaining safety in NextGen while reducing the cost of V&V
is to push V&V earlier in the lifecycle and use advanced complex system
V&V tools and processes.
Eric Neiderman, Human Factors Branch Manager, FAA: Human Factors of
Integration, Test, and Verification of Complex Systems
Theme: This presentation explored the issue of ensuring that human factors issues are
identified and resolved in complex systems.
 Human factors considerations are usability, trainability, and maintainability.
 Desirable to test human factors in unusual, off-nominal events.
o Understand the role and function of the person in rare, abnormal, unusual,
emerging situations.
 Human factors issues can best be understood through the use of…
o Tools
 Measurement, cognitive agents, and simulation capabilities
o Opportunities
 Human factors long-term R&D focus areas
 System resiliency
 Complexity
 Information
 Human-systems integration
 Safety
o Processes
 Evaluate the system in less than ideal situations and
circumstances.
o People
 Involve all disciplines
o Strategy
 Develop both short and long-term strategies in all areas of tools,
opportunities, processes, and people.
 Audience interaction:
o CASE has been used during the week as a place to rant about decisionmakers who don’t want to hear about SoS LVC testing in distributed
networks.
 Institute a mechanism for constructive action.
o Went for International Organization for Standardization (ISO) certification
on NextGen test procedures.
o How do you V&V systems early enough to make changes soon enough?
 Use a business case to convince decision-makers that there is
value in testing earlier.
25

 Need data to make business case. Must document better.
o Organizational structures are based on old models. Not agile enough.
Key Take-Away:
o Human factors can be systematically identified and resolved in complex
systems.
Group discussion in closing T2S3:
Statements below came from both panelists and members of the audience. No personal
attribution was consistently recorded.
 There has been a lot of complaining that decision-makers don’t want to consider
the use of SOS testing in a distributed environment.
o Perhaps CASE will provide a forum for more constructive action.
 We need to collect the data to document benefits of early V&V when we do it.
o Make the business case.
 For agile systems, the technical problems can be solved. Organizational
structures, processes, policies, etc., need to be changed.
o E.g., eliminate functional stove-piping
o Our organizational structures are antiquated.
o Not agile enough.
References (standards, best practices, guidebooks, etc.) related to V&V or I&T as
referenced in the introduction to this section.
 AIAA 117 Space Systems V&V
 1012-2012, IEEE Standard for System and Software Verification and Validation;
2012: http://standards.ieee.org/findstds/standard/1012-2012.html
 NASA IV&V Policy NPG 8730 DRAFT
 FAA AMS Policy for
V&V: http://fasteditapp.faa.gov/ams/do_action?do_action=LinkSection&contentU
ID=4&sectionNumber=2.1.7
 FAA V&V
Toolkit: https://employees.faa.gov/org/linebusiness/ato/operations/technical_ope
rations/best_practices/discipline/evaluation/
 FAA V&V Guidelines, October
2011: http://fast.faa.gov/docs/vandvguidelines.doc
 FAA T&E Process Guidelines, March
2012: http://fast.faa.gov/docs/teguidelines.doc
 FAA T&E Handbook, August
2010: http://www.faa.gov/about/office_org/headquarters_offices/ang/offices/tc/ini
tiatives/vnv/documents/publications/VVSPT-A2-PDD013_TnE_Handbook_v2.0.pdf
26



AIR FORCE INSTRUCTION 99-103, CAPABILITIES-BASED TEST AND
EVALUATION
DAU’s TEST AND EVALUATION MANAGEMENT GUIDE, JANUARY 2005
Documents that contain V&V/I&T ‘best practices’ information
o ISO 15288
o ISO/IEEE 12207
o NASA NPG 2820
o NASA Systems Engineering Handbook (6105)
o CMMI
o Mil Std 499B
o DAU Systems Engineering Fundamentals Guidebook
o DAU Defense Acquisition Guide
27
Day 3
CASE Track 2, Session 4: Lessons Learned in Integration, Test, and Verification
This session was chaired by David Dress, NASA Langley Research Center (Moderator)
and Steve Chan, NASA Johnson Research Center. There were five panelists, identified
below with their individual presentations.
Marshall Smith, Mission/Flight Test Strategy Lead for Exploration Systems
Development Division, NASA Headquarters: Ares I-X Lessons Learned Integration, Test and Verification
Theme: ARES I-X was a fast-paced program to provide early source data for NASA’s
ARES I vehicle, using resources from across the agency, to discover integration,
assembly, and test issues for ARES; provide a realistic simulation of the Ares
environments and flight control strategy; and provide flight validation of integrated aero,
structures, and performance models.
 Lessons-learned were discussed in four separate theme categories:
o Mission management
 Concentrated leadership/authority
 Schedule is king
 Minimize levels and boards – high decision velocity
 Make decisions quickly and live with them
 Documents and process are not the focus
o Test mindset – Enforce “good-enough” rationale
 Question standards, policies, procedures
 Developed simple alternatives (e.g., modal tests)
o Systems Engineering and Integration (SE&I)
 Objective and goal focused - question requirements and standards
 Small competent teams composed of elements and led by SE&I
 Tiger teams formed and dissolved as necessary
o Heritage hardware and facilities
 Don’t assume heritage will work with less effort
 Atlas Systems Integration Laboratory worked well for evaluating
heritage hardware in new environments.
 Key Take-Aways included:
o Make decisions when you have sufficient data; don’t wait for all of the data
o Strive for cheaper, faster, good enough.
o Make requirements subordinate to goals.
o Heritage doesn’t always work; it is a place to start.
28
Frank Rasor, Director, Flight Test Operations, Boeing (787 & 747): Product
Validation and Recent Flight Test Experiences
Theme: Lessons-learned were presented from two ongoing Boeing programs: 787 and
747-8. Each flight test program was expected to take 10 months. Each one took 21
months to complete, causing expensive retrofits to airplanes already on the ramp. This
presentation explored some of the reasons why.
 In static tests of the 787 structure, the wings which flexed 25 feet at the tips initially
showed no stress at the wing-body juncture. This was missed in instrumenting the
test article.
 In the 747-8, some of the same oscillations that were seen as noise and accepted in
older versions had to be eliminated because of new regulations.
o Landing gear doors were causing turbulence that created oscillation.
 Heavier aileron actuators had a buzz
 New materials were required to take gross weight to 1,005,000 lb.
o If a materials allowable test was missed, they pressed ahead
o They paid a price for this later in flight testing
 Key Take-Aways:
o Finish component tests before flight testing
o Program managers sometimes make bad decisions that cause problems for
subsequent PMs.
o More ground testing would have resulted in less flight testing.
o Referring to the Verification and Validation “V,” the way you arrive at and
manage the low point of the “V” influences how efficiently you move up the
right-hand-side with test.
John Muratore, Director, Mission Assurance, Space X (Falcon 9 and Dragon):
System Engineering: A Traditional Discipline in a Non-traditional Organization
Theme: Space X is a small company of 1800 people, most located in Hawthorne, CA
under one roof. They are developing several versions of Falcon launch vehicles and the
Dragon spacecraft. This presentation described the unique approach to SE in Falcon
V&V.
 Space X uses “iterative design.”
o Maintain continuous design heritage in development with rapid spiral
methodology.
o Huge development risk on the way down the “V,” rigorous SE on the way
back up.
o Rapid design-build-test cycles to inform design by experience.
o “Test like you fly, test what you fly.”
 Key Take-Aways:
o Space X’s innovative success is facilitated by its small size and relatively
small cost of systems.
o Rapid spiral development depends on applying lessons-learned.
 Immediate redesign when tests reveal problems enable program to
advance rapidly.
29
o People are better at solving problems than predicting them.
 Up front SE is balanced with learning through rapid spiral
development.
o The process depends upon rapid, constant communications among team
members; e-mails are answered immediately.
o Designing to satisfy the needs of multiple customers.
 Flexible design parameters to satisfy multiple mission needs.
o Maintain total control over what is built in-house.
 Multi-element optimization down to the last part.
o Space X uses a traditional test hierarchy, including in-space testing.
 Facilities were low-cost and were built in a very short time.
o It is difficult to build a creative high performance engineering culture.
o Too much organization, and too many rules and processes, can easily ruin
creativity and performance.
o SpaceX is achieving a good balance of creativity and systems engineering
for agility and affordability.
Brandon Rhatigan, Director, Engineering & Test Operations, F-35 Integration test
Force, Lockheed Martin: F-35 Lessons Learned in Integration and Test
Theme: The F-35 has 24.3 million lines of code (LOC) compared to other aircraft
averaging 9.4 million. This leads to increased complexity in integration and support.
This paper presented some of the lessons learned in testing the F-35.
 Test robustly, early and often.
o Data are presented for multiple programs that show increased lower-cost
ground testing early actually reduced higher-cost flight testing later.
 Employed “Fusion Testing”
o F-35 has an overwhelming amount of sensor data; too much for the pilot
to process individually
o F-35 invested heavily in labs and test beds.
o Fusion combines sensor data into usable information for the pilot
o Fusion testing means testing systems of sensors rather than single
sensors individually, but enabling the test engineer to look at individual
sensor performance as needed.
 Focus is on whether you can complete the mission, not on the
performance of a single sensor.
 Key Take-Away:
o Reliance on early ground testing and Fusion enabled the F-35 to
experience much better stability and maturity than legacy systems.
David Gilman, Senior Technical Advisor, Analytical Mechanics Associates, Inc.:
Lessons Learned - Inflatable Reentry Vehicle Experiment – 3
Theme: NASA has led the design of the Inflatable Reentry Vehicle and used a threestage sounding rocket from NASA’s Wallops Flight Facility to attain flight relevant
heating on the vehicle. This presentation discussed lessons learned.
30



The program encountered concerns over people becoming overstressed while
numerous steps were taken (presented in the briefing charts) to relieve stress on
the test team.
Some of the paths taken were not traditional (i.e., spending $250,000 on using
one facility when another facility would have been cheaper, in order to prevent
overloading the team).
Key Take-Aways:
o It is possible to keep people in high-stress situations if there is “light at the
end of the tunnel.”
o If there is no immediate end in sight, they tend to want to pull back on
work hours.
o It is prudent to give a day off occasionally to allow team members to
“recharge”.
Group discussion in closing T2S4:
Statements below came from both panelists and members of the audience. No personal
attribution was recorded.
 Is test-fix-test better than model-test-model?
o Space X uses the test-fix-test approach, constantly recalling boxes and
making fixes.
 Many PMs see early test failures (discoveries of system problems) as program
failures
o (Editorial note: Finding problems in a system being tested is not a test
failure, although that is the way it is often described and viewed by PMs
and external enablers.)
o We have gotten away from using testing to provide learning opportunities.
 We have become more conservative and avoid the risk of testing
in highly visible environments.
 Trading technical risk for political risk.
 Shutting down qualification facilities for black boxes is false economy.
o Maintain the capability to learn from flights, modify the box, re-qualify it, fly
it.
 “Good processes don’t ensure good performance, but bad processes can bring a
program to its knees.”
o Having the right people is more important than having the right processes.
 Use tiger teams to solve specific problems and disband them.
o Common leadership to stress problem-solving.
o Use good people, meet schedule, document results, disband.
 Regarding stressed team members:
o A stress-performance curve was mentioned. As you pass a point, things
begin to break down easily.
o Average age at Space X is 28; work week is 50 hours.
 Space X spends a lot of time documenting what pass and fail conditions are on a
test.
31

o “Nobody on the company succeeds if we are waiting on the ground to
solve an intractable problem.”
Getting testers involved early is the best way to go.
o Especially important when program starts are separated by long periods of
time.
o Both Boeing and L-M are trying to institutionalize this.
o Lead tester must be someone who is experienced in the entire
development process on another system.
 Difficult to find when program starts are infrequent.
32
Day 3 (Continued)
CASE Closing Session
The closing session was moderated by Laura McGill, Deputy Vice President of
Engineering, Raytheon Missile Systems, AIAA Vice President, Standards, General
Chair (CASE). Track speakers were: T1, Paul Collopy, University of Alabama,
Huntsville; T2, Allen Arrington, Sierra Lobo, NASA Glenn Research Center; T3, Sophia
Bright, Boeing. Note that track speakers’ comments as presented below include their
responses to other track speakers’ reports and are not presented in chronological order
of the session.
 There will be a website set up one month following this conference with briefing
notes, comment cards, and a high-level wrap-up white paper.
Track 1 Report – Paul Collopy, University of Alabama, Huntsville:
 Three kinds of complexity
o Interdependencies within the system
 Integration is messy; all pieces interact with all other pieces
 Decomposition is not possible
o Very large numbers of states and transitions
 NextGen is an example
 New validation approach needed
o Very large amount of design information
 Trying to fix every small problem that arises is a problem
 Can do “build and break” without decomposing system
 The form of the organization constrains development of systems
o This came up every day, in 50% of talks
o Structure matters
o The procedure for making design decisions affects the design
o Too formal a system of communications inhibits design
o The people side of a Work Breakdown Structure is hugely influential
 Ambiguity is essential in developing complex systems
o Good chief engineers are people who can work with ambiguity
o Reliance on computers has caused loss of judgment and understanding of
how systems work
o Comment from audience: Need both precision and ambiguity
o Comment from audience: Need the right questions to enable the right
answers (example: Why X significant digits?)
 T1 Key Takeaways:
o Learned how to be a better designer
33
o
o
o
o
o
o
o
 How to keep a system from becoming more complex than it needs
to be
 Bad architecture drives this
 Too many requirements drive us away from design elegance.
Look at test culture as a way to learn rather than a way to find someone to
blame.
Make designs more durable.
Define the “goodness” of a complex system.
 It should not be simply meeting requirements.
Recycling, re-using, or multi-functioning things do not always save cost
 Can increase complexity and drive cost up
We can’t possibly develop NextGen the way we have been taught to
develop systems
Cost overruns and delays are symptoms of over-complexity.
Audience comment : Complexity must buy its way into a platform.
Track 2 Report – Allen Arrington, Sierra Lobo, NASA Glenn Research Center:
(NOTE: Arrington’s comments are from the perspective of the test and verification
community)
 Engage the test community earlier, BUT…only if the test community understands
what’s going on in the design process
 We have tools. Are we using them properly?
 Are we training the next generation of engineers?
o We have an aging workforce; are we bridging the transition?
 Boeing initiates a new project every two-to-three years to keep trained
people in the pipeline.
o Audience comment: Do we always apply the logical progression of analysis,
ground test, flight test?
 “Audacious” comment: Will advances in computing allow skipping one
part – maybe ground test – in the future?
 Could AIAA construct a roadmap to get us to that point?
o The Track 2 Verification and Validation session focused on large numbers of
lines of code and simulation.
 Must ensure early test involvement for effective V&V.
 T2 Key Takeaways:
o Many good ideas, some require cultural transformation.
o Starting Monday:
 Better communication
 Attention to requirements fidelity and stability
 More up-front attention to analysis of alternatives
34
o Perhaps the AIAA Standards committee could take on the challenge of
standardizing definitions and nomenclature for these tracks.
Track 3 Report – Sophia Bright, Boeing:
 Situational awareness is very important for the systems engineer
o Full capabilities of a system emerge over time.
o Many questions need to be asked early in order to better handle complex
systems.
o Engineers must understand supportability and maintainability.
 Differentiate systemic issues from systematic issues.
o Incorporate into how we train our engineers.
 Communication is important.
o Engineers don’t do it well.
 Tools: Mental model; pictorial.
 Look for better/more effective ways.
 Listening is important for program managers, engineers, etc.
o Be able to know when and how to say no.
 Audience comment: Innovation can cause this to be an issue.
 Overall, acquisition issues are very complex.
o Issues can make our lives harder.

T3 Key Takeaways:
o We make systems more complex than they need to be.
 Focus on good architecture.
o Taxonomy – make sure we define terms before moving forward
 Better define complex vs. complicated.
 What is uncertainty?
o Training is important
o Need to develop a path forward for commercial use of space and air traffic
control
o Develop a process for transferring technology/knowledge throughout a
program, from design through operations
 Production needs to understand design
o There is a changing paradigm of costing sustainment
 What is real vs. perceived value?
 Seems to be left out of costing models
The Next CASE Event – Laura McGill moderated audience input:
 No more than three tracks
 Complexity is bigger than aerospace
35


o Bring in other disciplines
o Learn from their experiences
Facilitate more intermingling of tracks
o Encourage cross-attendance of PMs, testers, designers, budgeteers, etc.
o Step back and look at different ways to do things
 Cross-track
Budget is important
o Relate to complexity
o Keep at high-level
Closing Remarks – Michael Griffin, Chairman and CEO, Schafer Corporation,
AIAA President, Executive Chair (CASE):
 The systems engineering (SE) community should not forget:
o Good SE is about creating an elegant design
 Elegance consists of a design that
 Has efficacy (works);
 Is robust;
 Is efficient;
 Is as free as possible of unintended consequences.
o SE is about sociological considerations, but not only these
o SE is the link between analysis and design
 Schools teach engineering science
 How to analyze someone else’s design
 This is half the story
 SE includes creating that man-made world
 Designers and analysts don’t think the same way
 Analysts pick apart designs and find flaws
 Engineers link designers and analysts
o Look for “intelligent decomposition” of systems
 Reduce unintended consequences
o Engineers exist to turn ambiguity into knowledge
 Good engineers have a tolerance for ambiguity
 Can stand it longer than most
 Don’t turn ambiguity into soup too soon!
o Human cognition must be factored in
 SE is a team sport
 We have learned a lot about how humans decide things and the
effects of consequences on our cognition
o Confirmation biases: We look for information that supports our conclusions
and ignore other information.
36
o Distinguish between precision and correctness in the SE process.
o Sometimes we must “rise above principle.”
 Must we always be precise?
 It is better to know some things to four significant digits than
none to six.
Supplemental Information:
Ron Kohl, Moderator, Track2, Session 3: Observations, concerns and
suggestions resulting from CASE 2012
Although not reported at the closing session, Ron Kohl offered the following for
the report:
 Having the mid-morning and mid-afternoon breaks in a different building than the
technical sessions seemed to reduce the size of audience in the post-break
sessions.
 We may need to consider even more ‘non-conference, anti-conference’
approaches, methods, ‘tools’, take-home products. I think that our final product
was still more similar to, than different from, the standard conference model.
 More ‘practitioner’ oriented, ‘how to do it and why do it’ kind of info seems
warranted. We had some excellent things to worry about’ presentations, but
adding some ‘why bother’ and ‘consequences if you don’t’ info seems helpful in
order to allow practitioners to convince their management and leadership to
adopt some new or innovative approach/method.
 The name of our Conference, CASE, did not clearly communicate the nature of
the content. I think that we might want to consider more emphasis of the PM
and SE content of CASE in the name/title of the conference
 Complexity surfaced many times, in various contexts, and it seems that maybe
having a session (or more) about Complexity, how to recognize it, and what to do
about it, would be helpful. Paul C mentioned this item during the closing session.
 I think that separating our desired audience into PM, Chief Engineers and then
practitioners is perhaps too fine of a separation. I think that if we had two major
categories, namely PM and technical, then we could/would address all 3 groups
without needing to separate the Chief Engineer group from the other
‘practitioners’.
 Off-nominal behavior, rainy day scenarios and other fault-handling issues
surfaced many times and we may want to consider a session (or more) focused
on “What should my system do when bad things happen?” and how addressing
this question should surface in different parts of the lifecycle.
 Doing more V&V in the early part of the lifecycle (left side of the SE ‘V’). We may
want to have a session (or more) focused on just what could/should the V&V/I&T
37

community contribute to the left side of the ‘V’ activities. Some candidates could
be ‘supporting or forcing requirements reviews from a tester’s perspective’.
o During the closing session, someone (maybe Misty?) mentioned that we
should have more overlap, intermingling and/or cross-pollinating between
the development and the test tracks. Maybe having a half-day session (or
more) where the first half of the half-day is the devoted to the development
perspective of some topic/issue and the second half of the half-day
addresses the test perspective. For example (see 8 above), we could
have a session about ‘what developers would like to get from testing in the
early part of the lifecycle followed by ‘what the test community thinks it
could contribute to the early part of the lifecycle. And perhaps a grand
finale of ‘here are some V&V/test activities that could be done early/earlier
in the lifecycle (e.g. left side of the ‘V’).
The need to ‘justify’ adopting a new method. So maybe some more focus on the
business case (aka cost/benefit tradeoff, pros and cons, etc.) of why
management/leadership should consider adopting a new
method/technique/process would be helpful.
38
Recurring Topics:
Several topics, themes, or issues arose repeatedly in the presentations, often cutting
across sessions. An attempt has been made here to capture the most prevalent of
those.
 Definitions, taxonomy, and forums for interactions among disciplines are needed
in SE and V&V of complex systems and systems of systems.
o AIAA can play a key role here.
– ASI-WG
– CASE as a forum for constructive action
– Could AIAA Standards committee take on the challenge of
standardizing definitions and nomenclature for these tracks?
 Policies, processes, procedures, and organizational structures often inhibit
successful implementation of complex systems V&V.
o Experience trumps processes and procedures.
o Sound SE methodologies in SoS V&V mitigate risk.
 SE practices and methodologies are becoming more important in development,
verification and validation of systems as we move toward more increased
complexity in systems and systems of systems.
o Many test decision-makers are reluctant to test early and to support SoS
LVC testing in distributed networks.
 Integration of M&S into live T&E is imperative for SoS V&V.
o SoS complexities often preclude live testing because of cost, scheduling,
OITL, asset availability, or other issues.
 We need to maintain specialization as well as integration engineers.
 Most problems:
o Occur at the interfaces; most innovation occurs there as well.
o Are caused by lack of adequate planning, preparations, or definitions.
 Other root causes are resource limitations; turf / rice bowl disputes;
missing information from poorly-thought-out programs.
 Beware of requirements!
o They should be subordinate to goals or capabilities.
o Avoid requirements creep.
o Question them when they come as solutions.
o Not all great ideas are requirements.
o All levels of requirements must be linked and verifiable along the way.
 Integrate T&E into design process early
o Identify and fix problems early rather than late in a program.
o Late problem resolution is very expensive.
o More early GT results in less FT.
39


o Test-fix-test works; people are better at solving problems than predicting
them.
Efficiencies in live testing that result from integration of M&S or other innovations
place a strain on facility sustainment resources and user unit costs.
Communicate, communicate, communicate!
o Use effective pictures or mental models.
o Respond quickly to communications from other teams and disciplines;
they may be waiting for your wisdom.
40