Challenges of System of Systems
Testing In a Contemporary
Operational Environment
Test Technology Directorate (TTD)
US Army Operational Test Command
1
Purpose
Provide an overview of the
challenges to properly
operationally test systems of
systems given the contemporary
operational environment primarily
from a technology viewpoint.
US Army Operational Test Command
2
Definitions
Operational test (OT) Any testing conducted with the production or
production like system in realistic operational environments, with users
that are representative of those expected to operate, maintain, and
support the system
System of systems is a collection of task-oriented or dedicated
systems that pool their resources and capabilities together to obtain a
new, more complex, 'meta-system' which offers more functionality and
performance than simply the sum of the constituent systems.
US Army Operational Test Command
3
Definitions
• Operational environment is a composite of the conditions,
circumstances, and influences that affect the employment of military
forces and bear on the decisions of the unit commander.
• Contemporary operational environment (COE) is the overall
operational environment that exists today and in the near future (out
to the year 2020). The range of threats extends from smaller, lowertechnology opponents using more adaptive, asymmetric methods to
larger, modernized forces able to engage deployed U.S. forces in
more conventional, symmetrical ways. In some possible conflicts (or
in multiple, concurrent conflicts), a combination of these types of
threats could be especially problematic.
US Army Operational Test Command
Critical Variables
•
•
•
•
•
•
•
•
•
•
•
Physical environment
Nature and stability of the state
Military capabilities
Technology
Information
External organizations
Sociological demographics
Regional and global relationships
National will
Time
Economics
US Army Operational Test Command
The Challenge:
Replicating a Complex Environment
Airborne C2
ISR assets
JFACC/JAOC/
TACC
FAC(A)
aircraft
WOC
XXX
JFLCC
(Corps/MEF HQ)
ASOC/DASC
XX
JFMCC/
AREC/
SACC
CRC/TAOC
JCAS
aircraft
Senior
FSE/FSCC
TACP/AO
TOC/COC
FSE/FSCC
III/X
TOC/COC II
FSE/FSCC
JSOTF
Immediate Request for CAS
Process Through Execution
TACP/AO
JTAC
Friendly forces
Hostile
targets
US Army Operational Test Command
JFO/Observers
OTC’s Technology Challenge
Given:
• Rapidly changing technology
• System-of-systems more complex
• Size and scope of tests is increasing
• Resource constraints – Units, Space, Time, Funds
OTC’s Technology Challenge
Providing robust operational environments in which
to test the network-centric system-of-systems and
collecting the information needed for evaluation.
OTC’s Response:
• Technology – Strategy to build on what works to prepare for the future
• Expertise – Depth of experience and expertise is nurtured and shared
• Processes – Accredit sim/stim; certify instrumentation; validate data
management, accredit threat
US Army Operational Test Command
Use of Simulations
• Stimulate the player unit to cause actions to be
performed.
• Simulate a capability or elements that cannot be
provided due to resource constraints, safety restrictions,
or other limitations.
• Generate test designs and provide rehearsal and
modification capabilities, provide scenario or event list
generation, and support test control.
Source: TOPM 73-182
US Army Operational Test Command
8
Operational Testing:
Providing a Wrap Around
Airborne C2
Live
Virtual
Constructive
JFACC/JAOC/
TACC
ISR assets
FAC(A)
aircraft
WOC
XXX
JFLCC
(Corps/MEF HQ)
ASOC/DASC
XX
JFMCC/
AREC/
SACC
CRC/TAOC
JCAS
aircraft
Senior
FSE/FSCC
TACP/AO
TOC/COC
FSE/FSCC
III/X
TOC/COC II
FSE/FSCC
JSOTF
Immediate Request for CAS
Process Through Execution
TACP/AO
JTAC
Friendly forces
Hostile
targets
US Army Operational Test Command
JFO/Observers
Instrumentation Definition
• Instrumentation is the use of devices to measure properties,
monitor activity, collect data, store data, organize data, and
control actions
• Data requirements dictate instrumentation requirements
• Evaluator requirements dictate data requirements
US Army Operational Test Command
10
Instrumentation Issues
• Test platform configurations rarely consistent
• Multiple trips required to complete and test instrumentation
• Test platforms continue to evolve throughout the
development period
• Certification and VV testing requires planning and support
similar to record test
• Late system access and delayed system readiness delays
certification testing and the certification request
• Instrumentation use increases test cost
US Army Operational Test Command
11
Instrumentations Challenges
Instrumentation Intrusiveness and Ruggedness
US Army Operational Test Command
Questions?
US Army Operational Test Command
13
Key Definitions
• Model – A physical, mathematical, or otherwise logical
representation of a system, entity, phenomenon, or process.
• Simulation – A method for implementing a model over time.
• Stimulation – Providing the System Under Test with a
representation (quantity, quality, and necessary content) of input
forces, conditions, or data at the times required, to produce a
system response.
• Federation of Models & Simulations – A system of interacting M&S
with supporting infrastructure, based on a common understanding
of the objects portrayed in the system.
• Accreditation – The official certification that a model, simulation, or
federation of M&S and its associated data is acceptable (both
credible and suitable) for use for a specific purpose.
• Verification – The process of determining that a model
implementation and its associated data accurately represents the
developer’s conceptual description and specifications.
Source: TOPM 73-182
US Army Operational Test Command
14