System Test and Evaluation
Test and Evaluation (T&E) Throughout the Life Cycle
What is T&E and Why is it Needed?
T&E Throughout the Life Cycle
Learning Objective
 Describe a T&E process that
reduces risk by assessing if
the system:
o Meets requirements.
o Performs as intended in its
operational environment.
2
T&E Process – What, Why, and When
What is T&E?
Why is it T&E Activities
by
needed?
Concept
Life Cycle
Development
Phase
Phase
Engineering
Development
Phase
Post
Development
Phase
3
What is System Test and Evaluation?
A process to reduce risk by
assessing a system.
 Determines if the system meets
requirements and performs as intended
in its operational environment.
The process begins early in the
life cycle.
 Evaluates the feasibility of early
design approaches.
 Compares actual and expected system
performance.
 Creates a test approach to define
test activities implemented in each
life cycle phase.
 Evaluates operational effectiveness,
suitability, survivability, vulnerability, and
lethality of the system and its elements.
 Assures test activities generate
unbiased test results to share with
developers, users, and stakeholders.
4
Operational Effectiveness and Suitability
 Operational Effectiveness
o How well system meets the mission need when exercised in its
operational environment by the user (in a realistic scenario).
 Operational Suitability
o How well the system performs as intended in its environment.
5
What is Verification and Validation (V&V)?
Verification
Is the system built right?
system m eets requirem ents
Validation
Is the right system built?
system perform s as intended
 Proves compliance of a requirement by a
Method of Verification (Inspection,
Analysis, Demonstration, Test).
 Confirm the system meets the need or
intended purpose in the operational
environment (relates to CONOPS).
 Base on approved specifications, drawings,
parts, and items that set a configuration
baseline of the system.
 Assess operational effectiveness and
suitability, test system under realistic
conditions in an operational environment.
 Perform throughout life cycle, mostly
during Developmental T&E (DT&E).
 Perform in each phase using phase
products such as models and simulations.
source: NASA SE Handbook, Product Verification and Validation (2007)
6
Why T&E for Complex Systems?
 Experience with the system and its elements can
add a twist to solving problems.
 May reveal a need to change a concept or design,
or a system not fully testable prior to delivery.
 The challenge for T&E is to balance cost, schedule,
and performance against risks as twists occur.
7
Why do Twists Occur?
 Ignorance of natural phenomena or lack of
experience with technology
 Poor system engineering and test planning
or test objectives
 Testing reveals flawed concepts
or operational weaknesses
 Evolving or poor requirements
In 1996, what caused a failure of the first Ariane 5 test
flight? The second in 1997?
Were there other failures before it was retired in July
2023? What happened and why?
Ariane 5 successful launches = 95.7% (112/117)
8
References
Department of Defense. (2012). DoD test and evaluation management guide
(6th edition). U.S. Department of Defense.
Federal Highways Administration, Office of Operations. (2023).
Designing for transportation management and operations: A primer
(Chapter 2.5). U.S. Department of Transportation.
Kossiakoff, A., Sweet, W., Seymour, S., & Biemer, S. (2011).
Systems engineering – Principles and practices (2nd edition, Ch. 7-13).
J. Wiley. (1st edition published 2003).
National Aeronautics and Space Administration. (2007).
NASA systems engineering handbook (Rev 2, NASA SP-2016-6105).
NASA Headquarters, U.S. Government.
9
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
Title
T&E Throughout the Life Cycle
What is T&E and Why is it Needed?
Filename
1.1 T&E Definitions and Purpose
Author
J.Ziarko
Version
July 2023, V1.00
System Test and Evaluation
Test and Evaluation (T&E) Throughout the Life Cycle
Life Cycle Phases
T&E by Life Cycle Phase
Concept
Development
Engineering
Development
Illustration adapted from Kossiakoff, A. & Sweet, W. N. Systems Engineering: Principles and Practices, Wiley, 2003.
Post
Development
2
System Life Cycle Activities by Phase
Functional
Specs
Operational
Effectiveness
Concept
Development
Technology
Opportunities
Engineering
Development
Defined System
Concept
Define mission need,
CONOPs, and requirements.
Identify feasible system
concepts, select and verify a
concept that meets need.
Production
Specs
Operation &
Maintenance
Post
Development
Installed
System
Production
System
Translate system concept into
a validated physical system.
Build and test system to meet
requirements and perform as
intended.
Produce and deploy a system
that performs as intended in
its operational environment,
and assess system in its
operational environment.
Adapted from Kossiakoff, A. & Sweet, W. N., Systems Engineering: Principles and Practices, Wiley, 2003.
3
Systems Engineering Method - Traceability
source: Kossiakoff, A. & Sweet, W. N., Systems Engineering: Principles and Practices, Wiley, 2003.
4
T&E Activities in each Life Cycle
Life Cycle Activities
Test Activities
CONCEPT DEVELOPMENT





Need, CONOPS
Requirements
System Decomposition, Analysis
Trade Studies
System Concept





Mission Analysis, Needs Assessment
Requirements Validation Testing
Performance Evaluation
Technology Tests
Proof of Concept Tests
ENGINEERING DEVELOPMENT


Informal Testing
Formal Testing


Integration & Test
Qualification Testing (Developmental T&E)
POST DEVELOPMENT

Production & Deployment,
O&M, System Disposal

Operational Test and Evaluation (OT&E)
Operational Assessments
5
CONCEPT DEVELOPMENT
POST DEVELOPMENT
SE Life Cycle V
TEST APPROACH
Systems Engineering
Domain
VERIFY CONCEPT
Engineering
Design Domain
VALIDATE CONCEPT
DETAILED TEST PLANS &
PROCEDURES
VERIFY UNITS &
COMPONENTS
ENGINEERING DEVELOPMENT
source: US DOT Federal Highways Administration, Designing for Transportation Management and Operations: A Primer, Ch. 2.5,. 2023.
T&E Process Definitions
Test
Evaluation
Test & Evaluation
 A deliberate action or
experiment to find out
how well a system
works.
 Analyze design reviews
and test results to
assess system.
 Compare system and its
elements to requirements
and specifications through
testing.
 Gather and verify data
to analyze how well
developmental and
performance objectives
have been achieved.
 Compare actual system
performance to
expected performance.
source: DoD Test and Evaluation Management Guide, December 2012, p.76.
 Assess operational
effectiveness, suitability,
survivability, vulnerability
and lethality of system.
 Validate system is
operationally effective and
suitable for intended use.
7
T&E Process Follows Scientific Method
NEED
Purpose
TEST OBJECTIVES
Data and
Procedures
TEST
ACTIVITIES
ACTION
Conclusion
source: DoD Test and Evaluation Management Guide, December 2012, p.43.
TEST RESULTS
Results
8
Five Steps to the T&E Process
Assess capability needed to gain insights
into system operational effectiveness
and suitability to meet the need.
1
Need
Balance test results with
program information to
plan course of action,
including more tests.
Assess
5
2
Action
Plan
Compare actual outcomes (3)
to expected (2) outcomes.
Document in test report.
Test
Plan
4
3
Test
Results
Test
Activity
source: DoD Test and Evaluation Management Guide, December 2012, p.43.
Describe test objectives,
expected results, and
data and analytical tools
needed to conduct tests.
Prepare and conduct
tests activities to gather
sufficient data to support
analysis.
9
Creating Test Reports
Generates
test results
immediately
following a
test event
Preliminary
Reports
Produces periodic
reports of results
from significant
test events for
awareness of test
program progress
Final test results
documented of
system functions,
operational
effectiveness and
suitability
Status
Reports
Final
Reports
source: Kossiakoff et al, Systems Engineering Principles and Practices, Wiley, 2011.
10
References
Department of Defense (2012). DoD Test and Evaluation Management Guide
(6th edition). Washington D.C.: U.S. Department of Defense.
Federal Highways Administration, Office of Operations (2023).
Designing for Transportation Management and Operations: A Primer
(Chapter 2.5). Washington D.C.: U.S. Department of Transportation.
Kossiakoff, Alexander, W. Sweet, S. Seymour, S. Biemer (2011).
Systems Engineering – Principles and Practices (2nd edition, Ch. 7-13).
New York: J. Wiley. (1st edition published 2003).
National Aeronautics and Space Administration (2007).
NASA Systems Engineering Handbook (Rev 2, NASA SP-2016-6105).
Washington D.C.: NASA Headquarters, U.S. Government.
11
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
TITLE
1 T&E Throughout the Life Cycle
Life Cycle Phases
FILENAME
1.B Life Cycle Phases
AUTHOR
J.Ziarko
VERSION
July 2023, V1.00
System Test and Evaluation
Test and Evaluation (T&E) Throughout the Life Cycle
Concept Development Phase
Concept Development
Concept
Development
Engineering
Development
Post
Development
2
T&E During Concept Development
Create
Analyze, validate
Test Approach
Requirements
early in life cycle early in life cycle
 Describe approach
to test system,
subsystems, and
components.
 Often called the
Test and Evaluation
Master Plan
(TEMP).
Create Verification Prepare and
Cross Reference
conduct
Matrix (VCRM)
Proof of Concept
• Analyze before
hand-off to
development teams.
 Assign one Method
of Verification to
each requirement.
• Good requirements
are testable!
 Trace Test Cases to
one or more
requirements.
 Explore and define
concepts.
 Select and verify
system conceptual
design.
3
source: US DOT Federal Highways Administration. (2023). Designing for Transportation Management and Operations: A Primer. Ch. 2.5.
Test Planning
Parallels with
System Development
Need
Define capability to field
System Concept
Trade-offs in cost, schedule and
performance to develop a concept
Functional Design
Translate functional requirements
into system and subsystem functions
Detailed Design
Design subsystem components
source - Kossiiakoff et al. (2011). Systems engineering – Principles and practices. Ch. 13
5
The Test Approach
Concept
Development
Engineering
Development
System requirements are
analyzed and baselined.
Integration and Test - Informal Testing
Conducted by contractor, requirements
verified by Demonstration or Test.
System is decomposed.
Conceptual Design is
verified.
Baselined requirements handed-off.
Qualification Testing - Formal Testing
Conducted by contractor with customer,
verifies all requirements.
Post
Development
A fully assembled
system is tested in its
environment.
Operational T&E is
conducted.
The Test Approach documents all test activities throughout the life cycle.
It is used to create detailed Test Plans and Procedures in each phase.
6
Requirements Analysis
1
A key Systems
Engineering process
2
Early in the life cycle,
include test engineers
 The purpose is to transform stakeholder
needs into system requirements during
Concept Definition.
 Test engineers need to be included as
essential members of the system
requirements analysis team.
 System requirements form a baseline to
assess systems engineering activities
throughout each life cycle phase.
 Well written, testable requirements
reduce rework later in the life cycle when
handed-off to integration and test.
 System requirements are verified during
integration and testing.
 Good requirements are correct, verifiable,
feasible, complete, necessary,
unambiguous, singular, conforming, and
at the right system level.
7
The VCRM
 Create a Verification Cross Reference Matrix (VCRM)
o Documents system and subsystem level requirements.
o Supports analysis and assures traceability of requirements to test cases.
 In the VCRM
o Assign each requirement a unique ID.
o Assign ONE appropriate Method of Verification to each requirement.
o Assign test cases to cover a requirement or group of requirements.
o List the system “build” where the requirement is tested and verified.
Req
ID
SRS
Para
No.
Requirement Text
Verification
Method
Verification Level
Applicable
build
I&T
Test
Case ID
Qual
Test
Case ID
8
What are the Methods of Verification?
Inspection (I)
Visual examination of a
realized end product to
verify physical design
features or manufacturer
identification.
Demonstration (D)
Shows end product
performance capability
(requirements without data).
Analysis (A)
1
2
System Static
System is NOT Operating
System Dynamic
System is Operating
3
source: NASA Systems Engineering Handbook (2007, p.86)
4
Use when a prototype,
engineering model, or
fabricated, assembled,
integrated product is not
available.
Test (T)
Use end product to obtain
data via further analysis
(requirements with data).
9
NASA’s Search and Rescue (SAR) System
Proof of Concept
 Conducted during concept exploration
o To verify and validate feasibility of potential
concepts (alternative solutions) to meet the
need and winnow out alternatives.
 An Analysis of Alternatives is
conducted to select a System Concept.
o Conduct trade-off studies to select the
preferred system concept.
source: Johns Hopkins APL Technical Digest, May 2020
10
Example - Analysis of Alternatives (AoA)
What system or a mix of systems
provides the nation the best balance
between effectiveness and affordability
addressing NEO survey gaps?
What balance of space-based sensors
would best augment the existing and
planned ground-based capabilities and
provide the overall best solution for the
survey challenges?
source: Johns Hopkins Applied Physics Lab, AoA for Near Earth Objects, 2018
11
System Concept Implementation
Exit
From
Concept
Development
Exit
To
Engineering
Development
Preliminary Design Review (PDR)
approval.
Hand-off to development teams
who build and test the system.
 Preliminary Design (System Concept)
 The system concept is implemented.
 Functional A-Specs, System Architecture
 The product baseline is set.
 Baselined System Requirements
 Functions are allocated to
components, components integrated
into subsystems, and subsystems into
the whole system.
 Capability Development Document
 CONOPS, System Context
Concept
Development
Engineering
Development
12
Transition to Engineering Development
Engineering Development offers two paths leading
to validation of system capability.
1 Advanced Development
 Resolves uncertainties in the system
concept through analysis, simulation,
development, and prototyping.
 Reduces risks by validating the
functional design of unproven
subsystems and their components.
2 Engineering Design
 For major subsystems derived from
a proven predecessor.
 Or mature subsystems with reliably
predictable characteristics.
 Such as a new automobile model.
source: Kossiakoff et al. (2011). Systems engineering – Principles and practices. Wiley. Ch. 10
13
From Concept Development to
Engineering Development
Classic Systems Engineering Lifecycle Model
System Functional Specs
Concept
Development
Needs Analysis
Concept Exploration
Concept Definition
System Design Specs
Engineering
Development
Engineering
Development
Advanced Development
Engineering Design
Defined System Concept
Validated Development Model
Adapted from Kossiakoff et al. (2011). Systems engineering: Principles and practice. Wiley. Ch. 10.
14
References
Department of Defense. (2012). DoD test and evaluation management guide
(6th edition). U.S. Department of Defense.
Federal Highways Administration, Office of Operations. (2023).
Designing for transportation management and operations: A primer
(Chapter 2.5). U.S. Department of Transportation.
Kossiakoff, A., Sweet, W., Seymour, S., & Biemer, S. (2011).
Systems engineering – Principles and practices (2nd edition, Ch. 7-13).
J. Wiley. (1st edition published 2003).
National Aeronautics and Space Administration. (2007).
NASA systems engineering handbook (Rev 2, NASA SP-2016-6105). NASA
Headquarters, U.S. Government.
15
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
Title
T&E Throughout the Life Cycle
Concept Development Phase
Filename
1.3 Concept Development
Author
J.Ziarko
Version
July 2023, V1.00
System Test and Evaluation
Test and Evaluation (T&E) Throughout the Life Cycle
Engineering Development Phase
Engineering Development
Concept
Development
Engineering
Development
Post
Development
2
T&E During Engineering Development
Create
detailed test
plans and
procedures
 Implement
testing
components
and their units
in each
subsystem.
Prepare and use
System
Integration Lab
(SIL)
Conduct
Conduct
Informal Testing Formal Testing
Integration and Test
Qualification Testing
 Use lab to conduct
Informal and Formal
testing activities.
 Conducted by
developer (supplier
or contractor).
 Developer “run for
the record” with
customer.
 Includes facilities and
equipment.
 End result is a fully
integrated, stresstested mature
system.
 Developmental Test
and Evaluation
(DT&E).
 Desired outcome is
system acceptance.
3
source: US DOT Federal Highways Administration. (2023). Designing for Transportation Management and Operations: A Primer. Ch. 2.5.
Setting the Product Baseline
 Critical Design Review (CDR) completes detailed design.
o Focus is issues deemed critical in a Preliminary Design Review (PDR).
o Items which need additional analysis, breadboard/brassboard, prototype
tests, or simulations.
 Reviews detailed test plans and procedures.
Test
Plans
o Also reviews specifications, detailed engineering drawings, schematics,
data flow diagrams, prototypes, hardware, interface control drawings,
configuration control plan, quality assurance plan, etc.
o Interactions among Components are analyzed, may be conducted
separately for hardware and software Configuration Items (Cis).
o System can now be built and tested with the product baseline set.
source Kossiakoff et al. (2011). Systems engineering – Principles and practices. Wiley. Ch. 12
5
System Decomposition and Integration
Concept Development
Level
Needs Analysis
Concept Exploration
Concept Definition
System
Define system
capabilities, effectiveness
Identify, explore, synthesize
concepts
Define selected concept with
specifications
Subsystem
N/A
Define feasible requirements
Define physical and functional
architecture
Component
N/A
Visualize
Allocate functions to
components
Sub-Component, Part or Unit
N/A
N/A
N/A
Engineering Development
Level
Advanced Development
Engineering Design
Integration and Test
System
Validate concept
N/A
Test and evaluate
Subsystem
Validate subsystems
N/A
Integrate and Test
Component
Define specifications
Design and test
Integrate and test
Sub-Component, Part or Unit
Allocate functions to components
Design
Make or buy
N/A
source: adapted from Kossiakoff et al. (2011). Systems engineering – Principles and practices. Wiley. Ch. 7
6
SIL Capabilities
Described in Test Approach
SIL design determines capabilities and uses.
Conducts testing in a controlled environment.
Multi-platform facility
Provides expertise and resources to support
systems engineering test activities.
From power systems to hardware to
software to communications and more
Equipment includes servers, processors,
workstations, networks, simulators,
transceivers, data recorders, and more…
NASA Glenn Research Center
Space Environments Complex (SEC)
in Sandusky, Ohio
7
Two Major Test Activities use the SIL
Integration and Test
(Informal Testing)
Performed by contractor to verify a fully
integrated, stress-tested system.
Qualification Testing
(Formal Testing)
Managed by contractor with customer as
witness to assure the system is ready for
acceptance.
NASA Glenn Research Center
MARS Rover parachute
8
Informal Testing – Integration and Test
Integration Approach
Depicts how subsystems and their
components are integrated into the system,
through incremental “Build and Test”.
Integration Test cases
Cover requirements verified by Test OR
Demonstration.
9
Formal Testing Qualification Testing
Qualification Approach
Describes how to prepare and conduct a
“run for the record”.
Qualification Test Cases
Cover all requirements, one or more
requirements for each test case.
Each requirement is verified by only ONE
Method of Verification - Inspection,
Analysis, Demonstration, or Test.
source: DoD Test and Evaluation Management Guide, December 2012, p.86
10
What is Developmental T&E (DT&E)?
 Formal testing (Qualification Testing) conducted to
demonstrate Engineering Development is complete.
o Results from Informal Testing (Integration and Test) inform
Formal Testing (Qualification Testing).
o Includes dry runs by the contractor and a run for the record which
includes the customer as witness.
o Performed by the contractor to reduce risk and qualify the system so
it is ready for acceptance by the customer.
o Results are evaluated to verify risks have been minimized, the system
will meet specifications, and the system performs as intended.
source: DoD Test and Evaluation Management Guide, December 2012, p.122-123
11
From Engineering Development to
Post Development
Post
Development
Engineering
Development
Developmental T&E (DT&E)
 Conducted by development test team with
customer in the loop.
 Verifies against requirements and specs.
 Predicts performance expected from
model-based functional design.
 Demonstrates engineering design and
development process is complete.
 Results evaluated to assure system meets
requirements and performs as intended.
Operational T&E (OT&E)
 Conducted by independent organization,
not the acquirer or supplier.
 Validates system design, follows System
Qualification and Acceptance.
 Test results are compared to operational
requirements.
 Focused on mission accomplishment
rather than meeting specifications,
requirements, MOEs, and MOPs.
source: DoD Test and Evaluation Management Guide, December 2012, p.122-123
12
Transition to Production and Deployment
Engineering Development
Test and Evaluation
Plan
Engineering
Design
System Production
Specifications
Integration and
Evaluation
System integration
System Test
Operational Evaluation
Engineered
Components
Production and
Deployment
Production
System
source: Systems Engineering – Principles and Practices - Kossiiakoff et al Wiley, 2011 Ch. 13
Post Development
13
References
Department of Defense. (2012). DoD test and evaluation management guide
(6th edition). U.S. Department of Defense.
Federal Highways Administration, Office of Operations. (2023).
Designing for transportation management and operations: A primer
(Chapter 2.5). U.S. Department of Transportation.
Kossiakoff, A., Sweet, W., Seymour, S., & Biemer, S. (2011).
Systems engineering – Principles and practices (2nd edition, Ch. 7-13).
J. Wiley. (1st edition published 2003).
National Aeronautics and Space Administration. (2007).
NASA systems engineering handbook (Rev 2, NASA SP-2016-6105).
NASA Headquarters, U.S. Government.
14
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
Title
T&E Throughout the Life Cycle
Engineering Development Phase
Filename
1.4 Engineering Development
Author
J.Ziarko
Version
July 2023, V1.00
System Test and Evaluation
Test and Evaluation (T&E) Throughout the Life Cycle
Post Development Phase
Post-Development
Concept
Development
Engineering
Development
Post
Development
2
T&E During Post Development
Conduct
Operational
T&E (OT&E)
Conduct
Early Initial
OT&E (IOT&E)
 Field tests or
 Testing performed
Demonstrations
during Low Rate
exercise full
Production (LRP).
system capability  Assess if system is
in an operational
ready for FRP.
environment.
Conduct
Follow-on OT&E
(FOT&E)
Conduct
Live Fire T&E
(LFT&E)
 Testing performed
during Full Rate
Production (FRP).
 Validate
survivability
for defense
systems.
 Multiple copies of
system produced
and tested.
Focus is mission accomplishment following system acceptance by the customer.
Conducted by an independent organization from the acquirer and their supplier.
3
Post
Development
Production
 Produces copies of product, emphasis
on efficiency, economy, and quality.
 Technicians highly skilled automated
equipment and factory test operators
 Engineers are specialists in their
disciplines.
 Automated manufacturing is used and
configuration control is tight.
 Focus is process design, quality control,
tool/test equipment design and
calibration, factory supervision, and
troubleshooting.
NASA. Christer Fuglesang STS116 underwater
EVA simulation
4
OT&E Activities
Initial Operational Test and
Evaluation (IOT&E)
• Follows ORR, conducted on production
or production representative articles.
• Validates systems are operationally
effective and suitable for intended use.
• Supports decision to proceed beyond
Low-Rate Initial Production (LRIP).
NASA USAF KSFC
5
OT&E Activities
(cont.)
Follow-on Operational Test &
Evaluation (FOT&E)
• Conducted after Full-Rate Production
Decision Review (FRPDR) during
fielding or deployment with operational
support.
• Refines estimates from IOT&E,
evaluates changes and re-evaluates
system.
• Ensures system continues to meet
operational needs and retains
effectiveness in its environment.
High Speed Rail Hyperloop
6
T&E for Defense Systems Includes LFT&E
LFT&E Example
The Army conducted live fire testing of
old missiles to confirm older weapons
are still reliable and ready for use.
Plans are to reduce tests by one third
by 2030 as more tests move to the
“digital world”
US Army, White Sands Missile Range Dec.14, 2021
source: Defense News, July 12. 2022
7
Conducting Tests During OT&E
 Conducted by an independent organization.
o The system developer may participate as observers or for support.
o Developer may not influence the conduct or analysis of tests.
 Detailed test plans and procedures are reviewed.
o An analysis plan is prepared for each test event to specify how the
data will be processed to evaluate system.
o Includes test site preparation, configuration of the test equipment,
setup of the system, and step-by-step conduct of each test.
Test
Plan
o Test equipment must be calibrated and fully staffed.
o Adequate supplies of consumables and spare parts, transportation
and handling equipment, and technical data and manuals.
source: Kossiakoff et al. (2011). Systems engineering - Principles and practice. Wiley.
8
Test Scenarios for Field Tests
 Consists of a series of events
or specific test conditions.
o Objective is to validate all of the
system requirements in an efficient
manner.
o Test planning and sequencing
makes the most effective use of test
facilities and personnel.
o Each test builds on preceding tests.
Blue Angels
9
Operational Demos
 Conducted with extensive customer
observation and may include customer
operators who require training.
Operational Field Demo: revalidate requirements in
field setting.
Reliability Demo: operational demo to assess reliability
of data.
On-site Test Demo: repeat acceptance test on
customer operational site.
Environmental Demo: field operation in extreme
environmental conditions.
10
Example: Operational Demo
Boeing Orbital Flight Test-2
(Boe OFT-2)
 Starliner Spacecraft 2 launched 19 May 2022
 Docked with ISS - 21 May 2022.
 Landing, White Sands Missile
Range - 25 May 2022
Boeing Starliner
11
What is End-to-end Testing?
Exercises assembled system in its operational environment through expected modes
to verify data flows and performance. Outputs at end points match expected results.
source: NASA Systems Engineering Handbook (2007, p.94)
12
Operational
Assessment (OA)
Base on mission need, CONOPS and
operational requirements.
Assess ability to support OT&E.
Conduct any time using technology demos,
prototypes, mock-ups, models, simulators.
Independent Operational Test Organization
evaluates with user support.
 Operational Effectiveness - Ability of system to
accomplish mission when used in operations.
 Operational Suitability - Degree to which a
system placed in field is sustained satisfactorily.
 Survivability - Susceptibility, vulnerability, and
recoverability for effectiveness and suitability.
13
Deployment, Operations and Sustainment
•
Fielded systems are assessed once
deployed into operations.
•
Testing occurs during upgrades or
changes to the system, or to determine
if the system needs to be retired.
•
Systems are sustained until deemed
obsolete, disposed due to accidents or if
incapable of performing as intended.
NASA Hubble Telescope
14
Operational
Assessment and
Operational Test
 Normally conducted in each life
cycle phase.
 Keyed to decision reviews to
check system performs as
intended in its operational
environment.
 Continues to assess system
performance during in
Operations & Maintenance
NASA James Webb Telescope
15
Decision Reviews Keyed to
Operational Assessments (OA)
Decision Review
When Review is Conducted
System Requirements Review (SRS)
Before PDR
System Functional Review (SFR)
Before PDR
Preliminary Design Review (PDR)
Before CDR
Critical Design Review (CDR)
Before System Integration and Test
Test Readiness Review (TRR)
Before System Qualification Testing
System Verification Review (SVR)
Before PRR
Production Readiness Review (PRR)
Before Production
Operational Test Readiness Review (OTRR)
Before OT&E
16
source: US DOT Federal Highways Administration (2023), Ch. 2.5
In Sum – T&E Throughout the Life Cycle
 Concept Development
o Test Approach is created to plan test and evaluation activities.
o Requirements Analysis using the VCRM assigns a Method of Verification to
each requirement.
o Conceptual Design is selected and verified.
 Engineering Development
o Informal Testing and Formal Testing are conducted.
o Developmental Test and Evaluation (DT&E) results in a fully integrated,
tested system accepted by the customer.
 Post Development
o Operational Test and Evaluation (DT&E) determines if the system performs
as intended in its operational environment. Field tests are conducted.
o Operational Assessments and Tests occur throughout the life cycle,
continues once the system is fielded and until retired.
18
References
Department of Defense. (2012). DoD test and evaluation management guide
(6th edition). U.S. Department of Defense.
Federal Highways Administration, Office of Operations. (2023).
Designing for transportation management and operations: A primer
(Chapter 2.5). U.S. Department of Transportation.
Kossiakoff, A., Sweet, W., Seymour, S., & Biemer, S. (2011).
Systems engineering – Principles and practices (2nd edition, Ch. 7-13).
J. Wiley. (1st edition published 2003).
National Aeronautics and Space Administration. (2007).
NASA systems engineering handbook (Rev 2, NASA SP-2016-6105).
NASA Headquarters, U.S. Government.
19
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
Title
T&E Throughout the Life Cycle
Post Development Phase
Filename
1.5 Post Development
Author
J.Ziarko
Version
July 2023, V1.00
System Test and Evaluation
Requirements Traceability and Methods of
Verification, Part 1
Objectives
 Describe the importance of requirements in
planning a test and evaluation program and the
impact of bad requirements
 Define Verification and Validation. Compare and
contrast each.
 Describe the four types of Requirements
Verification Methods and describe where to use
them
2
How are Requirements Defined?
IEEE Definition
INCOSE Definition
EIA Standard Definition
A condition or
capability needed by
a user to solve a
problem or achieve
an objective.
The transformation of
a stakeholder, useroriented view of
desired capabilities
into a technical view
of a solution that
meets the operational
needs of the user.
Something that governs
what, how well, and
under what conditions a
product will achieve a
given purpose.
Characterization statement of desired system performance —
1.
What it is supposed to do
2.
How well it does it do it
3
Elements of a Requirement
1 Description
 A detailed
description or
characterization of
desired
performance.
Describes the
testable element of
a requirement.
2 Criteria
The threshold/published acceptable level of
performance (where on the yardstick)
 Quantitative measures are a parameter
(e.g., detection range) with a criterion of
≤ or ≥ some value and are preferred.
&
3 Grammar
Meaning
 Requirements
usually contain a
“shall” statement.
 “Must” and
“will” also appear.
 Qualitative measure uses a statement
like “capable of x, y, and z” and usually
have a criterion of “Y” or “N” (or some
level of relative goodness based on
some scale)
4
Types of Requirements
Operational
Helps determine
Operational
Validity of design
Functional
Describes how a
system must
behave
Performance
Defines how much a
function must be
executed, generally
measured in terms
such as quantity,
accuracy, coverage,
timeliness, or
readiness
Constraint
Defines the Non-Functional aspects of a
system or component, such as
restrictions on technology, resources or
techniques to be used (DO-178C)
Interface
Defines structural limiting values at the
interface, such as interface loads, forcing
functions, and dynamic conditions..
(NASA systems Engineering Handbook).
This can include structural, electrical,
human-machine
5
Characteristics of Good Requirements
for T&E
Complete
Unambiguous
Consistent
Verifiable
All external
behaviors are
defined
Every
requirement
has one and
only one
interpretation
No subset of
requirements
conflict with
each other
A cost-effective
finite process
exists to show
that each
requirement
has been
successfully
implemented
6
Characteristics of Good Requirements
for T&E (cont.)
Traceable
Modifiable
Origin of each
requirement is
clear and facilitates
referencing each
requirement within
lower-level
documentation
Changes to
requirements
can be made
easily,
completely, and
consistently
Ranked for
Importance
Ranked for
stability
Each
requirement
rated for
criticality to the
system
Each
requirement
rated for
likelihood to
change
Which ones are important to T&E – They all are!
7
Why Good Requirements are Important
for T&E
 Sets the stage for solid
evaluation of a system
 Hard to “prove” bad
requirements
If you can’t prove it, it doesn’t exist!
8
Verification and Validation of System
Design
1
Verification - Are we
building the system right?
 Performed at each step in the
process of building the system
 Provides confidence that design
solution addresses desired
capabilities
 Assures that systems are
developed IAW design
specifications
2
Validation - Are we building
the right system?
 Confirms the realized system
meets user’s needs and complies
with functional and other
requirements (approval)
 Ensures that system (component)
performs its intended purpose in
its intended environment
 Is operationally effective and
suitable for use
9
Examples of Requirements – Verifiable
or not?
Which of the following are verifiable system requirements?
1.
The software shall be user friendly.
no
2.
The system shall have an availability of .9995.
yes
3.
The user terminal display shall have a green background with pink
borders.
yes
4.
The radar shall process all incoming returns in less than 25 msec.
yes
5.
The Users Manual shall be comprehensive and address all situations.
no
6.
All user responses for critical system functions shall be verified by
repeating the prompt and making the user respond again.
7.
The maximum time from request to display of location in cold start-up
shall not exceed sixty (60) seconds.
yes
8.
The MTS shall provide a group of driver application and decision aids.
yes or no ?
yes or no ?
10
Methods of Verification
Inspection
Analysis
Demo
(Static)
(Static)
(Dynamic)
Test
(Dynamic)
11
Inspection – A Static Method of Verification
1 What is inspection?
 Examination of physical (not
functional) features, properties
or configurations to assure
compliance with requirements
/standards
2 When to use inspection
 To provide visual evidence of an
observable physical feature or
configuration against defined
criteria
 To show compliance with industry
standards, such as workmanship or
quality
 To show the use of COTS
12
Inspection – A Static Method of Verification (cont.)
Examples
 Each CCC workstation shall have
a high resolution, 15-inch or
larger color display (visual
inspection of display)
 The system shall operate in internal temperatures
from 10 to +35°C. (Inspection of vendor data
sheet)
13
Analysis – A Static Method of Verification
1 What is analysis?
 Use of analytical techniques in
lieu of (or in addition to)
testing to show compliance
with requirements
 Based on calculations or
measured data derived from
lower-level product
Verifications
2 When to use analysis
 When quantitative estimates of
performance are needed
 As interim verification step due to
unavailability of HW/SW
components for testing
 As alternative or preliminary step
to potentially destructive testing
 To process accumulated data from
other methods
14
Analysis – A Static Method of Verification (cont.)
Examples
The system shall have a Mean Time
Between Critical Failure (MTBCF)
during normal operations of 12,000
hours (calculation of MTBCF
equation using offboard data)
The CPU shall not exceed fifty percent
(50%) of processing throughput, physical
memory utilization, disk utilization, and
input/output throughput during any
maximum loaded system operation
(calculation of parameters using data
collected in various testing events)
15
Types of Analysis
Quantitative Analysis
Similarity Analysis
 Analysis accomplished using simplified
quantitative assessment that a
specification requirement is met
 Analysis accomplished to confirm that
a requirement is met by virtue of a
previous verification activity.
o
o
Example – time to perform a
functional task
Example - summation of
measurement values to achieve a
total evaluation
 Similarity analysis is applicable when
all of the following criteria are met:
o Same configuration as a previously
verified item
o Performs a similar function as the
previously verified item
o Environment and operating limits
are comparable
16
Demonstration – A Dynamic Method of Verification
1 What is demonstration?
 Observation of a functional
characteristic to determine compliance
with specified requirements
 A basic confirmation of performance
capability, differentiated from testing
by the lack of detailed data gathering
 Generally thought of as a Qualitative
dynamic verification
2 When to use demonstration
 When functional characteristics can be easily
observed
 To show normal operational modes or
functions (how the system/subsystem/unit
operates)
 When quantitative proof of performance is not
required
 When special equipment or sophisticated
instrumentation is not required
 When repeated trials or runs are not needed
17
Demonstration – A Dynamic Method of Verification
(cont.)
Example
 Upon power up, the MTS shall
query the user/driver for a
password and provide password
verification.
 The human-machine interface shall
use drop down windows for
functions and option
18
Test – A Dynamic Method of Verification
1 What is a test?
2 When to use a test
 Verification through actual system
operation under controlled conditions
or when subjected to specified
environments to evaluate
performance
 When requirements to be verified require
statistical confidence and power –
 Produces data at discrete points for
each specified requirement
 To verify critical functions, interfaces,
varied conditions or operational states
(requiring controlled repetition of runs)
 Typically, the most resource intensive
verification technique
 When your requirement has a
threshold you are most likely going
to verify by test
 Considered the Quantitative dynamic
verification method
19
Summary – Verification – Important Questions to Ask
Work with systems engineers to ensure
requirements are verifiable —
1. Do the defined requirements truly
define the user needs?
2. Do lower level requirements trace to
those user needs?
In the end it is about
determining that a
system works for the
user!
3. Do the system engineers understand
the implications of the stated
requirements regarding validation
and verification of those
requirements?
20
We’ll continue our conversation
about requirements traceability
and methods of verification in the
next video segment.
21
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Requirements Traceability and Methods of
Verification, Part 2
Considerations
for Customer
Needs
2
Customer Needs and Shaping the Program
Begin Early
T&E begins before any
hardware or software is
designed or integrated.
Evaluate for Match
Test engineers evaluate the
system conceptual design to
ensure designs to be produced
match to user needs.
Consider Verification & Validation
Assure yourself that customer needs can be
properly verified and validated.
3
Conceptual Validation
 Conceptual validation is an activity
that early T&E supports.
 This includes activities that
validate the documented CONOPs
that represents the user needs —
o Conducted early in the system
design process
• Models and simulations are
often used to aid in the
validation analysis
4
R equirem ents Validation
 Activities that validate the system
requirements represent the
CONOPs.
 Process establishes system
requirements that have neither
introduced new issues or left
issues out of the CONOPs.
 Heavily focuses on the systemto-external interfaces.
 Models are very useful — SySML
Internal Block Diagrams and
Activity Diagrams.
5
Customer Needs through CONOPs
 Environment
o Natural: All weather, all terrain, all locations
o Man-made: High G prior to use, mobile
 Stimuli
o Desired: Downed pilot’s report
o Undesired: Enemy jamming, enemy
deception
 Complications
o Security for pilot and rescue force
 Interactions Among System Elements
o Data: Codes, position, time
o Function: Position calculation,
communications, rescue coordination
CONOPS can form a solid
foundation for T&E Use Cases
6
Customers of T&E
 Who is the customer? (many lenses)
o Integration Engineers
(component/subsystem performance)
o “Specialty Design” Engineers – flutter,
vibe, etc.
o System architects – early desire to
verify conceptual design
 Ultimate customer – The User!
7
Summary
1. Good requirements are important
for planning a test and evaluation
program.
2. T&E plays a major role in Systems
Engineering. Getting T&E involved
early in the SE process is vital to a
successful program.
3. Understanding the customer and
user needs are key to the success
of a test program.
8
References
 Box, G. E. (1978). Statistics for Experimenters. New York: John Wiley.
 Buede, D. M. (2009). The Engineering Design of Systems. Hoboken: John Wiley and
Sons.
 Coleman, D. E., & Montgomery, D. C. (1993). A Systematic Approach to Planning a
Designed Industrial Experiment. Technometrics, 1-12.
 National Aeronautics and Space Administration. (1995). NASA Systems Engineering
Handbook. Washington, DC: National Aeronautics and Space Administration.
 Radio Technical Commision for Aeronautics. (2012). DO-178C Software
Considerations in Airborne Systems and Equipment Certification. Washington, DC:
Radio Technical Commision for Aeronautics.
9
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Test Objectives, Concepts and Environments, Part 1
Objectives
 Define Test objectives and Test
Concepts
 Describe the Verification Cross
Reference Matrix and the value it
brings to the T&E Engineer
 Describe the impact of the operational
environment on defining test
objectives and concepts
2
Why Test objectives?
 Overarching objective of T&E – Reconstituting the physical system according to it’s design and
demonstrating that it performs robustly in it’s operational environment
o Many stakeholders, each with different roles, have different needs out of system development

Team approach to developing a test program
is key

Purpose is to develop objectives for all, agree
on what determines “success” of objective
completion and gain commitment to the
program from all stakeholders (Box, 1978)
The objectives are the “chain”
that keeps everyone together
3
Guiding Principles for Unambiguous Test Objectives
 Keep test objectives unbiased
o Participation by diverse group
o Collaborative participation
 Keep objectives specific
 Objectives need to be measurable
 Objectives need to have practical
confidence
4
Test Concept
 Test Concept helps to assure the system will
operate as intended in it’s operational
environment
 Systems Engineering thinking applies to
creating logical, sound test concepts
 Test Concepts should be based on proposed
system capabilities, Interfaces and functions
o Is the system completely new or
revolutionary or a modification and
evolutionary?
The creation of a test
construct that will verify and
validate the new system
meets user needs and meets
contractual requirements (via
requirements doc or spec)
o Does the system exist but being used in a
completely new way (new customer base)
5
Test Concept (cont.)
 Derived from the system CONOPs, requirements and the system test objectives
o CONOPS articulate the intended system operating environment, external stimuli
and interactions with other systems
o Requirements provide functional and performance context to system operation
o Interfaces (intersystem and extrasystem)
• Candidate “natural test points”
• Interface examples: HMI, Environmental, Signals, data, logistics
6
Test Concept (cont.)
 Creation of logical “building blocks ”
or partitions
o The “crawl, walk, run”
philosophy
o Hierarchical structure executed
from the “bottom up”
o Map requirements to each
1.
2.
3.
Start with small
HWCIs/CSCIs/High risk items
(new interfaces/tech)
Integrate and evaluate
HWCIs/CSCIs into larger
entities (subsystem)
Integrate and evaluate
subsystems at the system
level
Avoid the “big bang” approach – you will not
be certain where to identify fault defects
7
When to Treat Interfaces as a Prerequisite
Function
requires
several nonco-temporal
data flows
And/Or
Several
interfaces
required to
support data
flows
Consider verifying interfaces first
8
When to Test Interfaces with Functionality
Interface test
is inherent in
function test
Functions
within the
system are
extensions of
others
9
Master Test Strategy
 Overarching project test strategy for
the lifecycle of the project:
o Test objectives
o T&E concepts
o Test domain roles and
responsibilities
o Required resources and required
test schedules
 “Contractual” in nature
 Test and Evaluation Master Plan
(TEMP) for DoD programs
 Developed very early in project
timeline
 Used as the foundational
document to support detailed test
plan development
10
Master Test Strategy (cont.)
Contains a T&E
Master
Schedule
aligned with
the overall
project
schedule
11
Master Test Strategy (cont.)
 Signatures required on a Master
Test Strategy
o Program Manager
o T&E Lead
o User Representatives
o Oversight Organizations
12
Test Plans
 Test plans document the “how to” for
test execution
1. Document data collection needs and
methods
2. Refine details contained within the
VCRM
3. Describe how the test resources are
allocated and scheduled
4. Provide the basis for test procedures
Test plans are the
bread and butter of
the test program
13
Test Plan Content
Scope
1
3
Tools &
Equipment
5
Test Events
Environments
2
4
Test Schedule
14
Test Plan Content (cont.) — Events
Test Events Describe …
Test Event objectives Test Configurations
Test Inputs
Expected Outcomes
Pass Criteria
Go/No Go Criteria
Required Data
Safety
This is the heart of your
plan. It makes clear how
you intend to utilize your
test site, test tools, and
test equipment and what
will be executed
15
Test Plan Content (cont.) — Personnel
Project Officer
Data Analysts
Test Engineer
The Team
Test Conductor
Specialty Engineers
Equipment Operator
Range Staff
16
We’ll continue our conversation about
test objectives, concepts and
environments in the next video
segment.
17
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Test Objectives, Concepts and Environments, Part 2
Data Collection & Analysis Plan
 Subset of the test plan
 Documents how test data will be
collected and analyzed
 Elements
o Analysis of personnel and
resources
o Collection methods
o Analysis methods
o Data configuration management
o Data reduction requirements
2
Requirements Traceability Matrix
 Also called
Verification CrossReference Matrix
(VCRM)
 Provides “one
stop” traceability
of requirements
o To Design
Specification
paragraphs
o Verification
cases
 Captures
verification
methods,
approach and
leads into further
test program
development
3
Information Clues a Skilled T&E Engineer
Looks for When Reading a VCRM
1. Testable requirements
2. Incorrect, poorly written and missing
requirements
3. I,A,D,T: are they appropriately defined
4. What are the major unknowns
5. What is important to get done early
6. Identify candidate risk reduction tests
7. What are the most expensive and
technically complicated tests
4
Test Environments
 Environments play a large role in robust
T&E design
o Systems we build and test need to
work in the real world
o Testing with considerations for
environments assures that the system
will perform in the presence of
expected environments
• As required
• When required
• For as long as required
o Testing to specification in a sterile
environment only buys part of the
evaluative solution set
5
Where to Identify the Environment for Test
1
CONOPS
MIL-STD-810G
2
Requirements documents and system
specifications
3
User Interviews
6
Importance of Being Thorough
 Not considering operating environment in design and T&E can lead to
disastrous consequences
o Space Shuttle Challenger
o Tacoma Narrows Bridge
o Ariane 5 Rocket
7
Types of Environmental Tests
Design Testing
Manufacturing
Testing
 Physical Testing
 Highly Accelerated Life Testing (HALT)
 Environmental Stress Screening (ESS)
 Highly Accelerated Stress Screening (HASS)
8
Types of Environments
Physical (MIL-STD-810)
 Temperature
 Shock and vibration
 Wind
 Rain
Electromagnetic (MILSTD-461/464)
 RF fields
 Lightning
 Spurious signals
 Humidity
 Fungus
 Salt fog
 Sand, dust and mud
 Acceleration
9
When to Specify Environmental Testing
 The contract requires
environmental testing
 The design is new
 A new application of an old
design
 Supplier history is unknown or
suspect
 The result of failure is
catastrophic
 Lower-level testing has not
been done
MIL-STD-810G
10
Benefield Anechoic Chamber, Edwards AFB
11
Lightning Tests, NAS Pax River
12
In Summary
 Team approach to developing a
test program is key
 Test Concept helps to assure the
system will operate as intended in
it’s operational environment
 Environments play a large role in
robust T&E design
13
References
 Coleman, D. E., & Montgomery, D. C. (1993). A Systematic Approach to Planning a
Designed Industrial Experiment. Technometrics, 1-12.
 Truitt, L. (2015). Developing Effective Test Objective Best Practice. Wright-Patterson
AFB: STAT Center of Excellence.
 System Engineering, Practice & Principals, Kossiakoff, Sweet, Seymour,
Biemer, 2011 Chapter 13
 DoD Test and Evaluation Management Guide Chapter 12 (January 2012, 6th Edition)
 MITRE System Engineering Guide, 2014 Test & Evaluation, pages 402-33
 Mil Standard 810, Environmental Engineering Considerations and
Laboratory Tests
14
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
MBSE and Architecture for T&E
Objectives
 Provide a refresher on MBSE
concepts
 Discuss the advantages of MBSE
for test and evaluation
 Discuss verification and validation
methods in a model based
engineering environment
2
MBSE – A Refresher
 MBSE uses digital models to describe and
represent all aspects of a system
o Requirements
o Use cases/how the system is to be
used
o Physical performance and behaviors
o Functional and physical designs
Systems Engineering
models can provide a
“Single Source of Truth”
which provides users a
view of the system through
their particular lens
Digital models tend to be
more precise and
transparent
3
SysML Framework
4
Advantages of MBSE
1.
MBSE can help define test
requirements early in the design
process, and to refine those test
requirements as increasing levels of
detail are added to a system design.
2.
MBSE may be used for development
of detailed test plans on all aspects
of the system.
3.
MBSE also holds the promise of
streamlining the test results
feedback loop.
5
Advantages of MBSE (cont.)
4.
Model-based approaches to design systems can
move verification and validation activities to an
earlier point in the systems engineering process
than occurs with traditional approaches.
5.
Facilitates verification of the conformance of
design models to their stated requirements by
use of the Requirements Block Diagram
stereotype -- <<verify>>
6.
The verification and validation process can
become model-driven, instead of focusing
primarily on interpretation of test results.
6
Verification and Validation of SE Models
1.
Automatic verification of models using Model
Checking techniques
o Works on the formal semantics of the model
• Structured Operational Semantics
(describes model meaning in small steps)
• Denotational Semantics (model meaning
be mathematical methods)
o Verification explores model state space –
looks for flaws in properties
2.
Static inspection of SysML structural models
o Requirements
o Physical design inspection
7
Verification and Validation of SE Models (cont.)
3. Analysis of SysML Behavioral Design Models
 Activity diagrams (state machine,
activity and sequence diagrams) for
basis of system behavior simulation
 Proper semantics tagging allow for
comparison of actual system behavior
vs expected behavior
8
MBSE Testing Standards
 Standard Profile for modeling test –
UML 2 Testing Profile (U2TP)
 System (SysML) modeling and test
modeling in one model based on
commonality to UML 2.0
 Consists of Architecture, Test Data,
Test Behavior and Test Times
 Architecture – Arbiter, scheduler,
system under test and test elements
9
Summary
 MBSE frameworks, schemas and
taxonomies support Verification
and Validation.
 Automated Verification techniques
can be heavily leveraged in an
MBSE environment.
 Standards exist to encourage
commonality among MBSE
schemas and Test toolsets
10
References
 Debbabi, M., Hassaine, F., Jarraya, Y., Soeanu, A., & Alawneh, L. (2010). Verification and
Validation in Systems Engineering. Berlin: Springer.
 Hause, M., Richards, D., Stuart, A., & Holt, J. (2010). Testing Safety Critical Systems with
SySML/UML. 15 IEEE International Conference on Engineering of Complex Computer Systems
(pp. 325-330). Oxford: IEEE Computer Society.
 Linhares, M. V., de Oliviera, R. S., Farines, J.M., & Vernadat, F. (2007). Introducing the Modeling
and Verification Process in SysML. IEEE Explore, 344-351.
 Friedenthal, S., Moore, A., & Steiner, R. (2009). A Practical Guide to SysML. Amsterdam: Elsevier.
 Piaszczyk, C. (2011, VOL 14 3). Model Based Systems Engineering with Department of Defense
Architectural Framework. Systems Engineering, pp. 305-326.
 Object Management Group. (2005). UML 2.0 Testing Profile Specification. Needham: Object
Management Group.
 Bjorkman, E., Sarkani, S., & Mazzuchi, T. (2013, Vol 16 No 3). Using Model-Based Systems
Engineering as a Framework for Improving Test and Evaluation Activities. Systems Engineering,
pp. 346-362.
11
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Statistical Techniques for Test Design and Planning
Part 1
Objectives
 Describe statistical testing and
importance of using for proper scoping
of test programs.
 Describe the types of statistical
methods used in test planning.
2
Statistical Test Designs - Defined
 Evaluation of Quantitative requirements – analysis of data to compare against a
threshold
 Test data is imperfect
o Data contain noise (instrumentation, random sampling)
o Data sets limited by practicality (cost, limitations to test)
 Consumers of test evaluations need to know our Confidence in our evaluations
o Driven by what we test
o … and also how MUCH we need to test
Statistical test designs help us determine the What and the How Much
3
Scientific Method – Test Design
Formulate a
probabilistic answer
to the question
Carry Out Data
Analyses
Identify the Question
Observation/
Question
Report
Conclusions
Research
Topic Area
Analyze Data
Hypothesis
Define the population to
which the question
applies
Test with
Experiment
Collect a Sample
4
Types of Statistics
 Descriptive Statistics
o Examples - Mean, Median, Range
o Analysis applies to the sample collected –
myopic point of view
o Based on univariate data
 Inferential Statistics
o Data collected from designed test events
o Evaluation used to reach meaningful
conclusions about system performance in the
“real world”
o Based on Bi- or Multivariate data
o Designs take into account “imperfect data”
Univariate
Bivariate
It only summarizes a
It only summarizes
single variable at a
two variable
time
It does not contain
It contains one
any dependent
independent variable
variable
Multivariate
It summarizes more
than 2 variables
It contains more
than one variable
The main purpose is
to study
relationships
Example:
Example: temperature temperature, Ice
sales, water bottle
and ice sales at the
disposal in beach
local market
trash cans
The main purpose is The main purpose is
to describe
to explain
Example:
temperature
5
Types of Inferential Test Designs
SUT –
System
under Test
OT Statistical Test and Analysis Handbook, Operational Test and Evaluation Force, Version 1.0 May 2020
6
Inferential Method – Response Variables (RV)
 When we use them
o Outcome of the measured requirement is
dependent on conditions (factors)
o There is a means to systematically control
the conditions
 RV design process litmus test
o Which conditions (factors) affect the RV?
o How much effect do the conditions have on
the RV?
o What is the predicted performance of the
system under the specific conditions the
user expects to see out of their system?
7
Data Distribution Types
Sharma, A. (2019). Understanding Different Types of Distributions You
will Encounter as a Data Scientist. From https://medium.com/mytake/
8
Four Phases of Inferential RV Design
Plan
Design
Test
Montgomery, D.C. (2008). Design and Analysis of Experiments. John Wiley and Sons
Analyze
9
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Statistical Techniques for Test Design and Planning
Part 2
Design of Experiments (DOE) - Defined
1. Statistical test design used to
characterize system performance
under systematic, controlled variation
of conditions

2. DOE creates efficient designs based
on desired confidence and power



Performance based on specific quantitative requirements (Response
Variables)
Conditions chosen on postulated impact on performance (Factors)
Power – probability of accurately identifying an effect on RV when
one exists
Confidence – Probability of accurately concluding an effect on RV
when does not exist
3. Allows for detailed definition of test
resources
2
Benefits of DOE
1. Provides efficient technique for determining
system/process responses to intentional
changes in candidate critical parameters.
2. Allows for study of multiple input factors
simultaneously.
3. Provides structured analysis of input factor
main effects and interactions.
3
DOE Terminology
Main Effects
Interactions
 The average change in
the response due to
moving an input
parameter from its low
or “off” level to its high
or “on” level while
holding all other factors
fixed.
 The failure of one input
parameter to produce the
same change in the response
at different levels of another
input parameter
 Ex: Interaction between type
of driver and type of beverage
4
Selecting the Experiment Design
https://www.synthace.com/blog/types-of-doe-design-a-users-guide
Your selection of design will influence test resources.
5
Response Variable Types
Continuous Variables
Discrete Variables

You count discrete data

Nominal or ordinal scales

Examples – any requirement
with yes/no, hit/miss,
success/failure answers
Continuous Variables
preferred for T&E

You measure continuous
data


Examples –requirements
measuring distance, height,
weight
Provide more useful
information about system
performance

Require less resources (510X less) than discrete
variables for equivalent
power
Work with Systems Engineers to convert discrete to continuous requirements.
6
Identification of Factors and Levels
 Conditions – 3
categories:
o Controlled
o Constant
o Recordable
 T&E, SE and Users work
together to choose
proper conditions list for
DOE consideration
 The number of
Controlled Conditions
selected directly drives
the size of test!
o Priority should be
given to those that
may have the
biggest performance
impact
7
DOE Design Constraints
Hard to Change Factors
 Example - Day vs Night
 Results in “Whole-Plot” designs – impacts
statistical power
Hard to Change Factor Impact Example
Very Hard to Change Factors
 Example – Summer vs Winter
 Create separate designs: One design for
Summer, another design for winter
 Eliminates interaction analysis between
summer and winter
Disallowed Combinations
 Unrealistic combinations of factor levels
 Example – Water Depth (500 ft), Water Temp
(> 80 Deg F)
 Combinations removed from run matrices
8
DOE Design Considerations
Replications
Repetition of runs for unique factor sets
Used to increase power and confidence
Used heavily with discrete RVs
Effect Size
Establishes “Test Sensitivity” among Factors
and test “noise”
In other words, how much difference between
factor and noise to say the factor affects the
RV? Often expressed as Signal to Noise Ratio
(SNR)
9
Analysis of Variance (ANOVA)
 DOE Factor selection influenced
by desire to understand main
effects and interactions
between factors
 Resulting data evaluation is
described in terms of Analysis
of Variance
10
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Statistical Techniques for Test Design and Planning
Part 3
Inferential Method – Confidence
Intervals
 When a test is needed to measure
performance against a requirement
where conditional variation is not
expected to influence results
 Population parameters help
characterize system performance
o Mean
o Confidence intervals around
the mean
2
Two-Sided Confidence Intervals
 Used when a need to describe likely range of
performance against the sample parameter based on
observed data
 Expectation for margins of error above and below the
sample parameter (ex, mean)
o Lower confidence limit
o Upper confidence limit
 Width of interval affected by:
o Sample size – larger the size, the smaller the
interval
o Confidence level – the higher the confidence the
larger the interval
o Standard deviation - the larger the standard
deviation, the larger the interval
https://digitalelearnings.com/confidence-interval/
3
One-Sided Confidence Intervals
 Used when a assessing a requirement for a “Greater Than” or “Less Than”
4
Inferential Methods – Hypothesis Testing
 Fundamental to all inferential statistics
 Can be used to address performance evaluations
of system upgrades
 Null hypothesis testing
o Two concerns: Null and Alternative
o Alternative hypothesis is the one of interest
o Positive assessment of system performance
is based on rejection of the Null
 Errors – result in potential incorrect conclusions
about test results
o Type 1 error (false positive)
o Type 2 error (false negative)
5
Summary of Test Types
National Defense Industrial Association, Test and Evaluation Division, Systems Engineering Division, Developmental Test &
Evaluation Committee Report, June 2013
6
Summary
 Statistical test designs aid in right-sizing
the system test program
 Using inferential statistics help to predict
system performance from limited
samples
 There are many types of statistical
methods. Knowing your desired results
will help in choosing the right type
7
References
 Box, G. E. (1978). Statistics for Experimenters. New York: John Wiley.
 Coleman, D. E., & Montgomery, D. C. (1993). A Systematic Approach to Planning a
Designed Industrial Experiment. Technometrics, 1-12.
 Ramert, Robert A (2019). Understanding the Signal to Noise Ratio in Design of
Experiments: STAT Center of Excellence
 Commander Operational Test and Evaluation Force: OT Statistical Test Design and
Analysis Handbook (2020)
 National Defense Industrial Association, Test and Evaluation Division, Systems
Engineering Division, Developmental Test & Evaluation Committee Report, June 2013
8
© The Johns Hopkins University 2024, All Rights Reserved.
System Test and Evaluation
Engineering Development Phase
System Integration Lab (SIL) Capabilities
Engineering Development
Concept
Development
Engineering
Development
Illustration adapted from Kossiakoff, A. & Sweet, W. N. Systems Engineering: Principles and Practices, Wiley, 2003.
Post
Development
2
T&E During Engineering Development
Create
detailed test
plans and
procedures
Prepare and use
System
Integration Lab
(SIL)
Conduct
Conduct
Informal Testing Formal Testing
Integration and Test
Qualification Testing
▪ Implement
testing
components
and their units
in each
subsystem.
▪ Use lab to conduct
Informal and Formal
testing activities.
▪ Conducted by
developer (supplier
or contractor).
▪ Developer “run for
the record” with
customer.
▪ Includes facilities and
equipment.
▪ End result is a fully
integrated, stresstested mature
system.
▪ Developmental Test
and Evaluation
(DT&E).
▪ Desired outcome is
system acceptance.
3
CONCEPT DEVELOPMENT
POST DEVELOPMENT
SE Life Cycle V
TEST APPROACH
Systems Engineering
Domain
VERIFY CONCEPT
Engineering
Design Domain
VALIDATE CONCEPT
DETAILED TEST PLANS &
PROCEDURES
VERIFY UNITS &
COMPONENTS
ENGINEERING DEVELOPMENT
source: US DOT Federal Highways Administration, Designing for Transportation Management and Operations: A Primer, Ch. 2.5,. 2023.
T&E During Engineering Development
▪ This is the phase when a contractor designs, builds and tests a system.
o This phase ends with an operational evaluation of the system, and a
transition to Production in the Post-Development phase.
• System Test and Evaluation is a major activity during this phase.
5
The Big Picture
Test and Evaluation is classically partitioned into three classes of activities
INFORMAL TESTING
FORMAL TESTING
Operational T&E
Subsystem/System Integration
Developmental System
Testing (Qualification Testing)
Validate that the system
meets operational
requirements and performs as
intended
Integrate components into
subsystems and subsystems into
the total system
Verify the system meets
specifications
Test Planning: defining test approach, test issues, test scenarios and procedures, and
test tools and equipment. Applies to all three of the other activities.
6
Test Activities by Testing Phase
System Integration and Test
activities
Test Planning Activities
▪ Review subsystem/system
requirements, and define test
requirements, approach, and strategies.
▪
Integrate tested components into subsystems by
sequential, incremental aggregation and testing
of these constituent elements.
▪ Define the test tools, test facilities and
test assets required to implement the
test program for all levels of the
system.
▪
Integrate subsystems into a total operational
system by the sequential, incremental
aggregation and testing of the constituent
elements
▪
Design and build necessary integration test
equipment and facilities.
▪
Demonstrate end-to-end operation.
7
Engineering Development - Test Program Steps
1.
Develop a test
plan, objectives,
test procedures,
test analysis
plan
2.
Develop/acquire
test equipment
and special test
facilities
3.
Conduct tests
4.
Analyze and
evaluate test
results
No
Defects
Found?
5.
Tests
Completed
Defects
Found?
Retest
Regression test
Correct
identified
design
deficiencies
Excerpt From: Alexander Kossiakoff, William N. Sweet, Sam Seymour & Steven M. Biemer.
“Systems Engineering Principles and Practice”, 2011
RISK BASED
Test
Completion
Criteria
8
Test Objectives During Engineering Development
STEP 1
Informal Testing
Integration and Test
STEP 2
Formal Testing
Qualification Testing
▪ Integrate the engineered components of
a new system into an operating whole
and test to ensure it “works”.
▪ Formally demonstrate that the system
meets all its operational requirements
o Validated production designs and
specifications
o Qualification for production and
subsequent operational use
Make it work
Prove it works
These are two distinctly different activities with different objectives and approaches.
9
Systems Integration Lab (SIL) defined
1. Conducts test activities and collects results in a
controlled lab environment.
A simulation, testing and
o For requirements verified by Test and
Demonstration (system is operating).
diagnostics facility that enables
o Reduces Risk by verifying requirements are
met and the system performs as intended.
separate parts of the systems
2. Requirements verified by Inspection and Analysis
(system is NOT operating) are handled in parallel.
o Analysis includes Modeling and Simulation
(M&S) and may use SIL resources but differs
from Simulators and Stimulators used when
the system is operating.
to be integrated and tested
under controlled conditions
10
Two Major Test Activities use the SIL
Integration and Test
(Informal Testing)
Performed by contractor to verify a fully
integrated, stress-tested system.
Qualification Testing
(Formal Testing)
Managed by contractor with customer as
witness to assure the system is ready for
acceptance.
NASA Glenn Research Center
MARS Rover parachute
11
NASA Glenn Research Center Space Environments
Complex (SEC) in Sandusky, Ohio
SIL Capabilities
▪ Design determines capabilities and uses.
o Described in Test Approach
o Conducts testing in a controlled environment.
▪ Multi-platform facility
o Provides expertise and resources to support
systems engineering test activities.
o From power systems to hardware to software
to communications and more
o Equipment includes servers, processors,
workstations, networks, simulators,
transceivers, data recorders, and more
12
SIL Design Determines its Capabilities and Uses
Systems engineers include SIL Capabilities in the Test Plan
1.
Intended Use, a description of SIL Capabilities
2.
Physical block diagrams for a system and its subsystems
3.
Physical Layout of the lab
4.
List of Lab Test Tools and Resources (includes processors,
applications, and tools such as simulators and stimulators)
5.
Authorizations to proceed or operate
6.
Scheduling and life cycle cost considerations
7.
Training for test engineers and users of SIL systems and tools.
13
Test Tools – Simulators and Stimulators
Simulators
Stimulators
(replicates real-world)
(Data input to jumpstart activity)
1. External GPS simulator used to obtain
positional coordinates
1.
2. Data storage simulator for data
storage and archive within a
Subsystem
For example, an aircraft ID and
Access codes to transmit from a pilot’s
ELU to Communication Satellite
system in a SAR system.
2.
3. Radio communication system for
communications, for example, between
a pilot’s ELU and a base station in a
Search and Rescue System (SAR)
Random Positional Coordinate
generator within the ELU
microprocessor for aircraft coordinate
calculation
3.
Status message generators inputs, for
example, between a base station and
ELU to evaluate response commands
and activities
4. Hardwired power source to simulate
the required system power without the
need for an isolated power source
14
Mock-ups
Iron Bird Test Facility
▪ An Iron Bird is an
example of a full-scale
mock-up of an aircraft
under development with
all flight components
attached.
Test environment for evaluating
functionality of avionics bus, and power
distribution networks.
Under specified conditions, records a wide
range of operational parameters at high
sampling rates.
Includes Engine testing, flight control labs,
hydraulic system integration test rigs,
electro-mechanical equipment with
hardware-in-the-loop tests
15
Environmental Testing
16
Environmental Effects on System Operability
Factor
Produces
Resulting In
Land, sea
clutter, false targets
Overload buffers, processing delays,
revised algorithms
Weather
attenuation
Reduced signal, noisy communication,
track gaps
Jamming
spurious signals
Communication loss, track gaps,
false tracks
Multipath
interfering signals
Communication loss, track gaps
Ducting
propagation paths
Communication loss, track gaps
17
Lightning Tests, Pax River, MD
18
EMC/EMI
Electromagnetic
compatibility (EMC)
Electromagnetic
interference (EMI)
▪ The ability of electrical equipment
and systems to function acceptably
in their electromagnetic
environment.
▪ The unintentional generation,
propagation and reception of
electromagnetic energy or
radiation which may cause
unwanted effects or physical
damage to electrical equipment.
IEEE EMC Society (EMC)
19
Radio Frequency (RF) Spectrum Overview
20
EMI/EMC Test Chamber Facilities Example
source: Sandia EM Brochure
21
Benfield Anechoic Chamber, Edwards AFB
Simulates an infinite or open space environment.
Chambers are used in electromagnetics and acoustics.
source: Edwards AFB 772nd Test Squadron
22
Real-world SILS
▪ NAVAIR Commons SIL
▪ NASA Radioisotope Power System (RPS) SIL
▪ NASA Power Systems Facility (PSF)
▪ Boeing 777 SIL (1990s)
23
NAVAIR Commons
SIL (CSIL)
▪
Multi-platform test facility
o Independent verification and validation
(IV&V). Shares infrastructure,
resources, and expertise.
▪
Government Oversight
o Independent HW-in-the-loop and SW
integration test activities.
▪
Systems Integration
o Addresses platform capability
requirements and sustainment, ensures
compatibility of functional and physical
interfaces.
▪
Qualification Testing
o Specializes in Avionic sub-system and
platform system testing prior to aircraft
ground and flight test.
24
NAVAIR Commons SIL
NASA Power Systems Facility (PSF)
The Power Systems Facility (PSF) houses testbeds where scientists and engineers verify critical
design concepts, test prototype hardware and software, and validate systems in real-time
simulations under actual loading and operating conditions.
Concentrator mirror assembly
in the high bay cleanroom
Power System Facility at Glen
Research Center
25
Example - NASA PSF Capabilities
1. Power Systems Development Lab
o Supports medium-power
electronics and systems, has a
small vacuum chamber.
2. Modular Power Systems Testbed
o A habitat-like environment to test
modular power components and
systems.
3. Power Electronics Laboratory
o Component development
integrated into a subsystem in the
Power Systems Testbed.
4. Power Electronics Prototyping Laboratory
o Manufacturing and assembling printed circuit
boards additive manufacturing (3D printed)
models, soldering and rework, and low-power
testing, troubleshooting of power electronic
circuits.
5. Power Systems Testbed
o Where system and integration tests are
performed, end-to-end systems tests
characterize the operation of all the elements
of a power system (sources, converters,
switchgear, and loads) connected together.
6. High Bay Cleanroom
o For Flight Hardware Development.
source: NASA RPS SIL https://www1.grc.nasa.gov/facilities/psf/
26
NASA Radioisotope Power System (RPS) SIL
▪ A source of electricity and power technology for
testing of space missions from Mars to the outer
planets.
o Development, testing, and validation of
electrical power systems and support systems
for aerospace applications in the International
Space Station (ISS) and other space and
aeronautic applications.
o Designed to provide end-to-end system
testing of radioisotope convertors.
o Emulates electrical characteristics of a
spacecraft system.
NASA RPS SIL, 2024 Housed in the Glen Research
Center (GRC) Power Systems Facility (PSF)
27
Boeing 777 SIL
▪ Included all the electrical power systems, electromechanical systems, avionics,
environment control systems, propulsion systems, and a portion of payload
electronics.
▪ Integration testing included o Realistic simulations of flight modes to support verification and validation of
production equipment before the first flight of the aircraft, and certification.
o Validated the correct performance of both the physical and functional
interfaces in electrical and electronic systems during concurrent operation of
multiple subsystems and failures.
source: IEEE Xplore Boeing's 777 Systems Integration Lab, October 2000,
IEEE Instrumentation and Measurement Magazine 3(3):13 – 18. DOI:10.1109/5289.863905
28
Interim Submittal for the SIL
▪ In this submittal there are 3 products:
o SIL Block Diagram
o SIL Physical Layout
o Test Tools List and Description
29
SIL Physical Block Diagram
▪ A block diagram showing the subsystem components and test tools that
integrates with the whole system.
▪ Break down the components sufficient to show where Test tools interface.
▪ Identify interfaces by protocol, fabric or material
30
Example SAR System SIL Physical Block Diagram
31
SIL Physical Layout
▪ This is a detailed floor plan of the SIL.
o Shows rooms and identifies their usage (work areas, storage, offices,
conference rooms, etc.
o Identifies where key components are located (workstations, servers, etc.)
o Includes main entrance, doorways, halls, etc.
o May include bays or mock-ups for vehicles or other platforms
32
Test Tools List and Description
1. Best shown in a table with the tool
title, description, and vendor
information.
2. Identifies by name each test tool
that integrates with the subsystem
components in the SIL.
3. Aligns with the physical block
diagram.
4. Provides all the information required
by the RFP for each tool.
5. If the tool is COTS, be prepared to
provide a vendor name and part
number.
6. Performance Parameters column is
only populated when there are specific
quantifiable performance attributes
required of the tool.
33
Other Test Assets
▪ These are test assets that do not interface with the system in the SIL but
are critical to the test approach.
▪ Include these in a separate table called “Other Test Assets.”
▪ Examples may be environmental chambers that are outsourced or mock-ups
that are not part of the SIL.
34
In Summary
1
2
3
▪ The SIL Design determines
its capabilities and uses.
▪ The Systems Engineer
includes SIL requirements
in the Test Plan.
▪ System Integration Lab
Engineers require very
deep technical experience.
o A Test Facility may have
multiple labs, each with a
specific capability.
o SILs may be designed to
support specific programs
or capabilities or may have
multiple key uses for
different programs and
projects.
o Intended use, Physical
block diagram and layout,
test tools and resources.
o Even entry level
engineers are extremely
well-qualified.
35
References (1)
▪ Boeing's 777 Systems Integration Lab, October 2000, IEEE Instrumentation and Measurement
Magazine 3(3):13 – 18. DOI:10.1109/5289.863905, IEEE Xplore
▪ Edwards AFB 772nd Test Squadron, 2024.
▪ IEEE EMC Resource Center, 2024, Home | EMC Society (EMC)
▪ MIL Standard 464, ELECTROMAGNETIC ENVIRONMENTAL EFFECTS REQUIREMENTS FOR
SYSTEMS, 2010, MIL-STD-464D | www.dau.edu
36
References (2)
▪ MIL Standard 461G, DEPARTMENT OF DEFENSE INTERFACE STANDARD, REQUIREMENTS FOR
THE CONTROL OF ELECTROMAGNETIC INTERFERENCE CHARACTERISTICS OF SUBSYSTEMS
AND EQUIPMENT, 2007, MIL-STD-461G | www.dau.edu
▪ MIL Standard 810 Rev H, ENVIRONMENTAL ENGINEERING CONSIDERATIONS AND
LABORATORY TESTS, 2022.
▪ NASA RPS SIL, Housed in the Glen Research Center (GRC) Power Systems Facility (PSF), 2024.
▪ National Telecommunications and Information Administration (NTIA) - US Frequency Allocation
Frequency Chart, January 2016– January_2016_spectrum_wall_chart_0.pdf
▪ Sandia National Lab, Electromagnetics – Research, 2024
37
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
TITLE
3 SI Capabilities
FILENAME
Engineering Development
SIL Capabilities
AUTHOR
J.Ziarko
VERSION
December 2024, V1.00
System Test and Evaluation
Informal Testing – Integration and Test
Engineering Development
Concept
Development
Engineering
Development
Illustration adapted from Kossiakoff, A. & Sweet, W. N. Systems Engineering: Principles and Practices, Wiley, 2003.
Post
Development
2
T&E During Engineering Development
Create
detailed test
plans and
procedures
Prepare and use
System
Integration Lab
(SIL)
Conduct
Conduct
Informal Testing Formal Testing
Integration and Test
Qualification Testing
▪ Implement
testing
components
and their units
in each
subsystem.
▪ Use lab to conduct
Informal and Formal
testing activities.
▪ Conducted by
developer (supplier
or contractor).
▪ Developer “run for
the record” with
customer.
▪ Includes facilities and
equipment.
▪ End result is a fully
integrated, stresstested mature
system.
▪ Developmental Test
and Evaluation
(DT&E).
▪ Desired outcome is
system acceptance.
3
Test and Evaluation Phases
The phases create a hierarchy from lower-level component test to operational testing
Operational Test and Evaluation
System Integration and Test (Informal Testing)
Subsystem Integration and Test
Component Test
Component Test
Time
Developmental Test and Evaluation (Formal Testing)
Subsystem Integration and Test
Component Test
Component Test
Component Test
4
Details by Phase and Integration Level
Process/Phase
Integration
Level
Operational test and
evaluation
System
Development test
and evaluation
System
System integration
and test
Subsystem
integration and test
Component test
System
Subsystem
Component
Environment
Objective
Real operational
environment
Simulated
operational
environment
Integration facility
(SIL)
Integration facility
(SIL)
Component Test
equipment
Demonstrated operational
performance
Demonstrated compliance
with operational
requirements
Fully integrated system
Fully integrated subsystem
Verified component
performance
Excerpt From: Alexander Kossiakoff, William N. Sweet, Sam Seymour & Steven M. Biemer.
“Systems Engineering Principles and Practice”, 2011
5
Responsibilities and Formalities
Process/Phase
Performed By
Formality
Definition
Operational test and
evaluation
Customer or customer
agent
Formal
Development test and
evaluation
Prime Contractor
Formal
System integration and
test
Prime Contractor
Formal Testing
Customer has approval
rights for all plans and
results and determines
sufficiency.
Subsystem integration and
test
Subsystem Contractor
Component test
Component contractor
Informal
Informal and Formal
(Formal when
subcontractor "sells off"
to prime contractor.)
Informal Testing
Contractor determines
sufficiency. No formal
customer involvement.
6
Integration and Test Objectives
▪ Create confidence that the system performs as intended and meets
requirements.
▪ Uncover potential design deficiencies.
▪ Acquire sufficient test data to identify sources of these deficiencies.
▪ Identify design faults which later result in program interruptions or delays.
o The approach must presuppose that defects will be encountered rather
than a lesser rigor approach that presupposes success.
Excerpt From: Alexander Kossiakoff, William N. Sweet, Sam Seymour & Steven M. Biemer.
“Systems Engineering Principles and Practice”, 2011
7
System Integration and Test Approach
▪ Conducted by the development contractor (supplier).
▪ System is exercised as it is integrated (build and test).
o I&T test cases trace to requirements verified by Demonstration or Test.
o Some Operational Tests occur for demonstrations in a controlled environment.
o Subsystem/system defects identified, and corrected.
o Risks are identified and mitigated.
▪ Test results from Informal testing prepare for the Test Readiness
Review (TRR) prior to Formal Testing.
Dr. Janice Ziarko
SE 101 – Module 5 V1.0 - September 2022
8
Integration and Test Realities
▪ No matter how thoroughly components are tested, STUFF happens.
o Unforeseen incompatibilities exist that were not found in lower-level
testing do not reveal themselves until the system elements are brought
together.
▪ Such discrepancies usually require changes/rework in some
components before the integrated system works properly.
o Also frequently require corresponding alterations in test equipment or
procedures.
▪ Integration test facilities are essential (System Integration Lab).
▪ Be flexible.
Excerpt From: Alexander Kossiakoff, William N. Sweet, Sam Seymour & Steven M. Biemer.
“Systems Engineering Principles and Practice”, 2011
9
Integration in the Life Cycle
Source: Systems Engineering – Principles and Practices - Kossiiakoff et al Wiley 2011 Ch. 13, from Figures13.1 and 13.2
10
Integration Sequence or Flow
▪ Avoid the BIG BANG approach!!!
o Incrementally integrate subsystems into the system.
o Reduce risk by integrating subsystem capability BEFORE subsystems are
completely developed.
▪ Integration typically proceeds in an orderly, stepwise manner
o Add one or two system elements at a time, and test to demonstrate proper
operation before proceeding to the next step.
o This maintains control of the process and simplifies diagnosis of discrepancies.
▪ Test equipment or tools simulate relevant functions of missing parts of the
system.
o Experience has repeatedly demonstrated that the provision of this capability is, in
the long run, quite cost-effective.
11
Fundamental Concept of System Integration
▪ Consists of taking developed subsystems and components from
internal and external development groups.
o “Plugging” them together to yield a functional system.
o Testing the integrated entity to establish confidence that it “works” (meets
requirements).
12
Coupling and Cohesion
▪ Successful integration of a complex system is easier when:
o Partitioning into subsystems exhibits low coupling (limited
complexity of interfaces).
o Subsystems display high cohesion (well-defined singular purpose
or functionality).
13
Test Cases are Connected to each Build
Traceability: Requirements map to Test cases.
14
Test Cases are Called out in the VCRM
▪ Baselined System/Subsystem Requirements (SRS) form the VCRM.
o Each requirement has a unique ID, the source for each requirement is listed.
▪ The build in the integration sequence is listed, along with Test Case IDs.
o Each requirement is traced to a test case.
o Each test case is traced to one or more requirements.
Require
-ment
Number
1
SRS
Section
Requirement
Verification
Method
Verification
Level
Applicable
I&T Build
I&T
Test Case ID
Qualification
Test Case ID
3.1.1
The SAR Base
Station shall
operate in three
modes: Idle,
Operational, and
Training.
D
Subsystem
Base-3
IT-3.1
QD_002
15
Prioritizing Tests
▪
▪
▪
Rarely have the “ideal” time and schedule for testing.
o
Often conducted under considerable stress because of time and cost
constraints.
o
Testing has finite bounds on time, expenses, capabilities.
Test planning requires prioritization of the schedule and equipment.
o
Allocate the available time and resources in the most efficient manner.
o
Test Prioritization is responsibility of systems engineering.
A major objective of Test and evaluation is to reduce risk
▪
Testing needs to be “Risk Based”.
o
Requires a careful balancing of a wide range of risks based on a comparative
judgment of possible outcomes in terms of performance, schedule, and cost.
16
Test Planning Considerations
Define
▪ Define test program
objectives to test
subsystems/system
against requirements.
o What features and
parameters must
be evaluated,
under what
conditions?
Review
Identify
Define
Identify
▪ Review
components
selected, design
issues involved in
the selection,
development test
results, and
resolution of
design issues.
▪ Identify interfaces
and interactions
between internal
components/
subsystems and
the system in its
environment.
▪ Define the
appropriate test
configurations for
testing the
components.
▪ Identify the test
inputs to stimulate
components and
anticipated test
outputs.
o What are the upper
and lower limits
and tolerances of
items under test?
17
Risk Based Test Completion Criteria
▪ Test completion criteria needs to based on an understanding of the
degree of “goodness” required of the system.
o What is “good enough? What is the risk if we field the system?
▪ Risk can be measured in terms potential of loss of life, loss of revenue,
loss of image and customer confidence, etc.
o Requires some form of “quantified reliability” as a basis and target.
o Requires ability to track and assess progress in achieving the target
goodness throughout the test program.
o The ability to quantify goodness is critical to the system engineer and test
leads in explaining status to management and customer.
18
Test Configuration Essentials
▪ A typical test configuration consists of:
1.
The system element (component or subsystem) under test
2.
A physical or computer model of the component or subsystem
3.
An input generator that provides test stimuli
4.
An output analyzer that measures element test responses
5.
Control and performance analysis units
6.
Monitored test points for fault diagnosis
7.
Subsystem integration builds, prior test results of components and parts
▪ Failures may be due to equipment, specifications, test plans, training, etc.
▪ Integration test facilities are prepared (Systems Integration Lab).
Source: Systems Engineering – Principles and Practices - Kossiiakoff et al Wiley 2011 Ch. 13
19
Defining Test Configurations
▪ Choice of specific test configurations in a test case is a complex balancing of risks.
o Ideal configuration - all components in the context of the total system in its
operating environment.
• Requires a prototype of the entire system in its environment, very costly.
o Minimum configuration - component with simple simulations of all its
interfacing elements.
• Less costly, but is it “good enough”?
o More practical middle ground - incorporate component under test in a
prototype subsystem
• Within a simulation of the remainder of the system and relevant part of the
operating environment.
20
Typical Test Configuration Overview
21
Typical Test Configuration – Test Control Unit
Test Control Unit interprets test manager commands into system inputs
22
Typical Test Configuration – Input Generator
Input Generator converts test commands into exact replicas, functionally and
physically, of the inputs that the system element is expected to receive.
23
Typical Test Configuration – Output Analyzer
Output Analyzer converts any outputs for the item under test that are not already
generated as quantitative physical measures into quantified test data
24
Typical Test Configuration – Input Generator
Element Model has the function of reproducing very precisely the component or subsystem
under test is expected to produce to each input, according to its performance specifications.
25
Typical Test Configuration – Performance Comparator
Performance Comparator matches the measured system element
outputs with the expected outputs from the element model
26
Typical Test Configuration – Test Conductor
Test Conductor provides test supervision by a test engineer supported by a control console
27
Interim Submittal – Integration Approach
▪
Define the sequence and inter-dependencies of the Builds.
o Define contents (capabilities and performance) of each Build.
o Depict the dependencies and sequence of the Builds in a Build Dependency Network.
Build Number
Build Name
Capabilities, Functions, and Test
Activities
Enter a unique
number to identify
each build
Create a
descriptive title for
each build
Describe the contents of each build in a
separate row.
• Describe in detail Build capabilities and
functions
• Summarize test activities in the Build
• List requirements verified by T or D in
each Build
28
Integration Sequence – Build Dependency Network
29
Interim Submittal - Integration Test Cases
▪ For each build defined in the Integration Approach submittal.
o Define the sequence of test cases within each Build showing the
dependencies of the individual test cases.
o Describe each Test Case in an Integration Test Case table.
• Draft at least 5 test cases for the first interim submittal.
• Final product is ~ 35 – 60 test cases, 3-5 test cases per Build.
30
Sample Test Case Dependency Network
31
Test Case Descriptions
▪ Each test case includes:
o Test Case ID, Applicable Build No. and Title for the test Case
o Test Case Title and Objective
o Test Environment/Set Up, Test Inputs, Test Outputs
o Data Reduction/Analysis (Requirements verified by Test, n/a for Demonstration)
o Pass/fail criteria for EACH requirement (Quantifiable where possible)
o Requirement ID (requirements verified by Test or Demonstration only)
Test
Case
ID
Build
No. and
Title
Test Test
Test
Test
Case Objective Environment/ Inputs
Title
Set UP
Test
Outputs
Data
Reduction/
Analysis
Pass/Fail
Criteria
REQ
ID
1
2
3
32
Sample Test Case Flow
33
In Summary
1.
Test and Evaluation can be partitioned into
3 classes of events or activities, each with
distinct and unique objectives.
2.
The primary objective of I&T is to make a
system “work”.
3.
Each subsystem is incrementally
integrated into the whole system, as the
subsystem capabilities are developed, to
minimize risk.
4.
Avoid the “Big Bang” approach which
increases risk. Integration and Test is a
systematic, incremental process to realize
the system.
5.
Always presuppose that stuff happens,
defects will be encountered.
34
References
▪ Kossiakoff, Alexander, W. Sweet, S. Seymour, S. Biemer (2011).
Systems Engineering – Principles and Practices (2 edition, Ch. 7-13).
nd
New York: J. Wiley. (1st edition published 2003).
▪ National Aeronautics and Space Administration (2007).
NASA Systems Engineering Handbook (Rev 2, NASA SP-2016-6105).
Washington D.C.: NASA Headquarters, U.S. Government
35
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
TITLE
3 InformalTesting
FILENAME
Engineering Development
Informal Testing
AUTHOR
J.Ziarko
VERSION
December 2024, V1.00
System Test and Evaluation
Formal Testing – System Qualification Testing
Engineering Development
Concept
Development
Engineering
Development
Illustration adapted from Kossiakoff, A. & Sweet, W. N. Systems Engineering: Principles and Practices, Wiley, 2003.
Post
Development
2
T&E During Engineering Development
Create
detailed test
plans and
procedures
Prepare and use
System
Integration Lab
(SIL)
Conduct
Conduct
Informal Testing Formal Testing
Integration and Test
Qualification Testing
▪ Implement
testing
components
and their units
in each
subsystem.
▪ Use lab to conduct
Informal and Formal
testing activities.
▪ Conducted by
developer (supplier
or contractor).
▪ Developer “run for
the record” with
customer.
▪ Includes facilities and
equipment.
▪ End result is a fully
integrated, stresstested mature
system.
▪ Developmental Test
and Evaluation
(DT&E).
▪ Desired outcome is
system acceptance.
3
CONCEPT DEVELOPMENT
POST DEVELOPMENT
SE Life Cycle V
TEST APPROACH
Systems Engineering
Domain
VERIFY CONCEPT
Engineering
Design Domain
VALIDATE CONCEPT
DETAILED TEST PLANS &
PROCEDURES
VERIFY UNITS &
COMPONENTS
ENGINEERING DEVELOPMENT
source: US DOT Federal Highways Administration, Designing for Transportation Management and Operations: A Primer, Ch. 2.5,. 2023.
Test and Evaluation Phases
The phases create a hierarchy from lower level component test to operational testing
Operational Test and Evaluation
System Integration and Test (Informal Testing)
Subsystem Integration and Test
Component Test
Component Test
Time
Developmental Test and Evaluation (Formal Testing)
Subsystem Integration and Test
Component Test
Component Test
Component Test
5
Qualification Testing Strategies
▪
A final test is conducted to prove the system meets expectations
before the system is accepted and paid for by the customer.
o
Qualification usually applies to a single system, whereas Acceptance test
usually applies to each system or subsystem/component being tested.
o
Dry runs are conducted prior to the “run for the record” with the customer.
o
Tests are witnessed and approved by the customer.
o
A Test Report is created, a formal document which records the test execution
results as a basis for (sub)system acceptance or required rework.
▪ More political and contractual in nature than technical.
o
Defects and problems should have been found during Integration and Test.
o
In the ideal sense, NOT a defect, debugging, problem identification activity.
6
Key Aspects of Qualification Testing
▪ Demonstrates completeness of a system (or subsystem).
o Performed by development contractor (supplier), but not the team who developed
the system, following integration and test.
o May be performed on a single component (hardware/software) or subsystem
stand-alone, or on integrated components and subsystems.
o For a component or subsystem, can be a gate approved for system integration.
o Components are under formal configuration management.
o Rigorous test plans and procedures are reviewed and approved by the Acquirer.
o Extensive dry runs are performed prior to the "run for the record“.
7
Qualification Testing - Verification and Validation
▪ Verifies requirements are met in the system being delivered to the
customer, based on System Requirements Specification (SRS).
o Build the system right.
▪ Validates the system performs as intended under highly controlled
conditions to meet stakeholder expectations.
o Build the right system
8
Requirements Traceability in Formal Testing
Requirements
Design
Document
Requirements
Specification
Requirements
Test Plan
and
Procedures
Traceability of requirements
to design implementation of
those requirements
Traceability of requirements
to specific test cases which
verify each requirement
9
Each Requirement has ONE Method of Verification
▪ Four primary Methods of Verification are:
o Inspection and Analysis (Static, System is NOT operating)
o Demonstration and Test (Dynamic, System IS operating)
▪ If there are multiple methods of verification documented for ONE
requirement, break it into multiple requirements.
o This makes no sense – what if the requirement passes one method and fails the
others? Can it be verified?
o
If a requirement truly needs independent multiple methods, consider a
hierarchy to methods of verification based on one method to one requirement.
10
Static Methods of Verification
Inspection
Visual examination of a realized system or
end product used to verify physical design
features or a manufacturer identification
▪ EXAMPLE Requirement: The server
subsystem shall be Commercial-Off-TheShelf (COTS).
▪ VERIFICATION: If the server subsystem is
COTS then each component will be
available to the general public and have a
catalog parts number. Provide vendor
documentation to show to the customer to
verify by inspection.
Analysis
Analytical techniques (modeling and
simulation) used to predict the suitability of a
design to meet expectations.
▪ EXAMPLE Requirement: The system shall
have no more than one critical failure in
3000 hours of operation.
▪ VERIFICATION: Hours needed to collect
data would be prohibitive. Analysis assesses
failure rate of components based on actual
or projected data of similar components,
and then statistically combines component
failure rates to derive a system failure rate.
source: NASA Systems Engineering Handbook, December 2007 (NASA/SP-2007-6105 Rev1)
11
Dynamic Methods of Verification
Demonstration
Performance capability assessed based on
observation (a go/no go decision without
detailed data gathering) and use of physical
models or mock-ups.
▪ EXAMPLE Requirement: The operator
workstation shall display the routes of
vehicles in transit.
▪ VERIFICATION: Test case defines test
environment and set-up, how system will be
stimulated (data inputs), expected outcome
(data outputs), and pass/fail criteria for each
requirement. N/A for data reduction/analysis.
To verify, observe routes on the display.
Test
Gathers and produces data at discrete points for
each specified requirement under controlled
conditions, the most resource-intensive
verification technique.
▪ EXAMPLE Requirement: The operator
workstation shall continuously track a
selected object within an accuracy of 0.1
miles anywhere in the CONUS.
▪ VERIFICATION: Test case defines test
environment and set-up, how system will be
stimulated (data inputs), expected outcome
(data outputs), analysis required on results,
and pass/fail criteria for each requirement.
12
Methods of Verification - Relative Cost
Test
Typically
More
Costly
Demonstration
Analysis
Inspection
Test by its very nature
will most likely
encompass
aspects of
Demonstration,
as well as Analysis
and Inspection
13
Why not just reuse the test cases from integration
and test for qualification testing?
▪ The reason is the difference in the starting point of the two phases.
o Integration and test assumes nothing works as you integrate components, and
defects were found and addressed.
• Inspection and analysis test cases can be conducted in parallel to Integration
and Test activities, and results can be reported to the customer.
o Qualification testing has confidence the system works from Integration and test
phase, focus is to convince the customer the system works as intended.
• Test and Demonstration test cases take on a more operational flavor and
incorporate multiple requirements grouped by scenarios.
14
Handling Inspection and Analysis Requirements
▪
Perform verification of inspection and analysis requirements as soon as support
data or artifacts are available.
o
▪
No need to wait until start of qualification test phase.
Data and artifacts to support Inspection and Analysis requirements verification can
be collected, documented, and analyzed throughout development.
o
Collected data and artifacts require customer review and approval to verify the
requirement.
o
Get customer sign-off on requirements as data is available.
15
Typical Formal Testing Flow of Events
Test Plan Development
Test Procedure Development
Dry Runs
TRR
Run for Record
Test Report
Results
Analysis
FCA
Test Results Evaluation (TRE)
PCA
Inspection and Analysis Requirements Verification
Note: FCE – functional configuration audit, PCA physical configuration audit
16
Qualification Testing Process
Integration
Phase
Refine
procedures
Dry
Runs
Test
Plans
and
Procedures
Qualification Phase
Acquirer
witness
Final
Dry
Run
Test Tools
(simulators, data generators,
data reduction, etc.)
Acquirer
witness
Test
Readiness
Review
Run
For
Record
Test results, other documentation
Software/Hardware configuration
Test
Results
Evaluation
Test
Report
17
Dry Runs
▪
Successful test execution requires that the portion of the system tested “works” and
that the test procedures, test tools and test data support the verification.
▪
Test plans and procedures must be rigorously executed and debugged in extensive
dry runs prior to customer witnessed run for record.
▪
Dry runs verify that not only the system/subsystem under test works, but test
procedures, test tools/SIL and test data are also ready for customer witness.
▪
Final dry run must be on the system and test configuration to be used during the
“run for record.”
▪
Focus is requirements verified by Test or Demonstration.
18
Test Readiness Review (TRR)
▪
Conducted immediately before formal test execution.
▪
Evaluates earlier testing to determine if system is ready for Formal Testing.
▪
Reviews results of Qualification Testing dry runs.
▪
Defines roles and responsibilities of test team.
▪
Defines the system/software configuration.
▪
Reports how all test tools (software and hardware) have been validated.
▪
Provides a road map of test conduct schedule.
▪
Defines success criteria for tests and re-tests.
19
Run for the Record
▪
Customer witnesses and approves results.
▪
Conducted strictly to procedures.
o Step by step execution without deviation.
o No glitches or failures are expected.
▪
Focus is requirements verified by Test or Demonstration.
20
Test Results and Test Report
▪
Off-line evaluation of test results can be performed when data is available.
▪
Analysis process is defined in the Test Plan.
▪
Results feed into the Test Report.
21
Test Results Evaluation (TRE)
▪
Provides closure to the Formal Qualification Testing phase.
o Conducted as immediately as possible after completion of test execution.
o Objective is that all parties who witnessed the test execution share a common
understanding of the results outcome before everyone goes home.
o Reviews overall “run for record” test activities.
o Reviews any problem reports written during tests for criticality and future resolution.
o Compares success of testing against established success criteria.
o Defines next step – failure, conditional success, or unconditional success.
Note: the TRE is not a commonly used or well documented review, but is very beneficial.
22
Two Audits after DT&E and before Production
Verifies system is complete and ready for production, transitioning from
the Engineering Development to Post-Development life cycle phase.
Functional Configuration Audit (FCA)
Physical Configuration Audit (PCA)
▪ Examines functional characteristics of the
configured product.
• Examines "as-built" configuration item against its
technical documentation to verify product baseline.
▪ Verifies the product has met the
requirements specified in its Functional
Baseline.
• Verifies design documentation matches the Configuration
Item (CI) specified in contract.
▪ Approved at Preliminary Design Review
(PDR) and Critical Design Review (CDR).
▪ May also be conducted concurrently with
the System Verification Review (SVR).
• Confirms manufacturing processes, quality control
system, measurement/test equipment, and training are
adequately planned, tracked, and controlled.
• Validates supporting processes used by contractor in the
production of the item, and other elements of the item
impacted or redesigned after SVR.
23
Logical Flow and Maturity of Test Planning
High Level
Test
Concepts
and Test
Classes
Detailed
Specific Test
Cases
Detailed
Test
Procedures
Test facilities, test schedules, test support, etc.
24
Detailed Test Plans and Procedures
▪ Typically, a detailed test plan includes procedures to perform each test.
▪ Hold off on development of test procedures until test cases has been
reviewed and approved by the customer.
▪ Due to the size of test procedures and potential to be a separate submittal,
a separate volume such as an appendix is typically used.
25
Detailed Test Plan Objectives
▪ Document details of planned testing to support review, critique, and re-use.
▪ Provides the following information:
▪
Test Concept
▪
Objectives and Requirements to be Satisfied
▪
Test Methods
▪
Responsible Activities Associated with Testing
▪
Measures Required and Recording Procedures to be Used
▪
Test Procedure is step-by-step detailed description of all actions to execute a test case
26
Data Item Description (DID)
▪
A guidance document which describes the format and content required of a contract
deliverable document.
o
DIDs are available for all types of documentation used in system development.
o
There are frequently multiple DIDs for the same document type.
o
The DID to be used for each deliverable document is specified in the Request
for Proposal (RFP) from the customer.
o
DIDs are “tailorable”, meaning they may be modified to suit a specific
application or program with approval of the customer.
27
Interim Submittal
Qualification Test Approach and Test Cases
▪ Review lecture content, the Group Project artifacts, and Group Project
examples from Office Hours.
▪ Complete a draft description of your qualification test approach.
o Describe the overall approach and plan for conducting qualification testing including
prerequisite conditions, overall plan and sequence, associated reviews and audits, etc.
▪ For draft test cases, complete examples for each method of verification.
o I and A requirements – 2 of each (use the template)
o D and T requirements – 3 to 5 total (use the template)
Note: For the Group Project final report, include ALL test cases for the entire system.
Typically about 30 to 60 test cases.
28
Qualification Test Case Templates
Requirements Verified by Inspection and Analysis
Test
Case
ID
Requirement
ID
Requirement
Inspected
(text)
Component
Inspected
Test
Case
ID
Requirement
ID
Requirement Analysis
Analyzed
Performed
(text)
1
2
3
1
2
3
Requirements Verified by Test and Demonstration
Test
Case
ID
Build
No. and
Title
Test Test
Test
Test
Case Objective Environment/ Inputs
Title
Set UP
Test
Outputs
Data
Reduction/
Analysis
Pass/Fail
Criteria
REQ
ID
1
2
3
29
In Summary
▪ Qualification Testing is a Formal Testing phase.
o It is not a debugging, problem identification phase, there is confidence the
system will work based on Integration and Test.
o Formal test plans require more rigor and detail than Informal test plans.
▪ There are four methods of verification for requirements.
o Inspection and Analysis requirements may be addressed as early as possible in
the development cycle.
o Test and Demonstration requirements are verified by exercising the system.
▪ Test Reports capture the details of all Test Results.
o Provides a basis for (sub)system acceptance or future work required.
o Documents evidence needed for a customer to accept the system and exit DT&E.
30
References
▪ Kossiakoff, Alexander, W. Sweet, S. Seymour, S. Biemer (2011).
Systems Engineering – Principles and Practices (2 edition, Ch. 10-13).
New York: J. Wiley. (1st edition published 2003).
nd
▪ National Aeronautics and Space Administration (2007).
NASA Systems Engineering Handbook (Rev 2, NASA SP-2016-6105).
Washington D.C.: NASA Headquarters, U.S. Government
31
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
TITLE
3 FormalTesting
FILENAME
Engineering Development
Formal Testing
AUTHOR
J.Ziarko
VERSION
December 2024, V1.00
System Test and Evaluation
Methods and Tools Applied to T&E
Topics
• SE Practices, Methods and Tools
• Measurement, Instrumentation, and
Data
2
SE Practices, Methods and Tools
3
SE Practices are tied to Methods and Tools
▪ The Life Cycle is a methodology and is tied with the “scientific” 4-step SE method.
o Methods of verification, traceability of requirements throughout the life cycle
o Decomposition, Cohesion and Coupling
•
Partitioning into subsystems exhibits low coupling (limited complexity of interfaces).
•
Subsystems have high cohesion or tight binding (well-defined singular purpose or
functionality).
o MBSE as a tools is primarily used to define the system architecture during development.
▪ SE Processes and Quality management (IEEE/ISO/IEC 15288 standard)
o Product Baselines and Configuration Management
o Risk identification and mitigation (IF…THEN statements, Risk Cube)
o Quality Management and Continuous Process Improvement
4
Systems Engineering Method - Traceability
source: Kossiakoff, A. & Sweet, W. N., Systems Engineering: Principles and Practices, Wiley, 2003.
5
CONCEPT DEVELOPMENT
POST DEVELOPMENT
SE Life Cycle V
TEST APPROACH
Systems Engineering
Domain
VERIFY CONCEPT
Engineering
Design Domain
VALIDATE CONCEPT
DETAILED TEST PLANS &
PROCEDURES
VERIFY UNITS &
COMPONENTS
ENGINEERING DEVELOPMENT
source: US DOT Federal Highways Administration, Designing for Transportation Management and Operations: A Primer, Ch. 2.5,. 2023.
Methods for Repeatable Test Results
Controlled Test
onditions
Test Methods and
Procedures
Configuration
Control
▪ Indoor/outdoor weather
conditions documented
(land, sea, air, space)
▪ Methods of verification
(I, A, D, T) with
pass/fail criteria for
each requirement
▪ For system items and
documents under test
▪ Indoor System Integration
Lab (SIL), leased labs, or
environmental test
chambers
▪ Outdoor field test
facilities, Test track, Test
Range, Proving grounds
▪ Written test
procedures including
dry runs to confirm
procedures are
feasible.
▪ HW/SW configuration
and test support items
needed to support
configurations.
▪ Document information
to describe what was
tested and to repeat
test if necessary.
8
Example - Methods to Organize Formal Testing
1
TRR
▪ Test Readiness Review
(TRR) conducted to
confirm test team is
ready for each test
activity and event.
▪ Confirm agreements
are in place to support
evaluation of defined
pass/fail criteria for
each requirement.
2
Test Teams
▪ Assign Test Teams for each
test activity or event.
▪ Test Lead and team conduct
tests for requirements
verified by Test (T) or
Demonstration (D)
3
Test Reports
▪ Document “as run” test
conditions and
procedures
▪ Record data and
observations to support
pass/fail criteria
evaluation of
requirements
▪ Test Lead and support for
requirements verified by
Inspection (I) or Analysis (A) ▪ Deliver fact-based
results and conclusions
9
Typical Tools for Test and Evaluation
Data Recorders and Data
Collection Tools
MBSE Tools
SysML, Cameo, CORE, DOORS
System and External
Interface Simulators
MATLAB, etc.
Statistical Analysis and
Design of Experiments
(DOE)
System T&E
Tools
Data Analysis and Data
Reduction Tools
Calibrated Test Equipment
10
A T&E Engineer Needs to Rely on Math Experts
▪
▪
▪ Probability, Statistics
▪ Topology
▪
▪ Average, mean, variance
▪
▪ Discrete number
▪
▪ Random number variable
▪
▪ Deterministic process
▪
▪ Stochastic process
▪
▪ Linear systems
▪
▪ Feedback control systems
▪
▪ Kalman filtering
▪
▪ Probability distribution function
▪
▪ Normal probability function
▪
▪ Gaussian
▪
▪ Binomial probability function
▪
▪
Calculus, Geometry
▪
Weibull probability function▪
▪
Markov process
▪
Decision tree
▪
Combinations
▪
Permutations
▪
▪
Convolution
▪
Six Sigma
▪
Conditional probability
▪
Bayes Theorem
▪
Central Limit Theorem
▪
Sample size
▪
▪
Sample mean
Bernoulli trials
Sample variance
Design of Experiments, DOE
Analysis of Variance, ANOVA
Hypothesis Testing
Pass/fail criteria
• Confidence level
False positive, False negative
Power
One-sided, Two-sided
• Repeatability
Degrees of freedom
Student-t test
• Bias error
Randomization
• Time tagging
Kaplan-Meier Curve
Bathtub Curve
MTBF
Reliability
Safety margin
• Uncertainty
• Production lot sampling
• Blocking
• 1-Factor
• 2-Factor
• Calibration error
• Uniform probability
distribution
• Bandwidth, aliasing
• Digital filters
• Analog filters
• Taguchi methods
Random sample
11
Example: Test Tools Required to Conduct Tests
SIL Test tools are listed
once form, fit, function
and support are known.
Connect tools as needed
for each test event.
12
Measurement, Instrumentation,
and Data
13
Measurement and Instrumentation are Complex
▪ System engineering complexity is often underestimated when planning
measurement and instrumentation capabilities.
o Expertise and support must be available for test planning, execution and reporting.
o Equipment (hardware/software) needs to be procured or developed to meet specific
needs for test activities.
▪ Covers a large scope that mirrors the entire system.
o Has a huge impact on generating useable, accurate test results.
o Planning includes accounting for measurement effects and errors.
o Plan for data collection, reduction, and analysis.
14
Example: Instrumentation Itself Can Create Errors
Analog/Digital Conversion: accuracy, time response, dynamic range, bandwidth, bits, DC offset, time alignment, time jitter, calibration
Measurement Sensor:
sensitivity, accuracy, time
response, dynamic range,
bandwidth, calibration
Data Analysis and Presentation: analysis software accuracy,
data display accuracy, post processing capabilities, real-time
data display, quick look data process capability
Digital Filtering &
Processing: sample rate, filter
bandwidth, delay, aliasing
Signal Transmission: channel
bit error rate, error correction
coding, signal dropout, latency,
time alignment
Data Recording: channel bit error rate, error correction
coding, signal dropout, latency, time alignment, dynamic
range
Measurement and Instrumentation Principles, Alan S Morris, 2001, Figure 1.2, page 9
15
Uncertainty and Risk
All measurements
have uncertainty
16
Data Reduction Issues
The Right Data and the Right Amount of Data
▪ Raw data by itself doesn’t always help much and can be a burden.
o Testers grapple with extracting actionable information from raw data.
o Too much data can obscure or make it hard to make good decisions.
▪ Due to the tremendous amount of data collected by measurement systems, it is
impossible to act on all of it (even if actionable information is clear).
o People sifting through the data to create actionable information is increasingly
unfeasible given the amount of data being generated and collected on servers.
o Automation and machine intelligence used for data processing and reduction is a
tool to manage data and remove subjectivity (Human in the Loop).
17
Conclusion
Data Collection, Reduction and Analysis
 Considered when engineering TOTAL system to answer a key question:
Is the system effective, suitable, survivable?
o
o
o
o
o
Supports other Test Planning documents.
Identifies data to be collected and analyzed, as well as logs and forms used.
Uses a Data Collection and Distribution Matrix (DCDM).
Data processing, output, archiving and retrieval are significant activities.
Identifies Organizations and Responsibilities, approved by Test Director.
 Tools and methods involve a considerable amount of systems engineering.
o
o
o
Handling data involves time, resources, and cost.
Improper Data Collection, Reduction and Analysis can result in failure of programs.
A Security Classification Guide for data security and labeling is important.
18
Data Collection Planning
Planning for Data Collection
and Data Reduction
▪ Are there built in test points? How is data being collected or generated?
o Test Instrumentation
o Environmental Data Collection
o Organic Data Collection
o Telemetry
o Reliability, Maintainability, and Availability Data – The hardest
o Test Observations (audio, video, screen shots, debriefs)
o Old Fashioned Way (Pen and Pencil)
19
Data Collection Systems
Stand Alone
Systems
Installed
Pickoffs
Signal monitors , Optical tracking
Designed as part of System
Time-Space-Position Information
(TSPI)
Interface Data - was it the right
signal, right timing?
Environmental Data Collection
System Status - Did subsystems
perform as intended?
Unique
Installation
For a Specific Test
Inputs/Outputs are realtime data (e.g. an
instrumented reentry body
to measure accuracy)
Enables abnormal
performance fault location
20
Data Reduction
 Data Transformed into a corrected, ordered, and simplified form.
o Digital information is derived empirically or experimentally.
o Reduces huge amounts of data down to the meaningful parts used in analysis.
o When derived from instrument readings, may transform from analog to digital.
o When observations are discrete but the underlying phenomenon is continuous,
smoothing and interpolation are often needed.
▪ Often undertaken in the presence of reading or measurement errors, which
must be understood before a most likely value is determined.
▪ When data is in digital form, involves editing, scaling, encoding, sorting,
collating, and tabular summaries.
21
Six Basic Data Reduction Activities
Sorting
Editing
Arrange items systematically by
ordering (a sequence defined by
some criterion) or categorizing
(group items with similar
properties).
Review and adjust collected data
to control the quality, manual or
automated.
Scaling
Standardize range of
independent variables or
features of data.
Encoding
Convert data into format required for
information processing needs
(includes software program compiling
and execution, transmission, storage,
and compression/decompression).
Collating
Data Reduction
Activities
Assemble written information
into a standard order (e.g.
numerical or alphabetical
order,) or extensions and
combinations thereof
Tabular summaries
Organize data into tables, graphs
or charts so logical and statistical
conclusions can be derived from
collected measurements
22
Data Reduction
Data Reduction and Analysis Can be a Huge Effort
▪ Identify resources during planning, assure expertise needed is available.
▪ Use existing facilities or design/build a new capability.
▪ Consider presentation of data, coupled with summaries or narratives.
o Order by size or rearrange table rows/columns so patterns are easy to see.
o Round drastically to one, or at most two, effective digits (effective digits are
ones that vary in that part of the data).
o Use layout and labeling to guide the eye, averages for visual focus.
23
Types of Data
Program/Sponsor
Decision
Classification Level Can
Change as Test
Changes
Un-scaled data or data
used to Scale Raw Data
is Unclassified
Measure
Temperature,
Pressure, Relative
Humidity, etc.
Evaluate performance
in operational
environment or
scenarios
Scaled Data could be
Secret (TS, TS SCI)
E.g. Conditions that
Impact Radar and
Communications
Classified
Environmental
Internal, installed in
the system
The more complex
the system, the more
data needed
Data recorded for
retrieval and analysis
on hard drives or
removable media
Organic
Data collection of
measurements at
remote or
inaccessible points
with automatic
transmission to
receiving equipment
Useful for missiles,
spacecraft, hostile
environments
Telemetry
24
In Summary
SE Practices, Methods,
and Tools
Measurement,
Instrumentation, and Data
▪ The system development life cycle
itself is a key method in the practice of
system engineering,
▪ Measurement and instrumentation
mirror the system and can contribute
to errors.
▪ The 4-step SE process is applied at
each phase and tier of the system.
▪ Some instrumentation is available,
some need to be designed or built.
▪ SE processes in the international
standard (15288) are tailored to adapt
for engineering a specific system.
▪ Data collection, reduction, and
analysis are a significant effort.
▪ Using tools such as MBSE requires
knowledge about what is needed to
create a model of the system.
▪ Garbage in, garbage out applies to
data quality, and what data is used for
modeling and assessing a system.
25
References
▪ Morris, A. and Reza Langari, Measurement and
Instrumentation Theory and Practice, 3rd Ed.
Elsevier Inc. 2020.
▪ Morris, Alan S., Measurement and
Instrumentation Principles, Elsevier Inc. 2001.
▪ NASA Measurement Quality Assurance
Handbook, ANNEX 2, 3, 4.
NASA-HDBK-8739.19-2, 19-3, 19.4. 2018-03-1
o ANNEX 2 - Uncertainty Analysis Methodology
o ANNEX 3 - Uncertainty Analysis Methodology
o ANNEX 4 - Uncertainty Analysis / SPC
26
© The Johns Hopkins University 2024, All Rights Reserved.
645.769
Lecture
TITLE
3 Methods and Tools Applied to T&E
FILENAME
Methods and Tools Applied to T&E
AUTHOR
J.Ziarko
VERSION
December 2024, V1.00