Carnegie Mellon University Notice
This video and all related information and materials (“materials”) are owned by Carnegie Mellon
University. These materials are provided on an “as-is” “as available” basis without any warranties and
solely for your personal viewing and use.
You agree that Carnegie Mellon is not liable with respect to any materials received by you as a result of
viewing the video, or using referenced websites, and/or for any consequences or the use by you of
such materials.
By viewing, downloading, and/or using this video and related materials, you agree that you have read
and agree to our terms of use (www.sei.cmu.edu/legal/).
© 2015 Carnegie Mellon University.
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
1
Copyright 2015 Carnegie Mellon University
This material is based upon work funded and supported by the Department of Defense under Contract No. FA8721-05-C-0003 with
Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development
center.
Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not
necessarily reflect the views of the United States Department of Defense.
NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS
FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER
EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR
PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE
MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT,
TRADEMARK, OR COPYRIGHT INFRINGEMENT.
This material has been approved for public release and unlimited distribution except as restricted below.
This material may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without
requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software
Engineering Institute at permission@sei.cmu.edu.
Carnegie Mellon® is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.
DM-0002727
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
2
A Taxonomy of Testing Types
Software Engineering Institute
Carnegie Mellon University
Pittsburgh, PA 15213
Donald G. Firesmith
© 2015 Carnegie Mellon University
Topics
Relevant Testing Challenges
Goals of Presentation
What is Testing?
Presentation Scope
Testing Types
Conclusion
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
4
Relevant Testing Challenges
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
5
Relevant Testing Challenges
Many testers are only aware of a minority of types of testing, let alone know how to
perform them.
Test managers and developers are aware of even fewer testing types.
The test strategies, project test plans, and test sections of system/software development
plans tend to identify only a very small number of types of testing (e.g., unit, integration,
system, and acceptance testing).
• Not planned → Not performed
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
6
Goals of the Presentation
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
7
Goals of the Presentation
Make it clear that:
• There are many different types of testing.
• Testing is a complex discipline with its own technical jargon.
• There is a lot of overlap between different classes of testing types.
• Think multiple classification (object-oriented design)
or multiple inheritance (object-oriented programming).
Get you to take a look at your:
• Test strategies and test plans and ask yourselves “Are they sufficiently complete?”
• Testers and other testing stakeholders and ask yourselves “Do they need additional
training in testing types.
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
8
Polling Question 1
How many different types of testing do you typically use on a project?
O 1-5
O 6-10
O 11-15
O 16+
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
9
What is Testing?
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
10
What is Testing?
Testing
The execution of an Object Under Test (OUT) under specific preconditions with specific
stimuli so that its actual behavior can be compared with its expected or required behavior
• Preconditions: pretest mode, states, stored data, or external conditions
• Stimuli:
- Calls, commands, and messages (control flows)
- Data inputs (data flows)
- Trigger events such as state changes and temporal events
• Actual Behavior:
- During Test:
– Calls, commands, and messages (control flows)
– Data outputs (data flows)
- Postconditions: post-test mode, states, stored data, or external conditions
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
11
Presentation Scope
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
12
Presentation Scope
Test
In Scope
Legend
Verification and Validation (V&V) Methods
Quality Control (QC)
Static
Evaluation
Dynamic
Other
T&E
Analysis
Static
Analysis
Dynamic
Analysis
Demonstration
Desk
Checking
“Inspection”
Inspection
Certification
Review
Peer
Review
Reuse
Warantee
WalkThrough
Formal
Review
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
13
The Taxonomy of Testing Types
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
14
Types of Testing
A type of testing is:
• A specific way to perform testing
• A class or subclass of testing
• Much narrower in scope than a testing paradigm
There are relationships between the various types of testing.
Most testers know:
• A lot about a few types of testing
• A little about some additional types of testing
• Very little about a sizable number of testing types
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
15
Polling Question 2
Have you ever seen a taxonomy of testing types
(i.e., a hierarchical categorization of different ways to test)?
O Yes
O No
O Not Sure
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
16
Types of Testing
What-Based Test Types
(What Gets Tested)
Object Under Test (OUT)-Based Test Types
Domain-Based Test Types
Temporal-Order-Based Test Types
16 Categories of Testing Types
Answering the 5W+2H Questions:
• What?
• When?
• Where?
• Who?
• Why?
• How?
• How Well?
These supertypes are not disjoint
(think multiple inheritance)!
When-Based Test Types
(When Testing Occurs)
Lifecycle-Based Test Types
Phase-Based Test Types
Built-In-Test (BIT) Types
Where-Based Test Types
(Why Testing Occurs)
Organization-Location-Based Test Types
Physical-Location-Based Test Types
Collaboration-Based Test Types
Test
Types
Who-Based Test Types
(Who Does Testing)
Organization-Based Test Types
Role-Based Test Types
Why-Based Test Types
(Why Testing Occurs)
Driver-Based Test Types
Reason-Based Test Types
Automation-Based Test Types
How-Based Test Types
(How Testing is Performed
Level-of-Scripting-Based Test Types
Technique-Based Test Types
How-Well-Based Test Types
(Quality Verified)
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Quality-Characteristic-Based Test Types
17
Types of Testing
WHAT is Tested
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
18
What: by Object Under Test (OUT)
Testing
What-Based Testing
OUT-Based Testing
Model
Testing
Hardware
Testing
Software
Testing
System
Testing
Data Center
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Tool/Environment
Testing
19
What: by Object Under Test (OUT) – Model Testing
Testing
What-Based Testing
OUT-Based Testing
Model Testing
Requirements
Model Testing
Architecture
Model Testing
Design Model
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
20
What: by Object Under Test (OUT) – Hardware Testing
Testing
What-Based Testing
OUT-Based Testing
Hardware Testing
Continuity Testing
Hardware Stress
Testing
Highly Accelerated
Life Testing (HALT)
HW Qualification
Testing
Power-off Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
21
What: by Object Under Test (OUT) – Software Testing
Testing
What-Based Testing
OUT-Based Testing
Software Testing
SW Unit [Component]
Testing
SW Integration
Testing
SW Application
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
22
What: by Object Under Test (OUT) – System Testing
Testing
What-Based Testing
OUT-Based Testing
Hardware-in-the-Loop
(HIL) Testing
System Testing
Subsystem Testing
System Integration Testing
System Testing
SoS Integration Testing
SoS Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Human-in-the-Loop
(HIL) Testing
Processor-in-the-Loop
(PIL) Testing
Software-in-the-Loop
(SIL) Testing
23
What: by Object Under Test (OUT) – Data Center Testing
Testing
What-Based Testing
OUT-Based Testing
Data Center Testing
Configuration
Testing
Failover and Restore
Testing
Integrated System
Testing (IST)
Network Traffic
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
24
What: by Object Under Test (OUT) – Tool / Environment Testing
Testing
What-Based Testing
OUT-Based Testing
Tool/Environment Testing
Development Tool
Testing
Development
Environment
Testing
Test Tool Testing
Test Environment
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
25
What:
by Domain
Testing
What-Based Testing
Domain-Based Testing
Domain-Independent
Testing
Many Types of
DomainIndependent
Testing
Domain-Specific
Testing
Crash Testing
Exhaust Emissions Testing
Flight Testing
Live Fire Testing
Sea Trials
Wind Tunnel Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
26
Types of Testing
WHEN Testing Occurs
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
27
When: by Temporal Order
Testing
When-Based Testing
Order-Based Testing
Order-By-Direction
Testing
Bottom-Up
Testing
Top-Down
Testing
Outside-In
Testing
Order-By-Component
Testing
FeatureBased Testing
Layer-Based
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
SubsystemBased Testing
28
When: by Lifecycle
Testing
When-Based Testing
Lifecycle-Based Testing
Waterfall
Testing
At-The-End
Testing
Incremental
Testing
V-Model
Testing
Continuous
Testing (CT)
Agile
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
DevOps
Testing
29
When:
by Phase
Testing
When-Based Testing
Phase-Based Testing
Developmental
Testing (DT)
Acceptance
Testing (AT)
Many Types of
Developmental
Testing
Business Acceptance Testing (BAT)
Customer Acceptance Testing (CAT)
Contract(ual) Acceptance Testing (CAT)
Beware of
Synonyms and
Almost Synonyms!
Operational
Testing (OT)
Operational
Effectiveness Testing
Initial Operational
Testing (IOT)
Operational
Suitability Testing
Follow-on Operational
Testing (FOT)
Data Center
Testing
Factory Acceptance Testing (FAT)
Qualification
Operational Testing
(QOT)
Operational Acceptance Testing (OAT)
Production Acceptance Testing (PAT)
Regulation Acceptance Testing (RAT)
Site Acceptance Testing (SAT)
User Acceptance Testing (UAT)
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
30
When: by
BIT Execution Time
Testing
When-Based Testing
Built-In Testing (BIT)
Power-Up
Built-In Testing
(PupBIT)
Interrupt-driven
Built-In Testing
(IBIT)
Ongoing
Built-In Testing
(OBIT)
Self-Testing via
Assertion
Checking
User-initiated
Built-In Testing
(UBIT)
Periodic
Built-In Testing
(PBIT)
Self-Testing via
Prognostics and
Health Management
(PHM) Subsystem
Shutdown
Built-In Testing
(SBIT)
Self-Testing via
Heartbeat
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
31
Types of Testing
WHY Testing is Being Performed
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
32
Why: by Driver
Testing
Why-Based Testing
Driver-Based Testing
RegulatoryCompliance Testing
(tests compliance)
Validates
System Complies
with Laws or
Regulations
NeedsDriven Testing
(tests why)
Verifies
System Meets
Stakeholder
Needs
Requirements-Driven
Testing
(tests what + how-well)
ArchitectureDriven Testing
(tests how)
Verifies
System Meets
System Requirements
Verifies
System
Conforms to
Architecture
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Design-Driven
Testing
(tests how)
Verifies
System
Conforms to
Design
33
Why: by
Reason
Testing
Why-Based Testing
Reason-Based Testing
Smoke
Testing
Reuse
Testing
Initial
Testing
COTS
Testing
Open
Source
Testing
Legacy
Testing
Retesting
Regression
Testing
Partial
Regression
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Error
Seeding
Complete
Regression
Testing
34
Types of Testing
WHO Performs Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
35
Who: by
Collaboration
Testing
Who-Based Testing
Collaboration-Based Testing
Individual Testing
Group Testing
Buddy
Testing
Flash Mob
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Pair
Testing
36
Who: by
Organization
Testing
Who-Based Testing
Organization-Based Testing
Development
Organization
Testing
Prime
Contractor
Testing
Subcontractor
Testing
Acquisition
Organization
Testing
COTS
Vendor
Testing
Independent
Test
Organization
Testing
Operations
Organization
Testing
DT
Organization
Testing
OT
Organization
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
User
Organization
Testing
37
Polling Question 3
Who performs testing on your projects? Check all that apply.
□ Project-internal Testers
□ Independent Testers
□ Developers
□ Specialty Engineers (e.g., performance, reliability, safety, security, human factors)
□ Quality Engineers
□ Others
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
38
Who: by Role
Testing
Who-Based Testing
Role-Based Testing
Operator
Testing
Tester
Testing
Developer
Testing
Requirements
Engineer Testing
Alpha Tester
Testing
Architect Testing
Embeded
Tester
Testing
Programmer
Testing
Human Factors
Engineer Testing
Independent
Tester
Testing
Buddy
Testing
Pair
Testing
Database
Admin
Testing
Network
Admin
Testing
Sys Admin
Testing
Safety Engineer
Testing
User
Testing
Beta
Tester
Testing
User as
Tester
Testing
Closed
Beta
Testing
Open
Beta
Testing
Security Engineer
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
39
Types of Testing
WHERE Testing is Performed
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
40
Where: by Organizational Location
Testing
Where-Based Testing
Organizational-Location-Based Testing
Insourced
Testing
Outsourced
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
41
Where: by Physical Location
Testing
Where-Based Testing
Physical-Location-Based Testing
Cloud
Testing
Distributed
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Local
Testing
42
Types of Testing
HOW Testing is Performed
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
43
How: by Level of Automation
Testing
How-Based Testing
Level-of-Automation-Based Testing
Manual Testing
Automated Testing
- Test Script Generation
- Test Data Generation
- Test Script Execution
- Test Report Generation
Record-Playback
Testing
Script-Based
Testing
Data-Driven
Testing
Action-Keyword
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Model-Based
Testing
44
How: by Level of Scripting
Testing
How-Based Testing
Level-of-Scripting-Based Testing
Scripted Testing
Unscripted Testing
Exploratory Testing
Monkey Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
45
How: by Technique
Testing
How-Based Testing
Test-Technique-Based Testing
Blackbox
Testing
Graybox
Testing
Whitebox
Testing
PatternsBased Testing
ExperienceBased Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Random Testing
(Cat on the Keyboard)
46
How: by Technique
- Blackbox Testing
Testing
How-Based Testing
Test-Technique-Based Testing
Blackbox Testing
Cause and
Effect Testing
Classification
Tree Testing
Combinatorial
Testing
Decision Table
Testing
End-to-End
Testing
Manual
Procedure
Testing
Requirements
Testing
Scenario
Testing
Syntax Testing
Risk-Based
Testing
State-Based
Testing
User Interface
Navigation
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
47
How: by Technique
- Graybox Testing
Testing
How-Based Testing
Test-Technique-Based Testing
Graybox Testing
Boundary
Value Testing
Equivalence
Class Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
48
How: by Technique
- Whitebox Testing Testing
How-Based Testing
Test-Technique-Based Testing
Whitebox Testing
Data Flow
Testing
All Definitions
Testing
Control Flow
Testing
All Uses
Testing
Branch
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Condition
Testing
Statement
Testing
49
How: by Technique
Testing
- ExperienceBased Testing
How-Based Testing
Test-Technique-Based Testing
Experience-Based Testing
Bug Hunt
Testing
Error Guessing
Testing
Exploratory
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
Galumphing
50
How: by Technique
- Random Testing
Testing
How-Based Testing
Test-Technique-Based Testing
Random Testing
(Cat on the Keyboard)
Fuzz
Testing
Monkey
Testing
Shoe
Testing
Stuck Key
Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
51
Types of Testing
HOW WELL Object Under Test
Functions
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
52
How Well:
by Quality
Characteristic
Testing
How-Well-Based Testing
Quality-Characteristic-Based Testing
by Quality Characteristic
Capacity Testing
Based on the associated
quality characteristic
and its associated
quality attributes:
• Uncover related defects
• Determine level of quality
Compatibility Testing
Backwards Compatibility Testing
Configuration Testing
Infrastructure Compatibility Testing
Consistency Testing
Mobile Testing
Correctness Testing
Data Migration Testing
Flexibility Testing
Internationalization Testing
Functionality Testing
PersonalizationTesting
Interoperability Testing
Performance Testing
Reliability Testing
Robustness Testing
Safety Testing
Scalability Testing
Security Testing
Usability Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
53
How Well: by Quality
– Capacity Testing
Testing
How-Well-Based Testing
Quality-Characteristic-Based Testing
Capacity Testing
Load Testing
Stress Testing
Volume Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
54
How Well: by Quality
– Reliability Testing
Testing
How-Well-Based Testing
Quality-Characteristic-Based Testing
Reliability Testing
Endurance (Stability) Testing
Reliability Enhancement Testing
Reliability Growth Testing
Reliability Mechanism Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
55
How Well: by Quality
– Robustness Testing
Testing
How-Well-Based Testing
Communication Error Testing
Quality-Characteristic-Based Testing
Hardware Error Testing
Human Error Testing
Robustness Testing
Software Error Testing
Error Tolerance Testing
System Error Testing
Fault Tolerance Testing
Fault Injection Testing
Failure Tolerance Testing
Failover and Recovery Testing
Environmental Tolerance Testing
Acceleration Tolerance Testing
Acoustic Tolerance Testing
Electromagnetic Compatibility (EMC)
Tolerance Testing
Pressure/Leakage Tolerance Testing
Pressure Cycling
Tolerance Testing
Radiation Tolerance Testing
Shock/Drop Tolerance Testing
Temperature Tolerance Testing
Temperature Cycling
Tolerance Testing
Vacuum Tolerance Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
56
How Well: by Quality
– Security Testing
Testing
How-Well-Based Testing
Quality-Characteristic-Based Testing
Security Testing
Access Control Testing
Anti-Spoofing Testing
Anti-Tamper Testing
EMSEC Testing
Encryption Testing
Infrastructure Testing
Penetration Testing
Blue Team Testing
Red Team Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
57
How Well: by Quality
– Usability Testing
Testing
How-Well-Based Testing
Quality-Characteristic-Based Testing
Usability Testing
A/B Testing
Accessibility Testing
Alpha Testing
Beta Testing
Content Usage Testing
Closed Beta Testing
Open Beta Testing
(Flash) Mob Testing
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
58
Conclusion
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
59
Conclusion
Most systems require quite a few different types of testing.
Most testers are not aware of the majority of the different types of testing.
• If you are not aware that it exists, then you don’t know whether you need it.
These types of testing can be organized into a taxonomy by the 5W + 2H questions.
This taxonomy has several uses:
• Ensure the test strategy is sufficiently complete with no important type of testing
overlooked.
• Organize testing types to make them and their relationships more understandable.
• Augment test training materials.
• Help categorize and understand limitations of testing tools.
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
60
A Taxonomy of Testing Types
SEI Webinar
© 2015 Carnegie Mellon University
61