Testing Workflow
In the Unified Process and
Agile/Scrum processes
Testing Workflow: Purpose
• Verify the result from implementation by
testing each build
– Plan the tests in each iteration
– Integration tests for every build within the
iteration
– System tests for the end of the iteration
• Design and implement tests by creating
– test cases that specify what to test
– test procedures that specify how to perform the
tests
– executable test components to automate the tests
Review: Kinds of testing
• Unit testing
– Does an individual software component (unit)
live up to its contract?
– Typically done by the same person who codes
the given unit
– Tests become part of regression test suite
• Integration testing
• System testing
• Regression testing
Where do unit tests fit into UP?
• Unit tests are part of the implementation
workflow
• They come before the integration testing
• They are typically designed and written by
the programmer who writes the given
module/unit
• (See separate slides on unit testing.)
Review: Kinds of testing
• Unit testing
• Integration testing
– Does a collection of modules function
correctly together as a group?
• System testing
• Regression testing
Review: Kinds of testing
• Unit testing
• Integration testing
• System testing
– Does the entire system meet the end user’s
requirements?
• Regression testing
Review: Kinds of testing
• Unit testing
– Shows that individual software components
(units) live up to their contracts
– Typically done by the same person who codes
the given unit
– Tests become part of regression test suite
• Integration testing
• System testing
• Regression testing
Testing Workflow: Flow
Testing Workflow in UP Phases
Testing Workflow in Phases
Generally, when there is an implementation
result, there is a test --- so Test at each build
Inception phase:
initial test planning, prototype testing
Elaboration phase:
test architectural baseline
Construction phase:
significant testing at each build
Transition phase:
re-test fixes and regression tests
Testing Workflow: Other Tasks
Regression tests:
in a new build, re-apply tests from old builds
to make sure nothing “broke” in new build
Evolve test model
create new test cases for next build
refine prior test cases into regression tests
remove obsolete tests and corresponding
test procedures and components
Testing Workflow: Models
Test model artifacts
Test cases
Test procedures
Test components
Test subsystem packages for complex test
Other artifacts
Test Plan
List of product Defects
Test Evaluation
Testing Workflow: Model
Describes how executable components (e.g.,
builds) in the implementation model are tested by
integration tests and system tests
Describes how specific system aspects are to be
tested
e.g., usability testing of user interface or
user manual
Testing Workflow: Test Case
Specifies one way to test the system
What to test (requirements)
With which inputs
Expected results
Test conditions and constraints
Test Case Examples
Test a use case or a scenario in a use case
(black box)
actor interaction, pre-conditions,
action sequences, post-conditions
Test a use-case realization (white box)
verify interaction between components
installation tests, configuration tests, negative
(abuse) tests, stress tests
Organize similar tests in input/result matrix
Family of tests for different paths through a use case
Special Tests
Verify the system can be installed on the customer platform
and that it executes correctly when installed
Configuration tests verify the system works correctly in
different configurations
Negative tests (abuse tests) discover what causes failure
Test outside the intended design, e.g., wrong
configuration, wrong input, etc.
Reveals weaknesses in design
Stress tests show problems with insufficient
or shared resources
Testing Workflow: Test Procedures
Specify how to perform one or several test cases or
parts of them
•Instructions for tester in manual test case
•Instructions for interacting with test automation tool to
create executable test components, then instructions to
integrate and execute those test components
•One test procedure may cover several test cases
•One test case may require several test procedures
•Test procedure instructions often reflect flow-of events
descriptions, including input values, how to enter input
values, and how to verify results
Testing Workflow: Test Components
Automate one or several test procedures or parts
Provide test inputs, control and monitor test
execution, and possibly analyze and report results
Developed with scripting languages, programming
languages, or test automation tools
test drivers, test harnesses, and test scripts
Complex test cases and procedures may require
formal analysis, design, and implementation modeling
of the test components
Testing Workflow: Test Plan
Describes testing strategies, resources and schedule
Strategy
What kind of tests to perform for each iteration
Test objectives
Required level of test and code coverage
Test acceptance criteria
Trade-off cost vs. improved quality and reduced risk
Test Plan may be defined in conjunction with the model
Artifacts from other workflows
analysis (black box)
design and implementation (white box)
“Tests” may include artifact reviews
Testing Workflow: Test Results
Results are the output data of the test and/or the
behavior of the system during test
Analysis of results: pass or defect
Defect: a system anomaly (software fault or a
defect found in artifact review)
• Capture symptom in sufficient detail to allow problem
resolution
• Record defect in problem and resolution tracking
system
Test evaluation: summary of overall test results
test-case coverage, code coverage
defects and defect status
Testing Workflow: Test Results
Testing Workflow: Workers
Test designer:
responsible for integrity of the test model
plan tests: test goals and schedule
select and define test cases, test procedures
evaluate test results
Component engineer:
responsible for development of test
components that automate some of the
test procedures
consider independent test
Testing Workflow: Workers
Integration tester:
perform integration tests and report defects
may be same person as system integrator in
implementation workflow
System tester:
perform system tests and report defects
focus on externally observable behavior
consider users, beta customers, etc. as testers
Testing Workflow – THE PLAN
Plan the Tests Strategy and Resources
Define Integration Tests
Define System Tests
Design Regression Tests
Define and Mutate Test Cases
Implement Tests
Evaluate Tests
Testing Workflow – THE PLAN
Plan the Tests Strategy and Resources
Describe testing strategy
Inputs: use case model, supplemental requirements,
design model
Develop test cases and procedures that
have a minimum of overlap
test the most important test cases
test high risk requirement items
Estimate resources
Estimate human and system resources
Schedule the testing effort
Testing Workflow – THE PLAN
Define Integration Tests
Derive test cases from use-case realizations – design
(component interaction)
Develop tests with minimal overlap that test an
interesting path or scenario through use case
Primarily consider interaction diagrams
Combinations of actor input, output and system
start state that lead to interesting scenarios that
employ the classes that participate in the
diagrams
Testing Workflow – THE PLAN
Define System Tests
Test system as a whole
Test combinations of use cases instantiated under
different conditions
Hardware configurations
System loads
Number of actors
Database size, etc.
Prioritize combinations
required or are likely to function in parallel
likely to influence each other if in parallel
involve multiple processes
frequently use system resources or
use them in complex and unpredictable ways
Testing Workflow – THE PLAN
Design Regression Tests
Modify test cases from earlier builds to be used
as regression tests in subsequent builds
Make more flexible and resilient to changes to
the system under test
Specifically address previously found defects
Automate where possible
Testing Workflow – THE PLAN
Define and Mutate Test Cases
Work test case by test case and suggest testing
procedures for each one
Reuse existing test cases
May need to modify them to be reusable
Align test procedures to subsystems under test
Minimize overlap
Isolates change to test resulting from change
to subsystem
Testing Workflow – THE PLAN
Implement Tests
Create test components from test procedures
With test automation tools
Perform or specify actions
Record actions as a test component
When programming explicitly
Design and implement using test procedures
Provide input and output data generation, visualization,
interpretation and management utilities
Use spreadsheets and database applications
Testing Workflow – THE PLAN
Evaluate Tests
Perform the integration tests relevant to a build
Manually perform test procedures
Compare test results with the expected test results
Investigate test results that deviate from the
expected
Faulty system and/or faulty test?
Report defects to the component engineer responsible for
the components likely to contain the fault
Report defects to the test designers for evaluation of overall
test results
Testing Workflow – THE PLAN
Evaluate Tests (cont)
Test designers prepare metrics of test acceptance with
respect to system quality level
Test completeness and coverage
Percentage of test cases executed
Percentage of code and paths tested
Reliability
Categorize defect severity
Analyze trends of discovered defects
Compare with prior experience to predict amount of testing
still required to reach an acceptable defect level
Testing Workflow – THE PLAN
Evaluate Tests (cont)
Suggest actions
Suggest further tests
Relax test criteria
Deliver parts of the system that passed;
rework and retest parts that failed
Document test completeness, reliability and actions
Testing Workflow – SUMMARY
Test Model
Test cases specify what to test
Test procedures specify how to perform tests
Test components automate test procedures
Test Plan
Test Evaluation
Defects to be fed into other core work