Paul Gerrard
paul@gerrardconsulting.com
Twitter: @paul_gerrard
Web: gerrardconsulting.com



Your test strategy challenges?
What is test strategy?
Test strategy approach






Test axioms as thinking tools
Using the First Equation of Testing
Testing in staged projects
Goals, risks and designing the test process
Goals, risks and coverage-based test reporting
Communicating test strategies


Case study exercise
Your challenges revisited (if we have time).

NB some slides are hidden and won’t be presented today.
Intelligent Testing and Assurance
Slide 3




This is a full day course (or 2 day, customised
for clients)
I won’t talk over every slide
We might not get through the whole Case
Study exercise in detail
But you WILL get some practice in many of
the Q&A and conversations about test
strategy.
Ever written a test strategy that
no one read?
Dig a Hole?
Test a system?


Before you can plan a test, you need to have a
lot of questions answered
The strategy…
 Presents some decisions that can be made ahead
of time
 Defines the process or method or information
that will allow decisions to be made (in project)
 Sets out the principles (or process) to follow for
uncertain situations or unplanned events.
Intelligent Testing and Assurance
Slide 8
1.
2.
3.
4.
5.
Success-based: test to show it works
Defect-based: test to find bugs
Coverage-based: analyse requirements and
code to achieve test coverage targets
Risk-based: use risk to focus testing and to
inform risk assessment
Goal-based: use business goals and risk to
focus testing and support decision-making.
FOCUS: from programmer  tester  stakeholder
Intelligent Testing and Assurance
Slide 9

Those who focus on risk and business goals:
 Sponsors, project stakeholders
 Business Users
 Project management

Those who focused on contractual aspects,
stage payments etc:
 Software suppliers
 Contracts people.
Intelligent Testing and Assurance
Slide 12

Those who focus on their risks and
responsibilities:





Suppliers
Developers
System Test Team
UAT Team
Those who focus on meeting business and
technical requirements:
 Technical Architects
 Operations
 Technical Support.
Intelligent Testing and Assurance
Slide 13
Strategy is a thought process
not a document
1.
1.1.
1.2.
1.3.
1.4.
1.5.
1.6.
2.
2.1.
2.2.
2.3.
2.4.
2.5.
3.
3.1.
3.2.
3.2.1.
3.2.2.
3.2.3.
3.2.4.
3.3.
3.4.
3.5.
3.6.
3.7.
3.8.
3.9.
3.10.
3.10.1.
3.10.2.
3.10.3.
3.10.4.
3.10.5.
3.10.6.
4.
4.1.
4.2.
4.3.
5.
5.1.
5.1.1.
5.1.2.
5.2.
5.2.1.
5.2.2.
5.3.
Introduction
Version History
Purpose
Scope
Background
Assumptions and Dependencies
Summary
Overview of User Testing
Objectives
User Testing and the Overall Project
Functional Requirements Testing
The Need for Technical Requirements Testing
Starting Early - Front-loading
User Test Policy
Baseline for Testing
Contract
Acceptance Criteria
XXX and Supplier Responsibilities
Testing and QA
Quality Plan
Testing Criteria
Risk Criteria
Starting Criteria
Policy for Re-Testing
Policy for Regression Testing
Completion Criteria
Handover to Production
Documentation Plan
User Test Strategy
Test Plan
Test Log
Incident Log
Error Log
Test Report
Functional Requirements Testing
Approach
Process
Special Application Needs
Technical Requirements Testing
Usability testing
Requirements
Conducting Usability Tests
Performance testing
Requirements for Performance Testing
Performance Test Cases
Conversion Testing
This is the table of
contents of a 51 page
document for an
acceptance test of an
outsourced
development (written
by me)
5.4.
5.4.1.
5.4.2.
5.5.
5.6.
5.7.
5.8.
5.9.
5.10.
5.11.
5.12.
5.13.
5.14.
6.
6.1.
6.1.1.
6.1.2.
6.1.3.
6.2.
6.2.1.
6.2.2.
6.2.3.
6.2.4.
6.2.5.
6.2.6.
7.
7.1.
7.2.
7.3.
8.
8.1.
8.1.1.
8.1.2.
8.1.3.
8.2.
8.3.
8.4.
8.5.
8.6.
8.7.
8.8.
8.9.
Security testing
Security Tests
Security Test Cases
Documentation Testing
Volume testing
Stress testing
Storage testing
Recovery Testing
Installation testing
Reliability Testing
Serviceability Testing
Portability Testing
Tests Not Required by the Users
User Test Infrastructure
Test Environment
Support
Roles and Responsibilities
Test Environment
Tools and Automation
Comparators
Test Data Generators
Capture/Replay Tools
Testing Information Systems
Database Query and Maintenance Facilities
Transaction Simulators
Schedule
Milestone Plan
Activities to be Resourced
Skills Required
User Test Execution
Acceptance Test Procedure
Pre-Test Meeting
During the Test
Post-Test Meeting
Software Delivery
Testing to Plan
Handling Failures
Logging Tests
Error Classification
Controlling Releases of New Versions
Regression Testing
Documentation
A large safety-related
system might have 150
pages of test strategy
supported by 10-20
other risk-related
documents
“Does size matter?”
Intelligent Testing and Assurance
Slide 15
Axioms
Early
Testing
Risks
Goals
Test
Strategy
Culture
Human
resource
Contract
Automation
Constraints
Process
(lack of?)
DeDuplication
Opportunities
Skills
Environment
Communication
User
involvement
Artefacts
Timescales
Intelligent Testing and Assurance
Slide 16





Formulated as a context-neutral set of rules
for testing systems
They represent the critical thinking processes
required to test any system
There are clear opportunities to advance the
practice of testing using them
Testers Pocketbook: testers-pocketbook.com
Test Axioms Website test-axioms.com
Intelligent Testing and Assurance
Slide 18
•
Test Axioms are not beginners guides
• They can help you to think critically about testing
• They expose flaws in other people’s thinking and their
arguments about testing
• They generate some useful by-products
• They help you to separate context from values
•
Interesting research areas!
• First Equation of Testing, Testing Uncertainty Principle,
Quantum Theory, Relativity, Exclusion Principle...
• You can tell I like physics
Intelligent Testing and Assurance
Slide 19
Stakeholder
Basis
Oracle
Fallibility
Scope
Value
Coverage
Never-Finished
Delivery
Good-Enough
Environment
Repeat-Test
Event
Design
Sequencing
Prioritisation
Intelligent Testing and Assurance
Slide 21
Delivery
Stakeholder
Repeat-Test
Sequence
Environment
Event
Never-finished
Value
Scope
Fallibility
Good-Enough
Design
Basis
Coverage
Prioritisation
Oracle
Intelligent Testing and Assurance
Slide 23
Summary:
Identify and engage the
people or organisations
that will use and benefit
from the test evidence we
are to provide
Consequence if ignored or
violated:
There will be no mandate
or any authority for testing.
Reports of passes, fails or
enquiries have no audience.
Questions:
 Who are they?
 Whose interests do they





represent?
What evidence do they
want?
What do they need it for?
When do they want it?
In what format?
How often?
Intelligent Testing and Assurance
Slide 24
Summary:
Choose test models to derive
tests that are meaningful to
stakeholders. Recognise the
models’ limitations and the
assumptions that the models
make
Questions







Are design models available to use as
test models? Are they mandatory?
What test models could be used to
derive tests from the Test Basis?
Which test models will be used?
Are test models to be documented or
are they purely mental models?
What are the benefits of using these
models?
What simplifying assumptions do these
models make?
How will these models contribute to the
delivery of evidence useful to the
acceptance decision makers?
How will these models combine to
provide sufficient evidence without
excessive duplication?
How will the number of tests derived
from models be bounded?
Consequence if ignored or

violated:
Tests design will be

meaningless and not credible
to stakeholders.
Intelligent Testing and Assurance
Slide 25
Summary:
Establish the need and requirements
for an environment and test data to
be used for testing, including a
mechanism for managing changes to
that environment – in good time.
Consequence if ignored or
violated:
Environments are not available in
time or are unsuitable for testing.
This will delay testing or cause tests
to be run in the wrong environment
and undermine the credibility of
evidence produced.
Questions






Who is responsible for the
acquisition, configuration and
support of test environments?
What assumptions regarding test
environments do our test models
make?
How will requirements for test
environments be articulated,
negotiated?
How will the validity and usability of
test environments be assured?
How will changes to environments
be managed, consistent with changes
in requirements and other
deliverables under test?
How will the state of environments,
including backed up and restored
versions be managed?
Intelligent Testing and Assurance
Slide 26
Axioms
Context
Values
Thinking +
Approach
• Not an equation in the
mathematical sense
• Need to consider three
key aspects and do a
lot of thinking
Intelligent Testing and Assurance
Slide 28


Given context, practitioners can promote
different approaches based on their values
Values are preferences or beliefs
 Pre-planned v exploratory
 Predefined v custom process
 Requirements-driven v goal-based
 Standard documentation v face-to-face comms.


Some contexts preclude certain practices
“No best practices”
Intelligent Testing and Assurance
Slide 30
The V-Model, W-Model and
Goal-Based Approaches
User
Acceptance
Test
Requirements
Is there ever a one-to-one relationship between baseline
Functional
System
documents
and testing?
Specification
Test
Physical
Design
Integration
Test
Where is the static testing (reviews, inspections, static analysis
Program
Unit
etc.)?
Specification
Test
Slide 33

Project documents:
 schedule, quality plan, test strategy, standards

Deliverables:
 requirements, designs, specifications, user documentation,
procedures
 software: custom built or COTS components, sub-systems,
systems, interfaces
 infrastructure: hardware, O/S, network, DBMS
 transition plans, conversion software, training...
Intelligent Testing and Assurance
Slide 34

Testing is the process of evaluating the deliverables
of a software project
 detect faults so they can be removed
 demonstrate products meet their requirements
 gain confidence that products are ready for use
 measure and reduce risk

Testing includes:
 static tests: reviews, inspections etc.
 dynamic tests: unit, system, acceptance tests etc.
Intelligent Testing and Assurance
Slide 35
Write
Requirements
Test the
Requirements
Install
System
Test the
Specification
Specify
System
Design
System
Test the
Design
Write
Code
System
Test
Build
System
Build
Software
Acceptance
Test
Integration
Test
Unit
Test
Slide 36
Write
Requirements
Test the
Requirements
Requirements
Animation
Scenario
Walkthroughs
Test the
Specification
Specify
System
Early Test
Case Preparation
Reviews
Install
System
Acceptance
Test
System
Test
Build
System
Inspections
Design
System
Test the
Design
Write
Code
Inspection
Build
Software
Integration
Test
Unit
Test
Static
Analysis
Slide 37
Write
Requirements
Test the
Requirements
Install
System
Test the
Specification
Specify
System
System
Test
Build
System
Business
Integration
Testing
System
Integration
Testing
Acceptance
Test
Usability
Testing
Design
System
Test the
Design
Build
Software
Integration
Test
Boundary
Value Testing
Equivalence
Partitioning
Write
Code
Unit
Test
Performance
Testing
Security
Testing
Exploratory
Testing
Path
Testing
Slide 38
Programme managed
Slide 39

The fundamental business objectives of the
system(s) to be built, implemented and used
 The benefits of undertaking the project
 The payoff(s) that underpin and justify the project

Risks are what threaten the goals of a project.
Intelligent Testing and Assurance
Slide 40

The test strategy must set out how:
 Achievements (the goals) of a project are
evidenced or demonstrated
 The risks that threaten goals will be explored, reassessed and deemed acceptable (or not)


We need to understand the goals and how
achievement will be measured
We need to understand (in particular,
product) risk and how they are explored and
exposed.
Intelligent Testing and Assurance
Slide 41
Every project has a network
of dependent interim and
ultimate goals threatened by
risks
The ultimate
business goal
Your strategy will identify
the test activities that will
measure goal achievement
and evidence these risks
GOAL
Intelligent Testing and Assurance
RISK
Slide 42


Italian dictionary: Risicare, “to dare”
Simple generic definition:
 “The probability that undesirable events will
occur”

In this tutorial, we will use this definition:
“A risk threatens one or more of a project’s
goals and has an uncertain probability”
Intelligent Testing and Assurance
Slide 44
Process Risk
Project Risk
variances in planning and
estimation, shortfalls in
staffing, failure to track
progress, lack of quality
assurance and configuration
management
resource constraints,
external interfaces, supplier
relationships, contract
restrictions
Primarily a management
responsibility
Planning and the development process
are the main issues here.
Product Risk
Testers are
mainly
concerned with
Product Risk
lack of requirements stability,
complexity, design quality,
coding quality, non-functional
issues, test specifications.
Requirements risks are the most significant risks
reported in risk assessments.
Slide 46


Do nothing!
Pre-emptive risk reduction measures





information buying
process model
risk influencing
contractual transfer
Where
testing fits in
Reactive risk reduction measures
 contingency plans
 insurance

But this all sounds highly theoretical – we could
never get this to work in my company!
Intelligent Testing and Assurance
Slide 50
Even penguins
know how to
manage risk!
Test
Phase/Activity
GOAL
Intelligent Testing and Assurance
RISK
Slide 67
Requirements
HL Design
Tech Design
Goal/Risk
Prog. Spec.
Code
Sub-System
System
•
•
•
•
•
Walkthrough
Review
Inspect
Prototype
Early test preparation
• Unit Test
• Static analysis
• Integration Test
• System Test
• Acceptance Test
• Non-functional
• Security
• Performance
• Usability
• Backup/recovery
• Failover/restart
• Volume
• Stress
• Etc. etc.
Intelligent Testing and Assurance
Slide 68
Goal/Risk
Test Objective
Goal/Risk
Test Objective
Technique
Goal/Risk
Test Objective
Technique
Goal/Risk
Test Objective
Technique
Goal/Risk
Test Objective
Sub-System Testing
System Testing
Goal/Risk
Test Objective
Technique
Goal/Risk
Test Objective
Technique
Intelligent Testing and Assurance
Slide 69



Planning relies on predictions of the future
but how can you predict test status at a
future date?
The answer is … you can’t
The Testing Uncertainty Principle:
 One can predict test status, but not when it
will be achieved;
 One can predict when a test will end, but
not its status.
Intelligent Testing and Assurance
Slide 71

Represents the overall readiness to commit to
going live considering:







The readiness of the solution
The readiness of the business
Ability to implement (and rollback, if necessary)
To live with the difficulties of early days
To support the system in it’s early days
The need to be compliant
Here’s a generic, but comprehensive set of
Acceptance Criteria for a Large Programme.
Intelligent Testing and Assurance
Slide 74
Steady State
Operation
1. The Solution
(system, process
and data) is ready
2. The
Organisation
(business and IT)
is ready
Implementation
Early Life
3. We are ready to 4. We are ready to
Implement the
Support the
solution (and roll
solution
back if necessary)
5.Operational
Risks are
understood and
mitigating actions
taken
6. We meet the necessary Regulatory and Compliance
requirements
Intelligent Testing and Assurance
Slide 75
Level 1 Criteria
The Solution (system, process and
data) is ready
Level 2 Criteria
•
•
•
The Organisation (business and
IT) is ready
•
•
•
•
We are ready to Implement the
solution (and roll back if
necessary)
•
We are ready to Support the
solution
•
•
•
•
•
Operational Risks are understood
and mitigating actions agreed.
•
•
We meet the necessary
Regulatory and Compliance
requirements
•
The users have proven that the solution (system and data) supports the business processes.
The quality level is high (demonstrated by a low quantity and severity of defects) with agreed
workarounds for significant defects.
The system performance is sufficient and it is reliable, stable and robust to failure.
New organisational structures are in place and necessary positions are filled.
Sufficient staff have been trained in the new solution and competency has been assessed to be
acceptable.
Third parties understand the changes and their readiness has been confirmed.
Benefit realisation plans are in place.
Implementation and roll-back plans have been adequately tested, rehearsed and
communicated.
Roles, responsibilities and escalation path over the cutover period have been agreed.
Temporary business workarounds during the cutover period have been agreed.
Early Life support processes and people are in place.
Sufficient transition and handover has been completed with the support and operations
groups to enable the solution to be supported in Early Life.
Processes and metrics are in place to provide early warning of operational problems during
Early Life.
For the significant operational risks mitigating actions have agreed including, where possible,
tested contingency plans.
Any residual risk (i.e. not fully mitigated) has been understood and accepted by senior
management.
The necessary regulatory and compliance approvals have been received (Audit, SOX, System
Security).
Intelligent Testing and Assurance
Slide 76


A test exit criteria is, in effect, a planning
assumption
If exit criteria are met on time or earlier, our
planning assumptions are sound:
We are where we want to be

If exit criteria are not met or not met on time,
our plan was optimistic:
Our plan needs adjustment, or we must
relax the criteria

What do test exit criteria actually mean?
Intelligent Testing and Assurance
Slide 77
Project/Test Phase
Test Driver
Test Obj.
Reqs
Design
Build
Integ
Systest
UAT
Trial
Prod.
Business goals
Coverage target
Objectives for each test
phase are easily
identified
Risks
Intelligent Testing and Assurance
Slide 79
today
start
Planned
end
all risks
‘open’ at
the start
residual risks
of releasing
TODAY
Progress through the test plan
Intelligent Testing and Assurance
Slide 80

Risk of release is known:
 On the day you start and throughout the test phase
 On the day before testing is squeezed
Progress through the test plan brings positive results
– risks are checked off, benefits available
 Pressure: to eliminate risks and for testers to
provide evidence that risks are gone
 We assume the system does not work until we have
evidence – “guilty until proven innocent”
 Reporting is in the language that management and
stakeholders understand.

Intelligent Testing and Assurance
Slide 81
Goal
Goal
Goal
Goal
Goal
Goal
Goal
Goal
Goal
Goal
Goal
Open
Risks
Closed
Open
Closed
Closed
Open
Closed
Open
Goals/Benefits available for release
Slide 82

Risk(s) that block every benefit are known:
 On the day you start and throughout the test phase
 Before testing is squeezed
Progress through the test plan brings positive results
– benefits are delivered
 Pressure: to eliminate risks and for testers to
provide evidence that benefits are delivered
 We assume that the system has no benefits to
deliver until we have evidence
 Reporting is in the language that management and
stakeholders understand.

Intelligent Testing and Assurance
Slide 83
Test Plan Identifier
Introduction
Test Items
Features to be Tested
Features not to be
Tested
6. Approach
7. Item Pass/Fail Criteria
8. Suspension Criteria
and Resumption
Requirements
1.
2.
3.
4.
5.
Test Deliverables
Testing Tasks
Environmental Needs
Responsibilities
Staffing and Training
Needs
14. Schedule
15. Risks and
Contingencies
16. Approvals
9.
10.
11.
12.
13.
Based on IEEE Standard 829-1998
Intelligent Testing and Assurance
Slide 85

Used as a strategy checklist
 Scarily vague (don’t go there)

Used as a documentation template/standard
 Flexible, not prescriptive, but encourages copy and
edit mentality (documents that no one reads)

But many many testers seek guidance on
 What to consider in a test strategy
 Communicating their strategy to stakeholders and
project participants
Intelligent Testing and Assurance
Slide 86












Items 1, 2 – Administration
Items 3+4+5 – Scope Management, Prioritisation
Item 6 – All the Axioms are relevant
Items 7+8 – Good-Enough,Value
Item 9 – Stakeholder,Value, Confidence
Item 10 – All the Axioms are Relevant
Item 11 – Environment
Item 12 – Stakeholder
Item 13 – All the Axioms are Relevant
Item 14 – All the Axioms are Relevant
Item 15 – Fallibility, Event
Item 16 – Stakeholder Axioms
Intelligent Testing and Assurance
Slide 87
1.
Stakeholder Objectives
 Stakeholder management
 Goal and risk management
 Decisions to be made and how






(acceptance)
 How testing will provide confidence
and be assessed
 How scope will be determined
2.
Delivery approach
3.
Plan (high or low-level)
4.
Design approach
 Test phases and sequence
 Sources of knowledge (bases and
oracles)
 Sources of uncertainty
 Models to be used for design and
coverage
 Prioritisation approach
Test sequencing policy
Repeat test policies
Environment requirements
Information delivery approach
Incident management approach
Execution and end-game approach






Scope
Tasks
Responsibilities
Schedule
Approvals
Risks and contingencies
Intelligent Testing and Assurance
Slide 88




It’s all about consensus
Strategy must address stakeholders’ concerns
Present strategy aligned with those concerns
“Their part” of the strategy sets out HOW it addresses
those concerns:





Will it evidence goal achievement?
Does it address the risks?
Does it set out MY and OTHERS’ responsibilities?
How do MY activities fit into the context of the larger plan?
Involve stakeholders in risks workshops, reviews and consult
them before presenting your proposal.
Intelligent Testing and Assurance
Slide 89

“Business goals and risk” focus:
 Project Stakeholders, management, users
 Are my goals being evidenced?
 Has every risk been identified?
 Has every risk been addressed?
 Is the right group addressing the risk?
 Will tests address MY concerns?
Intelligent Testing and Assurance
Slide 90

“Contractual aspects” focus:
 Software suppliers and Contract people
 Does the strategy match the contract?
 Are test activities aligned with stage payments?
 Is the strategy fair and allow me to be paid?
 Does the strategy impose the right level of test activity on
the supplier?
 How will testing demonstrate we get what we want?
Intelligent Testing and Assurance
Slide 91

“Risks and responsibilities” focus:
Suppliers, Developers, System Test Team, UAT Team
Do I know what I have to do in my testing?
Who covers the things I don’t cover?
When do I start testing?
When do I stop?
What evidence do I need to provide to address the
stakeholder risks?
 How do I report progress?






Intelligent Testing and Assurance
Slide 92

“Meeting business and technical requirements”
focus:
 Business analysts, Technical Architects, Operations, Technical




Support
How will testing show the architecture “works”?
How will testing give me confidence the system “works”?
How will testing demonstrate the system can be operated
and supported?
Is the system ready to be deployed?
Intelligent Testing and Assurance
Slide 93

Presentations
 Appropriate for management teams or larger groups
 Q&A can stay at a high level

Walkthroughs
 Smaller groups get more involved in the detail

In person
 Involve individuals in workshops, reviews, handover

Focus on the audience messages regardless of the
medium.
Intelligent Testing and Assurance
Slide 94

Helps stakeholders:
 They get more involved and buy-in
 The have better visibility of the test process

Helps testers
 Clear guidance on the focus of testing
 Approval to test against risks in scope
 Approval to not test against risks out of scope
 Clearer test objectives upon which to design tests.
Intelligent Testing and Assurance
Slide 96

Helps stakeholders:
 They have better visibility of the results available and the
risks that block results

Helps management:
 To see progress in terms of risks addressed and results
that are available for delivery
 To manage the risks that block acceptance
 To better make the release decision

Helps project managers, helps testers.
Intelligent Testing and Assurance
Slide 97
We work for the operator for a
national lottery
Slide 98
1.
Introduction to the project (30-40m)
1. Regulatory framework and stakeholders
2. Stakeholder objectives/benefits and concerns
2.
3.
4.
5.
6.
The lottery process (more risks) (20m)
Project Goal network and Test Process
(20m)
Test Phase Responsibilities (20m)
Test Phase Definition (20m)
What’s Left?
Intelligent Testing and Assurance
Slide 99




If there is an unknown – don’t get stuck
In the team, think of a possible scenario and
assume that
There won’t be time for me to invent answers
and explain
… and your ideas are probably more
interesting 
Intelligent Testing and Assurance
Slide 100
1.
On your own:
1. Read the Case Study overview (page 10-11
ONLY)
2. Read the regulatory framework
3. Learn about the stakeholders
4. Identify 1-2 goals and 3-4 risks
2.
As a TEAM, discuss your goals and risks
1. List them in your workbook page 12
2. If you need more space, use page 25 onwards…
Intelligent Testing and Assurance
Slide 101







Did everyone find the same goals/risks?
Is it hard to find goals?
Is it hard to find risks?
Should these goals/risks be in scope for
testing?
Which ones aren’t suitable for testing?
Does it depend on how you define testing?
Get your scope right/agreed!
Intelligent Testing and Assurance
Slide 102
1.
Read through the lottery process stage by
stage (page 13-15 ONLY)
1. Each stage has a short narrative
2.
Can you think of any new risks?
1. Add them to the table on page 12
Intelligent Testing and Assurance
Slide 103




How many risks do you think are there to be
documented?
How many are in scope for testing?
Who has the most risks of concern?
For each risk – is there a single mitigation or
more than one?
Intelligent Testing and Assurance
Slide 104





The project goal network on page 16 is an
outline of the goals of the project
Read and discuss the diagram as a team
Some test activities are shown, but are
merged with development activities
Mark up the Project Goal network with your
test phases or…
Use page 19 to sketch out the network of
goals and test phases that you think you need.
Intelligent Testing and Assurance
Slide 105





Did having the system schematic help?
Each test phase measures achievement
Each test phase is associated with a project
goal, activity or deliverable
If a test phase ‘passes’ then the goal is
achieved
Does exiting a test phase confirm that a
project goal has been achieved?
Intelligent Testing and Assurance
Slide 106

For each test phase, define some objectives
(goals to demonstrate or risks to be
mitigated) and assign responsibilities
Intelligent Testing and Assurance
Slide 107




Now you have a much clearer understanding
of the goals and risks
You have identified a set of test stages and
begun to assign goals/risks and responsibilities
Read page 20 guidelines for completing the
full test stage definitions
Create a full definition of one of your test
stages (I suggest a system or integration test
stage to start)
Intelligent Testing and Assurance
Slide 108
Lots!
No requirements! Do you need them (yet)?
No discussion of what the sources of knowledge
(basis and oracles) are or their quality/usefulness
 No discussion of test models


 How do we identify things to test?
 What coverage targets could be used; can we define
them; can we measure them?


No discussion of our capabilities (the operator)
and the supplier’s
No mention of environments or technology
Intelligent Testing and Assurance
Slide 110






No discussion of acceptance criteria
(consider slides 73-77 etc.)
No discussion of what evidence stakeholders
actually require and in what format
No mention of the incident process
No discussion of test execution tools
(developer or system level)
No discussion of test design/record-keeping
And lots more, I know…
Intelligent Testing and Assurance
Slide 111
Email me your test strategy
questions – answers are often
good ideas for blogs
Thank You!
Paul Gerrard
paul@gerrardconsulting.com
Web: gerrardconsulting.com
@paul_gerrard