THE
TESTING
OF
Advancing
Testing Using
Axioms
Paul Gerrard


Axioms – a Brief Introduction
Advancing Testing Using Axioms
 First Equation of Testing
 Test Strategy and Approach
 Testing Improvement
 A Skills Framework for Testers
 Quantum Theory for Testing

Close
Surely, there must be SOME things that
ALL testers can AGREE ON?
Or are we destined to argue
FOREVER?


Started as a ‘thought experiment’ in my blog in
February 2008
Some quite vigorous debate on the web
 ‘great idea’
 ‘axioms don’t exist’
 ‘Paul has his own testing school’



Initial 12 ideas evolved to 16 test axioms
Testers Pocketbook: testers-pocketbook.com
Test Axioms Website test-axioms.com
•
Some very useful by-products
• Test strategy, improvement, skills framework
•
Interesting research areas!
• First Equation of Testing, Testing Uncertainty Principle,
Quantum Theory, Relativity, Exclusion Principle...
• You can tell I like physics
There are no agreed definitions
of test or testing!
The words software, IT, program,
technology, methodology, vmodel, entry/exit criteria, risk –
do not appear in definitions
American Heritage Dictionary:
Test: (noun)
•
•
•
A procedure for critical evaluation;
A means of determining the presence, quality, or
truth of something;
A trial.
A testing stakeholder is someone
who is interested in the outcome
of testing;
You can be your OWN
stakeholder (e.g. dev and users)
Let’s look at a few of the test
axioms
Testing needs
stakeholders
Test design is
based on models
Testers need
sources of
knowledge to select
things to test
Testing needs a test coverage
model or models
Our sources of
knowledge are
fallible and
incomplete
The value of
testing is
measured by the
confidence of
stakeholder
decision making
Testing never
goes as planned;
evidence arrives
in discrete
quanta
“Ohhhhh... Look at that, Schuster... Dogs are so cute when they
try to comprehend quantum mechanics.”
Testing never
finishes; it stops
Axioms
+ Context
+ Values
+ Thinking
=Approach




Separation of Axioms, context, values and
thinking
Tools, methodologies, certification, maturity
models promote approaches without
reference to your context or values
No thinking is required!
Without a unifying test theory you have no
objective way of assessing these products.
Strategy is a thought process
not a document
Axioms
Early Testing Communication
DeDuplication
Risks
Test
Strategy
Goals
Culture
Human
resource
Opportunities
Contract
Constraints
Skills
Environment
Process
(lack of?)
Timescales
Automation
User
involvement
Artefacts
Summary:
Identify and engage the
people or organisations
that will use and benefit
from the test evidence we
are to provide
Consequence if ignored or
violated:
There will be no mandate
or any authority for testing.
Reports of passes, fails or
enquiries have no audience.
Questions:
 Who are they?
 Whose interests do they





represent?
What evidence do they
want?
What do they need it for?
When do they want it?
In what format?
How often?
Summary:
Choose test models to derive
tests that are meaningful to
stakeholders. Recognise the
models’ limitations and the
assumptions that the models
make
Questions







Consequence if ignored or
violated:
Tests design will be
meaningless and not credible
to stakeholders.


Are design models available to use as
test models? Are they mandatory?
What test models could be used to
derive tests from the Test Basis?
Which test models will be used?
Are test models to be documented or
are they purely mental models?
What are the benefits of using these
models?
What simplifying assumptions do these
models make?
How will these models contribute to the
delivery of evidence useful to the
acceptance decision makers?
How will these models combine to
provide sufficient evidence without
excessive duplication?
How will the number of tests derived
from models be bounded?
Test Plan Identifier
Introduction
Test Items
Features to be Tested
Features not to be
Tested
6. Approach
7. Item Pass/Fail Criteria
8. Suspension Criteria
and Resumption
Requirements
1.
2.
3.
4.
5.
Test Deliverables
Testing Tasks
Environmental Needs
Responsibilities
Staffing and Training
Needs
14. Schedule
15. Risks and
Contingencies
16. Approvals
9.
10.
11.
12.
13.
Based on IEEE Standard 829-1998












Items 1, 2 – Administration
Items 3+4+5 – Scope Management, Prioritisation
Item 6 – All the Axioms are relevant
Items 7+8 – Good-Enough,Value
Item 9 – Stakeholder,Value, Confidence
Item 10 – All the Axioms are Relevant
Item 11 – Environment
Item 12 – Stakeholder
Item 13 – All the Axioms are Relevant
Item 14 – All the Axioms are Relevant
Item 15 – Fallibility, Event
Item 16 – Stakeholder Axioms
1.
Stakeholder Objectives
 Stakeholder management
 Goal and risk management
 Decisions to be made and how






(acceptance)
 How testing will provide confidence
and be assessed
 How scope will be determined
2.
Design approach
 Sources of knowledge (bases and
oracles)
 Sources of uncertainty
 Models to be used for design and
coverage
 Prioritisation approach
Delivery approach
3.
Test sequencing policy
Repeat test policies
Environment requirements
Information delivery approach
Incident management approach
Execution and end-game approach
Plan (high or low-level)
4.






Scope
Tasks
Responsibilities
Schedule
Approvals
Risks and contingencies
Test process improvement is a
waste of time
There are no “practice” Olympics to determine the
best
 There is no consensus about which practices are
best, unless consensus means “people I respect also
say they like it”
 There are practices that are more likely to be
considered good and useful than others, within a
certain community and assuming a certain context
 Good practice is not a matter of popularity. It’s a
matter of skill and context.

Derived from “No Best Practices”, James Bach, www.satisfice.com
Actually
its 11
(most were not
software related)

Google search
 “CMM” – 22,300,000
 “CMM Training” – 48,200
 “CMM improves quality” – 74 (BUT really 11 – most
of these have NOTHING to do with software)

A Gerrard Consulting client…
 CMM level 3 and proud of it (chaotic, hero culture)
 Hired us to assess their overall s/w process and make
recommendations (quality, time to deliver is slipping)
 40+ recommendations, only 7 adopted – they couldn’t
change
 How on earth did they get through the CMM 3 audit?



Using process change to fix cultural or
organisational problems is never going to
work
Improving test in isolation is never going to
work either
Need to look at changing context rather than
values…
Context <- your context
+ Values <- your values
+ Thinking <- your thinking
=Approach <- your approach
Context <- someone else's
+ Values <- someone else's
+ Thinking <- someone else's
=Approach <- someone else's
Axioms <- recognise
+ Context <- could change?
+ Values <- hard to change
+ Thinking <- just do some
=Approach <- your approach


Axioms represent the critical things to think
about
Associated questions act as checklists to:
 Assess your current approach
 Identify gaps, inconsistencies in current approach
 QA your new approach in the future


Axioms represent the WHAT
Your approach specifies HOW








Mission
Coalition
Vision
Communication
Action
Wins
Consolidation
Anchoring
Changes identified
here
If you must use one,
this is where your ‘test
model’ comes into play
Axioms indicate WHAT to think
about...
...so the Axioms point to SKILLS
Summary:
Choose test models to derive tests that
are meaningful to stakeholders.
Recognise the models’ limitations and
the assumptions that the models make.
Consequence if ignored or violated:
Tests design will be meaningless and not
credible to stakeholders.
Questions:
 Are design models available to use as
test models? Are they mandatory?
 What test models could be used to
derive tests from the Test Basis?
 Which test models will be used?
 Are test models to be documented
or are they purely mental models?
 What are the benefits of using these
models?
 What simplifying assumptions do
these models make?
 How will these models contribute to
the delivery of evidence useful to the
acceptance decision makers?
 How will these models combine to
provide sufficient evidence without
excessive duplication?
 How will the number of tests
derived from models be bounded?

A tester needs to understand:
 Test models and how to use them
 How to select test models from fallible sources of





knowledge
How to design test models from fallible sources of
knowledge
Significance, authority and precedence of test models
How to use models to communicate
The limitations of test models
Familiarity with common models
Is this all that current certification provides?

Functional testers are endangered:
 Certification covers process and clerical skills
 Functional testing is becoming a commodity and is
easy to outsource

To survive, testers need to specialise:






Management
Test automation
Test strategy, design, goal- and risk-based
Stakeholder management
Non-Functional testing
Business domain specialists...


Intellectual skills and capabilities are more
important than the clerical skills
Need to re-focus on:






Testing thought processes (Axioms)
Real-world examples, not theory
Testing as information provision
Goal and risk-based testing
Testing as a service (to stakeholders)
Practical, hands-on, real-world training, exercises
and coaching.
If evidence arrives in discrete
quanta...
...can we assign a value to it?

Tests are usually run one by one
Every individual test has some significance
Some tests expose failures but ultimately we
want all tests to PASS
When all tests pass – the stakeholders are
happy, aren’t they?
Can we measure confidence?

But...





Testers cannot usually:





Prepare all tests they COULD do
Run ALL tests in the plan
Re-test ALL fixes
Regression-test as much or as often as required
How do we judge the significance of tests?
 To include them in scope for planning (or not)
 To execute them in the right order?
 To ensure the most significant tests are run?


What stakeholders want ultimately, is every
test to pass
The ideal situation is:
 We have run all our tests
 All our tests pass
 Acceptance is a formality

Not all tests pass though
 We track incidents, severity and priority – great
 But how do we track the significance or value of
tests that pass?

Significance varies by objective:
 Criticality of the business goal it covers
 Criticality of the risk it covers

Significance varies by precedent:
 The first end-to-end test pass is significant
 Subsequent e2e passes are less significant

Significance varies by functional dependence:
 A test of shared functionality is more important than
standalone functionality

Significance by stakeholder:
 Customers and sponsor tests are more significant
than developer tests.



Stakeholders usually know how to judge the
significance of failures when tests FAIL
So why don’t we assess the significance of
tests BEFORE we run them?
If we did that:
 We could scope and prioritise more effectively
 We would know exactly which tests provide
enough information for an acceptance decision
 Acceptance criteria would be taken seriously.

Using business goals, risks and coverage to drive
testing is ‘advanced’ - but it is still VERY CRUDE

Quantum Testing proposal:


Need to assign a micro-significance to all tests
Need to assess macro-significance to collections of tests
As tests are created and executed, evidence
increases incrementally
 Manage progress by monitoring EVIDENCE rather
than by counting test cases.






We and our stakeholders could know the
value of tests BEFORE we run them
Stakeholders would understand WHAT we
are doing and WHY
The problem of ‘enough testing’ becomes a
shared challenge (testers and stakeholders)
Caveats: we assign significance qualitatively
rather than numerically
Significance is RELATIVE rather than absolute!


Axioms are context-neutral rules for testing
The Equation of Testing
 Separates axioms, context, values and thinking
 We can have sensible conversations about process


Axioms and associated questions provide
context neutral checklists for test strategy,
assessment/improvement and skills
Quantum Testing aims to address the
question, “how much testing is enough?”
THE
TESTING
OF
testaxioms.com
testers-pocketbook.com
gerrardconsulting.com
Thank-You!