Black Box Software Testing
Part 12.
Introduction to
Test Documentation
Co-authors: Cem Kaner & James Bach
Acknowledgement
Some of this material is from the Third Los Altos Workshop on Software Testing (LAWST), February 7 and 8,
1998. I founded the LAWST and was co-organizer of LAWST 3.
At LAWST, we discussed test documentation (test planning strategies and materials). The agenda item was:
•
•
How do we know what test cases we have? How do we know which areas of the program are well covered and which are not?
How do we develop this documentation EFFICIENTLY? As many of you know, I despise thick test plans and I begrudge every
millisecond that I spend on test case documentation. Unfortunately, some work is necessary. My question is, how little can we get away
with, while still preserving the value of our asset?
The following people attended LAWST 3: Chris Agruss, James Bach, Karla Fisher, David Gelperin, Kenneth
Groder, Elisabeth Hendrickson, Doug Hoffman, III, Bob Johnson, Cem Kaner, Brian Lawrence, Brian Marick,
Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret Pettichord, Johanna Rothman, Jane Stepak, Melora
Svoboda, Jeremy White, and Rodney Wilson.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
499
Copyright Notice
These slides are distributed under the Creative Commons
License.
In brief summary, you may make and distribute copies of these
slides so long as you give the original author credit and, if you
alter, transform or build upon this work, you distribute the
resulting work only under a license identical to this one.
For the rest of the details of the license, see
http://creativecommons.org/licenses/by-sa/2.0/legalcode.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
500
Definitions
• The test planning documents include:
– The Testing Project Plan, which identifies classes of tasks and
broadly allocates people and resources to them;
– High-level designs for test cases (individual tests) and test
suites (collections of related tests);
– Detailed lists or descriptions of test cases;
– Anything else that you would put in a hard copy or virtual
binder that documents the collection of test cases that you will
develop and run.
• This collection of materials is sometimes called the test plan
and sometimes called the test documentation.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
501
Levels of Analysis
• 1-Variable
– Basic user interface verification
– Filters
– Many testers stop here, at the most superficial level of testing
• 2-Variable
– Shared or related filters
– Logical relationships among variables
• Multi-Variable
– The core problem is the vastness of the space of test cases. We
need a sampling strategy.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
502
Basic Test Documentation Components
• Lists:
– Such as lists of fields, error messages, DLLs
• Outlines: An outline organizes information into a hierarchy of
lists and sublists
– Such as the testing objectives list later in the course notes
• Tables: A table organizes information in two dimensions
showing relationships between variables.
– Such as boundary tables, decision tables, combination test tables
• Matrices: A matrix is a special type of table used for data
collection.
– Such as the numeric input field matrix, configuration matrices
» Refer to Testing Computer Software, pages 217-241. For more
examples, see page Testing Computer Software, page 218.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
503
Configuration Planning Table
Var 1
Var 2
Var 3
Var 4
Var 5
Config 1
V1-1
V2-1
V3-1
V4-1
V5-1
Config 2
V1-2
V2-2
V3-2
V4-2
V5-2
Config 3
V1-3
V2-3
V3-3
V4-3
V5-3
Config 4
V1-4
V2-4
V3-4
V4-4
V5-4
Config 5
V1-5
V2-5
V3-5
V4-5
V5-5
Config 6
V1-6
V2-6
V3-6
V4-6
V5-6
This table defines 6 standard configurations for testing. In later tests, the lab
will set up a Config-1 system, a Config-2 system, etc., and will do its
compatibility testing on these systems. The variables might be software or
hardware choices. For example, Var 1 could be the operating system (V1-1 is
Win 2000, V1-2 is Win ME, etc.) Var 2 could be how much RAM on the
computer under test (V2-1 is 128 meg, V2-2 is 32 meg., etc.). Var 3 could be
the type of printer, Var 4 the machine’s language setting (French, German,
etc.). Config planning tables are often filled in using the All Pairs algorithm.
504
Black Box Software Testing
Copyright © 2003
Cem Kaner & James Bach
Configuration Test Matrix
Test 1
Config 1
Config 2
Config 3
Config 4
Config 5
Pass
Pass
Pass
Pass
Pass
Fail
Pass
Pass
Pass
Pass
Pass
Test 2
Test 3
Pass
Pass
Pass
Test 4
Pass
Fail
Fail
Test 5
Fail
Pass
Fail
Config 6
Pass
Pass
Pass
This matrix records the results of a series of tests against the 6 standard
configurations that we defined in the Configuration Planning Table.
In this table, Config 1 has passed 3 tests, failed 1, and hasn’t yet been tested
with Test 2. Config 6 is still untested.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
505
Black Box Software Testing
Part 13.
Scripting:
An Industry Worst Practice
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
506
Scripting
• COMPLETE SCRIPTING is favored by people who believe that
repeatability is everything and who believe that with repeatable
scripts, we can delegate to cheap labor.
1 ____ Pull down the Task menu
2 ____ Select First Number
3 ____ Enter 3
4 ____ Enter 2
5 ____ Press return
6 ____ The program displays 5
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
507
Scripting: Another script format
Step
#
Check What to
?
do
What to
see
Design
notes
1.
____
Task menu
down
This starts the
blah blah test,
with the blah
blah goal
Black Box Software Testing
Pull down
task menu
Copyright ©
2003
Cem Kaner & James Bach
Observation
notes
508
Scripting: The Bus of Testing
– Scripting is the Greyhound Bus of software testing:
“Just relax and leave the thinking to us.”
– To the novice, the test script is the whole tour. The tester goes
through the script, start to finish, and thinks he’s seen what
there is to see.
– To the experienced tester, the test script is a tour bus. When
she sees something interesting, she stops the bus and takes a
closer look.
– One problem with a bus trip. It’s often pretty boring, and you
might spend a lot of time sleeping.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
509
Scripting (Manual Test Cases)
Worst of all worlds:
• Not automated
• Not maintainable
• No communication of analysis
• NOT REPEATABLE!
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
510
Scripting
Should we create scripts as pseudocode, in
preparation for test automation?
• Suggestion made at STAR 97
• But most (88%) of bugs were found during
creation of the scripts (my experience too).
Maybe the benefit lies in
structured exploration,
rather than regression.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
511
Never Do ANY kind of Scripting?
• Checklists (as distinct from scripts) have their place. For
example, think of releasing a product:
– Many very different tasks
– All of them must be done
– The task is rarely done, so many steps may be forgotten.
• I prefer to tell testers what to test (what issues to cover), not
how to do the tests. Teaching people “how” is a matter of
training, not something that I record time and time again in the
test plan. A checklist will sometimes be the right way to present
the list of issues.
• I think the level of detail is sufficient if I can successfully pass
the section to a reasonably experienced tester who is a little
familiar with the program and be confident that she can figure
out what the test cases are and how to run them.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
512
Black Box Software Testing
Part 14.
Requirements Analysis for
Test Documentation
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
513
Problems with the (allegedly) standard
approach
• It is essential to understand your requirements for
test documentation.
• Unless following a “standard” helps you meet your
requirements, it is empty at best, anti-productive at
worst.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
514
Problems with the (allegedly) standard approach
• IEEE Standard 829 for Software Test Documentation
– Test plan
– Test-design specification
– Test-case specification
•
•
•
•
•
•
•
Test-case specification identifier
Test items
Input specifications
Output specifications
Environmental needs
Special procedural requirements
Intercase dependencies
We often see
one or more
pages per
test case.
– Test-procedure specification
– Test-item transmittal report
– Test-log
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
515
Problems with the (allegedly) standard
approach
– What is the documentation cost per test case?
– What is the maintenance cost of the documentation, per test case?
– If software design changes create documentation maintenance
costs, how much inertia do we build into our system? How much
does extensive test documentation add to the cost of late
improvement of the software? How much should we add?
– What inertia is created in favor of invariant regression testing?
– Is this incompatible with exploratory testing? Do we always want
to discourage exploration?
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
516
Problems with the (allegedly) standard
approach
– What is the impact on high-volume test automation?
– How often do project teams start to follow 829 but then give it up
mid-project? What does this do to the net quality of the test
documentation and test planning effort?
– WHAT REQUIREMENTS DOES A STANDARD LIKE THIS
FULFILL?
– WHICH STAKEHOLDERS GAIN A NET BENEFIT FROM
IEEE STANDARD DOCUMENTATION?
– WHAT BENEFITS DO THEY GAIN, AND WHY ARE THOSE
BENEFITS IMPORTANT TO THEM?
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
517
Standards:
ISO 9000-3
• The basic thinking underlying ISO 9000 goes like this:
– Decide what you’re going to do (create and document a process).
– Do what you said you’d do.
– Document what you’ve done in a way that lets an auditor check
what you’ve done against your process.
• Risks
– It can be difficult to build in room for exploratory testing. To
some degree (perhaps a large degree), this depends on the auditor.
– The style and extent of documentation needed to clue in a third
party (auditor) might go beyond your internal needs. On finite
budgets, where do we draw the line?
– What is the best tradeoff between process development and test
execution? This must be a practical question, not a religious
question.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
518
Requirements
– There are many different notions of what a good set of test
documentation would include. Before spending a substantial
amount of time and resources, it’s worth asking what
documentation should be developed (and why?)
– Test documentation is expensive and it takes a long time to
produce. If you figure out some of your main requirements first,
you might be able to do your work in a way that achieves them.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
519
Defining documentation requirements
– Stakeholders, interests, actions, objects
•
•
•
•
–
–
–
–
Who would use or be affected by test documentation?
What interests of theirs does documentation serve or disserve?
What will they do with the documentation?
What types of documents are of high or low value?
Asking questions
Context-free questions
Context-free questions specific to test planning
Evaluating a plan
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
520
Discovering Requirements
• Requirements
– Anything that drives or constrains design
• Stakeholders
– Favored, disfavored, and neutral stakeholders
• Stakeholders’ interests
– Favored, disfavored, and neutral interests
• Actions
– Actions support or interfere with interests
• Objects
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
521
What is your group’s mission?
•
•
•
•
•
•
Find important problems
Assess quality
Certify to standard
Fulfill process mandates
Satisfy stakeholders
Assure accountability
•
•
•
•
•
•
Advise about QA
Advise about testing
Advise about quality
Maximize efficiency
Minimize time
Minimize cost
The quality of testing depends on which of these
possible missions matter and how they relate.
Many debates about the goodness of testing
are really debates over missions and givens.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
522
Test Docs Requirements Questions
• Is test documentation a product or tool?
• Is software quality driven by legal issues or by market
forces?
• How quickly is the design changing?
• How quickly does the specification change to reflect
design change?
• Is testing approach oriented toward proving conformance
to specs or nonconformance with customer expectations?
• Does your testing style rely more on already-defined tests
or on exploration?
• Should test docs focus on what to test (objectives) or on
how to test for it (procedures)?
• Should the docs ever control the testing project?
Black Box Software Testing
Copyright © 2003
Cem Kaner & James Bach
523
Test Docs Requirements Questions
• If the docs control parts of the testing project, should that
control come early or late in the project?
• Who are the primary readers of these test documents and how
important are they?
• How much traceability do you need? What docs are you tracing
back to and who controls them?
• To what extent should test docs support tracking and reporting
of project status and testing progress?
• How well should docs support delegation of work to new
testers?
• What are your assumptions about the skills and knowledge of
new testers?
• Is test doc set a process model, a product model, or a defect
finder?
524
Black Box Software Testing
Copyright © 2003
Cem Kaner & James Bach
Test Docs Requirements Questions
• A test suite should provide prevention, detection, and
prediction. Which is the most important for this project?
• How maintainable are the test docs (and their test
cases)? And, how well do they ensure that test changes
will follow code changes?
• Will the test docs help us identify (and revise/restructure
in face of) a permanent shift in the risk profile of the
program?
• Are (should) docs (be) automatically created as a
byproduct of the test automation code?
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
525
Example: Early vs. Late Planning
• Benefits of early planning
– Scheduling and staffing (unless things change)
– Early opportunity to review (if people can understand your work)
• Costs of early planning
– Any delays in finding bugs are expensive. You should start finding
bugs ASAP.
– Early investment in test case design is risky if the product design is
subject to change.
– You will keep learning over time. Planning early sometimes
precludes groups from inventing new tests later.
– Depth of testing should reflect risks, many of which are unclear until
mid-project.
• MY USUAL GOAL: Do just-in-time test planning without losing key
benefits of early work.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
526
Ultimately, write a mission statement
• Try to describe your core documentation requirements in one
sentence that doesn’t have more than three components.
• Examples:
– The test documentation set will primarily support our efforts to
find bugs in this version, to delegate work, and to track status.
– The test documentation set will support ongoing product and test
maintenance over at least 10 years, will provide training material
for new group members, and will create archives suitable for
regulatory or litigation use.
Black Box Software Testing
Copyright ©
2003
Cem Kaner & James Bach
527