Black Box Software Testing
Part 15.
Testing User Documentation
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 527
Copyright Notice
These slides are distributed under the Creative
Commons License.
In brief summary, you may make and distribute copies
of these slides so long as you give the original author
credit and, if you alter, transform or build upon this work,
you distribute the resulting work only under a license
identical to this one.
For the rest of the details of the license, see
http://creativecommons.org/licenses/bysa/2.0/legalcode.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 528
Documentation is an
Express Warranty
A warranty is a statement of fact, either articulated or
implied by law, respecting the quality or character of the
goods to be sold. Under the Uniform Commercial Code an
express warranty is:
2-313(a) Any affirmation of fact or promise made by
the seller to the buyer which relates to the goods
and becomes part of the basis of the bargain . . .
2-313(b) Any description of the goods which is
made part of the basis of the bargain . . .
2-313(c) Any sample or model which is made part of
the basis of the bargain.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 529
Documentation is an
Express Warranty
You can’t disclaim an express warranty -you are accountable for your claims.
Uniform Commercial Code 2-316 (1):
Words or conduct relevant to the creation of an express
warranty and words or conduct tending to negate or limit
warranty shall be construed whenever reasonable as
consistent with each other; but . . . negation or limitation
is inoperative to the extent that such construction is
unreasonable.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 530
Black Box Testing:
Testing Documentation
Doc testing is important because:
• Errors in the manual increase risks of legal liability.
• Testing the documentation improves the reliability
of the program.
• The documentation may be your mainstream test
plan and your most up-to-date specification.
• Confusion in the manual reflects confusion in the
program’s design.
Refer to Testing Computer Software, Chapter 10
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 531
Testing Documentation:
What to Test
•
Verify every statement of fact and every reasonable implication
• Check the placement and accuracy of figures
• Audit the completeness of the manual (check that every feature is
documented)
Track errors in the documentation in a way that is normal for the
Doc group. This probably doesn’t involve the bug tracking
system (but put code/doc mismatches there). If you give back
marked up manuscripts, keep photocopies of your markups.
Check your corrections against the next circulating draft of the
manual.
On average, you will cover 4 pages per hour in a reasonably
stable program. Your second pass will go more quickly because
the program is in better shape, but it will still take several
minutes per page because you will actually test every page.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 532
Testing Documentation:
Things to Say
• Your role is not editorial: you are not the
•
•
•
•
•
authority on style and layout.
Keep your tone non-judgmental.
Point out upcoming changes (design changes,
new error handling, data structures, etc.) that
might affect the manual. Mark these in
appropriate sections on the manuscript.
Name experts or references to consult when the
writer is confused.
Suggest examples.
Point to useful existing data files (for examples).
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 533
Testing Documentation:
Things to Say
•
When appropriate, you might do some writing. The writer
might or might not use what you have written. You might write
in two ways:
» words that you think belong in the manual “as is”
(you’re saying, “Here, say it like this and it will be
right.”)
» explanations that are background material for the
writer. Maybe a rough draft of what she’ll write.
• Note: the final review is a meeting just a few days before the
book goes to the printer. If you have many small-detail late
comments, offer the writer a chance to review them privately
with you a few days before the review.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 534
Documentation Testing:
On-Line Help
•
•
•
•
•
•
•
•
•
•
•
•
•
Contents of the help
Cross-reference jumps
Glossary lookups
Browse sequences
Graphic hotspots
Graphic display - color or resolution
Window size, e.g. compile at 1024x768 and display at 640x480
Procedure sequences
Balloon help / tool tips
Index
Search
Context jumps
Error messages
Refer to Testing
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
Computer Software, Chapter 10
All Rights Reserved. 535
Publisher Liability for
Content-Related Errors
Winter v. G.P. Putnam’s Sons, 938 F.2d 1033, (9th Circuit) 1991.
Winter became seriously ill from picking and eating mushrooms
after relying on The Encyclopedia of Mushrooms, published by
Putnam. Putnam did not verify the material in the book and did
not intentionally include the error.
• Putnam was not liable for errors in the book.
• Noted that Jeppesen cases consistently held publisher liable for
information error when information published was to be used as a tool.
• The court said that a software publisher might be liable for program that
does not work as intended.
ALM v. Van Nostrand Reinhold 480 NE.2d 1263, 1985. The Making
of Tools. Plaintiff used it, and a tool shattered, causing injury.
VNR not liable, but author of the book might be.
Liability might also attach if the program provides professional
services (such as tax preparation) and gives bad information.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 536
Warranties & Misrepresentations:
What Must You Test?
• Advertisements
• Published specifications
• Interviews
• Box copy
• Fax-backs
• Manual
• Help system
• Warranty
• Web pages
• Readme
• Advice given to customers on Internet, CompuServe or AOL
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 537
Black Box Software Testing
Part 16.
Managing Automated Software Testing
Several of these slides were developed by Doug Hoffman or in co-authorship with Doug Hoffman
for a course that we co-taught on software test automation.
Many of the ideas in this presentation were presented and refined in Los Altos Workshops on
Software Testing.
LAWST 5 focused on oracles. Participants were Chris Agruss, James Bach, Jack Falk, David
Gelperin, Elisabeth Hendrickson, Doug Hoffman, Bob Johnson, Cem Kaner, Brian Lawrence,
Noel Nyman, Jeff Payne, Johanna Rothman, Melora Svoboda, Loretta Suzuki, and Ned Young.
LAWST 1-3 focused on several aspects of automated testing. Participants were Chris Agruss, Tom
Arnold, Richard Bender, James Bach, Jim Brooks, Karla Fisher, Chip Groder, Elizabeth
Hendrickson, Doug Hoffman, Keith W. Hooper, III, Bob Johnson, Cem Kaner, Brian Lawrence,
Tom Lindemuth, Brian Marick, Thanga Meenakshi, Noel Nyman, Jeffery E. Payne, Bret
Pettichord, Drew Pritsker, Johanna Rothman, Jane Stepak, Melora Svoboda, Jeremy White, and
Rodney Wilson.
I’m indebted to James Whittaker, James Tierney, Harry Robinson, and Noel Nyman for additional
explanations of stochastic testing.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 538
Overview
• About Automated Software Testing
• The regression testing paradigm
• 19 common mistakes
• 27 questions about requirements
• Planning for short-term ROI
• 6 sometimes successful architectures
• Conclusions from LAWST
• Breaking away from the regression paradigm
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 539
Overview
In the mass-market software world, many efforts to automate testing have been
expensive failures. Paradigms that dominate the Information Technology and DoDdriven worlds don’t apply well to the mass-market paradigm. Our risks are different.
Solutions that are highly efficient for those environments are not at all necessarily
good for mass-market products.
The point of an automated testing strategy is to save time and money. Or to find more
bugs or harder-to-find bugs. The strategy works if it makes you more effective (you
find better and more-reproducible bugs) and more efficient (you find the bugs faster
and cheaper).
I wrote a lot of test automation code back in the late 1970’s and early 1980’s. (That’s
why WordStar hired me as its Testing Technology Team Leader when I came to
California in 1983.) Since 1988, I haven’t written a line of code. I’ve been managing
other people, and then consulting, seeing talented folks succeed or fail when
confronted with similar problems. I’m not an expert in automated test implementation,
but I have strengths in testing project management, and how that applies to test
automation projects. That level of analysis--the manager’s view of a technology and its
risks/benefits as an investment--is the point of this section.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 540
About Automated Testing
Source of test cases
• Old
• Intentionally new
• Random new
Size of test pool
• Small
• Large
• Exhaustive
Serial dependence among tests
• Independent
• Sequence is relevant
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 541
About Automated Testing
Evaluation strategy
• Comparison to saved result
• Comparison to an oracle
• Comparison to a computational or logical model
• Comparison to a heuristic prediction.
(NOTE: All oracles are heuristic.)
• Crash
• Diagnostic
• State model
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 542
Issues Faced in A Typical
Automated Test
• What is being tested?
• How is the test set up?
• Where are the inputs coming from?
• What is being checked?
• Where are the expected results?
• How do you know pass or fail?
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 543
The “Complete” Oracle
Test Inputs
Precondition Data
Precondition
Program State
Test Results
Test Results
Test
Oracle
System
Under
Test
Environmental
Inputs
Postcondition Data
Postcondition Data
Postcondition
Program State
Postcondition
Program State
Environmental
Results
Environmental
Results
Reprinted with permission of Doug Hoffman
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 544
Automated Software Test Functions
•
•
•
•
•
•
•
•
•
•
•
Automated test case/data generation
Test case design from requirements or code
Selection of test cases
Able to run two or more specified test cases
Able to run a subset of all the automated test cases
No intervention needed after launching tests
Automatically sets-up and/or records relevant test environment
Runs test cases
Captures relevant results
Compares actual with expected results
Reports analysis of pass/fail
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 545
Characteristics of
“fully automated” tests
•
A set of tests is defined and will be run together.
•
No intervention needed after launching tests.
•
Automatically sets-up and/or records relevant
test environment.
•
Obtains input from existing data files, random
generation, or another defined source.
•
Runs test exercise.
•
Captures relevant results.
•
Evaluates actual against expected results.
•
Reports analysis of pass/fail.
Not all automation is full automation.
Partial automation can be very useful.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 546
Capabilities of Automation Tools
Automated test tools combine a variety of
capabilities. For example, GUI regression
tools provide:
• capture/replay for easy manual creation of tests
• execution of test scripts
• recording of test events
• compare the test results with expected results
• report test results
Some GUI tools provide additional
capabilities, but no tool does everything well.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 547
Capabilities of Automation Tools
Here are examples of automated test tool capabilities:
• Analyze source code for bugs
• Design test cases
• Create test cases (from requirements or code)
• Generate test data
• Ease manual creation of test cases
• Ease creation/management of traceability matrix
• Manage testware environment
• Select tests to be run
• Execute test scripts
• Record test events
• Measure software responses to tests (Discovery Functions)
• Determine expected results of tests (Reference Functions)
• Evaluate test results (Evaluation Functions)
• Report and analyze results
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 548
Tools for Improving Testability by
Providing Diagnostic Support
• Hardware integrity tests. Example: power supply deterioration can
look like irreproducible, buggy behavior.
• Database integrity. Ongoing tests for database corruption, making
corruption quickly visible to the tester.
• Code integrity. Quick check (such as checksum) to see whether
part of the code was overwritten in memory.
• Memory integrity. Check for wild pointers, other corruption.
• Resource usage reports: Check for memory leaks, stack leaks,
etc.
• Event logs. See reports of suspicious behavior. Probably requires
collaboration with programmers.
• Wrappers. Layer of indirection surrounding a called function or
object. The automator can detect and modify incoming and outgoing
messages, forcing or detecting states and data values of interest.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 549
Automation Design
• Determine the goals of the automation
• Determine the capabilities needed to
achieve those goals
• Select automation components
• Set relationships between components
• Identify locations of components and events
• Sequence test events
• Evaluate and report results of test events.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 550
The Regression Testing Paradigm
The most commonly discussed automation approach:
• create a test exercise
• run it and inspect the output
• if the program fails, report a bug and try again later
• if the program passes the test, save the resulting outputs
• in future tests, run the program and compare the output to
the saved results. Report an exception whenever the
current output and the saved output don’t match.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 551
Is this Really Automation?
Analyze product
-human
Design test
-human
Run test 1st time
-human
Evaluate results
-human
Report 1st bug
-human
Save code
-human
Save result
-human
Document test
-human
Re-run the test
-MACHINE
Evaluate result
-MACHINE
(plus human if there’s any mismatch)
Maintain result
-human
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
Woo-hoo! We
really get the
machine to do
a whole lot of
our work!
(Maybe, but not
this way.)
All Rights Reserved. 552
Common Mistakes about Test
Automation
The paper (Avoiding Shelfware) lists 19 “Don’ts.”
For example,
Don’t expect to be more productive over the short term.
•
The reality is that most of the benefits from automation
don’t happen until the second release.
• It takes 3 to 10+ times the effort to create an automated
test than to just manually do the test. Apparent
productivity drops at least 66% and possibly over 90%.
• Additional effort is required to create and administer
automated test tools.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 553
GUI Automation is Expensive
•
Test case creation is expensive. Estimates run from 3-5 times the
time to create and manually execute a test case (Bender) to 3-10
times (Kaner) to 10 times (Pettichord) or higher (LAWST).
•
You usually have to increase the testing staff in order to generate
automated tests. Otherwise, how will you achieve the same
breadth of testing?
•
Your most technically skilled staff are tied up in automation
•
Automation can delay testing, adding even more cost (albeit
hidden cost.)
•
Excessive reliance leads to the 20 questions problem. (Fully
defining a test suite in advance, before you know the program’s
weaknesses, is like playing 20 questions where you have to ask
all the questions before you get your first answer.)
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 554
GUI Automation Pays off Late
• GUI changes force maintenance of tests
» May need to wait for GUI stablization
» Most early test failures are due to GUI changes
• Regression testing has low power
» Rerunning old tests that the program has passed is
less powerful than running new tests
» Old tests do not address new features
• Maintainability is a core issue because our main
payback is usually in the next release, not this one.
and
andSQM,
SQM,
SQM,LLC.
LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 555
Test Automation is Programming
Win NT 4 had 6 million lines of code, and 12
million lines of test code
Common (and often vendor-recommended)
design and programming practices for
automated testing are appalling:
• Embedded constants
• No modularity
• No source control
No documentation
• No requirements analysis
•
No wonder we fail
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 556
Requirements Analysis
Automation requirements are not just about the software
under test and its risks. To understand what we’re up to, we
have to understand:
• Software under test and its risks
• The development strategy and timeframe for the software under
test
• How people will use the software
• What environments the software runs under and their associated
risks
• What tools are available in this environment and their capabilities
• The regulatory / required recordkeeping environment
• The attitudes and interests of test group management.
• The overall organizational situation
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 558
Requirements Analysis
Requirement: “Anything that drives design
choices.”
The paper (Avoiding Shelfware) lists 27 questions.
For example,
Will the user interface of the application be
stable or not?
• Let’s analyze this. The reality is that, in many companies,
the UI changes late.
• Suppose we’re in an extreme case. Does that mean we
cannot automate cost effectively? No. It means that we
should do only those types of automation that will yield a
fast return on investment.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 559
You Can Plan for Short Term ROI
• Smoke testing
• Configuration testing
• Variations on a theme
• Stress testing
• Load testing
• Life testing
• Performance benchmarking
• Other tests that extend your reach
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 560
Six Sometimes-Successful
Automation Architectures
• Quick & dirty
• Equivalence testing
• Framework
• Data-driven
• Application-independent data-driven
• Real-time simulator with event logs
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 561
Quick and Dirty
• Smoke tests
• Configuration tests
• Variations on a theme
• Stress, load, or life testing
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 562
Equivalence Testing
A/B comparison
Random tests using an oracle
Regression testing is the weakest form
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 563
Framework-Based Architecture
Frameworks are code libraries that separate routine calls
from designed tests.
• modularity
• reuse of components
• hide design evolution of UI or tool commands
• partial salvation from the custom control problem
• independence of application (the test case) from user interface
details (execute using keyboard? Mouse? API?)
• important utilities, such as error recovery
For more on frameworks, see Linda Hayes’ book on automated
testing, Tom Arnold’s book on Visual Test, and Mark Fewster &
Dorothy Graham’s book “Software Test Automation.”
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 564
Data-Driven Tests
• Variables are data
• Commands are data
• UI is data
• Program’s state is data
• Test tool’s syntax is data
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 565
Data-Driven Architecture
•
•
•
•
•
In test automation, there are three interesting programs:
• The software under test (SUT)
• The automation tool that executes the automated test code
• The test code (test scripts) that define the individual tests
From the point of view of the automation software,
• The SUT’s variables are data
• The SUT’s commands are data
• The SUT’s UI is data
• The SUT’s state is data
Therefore it is entirely fair game to treat these implementation
details of the SUT as values assigned to variables of the automation
software.
Additionally, we can think of the externally determined (e.g.
determined by you) test inputs and expected test results as data.
Additionally, if the automation tool’s syntax is subject to change, we
might rationally treat the command set as variable data as well.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 566
Data-Driven Architecture
In general, we can benefit from separating the
treatment of one type of data from another with an
eye to:
• optimizing the maintainability of each
• optimizing the understandability (to the test case creator
or maintainer) of the link between the data and whatever
inspired those choices of values of the data
• minimizing churn that comes from changes in the UI,
the underlying features, the test tool, or the overlying
requirements
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 567
Data-Driven Architecture:
Calendar Example
Imagine testing a calendar-making program.
The look of the calendar, the dates, etc., can all be thought of
as being tied to physical examples in the world, rather than
being tied to the program. If your collection of cool calendars
wouldn’t change with changes in the UI of the software under
test, then the test data that define the calendar are of a different
class from the test data that define the program’s features.
–
Define the calendars in a table. This table should not be
invalidated across calendar program versions. Columns name
features settings, each test case is on its own row.
–
An interpreter associates the values in each column with a set
of commands (a test script) that execute the value of the cell
in a given column/row.
–
The interpreter itself might use “wrapped” functions, i.e.
make indirect calls to the automation tool’s built-in features.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 568
Data-Driven Architecture:
Calendar Example
This is a good design from the point of view of optimizing for
maintainability because it separates out four types of things that can
vary independently:
•
The descriptions of the calendars themselves come from real-world and
can stay stable across program versions.
•
The mapping of calendar element to UI feature will change frequently
because the UI will change frequently. The mappings (one per UI
element) are written as short, separate functions that can be
maintained easily.
•
The short scripts that map calendar elements to the program functions
probably call sub-scripts (think of them as library functions) that wrap
common program functions. Therefore a fundamental change in the
program might lead to a modest change in the software under test.
•
The short scripts that map calendar elements to the program functions
probably also call sub-scripts (think of them as library functions) that
wrap functions of the automation tool. If the tool syntax changes,
maintenance involves changing the wrappers’ definitions rather than
the scripts.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 569
Data Driven Architecture
Note with this example:
• we didn’t run tests twice
• we automated execution, not evaluation
• we saved SOME time
• we focused the tester on design and results, not execution.
Other table-driven cases:
• automated comparison can be done via a pointer in the
table to the file
• the underlying approach runs an interpreter against table
entries.
» Hans Buwalda and others use this to create a structure that is
natural for non-tester subject matter experts to manipulate.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 570
Application-Independent Data-Driven
• Generic tables of repetitive types
• Rows for instances
• Automation of exercises
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 571
Real-time Simulator
• Test embodies rules for activities
• Stochastic process
• Possible monitors
•
•
•
•
Code assertions
Event logs
State transition maps
Oracles
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 572
Think About:
• Automation is software development.
• Regression automation is expensive and can be
inefficient.
• Automation need not be regression--you can
run new tests instead of old ones.
• Maintainability is essential.
• Design to your requirements.
• Set management expectations with care.
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 573
GUI Regression Strategies:
Some Papers of Interest
Chris Agruss, Automating Software Installation Testing
James Bach, Test Automation Snake Oil
Hans Buwalda, Testing Using Action Words
Hans Buwalda, Automated testing with Action Words: Abandoning Record
& Playback
Elisabeth Hendrickson, The Difference between Test Automation Failure
and Success
Cem Kaner, Avoiding Shelfware: A Manager’s View of Automated GUI
Testing
John Kent, Advanced Automated Testing Architectures
Bret Pettichord, Success with Test Automation
Bret Pettichord, Seven Steps to Test Automation Success
Keith Zambelich, Totally Data-Driven Automated Testing
andSQM,
SQM,LLC.
LLC.
Copyright © 1994-2004 Cem Kaner and
All Rights Reserved. 574
Black Box Software Testing
Part 17
Test Strategy Planning
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 576
Test Strategy
“How we plan to cover the product so as to
develop an adequate assessment of quality.”
A good test strategy is:
• Diversified
• Specific
• Practical
• Defensible
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 577
Test Strategy
• Makes use of test techniques.
• May be expressed by test procedures and
cases.
• Not to be confused with test logistics, which
involve the details of bringing resources to bear
on the test strategy at the right time and place.
• You don’t have to know the entire strategy in
advance. The strategy can change as you learn
more about the product and its problems.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 578
Test Cases/Procedures
• Test cases and procedures should manifest
the test strategy.
• If your strategy is to “execute the test suite I
got from Joe Third-Party”, how does that
answer the prime strategic questions:
• How will you cover the product and
assess quality?
• How is that practical and justified with
respect to the specifics of this project and
product?
• If you don’t know, then your real strategy is
that you’re trusting things to work out.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 579
Diverse Half-Measures
• There is no single technique that finds all bugs.
• We can’t do any technique perfectly.
• We can’t do all conceivable techniques.
Use “diverse half-measures”-- lots of different
points of view, approaches, techniques, even
if no one strategy is performed completely.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 580
Heuristics from James Bach’s Test Plan
Evaluation Model, www.satisfice.com
1. Look for the important problems first
2. Focus mainly on areas of potential technical risk
3. Address configuration, operation , observation,
and evaluation of the product
4. Diversify test techniques and perspectives
5. Specify test data design and generation
6. Don’t just run preplanned tests
7. Test against stated and implied requirements
8. Collaborate with outside people for testing ideas
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 581
Heuristics from James Bach’s Test Plan
Evaluation Model, www.satisfice.com
9. Work with developers to improve product
testability
10. Highlight the non-routine, project-specific aspects
of the testing strategy and the testing project
11. Use people and tools for what they do best
12. Identify dependencies in the testing schedule
13. Keep testing off the critical path for release
14. Make the feedback loop with development short
15. Learn what people outside of testing think of the
product quality
16. Review all test materials
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 582
Heuristics for Test Plan Evaluation
Heuristic
1. Testing should be optimized to find
important problems fast, rather than
attempting to find all problems with
equal urgency.
Basis for the Heuristic
The later in the project that a
problem is found, the greater the
risk that it will not be safely fixed in
time to ship. The sooner a problem
is found after it is created, the lesser
the risk of a bad fix.
2. Test strategy should focus most
effort on areas of potential technical
risk, while still putting some effort
into low risk areas just in case the
risk analysis is wrong.
Complete testing is impossible, and
we can never know if our perception
of technical risk is completely
accurate.
3. Test strategy should address test
platform configuration, how the
product will be operated, how the
product will be observed, and how
observations will be used to evaluate
the product.
Sloppiness or neglect within any of
these four basic testing activities
will increase the likelihood that
important problems will go
undetected.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 584
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
4. Test strategy should be diversified
in terms of test techniques and
perspectives. Methods of evaluating
test coverage should take into
account multiple dimensions of
coverage, including structural,
functional, data, platform, operations,
and requirements.
No single test technique can reveal all
important problems in a linear
fashion. We can never know for sure
if we have found all the problems that
matter. Diversification minimizes the
risk that the test strategy will be blind
to certain kinds of problems.
Use diverse half-measures to go after
low-hanging fruit.
5. The test strategy should specify
how test data will be designed and
generated.
It is common for the test strategy to
be organized around functionality or
code, leaving it to the testers to
concoct test data on the fly. Often that
indicates that the strategy is too
focused on validating capability and
not focused enough on reliability.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 585
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
6. Not all testing should be prespecified in detail. The test strategy
should incorporate reasonable
variation and make use of the
testers’ ability to use situational
reasoning to focus on important,
but unanticipated problems.
A rigid test strategy may make it more
likely that a particular subset of problems
will be uncovered, but in a complex system
it reduces the likelihood that all important
problems will be uncovered. Reasonable
variability in testing, such as that which
results from interactive, exploratory
testing, increases incidental test coverage,
without substantially sacrificing essential
coverage.
7. It is important to test against
implied requirements—the full
extent of what the requirements
mean, not just what they say.
Testing only against explicit written
requirements will not reveal all important
problems, since defined requirements are
generally incomplete and natural language
is inherently ambiguous.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 586
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
8. The test project should promote
collaboration with all other
functions of the project, especially
developers, technical support, and
technical writing. Whenever
possible, testers should also
collaborate with actual customers
and users, in order to better
understand their requirements.
Other teams and stakeholders often
have information about product
problems or potential problems that
can be of use to the test team. Their
perspective may help the testers
make a better analysis of risk.
Testers may also have information
that is of use to them.
9. The test project should consult
with development to help them
build a more testable product.
The likelihood that a test strategy
will serve its purpose is profoundly
affected by the testability of the
product.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 587
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
10. A test plan should highlight the
non-routine, project-specific
aspects of the test strategy and
test project.
Virtually every software project
worth doing involves special
technical challenges that a good
test effort must take into account. A
completely generic test plan usually
indicates a weak test planning
process. It could also indicate that
the test plan is nothing but
unchanged boilerplate.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 588
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
11. The test project should use
humans for what humans do well
and use automation for what
automation does well. Manual
testing should allow for
improvisation and on the spot
critical thinking, while automated
testing should be used for tests
that require high repeatability, high
speed, and no judgment.
Many test projects suffer under the
false belief that human testers are
effective when they use exactingly
specified test scripts, or that test
automation duplicates the value of
human cognition in the test
execution process. Manual and
automated testing are not two forms
of the same thing. They are two
entirely different classes of test
technique.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 589
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
12. The test schedule should be
represented and justified in such a
way as to highlight any
dependencies on the progress of
development, the testability of the
product, time required to report
problems, and the project team’s
assessment of risk.
A monolithic test schedule in a test
plan often indicates the false belief
that testing is an independent
activity. The test schedule can stand
alone only to the extent that the
product the highly testable,
development is complete, and the
test process is not interrupted by
the frequent need to report
problems.
13. The test process should be
kept off of the critical path to the
extent possible. This can be done
by testing in parallel with
development work, and finding
problems worth fixing faster than
the developers fix them.
This is important in order to deflect
pressure to truncate the testing
process.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 590
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
14. The feedback loop between
testers and developers should be
as tight as possible. Test cycles
should be designed to provide
rapid feedback to developers
about recent additions and
changes they have made before a
full regression test is commenced.
Whenever possible testers and
developers should work physically
near each other.
This is important in order to maximize
the efficiency and speed of quality
improvement. It also helps keep testing
off of the critical path.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 591
Heuristics for Test Plan Evaluation
Heuristic
Basis for the Heuristic
15. The test project should employ
channels of information about
quality other than formal testing in
order to help evaluate and adjust
the test project. Examples of these
channels are inspections, field
testing, or informal testing by
people outside of the test team.
By examining product quality
information gathered through various
means beyond the test team, blind
spots in the formal test strategy can be
uncovered.
16. All documentation related to
the test strategy, including test
cases and procedures, should be
undergo review by someone other
than the person who wrote them.
The review process used should
be commensurate with the
criticality of the document.
Tunnel-vision is the great occupational
hazard of testing. Review not only helps
to reveal blind spots in test design, but
it can also help promote dialog and peer
education about test practices.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 592
Evaluating Your Plan:
Context Free Questions
Based on: The CIA’s Phoenix Checklists (Thinkertoys, p. 140)
and Bach’s Evaluation Strategies (Rapid Testing Course notes)
• Can you solve the whole problem? Part of the problem?
• What would you like the resolution to be? Can you picture it?
• How much of the unknown can you determine?
• What reference data are you using (if any)?
• What product output will you evaluate?
• How will you do the evaluation?
• Can you derive something useful from the information you have?
• Have you used all the information?
• Have you taken into account all essential notions in the problem?
• Can you separate the steps in the problem-solving process? Can you
determine the correctness of each step?
• What creative thinking techniques can you use to generate ideas? How
many different techniques?
• Can you see the result? How many different kinds of results can you see?
• How many different ways have you tried to solve the problem?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 593
Evaluating Your Plan:
Context Free Questions
•
•
•
•
•
•
•
•
•
•
•
What have others done?
Can you intuit the solution? Can you check the results?
What should be done?
How should it be done?
Where should it be done?
When should it be done?
Who should do it?
What do you need to do at this time?
Who will be responsible for what?
Can you use this problem to solve some other problem?
What is the unique set of qualities that makes this
problem what it is and none other?
• What milestones can best mark your progress?
• How will you know when you are successful?
• How conclusive and specific is your answer?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 594
Black Box Software Testing
Part 18.
Planning Testing Projects
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 596
Planning and Negotiating
the Testing Project
• The notes here focus on your process of negotiating
resources and quality level.
• Please see Testing Computer Software, Chapter 13 for a
detailed discussion of planning and testing tasks across
the time line of the project.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 597
Early Planning (1)
Things you can do very early in the project:
• Analyze any requirements documents for testability,
ambiguity.
• Facilitate inspection and walkthrough meetings.
• Prepare a preliminary list of hardware configurations
and start arranging for loaners.
• Ask for testing support code, such as debug monitors,
memory meters, support for testpoints.
• Ask for a clear error handling message structure.
• Discuss the possibility of code coverage measurement
(which will require programmer support).
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 598
Early Planning (2)
Things you can do very early in the project:
• Prepare for test automation. This involves reaching
agreements in principle on breadth of automation
and automation support staffing level.
• Order automation support equipment.
• Order external test suites.
• Learn about the product’s market and competition.
• Evaluate coding tools that facilitate automation (e.g.
test 3rd party custom controls against MS-Test)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 599
First Principles in Planning
• You can’t nail everything down.
• You will face difficult prioritization choices, and
many constraints will be out of your direct control.
• You can influence several constraints by opening up
your judgments to other stakeholders in the company.
This is more than getting a sign-off.
• Reality is far more important than your ability to cast
blame.
• Your task is to manage project-level risks. This
includes risks to the project’s cost, schedule, and
interpersonal dynamics, as well as to the product’s
feature set and reliability.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 600
Deciding Your Depth of Testing
You have three key objectives:
1 Achieve a uniform and well-understood minimum level of
testing.
2 Be explicit about the level of testing for each area:
» mainstream
» exploratory
» formally planned
» structured regression
3 Reach corporate agreement on the level of testing for each area
Just like bug deferrals, depth-of-testing restrictions
must rest on sound business decisions.
This should not be your decision to make.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 602
Identify the Testing Tasks
Develop a project task list with your staff. Remember, you are
looking for realism in estimates of complexity, difficulty, and time.
• Make the listing and estimation process public, and allow public
comment.
• Identify all potential sources of information.
• List all main functional areas, and all other cross-functional
approaches that you will take in testing the program.
• List every task that appears to need more than 1/2 day
(Probably you will do this by a group brainstorming session
on flipcharts, listing sources on one chart. Gather the sources.
List areas and approaches on a few charts. Make one chart for
each area of work, and list tasks for each chart. Tentatively
assign times to each task, possibly by a museum tour.)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 603
Estimate the Project Time
1 Assign time estimates to each task. Invite programmers, marketers,
and project managers to walk through the charts and provide their
own estimates. Explore the bases of differences. You might have
underestimated a task’s complexity. Or you might be seeing a priority
disagreement. Or wishful thinking.
2 Make your best-estimate task list. Provide the time needed to achieve
formal, planned testing for each area. Include estimates for structured
regression for key areas. This number is probably absurdly huge.
3 Add to the list your suggestions for time-cuts to guerrila-level and
mainstream-level testing for selected areas.
4 Circulate your lists to all stakeholders, call one or more planning
meetings, and reach agreements on the level of testing for each area.
Keep cutting tasks and/or adding staff until you reach an achievable
result.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 604
Estimate the Project Time
Don’t forget:
•
Budget for meetings and administrative overhead (10-30%).
•
Budget for bug regression (5-15%).
•
Budget for down time.
•
Budget for holidays and vacations.
•
Never develop a budget that relies on overtime. Leave the
overtime free for discretionary additional work by the
individual tester.
•
Don’t let yourself be bullied or embarrassed, and don’t bully
or embarrass your staff, into making underestimates. To
achieve a result, insist on cutting tasks, adding staff, or
finding realistic ways to improve productivity.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 605
Identify Dependencies & Milestones
You can’t start reviewing the manual until you
receive it, right? List these dependencies and
point out, in every project meeting, what is due to
you that is not yet delivered and holding you up.
Try to reach verifiable, realistic definitions for
each milestone. For example, if “gamma” is 6
weeks before release, the product can’t have
more than 6 weeks of schedule risk at gamma.
Negotiate a definition.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 606
Monitor Progress
•
Put the tasks and agreed-to durations on a timeline and
review progress of all staff every week.
•
When prerequisite materials aren’t provided, find other
tasks to do instead. When you can’t meet the schedule, due
to absent materials, publish new estimates.
•
When design changes invalidate your estimates, publish
new estimates.
•
If you’re falling consistently behind (e.g. due to
underestimated overhead), publish the problem and ask for
fewer tasks, more staff, or some other realistic solution.
•
If one of your staff has trouble with an area, recognize it and
deal with it.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 607
Control Late-Stage Issues
•
Watch out for late changes. Encourage people
to be cautious about late bug fixes, and supercautious about late design changes.
•
Provide interim and final deferred-bug reviews.
•
Take a final inventory of your testing coverage.
•
Carry out final integrity tests.
•
You may have to assess and reporting the
reliability of the tested product.
•
Plan for post-release processes. Develop a
release kit for product support.
•
Don’t forget the party.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 608
Black Box Software Testing
Part 19.
Metrics and Measurement Theory
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 610
We Aren’t Collecting Data.
Capers Jones, Patterns of Software Systems Failure & Success:
The number one root cause of cancellations, schedule slippages, and cost
overruns is the chronic failure of the software industry to collect accurate
historical data from ongoing and completed projects. This failure means
that the vast majority of major software projects are begun without
anyone having a solid notion of how much time will be required.
Software is perhaps the only technical industry where neither clients,
managers, nor technical staff have any accurate quantitative data
available to them from similar projects when beginning major
construction activities. . . .
A result that is initially surprising but quite common across the industry
is to discover that the software management community within a
company knows so little about the technology of software planning and
estimating that they do not even know of the kinds of tools that are
commercially available.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 611
Question
Imagine being on the job. Your local PBH
(pointy-haired boss) drops in and asks:
“So, how much testing have you
gotten done?”
Please write down an answer. Feel free to
use fictitious numbers but (except for the
numbers) try to be realistic in terms of the
type of information that you would provide.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 612
How do we Measure Extent of Testing?
Before we can measure something, we need some
sense of what we’re measuring. It’s easy to come up
with “measurements” but we have to understand the
relationship between the thing we want to measure
and the statistic that we calculate to “measure” it.
If we want to measure the “extent of
testing”, we have to start by
understanding what we mean by
“extent of testing.”
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 613
What is measurement?
Measurement is the assignment of
numbers to objects or events according
to a rule derived from a model or theory.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 614
The Question is
Remarkably Ambiguous
Common answers are based on the:
Product • We’ve tested 80% of the lines of code.
Plan • We’ve run 80% of the test cases.
Results • We’ve discovered 593 bugs.
Effort • We’ve worked 80 hours a week on this
for 4 months. We’ve run 7,243 tests.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 615
The Question is
Remarkably Ambiguous
Common answers are based on the:
Obstacles • We’ve been plugging away but we can’t be
efficient until X, Y, and Z are dealt with.
Risks • We’re getting a lot of complaints from beta
testers and we have 400 bugs open. The
product can’t be ready to ship in three days.
Quality of • Beta testers have found 30 bugs that we
missed. Our regression tests seem ineffective.
Testing
History • At this milestone on previous projects, we
had fewer than 12.3712% of the bugs found
across
still open. We should be at that percentage
projects
on this product too.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 616
A Framework for Measurement
A measurement involves at least 10 factors:
• Attribute to be measured
» appropriate scale for the attribute
» variation of the attribute
• Instrument that measures the attribute
» scale of the instrument
» variation of measurements made with this instrument
• Relationship between the attribute and the instrument
• Likely side effects of using this instrument to measure
this attribute
• Purpose
• Scope
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 617
Framework for Measurement
Attribute • Extent of testing – What does that mean?
Instrument • What should we count? Lines? Bugs? Test
cases? Hours? Temper tantrums?
Mechanism • How will increasing “extent of testing” affect
the reading (the measure) on the instrument?
Side Effect • If we do something that makes the measured
result look better, will that mean that we’ve
actually increased the extent of testing?
Purpose • Why are we measuring this? What will we do
with the number?
Scope • Are we measuring the work of one tester? One
team on one project? Is this a cross-project
metrics effort? Cross-departmental research?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 618
Simple measurement
You have a room full of tables that appear to
be the same length. You want to measure their
lengths.
You have a one-foot ruler.
You use the ruler to measure the lengths of a
few tables. You get:
• 6.01 feet
• 5.99 feet
• 6.05 feet
You conclude that the tables are “6 feet” long.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 619
Simple measurement (2)
Note the variation
• Measurement errors using the ruler
• Manufacturing variation in the tables
Note the rule:
• We are relying on a direct matching operation and
on some basic axioms of mathematics
» The sum of 6 one-foot ruler-lengths is 6.
» A table that is 6 ruler-lengths long is twice as long
as one that is 3 ruler-lengths long.
These rules don’t always apply. What do we do
when we have something hard to measure?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 620
Attributes and Instruments
Length
Ruler
Duration
Stopwatch
Speed
Ruler / Stopwatch
Sound energy
Sound level meter
Loudness
Sound level comparisons by humans
Tester goodness
??? Bug count ???
Code complexity
??? Branches ???
Extent of testing???
???
----Product coverage
?? Count statements / branches tested ??
----Proportion of bugs
found
?? Count bug reports or graph bug
curves??
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 621
Surrogate measures
"Many of the attributes we wish to study do not have
generally agreed methods of measurement. To
overcome the lack of a measure for an attribute, some
factor which can be measured is used instead. This
alternate measure is presumed to be related to the
actual attribute with which the study is concerned.
These alternate measures are called surrogate
measures."
Mark Johnson’s MA Thesis
“Surrogates” provide unambiguous assignments of
numbers according to rules, but they don’t provide an
underlying theory or model that relates the measure to
the attribute allegedly being measured.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 622
Consider bug counts
• Do bug counts measure testers?
• Do bug counts measure thoroughness of testing?
• Do bug counts measure the effectiveness of an
automation effort?
• Do bug counts measure how near we are to being
ready to ship the product?
How would we know?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 623
Bug counts and testers
To evaluate an instrument that is supposed to measure
an attribute, we have to ask two key questions:
• What underlying mechanism, or fundamental relationship,
justifies the use of the reading we take from this instrument as
a measure of the attribute? If the attribute increases by 20%,
what will happen to the reading?
• What can we know from the instrument reading? How tightly
is the reading traceable to the underlying attribute? If the
reading increases by 20%, does this mean that the attribute
has increased 20%. If the linkage is not tight, we risk serious
side effects as people push the reading (the “measurement”)
up and down without improving the underlying attribute.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 624
Bug counts and testers: mechanism?
Suppose we could improve testing by 20%.
This might mean that:
• We find more subtle bugs that are important but that
require more thorough investigation and analysis
• We create bug reports that are more thorough, better
researched, more descriptive of the problem and
therefore more likely to yield fixes.
• We do superb testing of a critical area that turns out
to be relatively stable.
The bug counts might even go down, even
though tester goodness has gone up.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 625
Bug Counts & Testers: Side effects
What if you could increase the count of reported bugs
by 20%?
If you reward testers for higher bug counts, won’t
you make changes like these more likely?
• Testers report easier-to-find, more superficial bugs
• Testers report multiple instances of the same bug
• Programmers dismiss design bugs as non-bugs that
testers put in the system to raise their bug counts
• No one will work on the bug tracking system or other
group infrastructure.
• Testers become less willing to spend time coaching
other testers.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 626
Bug counts and extent of testing?
Bugs Per Week
What Is This Curve?
Week
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 627
Bug counts and extent of testing
Attribute •
Instrument •
Mechanism •
Not sure. Maybe we’re thinking of percentage
found of the total population of bugs in this
product.
Bugs found. (Variations: bugs found this week,
etc., various numbers based on bug count.)
If we increase the extent of testing, does that
result in more bug reports? Not necessarily.
Side Effect •
If we change testing to maximize the bug count,
does that mean we’ve achieved more of the
testing?
Maybe in a trivial sense, but what if we’re finding lots of simple bugs
at the expense of testing for a smaller number of harder-to-find serious
bugs?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 628
Bug curves and extent of testing
Attribute • We have a model of the rate at which new bugs
will be found over the life of the project.
Instrument • Bugs per week. A key thing that we look at is the
agreement between the predictive curve and the
actual bug counts.
Mechanism • As we increase the extent of testing, will our bug
numbers conform to the curve? Not necessarily.
It depends on the bugs that are left in the product.
Side Effect • If we do something that makes the measured
result look better, will that mean that we’ve
actually increased the extent of testing? No, no,
no. See side effect discussion.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 629
Side effects of bug curves
Earlier in testing: (Pressure is to increase bug counts)
• Run tests of features known to be broken or incomplete.
• Run multiple related tests to find multiple related bugs.
• Look for easy bugs in high quantities rather than hard
bugs.
• Less emphasis on infrastructure, automation architecture,
tools and more emphasis of bug finding. (Short term
payoff but long term inefficiency.)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 630
Side effects of bug curves
Later in testing: (Pressure is to decrease new bug rate)
• Run lots of already-run regression tests
• Don’t look as hard for new bugs.
• Shift focus to appraisal, status reporting.
• Classify unrelated bugs as duplicates
• Class related bugs as duplicates (and closed), hiding key data
•
•
•
•
•
•
about the symptoms / causes of the problem.
Postpone bug reporting until after the measurement checkpoint
(milestone). (Some bugs are lost.)
Report bugs informally, keeping them out of the tracking system
Testers get sent to the movies before measurement checkpoints.
Programmers ignore bugs they find until testers report them.
Bugs are taken personally.
More bugs are rejected.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 631
Bug curve counterproductive?
Bugs Per Week
Shouldn't We Strive For This ?
Week
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 632
Code coverage
Coverage measures the amount of testing done of
a certain type. Since testing is done to find bugs,
coverage is a measure of your effort to detect a
certain class of potential errors:
• 100% line coverage means that you tested for every bug
that can be revealed by simple execution of a line of code.
• 100% branch coverage means you will find every error
that can be revealed by testing each branch.
• 100% coverage should mean that you tested for every
possible error. This is obviously impossible.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 633
Benefits of coverage
Before I attack coverage measures, let me
acknowledge that they are often useful.
• Many projects achieve low statement coverage, as little as
2% at one well-known publisher that had done (as measured
by tester-hours) extensive testing and test automation. The
results from checking statement coverage caused a complete
rethinking of the company’s automation strategy.
• Coverage tools can point to unreachable code or to code that
is active but persistently untested.
Coverage tools can provide powerful diagnostic
information about the testing strategy, even if they are
terrible measures of the extent of testing.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 634
Analysis: Statement/Branch Coverage
Attribute •
Extent of testing –
How much of the product have we tested?
Instrument •
Mechanism •
Count statements and branches tested
If we do more testing and find more bugs, does
that mean that our line count will increase? Not
necessarily. Example—configuration tests.
Side Effect •
If we design our tests to make sure we hit more
lines, does that mean we’ll have done more
extensive testing? Maybe in a trivial sense, but
we can achieve this with weaker tests that find
fewer bugs.
Purpose •
Not specified
Scope •
Not specified
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 635
Statement / branch coverage
just test the flowchart
You’re not testing::
» data flow
» tables that determine control flow in table-driven code
» side effects of interrupts, or interaction with background tasks
» special values, such as boundary cases. These might or might
»
»
»
»
»
»
not be tested.
unexpected values (e.g. divide by zero)
user interface errors
timing-related bugs
compliance with contracts, regulations, or other requirements
configuration/compatibility failures
volume, load, hardware faults
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 636
If we use “coverage”?
• If we improve testing by 20%, does this result in
a 20% increase in “coverage”? Does it necessarily
result in ANY increase in “coverage”?
• If we increase “coverage” by 20%, does this
mean that there was a 20% improvement in the
testing?
• If we achieve 100% “coverage”, do we really
think we’ve found all the bugs?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 637
Side effects and “coverage”
• Without a mechanism that ties changes in the
attribute being measured to changes in the reading
we get from the instrument, we have a “measure”
that is ripe for abuse.
• People will optimize what is tracked. If you track
“coverage”, the coverage number will go up, but (as
Marick has often pointed out) the quality of testing
might go down.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 638
Having framed the problem . . .
Do I have a valid, useful measure of the extent of
testing?
• Nope, not yet.
• The development and validation of a field’s measures
takes time.
In the meantime, what do we do?
• We still have to manage projects, monitor progress, make
decisions based on what’s going on - - - so ignoring the
measurement question is not an option.
• Let’s look at strategies of some seasoned test managers
and testers.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 639
One approach: Balanced scorecard
“Coverage” measures are popular because they
provide management with essential (if incorrect)
feedback about the progress of testing.
Rather than reporting a single not-very-representative
measure of testing progress, consider adopting a
“balanced scorecard” approach. Report:
•
•
•
•
a small number (maybe 10) of different measures,
none of them perfect,
all of them different from each other, and
all of them reporting progress that is meaningful to you.
Together, these show a pattern that might more
accurately reflect progress.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 640
One Approach: Balanced Scorecard
• For 101 examples of possible coverage measures,
that might be suitable for balancing, see
“Software Negligence and Testing Coverage” at
www.kaner.com. These are merged in a list with
over 100 additional indicators of extent of testing
in the paper, “Measurement Issues & Software
Testing”, which is included in the proceedings.
• Robert Austin criticizes even this balanced
scorecard approach. It will still lead to abuse, if
the measures don’t balance each other out.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 641
“Extent” as a
Multidimensional Problem
We developed the 8 aspects (or dimensions) of “extent
of testing” by looking at the types of measures of
extent of testing that we were reporting.
The accompanying paper reports a few hundred
“measures” that fit in the 8 categories
• product coverage
- plan / agreement
• effort
- results
• obstacles
- risks
• quality of testing
- project history
ALL of them have problems
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 642
“Extent” as a Multidimensional
Problem
So, look at testing progress reports actually used in
the field. Do we see focus on these individual
measures?
• Often, NOT.
• Instead, we see reports that show a pattern of information
of different types, to give the reader / listener a sense of
the overall flow of the testing project.
• Several examples in the paper, here are two of them:
» Hendrickson’s component map
» Bach’s project dashboard
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 643
Project Report /
Component Map (Hendrickson)
Page 1 --- Issues that need management attention
Page 2 --- Component map
Page 3 --- Bug statistics
Component Test Tester Total
Type
Tests
Planned /
Created
Tests
Passed /
Failed /
Blocked
Time
Budget
Time Projected Notes
Spent for Next
Build
We see in this report:
- Progress against plan
- Obstacles / Risks
- Effort
- Results
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 644
Bach’s Dashboard
Updated
Testing Dashboard
Area
Effort
Coverage
Coverage
Planned
Achieved
Build
11/1/00
32
Quality
Comments
1345, 1410
File/edit
High
High
Low

View
Low
Med
Med

Insert
Blocked
Med
Low

1621
We see coverage of areas, progress against plan, current effort, key results and
risks, and obstacles.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 645
Suggested Lessons
• Bug count metrics cover only a narrow slice of testing
progress. There are LOTS of alternatives.
• Simple charts can carry a lot of useful information and
lead you to a lot of useful questions.
• Report multidimensional patterns, rather than single
measures or a few measures along the same line.
• Think carefully about the potential side effects of your
measures.
• Listen critically to reports (case studies) of success with
simple metrics. If you can’t see the data and don’t know
how the data were actually collected, you might well be
looking at results that were sanitized by working staff
(as a side effect of the imposition of the measurement
process).
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 646
Black Box Software Testing
Part 20.
Defect Life Cycles
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 648
Why Track Defects?
• Get things that should be fixed, fixed
• Capture valuable information
• Identify responsibilities
• Provide data about quality
• Focal point for decision making
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 649
Tasks In Defect Tracking
• Quickly notify appropriate people
• Avoid forgetting about errors
• Assure important issues get resolved
• Minimize errors going unfixed due to
poor communication
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 650
What Are We Tracking?
•
Symptoms of possible errors
•
Severity of possible errors
•
Priority for fixing errors
•
Events related to the reported error
•
No duplicate reports (for single error)
•
Responsibilities
•
Resolutions
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 651
Forces Acting on Defect Tracking
•
•
•
•
•
•
•
•
•
•
•
•
Severity and Priority
Features and Quality
Duplicate reports
Project status reporting
Project management / Developer tasking
Metrics
Issue tracking and incident reporting
Irreproducible problems
Enhancement requests
Design errors
Deferring problems
Branches and release versions
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 652
Problem of Similar Bugs
You File a
Defect Report
You Ignor It
Never Fixed
It's a
It Gets Fixed
New Defect
(Consumer Risk)
Same Old Waste of Time
Gets Fixed
Defect
(Producer Risk)
Anyway
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 653
Defect Tracking Activities
•Submitting
•Assigning
•Fixing
•Verifying
•Resolving
•Administering
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 654
Example Defect Life Cycle
Issue
Perceived
Entered
Submitted
Accepted
Not an issue,
Duplicate,
Will not fix,
Cannot reproduce,
Withdrawn
Assigned
Fix Later
Not fixed
Finished
Deferred
It’s Later
Fixed
Fix verified
Closed
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 655
Defect Resolutions
• Common resolutions include:
• Pending: the bug is still being worked on.
• Fixed: the programmer says it’s fixed. Now you should check it.
• Cannot reproduce: The programmer can’t make the failure
•
•
•
•
•
happen. You must add details, reset the resolution to Pending,
and notify the programmer.
Deferred: It’s a bug, but we’ll fix it later.
As Designed: The program works as it’s supposed to.
Need Info: The programmer needs more info from you. She has
probably asked a question in the comments.
Duplicate: This is just a repeat of another bug report (XREF it on
this report.) Duplicates should not close until the duplicated bug
closes.
Withdrawn: The tester who reported this bug is withdrawing the
report.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 656
Project Planning
Part 21.
Testing Impossible Software Under
Impossible Deadlines
Cem Kaner, Software Testing Analysis East, 1999
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 658
Introduction
I didn’t come up with the idea for this talk--the
STAR folk suggested it as something that I might be
able to provide a few insights into. After all, this is a
chronic complaint among testers. . . .
Well, here goes. How should you deal with a
situation in which the software comes to you
undocumented on a tight deadline?
I think that the answer depends on the causes of
the situation that you’re in. Your best solution might
be technical or political or resume-ical. It depends
on how you and the company got into this mess. If
it is a mess.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 659
What Caused This Situation?
So, let’s start with some questions about causation:
• Why is the software undocumented?
• Why do you have so little time?
• What are the quality issues for this customer?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 660
Why Do You Have So Little Time?
A Few Possibilities
• Time-to-release drives competition
• Cash flow drives release date
• Key fiscal milestone drives release date
• Executive belief that you’ll never be satisfied, so your
•
•
•
•
•
schedule input is irrelevant
Executive belief that testing group is out of control and
needs to be controlled by tight budgets
Executive belief that you have the wrong priorities (e.g.
paperwork rather than bugs)
Executive belief that testing is irrelevant
Structural lack of concern about the end customer, or
Maybe you have an appropriate amount of time.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 661
Quality Issues For This Customer?
• In-house, “In-house” outsourced
• External, custom
• External, large package, packaged
• External, mass-market, packaged
• What is quality in this market? What are their costs of
failure?
» User groups, Software stores, Sales calls, Support calls
Over the long term, a good understanding of quality in
the market, and as it affects your company’s costs, will
buy you credibility and authority.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 662
Political Approaches: Buying Time
Many of the ideas in this section will help you if
you’re dealing with a single project that is out of
control.
If your problem is structural (reflects policy or
standard practice of the company), then some
of these ideas will be counter-productive.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 663
Buying Time:
Delaying a Premature Release -2
Take the high road
•
“I want to see knives in the chests, not in the backs.”
»
Trip Hawkins, founding President, Electronic Arts
•
Communicate honestly.
•
Avoid springing surprises on people.
•
Never sabotage the project.
•
Don’t become The Adversary: If you are nasty and
personal, you will become a tool, to be used by
someone else. And you will be disposable.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 664
Buying Time:
Delaying a Premature Release -3
•
•
•
Search for show-stoppers. If you can, dedicate
a specialist within your group to this task.
Circulate deferred bug lists broadly.
Consider writing mid-project and late-inproject assessments of:
»
»
»
»
extent of testing (by area)
extent of defects (by area, with predictions
about level of bugs left)
deferred bugs
likely out of box experience or reviewers’
experience
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 665
Buying Time:
Delaying a Premature Release -4
Do regular, effective status reporting. List:
• your group’s issues (deliverables due to you, things
slowing or blocking your progress, etc., areas in which
you are behind or ahead of plan)
•
project issues
•
bug statistics
Find allies (think in terms of quality-related costs)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 666
Signing Off on the Release?
Don’t sign off.
• Don’t accept the authority to sign off.
• Don’t pretend to be “QA” when you (obviously) are not.
• Position yourself as a technical information provider.
» Publish pre-release reports that provide data on the
stability of the product and the extent of completed testing
» Publish a report that lays out the likely issues that will be
raised by magazine reviewers (or other third parities)
when they see the product
» Let the people who want to ship prematurely own their
responsibility. Your responsibility is to provide them with
the information they need to make that decision.
•
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 667
Technical Approaches
• Find reference docs (you can get lots of data,
•
•
•
•
even if you can’t get specs.) See the notes on
this in the section on Spec-Driven testing
Re-use materials across projects.
Plan for exploratory testing.
Do cost-effective automation.
Use tools to find bugs faster
» web page syntax checkers, spiders, etc.
• Facilitate inspections
» (Get those bugs out before they reach you.)
• Cover the obvious legal issues.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 668
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
Non-digits
Maximum possible
number of chars
Empty field (clear
the default value)
UB number of
digits or chars
LB number of
digits or chars
Negative
0
UB of value + 1
LB of value - 1
UB of value
LB of value
Nothing
Reusable Test Matrix
Numeric Input Field
All Rights Reserved. 669
Group Management Approaches
Can you add head count?
• Will they let you?
• Can you absorb them?
» Do a work flow analysis to figure out what parts of each
task could be split off and delegated to a newcomer.
Stay organized
• Use functional outlines
• Or use a detailed project plan
• Create and publish explicit coverage reports
Get critical processes under control
• Example: release management
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 670
Black Box Software Testing
Part 22.
Career Planning for Testers
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 672
Career Planning for Testers
Tracks within testing:
•
•
•
Technical
» Automation programmer
» Automation architect
» Systems analyst
» User interface / human factors analyst / critic
» Test planner
» Subject matter expert
» Black box tester
Management
» Test lead, manager, director, internal consultant, external
consultant
Process management
» Metrics
» Software process improvement specialist
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 673
Certification of Testers
Several organizations will now give you “certification” in
software testing or software quality, including the American
Society for Quality, the Quality Assurance Institute and the
British Computer Society.
Should you be certified?
Training / studying for certification may or may not train you in
skills useful to your work. However, a certificate is a clear
statement that you care about your role in the profession.
In a tight market, such a certificate might improve your chances
of getting a job. I think that additional courses in programming
or work samples that show you have used test automation tools
(download a demo version and write your own tests if
necessary) are more likely to get you a good position.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 674
Career Planning for Testers
Tracks outside of testing (for example):
•
Programming
•
Program management
•
Marketing
•
Technical support
•
Documentation
•
Sales
•
Field support
•
Human factors analyst
•
Systems analyst
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 675
Career Planning for Testers
Tracks outside of testing
• People move back and forth between other fields
and testing. This doesn’t happen by accident. One
way to prepare to move to another field is by
picking tasks within testing that will educate you
in tasks that are performed in that other field.
• Management of multiple areas (e.g. testing and
something else) can lead to much more senior
management positions. Multi-functional managers
are often paid much better (along with promotion
faster) than managers of single areas.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 676
Career Planning for Testers:
Bodies of Knowledge
Based on the American Society for Quality Control’s Body of
Knowledge for the Certified Software Quality Engineer,
Software Testing Laboratories developed three useful
documents for describing the role of testers in a black box
testing organization:
•
The STL Body of Knowledge describes the knowledge that they
expect their staff to possess.
•
The STL Job Ladder is an overview of the knowledge and skill of
different levels of testing staff. The “Lead Track” is the track of
someone headed toward management. The “Engineer Track” is the
career track of an increasingly senior technical contributor.
•
The STL Body of Knowledge Map relates the knowledge areas in the
Body of Knowledge to the seniority levels in the STL Job Ladder.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 677
Career Planning for Testers:
Bodies of Knowledge
The two bodies of knowledge are quite different. Both list the
knowledge areas that they expect of a senior person in the
field, but ST Labs’ was developed by and for their testers,
whereas ASQ’s development was dominated by process
improvement specialists from (or who consult to) large
corporations, and includes relatively little on testing.
The ST Labs approach is useful in several ways:
•
You can work with your staff to plan their advancement as testers.
(Look for assignments that will grow their skills / knowledge in
desired ways).
•
You can work with candidates and hires who intend to stay in testing
only for a fixed time (say, a year or 18 months). They’re coming to
you with a plan to move to programming, tech support, marketing,
writing, something. You can review with them what they can work on
within testing that will help them get and then do well in that next job.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 678
Job Seeking in an Expanding Market
Part 22.
Cem Kaner, J.D., Ph.D., ASQ-CQE
Contact Information:
kaner@kaner.com
www.kaner.com (testing website)
www.badsoftware.com (legal website)
These slides were written in 1999 and early 2000 when the market was at its
peak. Some of the advice is dated now that the market has changed, but much
of it is still current, you just have to look harder for the right position.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 680
Factors to Consider
• What are your responsibilities to your current employer?
• What will make you happy?
• How much money?
• What about stock and benefits?
• What should your resume look like?
• How much should you stretch the truth?
• How do you build a reputation?
• Where can you find out about jobs?
• How should you manage recruiters?
• How can you learn about a company?
• How can you look good in an interview?
• How do you prepare to negotiate?
• How do you negotiate?
• Dealing with race, gender, age, etc.
• How do you set yourself up for success on the job?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 681
Responsibilities to Your Current
Employer
Loyalty?
•
•
•
•
Many employers are trying to appeal to the loyalty of their staff, to
keep them from leaving.
Pardon my cynicism, but many of the same people bragged about
controlling costs with layoffs when the market was tighter.
There are exceptions, such as Hewlett-Packard and IBM, that
undertake a strong sense of long-term responsibility to their staff.
You might reasonably feel a mutual obligation to these companies.
For the average company, if they want to appeal to your loyalty,
maybe you should ask for some golden handcuffs.
More generally
•
•
•
•
Give reasonable notice, but don’t give more than your employer
would give you, unless that serves other interests of yours
Preserve secrets
If you signed a non-compete contract, check with your lawyer
In general, what goes around comes around
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 682
What Will Make You Happy?
Great job markets are temporary. Take advantage
of it while you’ve got it. Now is the time to think
about what will make you happy, and go for it.
•
•
•
•
•
•
•
•
•
Money, stock, benefits, money
Opportunity for specific experiences / education
Opportunity for career growth
Opportunity to work with exceptional people (or to be a big
fish in a pond)
Social value of your work
Make room for your family, social life, education
Location
Specific allowances, support for your health
Other special circumstances
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 683
Money
Salary surveys
•
•
•
•
Many of these are lowball estimates
Few of these account for software testing as a separate area
There are several at websites like careerbuilder.com
Take geographic differences seriously.
Practice interviews
•
•
Interview with many companies
In several of the interviews, try out high numbers to test the top of
your market value. You’ll lose credibility in some of these interviews
(and thus lose the job) but you’ll learn a lot for your next jobs.
Employers will typically whine about how much you are
about to make, during a salary negotiation. Many will lay
a trip on you even if you are asking for half of the rate
they expected to pay.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 684
Stock & Benefits
Stock
•
•
•
Pre-IPO often means No-IPO.
We hear about stock option millionaires, but most become
hundredaires or thousandaires.
Look for factors that make it credible that this company will
go public in X timeframe:
» Profitability
» Consistent meeting of projections, no surprises to market
» Reasonably unique niche
Benefits
•
•
The usual medical, dental, education—what is important to
you?
Think carefully about overvaluing swimming pool, fitness
center, laundry room, dry cleaning pickup at your company.
Also, if they offer all this at the office, where is your life?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 685
Resume Tips
A resume INCLUDES and EXCLUDES. Your goal
is to appeal to the right sub-market.
Functional vs. historical
• You need buzz words but you are at risk in a functional
resume that is essentially a collection of buzz words.
Interviewers need history. Screeners need buzz.
Create several resumes for different market
segments
Length (1 page, 2 page, ???)
Emphasize what is special about you. Your hobby
might be your strongest selling point.
Verbosity is dangerous, people will tune out.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 686
Resume Tips
When should you send references with
your cover letter?
•
You are stretching to a job or are seeking work in a
discriminated situation. In general, include refs if
they make it substantially more likely that potential
employers will interview or hire you.
Write your COVER LETTER to specifically
respond to the ad. If it lists “requirements”
then point out the ones that you meet and
the ones on the list that you don’t meet but
really want to learn.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 687
Stretching the Truth?
Never stretch the truth
25% of American resumes contain false data.
Don’t join this 25%
Think of your manager
• Test managers are used to BS from others, and we
don’t like it.
• Probably completely intolerant of the false details,
and probably has a well-developed BS-detector.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 688
How Do You Build a Reputation?
•
•
•
•
•
Talks at local meetings
Teach classes at local universities / colleges
Go to conferences
Publish
Create a web site
The more attractive you are in the marketplace,
the higher your final salary will be.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 689
Where to Find Jobs
• Newspapers
• Magazines
• On-line services
• Internet search engines
• Recruiters
• Colleagues
• Spam (would you work for a
company that spams you?)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 690
Managing Recruiters
They are salespeople, typically on pure commission. The
more easily they think they can sell you, the more they’ll
work on your case.
Contact them regularly
•
Be consistent in the day / time that you contact them
• Remind them of you
• Ask them of openings that have recently crossed their desk that you
might be aware of.
Restrict their circulation of your resume
•
You must approve all sendings of your resume. Put this restriction in
writing. Refuse to deal with anyone who won’t honor this.
• Don’t tell them about other opportunities (or agree to tell them) (this
would have you giving another recruiter’s secrets to this recruiter).
Be aware of differences among recruiters
•
Executive recruiters vs. general purpose recruiters vs. outplacement
firm
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 691
Learn About the Company
During the search (or later)
•
•
•
•
•
•
•
Read the company’s web site, download demo software
Read the SEC filing
Deja News
Google, askjeeves.com, northern light
Stock sites that give investor info
Credit report (knowx.com)
Your peer network
•
In the phone screen
•
•
•
Ask for product literature
Ask for demo copies of software
Ask how else to find info about the company
•
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 692
Learn About the Company
Useful questions: I ask some of these of managers and some
of them of working staff. Use your judgment about who you
ask what:
•
•
•
•
•
•
•
•
•
What kinds of products and services does your company provide?
Can I see a demonstration of the key product?
What is special about your products and services? What are the key
strengths and weaknesses?
How did you develop the main product? What were the key
development tradeoffs? (Time vs Features vs. Cost vs. Reliability)
Who are your customers?
Who are your competition?
How do you learn about your customers?
How do you learn about your customers’ satisfaction with the overall
product, with the design, and with the defects?
Show me an organization chart (and where you are on it and where I
would be on it)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 693
Learn About the Company
• What is it like to work here?
• What do you do? What kinds of products and services
•
•
•
•
•
•
•
•
•
•
do you provide?
Can I see some examples?
Where do you fit in the product development process?
What do you like about your job?
What would you like to change?
How do you make time for your family?
How much control do you have over your own work?
Who designs the tests that you run? Who runs the tests
that you design?
Tell me about your test design process.
Can I see some test plans and test cases?
How do you feel about your pay / boss / colleagues?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 694
Learn About the Company
•
What courses or conferences did you take last year?
• What other training have you received?
• How do you learn new things?
• Describe three key things that you learned last year.
Keep your eyes open
• Are the interviewers tired?
• How is their furniture? How does it compare to the
company’s executives’ furniture?
• How much space / equipment / light do standard testers
get, and how much do higher ranking staff get?
• Look for congruencies and incongruencies of claims
regarding working conditions (e.g. 4-day work week)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 695
Looking Good in an Interview
What makes you attractive to them?
•
•
•
•
Skills, knowledge, aptitude, other (KSAO)
Your independence and confidence
Your reputation
If you had to look for a job today, what would be your unique
selling proposition? Are you happy with it? What kind of position
will it gain for you?
What makes you look bad?
•
•
Looking desperate
Arrogance
What convinces them that you are serious about them?
•
•
•
Background research
Saying that you want the job
Looking interested
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 696
Some characteristics of great testers
Alert
Attentive to detail
Analytical problem solver
Architect
Arrogance (usually, less is
better)
Artistic (knowledgeably
critique esthetic issues)
Assertive
Auditor
Author
Commitment (keep
promises, stick around)
Commitment to a task (do
what it takes)
Commitment to quality
Copes with difficult
circumstances
Courageous
Creative
Credible
Curious (inquisitive)
Customer focused
Decision maker (good
judgment)
Decisive
Not very defensive (able
to take criticism)
Diplomatic
Editor (criticize / improve
printed materials)
Effective with junior
testers
Effective with senior
testers
Effective with test
managers
Effective with
programmers
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 697
Some characteristics of great testers
Effective with nontesting managers
Empathetic
Empirical frame of
reference
Empowering
Energizing
Fast abstraction skills
Financially aware and
sophisticated
Finds bugs (intuitive
tester)
Flexible
Goal setting
Glue (promotes group
cohesiveness)
Humility
Integrity (honest; keeps
commitments)
Interpersonally
perceptive
Interviewer
Investigative reader
Leadership
Long term thinker
Meeting manager
Mentor
Multi-tasking
Organizer and planner
Persuasive
Politically perceptive
Policy and procedure
developer
Pragmatic
Programmer
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 698
Some characteristics of great testers
Protective (stands behind
his staff)
Punctual
Scholarly (collects
information, can back up
or evaluate arguments)
Sense of humor
Spoken communication
Strength of character
Subject matter expert
Substance abuser (not)
Team builder
Tolerant of ambiguity
Tolerant of different
development approaches
UI design
Versatile (many abilities)
Warm (makes the human
environment more
pleasant)
Written communication
Zealot (Rarely desirable in
large quantities.)
Catalyst
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 699
Looking Good in an Interview
What marketing materials are you bringing?
•
•
•
Resume, letters of reference, papers, printout of web
pages, other stuff that shows your vision
Work samples (beware of confidentiality)
Comments on their product
Practice interviewing
•
•
•
Interview lots of times
Interview with companies that are non-critical
Do mock interviews with friends
Send a follow-up letter
•
•
Astonishingly few people send these.
This is a marketing opportunity for you to spin the
meeting’s results.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 700
Preparing to Negotiate
Fisher, R. & Ertel, D. (1995) Getting Ready to Negotiate: The
Getting to Yes Workbook
The key things are
•
•
•
•
•
Knowing what you want
Knowing what they want
Knowing what they think of themselves
Knowing how they will present themselves as potential employers
Knowing what your alternatives are (your best alternative to a
negotiated agreement for this job)
PRACTICE NEGOTIATING
•
•
With friends
With potential employers. Plan to blow several away.
» Learn your market value
» Learn what’s out there
» Learn what seems to turn employers on and off.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 701
How Do You Negotiate?
You are negotiating a long term relationship that
you have to live with
If you don’t negotiate, you leave money on the table
that cuts back on your lifetime earning expectations
If you don’t negotiate, you leave your work open
instead of picking your assignment
If you don’t negotiate, you don’t learn how the
company will be when you do need to negotiate.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 702
How Do You Negotiate?
• Say what you want (non-monetary issues) or what
excites you, early in the interview.
• Refuse to provide previous salary info. Politely
reject queries about your salary expectations until
after the employer is excited about you.
• Be enthusiastic but don’t be over-eager.
• Speak about their product in their vocabulary.
• Find and point out ways in which you can help them.
Make proposals, show examples of your thinking.
• Let them know that you want the job.
• Talk about your stretch opportunity. If you have to
stretch for this job, let them see how they could
make you happy with the job (how it is a fit for you)
and how you could do it.
• Keep your eye out for intimidating styles from the
potential employer.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 703
How Do You Negotiate?
Chapman, J. (1996, 3rd Ed.) Negotiating Your Salary:
How to Make $1000 a Minute
Fisher, R., Ury, W., & Patton, B. (1991), Getting to Yes.
Freund, J.C. (1992), Smart Negotiating: How to Make
Good Deals in the Real World
O’Malley, M. (1998) Are You Paid What You’re Worth?
Miller, L.J. (1998), Get More Money on Your Next Job:
25 Proven Strategies for Getting More Money, Better
Benefits, & Greater Job Security.
Tarrant, J. (1997), Perks & Parachutes: Negotiating
Your Best Possible Employment Deal, from Salary and
Bonus to Benefits and Protection.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 704
Dealing with Race, Gender, Age, etc.
H1 is creating a discriminatory environment.
A.D.A gives you right to notice of tests
I think I see a widening pay differential. Much of the program
that I think is the problem is the negotiating style of the
individual rather than a policy of the company. Some
behaviors that turn out to have a severe differential
impact are engaged in by people who would not call
themselves (or normally be called) racist or sexist.
If you are a member of a group that is often treated as
disadvantaged / easy target, then you should definitely:
1. Read books on negotiation
2. Take a negotiating class (seriously consider trying Karrass, not the
nicey-nicey win/win stuff)
3. Do practice interviews
4. Join a group like Toastmasters
5. Find ways to compare notes with white men in comparable
positions. Knowledge is power.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 705
How Do You Set Yourself Up for
Success?
• Remember that if they don’t treat you well in the
interview, it won’t get better later.
• Don’t take a low rate to get in the door unless you are
brand new and plan to leave once you get experience.
They perceive you as more senior if they pay you more.
• If you have special needs, you must bargain for them
clearly and precisely up front. The alternative—bring it
up later—is often a loser.
• Get it in writing. Get any ambiguities cleared up in the
writing. Trust me doesn’t work well in many of these
ingredients.
• Be specifically aware of the possibility of becoming the
expendable 40+ manager who is out of work for a year.
(If you’re in this state, you have no fresh skills, no hot
new experience, you look easily replaced, and therefore
you find it intimidating to sell yourself.)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 706
Black Box Software Testing
Part 24.
Recruiting for Testers
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 708
Warning, warning: Legal issues
I am NOT well versed in employment law. Do NOT take this as
legal advice.
This talk raises a FEW legal issues, but only to get you thinking.
The rules vary by state. The rules are followed differently by
different companies. Talk to your HR department.
It is generally unwise to ask questions about age, race, sexual
orientation, political preference, national origin, pregnancy or
parental or marital status, disability (except for questions that are
provably directly tied to ability to perform the job), or anything
else that a normal person would consider private. Talk to your HR
department about what questions you are allowed to ask.
Be very cautious about checking a candidate’s credit record or
about insisting that she take a polygraph exam.
Beware of accidentally making unintended promises.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 709
Underlying principles
• Every hire should support the short term need and
•
•
•
•
•
the long term mission of the group.
Your team should be diverse.
With some creativity and tolerance, you can find
great people.
Hiring decisions should be made by consensus.
Use a behavioral strategy for gathering information
Tests should predict performance on the job.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 710
What is the mission of your group?
Every position in the testing group should help your
group realize its mission.
• The nature of each position is determined partially in terms
of work that has to be done over some foreseeable term and
partially in terms of the overall mission of the group.
• The skills, knowledge, and abilities required for each
position are partially determined by the specific nature of the
tasks to be accomplished over the short or intermediate term
and partially by the need to have people who can help the
group fulfill its mission.
» Example—the group needs a statistician but this will occupy
only 10% of that person’s time. You will hire for some “other”
position, but keep an eye out for statisticians.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 711
What is the mission of your group?
Examples of different missions:
• Assess and report the quality of the product? (So what’s
quality?)(What’s assessment? Probably need a person
skilled in statistics and measurement.)
• Discover, report and advocate for the repair of product
defects? (So what’s a defect?)
• Investigate the product’s conformance to a written
specification.
• Carry out some other limited set of functions, such as
looking only for coding errors (mismatch between the
programmer’s intended and actual behavior of the
program).
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 712
A Sample Job Description
The second level associate engineer has skill and
experience in software testing or development,
and a demonstrated ability to follow our internal
SQA practices.
We expect successful A2’s to create and execute
a systematic test plan for a routine project,
producing all relevant reports. We expect them to
have knowledge of typical technologies sufficient
to recognize quality problems in those areas.
Minimum 1 year directly relevant experience in
addition to required education and other
professional experience.
• (Copied from materials published by ST Labs)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 713
What position are you trying to fill?
The notion of “essential job functions”
• Job descriptions are generic. The specific
position you have to fill now has other details.
• Make a list of what you absolutely need, right now.
• Make a list of what else you want of a person in
this position.
• Think of a good candidate as someone who
meets all of the absolutely-needs and some
portion (maybe a percentage) of the want-tohaves.
• This turns into a memo that you distribute to
everyone interviewing for the position.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 714
Who are you looking for?
Knowledge—the body of information that the candidate will
need in order to perform effectively. For example, a
person might know or have training in test case design,
maybe even a course in creating test matrices.
Skills—involve proficiency at a specific task. For example, a
person might be really good at creating test matrices.
Abilities—potential to do a job. For example, a person might
be a very bright, systematic, analytical thinker, who you
would expect to be able to quickly become very good at
creating test matrices.
Other—other characteristics that are important to the job.
For example, a person might have personal integrity.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 715
Characteristics: The Need for Diversity
Diversity is essential. We need:
• subject matter experts
• programmers
• testers, test planners
• project managers, writers
Standard credentials are not the answer
• An example: Staffing for a financial application
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 716
Exercise: Characteristics of Candidates
Refer to the slides in the Career Planning section
that list a few characteristics of good testers.
Take 15 minutes to:
• Imagine recruiting for a mid-level tester in your test
group.
• Identify at least one skill, area of knowledge, or ability
that is important to the position (or to your company as
a whole), that is not listed on these slides.
• Circle (or write in) the top 10 characteristics that are
most important for this position
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 717
The Opportunity Hire
Some human examples of opportunity hiring
• Training requirement: Entertainment company, low pay.
Recruited Director-level employee as a tester. Key promise
to teach him about American business culture, goal of
opening a test lab.
• Temporary assignments:
» Marketeer (wanted training in product development)
» Tech support (offered expertise in my product, supervisory
experience; wanted a stint in PD)
• Work force: Left computing (project/product mgr) to do
construction project management. Came back to computing
without the most current skills.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 719
The Opportunity Hire
Opportunity hiring poses challenges. These
challenges may or may not be insurmountable. Here
are some hypothetical examples that have a loose
relationship to real problems I’ve had to manage.
• Programmer who wants to move groups today
• Programmer who seeks out “spare time” programming
assignments
• Executive who is too used to deferential treatment for
the rough and tumble of testing
• Family commitment led to time jealousy
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 720
Consensus-driven hiring
I follow three simple rules when hiring:
• Anyone in the company who wants to be part of the
interview process for a candidate is welcome.
• Example, talk with manufacturing mgr about doc mgr and then
invite to do the interview
• Of the people who have interviewed the candidate, anyone
in the testing group and any senior player from any other
group who will work with the candidate can veto the hiring
of this person.
• The veto policy must be actively managed so that vetoes
will not be based on race, religion, family situation, gender,
sexual orientation, age, disability, national origin, etc.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 721
Consensus-driven hiring
• Important (almost essential) for opportunity
hiring
• Group acceptance of special situation, at the start and
later
• Additionally
• More likely to discover serious problems
• More likely to reject based on those problems
• Provides support network for incoming candidate
• Key assumption:
It is a more serious mistake to hire badly
than to pass up a good candidate.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 722
Behavioral information gathering
• The general principle:
• The goal of the interview is to predict how the candidate will
behave if she joins your group.
• You can achieve this by (for example)
• asking questioning that elicit behavioral responses,
• examining work samples,
• testing,
• setting up situations that let you see how the candidate responds.
• Simple questioning often fails to elicit behavioral
information.
• Much of the behavior that you will see is independent of
the questioning.
• Example: The late interview candidate.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 723
Behavioral Interviewing
Questioning
• Closed vs. open-ended questions
• Factual vs. opinion questions
• Hypothetical vs. behavioral questions
Work samples
• Ask (during phone screen) if he has any work samples that he
can bring for review.
• The problem of confidential documents:
» Don’t request confidential documents.
» Before reviewing something that looks as if it might be
confidential, ask.
» Don’t review documents that are (or that you think probably are)
» Be diplomatic in your refusal to read them
» Listen to the candidate’s responses—this is a predictor of how
he will treat your confidential documents in the future.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 724
Behavioral Interviewing: Testing
Testing
• Puzzles
» Big practice effects
» Cultural fairness issues (may have a discriminatory
impact)
» I’m not convinced that they predict success in testing
» They are NOT IQ tests, though many people treat
them as if they were.
• Test cases
• A testing / training exercise
• Bug reports
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 725
Interview Technique
You’re the tester, here’s a CD. You have four hours
to test it. What will you do?
Question is, how do we gather information about
the product, what is the though process. “How can
you generate a test case without knowing what the
requirements are?” Go into a feature map after you
determine the market and the requirements.
Another example, use a print dialog that is broken
for interviewing. For example, leave a blank button
somewhere, see what they do with it.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 728
Black Box Software Testing
Part 25.
Learning Styles --- Teaching
Testers
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 730
Learning Styles
People learn in different ways. There are several models
of learning styles. Different presentation methods work
best for different styles of learners.
I’m not up to date on this literature. Back when I was a
grad student in psychology, we talked about
• Visual vs. auditory learners. Visual learners have more success in
lectures if they have a copy of every slide
• Active vs. passive learners. Some people learn better in lecture or
by reading / memorizing. Others learn better through exercises.
• Individual vs. group learners. Some people learn better on their
own, others on project teams.
• Concrete vs. conceptual learners. Some people learn better from
detailed stories, others from the formal presentation of the
underlying principles.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 731
Learning Styles
I’m still thinking about how to develop a course that
works well with the different styles. (I’m especially
interested in this for building on-line courses that are
worth teaching/taking).
For now, my approach involves:
• Lecture of key material
• Hard copies of every slide
• Many stories with rich detail to support key points
• Several assignments, which can be done by individuals or jointly
• Sample questions that can be studied by individuals or jointly.
And may soon involve
• Self-study drills
• A library of examples that students can read on their own.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 732
Learning Styles
There’s nothing magical about the division that I learned,
and several others are more popular in the literature.
For a useful introduction and discussion of several style
classifications, see R.M. Felder, Matters of Style,
http://www2.ncsu.edu/unity/lockers/users/f/felder/public/
Papers/LS-Prism.htm
Felder makes the point that the basic lessons for
instructional design end up being pretty similar across
the different classification schemes. The next slides,
taken verbatim from his paper, show his advice to
chemistry teachers:
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 733
Examples from Felder’s Matter of Style
“Here are some strategies to ensure that your courses present
information that appeals to a range of learning styles. These
suggestions are based on the Felder-Silverman model.
 Teach theoretical material by first presenting phenomena and problems that relate
to the theory (sensing, inductive, global). For example, don't jump directly into
free-body diagrams and force balances on the first day of a statics course. First
describe problems associated with the design of buildings and bridges and
artificial limbs, and perhaps give the students some of those problems and see
how far they can go before they get all the tools for solving them.
 Balance conceptual information (intuitive) with concrete information (sensing).
Intuitors favor conceptual information--theories, mathematical models, and
material that emphasizes fundamental understanding. Sensors prefer concrete
information such as descriptions of physical phenomena, results from real and
simulated experiments, demonstrations, and problem-solving algorithms. For
example, when covering concepts of vapor-liquid equilibria, explain Raoult's
and Henry's Law calculations and nonideal solution behavior, but also explain
how these concepts relate to barometric pressure and the manufacture of
carbonated beverages.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 734
Examples from Felder’s Matter of Style
• Make extensive use of sketches, plots, schematics, vector diagrams, computer
graphics, and physical demonstrations (visual) in addition to oral and written
explanations and derivations (verbal) in lectures and readings. For example, show
flow charts of the reaction and transport processes that occur in particle
accelerators, test tubes, and biological cells before presenting the relevant theories,
and sketch or demonstrate the experiments used to validate the theories.
• To illustrate an abstract concept or problem-solving algorithm, use at least one
numerical example (sensing) to supplement the usual algebraic example (intuitive).
For example, when presenting Euler's method for numerical integration, instead of
simply giving the formulas for successive steps, use the algorithm to integrate a
simple function like y = x2 and work out the first few steps on the chalkboard with
a hand calculator.
• Use physical analogies and demonstrations to illustrate the magnitudes of calculated
quantities (sensing, global). For example, tell your students to think of 100 microns
is about the thickness of a sheet of paper and to think of a mole as a very large
dozen molecules. Have them pick up a 100 ml. bottle of water and a 100 ml. bottle
of mercury before talking about density.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 735
Examples from Felder’s Matter of Style
• Occasionally give some experimental observations before presenting the general
principle, and have the students (preferably working in groups) see how far they can
get toward inferring the latter (inductive). For example, rather than giving the
students Ohm's or Kirchoff's Law up front and asking them to solve for an
unknown, give them experimental voltage/current/resistance data for several
circuits and let them try to figure out the laws for themselves.
• Provide class time for students to think about the material being presented (reflective)
and for active student participation (active). Occasionally pause during a lecture to
allow time for thinking and formulating questions. Assign "one-minute papers"
near the end of a lecture period, having students write on index cards the lecture's
most important point and the single most pressing unanswered question. Assign
brief group problem-solving exercises in class that require students to work in
groups of three or four.
• Encourage or mandate cooperation on homework (every style category). Hundreds
of research studies show that students who participate in cooperative learning
experiences tend to earn better grades, display more enthusiasm for their chosen
field, and improve their chances for graduation in that field relative to their
counterparts in more traditional competitive class settings.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 736
Examples from Felder’s Matter of Style
• Demonstrate the logical flow of individual course topics (sequential), but also point
out connections between the current material and other relevant material in the same
course, in other courses in the same discipline, in other disciplines, and in everyday
experience (global). For example, before discussing cell metabolism chemistry in
detail, describe energy release by glucose oxidation and relate it to energy release
by nuclear fission, electron orbit decay, waterfalls, and combustion in fireplaces,
power plant boilers, and automobiles. Discuss where the energy comes from and
where it goes in each of these processes and how cell metabolism differs. Then
consider the photosynthetic origins of the energy stored in C-H bonds and the
conditions under which the earth's supply of usable energy might run out.”
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 737
Blooms Taxonomy Slides
“Bloom's Taxonomy, created in 1956 by Dr. Benjamin
Bloom of the University of Chicago and his group of
educational psychologists, is a categorization of verbs
describing cognitive skills verbs into six classes
(knowledge, comprehension, application, analysis,
synthesis, and evaluation). The classes are ranked from
least complex (knowledge) to most complex (evaluation)
in terms of the level of thinking required for students to
achieve these objectives. In general, critical thinking skills
encompass only the three most complex categories
(analysis, synthesis, and evaluation).”
(From Michael Bowen’s web page,
http://207.233.105.16/curriculum/bloomtax.htm, to be
distributed in class.)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 738
Testing Training
• It is easy to teach to the first few levels (tests would have
students recall what was taught, define terms, make
simple applications).
• It is much tougher to teach the underlying skills of testing.
• Putting students through tests, exercises and exams has
been eye-opening in terms of how much they can learn
with personal involvement and how little they get from
traditional lectures.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 739
Example of a Task Breakdown
Bug Reporting
• Reproduce the bug
• Simplify the bug
• Follow-up testing
• Describe the screen
• Describe the sequence
• Determine customer impact
• Determine technical impact
What sub-tasks and skills are
involved in each of these?
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 740
Closing Notes
Given a group of motivated, bright people who have
appropriate background knowledge and belong to
your testing group or your testing class:
• Only some testers will learn well in lectures
• Only some testers will learn well by being thrown into a
project and being told to figure it out
• Only some testers will learn well from books
• Only some testers will learn well or work well on their own
and some will only learn well or work well on their own
Build training strategies to achieve the learning you
want to convey, accommodate multiple learning
styles, and push for mastery, not memory.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 741
Black Box Software Testing
Part 26.
Testing Resources on the Net
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 742
Testing Resources on the Net
Various Web Sites
DOUG HOFFMAN’S HOME PAGE
www.SoftwareQualityMethods.com
Consulting and training in strategy and tactics for software quality. Articles on software
testing, quality engineering, and management. Look here for updated links.
CEM KANER’S HOME PAGE
www.kaner.com
Articles on software testing and testing-related laws
JAMES BACH
www.satisfice.com
Several interesting articles from one of the field’s most interesting people.
BRETT PETTICHORD
www.io.com/~wazmo/qa.html
Several interesting papers on test automation. Other good stuff too.
BRIAN LAWRENCE
www.coyotevalley.com
Project planning from Brian Lawrence & Bob Johnson.
BRIAN MARICK
www.testing.com
Brian Marick wrote an interesting series of papers for CenterLine. This particular one is a
checklist before automating testing. The CenterLine site has a variety of other useful papers.
ELISABETH HENDRICKSON
www.QualityTree.com
Consulting and training in software quality and testing.
JOHANNA ROTHMAN
www.jrothman.com
Consulting in project management, risk management, and people management.
HUNG NGUYEN
www.logigear.com
Testing services, training, and defect tracking products.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 743
Testing Resources on the Net
Various Web Sites
LESSONS LEARNED IN SOFTWARE TESTING
www.testinglessons.com
This is the home page for Cem’, James’, and Bret’s book, Lessons Learned in Software Testing.
BAD SOFTWARE HOME PAGE
www.badsoftware.com
This is the home page for Cem’s book, Bad Software. Material on the law of software quality
and software customer dissatisfaction.
SOFTWARE QUALITY ENGINEERING
www.sqe.com
Several interesting articles on current topics.
SOFTWARE QUALITY ENGINEERING
www.stqe.com www.stickyminds.com
Articles from STQE magazine, forum for software testing and quality engineering.
QA DUDE’S QUALITY INFO CENTER
www.dcez.com/~qadude
“Over 200 quality links” -- pointers to standards organizations, companies, etc. Plus articles,
sample test plans, etc.
QUALITY AUDITOR
www.geocities.com/WallStreet/2233/qa-home.htm
Documents, links, listservs dealing with auditing of product quality.
THE OBJECT AGENCY
www.toa.com
Ed Berard’s site. Object-oriented consulting and publications. Interesting material.
RBSC (BOB BINDER)
www.rbsc.com
A different approach to object-oriented development and testing.
DILBERT
www.unitedmedia.com/comics/dilbert.
Home of Ratbert, black box tester from Heck.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 744
Testing Resources on the Net
Various Web Sites
SSQA
www.ventanatech.com/ssqa
Silicon Valley Software Quality Association is a local professional software QA
organization with monthly meetings, newsletter, more.
AMERICAN SOCIETY FOR QUALITY (ASQ) www.asq.org
National/international professional QA organization.
SILICON VALLEY SECTION OF (ASQ)
www.asq-silicon-valley.org
ISO
www.iso.ch
Describes ISO (International Organization for Standardization), with links to other
standards organizers
AMERICAN NATIONAL STANDARDS INSTITUTE
www.ansi.org
NSSN
www.nssn.org
National Standards Systems Network. Find / order various standards. Lots of links to
standards providers, developers and sellers.
IEEE Computer Society
www.computer.org
Back issues of IEEE journals, other good stuff.
SOFTWARE ENGINEERING INSTITUTE
www.sei.cmu.edu
SEI at Carnegie Melon University. Creators of CMM and CMMI.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 745
Testing Resources on the Net
Various Web Sites
CENTER FOR SOFTWARE DEVELOPMENT
www.center.org
Non-profit in San Jose with a big test lab and various other support facilities.
RELIABLE SOFTWARE TECHNOLOGIES
www.rstcorp.com
Consulting firm. Hot stuff on software reliability and testability. Big archive of downloadable papers.
Lots of pointers to other software engineering sites.
SOFTWARE TESTING INSTITUTE
www.ondaweb.com/sti
Membership-funded institute that promotes professionalism in the industry. BIG list of pointers to
resources in the industry (the Online STI Resource Guide).
SOFTWARE PRODUCTIVITY CENTRE
www.spc.ca
Methodology, training and research center that supports software development in the Vancouver BC
area.
CENTRE FOR SOFTWARE ENGINEERING
www.cse.dcu.ie
“Committed to raising the standards of quality and productivity within Ireland’s software development
community.”
EUROPEAN SOFTWARE INSTITUTE
www.esi.es
Industry organization founded by leading European companies to improve the competitiveness of the
European software industry. Very interesting for the Euromethod contracted software lifecycle and
documents.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 746
Testing Resources on the Net
Various Web Sites
QUALITY.ORG
www.casti.com
Links to quality control source materials.
CSST (CLIENT-SERVER SOFTWARE TESTING) www.cse.dcu.ie
D. J. Mosley’s home page. Lots of client / server publications.
FORMAL TECHNICAL REVIEW ARCHIVE
www.ics.Hawaii.edu/~johnson/FTR
Documents, links, listservs dealing with auditing of product quality.
SOCIETY FOR TECHNICAL COMMUNICATION
www.stc.org
Links to research material on documentation process and quality.
BUGNET
www.bugnet.com
Lists of bugs and (sometimes) fixes. Great source for data.
TRY THIS SOMETIME -- With a product you own, look up bugs in BugNet that doesn’t list a
workaround or a bugfix release, and replicate it on your computer. Then call the publisher’s tech
support group and ask if they have an upgrade or a fix for this bug. Don’t tell them that you found it in
BugNet. The question is, what is the probability that your publisher’s support staff will say, “Gosh,
we’ve never heard of that problem before.”
JPL TEST ENGINEERING LAB
tsunami.jpl.nasa.gov
Jet Propulsion Lab’s Test Engineering Laboratory. Includes the comp.software.testing archives.
Additionally, there are many sites for specialized work, such as sites for HTML compatibility tests of
browsers. The usual search tools lead you to the key sites (the list changes weekly.)
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 747
Testing Resources on the Net
Various Web Sites
SOFTWARE RESEARCH INC.
www.soft.com
Also a major consulting and toolbuilding firm. Organizes the Quality Week conference.
Publishes the TTN-Online newsletter. Excellent links.
QUALITY ASSURANCE INSTITUTE
www.qai.com
Quality control focused on the needs of Information Technology groups.
AETG WEBSITE
http://aetgweb.argreenhouse.com/
Home page for the AETG combinatorial testing product. Includes articles describing the theory
of the product.
UNIFORM COMMERCIAL CODE ARTICLE 2B
www.law.upenn.edu/bll/ulc/ulc.htm
These hold the drafts of the proposed Article 2B (UCITA), which will govern all sales of software.
This will become the legal foundation of software quality. Currently, the foundation looks like it will
be made of sand, Jell-O, and invisible ink.
SOFTWARE PUBLISHERS ASSOCIATION
www.spa.org
Software & Information Industry Association is the main software publishers’ trade association.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 748
Testing Resources on the Net:
Some News Groups
comp.software.testing
This covers issues of general interest to software testers. Plagued by too much
spamming, this is not quite as interesting a spot as it used to be.
comp.human-factors
User interface issues, usability testing, safety, man-machine reliability, design tools.
comp.software.international
Internationalization and localization issues
comp.software.config-mgmt
Various configuration management issues.
comp.software-eng
Discussions of all areas of software engineering, including design, architecture,
testing, etc. The comp.software-eng FAQ is the one that lists sources of bug tracking
systems, for example. (You’d think it would be in comp.software.testing, but
comp.software-eng got there first.)
comp.software.measurement
Not much there, but the goal is picking up metrics / measurements / data.
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 749
Testing Resources on the Net:
Some News Groups
comp.answers
The home of many, many FAQs (documents that answer Frequently Asked
Questions).
comp.jobs, comp.jobs.offered, ba.jobs, misc.jobs, etc.
A good way to check out market values for testers (or to find your new home when
you need one).
comp. risks
Daily discussions of newsworthy bugs.
comp.dcom.modems
Lots and lots and lots of discussion of modems.
misc.industry.quality
Various discussions of quality control paradigms and experiences.
alt.comp.virus
This covers viruses, explaining things at end-user and more technical levels. The
group often has very-up-to-date stuff. And like most alt-based groups, it seems to
have a lot of spam mixed in. . .
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 750
Testing Resources on the Net
Mailing Lists, etc.
swtest-discuss@convex.convex.com
Mail swtest-discuss-digest-request@rstcorp.com with
"subscribe" in the body of the message to subscribe.
baldrige, qs9000, many others
contact bill casti The Quality Czar <help@quality.org>
DEMING-L
contact LISTSERV@UHCCVM.UHCC.HAWAII.EDU
testing technology newsletter
contact ttn@soft.com, software research assocs
Copyright © 1994-2004 Cem Kaner and SQM, LLC.
All Rights Reserved. 751