Software Testing
Testing across the entire software
development lifecyle
Software test engineer
• Software test engineer is a professional who is
in charge of one or more technical test activities,
including
–
–
–
–
–
designing test inputs
producing test case values
running test scripts
analyzing results
reporting results to developers and managers
By Ammann and Offutt, 2008
Software test manager
• Test manager is in charge of one or more test
engineers. They
–
–
–
–
set test policies and processes
monitors the processes
interact with other managers on the project
help the engineers do their assignments
By Ammann and Offutt, 2008
Activities of a software test engineer
• Design tests by creating test requirements
– requirements are transformed into actual values and
scripts ready for execution
– executable test are run against the software
– the results are evaluated to determine if the tests find
a fault
• Powerful tool: formal coverage criterion
– Provides test engineers ways to decide what test
inputs to use during testing such that maximum
possible faults can be found
– Also provides stopping rules
By Ammann and Offutt, 2008
The mind of a tester
• Four different kinds of thinking shown by a good tester
(by Kaner et. al)
– Technical thinking: ability to model technology and understand
causes and effects
– Creative thinking: ability to generate ideas and see possibilities
– Critical thinking: ability to evaluate ideas and make inferences
– Practical thinking: ability to put ideas into practice
• An example of these kinds of thinking is found in a fable
called “The King’s Challenge”
By Everett and McLeod Jr., 2007 and Kaner et. al, 2002
The mind of a tester (cont’d)
• The King’s Challenge (a fable)
– Once upon a time, a mighty king wanted to determine
which of his three court wizards was the most
powerful
– So he put the three court wizards in the castle
dungeon and declared whoever escaped from his
respective dungeon cell first was the most powerful
wizard in all the kingdom
By Everett and McLeod Jr., 2007 and Kaner et. al, 2002
The mind of a tester (cont’d)
• Here is what each court wizards has done
– The first wizard immediately started chanting mystical poems to
open his cell door
– The second wizard immediately started casting small polished
stones and bits of bone on the floor to learn how he might open
his cell door
– The third wizard sat down across from his cell door and thought
about the situation for a minute. Then, he got up, walked over to
the cell door and pulled on the door handle. The cell door swung
open because it was closed but not locked
– Thus, the third wizard escaped his cell first and became known
as the most powerful wizard in all the kingdom
• What kinds of “tester” thinking did the third wizard
exercise in solving the king’s puzzle?
By Everett and McLeod Jr., 2007 and Kaner et. al, 2002
The mind of a tester (cont’d)
• Answer:
– Creative thinking: ability to generate ideas and see
possibilities
– Practical thinking: ability to put ideas into practice
By Everett and McLeod Jr., 2007 and Kaner et. al, 2002
Non-software testing at the user level
• Buying a car
– Use the automobile industry to find non-computer testing examples
that can be related to software testing
– Motivation for testing a car is to determine its quality or its
functionality before buying one
– As a user, you are not interested in performing all possible kinds of
test since you assume that manufacturer has done those tests for
you
– Important to realize that you limit your testing some way
– Limited test is referred as “test drive”
– To better understand the testing limits, examine what you do NOT
test followed by what you TEST
By Everett and McLeod Jr., 2007
Non-software testing at the user level
• Objectives of a Test Drive are NOT
– to break the car
• Seek guarantees and warranties that car
manufacturer has already proven that car is
“unbreakable” under normal driving conditions for x
thousand miles or y years (reliability)
– to improve the car’s design
• If you want design changes in the car, simply shop
for different model or different manufacturer
By Everett and McLeod Jr., 2007
Non-software testing at the user level (cont’d)
• What you test is determined by your transportation needs.
The needs become test drive objectives
• Test objectives are the measurable milestones in testing
which indicate that the testing activities have achieved the
desired goals
• Test drive objectives are translated into testing approaches
that validate whether the car on the dealer’s lot meets your
transportation objectives
By Everett and McLeod Jr., 2007
Non-software testing at the user level (cont’d)
• Objectives of a Test Drive ARE
–
–
–
–
–
to validate affordability
to validate attractiveness
to validate comfort
to validate usefulness
to validate performance
• Each of these objectives can be validated without trying to
break the car or redesign it.
• Each of these objectives are personal and you need to
prioritize them to in final your final decision
By Everett and McLeod Jr., 2007
Non-software testing at the user level (cont’d)
• Testing approaches include
– examining the sticker price and sale contract
– trying out radio, the air conditioner, and the lights
– trying acceleration, stopping, and cornering
• These testing approaches are referred to by common
terminology in the testing industry
– examine = static testing
(observe, read, review without driving the car)
– try out = functional and structural testing
(work different features of the car without driving the car)
– try = performance testing
(work different features of the car by actually driving the car)
By Everett and McLeod Jr., 2007
Non-software testing at the developer level
• Building a car
• Testing objectives of a new car to be built
– validate design via scale models
– validate operation of prototypes
– validate mass assembly plans from prototypes
• Starts with written requirements such as
–
–
–
–
–
Seats six
Carries five suite cases
Runs on regular gas
Consumes gas at a rate of 35 miles per gallon in highway
Has a top speed of 100 miles per hour
By Everett and McLeod Jr., 2007
Non-software testing at the developer level (cont’d)
•
These requirements are the nonnegotiable design and manufacturing
boundaries set by groups other than the designers
•
The manufacturer is responsible to build a new car which satisfies all the
requirements
•
New requirements,  more understandable test objectives
•
The auto design tester is responsible to validate the current state of the new
car against its requirements
–
If the new car doesn’t initially meet the requirements, then it’s the designer, not the tester who
must improve the design for full compliance
–
After the design changes are done, the tester needs to revalidate the revised design against
the requirements
•
Design, test, correct, retest cycle continues until the new car meets its
requirements and is completed before the car is manufactured
•
Requirements are essential for testing validation at every stage of developing
a new car
By Everett and McLeod Jr., 2007
Non-software testing at the developer level (cont’d)
• Testing approaches used while building new cars
–
–
–
–
–
–
plan the tests based on requirements and design specifications
examine blueprints and clay models
perform and analyze wind tunnel tests
perform and analyze safety tests
perform and validate prototype features
drive prototype and validate operations
• Specifications (blue prints or models) are the designer’s interpretation
of the requirements on how the design can be manufactured
• When specifications are validated against the requirements, all the
subsequent physical car assembly validation can be performed
against the specifications
By Everett and McLeod Jr., 2007
Non-software testing at the developer level (cont’d)
• Similar to the test drive, the car builder testing approaches
can be described by common testing terminology
– examine = static testing
(observe, read, review without driving the car)
– perform = functional and structural testing
(work different features of the car models, mock-ups, and manufactured car
subassemblies)
– drive = performance testing
(work different features of the car in the prototypes)
By Everett and McLeod Jr., 2007
Primary objectives of testing
• Testing objective 1: Identify the magnitude and sources of
development risk reducible by testing
– Company prepares a business case including expected benefits,
costs, and risks
– If the project is determined to be bad on investment, then the
project does not start
– If the benefits outweigh the cost and it’s good return on investment,
the benefits are compared to the risks
• If the risk is high but the likelihood of the risk occurrence is low, the project is on
• If the risk and the likelihood of the risk occurrence is high, the following
questions are asked:
– Can this risk be reduced by testing?
– If the risk can be reduced, how much can testing reduce it?
• If the risk factors are well known and quantifiable, it’s possible that testing can
reduce the probability of the risk occurrence
By Everett and McLeod Jr., 2007
Primary objectives of testing (cont’d)
• Testing objective 2: Perform testing to reduce identified
risks
– Test planning includes
• positive testing (things work as required)
• negative testing (things that break)
– Test planning highlights the risk areas such that the largest possible
percentage of the test schedule and effort (both positive and
negative testing) are allocated to reduce that risk
– Testing does not completely eliminate the risk since there are
always more scenarios to test than the allotted time and resources
to complete the tests
By Everett and McLeod Jr., 2007
Primary objectives of testing (cont’d)
• Testing objective 3: Know when testing is completed
– Since 100% testing is unrealistic, the tester must determine when
to stop testing
– The determination should start with the positive test items in the
test plan
– Complete as much of the risk-targeted testing as possible relative
to cost benefit break-even point
• $100,000 business risk, $50,000 testing cost to reduce the risk (NO!!!)
• A rule of thumb is 10-20% cost to benefit break-even point for testing
– $100,000 business risk, $1000-2000 testing cost (YES!!!)
– Tester must complete as many of the negative test items in the plan
with the remaining the testing budget after positive testing and risk
testing are completed
By Everett and McLeod Jr., 2007
Primary objectives of testing (cont’d)
• Testing objective 4: Manage testing as a standard project
within the development project
– Often testing is treated as simple skill that anyone can do without
planning, scheduling or resources
– Since business risk represents real dollar loss, real dollar testing is
required to reduce the risk
– Real dollar testing means that personnel with testing experience
should be become the testing team with access to management,
resources and schedules necessary to plan and complete the test
– Observations:
• The testing does not have to be a hit or miss
• Testers are a limited resource
By Everett and McLeod Jr., 2007
References
• P. Ammann and Jeff Offutt, Introduction to software testing,
Cambridge University Press, 2008
• G. D. Everett and R. McLeod Jr., Software Testing, Wiley InterScience, 2007
• C. Kaner, J. Bach, B. Pettichort, Lessons learned in software testing,
Wiley Computer Publishing, Addison-Wesley, 2002, pp. 286