CS 5150
Software Engineering
Lecture 17
Testing Large Systems
By Stephen Purpura
October 27, 2010
CS 5150
1
Testing Begins on Day 1
A professional project team begins with:
(a) An idea (usually mused over for a while between friends)
(b) A program/product manager
(c) A development manager
(d) A test manager
Today, we’re going to focus on how the Test Manager thinks
CS 5150
2
Good Test Managers
• Before the project starts, they build the first draft of the Test
Exit Criteria which answers:
“How do we know when we’re done?”
• The Test Manager:
– Drives the testing framework and approach
– Is a great communicator
– Understands the value of defect-fix-build-retest cycles that
work like clockwork. THINK BOTTLENECKS.
CS 5150
3
Example of a large scale system test: a browser
At one point, the second largest automated software
testing infrastructure in commercial software
development was used to test Internet Explorer.
The largest tested Windows.
Why so many computers?
CS 5150
4
Different Testing Methodologies
• In large projects there are many stakeholders.
• Usually, they believe in different schools or methodologies.
• You need to balance this from the start.
• An example of a methodology: Agile
http://pettichord.com/
http://www.io.com/~wazmo/papers/four_schools.pdf
• Five different schools (from pdf above):
– Analytic, Standard, Quality, Context-Driven, Agile
CS 5150
5
A Test Pass
A Test Pass: all of the actions involved in assessing whether a
software system meets its requirements.
This definition implies:
• All of the requirements are expressed as “tests” (usually
with instrumented indicators)
• The team documents all of the steps required to run the tests,
so that the test pass can be replicated.
• The team runs the test pass multiple times. Else why write it
all down?
CS 5150
6
Automated Testing Methods
Two different types of
(a) Scripted testing
•
Programs that manipulate internal or external interfaces
(b) “Screen Scrapers”
•
Webload (http://www.webload.org) can automate tests by
capturing mouse movements, keyboard events, etc.
Both methods can be “fed” into scripted load testing solutions like
WebLoad.
Note: UserTesting.com, while useful, is not ‘automated testing’.
CS 5150
7
The Test Harness
• Test Harness: the program that controls the automated testing
scripts.
– Usually first built for “smoke testing”
– Expands to replicate the maximum required load
(simultaneous transactions, MTBF, more scenarios)
• Smoke Testing: the minimal set of tests one runs to make
certain that daily check-ins to the source control system didn’t
break critical functionality in the daily build.
– I integrate the smoke test into the “make” command
CS 5150
8
Scenarios Make Great Test Cases
• Success and failure scenarios and use cases are the foundation
of great testing systems.
• Over the course of product development, you usually uncover
additional scenarios by watching people break the system.
• How does this happen …
CS 5150
9
Eating your own Dog Food
• “Eating your own dogfood” refers to a situation when a team
decides that it will use the software it is developing to conduct
its daily business and operations.
• If people use the system every day as part of their work, they
find problems with it. These problems frequently become new
scenarios for the test infrastructure.
• With operating systems and browsers, this is easy. With IT
systems (like human resource systems), this is less possible.
But get creative.
CS 5150
10
Additional Sources of Test Scenarios
•
•
•
•
•
•
•
•
•
•
•
CS 5150
Ergonomics
Usability labs
In-home testing
Customer Support
Consulting
Sales Engineers
“Mail a PC” to family members program
Drive to Fry’s and buy one of everything
Executive dogfood
Executive Mission Statements
Other
11
Dealing with Mixed User Interface Feedback
• Market segmentation (try dividing the feedback by
demographic groups to look for patterns)
• Example 1: Men over age 65; Income < Poverty Line
• Example 2: Ages 20 – 24; college educated;
– “Interface is not cool”
• Ask questions like:
– Why is this group buying the product?
– Which demos like the product and why?
– Is their issue related to the core “buying story”?
CS 5150
12
Beta Testing
• Beta Testing: early release of a product for the purpose of
accumulating feedback that can be integrated into the product.
•
•
•
•
Validates requirements
Uncovers buying issues
Generates testing scenarios
Users have a lot of diversity in both their uses and their
environments. Exploit the diversity to get better data about
the coverage of your test pass.
CS 5150
13
Permutations and Combinations
• How many versions of Microsoft Windows are there?
– 7? 35? 1100?
• A version is a build number plus an install set:
– Windows XP SP 2 with 7 of the 49 security patches
– Win64 vs. Win32
– Office installed vs. not installed
– IDE v SCSI v SATA drivers
– Video adapter driver
– Etc.
CS 5150
14
Durability
• How long should you make the tests run?
– Once?
– Multiple times?
– A number calculated based on analysis?
• Determining MTBF is important because it is a measure of
resilience (frailty). These tests frequently insert random events
in the system as a part of the test harness.
CS 5150
15
Acting on the Results
• Once you run the test pass, what do you do?
– One bug for each failure
• Bug counts go up because
– Test coverage increased
– Code quality went down
– Major regression
– Code base change that requires change in the test pass
• Bug counts go down because:
– Test team is not testing or losing creativity
CS 5150
16
End Game
• The End Game: the process of balancing feedback from sales,
marketing, executives, customers, engineering, testing, etc.
• Usually involves:
– A stabilization of bug find/fix ratios
– A code freeze
– A full test pass and subsequent analysis
– Discussions among the stakeholders that begin with:
Who would it effect if small corporate customers
couldn’t …
CS 5150
17
How to Decide When to Release
Imagine this scenario. What would you do?
The browser has a bug which causes rendering to be off by 3
pixels for certain combinations of tags within <DIV>s
The code works correctly on Windows 10 but not Windows 9.
Fixing the bug would require adding 2,000 lines of code to the
rendering engine
Failure to fix will cause W3C standards advocates to complain.
CS 5150
18
How to Decide When to Release
Imagine this scenario. What would you do?
3 months prior to the scheduled release, you are in a code freeze.
Privacy advocates push Congress to pass legislation that would
disable all cookies in the browser. The legislation passes.
The legislation makes it illegal to sell consumer products or
publish web sites that use cookies.
CS 5150
19
How to Decide When to Release
Imagine this scenario. What would you do?
The Final Four is in two weeks. You run ESPN’s web site
development and the bracket downloader fails on the version
of Windows 10 systems that was released yesterday.
CS 5150
20
Hiring a Test Manager
Test Managers are different
(a) Some are good at hiring testers
(b) Some are good at thinking through business process
(c) Some have deep technical background in important areas
(d) Some have relationships with important players
Test Managers need to be able to articulate arguments in terms
of the requirements. They also need to be able to argue for
the importance of their employees during personnel
reviews.
CS 5150
21
Thanks and Questions
• Thanks to the following people for assistance:
– Joel Maher (a great test engineer that I’ve worked with)
– Bill Walrond (a great PM who provided feedback)
• Questions?
CS 5150
22