Testing Computer Software
Mike Stewart
Microsoft
FoxPro User Group World Tour
Who Am I and Why Should You Care?




Software Test Lead for the Visual FoxPro team
Numerous Knowledge Base articles and white
papers for Microsoft
Consultant developing custom applications
using VFP, C++, Delphi and VB
Unix sysadmin and systems programmer
using C++
FoxPro User Group World Tour 2002
What this Session is Not




A lot of code examples – no example can fit
every situation
Testing the “Microsoft way” – this is all based
on software QA industry methodologies
One size fits all – everyone’s testing needs are
different
For people with aversions to blue Power Point
slides
FoxPro User Group World Tour 2002
Why Test?



Customer satisfaction – testers are customer
advocates
Cost – the later a bug is found, the more it
costs to fix
Quality – the earlier a bug is found, the more
likely it is to be fixed
FoxPro User Group World Tour 2002
What is the goal of testing?




Find bugs – more specifically, get bugs fixed
Verify that it works
Prevent errors before they get into the product
Provide information on risk – how solid is the
product?
FoxPro User Group World Tour 2002
Important Things to Remember


“A test that reveals a problem is a success. A
test that did not reveal a problem was a waste
of time.” - Kaner
You will not find all of the bugs



Myers’ 20 line program: 100 trillion paths
Program takes two inputs of four-digit number:
almost 400 million possibilities
Test something 10,000 iterations: what if it
fails at 10K + 1?
FoxPro User Group World Tour 2002
Exhaustive Testing

“The only exhaustive testing
there is, is so much testing
that the tester is exhausted!”
— Hetzel
FoxPro User Group World Tour 2002
“White Box” vs. “Black Box” Testing



White box testing treats the software as a box that
you can see into.
 You are looking into the box to see how it
works
 Assumes you have source code
Black box testing treats the software as a box you
cannot see into.
 Put in data (input) and see what comes out of
the box (output).
 No source code needed.
Black box testing on finds about 55% of the bugs
FoxPro User Group World Tour 2002
Why White Box Testing isn’t Covered




Doesn’t test the software in the manner in
which it will be used
It doesn’t do spec testing – it has no way of
knowing if it is functioning as designed or not
It does not find missing code – some estimate
this accounts for 30% of bugs
Not enough time! It’s a discipline of its own,
and we’d be here all day.
FoxPro User Group World Tour 2002
Steps in testing

Test planning





Write test plans
Spec and code reviews
Build verification – is it even stable enough to
test?
Function and system testing – run the full
suite of test cases
Media verification



No viruses
Bits on the media are the release bits
The product can be installed from the media
FoxPro User Group World Tour 2002
Essential Testing Issues

Establish a bug tracking system



Establish processes





ATS Web – free at
http://msdn.microsoft.com/vstudio
No sticky notes
Use cases
Test plans
Test cases
Bug entry and resolution
Automate, automate, automate!
FoxPro User Group World Tour 2002
Bug Tracking

Accountability



Metrics




Assures that the bug does not fall through the
cracks
When closed, we are assured that it is fixed.
How great is the severity of bugs found?
How fast are we closing bugs?
Are we ready to ship?
Liability

CYA – cover yourself against legal liability (I
am not a lawyer, however)
FoxPro User Group World Tour 2002
Test Plans



A test plan is a general “plan of attack”, not a
detailed list of test cases.
List areas to test, and how they will be tested.
Notepad File/Open example
“Notepad’s File/Open functionality will be tested
for various file types, file size limits, file name
limits and regional encoding. Various media
need to be considered, such as removable
media and networks.”
FoxPro User Group World Tour 2002
Test Cases



Each task can be compartmentalized into
separate automation programs, or
conglomerated into one big program.
Keep test cases atomized: test one particular
feature at a time.
Create a matrix, and build test cases from that


Could be something as simple as an Excel
spreadsheet
Example: the Notepad File/Open dialog
FoxPro User Group World Tour 2002
Test Cases for File/Open/Filename














Long file names – boundary testing
Single letter file names – boundary testing
Long extensions – boundary testing
Edit box text length – boundary testing
Invalid characters in file names – functional
Valid file names – functional
Wild card characters – functional
Edit keys (Ctrl-C, etc.) – functional or spec
Paste in binary from clipboard - pathological
MRU dropdown – functional
Path completion – functional
Large fonts – functional
Gain and lose focus – functional
High ASCII chars - functional
FoxPro User Group World Tour 2002
Bug Entry and Resolution

Minimum to track:





Severity
Priority
The tester who found it
The dev who fixed it
Enter good bugs:



The easier to reproduce, the more likely it is to
get fixed
Check on a second machine – could be a
configuration/OS issue
If it only happens on one OS, say so
FoxPro User Group World Tour 2002
Bug Entry and Resolution



If you’re org is tracking metrics, close
resolved bugs as quickly as possible
Set severity appropriately, and leave the
priority to the project manager(s)
Your org will have to decide how to set
severity and priority – everyone has different
needs
FoxPro User Group World Tour 2002
Testing Methodologies


Smoke testing – basic functionality
Exploratory testing



Testing while learning the product
Misses large areas of the product
Boundary conditions




Cannot test every value, so test for the most
important values
Hex boundaries and the boundary of field
length
Boundary -1, boundary, and boundary +1
Also test largest value allowed, if it is not a
hex value
FoxPro User Group World Tour 2002
Testing Methodologies

Stress




What happens when a large number of users
hit it?
Low memory, low disk space, large amounts of
data, repetitive
Bugs can often not be reproduced
Web Application Stress Tool is great for web
apps.
FoxPro User Group World Tour 2002
Testing Methodologies

Functional



Performance


Does it do what the spec says it should do?
How does it handle error conditions?
A discipline unto itself, worthy of an entire
session
Regression testing


Make sure nothing got broken
Does not find a lot of bugs
FoxPro User Group World Tour 2002
Testing Methodologies




Code Coverage – for VFP, use the Coverage
Profiler
Code coverage testing tells you what code has
not been hit, and therefore needs to have test
cases that hit this code.
You are unlikely to get 100% coverage
The VFP Coverage Profiler can also reveal
bottlenecks.
FoxPro User Group World Tour 2002
Testing Methodologies

Race conditions




Two pieces of code are colliding
Especially important in multi-threaded and
multi-user apps
Error conditions - does it fail gracefully?
UI


Tab order is easy to miss if you are a mouseaddicted developer
When testing as a dev, do not use the mouse,
because you know the mouse will work.
FoxPro User Group World Tour 2002
Testing Methodologies

Ad Hoc testing – “guerrilla testing”



Great for checking basic functionality
Can give broader testing than more formal
means
Hard to verify coverage
FoxPro User Group World Tour 2002
Automation


The Active Accessibility test harness included
with VFP 7.0
Other automation products:


Rational – Visual Test
Mercury Interactive – Testing suite with
various products for web and thick client
applications
FoxPro User Group World Tour 2002
Automation Tips

The test should be self-documenting




Do not use “pass” or “fail” as the test result
Test should say what it expected and what it
received, and whether it passed or failed.
Write “drivers” or hooks into your software if
needed – for example, VFP’s Active
Accessibility or Project Manager hooks
Tests should be self-sustaining – create your
own data, and clean up. Expect nothing.
FoxPro User Group World Tour 2002
When Do You Ship?


You will ship with bugs if your testing is
adequate.
Define show-stoppers




Define risk


Will customers find the bug?
Does it crash your product?
Will only pathologic testers find it?
Fixing a bug has a 60% risk of introducing new
bugs
When the show-stoppers are fixed, ship
FoxPro User Group World Tour 2002
Selling Testing to Client




Bulk contracts – sell the job, not the hours to
develop it, and the hours to test it.
Once the software is in the customer’s hands,
bugs are much more expensive to fix
How much does it cost to have 30 users find a
bug vs. a qualified, dedicated test effort at
$x/hour, that assures quality?
Other ideas?
FoxPro User Group World Tour 2002
“Things They Should Teach in the
Testing Stuff Class, but Don’t”



Run a high-color scheme like Marine HighColor
Use large fonts (Display/Settings/Advanced)
Throw away your mouse
FoxPro User Group World Tour 2002
More information




Testing Computer Software – Kaner
The Art of Software Testing - Myer
Software Quality Testing Engineering
magazine: http://www.stickyminds.com
STAR Conference – the DevCon for
professional software testers (www.stqe.com)
FoxPro User Group World Tour 2002
ADVISOR DEVCON Web Update Page
http://www.Advisor.com/CMF0109update
This session WILL have updates.
FoxPro User Group World Tour 2002
Thank you!
Please remember to fill out your
evaluation.
FoxPro User Group World Tour 2002