Just-In-Time Testing
Workshop
Robert Sabourin
AmiBug.Com, Inc.
Montreal, Canada
rsabourin@amibug.com
April 13, 2015
© Robert Sabourin, 2012
Slide 1
AmiBug.Com, Inc.
Overview
Welcome
Some Philosophy
Be Pre[pared
Testing Ideas
Test Triage
April 13, 2015
© Robert Sabourin, 2012
Slide 2
AmiBug.Com, Inc.
Just In Time Testing
• Robert Sabourin ,
Software Evangelist
• President
• AmiBug.Com Inc.
• Montreal, Quebec,
Canada
• rsabourin@amibug.com
• www.amibugshare.com
April 13, 2015
© Robert Sabourin, 2012
Slide 3
AmiBug.Com, Inc.
Just-In-Time Testing
Welcome
April 13, 2015
© Robert Sabourin, 2012
Slide 4
AmiBug.Com, Inc.
Just-In-Time Testing
• Pain points?
– What hurts?
– How Much?
April 13, 2015
© Robert Sabourin, 2012
Slide 5
AmiBug.Com, Inc.
Just-In-Time Testing
Just In Time For ________?
April 13, 2015
© Robert Sabourin, 2012
Slide 6
AmiBug.Com, Inc.
Just-In-Time Testing
Some Philosophy
April 13, 2015
© Robert Sabourin, 2012
Slide 7
AmiBug.Com, Inc.
Fundamental Question
• How do you know when you are finished?
April 13, 2015
© Robert Sabourin, 2012
Slide 8
AmiBug.Com, Inc.
Crosby on Quality
• “Quality is defined as conformance to
requirements”
• “Quality is not a measure of
GOODNESS”
– Phil B. Crosby, Quality is Free
April 13, 2015
© Robert Sabourin, 2012
Slide 9
AmiBug.Com, Inc.
Joseph Juran
“Quality is fitness for use”
Quality Control
Handbook
April 13, 2015
© Robert Sabourin, 2012
Slide 10
AmiBug.Com, Inc.
Gerald M. Weinberg
“Quality is value to some person”
Exploring Requirements
Quality Before Design
Dorset House
April 13, 2015
© Robert Sabourin, 2012
Slide 11
AmiBug.Com, Inc.
Edsger W. Dijkstra
• “Program testing can be used to show
the presence of bugs, but never to show
their absence”
April 13, 2015
© Robert Sabourin, 2012
Slide 12
AmiBug.Com, Inc.
Quiz
• Application screens
are selected with
three controls:
(a) has 5 options
(b) has 6 options
(c) has 2 options
• How many screens can a user choose?
April 13, 2015
© Robert Sabourin, 2012
Slide 13
AmiBug.Com, Inc.
Quiz
• Total Combinations
= 6 x 5 x 2 = 60
• To exercise each
combination once a
total of 60 tests
would be required.
April 13, 2015
© Robert Sabourin, 2012
Slide 14
AmiBug.Com, Inc.
Quiz
• How many tests
would be required to
exercise all possible
screens in every
possible order?.
Slide 15
April 13, 2015
© Robert Sabourin, 2012
AmiBug.Com, Inc.
Pop Quiz
• To exercise all screens in every
possible order would require
60! Test cases
60! = 60 x 59 x 58 x ... 3 x 2 x 1
60! ≈ 8.32 x 10**81
April 13, 2015
© Robert Sabourin, 2012
Slide 16
AmiBug.Com, Inc.
Pop Quiz
How many atoms are in
the observable
universe?
From 7.0 × 10**79
To
1.5 × 10**82
April 13, 2015
© Robert Sabourin, 2012
Slide 17
AmiBug.Com, Inc.
Control Flow Testing
• Model flow
– Create control flow
diagram
– Find basis paths
• Minimal set of transactions
• Exercise at least once
– Each step
– Each decision
April 13, 2015
© Robert Sabourin, 2012
Slide 18
AmiBug.Com, Inc.
Control Flow Testing
A
B
– Minimal basis paths
• N – number of nodes
• E – number of edges
• P – number of basis paths
• P=E–N+2
• McCabe Cyclomatic Complexity
C
D
N=9
E=10
P=10-9+2
P=3
E
F
G
H
I
April 13, 2015
© Robert Sabourin, 2012
Slide 19
AmiBug.Com, Inc.
Control Flow Testing
A
– Finding basis paths:
1. Start with a typical baseline
2. Flip first decision keep rest as
similar as possible
3. Continue flipping decisions on
baseline
4. After all decisions on baseline
have been flipped continue on
next path
5. Stop when all paths have been
exhausted
B
C
D
N=9
E=10
P=10-9+2
P=3
E
F
G
H
I
April 13, 2015
© Robert Sabourin, 2012
Slide 20
AmiBug.Com, Inc.
Control Flow Testing
April 13, 2015
© Robert Sabourin, 2012
Slide 21
AmiBug.Com, Inc.
April 13, 2015
© Robert Sabourin, 2012
Slide 22
AmiBug.Com, Inc.
ST
Di
AD
AC
AE
AG
X
L
Y
Z
AA
AB
AM
AF
AH AJ
AL
AI
AN
AK AO AP
AQ
M
K
O
A
J
B
C
N
H
P
Q
S
R
T
U
V
April 13, 2015
I
© Robert Sabourin, 2012
D
E
G
F
W
Slide 23
AmiBug.Com, Inc.
Control Flow Diagram to Identify Basis Paths
Basis Paths
p01
p02
p03
p04
p05
p06
p07
p08
p09
p10
p11
p12
p13
p14
p15
p16
p17
p18
p19
p20
p21
p22
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Start
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Dispatch
Nodes
Edges
Complexity
April 13, 2015
L
X
AC
AD
O
K
I
A
L
L
L
X
X
X
X
AC
AC
AC
AC
I
I
A
M
Y
AF
AE
P
P
J
B
M
M
M
Z
Z
Z
Z
AF
AF
AF
AF
J
J
B
N
AA
AJ
AF
Q
Q
C
C
N
N
N
AA
AA
AA
AA
AG
AH
AJ
AJ
C
C
R
P
AB
AI
AJ
R
R
D
D
P
P
P
AB
AM
AM
AM
R
R
AL
AL
D
D
Q
R
AK
AI
R
R
AK
R
H
H
S
S
S
R
R
AN
AN
P
P
T
T
T
Q
Q
Q
U
U
R
R
R
R
V
AP
AO
AQ
AQ
R
R
AM
R
E
E
R
F
G
W
R
R
R
45
65
22 E-N+2
© Robert Sabourin, 2012
Slide 24
AmiBug.Com, Inc.
Purpose of Testing
• Common definition:
– To find bugs before our customers do!
• Broader definition:
– The role of testing is to provide objective input to
facilitate business decisions!
– Keeps stakeholders aware of all issues or
concerns that relate to shipping a product!
April 13, 2015
© Robert Sabourin, 2012
Slide 25
AmiBug.Com, Inc.
Bug Defined
• To make our job more fun, whenever
we have a concern with software, we
call it a “bug”.
April 13, 2015
© Robert Sabourin, 2012
Slide 26
AmiBug.Com, Inc.
Just-In-Time Testing
• It’s all about people! (and the occasional
bug too)
April 13, 2015
© Robert Sabourin, 2012
Slide 27
AmiBug.Com, Inc.
Just-In-Time Testing
Context Drivers
April 13, 2015
© Robert Sabourin, 2012
Slide 28
AmiBug.Com, Inc.
Context Drivers - BTO
• Business
– Value
– To whom?
– Why?
• Technology
– Solutions
• Organization
– Corporate Structure
– Team Structure
– Roles and Responsibilities
April 13, 2015
© Robert Sabourin, 2012
Slide 29
AmiBug.Com, Inc.
April 13, 2015
© Robert Sabourin, 2012
Slide 30
AmiBug.Com, Inc.
Context Listeners
•
•
•
•
Find Sources
Monitor Drivers
Anticipate Change
React
April 13, 2015
© Robert Sabourin, 2012
Slide 31
AmiBug.Com, Inc.
Just In Time Testing
Get Ready, Get Set,
Cause here it comes
April 13, 2015
© Robert Sabourin, 2012
Slide 32
AmiBug.Com, Inc.
Just-In-Time Testing
Turbulence
April 13, 2015
© Robert Sabourin, 2012
Slide 33
AmiBug.Com, Inc.
Just-In-Time Testing
Unprepared
April 13, 2015
© Robert Sabourin, 2012
Slide 34
AmiBug.Com, Inc.
Just-In-Time Testing
Sharpen Testing Skills
• Thinker
• Detective
• Reporter
• Diplomat
• Negotiator
• Cheer Leader
• Pragmatist
April 13, 2015
© Robert Sabourin, 2012
Slide 35
AmiBug.Com, Inc.
Philosophy
We have precious little time to run tests!
We must always be prepared!
April 13, 2015
© Robert Sabourin, 2012
Slide 36
AmiBug.Com, Inc.
Time
April 13, 2015
© Robert Sabourin, 2012
Slide 37
AmiBug.Com, Inc.
Just In Time Testing
Test Triage
April 13, 2015
© Robert Sabourin, 2012
Slide 38
AmiBug.Com, Inc.
Plan to support change
Yoda
"No! Try not, Do. Or do not.
There is no try."
April 13, 2015
© Robert Sabourin, 2012
Slide 39
AmiBug.Com, Inc.
Testing Ideas
Plan to support change
• Collect all testing ideas you can find!
–
–
–
–
List
Sort
Organize
Shuffle
April 13, 2015
© Robert Sabourin, 2012
Slide 40
AmiBug.Com, Inc.
Testing Ideas
Plan to support change
• How to find them?
–
–
–
–
–
–
–
Does system do what it is suppose to do?
Does the system do things it is not supposed to?
How can the system break?
How does the system react to it’s environment?
What characteristics must the system have?
Why have similar systems failed?
How have previous projects failed?
April 13, 2015
© Robert Sabourin, 2012
Slide 41
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Collect testing ideas
• From testing ideas build a
series of testing objectives
– Each can be assigned as
work to testers
– Each can include all, part of,
or multiple testing ideas
April 13, 2015
© Robert Sabourin, 2012
Slide 42
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• I often use Index Cards
– Unique id
– One testing idea per card
– Colour indicates source
– Shuffled and reviewed
– Organized and reorganized
– Sorted, grouped, prioritized and collected
April 13, 2015
© Robert Sabourin, 2012
Slide 43
AmiBug.Com, Inc.
Capture testing ideas
Test Idea Sources
•
•
•
•
•
•
•
•
•
•
Capabilities
Failure Modes
Quality Factors
Usage Scenarios
Creative Ideas
States
Data
Environments
White Box
Taxonomies
April 13, 2015
© Robert Sabourin, 2012
Slide 44
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Investigative
approaches
– We become truffle
snorting pigs and try to
find useful information in
all evidence we discover
– We can even get good
ideas from out of date
sources
April 13, 2015
© Robert Sabourin, 2012
Slide 45
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Capabilities
– Use cases
– Functional
requirements
– Documented
requirements
– Implicit requirements
April 13, 2015
© Robert Sabourin, 2012
Slide 46
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Failure Modes
– What can break?
– Reaction to invalid input?
– How does software
behave in constrained
environment?
•
•
•
•
•
Memory
Disk Space
Network Bandwidth
CPU capacity
Shared resources
– Stress, Load, Volume
April 13, 2015
© Robert Sabourin, 2012
Slide 47
AmiBug.Com, Inc.
Adaptability
Accessibility
Auditability
Availability
Continuity
Dependability
Expandability
Functionality
Integrity
Interoperability
Maintainability
Operability
Portability
Reliability
Re-usability
Scalability
Security
Serviceability
Testability
Usability
Capture testing ideas
Application service provider
Automatic content generator
Customized access
Database access
Delivery
Document access
File sharing
Informational
Interactive
Transaction oriented
User-provided content
Workflow oriented
High Focus
Medium Focus
Low Focus
April 13, 2015
Quality Factors Importance
Different Application Types
© Robert Sabourin, 2012
Slide 48
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Usage Scenarios
– Identify classes of
users
– Identify how users
will use system
– Describe scenarios
– Use Story board or
similar approaches
– Identify variations
April 13, 2015
© Robert Sabourin, 2012
Slide 49
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Creative
approaches
–
–
–
–
Action verbs
Mind Maps
Soap Operas
Lateral Thinking
April 13, 2015
© Robert Sabourin, 2012
Slide 50
AmiBug.Com, Inc.
Testing Ideas
power up
service
needed
reset button
coin inserted
idle
inserting
coins
Capture testing ideas
coin return
no cups
OR no coffee
OR sensor jam
cup removed
coin return
make
coffee
button pushed
right amount
entered
user
choose
State Models
April 13, 2015
© Robert Sabourin, 2012
Slide 51
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Data
–
–
–
–
–
Flow
Structure
Create
Update
Change
April 13, 2015
© Robert Sabourin, 2012
Slide 52
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Environment
–
–
–
–
–
–
–
Hardware
Software
Operating systems
Locales
Browsers
Plug-ins
Co-dependent software
April 13, 2015
© Robert Sabourin, 2012
Slide 53
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• White Box
– Design
– Internal structure
– Code
April 13, 2015
© Robert Sabourin, 2012
Slide 54
AmiBug.Com, Inc.
Testing Ideas
Capture testing ideas
• Bug taxonomies
– Collections of possible bugs
– Appendix A of Testing
Computer Software, Kaner,
Falk, Nguyen
– Boris Biezer Taxonomy Otto
Vinter manages
– Shopping cart taxonomy Giri
Vijayaraghavan
April 13, 2015
© Robert Sabourin, 2012
Slide 55
AmiBug.Com, Inc.
April 13, 2015
© Robert Sabourin, 2012
Slide 56
AmiBug.Com, Inc.
Which test?
•Impact estimation
Triage testing ideas
–For each test idea guesstimate:
•
•
•
•
benefit of implementation
consequence of implementation
benefit for not implementing
consequence of not implementing
–How credible is the information?
April 13, 2015
© Robert Sabourin, 2012
Slide 57
AmiBug.Com, Inc.
Which test?
Test Idea Rejection – What If?
Triage testing ideas
–If the cost/benefit does not make business
sense then consider implementing:
• part of the test, could that lead to part of the
benefit at a more reasonable cost?
• more than the stated test, would that generate
more benefit?
• a different test than the stated idea, could that
generate more benefit for less cost?
April 13, 2015
© Robert Sabourin, 2012
Slide 58
AmiBug.Com, Inc.
Test Triage
• Test Triage Meeting
– Review Context
Triage testing ideas
• Business
• Technical
• Organizational
– New Information
• Test results
• Bug results
• New testing ideas
April 13, 2015
© Robert Sabourin, 2012
Slide 59
AmiBug.Com, Inc.
Test Triage
Triage testing ideas
• Allocate Testing Assignments to Testers
–
–
–
–
–
–
Make sure testers know context
Best thing to test
Best person to test it
Best people to explore it
Best lead
Are subject matter experts required
April 13, 2015
© Robert Sabourin, 2012
Slide 60
AmiBug.Com, Inc.
Test Triage
Life of a test idea
Triage testing ideas
a.
b.
c.
Comes into existence
Clarified
Prioritized
a.
b.
c.
d.
e.
f.
d.
Test Now (before further testing)
Test before shipping
Nice to have
May be of interest in some future release
Not of interest in current form
Will never be of interest
Integrate into a testing objective
April 13, 2015
© Robert Sabourin, 2012
Slide 61
AmiBug.Com, Inc.
Which test is next?
Triage testing ideas
• Questions
– Given state of project, state of business, state of
technology, our abilities, our experience and our
history, what we know and what we do not know,
what should we test next?
– How much effort are we willing to spend
continuing to test this project?
– Can we ship yet?
April 13, 2015
© Robert Sabourin, 2012
Slide 62
AmiBug.Com, Inc.
Deciding what not to
test?
Triage testing ideas
• Time pressure
– Should we skip a test?
– If test failed could system still be of
value to some stakeholder?
– If test was skipped could important
bugs have been otherwise found?
April 13, 2015
© Robert Sabourin, 2012
Slide 63
AmiBug.Com, Inc.
Guidelines and
Decisions
Get Started Right
• To each stakeholder
–
–
–
–
–
risk of failure
consequence of failure
value of success
how much certainty do we have
is it a wild guess or an absolute
truth?
April 13, 2015
© Robert Sabourin, 2012
Slide 64
AmiBug.Com, Inc.
Bottom Line
Get Started Right
• My experience is that it is better to
omit a test on purpose than to skip it
because you ran out of time or
forgot about it!
• Systematically collecting, evaluating
and triaging testing ideas helps me
decide what not to test - at least for
now?
April 13, 2015
© Robert Sabourin, 2012
Slide 65
AmiBug.Com, Inc.
Thank You
• Questions?
April 13, 2015
© Robert Sabourin, 2012
Slide 66
AmiBug.Com, Inc.