GUI testing
•
There are two major sets of GUI testing
1. “Usability” of the GUI
•
Looks (aesthetic, visual appeal) - - - artistic characteristic
–
–
–
–
•
Color
Shapes
Icons and Positions
Acronym/ Spelling/ Grammar
Navigation (natural flow) - - - business flow characteristic
– Past cultural and work experience (e.g. accessibility from menu or tool bar))
– Amount of (or number) of choices and combination of choices
– Shortcuts and options for experienced user
2. “Logic” of GUI
•
Coverage of the data inputs/outputs of GUI fields
–
–
–
–
–
Input box
Check box
Drop down window list
Button choices
Etc.
We will focus on the Functional Logic of GUI fields and the coverage of
data inputs in this lecture.
But first - - -Testing the Usability of GUI
• Usability is largely based on
– user perceptions and thus based on user feedback.
• Should not wait until the end of the development cycle:
– Perform “paper” inspection of interface requirements/design
(with users)
– Testing “early prototypes” with users to:
• Explore users’ mental models (looks, flow, etc.)
• Evaluate alternatives
• Validate the choice (in terms of speed, accuracy, etc.)
– Testing with functional code (incrementally with the
interfaces.)
– Perform a, possibly, formal usability test with :
• Good statistical sampling of users
• Usability expert to design the test and evaluate the results
Testing GUI Fields
•
In order to test a program or system, it usually
requires :
1.
2.
3.
4.
•
Starting the system (program)
Inputting some data
Let it execute
Observe the results (outputs and state)
What data do we “input?”
–
Note that inputting data is not limited to keying in
information. It includes clicking on a “button’ or
making a choice from a “drop down” list. For
example:
•
•
Clicking on a button is a binary event – clicked or not
clicked
If there are n buttons on a screen, then we have 2n
permutations of “inputs” that need to be considered.
Consider an Example
Cruise Meal Reservations
Key in # of people (max. 12):
Breakfast
Lunch
Dinner
For this meal reservation, we have 4 input fields.
- key in number of people
- 3 buttons to click for the meal or meals
What type and how many “inputs should we consider
for the test ?
Note: how do you think error or any message would/should be handled here?
Example (cont.)
-A) First, the numerical input field
allows “integer” inputs.
-B) Second, the integer must not
be greater than 12
-C) Third, there are 2 3 = 8
permutations for the 3 meal
buttons
Breakfast
button
True
Dinner
button
Lunch
button
False True
X
X
X
X
False True
X
X
X
X
X
X
X
X
X
X
False
X
X
X
X
X
X
X
X
X
X
These 8 combinations equates to 8 input
test cases. But rows are the 3 logical ones?
“Partitioning” Numerical Field Test (from example)
The “illegal” input test for numerical field. In this
case “number of people” has to be positive
integers. Input tests should include:
1.
–
–
–
Non-integer (e.g. decimal number)
Negative integer
non-numeric (characters)
special kind of “out
of bounds” test
The “legal” integer is “upper” bounded by 12:
2.
–
–
–
Integer 12 (inclusive boundary)
Integer 13 (one over the boundary)
Integer 11 (one inside the boundary)
more conventional
boundary value test
Is there a “lower” bound? - - - not clear
3.
–
–
–
Assume integer 1 (inclusive boundary)
0 (one outside the boundary)
2 (inside the boundary)
The different inputs considered for the numerical field equates to 9 test cases.
More on the Example
• Should we just add up the two separate sets of test cases and get 17
input tests?
– 8 from the 3 buttons
– 9 from the numerical field
• We can probably reduce some from the numerical field tests:
– Just do one inside the boundary instead of both sides of the boundary.
– One outside of the lower bound instead of 0 and a negative number.
• Do we need to ask if there is any relationship among the input fields?
– Should we ensure that the combination of legitimate numerical input (e.g. 8
people) is combined with all three buttons not clicked - - - to test the interrelationship?
– What type of error message, if any, should be issued if the user keys in 0
people and no buttons clicked?
– Should we worry about some other possible relationship that we did not
consider and thus perform 8 X 9 = 72 test cases? to see the different or
consistency of “error” messages?
• Message box
• Message content
• Opportunity to recover
1-Dimensional Boundary Value Analysis
• Our previous example of numerical field had an upper and a
lower bound of 12 and 1 respectively. There are two way to
consider the boundary value analysis.
1. Legitimate to illegitimate includes :
a) 2 (legitimate-inside boundary)
b) 1 (legitimate – on boundary)
c) 0 (illegitimate – outside the boundary)
d) 11 (legitimate –inside the boundary)
e) 12 (legitimate – on boundary)
f) 13 (illegitimate – outside the boundary)
legitimate
0
1
2
11
13
12
May choose to reduce to the set
with 5 inputs { 0, 1, 2, 12, 13}
2. Illegitimate to legitimate includes :
a) -1 (illegitimate-inside boundary)
illegitimate
b) 0 (illegitimate – on boundary)
c) 1 (legitimate – outside the boundary)
d) 14 (illegitimate – inside the boundary)
-1
e) 13 (illegitimate – on boundary)
0
f) 12 (legitimate – outside the boundary)
illegitimate
1
12
14
13
May choose to reduce to the set with 5 inputs {-1, 0, 1, 13, 14 }
or with only 4 inputs { 0, 1, 12, 13} - - - uncomfortable ?
Multi-Dimensional Boundary Value Analysis
(Case 1 – Partially Dependent)
Book check-out Date;
Book check-in Date:
- Assume that for these two fields we have performed the
boundary analysis and estimated the test cases:
- check-out date : n1 input test cases
- check-in date : n2 input test cases
-Then the total test cases will be (n1 + n2).
- But is there any inter-relationship?
- Yes, book checkout date must be before or same as
book check-in (return) date.
- ensure that that the 3 cases of a) =, b) check-out > check-in,
and c) check-out< check-in are in the ( n1+n2 ) input tests or
one needs to add these.
Multi-Dimensional Boundary Value Analysis
(Case 2 – Dependent)
Flight Time :
Flight Date :
Consider the situation where a discount is given to flights between:
- 6 PM to 12AM (inclusive) and
- 6/1/2005 through 12/31/2005 (inclusive)
Aside from the individual field level 1-dimensional boundary value analysis,
we need to take into account the “AND” logic for getting a discount. Individually,
this logic requires:
- {5:59pm, 6:00pm, 6:01pm, 11:59PM, 12:00AM, 12:01AM} for Flight Time
- {5/31/2005, 6/1/2005, 6/2/2005, 12/30/2005, 12/31/2005, 1/1/2006} for Flight Dates
( note that we have reduced the one duplication inside the boundary )
Multi-Dimensional Boundary Value Analysis
(Case 2 – Dependent) – cont.
Because of the ‘AND’ logic for the time and dates, we would consider
the (5 “ time” inputs X 5 “date” inputs) or a total of 25 combinations.
5:59 PM
6:00 PM
6:01 PM
5/31/05
(ill, ill)
(ill, leg)
(ill, leg)
6/01/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
6/02/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
12/31/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
1/01/06
(ill, ill)
(ill, leg)
(ill, leg)
12:00 AM
(ill, leg)
(ill, leg)
the entries are in the form : (date, time) where
- ill= illegitimate
- leg = legitimate
12:01 AM
(ill, ill)
(ill, ill)
Multi-Dimensional Boundary Value Analysis
(Case 2 – Dependent) – cont.
Do we need to run all 25 combinations ---- can we reduce the test cases?
5:59 PM
6:00 PM
6:01 PM
12:00 AM
12:01 AM
5/31/05
(ill, ill)
(ill, leg)
(ill, leg)
(ill, leg)
(ill, ill)
6/01/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
6/02/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
12/31/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
1/01/06
(ill, ill)
(ill, leg)
(ill, leg)
(ill, leg)
(ill, ill)
Cover all the borders scheme with a very limited saving - 24 test cases:
a) (ill,ill)  (leg,leg); b) (ill, leg)  (leg,leg); c) (leg,ill)(leg,leg)
Multi-Dimensional Boundary Value Analysis
(Case 2 – Dependent) – cont.
Do we need to run all 25 combinations ---- can we reduce the test cases?
5:59 PM
6:00 PM
6:01 PM
5/31/05
(ill, ill)
(ill, leg)
(ill, leg)
6/01/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
6/02/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
12/31/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
1/01/06
(ill, ill)
(ill, leg)
(ill, leg)
12:00 AM
(ill, leg)
(ill, leg)
12:01 AM
(ill, ill)
(ill, ill)
Cover some of the borders scheme with more savings – 12 test cases:
(ill,ill)  (leg,leg); (ill, leg)  (leg,leg); (leg,ill)(leg,leg)
Multi-Dimensional Boundary Value Analysis
(Case 2 – Dependent) – cont.
Do we need to run all 25 combinations ---- can we reduce the test cases more?
5:59 PM
6:00 PM
6:01 PM
5/31/05
(ill, ill)
(ill, leg)
(ill, leg)
6/01/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
6/02/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
12/31/05
(leg, ill)
(leg, leg)
(leg, leg)
(leg, leg)
(leg, ill)
1/01/06
(ill, ill)
(ill, leg)
(ill, leg)
12:00 AM
(ill, leg)
(ill, leg)
12:01 AM
(ill, ill)
(ill, ill)
Try Partitioned Data sets of (ill, ill); (ill, leg); (leg,ill); (leg,leg) – 4 test cases:
( NOT a good choice to use partitioned data sets on this 2-dimensional
boundary value analysis )
Multi-Dimensional Boundary Value
Analysis
• We have seen in our example of 2 inter-related data
fields, each having 5 potential inputs from boundary
value analysis, would result in 5 x 5 = 25 test cases
(even though the number may be reduced - - - with risk)
• In general, if we have Z inter-related data fields, each
with n test cases from individual boundary value
analysis, then we would have:
– n x n x - - - - x n (Z times) or
– nZ test cases
• if n= 5 and z = 3, that would mean 53 = 125 test cases
• If we have three independent screens like this it would mean 3 x
125 = 375 test cases !
• But if the 3 screens are dependent on each other, we may have as
much as 125 x 125 x 125 = 1,953,125 test cases! (imagine that!)
An Interesting Multi-Scenario Test Analysis
Enter
Flight Number :
OR
Departing Loc. :
Arriving Loc. :
Approximate Arrival Time:
Check Flight
Status
An Interesting Multi-Scenario Analysis
schedule dept-time
Flight Number
XYZ-126
actual dept-time
2:00PM
scheduled arr-time
actual arr-time
6:00pm
Test for : Prior to flight takes off
schedule dept-time
Flight Number
XYZ-126
2:00PM
actual dept-time
2:15PM
scheduled arr-time
actual arr-time
6:00pm
Need to test for : after flight takes off
What do you need to do to test this ?
How about actual arrival time ? ---- relation to dept time ?