Whitepaper
Exploratory Testing
“Why don’t we always use that?”
by
Dennis Joele
Dennis Joele works as a test designer at TriOpSys
and has designed and implemented various tests for
the Hydrographic Service of the Royal Navy, based on
structured testing. In this whitepaper, he will describe
a case study, in which Exploratory Testing has been
chosen. Since both methods are used within one
project, it is possible to make a comparison. The
results of this comparison caused the customer to
raise the question: "Why don’t we always use that?"
This question will also be answered.
Generally, we are accustomed to use structured testing. A
number of conditions should be met. For specification of
the tests, an adequate documented test basis and
sufficient preparation time should be available. If one of
these conditions is not met, another testing technique
might be helpful. This technique is known as Exploratory
Testing.
Whitepaper | Exploratory Testing
>
How it works
According to one of the founding fathers of the technique, Cem Kaner,
Exploratory Testing is defined as: "Any type of testing where the tester
arranges his test design during the execution of the test, and the
information obtained during the test is used to design new and improved
test cases.”
This implies that a number of steps have been taken: a part of the system is
explored (exploring), thought is being given to what should be tested (test
design) and the designed test is performed (test execution).
There are several disadvantages: the repeatability of the tests is difficult,
there is no visibility of the selected test techniques during test design, there
is no guarantee to ensure test coverage and the direction of the tests is
unknown in advance.
To address these disadvantages, James Bach expanded the technology
_____________
with a number of elements, creating a session based Exploratory Testing: a
“Exploratory testing
test charter, a session, notes and a debriefing.
is simultaneous
Test charter
learning, test design,
A test charter describes the task being performed and the goals:

What (scope)

What not (not within scope)

Why (questions to be answered)

How (brainstorm)

Expected Problems

Reference (diagrams/models)
and test execution.”
James Bach
___________
Optionally, the test charter may contain the test techniques which will be
used and the degree of test coverage to be expected. The test charter gives
the approximate boundary conditions of the test.
Session
A session takes place within a certain period of time, for example 2 hours
during which, based on the pre-defined test charter, the system is being
tested. Often, this type of session is carried out in pairs, for example a
tester and a domain expert. The tester ensures the main use of (implicitly)
test techniques and the domain expert complements the test cases with
his/her system and domain knowledge.
Page 2 of 7
Whitepaper | Exploratory Testing
>
Notes and debriefing
For later reflection of the session, notes are made during the session.
These include both the test design that is created during the test and the
test results. Naturally, findings are also noted. This improves the
repeatability of the tests.
After the session, the notes and findings will be shared with the author of
the test charter. Subsequently, it is determined whether the questions from
the test charter were answered.
When to apply
Under certain circumstances, it is possible to use Exploratory Testing. A
number of these conditions are listed below:

There is not enough documented test basis available

There is not enough time available to specify the tests

There is a need for complementation/diversification of formal testing

_____________
techniques
The technique has
Testers with sufficient test knowledge are available
also demonstrated
its worth when
The technique has also demonstrated its worth when quickly trying to find
quickly trying to find
out how a given system is working, to get a clear picture of the quality of a
out how a given
system, or to investigate a specific finding in more details.
system is working,
When not to apply
It is not a foregone conclusion that if one of the above mentioned conditions
is met, Exploratory Testing can be used automatically. There are also
situations in which this technique will prove itself unsuitable, such as:

The client demands provability of performed tests

Testing should be repeated able by others or with a test tool

Critical functionality which could cause great damage if not functioning
to get a clear
picture of the
quality of a system,
or to investigate a
specific finding in
more details.
_____________
properly

When tests are required to be specified extensively

If tests lack immediate feedback, as with batch functionality
Page 3 of 7
Whitepaper | Exploratory Testing
>
Exploratory Testing at the Hydrographic Service
of the Royal Navy
Exploratory Testing is applied in the development of a new civil and military
production system for making paper and electronic charts. The system
consists of several components, some of which are partly acquired
externally and partly developed internally. These components are
integrated into one system, using Workflow Management as an umbrella
layer. This ensures the monitoring of the execution of the business
processes. There are a total of five business processes that are automated
successively. For the first business process structured tests were already
performed.
Exploratory Testing has been introduced as an addition to/diversification of
the formal testing technique used, which is in this case the Process Cycle
Test. This test technique focuses on covering variations in the procedure.
_____________
Varying process steps are not covered, which is a possibility within
Exploratory Testing.
Exploratory Testing is a possibility if testers with sufficient test knowledge
are available. Normally, this requirement would be sufficient. However, the
hydrographic field is very specialized and limited subject matter knowledge
used to be available. To confront these difficulties, an end-user is involved
in the execution of tests.
For the first
business process
there structured
tests were already
performed.
___________
These two conditions were the starting point to use the technique.
Page 4 of 7
Whitepaper | Exploratory Testing
>
Results
In a time span of 2 hours, 11 findings in total were reported. For the related
subsystem a total of 52 findings were given a priority to Exploratory Testing.
These findings have come up during a completed Functional Acceptance
Test (FAT) and User Acceptance Test (UAT). Both took approximately 2
days, the FAT is performed by one person and another three end users
were involved in the UAT.
Effectively Exploratory Testing yielded a findings frequency of 11 findings/4
man-hour = 2.75 findings/man-hour. This calculation is accurate because
the entire time is effectively spent on the test itself. If the preparation time
for the test charter is included as well, this results in a frequency finding of
11 findings/4.5 man-hour = 2.44 findings/man-hour.
For the previous period of structurally testing, a calculation can also be
made, but it is less accurate. Here, the findings frequency is 52 findings/60
man-hours = 0.87 findings/man-hour. There is a factor of 0.75 used for the
time actually spent. In this calculation, the time required for the preparation
and reviewing of the Software Test Description (STD) is not yet included. If
this is set at two man-days, the findings frequency is reduced to 0.68
findings/man-hour.
_____________
For the previous
period of
structurally testing,
a calculation can
also be made, but it
(Translation graphics: red = number of findings per man-hour, excluding
documentation, purple = number of findings per man-hour, including
documentation)
is less accurate.
_____________
Page 5 of 7
Whitepaper | Exploratory Testing
>
Dividing both findings frequencies yields a factor of 2.75/0.87 = 3.16
(excluding documentation) and 2.44/0.68 = 3.59 (including documentation).
It seems that Exploratory Testing is more effective than the tests that were
previously performed. This may be stated because the differences in the
two calculations should be more than this factor, which seems unlikely. The
effect of the time required for documentation can be distinctly seen in the
second calculation. This ensures the stronger decrease of the effective
findings frequency of structured testing than of Exploratory Testing.
In this calculation, we only took the number of findings into consideration,
but not yet the nature of the findings. Eight of the findings from the FAT and
UAT were classified as 'blocking' or 'major' against three in the Exploratory
Testing session. This results in the following findings frequencies: 3
findings/4.5 man-hour = 0.67 findings/man-hour and 8 findings/76 man-hour
= 0.11 findings/man-hour. It has to be noted that two of the findings were
caused by regression. If these are excluded from the calculation,
_____________
Exploratory Testing remains to be twice as effective. Regardless of this, if
Besides the number
both methods have to be equally effective, the structured tests, in this
of findings, we take
specific case should yield more than 16.8 times the number of findings (76
at the nature of the
spent man-hour versus 4.5 spent man-hour). It may be stated that this
findings into
situation is highly unlikely.
consideration.
_____________
Page 6 of 7
Whitepaper | Exploratory Testing
>
Conclusion
Comparing the results of Exploratory Testing and structured testing shows
that if Exploratory Testing is used, the number of findings per man-hour
spent is more than three times higher. This observation is based on 76
hours spent in structured testing against 4.5 hours spent in Exploratory
Testing.
This means that in Exploratory Testing 1/3 of the number of hours spent is
sufficient to find the same number of findings. However, there are multiple
sessions required. This makes Exploratory Testing a lot more effective than
structured testing.
After reporting the results of the session, the head of the Technology &
Development department asked a simple but powerful question. "Nice
email, why don’t we always use Exploratory Testing?"
Of course, it is true that Exploratory Testing cannot be applied in all
situations. For instance, in the case of the testing of high-risk functionality
or when standards require the provability of tests performed. If the
conditions of Exploratory Testing are met, it may prove to be a very
effective way of testing.
_____________
This means that in
Exploratory Testing
1/3 of the number
of hours spent is
sufficient to find the
About TriOpSys
same number of
TriOpSys is specialised in designing, building and managing mission critical
IT systems, which are software systems that may never be offline.
Examples of these software systems are traffic and emergency centers (on
findings
____________
land, water and in the air), bulletin call centers, command centers,
command and control systems.
Our services cover three fields of expertise: IT Consultancy, Software
Development and Mission Critical Services, in three domains: Traffic
Management, Defense & Aerospace and Public Order & Safety and
Security.
Do you want more information about this whitepaper? Or are you looking for
advice? Please contact us at:
info@triopsys.nl
++31 346 – 58 17 80
www.triopsys.nl
Page 7 of 7