Testing Erlang-OTP with QuickCheck
Master of Science Thesis in the Programme Software Engineering and Technology
CRYSTAL CHANG DIN
Department of Computer Science and Engineering
CHALMERS UNIVERSITY OF TECHNOLOGY
UNIVERSITY OF GOTHENBURG
Göteborg, Sweden, June 2009
Supervisor: John Hughes
Department of Computer Science and Engineering
Chalmers University of Technology
SE-412 96 Göteborg
Sweden
Telephone + 46 (0)31-772 1000
Copyright © Crystal Chang Din, June 2009.
E-mail: crystal@student.chalmers.se
Contents
List of Figures .......................................................................................................................... iii
List of Tables............................................................................................................................ iv
Abstract ..................................................................................................................................... v
Acknowledgement ................................................................................................................... vi
1. Introduction .......................................................................................................................... 1
1.1 Purpose ......................................................................................................................... 1
1.2 Scope ............................................................................................................................. 1
1.3 Incompleteness............................................................................................................. 1
1.4 Tool ............................................................................................................................... 2
1.5 Contribution ................................................................................................................ 2
2. QuickCheck........................................................................................................................... 3
2.1 Generators ................................................................................................................... 3
2.2 Properties ..................................................................................................................... 4
2.3 Symbolic Representation ............................................................................................ 5
3. Testing Pure Functions ........................................................................................................ 6
3.1 Data Generators .......................................................................................................... 6
3.2 Data Structure Generators ......................................................................................... 7
3.3 Model Based Testing ................................................................................................... 9
3.4 Information Hiding ................................................................................................... 10
3.5 Case Study.................................................................................................................. 10
Case 1: ........................................................................................................................ 10
Case 2: ........................................................................................................................ 12
3.6 Findings ...................................................................................................................... 15
4. QuickCheck State Machine ............................................................................................... 24
4.1 Initial state ................................................................................................................. 24
4.2 Function “command”................................................................................................ 25
4.3 Function “precondition” ........................................................................................... 25
4.4 Function “postcondition” ......................................................................................... 25
4.5 Function “next_state” ............................................................................................... 25
5. Testing Imperative Functions ........................................................................................... 27
5.1 Module digraph ......................................................................................................... 27
Bug1 ............................................................................................................................ 27
i
Bug2 ............................................................................................................................ 32
Bug3 ............................................................................................................................ 33
5.2 Module dets ................................................................................................................ 34
Bug4 ............................................................................................................................ 34
Bibliography ........................................................................................................................... 39
Appendix ................................................................................................................................. 40
A. User Manu ................................................................................................................... 40
A-1 Pure Functions ................................................................................................... 40
A-2 Imperative functions .......................................................................................... 55
B. Complete Code ............................................................................................................ 56
ii
List of Figures
3-1: Transformation from decimal to Erlang number.......….…….…………………..9
3-2: Transformation from queue to list.…………………..…………………………10
4-1: A complete cycle of a QuickCheck state machine test case……………………24
4-2: Process during generation time.……………………………….……………….26
4-3: Process during run time.……………………………………………………..…26
5-1: Edge added.……………………………………………………………….....…31
5-2: Edge modified.……………………………………………………………....…31
iii
List of Tables
3-1: Size functions………………………………………...………………..…………11
3-2: Containment functions…………………………………………………………...14
3-3: Deletion functions………………………………………………………..………23
3-4: Comparison of (==) and (=:=).……...…………………………………………...23
5-1: Entire process of QuickCheck state machine (digraph)………………………….28
5-2: Entire process of QuickCheck state machine (dets)…..…………………………35
iv
Abstract
Erlang/OTP is a concurrent functional language developed and maintained by Ericsson
AB, and was released as open source in 1998. Though Erlang/OTP has been tested for
many years, it is still possible to find more bugs since some test cases have not been
written yet. The commercial version of QuickCheck for Erlang was recently developed
by Quviq AB. It is a tool replacing conventional test cases by writing specification of
properties under test that the tested function should fulfill. Moreover, QuickCheck also
provides libraries for testing operations containing side-effects, which are specified via
an abstract state machine. Based on that, QuickCheck is a testing tool mainly applied
in this thesis that focuses on Erlang/OTP data-structures.
In the first part, it starts with analyzing properties of pure functions in various datastructure modules. After that, we develop extensive test suites. As long as the
functions having the same property, they can be tested through the same property.
In the second part, we test imperative functions of the rest of data-structure modules
by using QuickCheck state machine. The bugs which were found in Erlang/OTP as
well as the method which was used to identify the bugs will also be described in the
later part.
v
Acknowledgement
This work will not be possible without kindly support from many people.
First of all, I would like to deeply thank my supervisor, professor Dr. John Hughes,
who guided me into such an exciting field. His broad and deep knowledge in
QuickCheck and software verification inspired me so much. I would like to express
my sincere thank to Nick Smallbone, who constantly gave me brilliant advices and
support when I met difficulties.
Professor Reiner Hähnle and professor Wolfgang Ahrendt provided me knowledge in
software verification field which is critical for QuickCheck. I want to thank Angela
Wallenburg who shared her experience of working in software verification. I would
like to thank Mary Sheeran who introduced me to ProTest project and QuickCheck. I
really appreciate the training provided by professor Thomas Arts about QuickCheck.
I would like to thank my dear family, who warmly encourage me to pursue my dream
in Sweden and support me with unconditional and infinite love. I also appreciate their
sincerely prayer. I would like to thank all my friends in Taiwan and in Sweden.
Because of them, I do not feel lonely abroad. I would like to particularly thank Cathy
Ni, who is my best friend and sister in Lord and a good company especially in Sweden.
Though our research fields are totally different, we worked together on our thesis and
supported each other from the beginning. Finally, I would like to thank my dear Lord,
who loves me and participates in my life in every moment. All I have is from Him.
vi
1. Introduction
1.1 Purpose
Message passing is the main idea in Erlang, which is used to develop telecom and
internet products such as switches, email gateways, instant messaging services, semistructured databases and game servers. Therefore, to design different data structures
for installing information wisely and make sure all of them operate as they should be
is a very challenging subject. Otherwise, several unpleasant situations could occur, e.g.
saved data are not consistent with the order of operations.
1.2 Scope
This thesis is separated into two parts for testing data-structures under the application
stdlib-1.16.1 of Erlang/OTP. Each data structure is implemented as an Erlang module.
The first part is about testing the modules consisting of pure functions.

key-value structure: dict module, orddict module and gb_trees module.

set structure: sets module, ordsets module and gb_sets module.

list structure: lists module, proplists module and string module

queue structure: queue module

array structure: array module
The second part is about testing the modules consisting of imperative functions.

directed-graph structure: digraph module

sets, ordsets, bag and duplicate_bag structure: ets module

sets, bag and duplicate_bag structure: dets module
1.3 Incompleteness
A data-structure is expected to perform some tasks in a specific manner. If the datastructure does not behave the way it is intended to, errors occur accordingly.
Therefore, to design an effective set of test cases that enable uncovering maximum
errors was the way been used to improve the quality of Erlang/OTP. However, writing
all test cases manually is not easy, especially when the complexity of requirements
increases. Some errors might still have not been found.
1
1.4 Tool
QuickCheck is the tool mainly used in this thesis. Together with a new phenomenon,
property-based test, the coverage of testing cases is increased enormously. In the
following chapters, we will describe how to write property-based test, how to
implement QuickCheck generators and how to define QuickCheck state machine for
testing imperative functions.
1.5 Contribution
Several libraries provide alternative implementations of a very similar interface,
enabling the same property specification used by more than one library. Before giving
an example, we introduce a way to present a function 1 .It says which module the
function belongs to and how many arguments the function has.
When we look through Erlang/OTP, it is not hard to notice the functionality, size of a
data structure, is included both in array module and in queue module. One is
array:size/1 and another is queue:len/1 respectively. We will show how to test them
by the same property specification in chapter 3.5. Based on this goal, we can develop
an extensive property-based test suite for as many pure functions as practicable.
Furthermore, we also find some interesting bugs of imperative functions in module
digraph and dets by using QuickCheck state machine. The bugs will be presented and
explained in chapter 4.
1
ModuleName：FunctionName∕NumberOfFunctionArgument.
2
2. QuickCheck
QuickCheck is a testing tool produced by the company Quviq, which was recently
founded by John Hughes and Thomas Arts. It provides the ability to test software in
an entirely different way from previous techniques. Instead of writing test cases that
map to use cases in a one-to-one fashion, QuickCheck enables programmers to
specify the expected behavior of the tested system. It is realized by writing
specification of properties as well as generators that automatically generates a
multitude of behaviors that the system might have. The automation of the test case
generation and execution can then be effectively checked against the properties that
ought to hold. A many-to-one approach of testing is therefore accomplished by
applying QuickCheck. In addition, the benefit from using QuickCheck is not end here.
As long as an error is found, QuickCheck also returns a minimum counterexample of
the error. It is called shrinking strategy. Based on that, the source of error becomes
easily identified.
2.1 Generators
Generators generate random pieces of data. QuickCheck provides a set of generators
of basic data types, which are defined by Quviq AB [5]. For instance,

binary() generates random binary values.

bool() generates Boolean values.

char() generates random characters.

list(G::gen(A)) generates a list of defined-type elements.

oneof(Gs::list(gen(A))) generates an element from the list arguments.

function1(G::gen(A)) generates a one-argument function with result generated
by G.
Besides, programmers can also specify their own generators. By the definition of
string type, a list of characters, a string generator can be defined as the following.
string() → list(char()).
Some possible results are also presented.

“f*”

[87,213,104,194,145,76]

[]
3
2.2 Properties
A property is written in a logical way in order to describe what a function should
fulfill. Let‟s start with an example of testing lists:any/2, which is defined as folloing
[3].
lists:any(Pred, List) → bool()
Returns true if Pred(Elem) returns true for at least one element Elem in List.
Types: Pred=fun(Elem) → bool() ; Elem=term() ; List=[term()]
It is obviously complicated to test this function by manually designing different lists
of terms as well as propositional functions. However, what we do with QuickCheck is
just writing the specification of this function in Code 2-1 saying what ought to hold
and also the generators of test data.
prop_any ()→
?FORALL ({F, L}, {function1 (bool ()), list(term ())},
lists:any (F,L)==not totally_false ([F(E)||E←L])).
Code 2-1: Property specification of lists:any/2
totally_false ([])→
true;
totally_false ([H|T]) →
case H of
true→false;
false→totally_false (T)
end.
Code 2-2: Totally false
This property can be read as: For all lists of terms and for all one-argument functions
returning Boolean values, it should hold that the result of lists:any/2 is equivalent to
not all element E in list L make Pred(E) false. We start by checking this property with
QuickCheck, which returns true after passing 100 automatically generated test cases.
1> eqc:quickcheck(lists_test:prop_any()).
…………………………………………………………………………………….
OK, passed 100 tests
true
In Code 2-3, we will show what a failing test looks like. In case we misread the lists
documentation and come to believe that lists:keydelete/3 deletes all the tuples from a
list which have the same key as the given one, then it should hold that the given key
cannot be found by applying lists:keymember/3 after deletion.
4
prop_deleteKey()->
?FORALL({Key, L} , {key(), list({key(),int()})},
lists:keymember(Key,1,lists:keydelete(Key,1,L))==false).
Code 2-3: Failing property specification
However, the property specification failed.
1>eqc:quickcheck(guess:prop_deleteKey()).
……………………………………………Failed! After 36 tests.
{e,[{a,2},{e,2},{f,9},{e,-7}]}
Shrinking…. (4 times)
{e,[{e,0},{e,0}]}
false
This shows that prop_deleteKey() does not behave as expected, i.e. the given key still
exists in the list after executing lists:keydelete/3, and the test failed after 36 test cases.
The counterexample is {e,[{a,2},{e,2},{f,9},{e,-7}]}, and the minimal
counterexample
after
shrinking,
is
{e,[{e,0},{e,0}]},
i.e.
lists:keymember(e,1,lists:keydelete(e,1,[{e,0},{e,0}])) does not return false. Further
investigation reveals that lists:keydelete/3 only deletes the first occurrence of a tuple
whose Nth element, which is the first one in this case, compares equal to the given
key, namely lists:keydelete(e,1,[{e,0},{e,0}])==[{e,0}].
2.3 Symbolic Representation
While function calls are the element of generators, symbolic representation of the
functions is used here for the purpose of debugging. An example will be shown in
chapter 3.2. It is basically Erlang tuple in the form {call, module, function,
[arguments]}, which clearly defines the module of function as well as the function
arguments. This symbolic representation can be evaluated later by QuickCheck eval
function, which replaces symbolic representation with the result of corresponding
function call.
5
3. Testing Pure Functions
This chapter will introduce the reader how to test pure functions of the data structure
in Erlang/OTP as well as explain the results after testing. A pure function always
evaluates the same result by the same argument value(s). The function result do not
depend on the state that may change as program execution proceeds, nor depend on
any external input from I/O devices.
3.1 Data Generators
QuickCheck provides a group of basic generators for testing data. Based on that,
programmers can design their own generators by need. An Erlang term is defined by
Joe Armstrong as a general Erlang data structure in Erlang/OTP. It can be number,
atom, reference, fun, port, pid, tuple, list, and binary [1]. We use it as an example to
show how to generate data generators. However, only number, atom, tuple, list, and
binary are chosen for the return types of the term generator in this thesis.
term()→
?SIZED(Size, term(Size)).
term(Size) →
oneof( [int()] ++
[real()] ++
[char()] ++
[bool()] ++
[list(term(Size-1)) || Size>0] ++
[?LET(List, list(term(Size-1)), list_to_binary(List)) || Size>0] ++
[?LET(List, list(term(Size-1)), list_to_tuple(List)) || Size>0]).
Code 3-1: Term generator
Here ?SIZED and ?LET are defined in QuickCheck 1.162 [5] as Erlang
macros. ?SIZED(Size, Generator) control the size of generation from 1 to about 40 so
that the longest list or tuple can up to length 40. Generator list(term(Size-1)) cannot
be directly used by function list_to_binary/1 or list_to_tuple/1 without binding it to an
actual value with ?LET(Pat,G1,G2), which generates a value from G1, binds it to Pat,
then generates a value from G2, which may refer to the variables bound in Pat.
Generator term() can generate one single element of number, character or Boolean
value; as well as lists, binaries and tuples by using recursive calls. The elements of the
lists, which binaries and tuples are transformed from, can be any types of term.



64
{14, true}
[[-55.686, 208], 3, 5.4, [<<192,102>>, false]]
6
3.2 Data Structure Generators
In order to test the functionality of a data-structure called D, it is necessary to
implement a generator for all possible data-structure D. In an Erlang module, the
return value of functions can be variant types. However, only the functions whose
return type is the same as the tested module are chosen to compose the data structure
generator. We would like to show an example of a simple dict-generator, which
consists of three kinds of commands.

dict: new()→Dict
Creates a new dictionary [3].

dict: store(Key, Value, Dict1)→Dict2
Stores a Key-Value pair in a dictionary [3].

dict: from_list(List)→Dict
Converts the key/value list List to a dictionary [3].
A dict-generator is defined in Code 3-2.
dict1()->
?SIZED(Size,dict1(Size)).
dict1(0) ->
dict:new();
dict1(Size) ->
?LET(Dict_,dict1(Size-1),
begin
oneof( [dict:new()] ++
[?LET({Key,Value},{key(),term()},dict:store(Key,Value,Dict_)) || Size>0 ] ++
[?LET(L,list({key(),term()}),dict:from_list(L))]) end).
Code 3-2: Simple dict structure generator 1
However, a possible dict is generated in the following format, which is very hard to
be understood. Also, the information about the sequence of generated commands is
lost.
1>eqc_gen:sample(test:dict1()).
{dict,3,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],
[[a,1,[2],<<>>,{}]],
[],
[[c|-7250470656]],
[[d|3]],
[],[],[],[],[],[],[],[],[],[],[]}}}
7
Instead, we rewrite this dict-generator with symbolic representation of commands.
dict2()->
?SIZED(Size,dict2(Size)).
dict2(0) ->
[{call,dict,new,[]}];
dict2(Size) ->
?LET(Dict_,dict2(Size-1),
begin
?LAZY(
oneof( [{call,dict,new,[]}] ++
[{call,dict,store,[key(),term(Size),Dict_]} || Size>0 ] ++
[{call,dict,from_list,[list({key(),term()})]}])) end).
Code 3-3: Simple dict structure generator 2
Macro ?LAZY is used for lazy evaluation so that symbolic function call
{call,dict,new,[]} will not be evaluated every time dict2/1 function is called.
2>eqc_gen:sample(test:dict2()).
{call,dict,store,
[e,{2.3},{call,dict,from_list,[[{f,<<34,249,182,220,20>>}]]}]}
This time, the generated result shows a dict generated recursively first by calling
dict:from_list/1 with a list, [{f,<<34,249,182,220,20>>}], as the argument, then by
calling dict:store/3 with a new key “e”, and a new value “{2.3}”, as the other two
arguments.
To evaluate this symbolic representation, we simply use the function, eqc_gen:eval/1,
in QuickCheck library.
3>eqc_gen:eval({call,dict,store,[e,{2.3},{call,dict,from_list,[[{f,<<34,249,182,220,20>>}]]}]}).
{dict,2,16,16,8,80,48,
{[],[],[],[],[],[],[],[],[],[],[],[],[],[],[],[]},
{{[],[],[],[],[],
[[e|{2.3}]],
[[f|<<32,249,182,220,20>>]],
[],[],[],[],[],[],[],[],[]}}}
Now, the result is clearly to be seen. According to this method, a complete dictgenerator can be defined by applying all the libraries whose return type is dict. All
possible dictionaries can be generated afterwards.
8
3.3 Model Based Testing
To make the testing trusty and easier, we use model based testing. Says we have a self
defined module called decimal which consists of mathematic functions. Also, there is
a way to transform a decimal number into Erlang number. Here is the test:
D1, D2  decimal 2
[sum(D1,D2)] ≡ [D1] + [D2],
[sub(D1,D2)] ≡ [D1] - [D2],
[mult(D1,D2)] ≡ [D1] * [D2],
[div(D1,D2)] ≡ [D1] / [D2],
[lt(D1,D2)] ≡ [D1] < [D2].
Figure 3-1: Transformation from decimal to Erlang number
We use Erlang numbers to model the decimal numbers. For example the function sum
has the same effect as plus(+). By this way, we can check if the function works
correctly by comparing the result with the based testing model. The methodology is
based on [2].
In this thesis, we choose module lists in application stdlib and module erlang in
application erts as the based testing model. A model-based transforming function of
queue module can be defined as the following, which transforms a queue into a list.
model(Queue) ->
queue:to_list(Queue).
Code 3-4: Model-based transforming function of queue module
As testing any function of queue module, the corresponding lists function is called.
Code3-5 shows how to test queue:in/2.
queue:in(Item, Q1) -> Q2
Inserts Item at the rear of queue Q1. Returns the resulting queue Q2 [3].
2 [D1]
means the transformation from a decimal number to an Erlang number.
9
prop_queue_in() ->
?FORALL({Item,Q} , {term(),queue()},
model(queue:in(Item,Q)) == model(Q)++[Item]).
Code 3-5: Property specification of queue:in/2
Figure 3-2: Transformation from queue to list
In this case, [Q] means the transformation from a queue to a list.
3.4 Information Hiding
Several functions in different modules provide alternative implementations of a very
similar interface and have the same property. Therefore, it is efficient to reuse the
same property specification for as many functions as possible. According to this goal,
saving the main differences between various modules in Erlang records is the strategy
we use. So that it is really easy to change from testing one function to another by
passing different Erlang records. Read examples in the following Case Study.
3.5 Case Study
A more complex model-based transforming function is defined in Code 3-6, in which
function Fun transforms either key-value structure into a list of keys or set structure
into a list of all set elements. It depends on the purpose.
model(Module, Fun, Original)->
Module:Fun(Original).
Code 3-6: Model-based transforming function of variant modules
Case 1:
The functionality of getting the size of a data structure is included in eight different
modules of Erlang/OTP, which are dict, orddict, gb_trees, sets, ordsets, gb_sets,
queue and array. In the following, it shows how to test these libraries with the same
property and use the concepts which have been presented, i.e. model based testing and
information hiding.
10
Table 3-1: Size functions
Module
Function
Definitions of Function
1) dict
size
Returns the number of elements in a Dict.
2) orddict
size
Returns the number of elements in an Orddict.
3) gb_trees
size
Returns the number of nodes in Tree.
4) sets
size
Returns the number of elements in Set.
5) ordsets
size
Returns the number of elements in Ordset.
6) gb_sets
size
Returns the number of elements in Set.
7) queue
len
Calculates and returns the length of queue Q.
8) array
size
Get the number of entries in the array Array.
We save the differences between teated mofules in an Erlang record, module. Record
field, name, holds the name of modules; record field, gen, holds the data structure
generators; and record field, length, holds the length of the data structure. This Enlang
record can be defined like this:
-record(module,{name, gen, length}).
This Erlang record is named module. Name, gen and length are three fields of the
record. An instance of module record for dict structure can be defined by assigning
dict to name, dict() to gen, and size to length.

dictr()->
#module{name=dict, gen=dict(), length=size}.
So does other seven structures.

orddictr()->
#module{name=orddict, gen=orddict(), length=size}.

gb_treesr()->
#module{name=gb_trees, gen=gb_trees(), length=size}.

setsr()->
#module{name=sets, gen=sets(), length=size}.

ordsetsr()->
#module{name=ordsets, gen=ordsets(), length=size}.

gb_setsr()->
#module{name=gb_sets, gen=gb_sets(), length=size}.
11

queuer()->
#module{name=queue, gen=queue(), length=len}.

arrayr()->
#module{name=array, gen=array(), length=size}.
Code 3-7: Variation in modules 1
The property test is written in Code 3-8, where erlang:length/1 of based testing model
is used to get the length of list.
prop_size(R)->
#module{name=Module, gen=Gen, length=Length}=R,
?FORALL(D_,Gen,
begin
D=eval(D_),
Module:Length(D) == erlang:length( model(Module,to_list,D) )
end).
Code 3-8: Property specification of the size of tested data structure
Now, we can test these eight functions by changing different Erlang records. 100
automatically generated test cases for each are passed.
1> eqc:quickcheck(combine:prop_size(combine:dictr())).
…………………………………………………………………………………….
OK, passed 100 tests
true
2> eqc:quickcheck(combine:prop_size(combine:orddictr())).
…………………………………………………………………………………….
OK, passed 100 tests
true
Case 2:
Though there are more differences between tested modules than one can imagine. We
only save the information, which can be reused also by other property specifications,
in Erlang records, and leave the rest differences assigned through the arguments of
property.
In this case, the Erlang record is extended for more information about the modules.
The field, mapping, stores a function name. Which the function transforms the data
structure into a type of list. For example, dict:fetch_keys/1 returns a list of all keys in
the dictionary but sets:to_list/1 returns a list of all set elements. The function stored in
the field, compare, is used to find out the existence of the given target in a list. Reader
can ignore the meaning of it at this moment.
12
-record(module,{name, gen, length, mapping, compare}).

dictr()->
#module{name=dict, gen=dict(), length=size, mapping=fetch_keys, compare=fun
match/2}.

orddictr()->
#module{name=orddict,gen=orddict(), length=size, mapping=fetch_keys, compare=fun
equal/2}.

gb_treesr()->
#module{name=gb_trees, gen=gb_trees(), length=size, mapping=keys, compare=fun
equal/2}.

setsr()->
#module{name=sets, gen=sets(), length=size, mapping=to_list, compare=fun match/2}.

ordsetsr()->
#module{name=ordsets, gen=ordsets(), length=size, mapping=to_list, compare=fun
equal/2}.

gb_setsr()->
#module{name=gb_sets, gen=gb_sets(), length=size, mapping=to_list, compare=fun
equal/2}.
Code 3-9: Variation in modules 2
match(_,[])->
false;
match(K,[H|T])->
case K=:=H of
true-> true;
false-> match(K,T)
end.
Code 3-10: Matching equal function
equal(_,[])->
false;
equal(K,[H|T])->
case K==H of
true-> true;
false-> equal(K,T)
end.
Code 3-11: Comparing equal function
Now we want to test the following seven functions.
13
Table 3-2: Containment functions
Module
Function
Definitions of Function
1) dict
is_key
This function tests if Key is contained in the dictionary Dict.
2) orddict
is_key
This function tests if Key is contained in the dictionary Orddict.
3) gb_trees is_defined
Returns true if Key is present in Tree, otherwise false.
4) sets
is_element
Returns true if Element is an element of Set, otherwise false.
5) ordsets
is_element
Returns true if Element is an element of Ordset, otherwise false.
6) gb_sets
is_element
Returns true if Element is an element of Set, otherwise false.
7) gb_sets
is_member
Returns true if Element is an element of Set, otherwise false.
The functionalities of these seven functions are quite similar. Therefore, we will try to
test them by the same property specification. However, those different function names
are used nowhere else than this property specification, so we just defined them
through the arguments of property.
prop_contain(R,Fun)->
#module{name=Module, gen=Gen, mapping=Mapping, compare=Compare}=R,
?FORALL({K,D_},{term(),Gen},
begin
D=eval(D_),
Module:Fun(K,D) == Compare(K,model(Module,Mapping,D))
end).
Code 3-12: Property specification of containment functionality
These seven functions were tested by changing R as the related Erlang records and
Fun as the name of tested functions. 100 randomly generated test cases for each are
passed.
1> eqc:quickcheck(combine:prop_contain(combine:dictr(),is_key)).
…………………………………………………………………………………….
OK, passed 100 tests
true
2> eqc:quickcheck(combine:prop_contain(combine:gb_treesr(),is_defined)).
…………………………………………………………………………………….
OK, passed 100 tests
true
3> eqc:quickcheck(combine:prop_contain(combine:setsr(),is_element)).
…………………………………………………………………………………….
OK, passed 100 tests
14
true
4> eqc:quickcheck(combine:prop_contain(combine:gb_setsr(),is_member)).
…………………………………………………………………………………….
OK, passed 100 tests
true
3.6 Findings
In order to make a test case used by as many libraries as possible, the characteristic of
each library should be analyzed first. Afterwards, the libraries are grouped into
several types according to testing strategy. In this process, it shows how libraries were
designed and what the Erlang/OTP users should be careful about.
Testing the functionality of deletion for module dict, orddict, gb_trees, sets, ordsets,
and gb_sets shows an example of this idea. In these cases, lists:delete/2,
lists:keymember/3 and lists:keydelete/3 are the functions used in model-based testing.
lists: delete(Elem, List1) -> List2
Returns a copy of List1 where the first element matching (=:=) Elem is deleted, if
there is such an element [3].
lists: keymember(Key, N, TupleList) -> bool()
Returns true if there is a tuple in TupleList whose Nth element compares equal (==) to
Key, otherwise false [3].
lists: keydelete(Key, N, TupleList1) -> TupleList2
Returns a copy of TupleList1 where the first occurrence of a tuple whose Nth element
compares equal (==) to Key is deleted, if there is such a tuple [3].
In Code 3-13, we write a property specification for both orddict:erase/2 and
gb_trees:delete_any/2 by their specification.
orddict:erase(Key, Orddict1) -> Orddict2
This function erases all items with a given key from a dictionary if the key is present
in the orddict, otherwise does nothing; returns new orddict [3].
gb_trees:delete_any(Key, Tree1) -> Tree2
Removes the node with key Key from Tree1 if the key is present in the tree, otherwise
does nothing; returns new tree [3].
15
prop_delete1(R, Fun) ->
#module{name=Module, gen=Gen}=R,
?FORALL({K,S1_},{term(),Gen},
begin
S1=eval(S1_),
S2=Module:Fun(K,S1),
L=model(Module,to_list,S1),
case lists:keymember(K,1,L) of
true ->model(Module,to_list,S2)==lists:keydelete(K,1,L);
false->model(Module,to_list,S2)==L
end
end).
Code 3-13: Property specification of Deletion 1
First we transform the orddict or gb_trees to a key-value list, and then check if the key
presents in the list. If it returns true, all the items with the same key are deleted from
the list. Otherwise, an unchanged list is returned. Meanwhile, we transform the
orddict or gb_trees, which is the result after deletion, to a key-value list. Finally, we
compare these two lists. If the comparison is always true, orddict:erase/2 and
gb_trees:delete_any/2 were implemented correctly.
1>eqc:quickcheck(combine:prop_delete1(combine:orddictr(),erase)).
…………………………………………………………………………………….
OK, passed 100 tests
true
2>eqc:quickcheck(combine:prop_delete1(combine:gb_treesr(),delete_any)).
…………………………………………………………………………………….
OK, passed 100 tests
true
Next, let‟s look at the definition of dict: erase/2.
dict: erase(Key, Dict1) -> Dict2
This function erases all items with a given key from a dictionary if the key is present
in the dict, otherwise does nothing; returns new tree [3].
It seems like dict:erase/2 has the same property with orddict:erase/2 and
gb_trees:delete_any/2. However, as we test it with the same test case, error message
arises immediately.
eqc:quickcheck(combine:prop_delete1(combine:dictr(),erase)).
……………………………..…Failed! After 17 tests
{3.0, {call,dict,store,[3,[-2524017590],{call,dict,new,[]}]}}
Shrinking. (1 times)
{3.0, {call,dict,store,[3,[],{call,dict,new,[]}]}}
false
16
What happened? The reason is that there are two kinds of comparison used in the
Erlang/OTP. One is comparing equal (==) and another is matching equal (=:=).
Number 3 is the same as number 3.0 with comparing equal (==) but they are different
with matching equal(=:=). In orddict module and gb_trees module, comparing equal is
used for comparing keys. However, dict module uses matching equal. From this
minimal counterexample, dict:erase/2 tries to delete key 3.0 from the dict,
{call,dict,store,[3,[],{call,dict,new,[]}]}, where the only key is 3. The result should be an
unchanged dict. However, by applying Code 3-13, lists:keymember/3 checks if key
3.0 is presented in the dict with comparing equal. True is returned. So that it continues
to use lists:keydelete/3, which also uses comparing equal, in order to delete the item.
Finally, the result is an empty dict and it is different from the expected unchanged dict.
To fix the problem, we implement a helper function, deletekeyvaluefromlist/2, in
which lists:member/2 is called to find the given key in this dict by using matching
equal. So does other parts of deletekeyvalueformlist/2, where matching equal is the
only use for key-comparison.
deletekeyvaluefromlist(K,L)->
case lists:member(K, keys_from_list(L)) of
true ->deletekeyvaluefromlist(K,L,[]);
false->L
end.
deletekeyvaluefromlist(_,[],Acc)->
Acc;
deletekeyvaluefromlist(Key,[{K,V}|T],Acc)->
case Key=:=K of
true ->Acc++T;
false->deletekeyvaluefromlist(Key,T,Acc++[{K,V}])
end.
Code 3-14: Delete Key-Value by Matching Equal
Accordingly, we modify the property specification into a new version.
prop_delete2(R)->
#module{name=Module, gen=Gen}=R,
?FORALL({E,S1_},{term(),Gen},
begin
S1=eval(S1_),
S2=Module:erase(E,S1),
model(Module,to_list,S2)==deletekeyvaluefromlist(E,model(Module,to_list,S1))
end).
Code 3-15: Property specification of Deletion 2
17
1>eqc:quickcheck(combine:prop_delete2(combine:dictr())).
…………………………………………………………………………………….
OK, passed 100 tests
true
Finally, 100 test cases are passed against this new property. Next, let‟s look at the
definition of gb_trees:delete/2
gb_trees:delete(Key, Tree1) -> Tree2
Removes the node with key Key from Tree1; returns new tree. Assume that the Key is
present in the tree, crashes otherwise [3].
The only different functionality from the former ones is that an error message will
arise as the given key is not present in the gb_trees. However, this reason is sufficient
for writing another property specification, which is presented in Code 3-16.
prop_delete3(R) ->
#module{name=Module,gen=Gen,compare=Compare}=R,
?FORALL({K,D1_},{term(),Gen},
begin
D1=eval(D1_),
case catch(Module:delete(K,D1)) of
{'EXIT',_}->
not equal(K,model(Module,keys,D1));
D2->
model(Module,to_list,D2)==lists:keydelete(K,1,model(Module,to_list,D1))
end
end).
Code 3-16: Property specification of Deletion 3
1>eqc:quickcheck(combine:prop_delete3(combine:gb_treesr())).
…………………………………………………………………………………….
OK, passed 100 tests
true
Up to now, we have tested all the delete functions of key-value structures. In the
following, we will start testing delete functions of sets structures. In Code3-17, we
write a property for sets:del_element/2.
sets:del_element(Element, Set1) -> Set2
Returns a new set formed from Set1 with Element removed. If Element is not an
element in Set1, nothing is changed [3].
18
prop_delete4(R, Fun) ->
#module{name=Module, gen=Gen}=R,
?FORALL({E,S1_},{term(),Gen},
begin
S1=eval(S1_),
S2=Module:Fun(E,S1),
L=model(Module,to_list,S1),
case lists:member(E,L) of
true ->model(Module,to_list,S2)==lists:delete(E,L);
false->model(Module,to_list,S2)==L
end
end).
Code 3-17: Property specification of Deletion 4
First, we transform the set to a list called L and remove the element from it if the
element is present in L. Otherwise, keep the list unchanged. Meanwhile, we transform
the set, which is the result after deletion, to a list. Finally, we compare these two lists.
If the comparison is always true, sets:del_element/2 is designed correctly.
1>eqc:quickcheck(combine:prop_delete4(combine:setsr(),del_element)).
…………………………………………………………………………………….
OK, passed 100 tests
true
100 test cases are passed against this property. Next, let‟s test ordsets:del_element/2,
gb_sets:del_element/2 and gb_sets:delete_any/2.
ordsets:del_element(Element, Ordset1) -> Ordset2
Returns a new ordset formed from Ordset1 with Element removed. If Element is not
an element in Ordset1, nothing is changed [3].
gb_sets:del_element(Element, Set1) -> Set2
gb_sets:delete_any(Element, Set1) -> Set2
Returns a new gb_set formed from Set1 with Element removed. If Element is not an
element in Set1, nothing is changed [3].
It seems like ordsets:del_element/2, gb_sets:del_element/2 and gb_sets:delete_any/2
have the same functionality with sets:del_element/2. However, as we test it with the
same test case, error messages arise immediately.
19
1>eqc:quickcheck(combine:prop_delete4(combine:ordsetsr(),del_element)).
……………………………………Failed! After 80 tests.
{3, {call, ordsets, from_list, [[3.0,24,3]]}}
Shrinking. (1 times)
{3, {call, ordsets, from_list, [[3.0]]}}
false
2>eqc:quickcheck(combine:prop_delete4(combine:gb_setsr(),del_element)).
……………………………………Failed! After 55 tests.
{3, {call, gb_sets, insert, [3.0, {call, gb_sets, from_list, [[72,77,<<>>]]}]}}
Shrinking. (1 times)
{3, {call, gb_sets, insert, [3.0, {call, gb_sets, from_list, [[]]}]}}
false
3>eqc:quickcheck(combine:prop_delete4(combine:gb_setsr(),delete_any)).
……………………………………Failed! After 23 tests.
{3, {call, gb_sets, balance, [{1, {3.0, nil, nil}}]}}
false
The reason is similar to the former key-value structure cases. In module ordsets,
comparing equal is used for comparing elements. Therefore, to delete 3 from ordsets,
where 3.0 is the only one element, the result should be an empty ordsets. However, in
Code 3-17, it checks if key 3 is presented in the ordsets by using lists:member/2,
which uses matching equal. Since number 3 is different from number 3.0, an
unchanged ordset is returned. Therefore, the result is different from the expected
empty ordsets. This problem is the same in module gb_sets.
To fix the problem, we implement a helper function deletefromlist/2, in which
comparing equal is the only use for element-comparison.
deletefromlist(E,L)->
case equal(E,L) of
true ->deletefromlist(E,L,[]);
false->L
end.
deletefromlist(_,[],Acc)->
Acc;
deletefromlist(E,[H|T],Acc)->
case E==H of
true ->Acc++T;
false->deletefromlist(E,T,Acc++[H])
end.
Code 3-18: Delete Element by Comparing Equal
20
Afterwards, we modify prop_delete4() into a new version in Code 3-19, which can
even test the function sets:del_element/2. Fun1 is the name of tested functions and
Fun2 is the functions used for deletion.
prop_delete5(R,Fun1,Fun2) ->
#module{name=Module, gen=Gen, compare=Compare}=R,
?FORALL({E,S1_},{term(),Gen},
begin
S1=eval(S1_),
S2=Module:Fun1(E,S1),
L=model(Module,to_list,S1),
case Compare(E,L) of
true ->model(Module,to_list,S2)==Fun2(E,L);
false->model(Module,to_list,S2)==L
end
end).
Code 3-19: Property specification of Deletion 5
1>eqc:quickcheck(combine:prop_delete5(combine:setsr(),del_element,
fun lists:delete/2)).
…………………………………………………………………………………….
OK, passed 100 tests
true
2>eqc:quickcheck(combine:prop_delete5(combine:ordsetsr(),del_element,
fun combine:deletefromlist/2)).
…………………………………………………………………………………….
OK, passed 100 tests
true
3>eqc:quickcheck(combine:prop_delete5(combine:gb_setsr(),del_element,
fun combine:deletefromlist/2)).
…………………………………………………………………………………….
OK, passed 100 tests
true
4>eqc:quickcheck(combine:prop_delete5(combine:gb_setsr(),delete_any,
fun combine:deletefromlist/2)).
…………………………………………………………………………………….
OK, passed 100 tests
true
21
Finally, 100 test cases for each are passed against this new property.
The last delete function in gb_sets module will be tested as the following.
gb_sets: delete(Element, Set1) -> Set2
Removes the element from Set1; returns new set. Assume that the Element is present
in the set, crashes otherwise [3].
The only different strategy from Code 3-19 is that an error message will arise as the
element is not present in the set. However, this reason is sufficient for writing another
property specification, which is presented in Code 3-20.
prop_delete6(R) ->
#module{name=Module,gen=Gen,compare=Compare}=R,
?FORALL({E,D1_},{term(),gb_sets()},
begin
D1=eval(D1_),
case catch(Module:delete(E,D1)) of
{'EXIT',_}->
not equal(E,model(Module,to_list,D1));
D2->
model(Module,to_list,D2)==deletefromlist(E,model(Module,to_list,D1))
end
end).
Code 3-20: Property specification of Deletion 6
100 test cases are passed against this property.
1>eqc:quickcheck(combine:prop_delete6(combine:gb_setsr())).
…………………………………………………………………………………….
OK, passed 100 tests
true
22
Table 3-3: Deletion Functions
Module
Function
1) orddict
erase
2) gb_trees
delete_any
3) dict
erase
4) gb_trees
delete
5) sets
del_element
6) ordsets
del_element
7) gb_sets
del_element
8) gb_sets
delete_any
9) gb_sets
delete
Table 3-3 presents these nine tested functions. There are some suggestions for
Erlang/OTP users.
Comparing equal is used in the modules whose data is saved in order, i.e. orddcit,
gb_trees, ordsets and gb_sets. However, the other two modules, dict and sets, do not
have this need. So that matching equal is applied to compare keys and elements
respectively. Nevertheless, it seems hard to explain the reason of using matching
equal for lists:member/2 and lists:delete/2 but comparing equal for lists:keymember/3
and lists:keydelete/3. Consequently, Erlang/OTP users should apply those functions
carefully according to this detail.
Table 3-4: Comparison of (==) and (=:=)
Comparing numbers
Matching
comparison
Equal
comparison
Module
3 is different from dict,
3.0
sets
3 is the same as 3.0
orddict,
gb_trees,
ordsets,
gb_sets
23
Model based testing
functions
lists:member/2,
lists:delete/2
lists:keymember/3,
lists:keydelete/3
4. QuickCheck State Machine
QuickCheck state machine generates imperative command sequence and executes
them by applying five main functions, which will be introduced in the following.
Figure 4-1: A complete cycle of a QuickCheck state machine test case [7]
4.1 Initial state
Just like the conventional testing where a function is always called to initialize test
cases, QuickCheck state machine provides several techniques to initialize the state
prior to both generation and execution of a command sequence. However, we only
introduce the one used in this thesis.
What is state? Vertices and edges are two main components of a directed graph which
can be either cyclic or acyclic. Except the cyclicity is defined for a digraph in the
beginning, vertices and edges can be added, deleted or modified by each invocated
operation, which can change the state of digraph. We define an Erlang record to trace
states. This record in the following is named state and contains three fields: vertex for
a list of vertices, edge for a list of edges, and cyclicity for either atom cyclic or acyclic.
-record(state, {vertex, edge, cyclicity}).
Says the tested digraph is cyclic. In the beginning, there should be no vertex and edge.
Hence, we can initialize the state as #state{vertex=[],edge=[],cyclicity=cyclic}. The
first two lines of Code 4-1 shows how to generate a command sequence based on this
initial state.
24
prop_statem_correct (Cyclicity)→
?FORALL (Cmds, commands (?MODULE, #state{vertex=[], edge=[], cyclicity=Cyclicity}),
begin
G=digraph:new ([Cyclicity]),
{H,S,Res} = run_commands(?MODULE,Cmds,[{digraph,G}]),
?WHENFAIL (io:format("~p\n~p\n~p\n", [H, S, Res]), Res==ok)
end).
Code 4-1: property of digraph module
A cyclic digraph can then be tested from the console by typing:
eqc:quickcheck(digraph_test:prop_statem_correct(cyclic)).
So does acyclic digraph.
eqc:quickcheck(digraph_test:prop_statem_correct(acyclic)).
4.2 Function “command”
The function command is called after initializing states during generation-time. It
generates command sequence in symbolic representation. New commands can be
recursively added into the sequence by the dependency of previous state and the
requirements of commands in the sequence. More examples will be shown in chapter
5.
4.3 Function “precondition”
After executing command generator during generation time, function precondition is
used to filter out commands by returning Boolean value. If the precondition for the
specific command is true, this command will be kept in the test case sequence.
Otherwise, it will be excluded.
4.4 Function “postcondition”
The function postcondition is called during execution-time after executing the
commands created during generation-time. It is used to determine whether the return
value of each executed commands in the test case sequence corresponding to the
specification of the tested imperative functions.
4.5 Function “next_state”
The state is changed by executing from one command to another. The next_state
25
function is called twice. If the precondition returns true, the state is updated with new
symbolic values in the generation-time; if the postcondition returns true, the state is
modified with actual values in the execution-time. Figure4-2 and figure4-3 are
designed according to [6]. The dialogue box stands for the states before and after each
tested command presented in square boxes.
Figure 4-2: Process during generation time
Figure 4-3: Process during run time
During generation time, QuickCheck only generate symbolic command. However,
during execution time each command is invocated.
26
5. Testing Imperative Functions
Different from pure functions, program state can be changed by a sequence of
imperative function commands. In this chapter, the bugs found by using QuickCheck
state machine in Erlang/OTP will be presented and explained to the reader.
5.1 Module digraph
The digraph module in Erlang/OTP implements imperative labeled directed graph.
The digraph module is defined by Ericsson AB [3] and presented as following.
Every vertex and edge can be annotated with additional information which is called
label. All the edges are directed. An edge e = (v1, v2) is said to emanate from vertex
v1 and to be incident on vertex v2. Therefore, edge e1=(v1, v2) is opposite to edge
e2=(v2, v1). Multiple edges between vertices are allowed. If the new created vertex or
edge already exists, the old one will be modified so that all the vertices and edges are
unique [3].
If there is an edge e emanating from vertex v1 to incident on vertex v2, v1 is an inneighbor of v2 and v2 is an out-neighbor of v1. The out-degree of a vertex is a
number of out-neighbors of that vertex. The in-degree of a vertex is a number of inneighbors of that vertex [3].
A path P from v[1] to v[k] in digraph (E, V) is a non-empty sequence
v[1], v[2], ..., v[k] of vertices in V such that there is an edge (v[i], v[i+1]) in E for
1 <= i < k. The length of the path P is k-1. P is a cycle if the length of P is not zero
and v[1] = v[k]. An acyclic digraph is a digraph that has no cycles [3].
Bug1
Let‟s test acyclic digraph by running this command.
1> eqc:quickcheck(digraph_test:prop_statem_correct(acyclic)).
………………………………………………………………………..
Shrinking------------(16 times)
[{init, {state, [], [], acyclic}},
{set, {var, 5}, {call, digraph, add_vertex, [{var, digraph}]}},
{set, {var, 7}, {call, digraph, add_vertex, [{var, digraph}]}},
{set, {var, 8}, {call, digraph, add_vertex, [{var, digraph}]}},
{set, {var, 19}, {call, digraph, add_edge, [{var, digraph}, {var, 5}, {var, 8}, 0]}},
{set, {var, 30}, {call, digraph, add_edge, [{var, digraph}, {var, 19}, {var, 7}, {var, 5}, []]}},
{set, {var, 32}, {call, digraph, edges, [{var, digraph}, {var, 5}]}}]
27
[{{state, [], [], acyclic}, [‘$v’|0]},
{{state, [{[‘$v’|0], []}], [], acyclic}, [‘$v’|1]},
{{state, [{[‘$v’|0], []}, {[‘$v’|1]}, []], [], acyclic}, [‘$v’|2]},
{{state, [{[‘$v’|0], []}, {[‘$v’|1], []}, {[‘$v’|2], []}], [], acyclic}, [‘$e’|0]},
{{state, [{[‘$v’|0], []}, {[‘$v’|1], []}, {[‘$v’|2], []}], [{[‘$e’|0], [‘$v’|0], [‘$v’|2], 0}], acyclic},
[‘$e’|0]},
{{state, [{[‘$v’|0], []}, {[‘$v’|1], []}, {[‘$v’|2], []}], [{[‘$e’|0], [‘$v’|1], [‘$v’|0], 0}], acyclic},
[[‘$e’|0], [‘$e’|0]]}]
{postcondition, false}
false
Before explaining this failing test case, let‟s see the definition of function
digraph:add_vertex/1, digraph:add_edge/4, digraph:add_edge/5 and digraph:edges/2.
digraph:add_vertex(G) -> vertex()
Creates a vertex using the empty list as label, and returns the created vertex. The
created vertex is represented by the term ['$v' | N], where N is an integer >= 0 [3].
digraph:add_edge(G, E, V1, V2, Label) -> edge() | {error, Reason}
Creates (or modifies) the edge E of the digraph G, using Label as the (new) label of
the edge. The edge is emanating from V1 and incident on V2. Returns E [3].
digraph:add_edge(G, V1, V2, Label) -> edge() | {error, Reason}
Creates an edge which is represented by the term ['$e' | N], where N is an integer >= 0
[3].
digraph:edges(G, V) -> Edges
Returns a list of all edges emanating from or incident on V of the digraph G, in some
unspecified order [3].
In Table 5-1, it goes through this failing test case by seeing the entire process in
QuickCheck state machine.
Table 5-1: Entire process of QuickCheck state machine (digraph)
Command generation time:
Cycle 1
Cycle 2
Cycle 3
Initial state
Command
Precondition
Next state
Command
Precondition
Next state
Command
Precondition
Next state
{state, [], [], acyclic}
{call, digraph, add_vertex, [{var, digraph}]}
true
{state, [{{var,5},[]}], [], acyclic}
{call, digraph, add_vertex, [{var, digraph}]}
true
{state, [{{var,5},[]},{{var,7},[]}], [], acyclic}
{call, digraph, add_vertex, [{var, digraph}]}
true
{state, [{{var,5},[]},{{var,7},[]},{{var,8},[]}], [], acyclic}
28
Cycle 4
{call, digraph, add_edge, [{var, digraph}, {var, 5}, {var, 8},
0]}
true; (the edge would not create a cycle in this acyclic
digraph and both {var,5} and {var,8} are vertices of the
digraph {var,digraph})
{state, [{{var,5},[]},{{var,7},[]},{{var,8},[]}], [{{var,19},
{var,5}, {var,8}, 0}], acyclic}
{call, digraph, add_edge, [{var, digraph}, [„$e‟|0], {var, 7},
{var, 5}, []]}
true; (the edge would not create a cycle in this acyclic
digraph and both {var,7} and {var,5} are vertices of the
digraph {var,digraph})
{state, [{{var,5},[]},{{var,7},[]},{{var,8},[]}], [{{var,19},
{var,7}, {var,5}, []}], acyclic}
{call, digraph, edges, [{var, digraph}, {var, 5}]}
true
unchanged
Command
Precondition
Next state
Cycle 5
Command
Precondition
Next state
Cycle 6
Command
Precondition
Next state
Command execution time:
Cycle 1
Initial state
Invocation
Postcondition
Cycle 2
Next state
Invocation
Postcondition
Cycle 3
Next state
Invocation
Postcondition
Cycle 4
Next state
Invocation
Postcondition
Next state
Cycle 5
Invocation
Postcondition
Next state
Cycle 6
Invocation
Postcondition
{state, [], [], acyclic}
digraph:add_vertex(G)
true
{state, [{[„$v‟|0],[]}], [], acyclic}
digraph:add_vertex(G)
true
{state, [{[„$v‟|0],[]},{[„$v‟|1],[]}], [], acyclic}
digraph:add_vertex(G)
true
{state, [{[„$v‟|0],[]},{[„$v‟|1],[]},{[„$v‟|2],[]}], [], acyclic}
digraph:add_edge(G,[„$v‟|0],[„$v‟|2],0)
true
{state, [{[„$v‟|0],[]},{[„$v‟|1],[]},{[„$v‟|2],[]}],
[{[„$e‟|0],[„$v‟|0],[„$v‟|2], 0}], acyclic}
digraph:add_edge(G,[„$e‟|0],[„$v‟|1],[„$v‟|0],[])
true
{state, [{[„$e‟|0],[]},{[„$v‟|1],[]},{[„$v‟|2],[]}], [{[„$e‟|0],
[„$v‟|1],[„$v‟|0], []}], acyclic}
digraph:edges(G,[„$v‟|0])
false
An Erlang record, -record(state, {vertex, edge, cyclicity}), is used to store the state of
this state machine with a list of vertices, a list of edges and the cyclicity of the digraph.
In the beginning, {init, {state, [], [], acyclic}} defines the initial state as an empty
acyclic digraph. At the end, {postcondition, false} shows that it fails during verifying
29
the specification for executed command, which is digraph:edges/2 in this case. The
middle part shows the detail of how the postcondition failed. It can be divided into
three parts. First is the history of the test command sequence. {set, {var, 5}, {call,
digraph, add_vertex, [{var, digraph}]}} means to set the symbolic representation,
{call, digraph, add_vertex, [{var, digraph}]}, to the variable {var, 5}. We can see the
failed sequence starts with function add_vertex/1 and then add_vertex/1, add_vertex/1,
add_edge/4, add_edge/5, finally edges/2. The second part shows the sequence of
changed states. The state is always modified as executing next command. For
example, the first half part of {{state, [], [], acyclic}, [„$v‟|0]} shows the pre state for
executing command {set, {var, 5}, {call, digraph, add_vertex, [{var, digraph}]}} and
the second half part shows the return value of this add_vertex function. However, we
can see the state is actually changed by adding a new element {[„$v‟|0], []} to the
vertex list from {{state, [{[„$v‟|0], []}], [], acyclic}, [„$v‟|1]} when executing next
command.
Because the result of digraph:edges/2 is [[„$e‟|0], [„$e‟|0]]. It reveals function
digraph:add_edge/5 did not correctly modify the same named edge and keep [„$e‟|0]
unique.
To clarify the actual function of digraph:add_edge/5, the following example is chosen
in this case.
First, we need an empty digraph G.
1>G=digraph:new().
{digraph,53261,57358,61455,true}
Four vertices are added into digraph.
2>digraph:add_vertex(G).
['$v'|0]
3>digraph:add_vertex(G).
['$v'|1]
4>digraph:add_vertex(G).
['$v'|2]
5>digraph:add_vertex(G).
['$v'|3]
Add edge ['$e'|0], which emanates from vertex [„$v‟|0] and incident on vertex [„$v‟|1]
without label, by using the function digraph:add_edge/5.
4>digraph:add_edge(G,['$e'|0],['$v'|0],['$v'|1],{}).
['$e'|0]
Up to now, the digraph G can be showed as Figure 5-1.
30
Figure 5-1: Edge added
Figure 5-2: Edge modified
Next, add a directed edge emanates from vertex [„$v‟|2] and incident on vertex [„$v‟|3]
without label. As all the vertices and edges in the digraph should be unique, edge
[„$e‟|0] is modified by new assignments.
5>digraph:add_edge(G,['$e'|0],['$v'|2],['$v'|3],{}).
['$e'|0]
Base on the principle of uniqueness, digraph G should be changed to Figure 5-2
afterwards.
Now, we can test the characteristics of digraph G.
The number of edges in total.
6>digraph:no_edges(G).
1
The two vertices which edge [„$e‟|0] is between.
7>digraph:edge(G,[‘$e’|0]).
[[‘$e’|0],[‘$v’|2],[‘$v’|3],{}]
The out-degree of vertex [„$v‟|2]
8>digraph:out_degree(G,['$v'|2]).
1
All the edges which emanate from vertex [„$v‟|2]
9>digraph:out_edges(G,['$v'|2]).
[[‘$e’|0]]
All the out-neighbours of vertex [„$v‟|2]
10>digraph:out_neighbours(G,[‘$v’|2]).
[[‘$v’|3]]
The in-degree of vertex [„$v‟|3]
11>digraph:in_degree(G,['$v'|3]).
1
31
All the edges which are incident on vertex [„$v‟|3]
12>digraph:in_edges(G,['$v'|3]).
[[‘$e’|0]]
All the in-neighbours of vertex [„$v‟|3]
13>digraph:in_neighbours(G,[‘$v’|3]).
[[‘$v’|2]]
Up to now, all the test cases return correct results. However, bugs are found in the
following cases.
The out-degree of vertex [„$v‟|0]
14>digraph:out_degree(G,['$v'|0]).
1
All the edges which emanate from vertex [„$v‟|0]
15>digraph:out_edges(G,['$v'|0]).
[[‘$e’|0]]
All the out-neighbours of vertex [„$v‟|0]
16>digraph:out_neighbours(G,[‘$v’|0]).
[[‘$v’|3]]
The in-degree of vertex [„$v‟|1]
17>digraph:in_degree(G,['$v'|1]).
1
All the edges which are incident on vertex [„$v‟|1]
18>digraph:in_edges(G,['$v'|1]).
[[‘$e’|0]]
All the in-neighbours of vertex [„$v‟|1]
19>digraph:in_neighbours(G,[‘$v’|1]).
[[‘$v’|2]]
Obviously, edge [„$e‟|0] has been completely modified. However, it is still recorded
as an out-edge of vertex [„$v‟|0] and an in-edge of vertex [„$v‟|1]. Therefore, [„$v‟|3]
is an out-neighbor of [„$v‟|0] and [„$v‟|2] is an in-neighbor of [„$v‟|1]. This bug also
exists in cyclic digraph.
The same bug makes another case crushed. Bug2 presents this.
Bug2
1>G=digraph:new().
{digraph,40976,45073,49170,true}
32
2>digraph:add_vertex(G).
[‘$v’|0]
3>digraph:add_vertex(G).
[‘$v’|1]
4>digraph:add_edge(G,[‘$e’|0],[‘$v’|0],[‘$v’|0],{}).
[‘$e’|0]
5>digraph:add_edge(G,[‘$e’|0],[‘$v’|1],[‘$v’|0],{}).
[‘$e’|0]
6>digraph:del_edge(G,[‘$e’|0]).
true
7>digraph:out_neighbours(G,['$v'|1]).
[]
8>digraph:out_neighbours(G,['$v'|0]).
** exception error: bad argument
in function ets:lookup_element/3
called as ets:lookup_element (45073,['$e'|0],3)
in call from digraph:collect_elems/4
[„$e‟|0] is still recorded as an our-edge of [„$v‟|0] after being modified. Therefore, as
trying to get the out-neighbours of [„$v‟|0], the system intends to find it out in the
opposite of [„$e‟|0], which, however, has been deleted. Hence, exception arises
accordingly.
Bug3
The function digraph:del_path(G,V1,V2) -> true deletes edges from the digraph G
until there are no path from the vertex V1 to the vertex V2 [3]. However, it cannot
work at all.
1>D=digraph:new().
{digraph,40973,45070,49167,true}
2>digraph:add_vertex(D).
[‘$v’|0]
3>digraph:add_vertex(D).
[‘$v’|1]
4>digraph:add_edge(D,[‘$v’|0],[‘$v’|1]).
[‘$e’|0]
digraph:get_path(G, V1, V2) -> Vertices | false
Tries to find a simple path from the vertex V1 to the vertex V2 of the digraph G.
Returns the path as a list [V1,....., V2] of vertices, or false if no simple path from V1
to V2 of length one or more exists [3].
5>digraph:get_path(D,[‘$v’|0],[‘$v’|1]).
[[‘$v’|0],[‘$v’|1]]
33
However, an exception arises by deleting this path. The process which controls the
digraph terminates.
6>digraph:del_path(D,[‘$v’|0],[‘$v’|1]).
** exception error: bad argument
in function ets:lookup_element/3
called as ets:lookup_element (45070,['$e'0|],3)
in call from digraph:collect_elems/4
in call from digraph:get_path/3
in call from digraph:del_path/3
5.2 Module dets
Module dets provides a term storage on file. It is a collection of objects, which are
tuples, with the key at the same position. Tables are divided into three different types,
set, bag and duplicate_bag. A set can only have one object associated with each key.
A bag or duplicate_bag can have many objects associated with each key. But the same
object can be more than one in duplicate_bag.
Bug4
This bug appears in testing type duplicate_bag under module dets. We can simplify
the test cases by assign 1 as the key-position in the tuple.
1>eqc:quickcheck(dets_tests:prop_statem_correct(duplicate_bag,1)).
……………………………………………………………………….
{postcondition, false}
Shrinking------------------(10 times)
[{init, {state, duplicate_bag, 1, []}},
{set, {var,13}, {call, dets, insert, [{var, tab}, {3, 0}]}},
{set, {var,14}, {call, dets, insert_new, [{var,tab}, [{3, 0}, {3, 0}]]}},
{set, {var,15}, {call, dets, lookup, [{var, tab}, 3}}]
[{{state, duplicate_bag, 1, []}, ok},
{{state, duplicate_bag, 1, [{3, 0}]}, false},
{{state, duplicate_bag, 1, [{3, 0}]}, [{3, 0}, {3, 0}]}]
{state, duplicate_bag, 1, [{3, 0}]}
{postcondition, false}
false
Before explaining this test case, let‟s see the definition of function dets:insert/2,
dets:insert_new/2 and dets:lookup/2.
34
dets:insert(TableName, Objects) -> ok | {error, Reason}
Inserts one or more objects into the table Name. If there already exists an object with
a key matching the key of some of the given objects and the table type is set, the old
object will be replaced [3].
dets:insert_new(TableName, Objects) -> Bool
Inserts one or more objects into the table Name. If there already exists some object
with a key matching the key of any of the given objects the table is not updated and
false is returned, otherwise the objects are inserted and true returned [3].
dets:lookup(TableName, Key) -> [Object] | {error, Reason}
Returns a list of all objects with the key stored in the table [3].
In Table 5-2, it goes through this failing test case by seeing the entire process of
QuickCheck state machine.
Table 5-2: Entire process of QuickCheck state machine (dets)
Command generation time:
Cycle 1
Cycle 2
Cycle 3
Initial state
Command
Precondition
Next state
Command
Precondition
Next state
Command
Precondition
Next state
{state, duplicate_bag, 1, []}
{call, dets, insert, [{var, tab}, {3, 0}]}
true
{state, duplicate_bag, 1, [{3,0}]}
{call, dets, insert_new, [{var,tab}, [{3, 0}, {3, 0}]]}
true
{state, duplicate_bag, 1, [{3,0}]}
{set, {var,15}, {call, dets, lookup, [{var, tab}, 3}}
true
unchanged
Command execution time:
Cycle 1
Cycle 2
Cycle 3
Initial state
Invocation
Postcondition
Next state
Invocation
Postcondition
Next state
Invocation
Postcondition
{state, duplicate_bag, 1, []}
dets:insert(D,{3,0})
true
{state, duplicate_bag, 1, [{3, 0}]}
dets:insert_new(D,[{3,0},{3,0}])
true; (If there already exists some object with a key
matching the key of any of the given objects the table is
not updated and false is returned, otherwise the objects are
inserted and true returned.)
{state, duplicate_bag, 1, [{3, 0}]}
dets:lookup(D,3)
false
35
An Erlang record, -record(state, {type, keypos, list}), is used to store the state of this
state machine with the type of tested data-structure, the key position of tuple and a list
of latest tuples in the actual table. In the beginning, {init, {state, duplicate_bag, 1, []}}
defines the initial state as an empty duplicate_bag with the key position at 1. At the
end, {postcondition, false} shows that it fails during verifying the specification for
executed command, which is dets:lookup/2 in this case. The middle part shows the
detail of how the postcondition failed. It can be divided into three parts. First is the
history of the test command sequence. {set, {var,13}, {call, dets, insert, [{var, tab},
{3, 0}]}} means to set the symbolic representation, {call, dets, insert, [{var, tab}, {3,
0}]}, to the variable {var,13}. We can see the failed sequence starts with function
dets:insert/2 and then dets:insert_new/2, finally dets:lookup/2. The second part shows
the sequence of changed states. The state is always modified as executing next
command. For example, the first half part of {{state, duplicate_bag, 1, []}, ok} shows
the pre state for executing command {call, dets, insert, [{var, tab}, {3, 0}]} and the
second half part shows the return value of this insert function. However, we can see
the state is actually changed by adding a new element {3, 0} to the list from {{state,
duplicate_bag, 1, [{3, 0}]}, false} as executing the next command.
As 3 is already the key of stored element {3, 0}, the list of elements [{3, 0}, {3, 0}]
will not be inserted into the table by calling function insert_new. Therefore, it returns
false, which can be seen in the second part of {{state, duplicate_bag, 1, [{3, 0}]},
false}. Though {{state, duplicate_bag, 1, [{3, 0}]}, [{3, 0}, {3, 0}]} shows the state is
not changed accordingly, function lookup returns [{3, 0}, {3, 0}] but not the expected
value [{3, 0}].
Now, we test this command sequence directly from the console.
1> dets:open_file(tab,[{type,duplicate_bag}]).
{ok,tab}
2> dets:insert(tab,{3,0}).
ok
3> dets:insert_new(tab,[{3,0},{3,0}]).
false
4> dets:lookup(tab,3).
[{3,0}]
What a surprise!! dets:lookup/2 returns correct answer from the console. According to
the observation, the delayed time between each command by typing them one by one
from the console is actually much longer than executing the same commands in a
sequence by QuickCheck. It is probably the source of error? Based on this guessing,
we write Code 5-1 and execute it by setting different delayed time between executing
dets:insert/2 and dets:insert_new2.
36
weird(Delay) ->
dets:open_file(tabName, [{type, duplicate_bag}]),
dets:delete_all_objects(tabName),
dets:insert(tabName, {3,0}),
receive after Delay -> ok end,
dets:insert_new(tabName, [{3,0},{3,0}]),
X = dets:lookup(tabName, 3),
dets:close(tabName),
X.
Code 5-1: Testing delayed error
Now, let‟s set different delayed time.
1> dets_test:weird(2500).
[{3,0},{3,0}]
2> dets_test:weird(2750).
[{3,0},{3,0}]
3> dets_test:weird(2900).
[{3,0},{3,0}]
4> dets_test:weird(2950).
[{3,0},{3,0}]
5> dets_test:weird(2999).
[{3,0},{3,0}]
6> dets_test:weird(3000).
[{3,0}]
This bug is even stranger because we don't need to use a concurrent program to find it.
As the delayed time becomes longer than 3000 milliseconds, function lookup returns
the correct answer. Therefore, delayed time is the source of error. However, we
cannot explain this.
This error is caused not only according to the delayed time between executing
dets:insert/2 and dets:insert_new/2. The same thing happens if we use
dets:insert_new/2 instead of dets:insert/2.
37
6. Conclusion
QuickCheck increases the possibility of finding failing test cases by self defined
generators and property-based testing. Through applying QuickCheck, we have tested
almost all the pure functions in dict module, orddict module, gb_trees module, sets
module, ordsets module, gb_sets module, lists module, proplists module, string
module, queue module and array module; as well as the imperative functions in
digraph module, ets module and dets module. All the pure functions that have been
tested by QuickCheck got the passing results, which provides Erlang programmers
confidence of using those libraries. As well as, we also implemented generic
properties for more than one tested functions. Furthermore, the found bugs among
imperative functions have been reported to the Erlang/OTP programming team.
38
Bibliography
[1] Armstrong, J. (2007). Programming Erlang. United States of America: The
Progmatic Programmers.
[2] Arts, T., Castro, L. M., & Hughes, J. (2008). Testing Erlang Data Types with
Quviq QuickCheck. Erlang'08. Victoria, BC, Canada: ACM.
[3] Ericsson AB. (2009). STDLIB Reference Manual. Retrieved 2009, from
Erlang/OTP R13B: http://erlang.org/doc/index.html
[4] Granberg, A., & Jernberg, D. (2007). NBAP message construction using
QuickCheck. Kista: Linköpings Universitet.
[5] Quviq AB. (2009). QuickCheck 1.162 . Quviq.
[6] Quviq AB. (2008). QuickCheck for Erlang Users. Stockholm: Quviq.
[7] Wang, J., & Yeoh, S. S. (2009). Testing Software Block with QuickCheck.
Göteborg: Chalmers University and Technology.
39
Appendix
A. User Manu
A-1 Pure Functions
dict module (combine.erl)
Function names
append
append_list
erase
fetch
fetch_keys
filter
find
from_list
is_key
map
merge
size
store
update/3
update/4
update_counter
QuickCheck testing commands
eqc:quickcheck(combine:prop_append1(combine:dictr()))
eqc:quickcheck(combine:prop_append_list1(combine:dictr()))
eqc:quickcheck(combine:prop_delete2(combine:dictr()))
eqc:quickcheck(combine:prop_fetch_get(combine:dictr() , fetch))
eqc:quickcheck(combine:prop_fetch_keys(combine:dictr() , fetch_keys))
eqc:quickcheck(combine:prop_filter1(combine:dictr()))
eqc:quickcheck(combine:prop_get_value(combine:dictr() , fetch , find))
eqc:quickcheck(combine:prop_from_list1(combine:dictr()))
eqc:quickcheck(combine:prop_contain(combine:dictr() , is_key))
eqc:quickcheck(combine:prop_map(combine:dictr()))
eqc:quickcheck(combine:prop_merge(combine:dictr()))
eqc:quickcheck(combine:prop_size(combine:dictr()))
eqc:quickcheck(combine:prop_insert1(combine:dictr() , store ,
fun combine:insertkeyvaluefromlist/4))
eqc:quickcheck(combine:prop_update3(combine:dictr()))
eqc:quickcheck(combine:prop_update4(combine:dictr()))
eqc:quickcheck(combine:prop_update_counter(combine:dictr()))
40
orddict module (combine.erl)
Function names
QuickCheck testing commands
eqc:quickcheck(combine:prop_dict_is_ordered(combine:orddictr()))
append
append_list
erase
fetch
fetch_keys
filter
find
from_list
is_key
map
merge
size
store
update/3
update/4
update_counter
eqc:quickcheck(combine:prop_append2(combine:orddictr()))
eqc:quickcheck(combine:prop_append_list2(combine:orddictr()))
eqc:quickcheck(combine:prop_delete1(combine:orddictr() , erase))
eqc:quickcheck(combine:prop_fetch_get(combine:orddictr() , fetch))
eqc:quickcheck(combine:prop_fetch_keys(combine:orddictr() ,
fetch_keys))
eqc:quickcheck(combine:prop_filter1(combine:orddictr()))
eqc:quickcheck(combine:prop_get_value(combine:orddictr() , fetch ,
find))
eqc:quickcheck(combine:prop_from_list1(combine:orddictr()))
eqc:quickcheck(combine:prop_contain(combine:orddictr() ,is_key))
eqc:quickcheck(combine:prop_map(combine:orddictr()))
eqc:quickcheck(combine:prop_merge(combine:orddictr()))
eqc:quickcheck(combine:prop_size(combine:orddictr()))
eqc:quickcheck(combine:prop_insert1(combine:orddictr() , store ,
fun lists:keystore/4))
eqc:quickcheck(combine:prop_update3(combine:orddictr()))
eqc:quickcheck(combine:prop_update4(combine:orddictr()))
eqc:quickcheck(combine:prop_update_counter(combine:orddictr()))
41
gb_trees module (combine.erl)
Function names
QuickCheck testing commands
eqc:quickcheck(combine:prop_dict_is_ordered(combine:gb_treesr()))
delete
delete_any
enter
from_orddict
get
is_defined
is_empty
keys
lookup
size
take_largest,
largest
take_smallest,
smallest
values
eqc:quickcheck(combine:prop_delete3(combine:gb_treesr()))
eqc:quickcheck(combine:prop_delete1(combine:gb_treesr() , delete_any))
eqc:quickcheck(combine:prop_insert1(combine:gb_treesr() , enter ,
fun lists:keystore/4))
eqc:quickcheck(combine:prop_from_list3(combine:gb_treesr() ,
from_orddict))
eqc:quickcheck(combine:prop_fetch_get(combine:gb_treesr() , get))
eqc:quickcheck(combine:prop_contain(combine:gb_treesr() , is_defined))
eqc:quickcheck(combine:prop_is_empty(combine:gb_treesr()))
eqc:quickcheck(combine:prop_fetch_keys(combine:gb_treesr() , keys))
eqc:quickcheck(combine:prop_get_value(combine:gb_treesr() , get ,
lookup))
eqc:quickcheck(combine:prop_size(combine:gb_treesr()))
eqc:quickcheck(combine:prop_smallest1(combine:gb_treesr()))
eqc:quickcheck(combine:prop_smallest1(combine:gb_treesr()))
eqc:quickcheck(combine:prop_values(combine:gb_treesr()))
42
sets module (combine.erl)
Function names
add_element
del_element
filter
from_list
intersection/1
intersection/2
is_element
is_set
is_subset
size
subtract
union/1
union/2
QuickCheck testing commands
eqc:quickcheck(combine:prop_insert2(combine:setsr() , add_element))
eqc:quickcheck(combine:prop_delete5(combine:setsr(), del_element ,
fun lists:delete/2))
eqc:quickcheck(combine:prop_filter2(combine:setsr()))
eqc:quickcheck(combine:prop_from_list2(combine:setsr()))
eqc:quickcheck(combine:prop_intersection_many_sets(combine:setsr()))
eqc:quickcheck(combine:prop_intersection_two_sets(combine:setsr()))
eqc:quickcheck(combine:prop_contain(combine:setsr() , is_element))
eqc:quickcheck(combine:prop_is_set(combine:setsr()))
eqc:quickcheck(combine:prop_is_subset(combine:setsr()))
eqc:quickcheck(combine:prop_size(combine:setsr()))
eqc:quickcheck(combine:prop_subtract1(combine:setsr()))
eqc:quickcheck(combine:prop_union_many_sets(combine:setsr()))
eqc:quickcheck(combine:prop_union_two_sets(combine:setsr()))
43
ordsets module (combine.erl)
Function
names
QuickCheck testing commands
eqc:quickcheck(combine:prop_set_is_ordered(combine:ordsetsr()))
add_element
del_element
filter
from_list
intersection/1
intersection/2
is_element
is_set
is_subset
size
subtract
union/1
union/2
eqc:quickcheck(combine:prop_insert2(combine:ordsetsr() , add_element))
eqc:quickcheck(combine:prop_delete5(combine:ordsetsr() , del_element ,
fun combine:deletefromlist/2))
eqc:quickcheck(combine:prop_filter2(combine:ordsetsr()))
eqc:quickcheck(combine:prop_from_list2(combine:ordsetsr()))
eqc:quickcheck(combine:prop_intersection_many_sets(combine:ordsetsr()))
eqc:quickcheck(combine:prop_intersection_two_sets(combine:ordsetsr()))
eqc:quickcheck(combine:prop_contain(combine:ordsetsr() , is_element))
eqc:quickcheck(combine:prop_is_set(combine:ordsetsr()))
eqc:quickcheck(combine:prop_is_subset(combine:ordsetsr()))
eqc:quickcheck(combine:prop_size(combine:ordsetsr()))
eqc:quickcheck(combine:prop_subtract2(combine:ordsetsr() , subtract))
eqc:quickcheck(combine:prop_union_many_sets(combine:ordsetsr()))
eqc:quickcheck(combine:prop_union_two_sets(combine:ordsetsr()))
44
gb_sets module (combine.erl)
Function
names
QuickCheck testing commands
eqc:quickcheck(combine:prop_set_is_ordered(combine:gb_setsr()))
add
add_element
del_element
delete_any
delete
difference
filter
from_list
from_ordset
insert
intersection/1
intersection/2
is_element
is_empty
is_member
is_set
is_subset
size
singleton
subtract
take_largest,
largest
take_smallest,
smallest
union/1
union/2
eqc:quickcheck(combine:prop_insert2(combine:gb_setsr() , add))
eqc:quickcheck(combine:prop_insert2(combine:gb_setsr() , add_element))
eqc:quickcheck(combine:prop_delete5(combine:gb_setsr() , del_element ,
fun combine:deletefromlist/2))
eqc:quickcheck(combine:prop_delete5(combine:gb_setsr() , delete_any ,
fun combine:deletefromlist/2))
eqc:quickcheck(combine:prop_delete6(combine:gb_setsr()))
eqc:quickcheck(combine:prop_subtract2(combine:gb_setsr() , difference))
eqc:quickcheck(combine:prop_filter2(combine:gb_setsr()))
eqc:quickcheck(combine:prop_from_list2(combine:gb_setsr()))
eqc:quickcheck(combine:prop_from_list3(combine:gb_setsr() ,
from_ordset))
eqc:quickcheck(combine:prop_insert3(combine:gb_setsr()))
eqc:quickcheck(combine:prop_intersection_many_sets(combine:gb_setsr()))
eqc:quickcheck(combine:prop_intersection_two_sets(combine:gb_setsr()))
eqc:quickcheck(combine:prop_contain(combine:gb_setsr() , is_element))
eqc:quickcheck(combine:prop_is_empty(combine:gb_setsr()))
eqc:quickcheck(combine:prop_contain(combine:gb_setsr() , is_member))
eqc:quickcheck(combine:prop_is_set(combine:gb_setsr()))
eqc:quickcheck(combine:prop_is_subset(combine:gb_setsr()))
eqc:quickcheck(combine:prop_size(combine:gb_setsr()))
eqc:quickcheck(combine:prop_singleton(combine:gb_setsr()))
eqc:quickcheck(combine:prop_subtract2(combine:gb_setsr() , subtract))
eqc:quickcheck(combine:prop_smallest2(combine:gb_setsr()))
eqc:quickcheck(combine:prop_smallest2(combine:gb_setsr()))
eqc:quickcheck(combine:prop_union_many_sets(combine:gb_setsr()))
eqc:quickcheck(combine:prop_union_two_sets(combine:gb_setsr()))
45
queue module (queue_test.erl ; combine.erl)
Function names
QuickCheck testing commands
out
eqc:quickcheck(queue_test:prop_out())
out_r
eqc:quickcheck(queue_test:prop_out_r())
snoc
eqc:quickcheck(queue_test:prop_snoc())
in
eqc:quickcheck(queue_test:prop_in())
join
eqc:quickcheck(queue_test:prop_join())
split
eqc:quickcheck(queue_test:prop_split())
reverse
eqc:quickcheck(queue_test:prop_reverse())
from_list
eqc:quickcheck(combine:prop_from_list3(combine:queuer() , from_list))
size
eqc:quickcheck(combine:prop_size(combine:queuer()))
filter
eqc:quickcheck(combine:prop_filter2(combine:queuer()))
is_empty
eqc:quickcheck(combine:prop_is_empty(combine:queuer()))
in_r
eqc:quickcheck(combine:prop_insert_front(in_r))
cons
eqc:quickcheck(combine:prop_insert_front(cons))
get
eqc:quickcheck(combine:prop_front_item(get))
head
eqc:quickcheck(combine:prop_front_item(head))
last
eqc:quickcheck(combine:prop_end_item(last))
daeh
eqc:quickcheck(combine:prop_end_item(daeh))
get_r
eqc:quickcheck(combine:prop_end_item(get_r))
peek
eqc:quickcheck(combine:prop_peek(peek , fun erlang:hd/1))
peek_r
eqc:quickcheck(combine:prop_peek(peek_r , fun lists:last/1))
drop
eqc:quickcheck(combine:prop_except_front(drop))
tail
eqc:quickcheck(combine:prop_except_front(tail))
init
eqc:quickcheck(combine:prop_except_end(init))
lait
eqc:quickcheck(combine:prop_except_end(lait))
drop_r
eqc:quickcheck(combine:prop_except_end(drop_r))
46
array module (array_test.erl ; combine.erl)
Function names
QuickCheck testing commands
fix
eqc:quickcheck(array_test:prop_fix())
from_list
eqc:quickcheck(array_test:prop_from_list())
from_orddict
eqc:quickcheck(array_test:prop_from_orddict())
get
eqc:quickcheck(array_test:prop_get())
is_fix
eqc:quickcheck(array_test:prop_is_fix())
map
eqc:quickcheck(array_test:prop_map())
relax
eqc:quickcheck(array_test:prop_relax())
reset
eqc:quickcheck(array_test:prop_reset())
resize/1
eqc:quickcheck(array_test:prop_resize1())
resize/2
eqc:quickcheck(array_test:prop_resize2())
set
eqc:quickcheck(array_test:prop_set())
sparse_map
eqc:quickcheck(array_test:prop_sparse_map())
sparse_size
eqc:quickcheck(array_test:prop_sparse_size())
sparse_to_list
eqc:quickcheck(array_test:prop_sparse_to_list())
sparse_to_orddict
eqc:quickcheck(array_test:prop_sparse_to_orddict())
size
eqc:quickcheck(combine:prop_size(combine:arrayr()))
47
proplists module (proplists_test.erl)
Function names
QuickCheck testing commands
append_values
eqc:quickcheck(proplists_test:prop_append_values())
compact
eqc:quickcheck(proplists_test:prop_compact())
delete
eqc:quickcheck(proplists_test:prop_delete())
expand
eqc:quickcheck(proplists_test:prop_expand())
get_all_values
eqc:quickcheck(proplists_test:prop_get_all_values())
get_bool
eqc:quickcheck(proplists_test:prop_get_bool())
get_keys
eqc:quickcheck(proplists_test:prop_get_keys())
get_value
eqc:quickcheck(proplists_test:prop_get_value())
is_defined
eqc:quickcheck(proplists_test:prop_is_defined())
lookup
eqc:quickcheck(proplists_test:prop_lookup())
lookup_all
eqc:quickcheck(proplists_test:prop_lookup_all())
property/1
eqc:quickcheck(proplists_test:prop_property1())
property/2
eqc:quickcheck(proplists_test:prop_property2())
split
eqc:quickcheck(proplists_test:prop_split())
substitute_aliases
eqc:quickcheck(proplists_test:prop_substitute_aliases())
substitute_negations
eqc:quickcheck(proplists_test:prop_substitute_negations())
unfold
eqc:quickcheck(proplists_test:prop_unfold())
48
lists module (lists_test.erl)
Function names
QuickCheck testing commands
all
eqc:quickcheck(lists_test:prop_all())
any
eqc:quickcheck(lists_test:prop_any())
append/1
eqc:quickcheck(lists_test:prop_append1())
append/2
eqc:quickcheck(lists_test:prop_append2())
delete
eqc:quickcheck(lists_test:prop_delete())
dropwhile
eqc:quickcheck(lists_test:prop_dropwhile())
duplicate
eqc:quickcheck(lists_test:prop_duplicate())
filter
eqc:quickcheck(lists_test:prop_filter())
flatmap
eqc:quickcheck(lists_test:prop_flatmap())
keydelete
eqc:quickcheck(lists_test:prop_keydelete())
keyfind
eqc:quickcheck(lists_test:prop_keyfind())
keymap
eqc:quickcheck(lists_test:prop_keymap())
keymember
eqc:quickcheck(lists_test:prop_keymember())
keymerge
eqc:quickcheck(lists_test:prop_keymerge())
keyreplace
eqc:quickcheck(lists_test:prop_keyreplace())
keysearch
eqc:quickcheck(lists_test:prop_keysearch())
keysort
eqc:quickcheck(lists_test:prop_keysort())
keytake
eqc:quickcheck(lists_test:prop_keytake())
last
eqc:quickcheck(lists_test:prop_last())
map
eqc:quickcheck(lists_test:prop_map())
max
eqc:quickcheck(lists_test:prop_max())
member
eqc:quickcheck(lists_test:prop_member())
merge/1
eqc:quickcheck(lists_test:prop_merge_many())
merge/2
eqc:quickcheck(lists_test:prop_merge())
merge3
eqc:quickcheck(lists_test:prop_merge3())
min
eqc:quickcheck(lists_test:prop_min())
nth
eqc:quickcheck(lists_test:prop_nth())
nthtail
eqc:quickcheck(lists_test:prop_nthtail())
partition
eqc:quickcheck(lists_test:prop_partition())
prefix
eqc:quickcheck(lists_test:prop_prefix())
reverse/1
eqc:quickcheck(lists_test:prop_reverse1())
49
reverse/2
eqc:quickcheck(lists_test:prop_reverse2())
seq/2
eqc:quickcheck(lists_test:prop_seq1())
seq/3
eqc:quickcheck(lists_test:prop_seq2())
sort
eqc:quickcheck(lists_test:prop_sort())
split
eqc:quickcheck(lists_test:prop_split())
splitwith
eqc:quickcheck(lists_test:prop_splitwith())
sublist/2
eqc:quickcheck(lists_test:prop_sublist1())
sublist/3
eqc:quickcheck(lists_test:prop_sublist2())
subtract
eqc:quickcheck(lists_test:prop_subtract())
suffix
eqc:quickcheck(lists_test:prop_suffix())
sum
eqc:quickcheck(lists_test:prop_sum())
takewhile
eqc:quickcheck(lists_test:prop_takewhile())
ukeymerge
eqc:quickcheck(lists_test:prop_ukeymerge())
ukeysort
eqc:quickcheck(lists_test:prop_ukeysort())
umerge/1
eqc:quickcheck(lists_test:prop_umerge_many())
umerge/2
eqc:quickcheck(lists_test:prop_umerge())
umerge3
eqc:quickcheck(lists_test:prop_umerge3())
uzip
eqc:quickcheck(lists_test:prop_uzip())
uzip3
eqc:quickcheck(lists_test:prop_uzip3())
usort
eqc:quickcheck(lists_test:prop_usort())
zip
eqc:quickcheck(lists_test:prop_zip())
zip3
eqc:quickcheck(lists_test:prop_zip3())
zipwith
eqc:quickcheck(lists_test:prop_zipwith())
zipwith3
eqc:quickcheck(lists_test:prop_zipwith3())
erlang module (lists_test.erl)
Function names
QuickCheck testing commands
length
eqc:quickcheck(lists_test:prop_length())
50
string module (string_test.erl)
Function names
QuickCheck testing commands
chars
eqc:quickcheck(string_test:prop_chars())
chr
eqc:quickcheck(string_test:prop_chr_rchr(chr))
concat
eqc:quickcheck(string_test:prop_concat())
copies
eqc:quickcheck(string_test:prop_copies())
equal
eqc:quickcheck(string_test:prop_equal())
left
eqc:quickcheck(string_test:prop_left())
len
eqc:quickcheck(string_test:prop_len())
rchr
eqc:quickcheck(string_test:prop_chr_rchr(rchr))
right
eqc:quickcheck(string_test:prop_right())
sub_string
eqc:quickcheck(string_test:prop_sub_string())
sub_word
eqc:quickcheck(string_test:prop_sub_word())
substr
eqc:quickcheck(string_test:prop_substr())
words
eqc:quickcheck(string_test:prop_words())
51
The functions in module dict, orddict, gb_trees, sets, ordsets, gb_sets, queue and array,
are grouped for testing. As long as the functions having the same property, they can
be tested through the same property-based test case, which is written in the file,
combine.erl.
Property specification
Module
Function
Complete property specification
prop_is_set(R)
set
is_set
prop_is_set(combine:setsr())
ordset
is_set
prop_is_set(combine:ordsetsr())
gb_set
is_set
prop_is_set(combine:gb_setsr())
dict
size
prop_size(combine:dictr())
orddict
size
prop_size(combine:orddictr())
gb_trees
size
prop_size(combine:gb_treesr())
sets
size
prop_size(combine:setsr())
ordsets
size
prop_size(combine:ordsetsr())
gb_sets
size
prop_size(combine:gb_setsr())
queue
len
prop_size(combine:queuer())
array
size
prop_size(combine:arrayr())
dict
from_list
prop_from_list1(combine:dictr())
orddict
from_list
prop_from_list1(combine:orddictr())
sets
from_list
prop_from_list2(combine:setsr())
ordsets
from_list
prop_from_list2(combine:ordsetsr())
gb_sets
from_list
prop_from_list2(combine:gb_setsr())
gb_trees
from_orddict
prop_from_list3(combine:gb_treesr() , from_orddict)
gb_sets
from_ordset
prop_from_list3(combine:gb_setsr() , from_ordset)
queue
from_list
prop_from_list3(combine:queuer() , from_list)
dict
is_key
prop_contain(combine:dictr() , is_key)
orddict
is_key
prop_contain(combine:orddictr() ,is_key)
gb_trees
is_defined
prop_contain(combine:gb_treesr() , is_defined)
sets
is_element
prop_contain(combine:setsr() , is_element)
ordsets
is_element
prop_contain(combine:ordsetsr() , is_element)
gb_sets
is_element
prop_contain(combine:gb_setsr() ,is_element)
gb_sets
is_member
prop_contain(combine:gb_setsr() , is_member)
dict
store
prop_insert1(combine:dictr() , store ,
prop_size(R)
prop_from_list1(R)
prop_from_list2(R)
prop_from_list3(R,Fun)
prop_contain(R,Fun)
prop_insert1(R,Fun ,Fun2)
fun combine:insertkeyvaluefromlist/4)
orddict
store
prop_insert1(combine:orddictr() ,store ,
fun lists:keystore/4)
gb_trees
enter
prop_insert1(combine:gb_treesr() , enter ,
fun lists:keystore/4)
prop_insert2(R,Fun)
sets
add_element
prop_insert2(combine:setsr() , add_element)
ordsets
add_element
prop_insert2(combine:ordsetsr() ,add_element)
52
prop_delete1(R,Fun)
gb_sets
add_element
prop_insert2(combine:gb_setsr() ,add_element)
gb_sets
add
prop_insert2(combine:gb_setsr() , add)
orddict
erase
prop_delete1(combine:orddictr() , erase)
gb_trees
delet_any
prop_delete1(combine:gb_treesr() ,delete_any)
sets
del_element
prop_delete5(combine:setsr(), del_element ,
prop_delete5(R,Fun1, Fun2)
fun lists:delete/2)
ordsets
del_element
prop_delete5(combine:ordsetsr() , del_element ,
fun combine:deletefromlist/2)
gb_sets
del_element
prop_delete5(combine:gb_setsr() , del_element ,
fun combine:deletefromlist/2)
gb_sets
delete_any
prop_delete5(combine:gb_setsr() , delete_any ,
fun combine:deletefromlist/2)
prop_intersection_two_sets(R)
prop_intersection_many_sets(R)
prop_union_two_sets(R)
prop_union_many_sets(R)
prop_filter1(R)
prop_filter2(R)
prop_is_subset(R)
prop_subtract2(R,Fun)
prop_is_empty(R)
sets
intersection/2
prop_intersection_two_sets(combine:setsr())
ordsets
intersection/2
prop_intersection_two_sets(combine:ordsetsr())
gb_sets
intersection/2
prop_intersection_two_sets(combine:gb_setsr())
sets
intersection/1
prop_intersection_many_sets(ombine:setsr())
ordsets
intersection/1
prop_intersection_many_sets(combine:ordsetsr())
gb_sets
intersection/1
prop_intersection_many_sets(combine:gb_setsr())
sets
union/2
prop_union_two_sets(combine:setsr())
ordsets
union/2
prop_union_two_sets(combine:ordsetsr())
gb_sets
union/2
prop_union_two_sets(combine:gb_setsr())
sets
union/1
prop_union_many_sets(combine:setsr())
ordsets
union/1
prop_union_many_sets(combine:ordsetsr())
gb_sets
union/1
prop_union_many_sets(combine:gb_setsr())
dict
filter
prop_filter1(combine:dictr())
orddict
filter
prop_filter1(combine:orddictr())
sets
filter
prop_filter2(combine:setsr())
ordsets
filter
prop_filter2(combine:ordsetsr())
gb_sets
filter
prop_filter2(combine:gb_setsr())
queue
filter
prop_filter2(combine:queuer())
sets
is_subset
prop_is_subset(combine:setsr())
ordsets
is_subset
prop_is_subset(combine:ordsetsr())
gb_stes
is_subset
prop_is_subset(combine:gb_setsr())
ordsets
subtract
prop_subtract2(combine:ordsetsr() , subtract)
gb_sets
subtract
prop_subtract2(combine:gb_setsr() , subtract)
gb_sets
difference
prop_subtract2(combine:gb_setsr() ,difference)
gb_trees
is_empty
prop_is_empty(combine:gb_treesr())
gb_sets
is_empty
prop_is_empty(combine:gb_setsr())
queue
is_empty
prop_is_empty(combine:queuer())
53
prop_fetch_get(R,Fun)
prop_get_value(R,Fun1, Fun2)
prop_fetch_keys(R , Fun)
prop_merge(R)
prop_update3(R)
prop_update4(R)
prop_update_counter(R)
prop_map(R)
prop_insert_front(Fun)
prop_front_item(Fun)
prop_end_item(Fun)
prop_peek(Fun1,Fun2)
prop_except_front(Fun)
prop_except_end(Fun)
dict
fetch
prop_fetch_get(combine:dictr(),fetch)
orddict
fetch
prop_fetch_get(combine:orddictr(),fetch)
gb_trees
get
prop_fetch_get(combine:gb_treesr(),get)
dict
find
prop_get_value(combine:dictr() , fetch , find)
orddict
find
prop_get_value(combine:orddictr(),fetch,find)
gb_trees
lookup
prop_get_value(combine:gb_treesr(),get,lookup)
dict
fetch_keys
prop_fetch_keys(combine:dictr() , fetch_keys)
orddict
fetch_keys
prop_fetch_keys(combine:orddictr() , fetch_keys)
gb_trees
keys
prop_fetch_keys(combine:gb_treesr() , keys)
dict
merge
prop_merge(combine:dictr())
orddict
merge
prop_merge(combine:orddictr())
dict
update/3
prop_update3(combine:dictr())
orddict
update/3
prop_update3(combine:orddictr())
dict
update/4
prop_update4(combine:dictr())
orddict
update/4
prop_update4(combine:orddictr())
dict
update_counter
prop_update_counter(combine:dictr())
orddict
update_counter
prop_update_counter(combine:orddictr())
dict
map
prop_map(combine:dictr())
orddict
map
prop_map(combine:orddictr())
queue
in_r
prop_insert_front(in_r)
cons
prop_insert_front(cons)
get
prop_front_item(get)
head
prop_front_item(head)
last
prop_end_item(last)
daeh
prop_end_item(daeh)
get_r
prop_end_item(get_r)
peek
prop_peek(peek,fun erlang:hd/1)
peek_r
prop_peek(peek_r,fun lists:last/1)
drop
prop_except_front(drop)
tail
prop_except_front(tail)
init
prop_except_end(init)
drop_r
prop_except_end(drop_r)
liat
prop_except_end(liat)
queue
queue
queue
queue
queue
54
A-2 Imperative functions
digraph module (digraph_test.erl)
Type of digraph
QuickCheck testing commands
cyclic digraph
eqc:quickcheck(digraph_test:prop_statem_correct(cyclic))
acyclic digraph
eqc:quickcheck(digraph_test:prop_statem_correct(acyclic))
ets module (ets_test.erl)
Types of
QuickCheck testing commands
structure
set
eqc:quickcheck(ets_test:prop_statem_correct(set,int()))
ordered_set
eqc:quickcheck(ets_test:prop_statem_correct(orderedset,int()))
bag
eqc:quickcheck(ets_test:prop_statem_correct(bag,int()))
duplicate_bag
eqc:quickcheck(ets_test:prop_statem_correct(duplicate_bag,int()))
int()>=1, which defines the key position in the tested structure
dets module (dets_test.erl)
Types of
QuickCheck testing commands
structure
set
eqc:quickcheck(ets_test:prop_statem_correct(set,int()))
bag
eqc:quickcheck(ets_test:prop_statem_correct(bag,int()))
duplicate_bag
eqc:quickcheck(ets_test:prop_statem_correct(duplicate_bag,int()))
int()>=1, which defines the key position in the tested structure
As testing module dets a named file is generated and initialized by the parameters of
the testing command from console. This file should always be deleted by hand from
the disk before testing next command with different parameters.
55
B. Complete Code
http://www.cse.chalmers.se/~nicsma/code.zip
56