Using JML Runtime Assertion Checking
to Automate Metamorphic Testing in
Applications without Test Oracles
Christian Murphy, Kuang Shen, Gail Kaiser
Columbia University
Problem Statement

Some applications (e.g. machine learning,
simulation) do not have test oracles that indicate
whether the output is correct for arbitrary input

Oracles may exist for a limited subset of the input
domain, and gross errors (e.g. crashes) can be
detected with certain inputs or other techniques

However, it is difficult to detect subtle
(computational) errors for arbitrary inputs
Chris Murphy – Columbia University
2
Observation

Even when there is no oracle in the general
case (i.e. knowing the relationship between a
particular input and its output), it may be
possible to know relationships between sets
of inputs and their corresponding outputs

One such approach that takes advantage of
this is “Metamorphic Testing” [Chen ’98]
Chris Murphy – Columbia University
3
Metamorphic Testing

An approach for creating follow-up test cases
based on those that have not revealed any defects

If input x produces output f(x), then the function’s
“metamorphic properties” are used to guide a
transformation function t, which is applied to
produce a new test case, t(x)

We can then predict the expected value of
f(t(x)) based on the value of f(x) obtained
from the actual Chris
execution
Murphy – Columbia University
4
Metamorphic Testing Example

Consider a function std_dev(A[ ])
If we permute the elements in A, the result
should stay the same
 If we multiply each element in A by 2, the
result should be doubled


These properties can be used to create a
“pseudo-oracle” for the function
Chris Murphy – Columbia University
5
Metamorphic Testing without an Oracle

When a test oracle exists, we can know
whether f(t(x)) is correct
we have an oracle for f(x)
 So if f(t(x)) is as expected, then it is correct
 Because

When there is no test oracle, f(x) acts as a
“pseudo-oracle” for f(t(x))
f(t(x)) is as expected, it is not necessarily
correct
 However, if f(t(x)) is not as expected, either
f(x) or f(t(x)) (or both) is wrong
 If
Chris Murphy – Columbia University
6
Our Solution

To address the problem of applications that have
no test oracle, we use Metamorphic Testing at
the function level

To specify the metamorphic properties, we use
extensions to the Java Modeling Language (JML)
and a pre-processor called Corduroy

To automate the process and address the need for
initial input values, we check the properties at
runtime
Chris Murphy – Columbia University
7
Related Work

Applying Metamorphic Testing to “non-testable
programs” (applications without test oracles)
 Chen

IST’02, Chen ISSTA’02, Chan JWSR’07
Checking algebraic specifications at runtime
 Sankar

ISSTA’91, Nunes ICFEM’06
Checking program invariants at runtime
 Flanagan
FME’01, Hangal ICSE’02
Chris Murphy – Columbia University
8
JML Basics

Behavioral interface specification language that
embraces the Design by Contract approach

Developers can specify:
 preconditions
(“requires”)
 postconditions (“ensures”)
 arbitrary assertions

Specifications can be checked dynamically using
JML Runtime Assertion Checking tools
Chris Murphy – Columbia University
9
Process
Specify functions’ metamorphic
properties using extension to JML
Corduroy
Methods that check
metamorphic properties
Pure JML specifications
that invoke test methods
Chris Murphy – Columbia University
10
Metamorphic Properties in JML
/*@
@ensures \result <= 1 &&
\result >= -1;
@meta sine(x + 2 * PI) == \result
@meta -1 * sine(-x) == \result
*/
public double sine (double x) {
...
}
Chris Murphy – Columbia University
11
Extension to JML: Array Functions
/*@
@ensures \result >= 0;
@meta std_dev(\shuffle(A)) ==
\result
@meta std_dev(\multiply(A, 2)) ==
\result * 2
*/
public double std_dev(double[] A) {
...
}
Chris Murphy – Columbia University
12
Extensions to JML

Array/Collection Functions
 Shuffle/permute
the order of the elements
 Multiply each element by a constant
 Add a constant to each element
 Include a new element
 Exclude an element

Addressing non-determinism
 Check
in range of possible values
 Check in a set of possible values
Chris Murphy – Columbia University
13
Model
Input(s)Input(s)
Caller
Function
Check
Metamorphic
Properties
Callee
Function
OutputOutput
Chris Murphy – Columbia University
14
Implementation
/*@
@ensures
__metaTest_average(A,
\result);
@meta
average(\multiply(A,
2)) ==
\result * 2
*/
public double average(double[] A) {
...
}
protected boolean __metaTest_average
(double[] A, double result) {
return JMLDouble.approximatelyEqualTo
(average(Corduroy.multiply(A, 2)),
result * 2);
}
Chris Murphy – Columbia University
15
Case Studies

We investigated the WEKA and RapidMiner
toolkits for Machine Learning in Java

For WEKA, we tested four apps:
 Naïve
Bayes, Support Vector Machines (SVM),
C4.5 Decision Tree, and k-Nearest Neighbors

For RapidMiner, we tested one app:
 Naïve
Bayes
Chris Murphy – Columbia University
16
Experimental Setup

For each of the five apps, we specified 4-6
metamorphic properties of selected methods
(based on our knowledge of the expected
behavior of the overall application)

Testing was conducted using data sets from
UCI Machine Learning Repository

Goal was to determine whether the
properties held as expected
Chris Murphy – Columbia University
17
Findings

Discovered defects in WEKA k-NN and
WEKA Naïve Bayes related to modifying the
machine learning “model”
 This
was the result of a variable not being
updated appropriately

Discovered a defect in RapidMiner Naïve
Bayes related to determining confidence
 There
was an error in the calculation
Chris Murphy – Columbia University
18
Limitations

Specifying and checking the properties was
simplified by use of the tool, but still requires
some knowledge of the app to determine the
properties in the first place

The approach works well for functions that
take input and produce output, but in some
case requires more complex properties than
can be expressed in JML
Chris Murphy – Columbia University
19
Future Work

Reducing testing time by checking
properties in parallel

Implementations for other languages

More empirical studies to determine
effectiveness and applicability to other
domains
Chris Murphy – Columbia University
20
Summary

We have presented a testing approach that uses
metamorphic testing of individual functions in
applications that do not have test oracles

These are specified via an extension to the JML
specification language

We have also presented an implementation
framework called Corduroy, which converts the
specification of metamorphic properties into test
methods
Chris Murphy – Columbia University
21
Using JML Runtime Assertion Checking
to Automate Metamorphic Testing in
Applications without Test Oracles
Chris Murphy
cmurphy@cs.columbia.edu
Categories of Metamorphic Properties







Additive: Increase (or decrease) numerical values
by a constant
Multiplicative: Multiply numerical values by a
constant
Permutative: Randomly permute the order of
elements in a set
Invertive: Reverse the order of elements in a set
Inclusive: Add a new element to a set
Exclusive: Remove an element from a set
ML apps such as ranking, classification, and
anomaly detection exhibit these properties
[Murphy SEKE’08]
Chris Murphy – Columbia University
23
Specifying More Complex Properties
/*@
@meta test_myFunc(x, \result)
*/
public double myFunc (int x) {
...
}
private boolean test_myFunc
(int x, double result)
{
...
}
Chris Murphy – Columbia University
24
Addressing Side Effects
// holds result of last call to “average”
private double value;
/*@
@assignable value;
@meta average(\multiply(A, 2)) == value * 2
*/
public void average(double[] A) {
...
value = ...
// no return value
}
Chris Murphy – Columbia University
25
Addressing Side Effects
protected boolean __metaTest_average
(double[] A)
{
double __value = value; // backup
try {
return JMLDouble.approximatelyEqualTo
(average(Corduroy.multiply(A, 2)),
__value
value * 2);
}
finally {
value = __value; // restore
}
}
Chris Murphy – Columbia University
26