Using Runtime Testing to Detect
Defects in Applications
without Test Oracles
Chris Murphy
Columbia University
November 10, 2008
About Me

3rd-year PhD student at Columbia University

Advised by Prof. Gail Kaiser

Research interests:
 Software
Testing
 Computer Science Education
 Computer-Supported Cooperative Work
Chris Murphy – Columbia University
2
Introduction

This thesis addresses the problem of testing
complex, highly configurable systems, particularly
those without “test oracles” that indicate what the
correct output should be for arbitrary input

We adapt a technique that leverages built-in
“pseudo-oracles” and perform testing in the
deployment environment in order to address
limitations regarding defects that reveal themselves
only in certain states or with certain input data
Chris Murphy – Columbia University
3
Overview
Problem Statement & Requirements
 Approach & Hypotheses
 Model & Architecture
 Feasibility & Preliminary Results
 Related Work
 Expected Contributions

Chris Murphy – Columbia University
4
Problem Statement

Some applications, such as in Machine
Learning, do not have test oracles for the
general case

Even if certain defects may be detectable,
others can only be revealed as a result of
particular input values, configurations,
application states, or runtime environments
that may not have been tested prior to
deploying a software product
Chris Murphy – Columbia University
5
Observation

Even when there is no oracle in the general
case, there can still be a limited subset of
inputs such that:
 they
can at least reveal certain types of defects,
e.g. catastrophic failures (crashes), and/or
 the expected output can actually be known

These inputs may be generated based on
past inputs and their respective outputs, as in
“Metamorphic Testing” [Chen ’98]
Chris Murphy – Columbia University
6
Metamorphic Testing

Originally designed as an approach for creating follow-up
test cases based on those that have not revealed any
defects

If input x produces output f(x), then the function’s
“metamorphic properties” are used to guide a transformation
function t, which is applied to produce t(x)

We can then predict the expected value of f(t(x)) based
on the known value of f(x)

If f(t(x)) is not as expected, then a defect exists
Chris Murphy – Columbia University
7
Metamorphic Testing Example

Anomaly-based network intrusion detection systems
build a “model” of normal behavior

In some cases, the model may consider the byte
distribution of data in the incoming network packet;
anything deemed anomalous causes an alert

We cannot know a priori whether a particular packet
should cause an alert

However, if we permute the order of the bytes, the result
(anomalous or not) should be the same as for the
original packet, since the system only considers the
distribution of the Chris
bytes
Murphy – Columbia University
8
Proposed Approach

To address the problem of functions and/or
applications that have no test oracle, we use
Metamorphic Testing

In such applications, to reveal defects that
are dependent on input data, configuration,
application state, or the runtime environment,
we continue Metamorphic Testing in the field,
after deployment and during actual usage
Chris Murphy – Columbia University
9
Approach Details

Initial input/output pairs are taken from actual executions

We cannot know whether the output is correct but we at
least know that the input is something that comes up in
practice, and is useful as a valid test case

We then apply “metamorphic properties” to get test input, so
that we should be able to predict the test output

Although we cannot know whether the test output is correct
either, if it is not as predicted then there is a defect

Since this runs in the field, we have to ensure that users
don't notice this testing, e.g. see the test output, experience
a sudden performance lag, etc.
Chris Murphy – Columbia University
10
Hypotheses
1.
For programs that do not have a test
oracle, conducting Metamorphic Testing
within the context of the application
running in the field can reveal defects that
would not ordinarily otherwise be found
2.
This can be done without affecting the
application state from the users’
perspective, and with minimal
performance overhead
Chris Murphy – Columbia University
11
Proposed Model
1.
Automated Metamorphic System Testing:
Conducts system-level Metamorphic Testing in
the deployment environment
2.
Metamorphic Runtime Checking: A separate
testing technique that, for individual units
(functions), supports the execution of
Metamorphic Tests that are executed “from
within” the context of the running application
Chris Murphy – Columbia University
12
Automated Metamorphic System Testing

Checks that the metamorphic properties of the
entire system hold after execution

Treats the application as a black box

Multiple invocations run in parallel, and results are
compared upon completion

User only sees output from the “original” invocation
Chris Murphy – Columbia University
13
Chris Murphy – Columbia University
14
Amsterdam: Automated Metamorphic System Testing Framework

Metamorphic properties are specified in XML
 Input transformation
 Runtime options
 Output comparison

Framework provides out-of-box support for
numerous transformation and comparison
functions but is extendable to support custom
operations

Additional invocations are executed in parallel in
separate sandboxes that have their own virtual
execution environment
Chris Murphy – Columbia University
15
Metamorphic Runtime Checking

For individual units (functions), we check whether
the metamorphic properties hold as the application
is running, using actual input from real executions
and the application’s current state

Function arguments are modified according to the
metamorphic properties
Function is called again (in an isolated sandbox)
with the new input
Outputs are compared


Chris Murphy – Columbia University
16
Function foo
is about to
be executed
Run a
pre-test?
no
Execute foo
Run a
post-test?
no
yes
yes
Create a
sandbox
for the test
Create a
sandbox
for the test
Program
continues
Execute
preTestFoo
Record
success/failure
Chris Murphy – Columbia University
Execute
postTestFoo
Record
success/failure
17
Foundation: In Vivo Testing

To facilitate testing “from within” a running program,
we will extend In Vivo Testing [Chu ICST’08]

In Vivo Tests are analogous to unit tests but they
test from within the context of the running
application as it executes in the deployment
environment, as opposed to a clean slate

They test that sequences of actions produce the
expected results, no matter what the configuration,
state, or runtime environment
Chris Murphy – Columbia University
18
Complementary Testing Approaches

Metamorphic Testing addresses a limitation
of In Vivo Testing:
 The

need for a test oracle
In Vivo Testing addresses some limitations
of Metamorphic Testing:
 Availability
of initial test data
 Detecting defects that only appear in certain
states, configurations, or environments; or
occur only intermittently
Chris Murphy – Columbia University
19
Columbus: Metamorphic Runtime Checking Framework

Tests (specifications of metamorphic properties)
are written by developers, then select
components of the application are instrumented
with tests at compile-time

Configuration includes: probability of running a
test for each function; maximum number of
concurrent tests; action to take when test fails;
whether to assign tests to separate
processor/core

Test sandbox can be created by simple “fork” or
by creating a virtual execution environment
Chris Murphy – Columbia University
20
Preliminary Results

Identified categories of metamorphic properties in
the domain of Machine Learning

Detected defects with Metamorphic System Testing

Detected defects with In Vivo Testing

Detected defects with Metamorphic Runtime
Checking
Chris Murphy – Columbia University
21
Categories of Metamorphic Properties







[Murphy SEKE’08]
Additive: Increase (or decrease) numerical values
by a constant
Multiplicative: Multiply numerical values by a
constant
Permutative: Randomly permute the order of
elements in a set
Invertive: Reverse the order of elements in a set
Inclusive: Add a new element to a set
Exclusive: Remove an element from a set
ML apps such as ranking, classification, and
anomaly detection exhibit these properties
Chris Murphy – Columbia University
22
Feasibility: Metamorphic System Testing

We performed system-level metamorphic
testing on various types of Machine Learning
applications [Murphy SEKE’08]

Detected previously-unknown defects in a
real-world network intrusion detection system

However, this testing was not automated:
inputs were modified with one-off scripts and
outputs were compared manually
Chris Murphy – Columbia University
23
Feasibility: In Vivo Testing

We have previously developed an implementation
of the In Vivo Testing framework for Java
applications called Invite [Chu ICST’08]

Targeted towards applications in which defects
were not obvious to the user (not necessarily those
without test oracles)

Detected known defects in OSCache that were not
found by traditional unit tests

Uses “fork” to create new processes for sandbox
Chris Murphy – Columbia University
24
Feasibility: Metamorphic Runtime Checking

We have developed a system by which
functions’ metamorphic properties are
specified using an extension to JML

Specifications converted into metamorphic
unit tests by a tool called Corduroy [Murphy ’08]
 tests

run using JML Runtime Assertion Checking
Detected defects in WEKA and RapidMiner
machine learning toolkits
Chris Murphy – Columbia University
25
Related Work: Absence of Oracles
Pseudo-oracles [Davis ACM’81]
 Testing non-testable programs [Weyuker TCJ’82]


Overview of approaches [Baresi ’01]
 Embedded
assertion languages
 Extrinsic interface contracts
 Pure specification languages
 Trace checking & log file analysis

Using metamorphic testing [Chen JIST’02]
Chris Murphy – Columbia University
26
Related Work: Testing in the Field

Perpetual Testing [Osterweil QW’96]
Gamma [Orso ISSTA’02]
 Skoll [Memon ICSE’04]
 Cooperative Bug Isolation [Liblit RAMSS’04]

Failure-Oblivious Computing [Rinard OSDI’04]
 Security systems that monitor for errors

Chris Murphy – Columbia University
27
Methodology (1)

To further demonstrate feasibility, we will conduct
Automated Metamorphic System Testing and
Metamorphic Runtime Checking on real-world
Machine Learning applications as they run under
normal operation in the field

We will also show that certain defects would not
ordinarily have been detected by using
Metamorphic Testing (or other techniques) prior to
deployment
Chris Murphy – Columbia University
28
Methodology (2)

To show that our testing approach advances
the state of the art in testing applications that
have no test oracle, we will compare it to
other techniques that could be used to
address this same problem
 Symbolic
execution
 Model checking
 Program invariants
 Formal specification languages
Chris Murphy – Columbia University
29
Expected Contributions
1.
Automated Metamorphic System Testing
and a testing framework called Amsterdam
2.
Metamorphic Runtime Checking and a
testing framework called Columbus
3.
A set of guidelines for assisting the
formulation and specification of
metamorphic properties
Chris Murphy – Columbia University
30
Using Runtime Testing to
Detect Defects in Applications
without Test Oracles
Chris Murphy
cmurphy@cs.columbia.edu
Chris Murphy – Columbia University
31