The In Vivo Testing Approach
Christian Murphy, Gail Kaiser, Ian Vo, Matt Chu
Columbia University
Problem Statement

It is infeasible to fully test a large system
prior to deployment considering:
 different
runtime environments
 different configuration options
 different patterns of usage

This problem may be compounded by
moving apps from single-CPU machines to
multi-core processors
Chris Murphy, Columbia University
2
Our Solution

Continually test applications executing in the
field (in vivo) as opposed to only testing in
the development environment (in vitro)

Conduct the tests in the context of the
running application

Do so without affecting the system’s users
Chris Murphy, Columbia University
3
int main ( ) {
...
...
...
foo(x);
...
...
test_foo(x);
}
Chris Murphy, Columbia University
4
Contributions

A new testing approach called in vivo
testing designed to execute tests in the
deployment environment

A new type of tests called in vivo tests

An implementation framework called Invite
Chris Murphy, Columbia University
5
Related Work

Perpetual testing [Clarke SAS’00]
Skoll [Memon ICSE’04]
 Gamma [Orso ISSTA’02]
 CBI [Liblit PLDI’03]


Distributed In Vivo Testing [Chu ICST’08]
Chris Murphy, Columbia University
6
Example of Defect: Cache
private int numItems = 0, currSize = 0;
private int maxCapacity = 1024; // in bytes
public int
getNumItems()
{
Number
of
Their size
return
numItems;
items in
(in bytes)
}
the cache
Maximum
public boolean addItem(CacheItem
i) throws ...
capacity
{
Should only be incremented
numItems++;
within “if” block
if (currSize + i.size < maxCapacity) {
add(i); currSize += i.size;
return true;
} else { return false; }
}
Chris Murphy, Columbia University
7
Insufficient Unit Test
public void testAddItem() {
Cache c = new Cache();
assert(c.addItem(new CacheItem()))
assert(c.getNumItems() == 1);
assert(c.addItem(new CacheItem()))
assert(c.getNumItems() == 2);
}
1. Assumes an empty/new cache
2. Doesn’t take into account various states
that the cache can be in
Chris Murphy, Columbia University
8
Defects Targeted
1.
2.
3.
4.
5.
Unit tests that make incomplete
assumptions about the state of objects in
the application
Possible field configurations that were not
tested in the lab
A legal user action that puts the system in
an unexpected state
A sequence of unanticipated user actions
that breaks the system
Defects that only appear intermittently
Chris Murphy, Columbia University
9
Applications Targeted

Applications that produce calculations or
results that may not be obviously wrong
 “Non-testable
programs”
 Simulations

Applications in which exta-functional
behavior may be wrong even if output is
correct
 Caching
systems
 Scheduling of tasks
Chris Murphy, Columbia University
10
In Vivo Testing: Process
1.
Create test code (using existing unit tests or
new In Vivo tests)
2.
Instrument application using Invite testing
framework
3.
Configure framework
4.
Deploy/execute application in the field
Chris Murphy, Columbia University
11
Model of Execution
Function is
about to be
executed
Run a test?
NO
Execute
function
Rest of
program
continues
Yes
Create
sandbox
Run test
Stop
Fork
Chris Murphy, Columbia University
12
Writing In Vivo Tests
/* Method to be tested */
public boolean addItem(CacheItem i) { . . . }
/* In
JUnit
Vivo style test */
public boolean
void
testAddItem()CacheItem
{
i) {
Cache c = new
Cache();
this;
int oldNumItems = getNumItems();
i)) CacheItem()))
if (c.addItem(new
assert
1);
return (c.getNumItems() == oldNumItems+1;
else return true;
}
Chris Murphy, Columbia University
13
Instrumentation
/* Method to be tested */
public boolean __addItem(CacheItem i) { . . . }
/* In Vivo style test */
public boolean testAddItem(CacheItem i) { ... }
public boolean addItem(CacheItem i) {
if (Invite.runTest(“Cache.addItem”)) {
Invite.createSandboxAndFork();
if (Invite.isTestProcess()) {
if (testAddItem(i) == false) Invite.fail();
else Invite.succeed();
Invite.destroySandboxAndExit();
}
}
return __addItem(i);
}
Chris Murphy, Columbia University
14
Configuration

Each instrumented method has a set
probability ρ with which its test(s) will run

To avoid bottlenecks, can also configure:
 Maximum
allowed performance overhead
 Maximum number of simultaneous tests

Also, what action to take when a test fails
Chris Murphy, Columbia University
15
Case Studies

Applied testing approach to two caching systems
 OSCache
2.1.1
 Apache JCS 1.3

Both had known defects that were found by users
(no corresponding unit tests for these defects)

Goal: demonstrate that “traditional” unit tests would
miss these but In Vivo testing would detect them
Chris Murphy, Columbia University
16
Experimental Setup
An undergraduate student created unit tests
for the methods that contained the defects
 These tests passed in “development”

Student was then asked to convert the unit
tests to In Vivo tests
 Driver created to simulate real usage in a
“deployment environment”

Chris Murphy, Columbia University
17
Discussion

In Vivo testing revealed all defects, even
though unit testing did not

Some defects only appeared in certain states,
e.g. when the cache was at full capacity
 These
are the very types of defects that In Vivo
testing is targeted at

However, the approach depends heavily on
the quality of the tests themselves
Chris Murphy, Columbia University
18
Performance Evaluation

We instrumented three C and two Java
applications with the framework and varied
the value ρ (probability that a test is run)

Applications were run with real-world inputs
on a dual-core 3GHz server with 1GB RAM

No restraints were placed on maximum
allowable overhead or simultaneous tests
Chris Murphy, Columbia University
19
Experimental Results
90
80
Time (seconds)
70
60
C 1 (11,065 tests)
50
C 2 (26,791 tests)
C 3 (4,718 tests)
40
Java 1 (13,694 tests)
Java 2 (2,300 tests)
30
20
10
0
0%
25%
50%
75%
Chris Murphy, Columbia University
100% percent of function
calls resulting in tests
20
Discussion

Percent overhead is not a meaningful metric
since it depends on the number of tests run
 More
tests = more overhead
 Short-running programs with lots of tests will
have significantly more “overhead” than longrunning programs
For C, the overhead was 1.5ms per test
 For Java, around 5.5ms per test

Chris Murphy, Columbia University
21
Future Work

Ensure that test does not affect the external
system state (database, network, etc.)

Adjust frequency of test execution based on
context or resource availability (CPU usage,
number of threads, etc.)

Apply approach to certain domains, e.g.
security testing
Chris Murphy, Columbia University
22
Conclusion

We have presented a new testing approach
called in vivo testing designed to execute
tests in the deployment environment

We have also presented an implementation
framework called Invite

In Vivo testing is an effective technique at
detecting defects not caught in the lab
Chris Murphy, Columbia University
23
The In Vivo Testing Approach
Christian Murphy, Gail Kaiser, Ian Vo, Matt Chu
Columbia University