Testing Concurrent Programs
Rice Computer Science Club
Mathias Ricken
Rice University
October 4, 2007
Moore’s Law
*
*
Timeliness
CPU clock frequencies stagnate
Multi-Core CPUs provide additional
processing power
– Multiple threads needed to use multiple cores
Writing concurrent programs is difficult!
Programming Examples
Unit Testing
Unit tests…
– Test a part, not the whole program
– Occur earlier
– Automate testing
– Serve as documentation
– Prevent bugs from reoccurring
– Help keep the shared repository clean
Effective with a single thread of control
Foundation of Unit Testing
Unit tests depend on deterministic
behavior
Known input, expected output…
Success  correct behavior
Failure 
flawed code
Outcome of test is meaningful
Problems Due to Concurrency
Thread scheduling is nondeterministic and
machine-dependent
– Code may be executed under different schedules
– Different schedules may produce different results
Known input, expected output…
Success 
Failure 
correct behavior in this schedule,
may be flawed in other schedule
flawed code
Success of unit test is meaningless
Possible Solutions
Programming Language Features
– Ensuring that bad things cannot happen
– May restrict programmers
Lock-Free Algorithms
– Ensuring that if bad things happen, it’s ok
– May limit data structures available
Comprehensive Testing
– Testing if bad things happen in any schedule
– Does not prevent problems, but does not limit
solutions either
Contributions
Improvements to JUnit
– Detect exceptions and failed assertions in
threads other than the main thread
Annotations for Concurrency Invariants
– Express complicated requirements about
locks and threads
Tools for Schedule-Based Execution
– Record, deadlock monitor
– Random delays, random yields
Improvements to JUnit
Uncaught exceptions and failed assertions
– Not caught in child threads
Sample JUnit Tests
public class Test extends TestCase {
public void testException() {
throw new RuntimeException("booh!");
}
} public void testAssertion() {
Both tests
assertEquals(0, 1);
fail.
}
}
if (0!=1)
throw new AssertionFailedError();
Problematic JUnit Tests
Main
thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!");
}
}).start();
Child
thread
}
}
end of
Main
thread
spawns
test
success!
Child
thread
uncaught!
Problematic JUnit Tests
Main
thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!");
}
}).start();
Child
Uncaught exception,
thread
}
test should fail but
}
does not!
Improvements to JUnit
Uncaught exceptions and failed assertions
– Not caught in child threads
Thread group with exception handler
– JUnit test runs in a separate thread, not main thread
– Child threads are created in same thread group
– When test ends, check if handler was invoked
Thread Group for JUnit Tests
Test
thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!");
}
}).start();
Child
thread
}
invokes
}
TestGroup’s Uncaught
Exception Handler
Thread Group for JUnit Tests
Test
thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!");
}
}).start();
Child
thread
}
spawns and waits
resumes
} Main
failure!
thread
Test
thread
spawns
Child
thread
end of test
uncaught!
invokes
group’s
handler
check
group’s
handler
Improvements to JUnit
Uncaught exceptions and failed assertions
– Not caught in child threads
Thread group with exception handler
– JUnit test runs in a separate thread, not main thread
– Child threads are created in same thread group
– When test ends, check if handler was invoked
Detection of uncaught exceptions and failed
assertions in child threads that occurred before
test’s end
Past tense: occurred!
Child Thread Outlives Parent
Test
thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!");
}
}).start();
Child
thread
}
spawns and waits
resumes
} Main
failure!
thread
Test
thread
spawns
Child
thread
end of test
uncaught!
invokes
group’s
handler
check
group’s
handler
Child Thread Outlives Parent
Test
thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!");
}
}).start();
Child
thread
}
check group’s
resumes
handler
} Main spawns and waits
success!
thread
Test
thread
spawns
Too late!
end of test
Child
thread
uncaught!
invokes
group’s
handler
Improvements to JUnit
Child threads are not required to terminate
– A test may pass before an error is reached
Detect if any child threads are still alive
– Declare failure if test thread has not waited
– Ignore daemon threads, system threads (AWT, RMI,
garbage collection, etc.)
Previous schedule is a test failure
– Should be prevented by using Thread.join()
Enforced Join
Test
thread
public class Test extends TestCase {
public void testException() {
Thread
t = new Thread(new
{
new Thread(new
Runnable() Runnable()
{
public
public void
void run()
run() {
{
throw
new RuntimeException("booh!");
throw
new
throw new RuntimeException("booh!");
RuntimeException("booh!");
}
}
});
});
t.start();
t.start(); …
… t.join();
t.join(); …
Child
}
thread
}
Improvements to JUnit
Child threads are not required to terminate
– A test may pass before an error is reached
Detect if any child threads are still alive
– Declare failure if test thread has not waited
– Ignore daemon threads, system threads (AWT, RMI,
garbage collection, etc.)
Previous schedule is a test failure
– Should be prevented by using Thread.join()
Testing ConcJUnit
Replacement for junit.jar or as plugin JAR for
JUnit 4.2
Available as binary and source at
http://www.concutest.org/
Results from DrJava’s unit tests
– Child thread for communication with slave VM still
alive in test
– Several reader and writer threads still alive in low
level test (calls to join() missing)
Conclusion
Improved JUnit now detects problems in
other threads
– Only in chosen schedule
– Needs schedule-based execution
Annotations ease documentation and
checking of concurrency invariants
– Open-source library of Java API invariants
Support programs for schedule-based
execution
Future Work
Schedule-Based Execution
– Replay given schedule
– Generate possible schedules
– Dynamic race detection
– Probabilities/durations for random
yields/sleeps
Extend annotations to Floyd-Hoare logic
– Preconditions, postconditions
– Representation invariants
Many Thanks To…
My advisor
– Corky Cartwright
My committee members
– Walid Taha
– Bill Scherer
NFS and Texas ATP
– For partially providing funding
Rice Computer Science Club
Extra Slides
Tractability of Comprehensive Testing
Test all possible schedules
– Concurrent unit tests meaningful again
Number of schedules (N)
– t: # of threads, s: # of slices per thread
detail
Extra: Number of Schedules
Product of s-combinations
For thread 1: choose s out of ts time slices
For thread 2: choose s out of ts-s time slices
…
For thread t-1: choose s out of 2s time slices
For thread t-1: choose s out of s time slices
Writing s-combinations
using factorial
Cancel out terms in denominator and next numerator
Left with (ts)! in numerator and t numerators with s!
back
Tractability of Comprehensive Testing
If program is race-free, we do not have to
simulate all thread switches
– Threads interfere only at “critical points”: lock
operations, shared or volatile variables, etc.
– Code between critical points cannot affect outcome
– Simulate all possible arrangements of blocks
delimited by critical points
Run dynamic race detection in parallel
– Lockset algorithm (e.g. Eraser by Savage et al)
Critical Points Example
Local Var 1
lock
access unlock
Thread 1
All lock
accesses
access unlock
protected by
lock
Shared Var
Lock
Thread 2
All accesses
protected by
lock
lock
Local Var 1
access unlock
All accesses
protected by
lock
Local variables
don’t need
locking
Fewer Schedules
Fewer critical points than thread switches
– Reduces number of schedules
– Example:
Two threads, but no communication
N=1
Unit tests are small
– Reduces number of schedules
Hopefully comprehensive simulation is tractable
– If not, heuristics are still better than nothing
Limitations
Improvements only check chosen
schedule
– A different schedule may still fail
– Requires comprehensive testing to be
meaningful
May still miss uncaught exceptions
– Specify absolute parent thread group, not
relative
– Cannot detect uncaught exceptions in a
program’s uncaught exception handler (JLS
limitation)
details
Extra: Limitations
May still miss uncaught exceptions
– Specify absolute parent thread group, not
relative (rare)
Koders.com: 913 matches ThreadGroup vs.
49,329 matches for Thread
– Cannot detect uncaught exceptions in a
program’s uncaught exception handler (JLS
limitation)
Koders.com: 32 method definitions for
uncaughtException method
back
Extra: DrJava Statistics
2004
Unit tests
passed
failed
not run
Invariants
met
failed
% failed
KLOC
“event thread”
736
610
36
90
5116
4161
965
18.83%
107
1
2006
881
881
0
0
34412
30616
3796
11.03
129
99
back
Concurrency Invariants
Has to be called in event thread
– TableModel, TreeModel
May not be called in event thread
– invokeAndWait()
Have to acquire readers/writers lock
– AbstractDocument
– DrJava’s documents
Invariants Difficult to Determine
May be found in
– Javadoc comments
– Only in internal comments
– Whitepapers
Often not documented at all
Errors not immediately evident
Impossible to check automatically
Java Annotations
Add invariants as annotations
@NotEventThread
public static void invokeAndWait(
Runnable r) { … }
Process class files
– Find uses of annotations
– Insert bytecode to check invariants at method
beginning
Advantages of Annotations
Java Language constructs
– Syntax checked by compiler
Easy to apply to part of the program
– e.g. when compared to a type system change
Light-weight
– Negligible runtime impact if not debugging (slightly
bigger class files)
Automatic Checking
Predicate Annotations
In annotation definition, specify static
boolean Java method
– Method must be callable from every context
 completely static and public
Data in annotation, method arguments and
value of this passed when method
invoked
Predicate Annotation Example
@PredicateLink(value=Predicates.class,
Refers to
method="example",
Predicates
arguments=true)
.example
public @interface ExampleAnnotation {
String foo;
}
Definition
Predicate Annotation Example
public class TestCode {
@ExampleAnnotation(foo="test")
public void test(int param) { … }
}
…
TestCode t = new TestCode(); t.test(5);
Usage
Call
Predicate Annotation Example
public class Predicates {
public static boolean example(
Object this0,
int param,
String foo) {
return (foo.length()<param);
}
Predicate Annotation Example
@PredicateLink(value=Predicates.class,
method="example",
arguments=true)
public @interface ExampleAnnotation {
String foo;
}
public class TestCode {
@ExampleAnnotation(foo="test")
public void test(int param){…}
}
…
TestCode t = new TestCode();
t.test(5);
public class Predicates {
public static boolean example(
Object this0,
int param,
String foo) {
return (foo.length()<param); // this0==t, param==5, foo=="test"
}
Invariant Annotation Library
@OnlyEventThread, @NotEventThread
@OnlyThreadWithName
@NotNullArgument
@DistinctArguments, @SameArguments
@OnlySynchronizedThis,
@NotSynchronizedThis
@OnlySynchronizedArgument,
@NotSynchronizedArgument
etc. (ca. 80 annotations)
Problem: Multiple Annotations
Java does not allow the same annotation
class multiple times
@OnlyThreadWithName("foo")
@OnlyThreadWithName("bar") // error
void testMethod() { … }
Conjunctions, disjunctions and negations?
Annotation Subclasses?
Let annotation extend a supertype?
public @interface Invariant { }
public @interface OnlyThreadWithName
extends Invariant { String name(); }
public @interface And extends Invariant {
Invariant[] terms();
}
Subtyping not allowed for annotations
Generic Annotations?
Write @And as generic annotation?
public @interface And<T> {
T[] terms();
}
public @interface OnlyThreadWithName {
String name();
}
Generics not allowed in annotations
Work-Around
Different meta-annotation, @Combine
@Combine(Combine.Mode.AND)
public @interface SeveralNames {
OnlyThreadWithName[] value();
}
@SeveralNames({@OnlyThreadWithName("foo"),
@OnlyThreadWithName("bar")})
void testMethod() { … }
Combine Annotations
May only contain invariant annotations
– Predicate annotations
– Combine annotations
– Arrays of the above
Predicate method automatically generated
– Calls member predicate methods
– Accumulates using AND, OR or NOT
NOT first negates, then uses AND
Default mode is OR
De Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)
Invariant Inheritance
Invariants on a method
– Apply to the method and all overriding methods in
subclasses
Invariants on a class
– Apply to all methods introduced in that class or
subclasses
 Methods can have invariants defined elsewhere
All annotations describe requirements for the
client (and, due to subclassing, for subclasses)
– Allows frameworks to describe requirements
– Description “thread-safe” is often wrong
Invariant Subtyping
To maintain substitutability, subclasses may not
strengthen invariants
Invariants can be modeled as special input
parameter
– Tuple of invariants (“record” in λ calculus [Pierce])
– Subtyping rules for records declare the “wider” record
as subtype
– In function types, parameter types are contravariant
I0 = {}, I1 = {inv1}, I2 = {inv1,inv2}, I2 <: I1 <: I0
F0 = I0 → ·, F1 = I1 → ·, F2 = I2 → ·, F0 <: F1 <: F2
Invariant Subtyping
Analyze methods with invariants as parameter
class A {
void f()
{ … };
}
class B
extends A {
@Inv1
void f()
{ … };
}
Invariants subtyping:
class C
extends B {
@Inv2
void f()
{ … };
}
A <@ B <@ C
IA = {}, IB = {inv1}, IC = {inv1,inv2}; IC <: IB <: IA
FA = IA → ·, FB = IB → ·, FC = IC → ·; FA <: FB <: FC
Java subtyping:
C <: B <: A
Detection of Subtyping Problems
If Java subtyping and invariant subtyping
disagree (A <: B but B <@ A)
– Substitutability not maintained
– Statically emit warning
Detect if client subclasses do not use
framework classes as prescribed
– Safer multithreaded frameworks
Java API Annotations
Started to annotate methods in Java API
– 30 whole classes, 44 individual methods
Community project at
http://community.concutest.org/
– Anyone can suggest annotations
– Vote on suggested annotations
– Browse by class or annotation type
Annotations can be extracted into XML
– Share annotations
– Add checks without needing source code
Testing Invariant Checker
Annotated two DrJava versions
– 3/26/2004
– 9/2/2006
Ran test suite, logged invariant violations
– 2004: 18.83% failed
– 2006: 11.03% failed
2006 version easier to annotate
– Better documentation of invariants