Poirot – A Concurrency Sleuth
Shaz Qadeer
Research in Software Engineering
Microsoft Research
Concurrent programming is difficult
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
IO_REQUEST_PACKET *irp;
irp->Cancel = FALSE;
irp->CancelRoutine = NULL;
Normal
Cancellation
…
if (irp->Cancel) {
IoCompleteIrp(irp);
} else {
IoSetCancelRoutine(irp, CancelRoutine);
IoMarkIrpPending(irp);
}
…
…
irp->Cancel = TRUE;
fn = IoSetCancelRoutine(Irp, NULL);
if (fn) {
fn(irp);
}
…
Fatal error!
void CancelRoutine(IRP *irp) {
IoCompleteIrp(irp);
}
Concurrent programming is difficult
• Multiple loci of control resulting in non-local
control flow
• Code difficult to understand and review
What about verification?
• Assertion-based modular reasoning becomes complicated due
to non-local interactions
– Floyd-Hoare morphs into Owicki-Gries
• Even with simple (finite) abstractions, the presence of
concurrency makes the analysis computationally very
expensive
Sequential
Concurrent
Single
Procedure
P-time
PSPACE-complete
Multi
Procedure
P-time
Undecidable
What about testing?
Scheduling nondeterminism
• Uncontrollable
• Unobservable
• Exponential
Thread 1
Thread n
x = 1;
…
…
…
…
…
x = k;
x = 1;
…
…
…
…
…
x = k;
…
• Number of executions = O( nnk )
• Exponential in both n and k
Concurrency is important
• More than ever before
• Increasing importance of communicating systems
– networked devices
– cyber-physical systems
• Distributed programs running on the cloud
– EC2, Azure, AppEngine, …
• Parallel programs running on multicores and GPUs
– TBB, TPL, CUDA, AMP, …
Concurrency testing with CHESS
• Deterministic scheduling
– make scheduling choices observable and
controllable
• Search prioritization
– combating the combinatorial explosion of possible
schedules
Deterministic scheduling
Program
CHESS
While(not done){
TestScenario()
}
Tester Provides a Test Scenario
TestScenario(){
…
}
Win32 API
Kernel:
Threads, Scheduler,
Synchronization Objects
CHESS runs the scenario in a loop
• Each run is a different interleaving
• Each run is repeatable
Search prioritization (I)
• Given p ≥ 0, generate all schedules with up to
p preemptions
• Pseudo-polynomial number of schedules
– polynomial in preemption bound and schedule
points
– exponential in number of threads
• Many bugs with fewer than 2 preemptions
• Simple error traces for easier debugging
Search prioritization (II)
• Given p ≥ 0 and deterministic schedulers S0, …,
Sp-1, schedule according to S0, …, Sp-1 in
sequence moving from one to next
nondeterministically
– e.g., round-robin non-preemptive scheduling with
p different round-robin orders
• Polynomial number of schedules
• Testers can innovate by designing domainspecific deterministic schedulers
CHESS is available
• Used internally by Microsoft product groups
and externally by Microsoft customers
• Binary and source code available at:
– http://chesstool.codeplex.com
Limitations of CHESS
• Exposing and gaining control of scheduling
choices is difficult
– most implementation effort and user frustration
due to this problem
• Testing components that interact extensively
with the environment is difficult
• Input coverage is not addressed
Static program exploration with Poirot
• Symbolic instead of concrete execution
• C: Source code for software component
• E: Model for environment and scheduler
• Explore behaviors of C+E
– for all symbolic inputs
– for all scheduling choices
Demo: Asynchronous File I/O
Disk
Request queue
tail
In-memory cache
head
cache
cacheSize
AsyncRead(…) {
DiskReader(…) {
DiskReader(…) {
}
}
}
Poirot architecture
C  Boogie
Concurrent
C Program
Corral
.NET  Boogie
Concurrent
.NET Program
Trace Viewer
Concurrent
Boogie Program
Coverage Report
Searching with Corral
Abstraction
Concurrent
Boogie Program
Stratified
Search
Sequentialization
Error Trace
Sequential
Boogie Program
Concurrent
Boogie Program
Refinement
Coverage Report
Abstraction
• Set of global variables G
• Set of tracked variables T
• Drop writes to variables in G-T
• Replace reads to variables in G-T with
nondeterministic values
Searching with Corral
Abstraction
Concurrent
Boogie Program
Stratified
Search
Sequentialization
Error Trace
Sequential
Boogie Program
Concurrent
Boogie Program
Refinement
Coverage Report
Refinement
• Path p
– feasible if only variables in T are tracked
– infeasible if all variables in G are tracked
• Expand tracked set T to U such that p
infeasible while tracking only variables in U
• Naïve algorithm: linear scan of G-T
• New divide-and-conquer algorithm
– best case log(|G-T|)
– worst case 2*|G-T|
Searching with Corral
Abstraction
Concurrent
Boogie Program
Stratified
Search
Sequentialization
Error Trace
Sequential
Boogie Program
Concurrent
Boogie Program
Refinement
Coverage Report
Sequentialization (I)
• Given a concurrent program P, construct a
sequential program Q such that Q  P
• Drop each occurrence of async-call
• Convert each occurrence of async-call to call
Sequentialization (II)
• Given a concurrent program P, construct a
family of programs Qi such that
– Q0  Q1  Q2  …  P
– i Q i = P
• Even better if interesting behaviors of P
manifest in Qi for low values of i
Context-bounding
• Under-approximation parameterized by K ≥ 0
– executions in which each thread gets at most K
contexts to execute
• As K  , we get all behaviors
• Can we create sequentializations for contextbounding?
Sequentializing context switches
Shared Memory
Execution:
(s1, l1)
T1
T1
T2
Local Memory
Local Memory
T1
(s2, l2)
T2
T1
s2
l2
T2
T2
T1
T1
Guess and verify
(s1, l1)
T1
(s2, l2)
(s3, l2)
T1
Guess the effect of T2
T2
Verify the guess
• Make copies of global variables
• Source-to-source translation
– linear in program size and K
• Generalizes to dynamically-created threads
Searching with Corral
Abstraction
Concurrent
Boogie Program
Stratified
Search
Sequentialization
Error Trace
Sequential
Boogie Program
Concurrent
Boogie Program
Refinement
Coverage Report
Stratified search
Convert loops to recursive calls
Call tree given recursion bound r
assert no bug
main
T
L
assert
Summaries(L)
VC(p)
no bug
VC(T)
…
Poirot status
• Medium-sized C programs
– up to 20K low-level systems code
– reports precise traces at scale
• Small .NET programs
– bytecode to Boogie translator in progress
• Try: http://www.rise4fun.com/Poirot
• Download available
Why bounded search?
Data: Boolean, Integers, Arrays
Control: Sequencing, Choice, Iteration, Call, Async-Call
Sequencing
Choice
NP-complete
Sequencing
Choice
Iteration
+ bound
Call
Async-call
Sequencing
Choice
Iteration
Call
Async-call
Decidable
PSPACE-hard
Undecidable
Advances in SAT/SMT-solvers have made this problem tractable
Rationale: It is better to fail at the simpler problem!
HAVOC verifier deployed for security analysis in Windows/IE
Poirot collaborators
Akash Lal, MSR Bangalore
Shuvendu Lahiri, MSR Redmond
Questions