>> Madan Musuvathi: It's my great pleasure to introduce Neha Rungta. She is a
PhD candidate from Brigham Young University, and she works with Eric Mercer.
And she's also worked a lot with [inaudible] who, you know, who was at NASA for
a while and now is is at Seven Networks.
Now, she is really interested in using guided search and randomization
techniques to improve -- to address the states base explosion problem that
shows up in model checking and concurrency testing. And she received the
Google Anita Borg Scholarship in 2007, and she developed a guided test
framework for the Java Pathfinder model checker as part of the Google Summer
of Code. So Neha.
>> Neha Rungta: Thank you, Madan. Thank you for the very kind introduction.
As Madan said, I am Neha Rungta, and today my talk is going to be on
abstraction guided symbolic execution for essentially testing concurrent
programs. And this is a joint work with my providers, Eric Merced and Willa Vicer
[phonetic], who is currently at Seven Networks.
So just to kind of provide introduction, the problem that we are trying to solve and
even in this contrived example it illustrates essentially the kind of errors or the
problems that we are trying to detect. So in this simple program, we have two
threads, thread A and B, that operate on the shared variable element and just try
to see initially the element has an integer variable that is initialized to one. And
what thread A does is acquires a lock on element, checks its value, and we see
what the function does, and unlocks the element.
All check is doing is saying if the variable that is contained within the element
object is greater than 9, then throw an exception. While thread B has a data
input variable, X, and if it's -- if its value is greater than 18, then it calls this series
of acquires a lock on element, resets its value, and unlocks. And all reset does is
sets it to 11, which is, you know, not much greater than 9 and kind of is the
condition.
So what are the conditions that will lead to this exception that we are trying to
raise? If you simply execute thread A that acquires the lock, checks the value,
unlocks the variable, then no exception because it is initialized to one and it's
never greater than 9. In another scenario, if we start executing thread B which
has a data input, it seeks the input to the value X and checks if that's greater than
18, and just before the lock if it contact switches and goes to running thread A
again, this exception still won't be raised. However, in the final scenario, if thread
B actually runs to completion where it says find the data input which is greater
than 18 which acquires the lock on element, it resets it and then unlocks it, and
finally if we run thread A you have our exception.
So among all these, even in this little contrived example among the different
scenarios it was a very specific, you know, data schedule, data variable and
thread schedule that led to us leading to this exception state. So the problem
that we are trying to address is how do we automate finding the error? In this
case, it was the exception. And it can translate to any concurrency error such as
deadlocks, race conditions or any other reachability properties.
There is a huge body of work in these areas, and this is a very small smattering
of it. It doesn't even do justice, but I had to fit it on one slide. So we have the
static analysis techniques where -- which try -- which report possible errors that
may exist in the program using more imprecise analysis in order to scale to larger
systems.
We have Jlint and FindBugs that do more for pattern matching things while
Spec# and ESP Java do a little more on the verification end and are a little more
precise than Jlint and FindBugs while Racer X and Chord sort of lie in between.
We have the model checking world which has very precise sound and reports all
possible errors, but unfortunately it didn't scale to large systems, we have our,
you know, Java Pathfinder, the slam toolkit, blast, spin, [inaudible] we have a
whole plethora of model checkers there.
Dynamic analysis kind of does the instrument code and monitor at runtime. It
may miss certain errors like contest race finder. I didn't know where to kind of
put chess so for now it's in dynamic analysis. It could very well go in, you know,
model checking because it's more systematic.
But and on the other end of the spectrum we have symbolic execution where we
have -- with substitute symbolic values for data variables and tries to detect
feasible paths through the program to find errors. It does have a high constraint
solving cost associated with it, which makes it kind of ineffective in concurrent
programs, but it's been shown to be very effective in sequential programs and
test generations.
So among these boxes, the one thing that if you see is missing is testing and my
initial slide was about testing, is because there isn't much on testing concurrent
programs. All the tests generation, unit testing, there is chess that does, you
know, some of the unit testing for concurrent programs, but in general, you know,
there isn't a huge literature for concurrency the way we find in sequential. If we
see sequential testing, we could possibly spend days and days that's just gone
into testing sequential programs.
So that is the paradigm that, you know, we are trying to go toward and find our -find solutions in that problem area. So sort of the systematic verification
technique kind of does this. You have a program state with -- in a concrete
where you have different threads, each thread has its own runtime stack, it's local
variables, what the current program location it is at. And then we have a set of
he -- we have a global variables in the heap which when we get into a runtime
environment, it systematically can give us all possible, possible successors of
that state. So something like even chess, chess is basically a modified runtime
environment which can systematically allow you to give the successors, Java
Pathfinder a modified Java virtual machine that gives you that control, even spin.
Now, when we add an additional like data input, we've just added to our program
state the data input variables and their symbolic representations. And also a
path constraint that represents the set of constraints over these data input
variables that need to be satisfied for a path to be feasible. So what are some of
the tools that allow such a kind of a program state?
Actually not too many. We have a symbolic extension of the Java Pathfinder that
lets us do that. Something maybe a combination of chess and packs possibly
could let us do a very similar thing. And then there is concurrent concolic testing
JQ tool developed by [inaudible] that does a similar kind of things.
So sort of in high level what it -- to systematically generate the successors given
a state, it will explore all possible thread schedules, so in this case if there are
three threads. For each thread it -- to check for data non determinism what it
checks is if I'm at a conditional branch statement that contains symbolic variables
of primitive data type, at that point there can be two possible successors. One
where we find assignments to the data input variables, where the true branch can
be executed, and, two, where we can find a set of assignments where the false
branch can be executed, the negation of the constraint can be executed.
Now, for complex data types, there are different techniques in literature that have
proposed how do you handle symbolic execution for complex data input
structures. The generalized symbolic execution technique does -- you know,
whenever you -- it's sort of in lazy initialization. When you reference an
unintialized data object, you can either -- it can either be null, it can be either an
instantiation of the new, new type, so any of its subtypes because you in terms of
dynamic method and location you really don't know which type it's going to be.
And also point to any of the existing subtypes that of same.
So if we had a Fu object of type T, at this point O may also point to that type T.
And if it's none of these two, where it's -- you can just generate a single
successor state where thread T0 executed a step, if it's not a conditional branch
or a data input but complex data types. Most common way of exploring this is
sort of in a depth first search where you systematically explore the states and the
transition graph along a single path. It's sort of created on the fly and until it hits
the end node in the transition graph and then backtracks, goes to the next state
in the search stack and traverses the next stack until we actually reach the target
location.
The problem is doing such an exhaustive search is very expensive, and you tend
to explore a lot of states. You may run out of time or memory before you even
find your target state and so even are randomization, like you can randomize
this, not go systematically, but still not be able to find. So what guided search
tries to do is, you know, start from our initial state and generate the successors of
the state and assign a heuristic rank to each of these states and the state with
the lowest heuristic rank expanded first. And we keep expanding it until it
reaches the target state.
So we think a heuristic estimate is efficient if it actually explores fewer states
before error discovery or before reaching the target state. That is what we define
our fewer states that at least exhaustive or random. So that's our lower bar of,
you know, performance major.
So in this work what we say is sort of use an abstraction guided symbolic
execution technique that has a set of target locations as the input to the
technique. And I'll get back to how we generate these target locations. From this
target locations we generate an abstract system of program location. A program
location is just uniquely identified by say the program counter or like a line of
code in your program. It has no data or thread information. And what we add
only the locations that determine the reachability of the target locations.
We actually do not perform any verification on the abstract system, which is sort
of the difference between this abstraction guided refinement and what we view in
the model community as abstraction guided refinement because generally the
verification is performed on the abstract system itself.
So we just use the abstract system to guide the symbolic execution, to guide the
state generation in the runtime environment. And we use heuristic to pick the
thread schedules and data, input data values and we keep refining the abstract
system when we cannot -- when we are stuck, where we cannot proceed further
execution and we'll give that with an example.
So this is just sort of very high level picture of the technique, where it says, okay,
you have a target location, a set of target -- it can be a set of target locations and
we look -- we compute some control and data dependence analysis to say, okay,
this target the control dependent on this conditional statement being true, and
there's two definitions of this variable A that actually reach this conditional
branch, not killed along the path to the conditional branch.
And that is the input to this guided symbolic execution technique which says,
okay, which of the threads am I going to drive? I'm going to look at this
information, use my heuristics to decide and also during the control flow things,
which branch do I need to go down on? And when I can go down a certain
branch, I will refine the abstract system by adding more locations into the
abstract system.
So how do we generate these input target locations? It can be as simple as a
user defined reachability property. Can I ever get to this statement where this
exception is raised? Because in a concurrent system there may be like three or
four threads or that operate on different variables that actually happen to
determine the reachability of this target location.
The other thing is static analysis tools give you a lot of possible interesting
locations. Like for example, the lock set analysis will give you cyclic lock
dependencies, where each of the threads need to be at a lock acquisition
location that may result in a deadlock. So there are two possible conditions
where can we even -- can the threads even reach those locations? They may be
getting locked and may not ever reach because the static analysis mostly ignores
the runtime environment of the program and it's more on the source code. So it's
checking the feasibility. Can two threads be at the critical section at the same
time? Things like that, a lot of the static analysis tools will sort of give possible
errors, and we can use those to generate a reachability property into. Yes?
>>: So when you say target location, is it a single PC or ->> Neha Rungta: It's a single PC. So it's a set of target locations means a set of
single PC locations, program locations. So ->>: But that [inaudible] what you need is a pair, right, you want two threads to be
->> Neha Rungta: So it will just be a set of two locations. So in a accepts we're
just not ordering them because -- by putting them in two places, by putting in a
set, it's just, you know, [inaudible].
>>: I'm just trying to say that because just between one -- either one of the PCs
[inaudible].
>> Neha Rungta: No, is goal is to have one thread at each target location in the
set. That's the goal, given the input. And that may actually, may or may not
result in the error, but right now that's the reachability goal that we are trying to
test for, because even they may all be there and still not -- it still may not
deadlock because it's maybe the aliasing on the objects is not right or can be a
set of different reasons why it still may not occur.
So like I said in our abstract system, we have a set of program locations. Now,
these program locations just are essentially a subset of the control locations that
you find in the control flow graphs of your system. It's basically a courser
abstraction of the control flow graph. And we use -- and we only keep the
locations that actually determine the reachability of the target locations and again
contain no data or thread information. Just a very quick instruction to control flow
graphs. A control flow graph has a start node end node, you know, as branch,
branch statements where it can go to two possible successors, call sites which
can invoke another procedure and, you know, goes -- has this call site edge. So
there are different kinds of locations in the abstract system. One is of course the
target locations, and all possible start locations of the program, where all can the
program start at? We add call sequences that lead from the start of the program
to the target locations and we have branch statements that determine the
reachability of these call sites and these target locations. Definitions of variables
that are in the branch predicates of the conditional statements that we just
added, and synchronization locations. And I know that's a lot. We'll just go
through each of them like individually.
So returning to our example, this through exception is our target location. So we
add that to our abstract system, and there are two possible start locations. One,
where thread A is running, and, two, where thread B is running. Start, start
locations of those.
And this is again just a single program location and has no thread information.
But it's just a start location in the program. So to add call sites and start location,
we go through sequences of call sites from beginning of the start of the program
that leads to the procedure that contains a target location and we add the call site
and the start location of the call E. So in this, we have function check that leads
from a start of the program to the function that contains this exception. So we
add those corresponding program locations. Here is the call and here is the start
corresponding to the call.
Then we add conditional statements. We compute a control dependence
analysis to see are there conditional statements that affect the affect the
reachability. And we do it as nested levels so there may be another conditional
dependence, controlled conditional statement that determines the reachability of
the next conditional statement.
And we say okay, in this case we have a conditional statement if element dot E is
greater than 9. The control dependent on this conditional statement being true,
hence we add that to our abstract system.
The other thing we add is there are variables in the branch predicate of the
conditional statements that we just added. What are the definitions of these
variables that reach the conditional statement? When I say reach, it's reaching
definitions and data dependence analysis where from that point there exists a
path where there's no redefinition of the variable along the path. And for alias
information we computed based on a, you know, maybe an alias, we do a
conservative estimate, and for now we restrict it in our initial creation of the
abstract system, we restrict the definition along intraprocedural path where it's
within a single procedure so we don't end up with a lot of paths in our abstract
system that we need to test that lead to the target location.
So in this example, we actually don't have a definition of element dot E within this
procedure so nothing is added and it's the same as before. Yes?
>>: [inaudible] in computing the ->> Neha Rungta: No, we don't assume that. So we can be looking for, you
know, there may be another data race, but you know, we don't really care if there
is or not.
>>: [inaudible].
>> Neha Rungta: Well, if there is -- so since the abstract system, it didn't really
look at thread interactions. So if just in the control flow graph, there is a path
along -- so another thread can kill the definition on a totally different path in your
control flow route. And that's valid. We are just looking for a single path through
the control flow system that says along that path is. So since we don't keep any
of the thread interactions between the things, so -- and there do exist those, so,
you know, and that's part of the refinement like, you know, maybe that other
thread needed to, you know, redefine that variable for us to really go to that
location. So sort of you do that automatically, though, as a, you know,
refinement procedure. When we can go down the conditional branch or can get
to our target location.
So finally we have the synchronization operation where if there is a lock acquired
at a -- along a path to the location that we just added in our abstract system and
the corresponding lock [inaudible] locations. And for now we're just looking at
explicit locking so we're like lock A or unlock A. So in this case, we've added
check, so along this path we acquire lock and so its corresponding thing is also
unlock. And so those are the final after we've generated our abstract system,
these are the locations in our abstract system.
Now, note, the one we generated thread B had nothing to do with it, at least in
our initial -- the way we defined our initial generation of our abstract system, so
there is no elements from thread B. And we add edges, and of course we -- you
know, keep generating these locations until we reach a fixed point and we are
guaranteed to terminate even in presence of loops or recursion because there's
no data or thread information. We just add a cycle in your abstract system and
you're done. And it's just unique program locations. So we are guaranteed to
terminate.
Then we add the edges between. And these edges are kind of just based on the
control flow edges. Like if there exists a path from A, just use that to basically
collapsing all the intermediate states that do not exist in the abstract system and
add an edge between one node to another is how we add the edges. So now the
input to the guided symbolic execution is actually a single trace that leads from
the start of the program one of the starts of the program, to a target location.
And in this case, there's only one pad that leads from the start to this target
location going from thread A start to the exception. This is the same abstract
trace.
Now, when there are multiple target locations like for like the deadlock, there may
be two -- there are two target locations so in that case we'll have an abstract
trace set where there exists an abstract trace from the start of the program to
each of the target locations. Now, since there is a, you know, branching and
there may be multiple abstract -- traces to a single target location, what we do is,
you know, generate them separately and attribute them on different computation
nodes because what we want to real test is the feasibility of one, is one of the
things reachable. So we just distribute it on different computation nodes and say
right now we're not checking for like completeness, but as soon as one finds the
target location or the tar goal state because all the threads are at their
corresponding target locations we can say yes or error trace or here's a trace to,
you know, the goal state.
So the intuition between what guided symbolic execution ->>: [inaudible].
>> Neha Rungta: So what we do when we have loops is kind of do not consider
the cycle. So to generate the finite path we actually just remove any back edges
in our abstract system. Now, that loop actually says there's a cyclic dependency
between different locations. So -- and our heuristics are actually smart enough to
keep doing -- going through the cyclic dependencies as many times as they need
to be to drive to the next location outside the cyclic dependencies. So we
actually can get out without having to, you know, figure out a bound as to how
many times to unroll the dependency and just while generating the abstract trace
remove any back edges in the abstract system.
Yes?
>>: [inaudible] certain number of [inaudible].
>> Neha Rungta: Well, it may be. Like if there is -- if your cyclic -- if you're data
dependency and control dependency is safe within the loop, right, it's a
conditional statement, so in your abstract system there will be a loop, like in this.
>>: [inaudible] loops, that's the answer.
>> Neha Rungta: Right I'll literally ->>: It's not sound by any means?
>> Neha Rungta: No. No. Yes.
>>: And also because you work on the extractable, you may have many more
roots that have the concrete levels so you have spurious roots.
>> Neha Rungta: Right. And the verification is not on the abstract system.
>>: Right. You are [inaudible] abstract traces so you can't have -- if you look at
the control flow graph of the program, you can have lots of loops->> Neha Rungta: Right.
>>: [inaudible] which actually may not be feasible.
>> Neha Rungta: Which may not be feasible. Or you know, even if ->>: That's what basically you were talking about ->> Neha Rungta: Right. Right. And, you know, even if say the concrete
execution needs to, you know, execute it three times, it's a certain loop before it
can get to the next locations in the abstract system, we have it like a heuristic
that keeps driving the thread to the next location in the abstract system. So
essentially it will keep, you know, driving the thread through the concrete loop
like in the concrete program behavior space, even though the loop is doesn't
exist in the abstract trace that I've given the symbolic execution that it's guiding
for.
>>: [inaudible] generate [inaudible].
>> Neha Rungta: So the traces is just to say drive each -- and I'll just come to
right back in. So the idea is so here's your abstract trace, L0 -- and for simplicity,
let's assume we have just one target location, in this case L1. So what we want
to create is a corresponding sequence of program states which have the thread
and data and all the information in our runtime environment we want to create
this corresponding sequence of program state that contains a location for each of
the abstract trace locations. So where -- and I'll give it a little more in detail. So
where the -- of course the size of the program state is much larger than your
abstract trace, and what it -- to abstract and what it means to map is here's your
program state. We just saw all the threads, local variables, data input variables,
everything. And we push it through an abstraction function and will give us a
single program location which matches the single program location is the current
program location of the most recently executed thread in your runtime
environment.
So when you're given a program state, it gives you a successor back and the
program location of your most recently executed thread is essentially the abstract
state, the abstract location, the abstract state for this entire huge program state.
And what we try to do is try -- use that abstraction function to map from my
concrete trace feasible execution trace for a better term to corresponding. So in
this case, S0 maps L0 while S2 maps to L1. That's what we're trying to do. Like
find these sequence of states within the sequence of program states in our
feasible execution path generated by the symbolic execution technique.
Now, there may be states that do not have any mapping to any program location
in the abstract trace. But that's fine because we are trying to fill in the locations
automatically between the two interesting points that we real care about. Does
that answer your question? Or are we using ->>: So [inaudible] is a concrete.
>> Neha Rungta: Yes, is a feasible execution.
>>: Is a concrete [inaudible].
>> Neha Rungta: Right.
>>: And you're using [inaudible] to guide ->> Neha Rungta: To guide. To generate this.
>>: [inaudible].
>> Neha Rungta: Right. Of course. Yes. So what we do is we use what is
called a greedy depth first search at each successor. We just look at its
immediate -- at each state, we just look at its immediate successors rank those
and explore the best possible successors and keep doing that until, you know,
we find our target state. And we don't consider backtracking states until we
reach the end of the path, which is a little different than traditional greedy best
first search, you know, where you look at the global best search because we are
trying to generate a path. That's why we kind of, you know, look at the best
successor E level.
So how do we real guide our search? So we have a sequence of locations. This
is our locations in our abstract trace. L1, L2, through LM. Now, we have a
program state as zero that generates three possible successors, S1, S2, and S3.
Now, what the line demonstrates is I have observed location L2 in program
states S1 and S2. Which, you know, there may be two different threads that
have, you know, executed and are at L2 at right now from the state S0. So we
prefer states that are -- that I've observed a location from our abstract state over
those that have not. So now in case well where we have two states that have
observed the same location, we can just randomly pick one and generate its
successors.
Now, in the case that they have not observed any more new locations from the
abstract trace, we have our work on distance estimate heuristics that computes a
estimate on the distance to the next location on the abstract trace and that is the
heuristic estimate. And I have a little more on how it's computed because it's a
lot of the previous work is on that and then the slides. And if you have time, you
know, I can possibly go over how that distance estimate is computed.
And we keep, you know, going and doing that until we find the target state. Now,
for refinement, if the -- if my location in my abstract state is a conditional
statement, FY, and I know I need to execute the true branch of the statement but
the successors that are generated, say -- say if Y is symbolic, there's no
satisfying assignment that will lead to the true branch, or also even if it's
concrete, you know, it's just not there, even in the single state that I've generated
is not the successor for the two branch, in fact, it's, you know, for the false
branch. At that point is when I refine. Going back to our early example, we have
this conditional branch where if element dot E is greater than 9, that is what
caused us to reach this exception, and say in our program execution we reach
this location where we find that it's not true, so at that point we refine, we add
variables, we first get the variables in the branch predicate that we are looking at.
And they may be global or thread local variables. And we perform an
intraprocedural dataflow analysis.
So and that will get things like there may be another thread that you now, can
access the definition that may also reach to this branch condition and now we
just propagate the alias information to different procedures and it is a little more
expensive analysis and so we wait until we say oh, we really can get to it.
So in this case, like I said, we see there is a definition of E at this location. So we
add that to our abstract system because we say this seems like an interesting
definition that I may need to go to before I check the value of this conditional
statement again. And I recompute the fix point. And for this little contrive
example it's pretty much all the program locations get added to the abstract
system that was there earlier, but in larger systems, you know, at least in our
experience we did not -- we found that even with refinements you don't end up
adding the entire control flow graph of your system, there's still a small subset of
the entire control flow graph of the system. Yes?
>>: [inaudible].
>> Neha Rungta: It can be anywhere between -- and of course it's dependent on
the program. Like for our Java concurrency errors the reduction was about 80 to
like 90 percent like 80 to 90 percent of program locations from your program.
And those only -- even after, you know, refinement you had anywhere between,
you know, ten to five percent. And it was a little program dependent of course
because of the dependencies. If you have a very disjoint system, you will -- you
will probably see a, you know, greater reduction. So in terms of, you know, how
the variables interact, that does have a little end to that.
>>: Just like before [inaudible] you wouldn't have known that -- essentially we
have an abstract [inaudible] and then we do a exploration to see if we can find ->> Neha Rungta: Through that trace.
>>: Through that trace. And so you have to complete the [inaudible] before you
start the refinement.
>> Neha Rungta: No. We do it sort of eager refinement what we call it.
>>: [inaudible] maybe you don't need a refinement at all.
>> Neha Rungta: And that's possible. And you know, we have those other
traces that the other computation nodes are chugging on. And there may be a
path, you know, that's gone through another definition that leads to this
conditional statement already there in our initial set. And at that point, you know,
before this can reach the refine that says oh, by the way, I found this target
location so everybody else can stop. So the sort of distributed. So one node is
really targeted towards finding a path, towards finding a feasible path along that
one particular abstract trace and doesn't sort of backtrack along the abstract
system to find another path to the target location.
>>: So [inaudible].
>> Neha Rungta: Actually, no, it's sort of an eager refinement where whenever I
reach -- whenever my program execution reaches a corresponding conditional
statement and the abstract trace, which it cannot guide along the right successor,
like true or false. At that point refine. Try to redefine one of the variables in the
branch predicate and try to execute that branch again. So it's sort of a very
eager refinement. And sort of, you know, this is a, you know, first pass at a
refinement. There's obviously a lot more precisely that we can do the refinement
and add more precision, even the abstract. So this is a, you know ->>: [inaudible] in this example they might have a loop in your abstract trace that
you pruned that we've gone around and looked [inaudible] wouldn't have known.
>> Neha Rungta: Right. I wouldn't have known that. And sort of the additional -there's a definition that loop that will get added to the abstract trace, you know,
on the refinement. So because it says oh, you know here's a redefinition that
might lead you again. So.
>>: [inaudible] refinement this is -- what's the difference with the predicate
abstraction and the refinement by adding predicates, it's essentially the same
thing or there are major differences here?
>> Neha Rungta: Well, of course there's no predicates, you know, no -- it's not
on date of values.
>>: So when you [inaudible] variables and you ->> Neha Rungta: No, we don't add variables. Only locations.
>>: So ->> Neha Rungta: So a single -- so this E dot 11 ->>: [inaudible].
>> Neha Rungta: It's only on the control flow. No variables. Because there is no
verification ->>: Okay. Now, if you add for instance let's say I have a point of path from here
to here and I add basically some point for location where there are conditional
statements. So what happened to variables that are testing those conditions?
>> Neha Rungta: We don't add those. We don't add like the variables. So in
this case where we've added these equal to 11, this is a program state. Like this
doesn't actually contain the value of E. It's just a program location where it says,
you know, in -- where E is equal to 11. So I guess the biggest differences, we
don't actually verify the abstract system based on what the predicate abstraction
does and the abstract system has no data values at all. And neither does the
refinement add any data values per se. It's only program locations that are
added.
>>: [inaudible] what you're doing as [inaudible] progressive slicing, static slicing.
So for example, if you did all refinements in [inaudible] you end up with a static
slice of the program.
>> Neha Rungta: Right.
>>: And [inaudible].
>> Neha Rungta: Yes. I think that's a good comparison.
>>: So [inaudible].
>> Neha Rungta: And slides for maybe, you know, a concurrent program and
use -- and again, the -- and if my understanding is right, we're not, you know,
executing that on that slice or on that abstraction, on this abstraction. It's just
sort of a set of interesting points. It's almost like you have a fingerprint and you
don't need to store the entire fingerprint to identify whose fingerprint there is.
You have this seven or eight key locations on your fingerprint that is good
enough to identify, you know, which ->>: [inaudible] so you start with an abstract control flow graph and you refine.
>> Neha Rungta: Right.
>>: But for the [inaudible] execution itself, what variables are you -- so there you
don't have a refinement step, it's a single -- you have a single execution is like an
extract interpretation.
>> Neha Rungta: Right.
>>: So one single abstract domain.
>> Neha Rungta: Right.
>>: Fixed for the whole thing?
>> Neha Rungta: Right.
>>: Got it.
>> Neha Rungta: We don't refine on the symbolic execution, no. So after adding
that definition to that abstract system, what we do is update the abstract trace.
And right now we just -- again, there may be many paths to different definitions or
to the -- to variables in the branch predicate that are getting redefined. And you
know, we have to check for lock dependencies between those definitions to see
whether or not we can get to it. So in this case, we have another path to the
definition of E 11. Now, this goes through thread B which goes if X is greater
than 18 lock reset and you know this is the instructions and the reset.
And we checked for log dependencies. Here's why. So if in the refined abstract
trace to the data definition, if it acquires a lock on the same object as the lock -as if there is a lock acquired by, you know, our initial trace to the target location,
we have to add an additional unlock, find the corresponding unlock for this and
add it after the data definition because if we do drive or thread right to right here
and then try to go execute this thread, it will get blocked at this lock because
another thread has this same lock. So we kind of make that check and add that
unlock element. And sort of this is our new refined abstract trace. What it does
is, you know, we have the definition. I did my unlock and now go reexecute this
original abstract trace.
And -- sorry.
>>: [inaudible].
>> Neha Rungta: So in our original -- since I'm right now restricting to explicit
locking for each corresponding lock that I've the abstract system, I've also added
the unlock for that.
>>: [inaudible]. And you don't know which [inaudible].
>> Neha Rungta: Well, in the explicit lock you know you do kind of know which it
is -- which object it's acquired a lock on. And the corresponding unlock for that,
you can map statically.
>>: [inaudible].
>> Neha Rungta: Because you have synchronized locks. Like in Java.
>>: Right. Okay.
>> Neha Rungta: Right. So you have the synchronized lock and that the block
ending for that is for the one that is acquired. And then we sort of restart our
symbolic execution back from S0. And so sort of an eager refinement technique.
So just to sort of a quick overview of the heuristic that we use, and a lot of this
heuristic is actually not part of this work that I just presented but have been part
of like a lot of the past research that I've done in. So some of the heuristics, so
how do we rank? So at the very first level we check at here's a program state SI.
A long a path from how many location has it observed. So the heuristic one for
SI is N, which is the length of the abstract trace minus the number of locations
that it has observed. Along a path from S0 through SI. So if it is already mapped
to -- if there's a -- if there exists a state that has S1 that matches L0, which
means I've observed one location from my abstract trace, so my heuristic value is
N minus 1. N -- a lower heuristic value is better because you explore something
that has a lower heuristic value and chances of going to an error faster.
At the second level, we estimate -- you know, we map the current state to its
corresponding abstract state, and again, this is like our -- a lot of our previous
work. And compute it on a control flow representation. So we have LI, which is
the abstraction for the SI, and this is the entire control flow system of the
program. And compute an estimate along, you know, by minimizing the number
of edges in the control flow graph that are required to reach the target location.
So in this case, this is two. And finally, we also rank data non determinism where
when we are initializing a symbolic variable of a complex data type, so where you
have, you know, null, near T, all subtypes of T and existing objects, at that point
we look at our control and our abstract trace. So we have O calling Fu and in the
class X dot Fu. So we really need -- we really don't need to bother about the one
with just other ones that are not of the same class because we know we need to
go precisely into that class to get to our target location. So that allows us to like
intelligently take and prune paths so we can say, okay, here's a new T, that is,
you know, we'll assign that as zero and all the others as one.
So sort of the information in our abstract trace essentially allows this, even, you
know, heuristic which is, you know, kind of obvious, thank you. Of course you'd
want to pick the classes that you know you want to go to, but the information in
the abstract trace allows us to do that in an exhaustive search you really don't
know where you're kind of going without some help.
So as far as experimental results we use the symbolic extension of Java
Pathfinder for those like it is a modified JVM that operates on Java byte code.
We did turn on dynamic parser reduction. The abstraction refinement heuristic
computation, everything is performed on Java byte code. And of course JPF
operates on Java byte code. And the libraries are all -- since everything is, you
know, byte code, any library that it uses also become part of the multi threaded
system so we don't have to worry about, oh, you know what if you don't have
source for third party libraries.
We have a set of benchmarks, and we have a set of examples from the JDK 14
concurrent library. And we do create a very generic environment around these
libraries to be able to execute code at these libraries. Let's see. So some of the
results is in -- and for all these examples are for there are certain like dynamic
analysis tools like con tests, even after running for few hours, few trials they're
not able to find like the error state or even exhaustive search within JPF, where
you just turn on that first search with a bound is not able to find.
So we picked at least that's our threshold where you cannot find the error with an
exhaustive search technique within JPF or that first search.
>>: [inaudible].
>> Neha Rungta: No, contest is not. So what we did is actually close the
environment, added the data like the correct data that actually needs to be -- I -and even then, even at -- it wasn't even able to find simply the thread schedule
that was needed to find the concurrency errors. Because -- and what we've done
is kind of in order to mimic a little more of the real -- mimic. It's not of course the
real word, but we have like say 20 threads running. But it's just the interaction of
two threads that actually need to get to the, you know, error state. So when you
run it in a dynamic with no control like with no just sort of adding interpretations
like contest does to the runtime environment, it's doing that to all the threads. It's
not using any -- I mean, it's not doing it to any sort of way to say what is a better
way to do it. In that sense. So that's why it didn't -- so those are the models. It's
like if you make a simple enough model with just two threads doing 10
operations, you know, most tools, techniques will be able to final that error. But
when you have a lot of interactions going on, you know, a lot of threads, even
operating in a small system with say hundred lines of source code, the behavior
space becomes so large that you know without something a little more, you
know, guided sort of or, you know, with information it didn't fall.
>>: [inaudible].
>> Neha Rungta: So this is like actually the number of threads, total number of
threads, the parameters is actually the total number of threads in the system.
>>: [inaudible].
>> Neha Rungta: No. It's like 9 plus one. So it's like 10 threads. So each
number represents the number of symmetric threads. So 9 symmetric threads
and there's another symmetric thread and things like that. And these have two
threads, you know. We can add more threads. And that makes, you know,
exhaustive and contest even worse, so -- but even with two threads, you know,
and given -- even given the right data input. So any dynamic analysis we did
was with, you know, closing the environment in respect to data input.
And so like originally there were we said I had said that, you know, there -- and
this is a fairly good reduction, you know. You have -- you just look at this is the
total running time, the total time taken to generate the abstract system that, the
time taken to do state generation, the time taken to the refinement. So this is
total time. And to like wall clock time, end to end. And so but for what we do is
we had when we said, you know, there can be multiple abstract traces to the
target location, we do distribute them on a large number of nodes and as soon as
one says oh, found one, we terminate the others. And to progressively -- and
even then you can still have a large number of abstract traces.
So to pick our initial state what we'll do is restrict the number of call sites. So
along the shortest number of call sites, let's start with that. So, you know, we can
kind of intelligently even pick -- let's -- and if you don't find it, we can keep moving
on.
So here's the total trace length for our -- so you see, it's not that much and -yes?
>>: [inaudible].
>> Neha Rungta: Yes. For one, yes.
>>: So how many nodes did you [inaudible].
>> Neha Rungta: So did I report it here? So since we like pick initial -- I've not
reported here. But what we do is pick initially abstract traces. We restrict it to
the shortest path, paths that contain the smallest number of call sites from the
start to the target locations. So we ended up running between 5 and 17 for our
set. And we found our errors within those, like initial set. And so like the
smallest example had 5. So there were five places that had the shortest number
of call sites that led from the start to the target. And for you know at least our
benchmarks we found within the shortest number of call sites. But we can
progressively keep adding more call sites to it. So the biggest one I think was
17.
>>: So in a sense of you have to do the target locations. So you knew there was
an error and you know ->> Neha Rungta: Okay. So I had to give the target locations and in the sense
where say Jlint told me there is a possible deadlock between these two things.
So I said, okay, you know, find those, you know, give it to my sort of static
intermediate thing that actually finds the corresponding source in the program
and, you know, spits it out as, you know, a set of target locations. And so I don't
know whether they're reachable or not.
And, well, in these benchmarks, they are. And so one of the future work things is
so what can we learn when they're not. And we're doing so much work, I mean,
and it's very feasible that it's not reachable. Can we like rank sort of the
feasibility or rank what some measure. Can we assign like some number, like
oh, we think there is a .9 probability that there's a really, you know, feasible but,
you know, there's this pesky little branch condition that, you know, I can go down.
I don't know, can the user give me any help or, you me, at least give more
information to the user about what its feasibility can be rather than saying, you
know, I don't know. Right now it just says I don't know if it didn't find something
that's feasible.
And this is the number of trace lens. And this is the number of refinements it did.
And in the number of refinements it says you so there's just like one or two
refinements that it did kind of found the error, which is as I was talking about
them this is quite similar to their observation in the iterative context bounded
thing where you don't need a lot of things to happen, like in terms of where it
needs to context switch for you to find the concurrency errors. So one of the
things that we want to do is evaluate against, you know, iterative mounded
context to see if you have a large enough program even, you know, can it -- can
this technique help, even when, you know, you know, when it's not trying to do
coverage like iterative bounded context does. Yes?
>>: [inaudible] how you measure the length of the trace?
>> Neha Rungta: The number of nodes in the trace is sort of the -- the number
of -- yeah. Abstract nodes.
>>: [inaudible] control nodes.
>> Neha Rungta: Control nodes, yes. In a given trace. So of course future work
->>: [inaudible] you know, I don't remember that on top my head. It's part of the
JPF symbolic execution engine. But I can look it up. Okay. So of course
compare feature work compared with iterative bounded context. The abstraction
and refinement right now is actually at the core heuristic base things. We are
looking into traces work with compositional symbolic execution to see if we can,
you know, harness some of these techniques to better abstract, better refine.
Can we do test case generation using the abstract model? But, you know, the
hard part is what would we run those tests. There is, I don't know, run those
tests with. But that's definitely very, you know, avenue for feature work.
The other thing is how do we rank likelihood of reaching a target location when
the paths of target are not found in that execution techniques. And of course we
want to look at richer synchronization constructs because right now we're just
looking at synchronization blocks, blocking so which you know here's an
acquired. But what happens in terms of weights and sleeps and notifies in C#
that's pulse [inaudible]. So that's it.
[applause].
>> Neha Rungta: Yes?
>>: [inaudible] I guess you know what synergy unable.
>> Neha Rungta: Right. Yeah.
>>: [inaudible] abstraction because it's [inaudible] abstraction [inaudible]
basically I believe since it's a form of -- I mean, it's very general of course, pretty
good abstraction but you add predicates that say control location that's a
[inaudible] class of predicates.
>> Neha Rungta: Right.
>>: Unless something's actually the way it works.
>> Neha Rungta: [inaudible].
>>: Then that's the way they do their abstraction refinement in -- I mean.
>> Neha Rungta: Right.
>>: Or synergy [inaudible] but ->> Neha Rungta: The thing, the difference in -- and I'm not sure is how those
techniques would kind of work in concurrent performance like sort of ->>: But here, I mean this [inaudible].
>> Neha Rungta: Right.
>>: It also [inaudible].
>> Neha Rungta: Right.
>>: But if you just look at sequential programs, forget about [inaudible].
>> Neha Rungta: Right.
>>: But look at what you do, then presenting that would be also a [inaudible] if
you present that to people that work on synergy.
>> Neha Rungta: Right.
>>: So any way you slice it, you put away concurrency. But you can't --
>> Neha Rungta: Yes. But of course there's nothing.
>>: So sometimes there is ->> Neha Rungta: Well, and I guess part of the thing is, you know, and maybe it's
like this kind of component. Well, okay, so my thing was, okay, there's a lot of
good things for sequential programs. I mean, you know, there's ->>: A lot of stuff.
>> Neha Rungta: There's a lot of good things for sequential programs. Like
predicate abstraction like CGar works great for sequential programs. And so I'm
like, okay, then but CGar doesn't quite work as well for concurrency. At least
from the people I've talked to I've kind of, you know, expressed that opinion,
okay.
>>: [inaudible] also didn't work always, now, [inaudible] I mean if you cannot final
basically an abstraction quickly.
>> Neha Rungta: Right.
>>: That's a -- less than a higher predicate then forget it, I mean, it's never going
to finish.
>> Neha Rungta: I think that's ->>: So now if you want to check very abstract properties, very guided and they
exist, easy to find in abstraction, they will find it. If you have like a million line
program and you checked like [inaudible] forget it. I mean, this never going to -these techniques can [inaudible].
>> Neha Rungta: Right. And I think part of that is even with my technique. You
know, I've tested all the things that, you know, there exists this path to these, you
know, target locations but what if it didn't or, you know, if it's very hard or like I -- I
haven't looked into, you know, what are the kind of things where there exists but
this technique is not able to find, like [inaudible] with naming with richer
synchronization things that might be [inaudible] of really control which -- where
the lock is and who's having the lock box, things like that.
>>: So can you go back to the example you had in your first line.
>> Neha Rungta: Okay.
>>: So one thing I wanted to -- always wanted to know was, like if you look at the
example, you need two things to happen for them both to happen. But you
actually need three things. So first somebody has to [inaudible] then you have to
run thread A and thread B. Right? That is -- and then second, somebody has to
tell you that thread B has to run with an input that you [inaudible] and third you
need the actually [inaudible]. Now, I'm actually wondering how important or how
likely do you see the case that the input values are [inaudible].
>> Neha Rungta: Okay. So I actually have an example with the JDK library.
Sort of a real, more real world example that's, you know, where it matters. So
now going -- so this is just a container classes the hash table and you know, the
vector classes, the GEK library. And your [inaudible] and if you run sort of a
static lop side [inaudible] it will say this may cause a [inaudible] if there's a
[inaudible] that is acquired a lock that's right here and there is a thread that's
acquired a lock right here. That's the only information your static analysis can
give you. Okay. So there is -- it may cause deadlock.
So and then the question becomes does the deadlock really exist? So if you
have say two threads, you better operate you know then have access to this low
variable hash table H and a variable back to variable B when it -- and you know,
they both executed their first line of code, where they have locks on their
respective objects because it's, you know, synchronized and just acquired the
lock on itself. When a thread zero will try and say, oh, let something that's
contained within the hash table, let me get its hash because that will [inaudible]
that hash function is returning.
And if and only if your hash table contains the vector B that's being operated by
another sort of a circular data dependency between containers, it will say thread
is locked because it's trying to acquire a lock that's already held by thread T1,
which is saying when -- what thread T1 is saying is let me call the equals
because I'm checking whether it's an index off. And what index off -- what the
equals function is, it says oh, you know strike 0 icon acquired because I'm
blocked you know the thread 0 has the lock on 8, which is -- so there's this
circular data dependency between the containers.
And also they both have to check after acquiring the lock on the respective
things. Because you know, if one kind of just finished to completion and the
other didn't, it was, you know, that would [inaudible] and we would not have
been, you know, [inaudible]. So this kind of error is like right now I have studied
more just a library code. Specially true in library code where say the designers
and developers to think of certain input. And we can go back and kind of read
documentation, are they even saying that don't use it this way? And want there.
So we're like, okay, to assume that this is a real error given the day he came and
-- or at least in that usage like at least we can say oh, by the way, don't use
circular, you know, data containers or, you know, do your own checking if you're
even doing it. Even if you're -- you know, don't rely on the locks that -- because
these libraries are trying to save, so in that sense we should not have the
deadlock this action gives us. And there are similar other examples like you
[inaudible] from [inaudible] that result from both data and thread.
So I don't -- there's of course errors, concurrence errors, concurrency errors that
are just caused by the transmittals or they're just caused by data values like you
know, given that specific data about any thread schedule can lead to the error.
And [inaudible] also like we're really interested in this problem it's both data and
thread that actually lead to the error.
>>: When you perform [inaudible] programs are in smaller, have you had -- have
you tried -- because they're smaller [inaudible] try exhaustive search?
>> Neha Rungta: An exhaustive doesn't work.
>>: Not even on these.
>> Neha Rungta: No.
>>: [inaudible].
>> Neha Rungta: Mid 20 threads just blows up in your face even. So because
it's not just the source lies, it's [inaudible].
>>: Can you glue the 20 threads to ->> Neha Rungta: No, it's kind.
>>: [inaudible].
>> Neha Rungta: No. No. For the error you just need the two threads. That's
the 20 threads are to sort of say this is a very course mimic time to mimic the real
world where, you know, if you extract the -- from any real world system, the
really, the things that cause the error and run along, you know, create a model of
that and run the model checker and we'll find error. The goal is to try to go to
those, you know, programs where they do have those, you know, 20 or 50
threads in a [inaudible] running. But only two really leading to the error. Where it
can be like really just guide and test those two to the error instead of, you know,
just systematic. Because that's very systematic. And that's where we'd like to go
is actually get some real world programs like that, instead of just doing it -- you
can say here's a real world program that actually runs through the threads and,
you know, [inaudible]. So that's our end goal.
>>: So one of the biggest problems that we have is to find a [inaudible] for the
second [inaudible] do you have a stress test that crashes, never crashed in the
past three months, crashes now. So you actually have this [inaudible] so you
know actually what the [inaudible] and now I'm going to give you the same
program, the same inputs, no. And you know, your probably was like trying to
figure out how to reach that [inaudible] that's an important ->> Neha Rungta: So you're saying even given these [inaudible] it's hard to
recreate? Given a stack to examine. [inaudible] well, if, there is a lot of ifs in
here because the biggest problem is [inaudible] sort of closing, you know, trying
to get that program running into like JPF.
>> Madan Musuvathi: Thank you.
[applause]