Efficient Symbolic Execution of Value-based Data
Structures for Critical SystemsDear Patrice
We regret to inform you that your paper
has not been accepted for FM 2011.
We hope that you will find the reviews valuable in helping you to improve you
paper for submission elsewhere.
We hope to see you at FM in Limerick.
Michael & Wolfram
---------------------------- REVIEW 1 -------------------------TITLE: Efficient Symbolic Execution of Value-based Data Structures
for Critical Systems
PAPER: 99
OVERALL RATING: 0 (borderline paper)
REVIEWER'S CONFIDENCE: 2 (medium)
The paper presents a new symbolic execution technique for Spark/Ada
programs.
The technique has been implemented and the paper provides a good number of
experiments.
A lot of work went into the research behind this paper.
[Comment]
Unfortunately, the presentation in the middle of the paper falls short and fails to
convey the main contribution of the paper. In particular, the crucial difference
between the logical and graph-based representation is not explained very
well.
DETAILED COMMENTS:
page 3: Yikes -> Yices
"SymExe does not merge state information" -> why? Is it too difficult/expensive?
page 4, assert/assume -> what is the difference? Figure on page 5 shows that
the difference is not raising an error in assume.
page 4, concrete & symbolic values: Isn't a concrete value simply a symbolic
value which represents a singleton set of values? I guess there is internally a
different representation, though?
page 5: what exactly is the AssertionError state?
I wonder why you do not separate expressions (no side-effect) and statements
(no values). This would make the presentation cleaner (assuming your
expressions really don't have side-effects).
I find the formalisation a bit more cumbersome than necessary. For the reader
the problem is shifted from understanding an intuitive core language and its
effect on a logical state description to understanding a much less intuitive syntax:
I think it would be better to provide a higher level-description and assume the
reader knows how one can encode state updates/.../etc and then focus on the
important aspects : precisely how states are represented, which logic do you
use,...
line -7: add/update and override: what is the difference? override is add/update ?
I guess you mean assignment instead of override.
page 7:
I find this page not very well explained. What are you trying to convey. It is
difficult to extract the core of your encoding and constraint solving procedure.
You basically seem to translate one notation into another one (aupdt,...) which is
not fully explained, and as such not much has been gained for the reader.
pages 8/9: Graph-Based Array Semantics: Again, I find the presentation hard to
follow. I wonder whether you could not try a more high-level/abstract
presentation, and focussing on the difference with the logical array semantics.
This is the core of your paper, and unfortunately not very clear. Can this not be
recast as an alternate decision procedure for arrays?
page 12/13: experiments. I think you should provide more explanations about the
experiments and their interpretation (a common problem in papers ;-)).
---------------------------- REVIEW 2 -------------------------TITLE: Efficient Symbolic Execution of Value-based Data Structures
for Critical Systems
OVERALL RATING: 0 (borderline paper)
REVIEWER'S CONFIDENCE: 3 (high)
Summary of contents
Symbolic execution tracks the symbolic values instead of concrete values
through the program, where one symbolic run can represent multiple executions
using concrete values. This paper mainly presents two approaches for reasoning
about complex data structures like arrays that are statically allocated and valuebased, aiming at the bounded verification of the functional contract.
The logic approach basically delegates the reasoning about array to existing
decision procedures for array theories. On the other hand, the Graph-based
approach directly decomposes the symbolic constraints on arrays into basic data
structures during execution; as a result, the decision procedures applied only
need to handle scalar values.
The authors formalized and implemented the two approaches within the SPARK
language and tool framework, and evaluated their performance on some actual
examples. The experiment results provided in the paper show that the graphbased approach offers better performance compared to using the array decision
procedures.
=================
Evaluation
First, the presentation of this paper is basically acceptable, except that the
description of formal semantic rules for the array operations by two different
approaches may be more complex than necessary, and make it hard to read.
More detailed comments follow.
Secondly, the two approaches are not significantly novel, while they are
targeted at the SPARK framework and bounded verification which may increase
their utility in the real critical embedded applications.
Third, the empirical evaluations are ok, but are not strongly convincing. The
example applications being verified are rather small. More discussions of
the relevant impacts on the experiment results besides the two approaches
might be better, such as the underlying DPs, the optimization techniques utilized.
Technical comments/questions:
1. The introduction section is basically ok. But the unique features of the graphbased approach are not clearly pointed out, the use of term "explicit-state
representation" can be confusing, not symbolic any more? Maybe using a small
example could better illustrate the uniqueness of the graph-based approach
compared to the logic approach.
2. In the background section, it is good that the advantages of bounding
mechanism in embedded applications are discussed. Although the basic
characteristics of symbolic execution are presented, it may better to use some
example that can also illustrate the different operations of the logic and graphbased approaches. They are the important concepts of this paper.
3. In the formalization section,
- The optimization presented at the middle of page 6 is essentially the constant
propagation method in compilation?
- Formal semantics are supposed to ensure the rigorousness, but not greatly
decrease the readability. The rules in Figure 3 and 4 may use unnecessary
complex symbols. For instance, it might be better that "(pi) ++ [<a,l>]" in Figure 3
can be rewritten in other clearer form, as such symbolic representation has not
been explained beforehand.
- The rules in Figure 4 are not concise, hard to read, especially many auxiliary
functions.
- On page 9, the description of three array lookup rules for the graph-based
approach are not easy to read, may be better if being revised. For example, the
case scenario of each rule shall be given definitely, the corresponding operations
had better be concise and justified. Moreover, these three rules are the basic
different points of this approach compared with the logic approach. Some
example may also help.
- On page 10, the helper function of the Compare procedure is presented. It
might be interesting to discuss more about the general purpose and usage of
helper functions in verification, as they are also used in the example applications
for evaluation in the next section.
4. In the evaluation section,
- According to the statistics of the application examples under verification, their
scale are rather small. It is not quite convincing that these examples are the
representative applications.
- The runtime of many instances are less than 1 second, small errors in actual
executions may cause great differences. So, the claimed significant performance
improvement by the graph-based approach may not be solidly supported.
- Several newer versions of Z3 have come out this year, it may be worthwhile to
try them.
- It is necessary to present clearly what optimization techniques are used for the
graph-based approach besides its basic procedure. Otherwise, readers may
doubt the justice of the comparisons. And why they are they not applicable or not
effective for the logic approach?
- On page 13, the paper discussed that the optimized approach gain more
knowledge for faster concrete execution. Are such knowledge the application
specific, or general? Why the underlying decision procedures cannot learn such
knowledge? Might the way of calling decision procedures (DP) in the logic
approach applied in this paper block DPs from learning such knowledge, but in
fact existing DPs are capable of such learning?
- In the paper "What's decidable about arrays?" by Aaron Bradley et al., they
could reason most sorting operations on arrays very efficiently and implemented
in CVC-lite. It may be interesting to refer their work as well.
- It might be very interesting to discuss further how to apply DPs efficiently, like
how to avoid their weakness and make full use of their advantages; or whether
the optimized techniques used in this paper can guide enhancing the
performance of existing DPs.
---------------------------- REVIEW 3 -------------------------TITLE: Efficient Symbolic Execution of Value-based Data Structures
for Critical Systems
PAPER: 99
OVERALL RATING: -2 (reject)
REVIEWER'S CONFIDENCE: 3 (high)
[Comments]
The paper proposes and evaluates the application of symbolic execution for
verifying SPARK/ADA programs. The operational semantics of a so-called core
of the SPARK language is presented, then two semantics for an extensions of
the core language with arrays are presented. The first one is straight forward,
relying on the functional representation of the arrays and SMT support for a
theory of arrays. The second one handles array manipulation directly in the
semantics, through a graph-based symbolic representation, thus reducing the
burden on the external decision procedures. The two representations are then
evaluated on several benchmarks, including code from industry code bases. The
evaluation shows that the graph-based representation is better than its logical
counterpart.
The text of the paper, though quite technical, is relatively easy to follow. I cannot
say the same about the figures, which are full of symbols and notations, which
are manageable at first, but become quite horrendous to read exactly in the most
interesting part (i.e., the graph-based operational semantics of arrays). I
understand that the authors wanted to fit everything on one page, but maybe
they abstracted it too much to do so.
In my opinion the first parts of the operational semantics (including the logicalbased semantics for arrays) are quite straightforward. Moreover, while the
authors say the language they give semantics to is a "SPARK/ADA-like core
language", it does not seem to me much more than a bytecode language, which
could be as much SPARK-like as any language. The space saved by reducing
them to a minimum could have been used to clarify the admittedly
interesting graph-based semantics
Regarding related work, since the authors' contribution falls into using symbolic
execution for program verification, I believe that they should compare with other
related approaches such as Dynamic Logic (e.g., the Key Project) and Matching
Logic who also rely on symbolic execution for program verification.
To summarize, I believe this paper has potential and interesting results but is
premature for publication.
Minor comments:
page 8: text if going beyond page limits.
page 9, last par.: "applies when i is materialized" => "applies when i is NOT
materialized" [PC: Good catch on the reviewer’s part]
page 10, 2nd par: "the state IS will be ignored"
Table 1, all sorting algorithms, row 7: it is very confusing, and I believe
unfortunate that you have used 0 to represent time being exceeded. The way it
looks at first sight is that the logical approach finished instantly, while the other
took over 200 seconds. Better put dashes, or even infinity, than 0.