The ’Toccata’ project team creation: an
evaluation
Author’s name undisclosed
Abstract
This document is an evaluation of the proposal for the creation of the ’Toccata’
INRIA project-team lead by Claude Marché, based on the ”Toccata Project
Formally Verified Programs, Certified Tools and Numerical Computations” document.
1
Interest of the project
The project is oriented toward three clearly identified goals: certified programs, certified numerical programs and certified tools. Up to now, only the
hardware industry, and the regulated critical software industry were using
formal methods to develop their IPs and software.
These methods are not easily available to mainstream industrial usage: difficult to use, difficult to maintain, and in the non-critical software domain,
the most sophisticated tools used by the engineering staff are static analyzers
based on abstract interpretation.
The project has chosen an original approach based on deductive verification,
using both automated and interactive theorems provers, that is amenable to
an industrial usage.
Thus, all the project’s goals have important scientific, industrial and economic interest.
1
2 Reachability of the announced goals
2
2
Reachability of the announced goals
The project relies on the previous ’Proval’ project extensive experience in
the areas of certified programs and addresses the problem of certifying tools
themselves.
Certifying tools seems a large undertaking: a key enabler to use them in
an industrial context, but probably also an important source of engineering
workload, that may have been underestimated in the staffing.
On the other hand, this may be an important source of feedback and progress,
comparable to the one that can be obtained when bootstrapping a compiler.
3
Adequacy of the resources with the objectives
The former scientific and software production of the permanent members of
the team is quite important, and I have no doubt about the team ability to
produce what is announced.
I also think that this research domain is quite attractive to PhD students
and post-doctorates.
4
Position of the project-team within the national and
international community
The research environment is well described, and as far as I know, almost
comprehensive.
The originality of the positioning of the project team to use both automated
and interactive theorem proving makes for quite a unique positioning in the
software verification community.
The only point I would mention is that I was surprised not to read about the
Agda community as being part of the identified competitors to the ’Toccata’
project.
5 Proposed software development
5
3
Proposed software development
The proposal enumerates an impressive set of legacy software, and suggests
a few new developments. The roles of the various software and components
is clearly articulated, thanks also to ’Figure 1’.
However the following points should be better addressed:
• It was unclear to me from the description to understand what exactly
is the ’design a language and environment for both programming and
proving.’ (3.1.3): it is not enough specified.
• I don’t understand clearly the goal in 3.2 ’promote our future Why3
environment so that others could develop certified tools’: it is not
articulated enough, and possibly overambitious.
• I am puzzled by the porting the Gappa kernel to Coq or Why3: it looks
to me that the team is supporting Why3 over Coq, so that Why3 is a
more natural choice in the project team context. Then it seems to me
that there is a long way to reach this goal, since it probably requires
the development of many non-trivial sub-components to support the description of bit-level computations, fixed-point computations, floatingpoint computations as described by the IEEE754 standard.
I could not clearly establish the allocation of the resources to the developments, but my first general impression is that the team as it is described is
under-staffed to support the large diversity of already existing software.
There are a few risks with regard to the ownership of some components, that
seem developed and maintained by only one responsible (for instance, Gappa
is such a tool).
But an interesting point that I noticed very recently is the support of ’AltErgo’ by the OcamlPro company, which is a good signal for the industry to
use this tool.
6
Perspectives of application and industrial transfer
The industrial transfer of some of the tools presented in the critical software industry seems already an on-going activity: in this strongly regulated
area the industrial understand well the concept of ’certified programs’ and
’certified tools’.
7 Cooperations
4
A more important challenge is to bring these certification tools in a less
demanding industry, but in critical parts of the components they are using:
for instance instability in device drivers has led Microsoft to take serious
actions with regard to their driver certification procedure using a modelchecking approach: can a deductive verification approach be competitive
with this ? Can such an approach be applied to open-source software drivers
?
Frama-C plugins seems the good entry point for this kind of usage, though
in the mid-term, Frama-C will suffer from the lack of C++ support: some
industrial kernel-level components are indeed written in C++.
Another comment is that though C/C++ languages are not well suited for
developing formal certification tools, their usage is dominant in the embedded
software industry, where it seems unrealistic to displace them: to have an
impact, supporting both of these languages is mandatory.
Finally, the documents seems to suggest that there is interest for the methods described in the ’mobile phone’ industry, which I believe is not precise
enough: I would understand that there is interest for formal methods for the
’trusted boots’ or ’secure environment domains’, but this should be clarified.
7
Cooperations
The situation of the project with regard to national and international cooperations as it is described is quite satisfactory.
One point unclear to me in the document, at (4.3.4), was to understand if the
collaborations enumerated here were no more active at all, or were continued
in the absence of a formal contract.
For instance I would be surprised if the collaborations with the INRIA Aric
(Lyon) or Caramel (Nancy) team were discontinued, given the interest of the
’Toccata’ project for numerical tools, and its fundamental production in the
domain ( Flocq library,...).
I would also have expected some interactions with other teams specializing
in program verification through other static analysis methods such as David
Monniaux’s ’Stator’ project, or teams more oriented towards floating-point
computations analysis such as Matthieu Martel’s ’Dali’ project.
8 Project proposal document
8
5
Project proposal document
The project proposal is solid and well-written, it addresses all the aspects
that I needed to write this evaluation and gives appropriate references.
9
Conclusion
I strongly recommend accepting this proposal for the creation of the ’Toccata’ INRIA project-team as the continuation of the Proval project-team.
It is well-defined and solidly staffed with very competent researchers having
a recognized international track record, that have demonstrated their ability
to have an impact on both theoretical aspects of their field at the highest international level, and produce concrete software tools, usable outside
academia.
I expect its scientific outcome to be significant, and that it will help making
the deductive verification approach usable in the industry, which seems so
far to prefer static analysis tools based on abstract interpretation.
It would be instructive to revisit the various strengths and weaknesses of
state of the art techniques based on abstract interpretation, model-checking
and deductive verification at the end of the project.
A
Suggestions
I would suggest that the following points could be addressed:
• Certification of some elementary mathematical functions software implementation as they are defined in the IEEE754-2008 standard (Certified Components, Numerical Programs).
• Certification of parallel programs under various memory models, specifically lock-free algorithms under weak memory models that are more
and more common in low-power SMP architectures (Extra Exploratory
Axes of Research, Concurrent Programming).
• More work on the certification of machine-level code or intermediate
representations (.NET CLR of llvm bitcode): there is always a tension
between proofs of programs written in high level languages and trust
in the compiler that will translate to machine code. The Compcert
A Suggestions
6
project is doing a great job to provide certified compilers, but floatingpoint support was missing until very recently. Thus, WYSINWYX, as
coined by Thomas Reps, can still be a problem when such a compiler
is not available.
I would suggest that the following risks could be addressed:
• Usability of the tools: slow tools are not well accepted in the industry
and difficult to integrate in modern software engineering context (short
automatic continuous integration loops, for instance). It would be nice
to design tools whose performance scales well when used on large SMP
commodity machines.
I would suggest that the following impact points could be improved:
• Impact on programming languages standardization committees (for instance C++14): some features of the libraries and language could be a
nuisance or bring important benefit in the ability to verify some classes
of program. Also, for instance the ’C++11’ memory model discussions
have been largely supported by formal verification techniques, and this
experience has stimulated research in this area.
• Impact on numerical standards committees (for instance the IEEE Interval Standard Working Group - P1788). A tool such as Gappa most
probably implements some forms of interval arithmetic and the committee could benefit from this experience, conversely the definition of
the operations’ semantic could be useful to check to what degree they
are amenable to formal proofs, and their semantic usable in future verification tools.
I would suggest that the following communication points could be improved:
• Communication and promotion of the developed techniques and tools,
for instance by participating in INRIA seminars such as the ”Validation
formelle de systèmes industriels critiques”.
I am conscious that these suggestions are most probably too numerous and
possibly not exactly in the project’s scope or even not realistic, but I provide
them in case they could inspire something useful for the project or future
instances of the project.