Benefits of Bounded Model
Checking at an Industrial Setting
intel
F.Copty, L. Fix, R.Fraer, E.Giunchiglia*, G. Kamhi,
A.Tacchella*, M.Y.Vardi**
Intel Corp., Haifa, Israel
*Università di Genova, Genova, Italy
**Rice University, Houston (TX), USA
Technical framework

Symbolic Model Checking (MC)
• Over 10 years of successful application in formal
verification of hardware and protocols
• Traditionally based on reduced ordered Binary Decision
Diagrams (BDDs)

Symbolic Bounded Model Checking (BMC)
• Introduced recently, but shown to be extremely effective
for falsification (bug hunting)
• Based on propositional satisfiability (SAT) solvers
Open points

Why is BMC effective?
• Because the search is bounded, and/or...
• ...because it uses SAT solvers instead of BDDs?

What is the impact of BMC on industrial-size
verification test-cases?
• Traditional measures: performance and capacity
• A new perspective: productivity
Our contribution

Apples-to-apples comparison
• Expert’s tuning both on BDDs and SAT sides
 optimal setting for SAT by tuning search heuristics
• BDD-based BMC vs. SAT-based BMC
 using SAT (rather than bounding) is a win

A new perspective of BMC on industrial test-cases
• BMC performance and capacity
 SAT capacity reaches far beyond BDDs
• SAT-based BMC productivity
 greater capacity + optimal setting = productivity boost
Agenda

BMC techniques
• Implementing BDD-based BMC
• SAT-based BMC: algorithm, solver and strategies

Evaluating BMC at an industrial setting
• BMC tools: Forecast (BDDs) and Thunder (SAT)
• Measuring performance and capacity
 In search of an optimal setting for Thunder and Forecast
 Thunder vs. Forecast
 Thunder capacity boost
• Measuring productivity

Witnessed benefits of BMC
BFS traversal
Initial states
Buggy states
Counterexample trace
From BDD-based MC to BMC
Adapting state-of-the-art BDD techniques to BMC

Bounded prioritized traversal
•
•
•
•

When the BDD size reaches a certain threshold...
... split the frontier into balanced partitions, and...
... prioritize the partitions according to some criterion
Ensure bound is not exceeded
Bounded lazy traversal
• Works backwards
• Application of bounded cone of influence
SAT-based BMC
I (s)
I ( s0 ) 

Sat
k 1
i 0
T ( s i , s i 1 ) 

k 1
i 0
B( si )
B(s)

T ( s, s ' )
Bound (k=4)
k 1
i 0
SAT solver
T ( s i , s i 1 )
Unsat
Increase k?
SAT solvers




Input: a propositional formula F( x1, ..., xn )
Output: a valuation v = v1, ..., vn with
vi  {0,1} s.t. F( v1, ..., vn ) = 1
A program that can answer the question
“there exists v s.t. F( v ) = 1” is a SAT solver
Focus on solving SAT
• By exploring the space of possible assignments
• Using a sound and complete method
 Stålmarck’s (patented)
 Davis-Logemann-Loveland (DLL)
DLL method
s = {F,v} is an object
next  { SAT, UNSAT, LA, LB, HR } is a variable
DLL-SOLVE(s)
1
next  LA
2
repeat
3
case next of
4
LA : next  LOOK-AHEAD(s)
5
LB : next  LOOK-BACK(s)
6
HR : next  HEURISTIC(s)
7
Until next  { SAT, UNSAT }
8
return next
HR, LB
or SAT
LA
or UNSAT
LA
or SAT
SIMO: a DLL-based SAT solver


Boolean Constraint Propagation (BCP) is the only
Look-Ahead strategy
Non-chronological Look-Back
• Backjumping (BJ): escapes trivially unsatisfiable subtrees
• Learning: dynamically adds constraints to the formula

Search heuristics
• Static: branching order is supplied by the user
• Dynamic
 Greedy heuristics: simplify as many clauses as possible
 BCP-based: explore most constrained choices first
• Independent (relevant) vs. dependent variables
SIMO’s search heuristics
Selection
Scoring
Propagation
Moms
All
All
Morel
Relevant
Relevant
Unit
All
All
All
Unirel
All
Relevant
All
Unirel2
Relevant
Relevant
All
Forecast: BDD-based (B)MC
Directives
intel Forecast
Property (ForSpec)
Model (HDL)
Spec Synthesis
RTL synthesis
Model Checking Algorithms
Interface to BDD engines
Intel’s BDD
CUDD
CAL
Proof/Counterexample
…
Thunder: SAT-based BMC
Directives
intel Thunder
Property (ForSpec)
Model (HDL)
Spec Synthesis
RTL synthesis
Formula generation
+
+
Interface to SAT engines
SIMO
Prover
SATO
Proof/Counterexample
GRASP
Performance and capacity

Performance (what resources?)
• CPU time
• Memory consumption

Capacity (what model size?)
• BDD technology tops at 400 state variables (typically)
• SAT technology has subtle limitations depending on:
 The kind of property being checked
 The length of the counterexample
Measuring performance

Benchmarks to measure performance are
• Focusing on safety properties
• Challenging for BDD-based model checking
• In the capacity range of BDD-based model checking

In more detail
• A total 17 circuits coming from Intel’s internal selection
with known counterexample minimal length k
• Using 2 formulas per circuit with Thunder/SIMO flow
 A satisfiable instance (falsification) at bound k, and
 An unsatisfiable instance (verification) at bound k-1
An optimal setting for Thunder

1200

1000
• Moms (M) and Morel (MR)
• Unit (U), Unirel (UR) and
Unirel2 (UR2)
800
CPU time (s)
M
MR
U
UR
UR2
600
400


Instances (total 26)
SIMO admits a single optimal
setting (UR2)
• Faster on the instances solved
by all the heuristics (16)
• Solves all instances in less
than 20 minutes of CPU time
200
0
With BJ + learning enabled...
... we tried different heuristics
Unirel2 is the default setting
with the Thunder/SIMO flow
Bounded traversal in Forecast

8000
7000
CPU time (s)
6000
ABL
ABP
AUP
SBL
SBP
SUP
5000
4000
3000
2000
1000
With automatically derived
initial order
• Bounded lazy (ABL)
• Bounded prioritized (ABP)
• Unbounded prioritized (AUP)

 bounding does not yield
consistent improvements!
With semi-automatically
derived initial order
• Bounded settings (SBL, SBP)
• Unbounded prioritized (SUP)
0
Instances (total 13)
 bounding does not yield
consistent improvements!
An optimal setting for Forecast?

7000
• Best approximates the notion of
default setting in Thunder
• AUP is the the best among A’s
6000
5000
CPU time (s)
Default setting is AUP
4000
AUP
ST
3000

Tuned setting (ST)
• Semi-automatic intial order
• Specific combinations of:




2000
1000
0
Instances (total 17)

Unbounded traversal
Prioritized traversal
Lazy strategy
Partitioning the trans. relation
No single optimal tuned
setting for Forecast
Thunder vs. Forecast

7000
6000

CPU time (s)
5000
AUP
UR2
ST
4000
3000

2000
1000
0
Instances (total 17)
Forecast default AUP is
worse than Thunder UR2
Forecast tuned ST
compares well with
Thunder UR2
Forecast ST time does
not include:
• Getting pruning directives
• Finding a good initial order
• Getting the best setting
Measuring capacity

The capacity benchmark is derived from the
performance benchmark
• Getting rid of the pruning directives supplied by the
experienced users
• Enlarging the size of the model beyond the scope
of BDD-based MC

Unpruned models for this analysis…
• …have thousands sequential elements (up to 10k)
• …are out of the capacity for Forecast
Thunder capacity boost
Latches+Inputs
Latches+Inputs Variables in
(after pruning) SAT formula
Thunder
CPU time
Circuit 1(5)
12011
152
6831
6.10
Circuit 1(4)
12011
152
5403
5.10
Circuit 2(7)
7054
661
24487
96.10
Circuit 2(6)
7054
661
20552
16.37
Circuit 3(11)
6586
1129
119248
78.61
Circuit 3(10)
6586
1129
107838
68.20
Circuit 4
9704
1069
21351
29.39
Circuit 5
17262
5542
Circuit 6
6832
2936
121786
576.24
Circuit 7
3321
532
35752
73.32
Circuit 8
1457
1012
50758
267.91
TIMEOUT
Measuring productivity

Productivity decreases with user intervention
• Need to reduce the model size
• Need to find a good order on state variables
• Need to find a good tool setting

No user intervention  no productivity penalty
• Using Thunder/SIMO BMC flow:
 Dynamic search heuristic: no need for an initial order
 Single optimal setting: Unirel2 (with BJ and learning)
 Extended capacity: no manual pruning
• Comparison with Forecast BMC flow indicates that SAT
(rather than bounding) is the key for better productivity
Witnessed benefits of BMC

A single optimal setting found for Thunder using
SIMO: Unirel2 with backjumping and learning

SAT (rather than bounding) turns out to be the key
benefit when using BMC technology

A complete evaluation
• Performance of tuned BDDs parallels SAT
• Impressive capacity of SAT vs. BDDs
• SAT wins from the productivity standpoint
Useful links

The version of the paper with the correct
numbers in the capacity benchmarks:
www.cs.rice.edu/~vardi
www.cs.rice.edu/~tac

More information about SIMO:
www.cs.rice.edu/CS/Verification
www.mrg.dist.unige.it/star