Using Satisfiability Solvers to Test Network Configurations1
A comparison of three solvers’ performances
Mendy Fisch (mendy@princeton.edu)
Dr. Gary Levin, Telcordia Technologies, Inc. (glevin@telcordia.com)
Contents
Summary – 1
Description of the problem – 2
Input methods for network configurations – 3
Running times for each solver – 3
Acknowledgements – 6
References - 6
Summary
This brief report describes the performance of various satisfiability solvers
when used to test configurations of network addresses. The satisfiability solvers
were fed files detailing a desired configuration of network addresses and asked to
determine if the configuration were possible. Given a possible configuration, the
solvers were asked to output numerical assignments for the network addresses that
would satisfy the configuration requirements, if they were capable of doing so.
This research was conducted with the eventual goal of finding a solver that
could solve large instances of this problem using reasonable time and space. This
could have useful commercial applications as part of a program that checks existing
network configurations or configures network configurations according to desired
parameters.
All of the solvers tested appeared to take at least O(N^3) time to solve test
configurations that were fed to them, where N is the number of subnets in the
configuration. Physical memory was a major limiting factor. Running on a machine
with 1536 MB of RAM, the solvers ran out of memory when testing large instances
of the problem – which caused thrashing and page faults.
The solvers tested included the Kodkod constraint solver for relational logic
(http://web.mit.edu/emina/www/kodkod.html) utilizing both the MiniSat and
MiniSatProver SAT solvers (http://minisat.se/), CVC3, an automatic theorem prover
for SMT problems (http://www.cs.nyu.edu/acsys/cvc3/), and Boolector, an SMT
solver for bitvectors (http://fmv.jku.at/boolector/), which was the winner in the
bitvector division of the 2008 Satisfiability Modulo Theories Competition
(http://smtcomp.org/).
Kodkod and Boolector performed similarly in tests – both took about O(N^3)
time and caused the machine to thrash or run out of memory for files containing 150
1
This material is based upon work supported by Air Force Rome Laboratories (AFRL)
under contract FA8750-07-C-0030, funded by Dr. Carl Landwehr. Any opinions, findings
and conclusions or recommendations expressed in this material are those of the authors
and do not necessarily reflect the views of Air Force Rome Laboratories.
1
or more subnets. CVC began running out of memory on files with 50 or more
subnets, so it was difficult to get an estimate for the order of the runtime of the
problem under CVC.
While cubic runtime is certainly not optimal, it seems that the first bottleneck
is physical memory. A program that drains the machine’s memory will not evaluate
in any amount of time. The first challenge for SAT solver developers should be to
optimize the solvers for space; runtime can be addressed afterwards.
Description of the problem
The aim of the project is to utilize satisfiability solvers to determine whether
given configurations of network addresses are valid, and generate numerical
assignments for unassigned network addresses if the configuration is valid. This is
accomplished by expressing the desired hierarchy of IP addresses as a logical
statement and running the statement through a satisfiability solver to generate a set
of addresses that will satisfy the configuration.
The most important operation in the problem is checking whether a given IP
address falls within the range of a given subnet address – that is, whether it would
be possible for this IP address to be below the subnet address in the network
hierarchy and hence “contained” in the subnet.
Linking “contained” operations together with the logical operators AND,
NOT, and EQUALS allows us to describe a hierarchical network of IP addresses,
some representing large subnets, others representing smaller subnets, and still
others representing individual machines.
One way to check if one IP address is “contained” within a subnet involves
performing bitvector operations on the address and the subnet. Given a subnet
address A and a mask M that indicates which bits in A are host-specific, as well as an
IP address B and a mask N that indicates which bits in B are host specific, the
“contained” statement can be expressed as:
(M ≥ N) AND A & (-1 << M) = B & (-1 << M)
where A and B are 32-bit integers, M and N are numbers between 0 (no bits are
significant) and 32 (all bits are significant), & is a bitvector AND operation, and (-1
<< M) is an instruction to left-shift a 32-bit integer with all bits on by M bits.
An advantage of this method is that many satisfiability solvers are optimized
for bitvector operations - or, in the case of Boolector, are only able to work with
bitvectors.
Another way to implement “contained” would be to avoid using bit-whacking
operations and stick to integer comparisons. In this case, we would need to
transform the subnet address (A) and the address we want to test (B) into network
ids by computing A – (A modulo M) and B – (B modulo N), where M is A’s mask and
N is B’s mask. Then, we could express the contained statement as:
networkId(A) ≤ networkId(B) AND networkId(A) + 2M ≥ networkdId(B) + 2N
Though we tried to program this method in CVC3, we failed because there
was no way to implement the modulo function in CVC. Running a sample file
through CVC (see http://www.princeton.edu/~mendy/CVC/SMT-002.txt)
generated the following error:
“*** fatal error in theory_core.cpp:325 (false)
2
Equivalence classes didn't get merged.”
Input methods for network configurations
Network configurations were expressed in postfix notation, and translated to
the appropriate format for each satisfiability solver. For an example of a small file in
the postfix notation (Qunatifier Free Format), see
http://www.princeton.edu/~mendy/QFF/QFF-002.txt. (For more examples, see the
directory http://www.princeton.edu/~mendy/QFF.)
The java program http://www.princeton.edu/~mendy/parser_bitvector.java
translates QFF files to CVC3’s presentation input language, implementing
“contained” using the bitwhacking method. All of the BIN-* files in the directory
http://www.princeton.edu/~mendy/CVC/ were generated using this translator.
The translator http://www.princeton.edu/~mendy/parser_smt.java
translates QFF files to CVC3’s presentation input language, but implements
“contained” using integer comparisons. An example of a file generated using this
translator is http://www.princeton.edu/~mendy/CVC/SMT-002.txt. CVC fails to
evaluate this file, for the reasons described above.
To prepare files for Boolector, we translate files first to CVC’s presentation
input language, and then use CVC’s translate function to translate the file to the
SMT-LIB format that Boolector understands. The file
http://www.princeton.edu/~mendy/parser_bitvector_noint.java outputs files that
can be translated by CVC to SMT-LIB format. It differs from parser_bitvector.java
because it ensures all bitvectors used are of the same size and changes all integer
constraints (i.e. 4 = 4) into bitvector constraints. Files in
http://www.princeton.edu/~mendy/Boolector/ were generated by running the
QFF files through the parser_bitvector_noint translator and feeding the results to
CVC3’s translator (cvc3 +translate –output-lang smt). Writing a more direct
translator may have been more efficient, but was unnecessary since we are seeking
to compare solvers rather than come up with efficient translation methods.
Running times for each solver
Following is a table summarizing the performance of each solver on the given
input files. The solvers were run on a VMware virtual machine running Red Hat
Enterprise Linux 4, utilizing 1536 megabytes of RAM and 2 2.4 GHZ Intel Core Duo
processors. For programs requiring Java, Java was assigned 1500 MB of RAM (using
–Xmx1500m). Where it appeared that memory limitations were causing a
significant slowdown in the program, this fact is noted.
The files used can be found using the links provided above. Each filename
indicates how many subnets it is checking. Files ending in const4 replace all of the
mask variables with 4, and files ending in eq4 append the condition that each mask
variable equal 4. This makes for a faster runtime.
Boolector
File
002
Solving
0.039
3
050-const4
050-eq4
050
100-const4
100-eq4
100
150-const4
150-eq4
150
200-const4
200-eq4
200
1.739
1.612
10.888
5.822
6.003
91.441
12.590
14.728
* out of memory error ([btormem] out of memory in 'btor_realloc')
46.826 * observed memory limitations using vmstat
594.158 * observed memory limitations using vmstat
* out of memory error ([btormem] out of memory in 'btor_realloc')
KodKod - MiniSat
File
002
050-const4
050-eq4
050
100-const4
100-eq4
100
150-const4
150-eq4
150
200-const4
200-eq4
200
Solving
0.008
0.212
0.935
5.728
0.559
4.850
55.950
1.150
10.204
1945.287 * observed memory limitations using vmstat
3.319
*** stopped after 30 minutes, observed memory limitations
*** stopped after 30 minutes, observed memory limitations
KodKod - MiniSatProver
File
Solving
002
0.012
050-const4 0.205
050-eq4
2.076
050
162.813 * observed memory limitations using vmstat
100-const4 1.026
100-eq4
10.717
100
*** stopped after 30 minutes, observed memory limitations
150-const4 2.249
150-eq4
*** stopped after 30 minutes, got out of memory error message
150
*** stopped after 30 minutes, got out of memory error message
200-const4 76.358
200-eq4
*** stopped after 30 minutes, got out of memory error message
200
*** stopped after 30 minutes, got out of memory error message
4
CVC3
002
050-const4
050-eq4
050
100-const4
11.624
662.809 * observed memory limitations
*** stopped after 30 minutes, observed memory limitations
*** stopped after 30 minutes, observed memory limitations
*** stopped after 30 minutes, observed memory limitations
I was able to observe memory limitations by running the vmstat command
on the Linux machine. If there were large numbers in the “swpd” column (for virtual
memory usage), the “si” column (for page-ins), and the “so” column (for page outs),
there was likely not enough physical memory, and paging was probably causing the
runtime of the program to become significantly slower. Following is a sample
vmstat printout indicating low memory. Notice the large values in the “swpd,” “si,”
and “so” fields. If the program were not demanding more memory than was
available, these fields would probably have contained zeroes.
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu---r b swpd free
buff cache si so
bi bo in
cs us sy id wa
3 1 229444 17464 488 21288 46 116 151 288 519 168 8 57 25 9
4 0 239292 14456 216 19864 363 1020 906 1057 999 222 4 73 4 19
3 0 240636 14400 220 19472 922 466 1786 507 1037 259 2 44 12 41
0 4 253172 15672 224 21456 1218 1289 2337 1397 1071 327 2 35 12 51
2 6 258496 14656 232 18044 835 557 1570 675 1068 298 2 56 8 35
Occasionally, Java or Boolector generated an error message when it reached
a memory limit. Kodkod did this when solving 200-eq4 and 200 with MiniSatProver.
The error message read:
“Free swap:
393216 pages of RAM
163824 pages of HIGHMEM
4593 reserved pages
6151 pages shared
501 pages swap cached
Out of Memory: Killed process 19695 (java)”
Each time I received this message, my machine froze and I needed to restart
it. Boolector also outputted an error message for 200 and 150, “[btormem] out of
memory in 'btor_realloc.'”
Looking at the times that were not affected by memory limitations (we can
really only do this for Kodkod with MiniSat and Boolector), it seems that the running
time is cubic for the regular files (with unassigned mask variables) and quadratic for
the files with assigned mask variables. T(100)/T(50) for Boolector = 9.441/10.888 =
8.4, and T(100)/T(50) for Kodkod = 55.950/5.728 = 9.77. Both of these ratios are
5
around 9, indicating cubic time. Most of time ratios for doubling files with assigned
mask variables are around 4, indicating quadratic time.
Though using cubic or quadratic time is often a cause for concern, it is clear
that the biggest problem at the moment is the amount of memory taken up by the
solvers. Either the solvers should be adjusted to consume less memory per variable,
or the configuration files should be rewritten to contain fewer variables.
Acknowledgements
This work was performed under guidance of Professor Sharad Malik at Princeton,
Professor Clark Barrett at New York University and Dr. Sanjai Narain at Telcordia.
The idea of using Kodkod for network configuration was developed in Telcordia's
ConfigAssure project. The Kodkod system was created by Emina Torlak at MIT under
guidance of Professor Daniel Jackson.
References
1. S. Narain, G. Levin, V. Kaul, S. Malik. “Declarative Infrastructure Configuration
Synthesis and Debugging.” Journal of Network Systems and Management, Special Issue
on Security Configuration, eds. Ehab Al-Shaer, Charles Kalmanek, Felix Wu. 2008.
Available online:
http://www.argreenhouse.com/papers/narain/DeclarativeConfiguration.pdf
2.
Alloy: http://alloy.mit.edu/
3.
Kodkod: http://web.mit.edu/emina/www/kodkod.html
4.
Chaff: http://www.princeton.edu/~chaff
5.
MiniSat: http://minisat.se/
6. Y. Mahajan, Z. Fu, S. Malik. “Zchaff2004, An Efficient SAT Solver.” Proceedings of 7th
International Conference on Theory and Applications of Satisfiability Testing (SAT). 2004.
6