How to Solve #P-Complete
Counting Problems Fast
Presented by:
Andrey Dolgin
Agenda
 Introduction
 #P-complete Problem Gallery
 Theoretical Foundations
 Numerical results
Introduction
 Many problems in
computer science,
engineering, biology,
physics are :
 χ : Set of objects,
 χ ∗ : Subset of special type
of objects.
 How many special objects
are there, i.e. what is | χ ∗|?
If finding χ ∗ is NP-Complete, then counting | χ ∗| is #P -Complete.
#P-complete Problem Gallery
 The Satisfiability problem. How many sets of
Boolean variables satisfy a given set of Boolean
clauses?
 The Hamiltonian cycle problem: How many
Hamiltonian cycles has a graph?
 The Connectivity problem: Given two vertices in a
graph, how many distinct paths are there between
them?
 The k-coloring problem. Given k distinct colors, in
how many different ways can we color the nodes,
so that each two adjacent nodes have a different
color?
Theoretical Foundations :
Counting via Monte Carlo
 Counting by rare event techniques
 Crude Monte Carlo
f (X) 
1
| |
|  | l |  ||  |  I{X * } f ( X ) |  |  f ( I{X * } )
*
X
 Importance sampling
g (X) 
1
| * |

g (X)
1 
|  ||  |  I{X * } f (X ) |  |  I{X *} f (X )
 Eg  I{X *}

g (X)
g (X) 
X
X

*
Theoretical Foundations :
Variance Minimization Method
f ( X, v )
f (X)
1
l   f [ H ( X)]   g [ H ( X)
]  lˆ 
g (X)
N
N
f ( X k , u)
H
(
X
)

k
f (X k , v)
k 1
W (X k ; u, v)
 Variance minimization
min v V (v )  min v
N
1
2
2
2
V
[
H
(
X
)
W
(
X
;
u
,
v
)]

min
{

[
H
(
X
)
W
(
X
;
u
,
v
)]


v
v
u [ H ( Y )]}
2
N
N
f ( X , w)
1
ˆ
min v V (v )  min v
N
N
2
H
 (X k )W (X k ; u, v)W (X k ; u, w)
k 1
Numerical
Theoretical Foundations :
Cross-Entropy (CE) Method
l  Pf ( S ( X )   )
 g=f(X,v) belongs to the same parametric family as f(X,u)
 Kullback-Leibler or cross-entropy distance minimization
g( X )
D( f , g )   f [ log
]   f ( x) log f ( x)dx   f ( x) log g ( x)dx
f (X )
*
g
( X )  H ( X ) f ( X, u ) / l
 Parametric minimization
1
max v  g ( x) log f ( x, v)dx  max v  I{S ( X ) } f ( x, u ) log f ( x, v)dx
l
*
f ( X , w)
1
max v
N
N
I
k 1
W ( X k ; u, w) log f ( X k , v )
{ S ( X )  }
Analytical
Theoretical Foundations :
CE for counting
 Problem
“curse of dimensionality”
l  Pf ( S ( X )   )   f I{S ( X ) }
n
W ( X, u, pt 1 )   Wi ( X i , ui , pt 1,i )
 Solution, iteration t
ˆt  S( (1  ) N )

N
~
pt ,i 
 % of indicator
 Smoothing
i 1
p t  ~
p t  (1   ) ~
p t 1
I
I
W ( X k ; u, pt 1 )
k 1 { X ki 1} {S ( X k )ˆt }
N
k 1 {S ( X k )ˆt }

I
W ( X k ; u, pt 1 )
Theoretical Foundations :
Minimum Cross-Entropy(MinxEnt)
min p {D( p, u)}  min p {

p( x1 ,..., xn ) ln
x1 ,..., xn
min p {L( p,λ,  )}  min p { p( X) ln
X
p( x1 ,..., xn )
}, s.t. p S (X)   ,  p( x1 ,..., xn )  1
u( x1 ,..., xn )
x1 ,..., xn
p( X)
 λ( p( X)S ( X)   )   ( p( X)  1)}
u ( X)
X
X
 u S ( X ) exp(  S ( X ) )

 u exp(  S ( X ) )
u( x1 ,..., x n ) exp(  S ( x1 ,..., x n ) )
p( x1 ,..., x n ) 
 u exp(  S ( X ) )
λ
Generate RV?
Theoretical Foundations :
Exponential Change of Measure(ECM)
wx
wx
M
(
w
)


[
e
]
f ( x)
e
f ( x)
 wx
f
W ( x, f , g ) 
 M ( w) e
g ( x) 
g ( x)
M ( w)
f ( x)  (2 )
1/ 2 
f  exp(  )
f  Ber ( p)
e
( x   )2
2
ft ( x)  (2 )
t

M (t )  1  
 
1
f ( x)  p x (1  p)1 x , x  0,1
1
f t ( x)  ( pe t ) x (1  p)1 x
Mt
 p  et 

  t
p

e

1

p


x
1/ 2

tx
( x   )2
2
( x  (   t ))2
f ( x )e
etx
1/ 2 e
1/ 2 
 (2 )
 (2 ) e 2
t t2 / 2
M (t )
e
f t ( x) 
e tx e  x
t 

1  


1
 (  t )  e  (   t ) x
M (t )  E f (etX )  p  et  1  p
 1 p 

  t
p

e

1

p


1 x
Ber (( pet ) /( pet  1  p))
Theoretical Foundations :
Exponential Change of Measure(ECM)
 Like one dimensional MinxEnt for each dimension
SEPARATELY
wx
M ( w)   f [e wx ]
f ( x)
W ( x, f , g ) 
 M ( w)e  wx
g ( x)
Et* X  a
P{ X  a}
 Multiple parameters
e f ( x)
g ( x) 
M ( w)
Exp( )  Exp(  w)
( ,  )  (  w,  )
N (  ,  2 )  N (   w,  2 )
Ber ( p )  Ber ( pe w /( pe w  1  p ))
Geo( p )  Geo(1  (1  p )e w )
Theoretical Foundations :
Parametric MinxEnt(PME)
 Parametric like CE but exponent instead of indicator
m
min p {D(p, u)}  min p{ p j ln
j 1

pj
uj
m
},
s.t.
p S (X)   ,  p j  1
j 1
ˆ)
S
(
X
)
exp(

S
(
X
)

k
k
t
k 1
 ˆt
N
exp(  S ( X )ˆ )
̂t
N

k 1


k
t
ˆ )W ( X , u, ~
I
exp(

S
(
X
)

pt 1 )
{
X

1
}
k
t
k
ki
k 1
N
exp(  S ( X )ˆ )W ( X , u, ~
p )
N
~
pt ,i

k 1
k
t
k
t 1
 If S(X) is block-wise separable, size of W reduces
Theoretical Foundations :
Block-wise separability 2-SAT
1
-1
0
0
1
1
0
0
0 0
S(X)=S1(X1,X2)+S2(X3,X4)
0 0
W ( X , u, ~p t 1 )  W1 ( X 1 , X 2 , u1 , u 2 , ~p t 1,1 , ~p t 1,2 )W2 ( X 3 , X 4 , u 3 , u 4 , ~p t 1,3 , ~p t 1,4 )
1 1
1 -1  u [F1 ( X 1 , X 2 )F2 ( X 3 , X 4 )]   u [F1 ( X 1 , X 2 )] u [F2 ( X 3 , X 4 )]
X exp{ ˆ S ( X , X )}W ( X , X , u , u , ~
p ,~
p )


 exp{ˆ S ( X , X )}W ( X , X , u , u , ~p , ~p )
X exp{ ˆ S ( X , X )}W ( X , X , u , u , ~
p ,~
p
)


 exp{ˆ S ( X , X )}W ( X , X , u , u , ~p , ~p )
N
~
pt ,1
k 1
N
t 1 1
k1
t 1 1
k 1
k1
k1
k2
k2
1
1
k1
k1
k2
k2
1
1
2
t 1,1
t 1,1
2
t 1, 2
t 1, 2
N
~
p t ,3
k 1
N
k 1
t 1
k3
t 1
2
2
k3
k3
k4
k4
k3
k3
k4
k4
3
3
4
4
t 1, 3
t 1, 3
t 1, 4
t 1, 4
Theoretical Foundations :
PME for counting
 Problem
l  Pf ( S ( X )   )   f I{S ( X ) }
| |
pt 1,i S( i )
 Solution, iteration t

i c
ˆt 
| |
 % of mean
p

t
i  ct
t 1,i
ˆ)
S
(
X
)
exp(

S
(
X
)

k 1
k
k
t
 ˆt
N
exp(  S ( X )ˆ )
N

k 1
~
pt ,i


k
̂t
t
ˆ )W ( X , u, p )
I
exp(

S
(
X
)

{
X

1
}
k
t
k
t 1
ki
k 1
N
exp(  S ( X )ˆ )W ( X , u, p )
N
 Smoothing

k 1
pt  ~
pt  (1   ) ~
pt 1
k
t
k
t 1
Numerical results :
SAT
Good news:
SAT is solvable
Theoretical Foundations :
Satisfiability(SAT) Problem
Count all truth assignments {x1,…,xn} such that
all clauses {C1,…,Cm} are true.
Theoretical Foundations :
Satisfiability Problem
Theoretical Foundations :
SAT with Conditioning
Numerical results :
Screened CE 25x100 3-SAT
Numerical results :
PME 25x100 3-SAT
Numerical results :
Screened CE 75x325 3-SAT
Numerical results :
PME 75x325 3-SAT
Numerical results :
PME 75x325 3-SAT Dynamics
Numerical results :
Screened CE 50x218 3-SAT
Numerical results :
PME 50x218 3-SAT
Numerical results :
Conditional PME 50x218 3-SAT
Numerical results :
PME 100x110 2-SAT
Numerical results :
PME 100x110 2-SAT Dynamics
THANK YOU