# Probabilistic Inference Heuristics for Satisfiability by mifei

VIEWS: 5 PAGES: 50

• pg 1
```									The satisfiability threshold and clusters of solutions in the 3-SAT problem
Elitza Maneva IBM Almaden Research Center

3-SAT
Variables: x1, x2, …, xn take values {TRUE, FALSE}

Constraints: (x1 or x2 or_not x3) , (not x2 or x_ or not x6), … 4 _ (x1  x2  x3)  ( x2  x4  x6)  … 1
x1 x2 0 x4 x5 x8

0

x3 x6

x7

1

0

Random 3-SAT
x1 x2 x3 x4 x5 x6 x7 x8

n

m = n
Red = proved, green = unproved
Random walk
Survey propagation Satisfiable Belief propagation Satisfiable Myopic PLR
0 1.63

Not satisfiable Not satisfiable



3.52

3.95

4.27

4.51

Rigorous bounds for random 3-SAT
2002 2002 Kaporis Hajiaghayi Kirousis Sorkin Lalas

3.52

1999: [Friedgut] there is a sharp threshold of satisfiability c(n)

Rigorous bounds for random 3-SAT 
0 1.63 3.52 4.51 5.19

Pure Literal Rule Algorithm: If any variable appears only positive or only negative assign it 1 or 0 respectively Simplify the formula by removing the satisfied clauses Repeat _ _ _ _ _ (x1  x2  x3)  ( x2  x4  x5)  (x1  x2  x4)  (x3  x4  x5) 1 1 0 1

Rigorous bounds for random 3-SAT 
0 1.63 3.52 4.51 5.19

Myopic Algorithms:
Choose a variable according to # positive and negative occurrences Assign the variable the more popular value Simplify the formula by 1. removing the satisfied clauses 2. removing the FALSE literals 3. assigning variables in unit clauses 4. assigning pure variables Repeat Best rule: maximum |# positive occurr. – # negative occurr.|

Rigorous bounds for random 3-SAT 
0 1.63 3.52 4.51 5.19

E [# solutions] = 2n  Pr [00…0 is a solution] = = 2n  (1-1/8)m = = (2  (7/8))n For >5.191, E [# solutions]  0, so Pr [satisfiable]  0

Rigorous bounds for random 3-SAT 
0 1.63 3.52 4.51 4.67 5.19

E [# positively prime solutions]  0

Positively prime solution: a solution in which no variable assigned 1 can be converted to 0.
Fact: If there exists a solution, there exists a positively prime solution.

Rigorous bounds for random 3-SAT 
0 1.63 3.52 4.51 4.67 5.19

E [# symmetrically prime solutions]  0

Random 3-SAT
x1 x2 x3 x4 x5 x6 x7 x8

n

m = n
Red = proved, green = unproved
Random walk
Survey propagation Belief propagation Satisfiable Myopic PLR
0 1.63



3.52

3.95

4.27

4.51

Random Walk Algorithms
[Alekhnovich, Ben-Sasson `03] [Seitz, Alava, Orponen `05] [Ardelius, Aurell `06] ASAT: Pick an unsatisfied clause Pick a variable in the clause Flip it only with prob. p if number of unsatisfied clauses does not increase Experiment: Takes O(n) steps for  < 4.21.

Simple Random Walk:
Pick an unsatisfied clause Pick a variable in the clause Flip the variable

Theorem: Finds a solution in O(n) steps for  < 1.63.

Random 3-SAT
x1 x2 x3 x4 x5 x6 x7 x8

n

m = n
Red = proved, green = unproved
Random walk
Survey propagation Belief propagation Satisfiable Myopic PLR
0 1.63



3.52

3.95

4.27

4.51

We can find solutions via inference
Suppose the formula is satisfiable. Consider the uniform distribution over satisfying assignments: Pr[x1, x2, …, xn]   (x1, x2, …, xn)

Simple Claim: If we can compute Pr[xi=1], then we can find a solution fast. Decimation: Assign variables one by one to a value that has highest probability.

Fact: We cannot hope to compute Pr[xi=1] exactly
Heuristics for guessing the best variable to assign:
1. Pure Literal Rule (PLR): Choose a variable that appears always positive / always negative. 2. Myopic Rule: Choose a variable based on number of positive and negative occurrences. 3. Belief Propagation: Estimate Pr[xi=1] by belief propagation and choose variable with largest estimated bias.

Computing Pr[x1=0] on a tree formula
108 192 36 48 1 1

x1

3 4

12 12

1 1

1 1

3 4

4 3

#Solns with 0 #Solns with 1

1 1

1 1

1 1

1 1

#Solutions with 0 #Solutions with 1

Vectors can be normalized
.36 .64 .43 .57 .5 .5 .43 .57

x1

.43 .57

.5 .5

.5 .5

.5 .5

.57 .43

.5 .5

.5 .5

.5 .5

.5 .5

Vectors can be normalized … and thought of as messages
x1

What if the graph is not a tree?

Belief propagation

Belief propagation
x3 x2 (x1, x2 , x3) x1 x11 Pr[x1, …, xn]  Πa a(xN(a))

x4

x5

x6

x10

x9

x8

x7

Belief Propagation [Pearl ‘88]
x1 x2 x3 x4 x5 x6 x7

n

m Given: Pr[x1 …x7]  a(x1, x3)  b(x1, x2)  c(x1, x4) … i.e. Markov Random Field (MRF) Goal: Compute Pr[x1] (i.e. marginal) Belief programming algorithm. bN(i)/c It is exact only when the recurrence relation holds, i.e.: M c  i (x ) = Σ  (x )  Π M j c (xj) : tree. j N(c)\i 1. if the graph iis xaj N(c)\i c N(c) 2. if the Estimated marginals:like a tree: large cycles graph behaves
j

Message passing rules: M i  c (xi) = Π M b  i (xi) propagation is a dynamic

i(xi) = Π

cN(i)

M c  i (xi)

Applications of belief propagation
• • • • • • Statistical learning theory Vision Error-correcting codes (Turbo, LDPC, LT) Lossy data-compression Computational biology Sensor networks

Limitations of BP
x1 x2 x3 x4 x5 x6 x7 x8

n

m = n
Random walks
Survey propagation Belief propagation Satisfiable Myopic PLR
0 1.63



3.52

3.95

4.27

4.51

Reason for failure of Belief Propagation
• Messages from different neighbors are assumed to be almost independent  i.e. there are no long-range correlations

Random walks

No long-range propagationLong-range Survey correlations correlations Belief propagation exist
Myopic PLR
0 1.63



3.52

3.95

4.27

4.51

Reason for failure of Belief Propagation
• Messages from different neighbors are assumed to be almost independent  i.e. there are no long-range correlations Fix: 1-step Replica Symmetry Breaking Ansatz • The distribution can be decomposed into ―phases‖ • There are no long-range correlations within a phase • Each phase consists of similar assignments – ―clusters‖ • Messages become distributions of distributions • An approximation yields 3-dimensional messages: Survey Propagation [Mezard, Parisi, Zecchina ‗02] • Survey propagation finds a phase, then WalkSAT is used to find a solution in the phase

Reason for failure of Belief Propagation
• Messages from different neighbors are assumed to be almost independent  i.e. there are no long-range correlations Fix: 1-step Replica Symmetry Breaking Ansatz • The distribution can be decomposed into ―phases‖ Pr[x1, x2, …, xn] =  p  Pr [x1, x2, …, xn]

fixed variables

Space of solutions
Satisfying assignments in {0, 1}n
0110

phases

1011

01101

Survey propagation

.12 .81 .07

0 1 

Survey propagation
x1 x2 x3 x4 x5 x6 x7

I‘m 0 with prob 10%, 1 with prob 70%, whichever (i.e. ) 20%

x8

You have to satisfy Mu me jc Mci=  with prob. 60% ———————— u s  jN(c)\i

M

j c+M j  c+M jc

Muic = (1Msic = (1Mic =

b  Nua (i)



(1- Mbi ))

b  Nsa (i)



(1-Mbi) (1-Mbi)

b


Nsc (i)

(1- Mbi )) 
b

Nuc

(i)

b  N(i)\c



(1- Mbi )

Combinatorial interpretation
• Can survey propagation be thought of as inference on cluster assignments? Not precisely, but close. • We define a related concept of core/cover assignments • Assignments in the same cluster share the same core • However, different cluster may have the same core

Finding the core of a solution
0
1

1 0 0 0

0
1

Finding the core of a solution
0
1

1 0 0 0

0
1

Finding the core of a solution
0
1

1 0 0 0

0
0

unconstrained variables

Finding the core of a solution
0
1

1 0 0 0

0


Finding the core of a solution
0
1

1 0  0

0


Finding the core of a solution
0


1 0  0

0


Finding the core of a solution



1 0  0

0


Such a fully constrained partial assignment is called a cover.

Extending the space of assignments
{0, 1}n assignments
0110

Partial assignments {0,1,}n

core

# stars

core

1011

01101



Survey propagation is a belief propagation algorithm
[Maneva, Mossel, Wainwright ‘05] [Braunstein, Zecchina ‘05] Theorem: Survey propagation is equivalent to belief propagation on the uniform distribution over cover assignments.

But, we still need to look at all partial assignments.

Peeling Experiment for 3-SAT, n =105
100000 90000 80000


2 2.5 3 3.5 4 4.1 4.2

# unconstrained

70000 60000 50000 40000 30000 20000 10000 0 0 20000 40000 60000 80000

100000

# stars

Clusters and partial assignments
{0, 1}n assignments
0110

Partial assignments

01101 1011

01101

# stars



Definition of the new distribution
Partial assignments Formula
0111 1111 1010

1

n()

011 

011 11

111

111
11

010 100

3

2

no()

11

4

1

1 

1

1

1. Includes all assignments without contradictions or implications 2. Weight of partial assignments:

Pr[]  

n()

(1- )

no()

Vanilla BP
3. A family of belief propagation algorithms: 0


1

SP

Pr[]   n() (1- ) no()
{0, 1}n assignments
0110

Partial assignments =0 =1
core

# stars

1011

core Vanilla BP
0

01101



SP
1

This is the correct picture for 9-SAT and above.  [Achlioptas, Ricci-Tersenghi ‗06]

Clustering for k-SAT
What is known?
2-SAT: a single cluster 3-SAT to 7-SAT: not known 8-SAT and above: exponential number of clusters (with second moment method) [Mezard, Mora, Zecchina `05] [Achlioptas, Ricci-Tersenghi `06] 9-SAT and above: clusters have non-trivial cores (with differential equations method) [Achlioptas, Ricci-Tersenghi `06]

Convex geometry / Antimatroid
0111 1111 1010

011

011

111

111

010

100

11

11

11

1

1

1

1



Total weight is 1 for every 

Rigorous bounds for random 3-SAT 
0 3.52 4.51 4.9 5.19

E [total weight of partial assignments]  0 ( = 0.8) Fact: If there exists a solution, the weight of partial assignments is at least 1.

Rigorous bounds for random 3-SAT 
0 3.52 4.453 4.51 5.19

Theorem [ Maneva, Sinclair ] For  > 4.453 one of the following holds: 1. there are no satisfying assignments with high probability; 2. the core of every satisfying assignment is (,,…,).

Random 3-SAT
x1 x2 x3 x4 x5 x6 x7 x8

n

m = n
Red = proved, green = unproved
Random walk
Survey propagation Belief propagation Satisfiable Myopic PLR
0 1.63



3.52

3.95

4.27

4.51

Challenges
• Improve the bounds on the threshold • Prove algorithms work with high probability

• Find an algorithm for certifying that a formula with n clauses for large  has no solution

Thank you

```
To top