# Graph Cuts and Discrete Optimization

Document Sample

```					MAP Estimation Algorithms in
Computer Vision - Part II

M. Pawan Kumar, University of Oxford

Pushmeet Kohli, Microsoft Research
Example: Image Segmentation
E: {0,1}n → R
E(x) =   ∑ ci xi + ∑ cij xi(1-xj)            0 → fg
1 → bg
i         i,j
n = number of pixels

Image (D)
Example: Image Segmentation
E: {0,1}n → R
E(x) =     ∑ ci xi + ∑ cij xi(1-xj)            0 → fg
1 → bg
i                i,j
n = number of pixels

Unary Cost
(ci)
Dark (negative)   Bright (positive)
Example: Image Segmentation
E: {0,1}n → R
E(x) =   ∑ ci xi + ∑ cij xi(1-xj)            0 → fg
1 → bg
i         i,j
n = number of pixels

Discontinuity
Cost (cij)
Example: Image Segmentation
E: {0,1}n → R
E(x) =   ∑ ci xi + ∑ cij xi(1-xj)            0 → fg
1 → bg
i         i,j
n = number of pixels

x* = arg min E(x)
x

How to minimize
E(x)?
Global Minimum (x*)
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut
and energy minimization?

What problems can we solve
using st-mincut?

st-mincut based Move algorithms

Outline of the Tutorial

The st-mincut problem

Connection between st-mincut
and energy minimization?

What problems can we solve
using st-mincut?

st-mincut based Move algorithms

The st-Mincut Problem

Source

2            9          Graph (V, E, C)
1                Vertices V = {v1, v2 ... vn}
v1            v2         Edges E = {(v1, v2) ....}

2                   Costs C = {c(1, 2) ....}
5             4

Sink
The st-Mincut Problem
What is a st-cut?

Source

2            9
1
v1            v2
2
5             4

Sink
The st-Mincut Problem
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source

2            9      What is the cost of a st-cut?
1                 Sum of cost of all edges
v1            v2           going from S to T
2
5            4

Sink

5 + 2 + 9 = 16
The st-Mincut Problem
What is a st-cut?
An st-cut (S,T) divides the nodes
between source and sink.
Source

2            9      What is the cost of a st-cut?
1                 Sum of cost of all edges
v1            v2           going from S to T
2
5             4
What is the st-mincut?
Sink                     st-cut with the
minimum cost

2 + 1 + 4 = 7
How to compute the st-mincut?

Solve the dual maximum flow problem
Compute the maximum flow between
Source                 Source and Sink
Constraints
2             9
Edges: Flow < Capacity
1
v1             v2          Nodes: Flow in = Flow out
2
5              4
Min-cut\Max-flow Theorem
Sink
In every network, the maximum flow
equals the cost of the st-mincut
Maxflow Algorithms

Flow = 0
Augmenting Path Based
Algorithms
Source

2              9     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
5               4

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 0
Augmenting Path Based
Algorithms
Source

2              9     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
5               4

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 0 + 2
Augmenting Path Based
Algorithms
Source

2-2             9      1. Find path from source to sink
with positive capacity
1
v1             v2
2. Push maximum possible flow
2               through this path
4
5-2
3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 2
Augmenting Path Based
Algorithms
Source

0              9     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
3               4

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 2
Augmenting Path Based
Algorithms
Source

0              9     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
3               4

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 2
Augmenting Path Based
Algorithms
Source

0              9     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
3               4

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 2 + 4
Augmenting Path Based
Algorithms
Source

0             5      1. Find path from source to sink
with positive capacity
1
v1             v2
2. Push maximum possible flow
2               through this path
3              0

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 6
Augmenting Path Based
Algorithms
Source

0              5     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
3               0

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 6
Augmenting Path Based
Algorithms
Source

0              5     1. Find path from source to sink
with positive capacity
1
v1              v2
2. Push maximum possible flow
2               through this path
3               0

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 6 + 1
Augmenting Path Based
Algorithms
Source

0             4      1. Find path from source to sink
with positive capacity
1-1
v1             v2
2. Push maximum possible flow
2+1               through this path
2              0

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 7
Augmenting Path Based
Algorithms
Source

0              4     1. Find path from source to sink
with positive capacity
0
v1              v2
2. Push maximum possible flow
3               through this path
2               0

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
Maxflow Algorithms

Flow = 7
Augmenting Path Based
Algorithms
Source

0              4     1. Find path from source to sink
with positive capacity
0
v1              v2
2. Push maximum possible flow
3               through this path
2               0

3. Repeat until no path can be
Sink
found

Algorithms assume non-negative capacity
History of Maxflow Algorithms
n:   #nodes
Augmenting Path and Push-Relabel
m:   #edges
U: maximum
edge weight

Algorithms
assume non-
negative edge
weights

[Slide credit: Andrew Goldberg]
History of Maxflow Algorithms
n:   #nodes
Augmenting Path and Push-Relabel
m:   #edges
U: maximum
edge weight

Algorithms
assume non-
negative edge
weights

[Slide credit: Andrew Goldberg]
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

1000             1000
1
a1                       a2

0
1000         1000
Sink
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

1000             1000
1
a1                       a2

1000         1000         Augmenting
Paths
Sink
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

1000             1000
1
a1                       a2

1000         1000         Augmenting
Path
Sink
Augmenting Path based Algorithms

Ford Fulkerson: Choose any augmenting path

Source

999              1000
0
a1                       a2

1
1000         999
Sink
Augmenting Path based Algorithms
n:   #nodes

Ford Fulkerson: Choose any augmenting path          m:   #edges

Source

999               1000
0
a1                        a2

1
1000           999
Sink

We will have to perform 2000 augmentations!
Worst case complexity: O (m x Total_Flow)
(Pseudo-polynomial bound: depends on flow)
Augmenting Path based Algorithms
n:   #nodes

Dinic: Choose shortest augmenting path   m:   #edges

Source

1000            1000
1
a1                       a2

0
1000         1000
Sink

Worst case Complexity: O (m n2)
Maxflow in Computer Vision

• Specialized algorithms for vision
problems
– Grid graphs
– Low connectivity (m ~ O(n))

• Dual search tree augmenting path
algorithm
[Boykov and Kolmogorov PAMI 2004]
• Finds approximate shortest augmenting
paths efficiently
• High worst-case time complexity
• Empirically outperforms other
algorithms on vision problems
Maxflow in Computer Vision

• Specialized algorithms for vision
problems
– Grid graphs
– Low connectivity (m ~ O(n))

• Dual search tree augmenting path
algorithm
[Boykov and Kolmogorov PAMI 2004]
• Finds approximate shortest augmenting
paths efficiently
• High worst-case time complexity
• Empirically outperforms other
algorithms on vision problems
• Efficient code available on the web
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut
and energy minimization?

What problems can we solve
using st-mincut?

st-mincut based Move algorithms

St-mincut and Energy Minimization
S   st-mincut
Minimizing a Qudratic
Pseudoboolean
function E(x)
T

Functions of boolean variables          Pseudoboolean?

E: {0,1}n → R

E(x) =   ∑ ci xi + ∑ cij xi(1-xj)            cij≥0
i         i,j

Polynomial time st-mincut algorithms
require non-negative edge weights
So how does this work?

Construct a graph such that:
1. Any st-cut corresponds to an assignment of x
2. The cost of the cut is equal to the energy of
x : E(x)

S   st-mincut

E(x)

T
Solution
Graph Construction

E(a1,a2)

Source (0)

a1              a2

Sink (1)
Graph Construction

E(a1,a2) = 2a1

Source (0)

2

a1               a2

Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1

Source (0)

2

a1                a2

5
Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2

Source (0)

2          9

a1                   a2

5           4
Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2

Source (0)

2            9

a1                     a2

2
5             4
Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2           9
1
a1                    a2

2
5            4
Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2           9
1
a1                    a2

2
5            4
Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2           9         Cost of cut = 11

1
a1                    a2    a1 = 1 a2 = 1

2
E (1,1) = 11
5            4
Sink (1)
Graph Construction

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2           9         st-mincut cost = 8

1
a1                    a2     a1 = 1 a2 = 0

2
E (1,0) = 8
5            4
Sink (1)
Energy Function Reparameterization

Two functions E1 and E2 are reparameterizations if

E1 (x) = E2 (x) for all x

For instance:
a1     ā1   1+ 2a1 + 3ā1   3 + ā1
E1 (a1) = 1+ 2a1 + 3ā1
0      1            4        4
E2 (a1) = 3 + ā1
1      0            3        3
Flow and Reparametrization

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2           9
1
a1                    a2

2
5            4
Sink (1)
Flow and Reparametrization

E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

2           9
2a1 + 5ā1
1
a1                    a2
= 2(a1+ā1) + 3ā1
2
= 2 + 3ā1
5            4
Sink (1)
Flow and Reparametrization

E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (0)

0           9
2a1 + 5ā1
1
a1                    a2
= 2(a1+ā1) + 3ā1
2
= 2 + 3ā1
3            4
Sink (1)
Flow and Reparametrization

E(a1,a2) = 2 + 3ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2

Source (1)

0           9
9a2 + 4ā2
1
a1                    a2
= 4(a2+ā2) + 5ā2
2
= 4 + 5ā2
3            4
Sink (0)
Flow and Reparametrization

E(a1,a2) = 2 + 3ā1+ 5a2 + 4 + 2a1ā2 + ā1a2

Source (1)

0           5
9a2 + 4ā2
1
a1                    a2
= 4(a2+ā2) + 5ā2
2
= 4 + 5ā2
3            0
Sink (0)
Flow and Reparametrization

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

Source (1)

0           5
1
a1                    a2

2
3            0
Sink (0)
Flow and Reparametrization

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2

Source (1)

0           5
1
a1                    a2

2
3            0
Sink (0)
Flow and Reparametrization

E(a1,a2) = 6 + 3ā1+ 5a2 + 2a1ā2 + ā1a2
3ā1+ 5a2 + 2a1ā2

= 2(ā1+a2+a1ā2) +ā1+3a2
Source (1)
= 2(1+ā1a2) +ā1+3a2
0           5
F1 = ā1+a2+a1ā2
1
a1                    a2        F2 = 1+ā1a2

2                 a1     a2   F1    F2
3            0         0      0    1     1
0      1    2     2
Sink (0)     1      0    1     1
1      1    1     1
Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2
3ā1+ 5a2 + 2a1ā2

= 2(ā1+a2+a1ā2) +ā1+3a2
Source (1)
= 2(1+ā1a2) +ā1+3a2
0            3
F1 = ā1+a2+a1ā2
3
a1                      a2        F2 = 1+ā1a2

0                  a1     a2   F1    F2
1              0         0      0    1     1
0      1    2     2
Sink (0)      1      0    1     1
1      1    1     1
Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

Source (1)

0            3
3                     No more
a1                      a2
augmenting paths
0                     possible
1              0
Sink (0)
Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2                    Residual Graph
(positive coefficients)

Source (1)
Total Flow
bound on the           0           3
optimal solution
3
a1                    a2

0
1            0
Sink (0)
Inference of the optimal solution becomes
trivial because the bound is tight
Flow and Reparametrization

E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2                    Residual Graph
(positive coefficients)

Source (1)
Total Flow
bound on the           0           3         st-mincut cost = 8
optimal solution
3
a1                    a2     a1 = 1 a2 = 0

0
E (1,0) = 8
1            0
Sink (0)
Inference of the optimal solution becomes
trivial because the bound is tight
Example: Image Segmentation
E: {0,1}n → R
E(x) =   ∑ ci xi + ∑ cij xi(1-xj)       0 → fg
1 → bg
i         i,j

x* = arg min E(x)
x

How to minimize
E(x)?
Global Minimum (x*)
How does the code look like?

Graph *g;

For all pixels p
Source (0)
/* Add a node to the graph */

/* Set cost of terminal edges */
set_weights(nodeID(p), fgCost(p), bgCost(p));

end

end

g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p));         Sink (1)
// is the label of pixel p (0 or 1)
How does the code look like?

Graph *g;

For all pixels p
Source (0)
/* Add a node to the graph */

/* Set cost of terminal edges */                bgCost(a1)      bgCost(a2)
set_weights(nodeID(p), fgCost(p), bgCost(p));

end
a1                  a2
end
fgCost(a1)      fgCost(a2)
g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p));                      Sink (1)
// is the label of pixel p (0 or 1)
How does the code look like?

Graph *g;

For all pixels p
Source (0)
/* Add a node to the graph */

/* Set cost of terminal edges */                 bgCost(a1)               bgCost(a2)
set_weights(nodeID(p), fgCost(p), bgCost(p));
cost(p,q)
end
a1                           a2
end
fgCost(a1)               fgCost(a2)
g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p));                              Sink (1)
// is the label of pixel p (0 or 1)
How does the code look like?

Graph *g;

For all pixels p
Source (0)
/* Add a node to the graph */

/* Set cost of terminal edges */                 bgCost(a1)                bgCost(a2)
set_weights(nodeID(p), fgCost(p), bgCost(p));
cost(p,q)
end
a1                            a2
end
fgCost(a1)                fgCost(a2)
g->compute_maxflow();

label_p = g->is_connected_to_source(nodeID(p));                               Sink (1)
// is the label of pixel p (0 or 1)
a1 = bg a2 = fg
Image Segmentation in Video
s

E(x)         w pq
x*

t   =1

Image         Flow          Global Optimum
Image Segmentation in Video

Image

Global
Flow                             Optimum
Dynamic Energy Minimization
minimize
EA                                  SA

Can we do                              Recycling
better?                               Solutions

EB                                  SB
computationally
expensive operation

Boykov & Jolly ICCV’01, Kohli & Torr (ICCV05, PAMI07)
Dynamic Energy Minimization
minimize
EA                             SA
Reuse flow

differences
A and B            between                EB*     Simpler
energy
similar           A and B
cheaper
operation

EB                             SB
computationally        3 – 100000
Reparametrization
expensive operation     time speedup!

Kohli Torr (ICCV05, PAMI07)
Boykov & Jolly ICCV’01, Kohli & & Torr (ICCV05, PAMI07)
Dynamic Energy Minimization
Original Energy
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 2a1ā2 + ā1a2
Reparametrized Energy
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2

New Energy
E(a1,a2) = 2a1 + 5ā1+ 9a2 + 4ā2 + 7a1ā2 + ā1a2
New Reparametrized Energy
E(a1,a2) = 8 + ā1+ 3a2 + 3ā1a2 + 5a1ā2

Kohli Torr (ICCV05, PAMI07)
Boykov & Jolly ICCV’01, Kohli & & Torr (ICCV05, PAMI07)
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut
and energy minimization?

What problems can we solve
using st-mincut?

st-mincut based Move algorithms

Minimizing Energy Functions

• General Energy Functions
– NP-hard to minimize                      MAXCUT     NP-Hard
– Only approximate minimization
possible                                    Submodular
Functions

• Easy energy functions
– Solvable in polynomial time
– Submodular ~ O(n6)
Functions defined on trees

Space of Function
Minimization Problems
Submodular Set Functions
Let E= {a1,a2, .... an} be a set
2|E| = #subsets of E

Set function f  2|E|  ℝ
Submodular Set Functions
Let E= {a1,a2, .... an} be a set
2|E| = #subsets of E

Set function f  2|E|  ℝ is submodular if

f(A) + f(B)  f(AB) + f(AB) for all A,B  E

E
A                 B

Important Property
Sum of two submodular functions is submodular
Minimizing Submodular Functions

• Minimizing general submodular functions
– O(n5 Q + n6) where Q is function evaluation time
[Orlin, IPCO 2007]

• Symmetric submodular functions
– E (x) = E (1 - x)
– O(n3) [Queyranne 1998]

– Can be transformed to st-mincut
– One node per variable [ O(n3) complexity]
– Very low empirical running time
Submodular Pseudoboolean Functions
Function defined over boolean vectors x = {x1,x2, .... xn}

Definition:
• All functions for one boolean variable (f: {0,1} -> R) are submodular

• A function of two boolean variables (f: {0,1}2 -> R) is submodular if
f(0,1) + f(1,0)  f(0,0) + f(1,1)

• A general pseudoboolean function f  2n  ℝ is submodular if all
its projections fp are submodular i.e.

fp(0,1) + fp (1,0)  fp (0,0) + fp (1,1)
Functions

E(x) =   ∑ θi (xi)   +   ∑ θij (xi,xj)
i              i,j

For all ij   θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1)
Functions

E(x) =    ∑ θi (xi)   +   ∑ θij (xi,xj)
i              i,j

For all ij   θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1)

Equivalent (transformable)

E(x) =      ∑   ci xi +   ∑ cij xi(1-xj)         cij≥0
i             i,j

i.e. All submodular QPBFs are st-mincut solvable
How are they equivalent?
A = θij (0,0)            B = θij(0,1)             C = θij (1,0)           D = θij (1,1)

xj
0        1                  0     1              0     1              0      1
B+C
0    A         B            0     0     0         0    0    D-C       0      0
-A-D
xi                       = A +                     +                   +
1    C         D            1    C-A C-A          1    0    D-C       1      0     0

C-A                D-C

θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0)            B = θij(0,1)             C = θij (1,0)           D = θij (1,1)

xj
0        1                  0     1              0     1              0      1
B+C
0    A         B            0     0     0         0    0    D-C       0      0
-A-D
xi                       = A +                     +                   +
1    C         D            1    C-A C-A          1    0    D-C       1      0     0

C-A                D-C

θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0)            B = θij(0,1)             C = θij (1,0)           D = θij (1,1)

xj
0        1                  0     1              0     1              0      1
B+C
0    A         B            0     0     0         0    0    D-C       0      0
-A-D
xi                       = A +                     +                   +
1    C         D            1    C-A C-A          1    0    D-C       1      0     0

C-A                D-C

θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0)            B = θij(0,1)             C = θij (1,0)           D = θij (1,1)

xj
0        1                  0     1              0     1              0      1
B+C
0    A         B            0     0     0         0    0    D-C       0      0
-A-D
xi                       = A +                     +                   +
1    C         D            1    C-A C-A          1    0    D-C       1      0     0

C-A                D-C

θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij
How are they equivalent?
A = θij (0,0)            B = θij(0,1)             C = θij (1,0)           D = θij (1,1)

xj
0        1                  0     1              0     1              0      1
B+C
0    A         B            0     0     0         0    0    D-C       0      0
-A-D
xi                       = A +                     +                   +
1    C         D            1    C-A C-A          1    0    D-C       1      0     0

C-A                D-C

θij (xi,xj) = θij(0,0)
+ (θij(1,0)-θij(0,0)) xi + (θij(1,0)-θij(0,0)) xj
+ (θij(1,0) + θij(0,1) - θij(0,0) - θij(1,1)) (1-xi) xj

B+C-A-D  0 is true from the submodularity of θij
Functions
x in {0,1}n

E(x) =   ∑ θi (xi)    +   ∑ θij (xi,xj)
i               i,j

For all ij   θij(0,1) + θij (1,0)  θij (0,0) + θij (1,1)

Equivalent (transformable)

st-mincut
S

T
Minimizing Non-Submodular Functions

E(x) =      ∑ θi (xi)    +   ∑ θij (xi,xj)
i               i,j

θij(0,1) + θij (1,0)   ≤θ   ij   (0,0) + θij (1,1) for some ij

• Minimizing general non-submodular functions is
NP-hard.

• Commonly used method is to solve a relaxation
of the problem

[Slide credit: Carsten Rother]
Minimization using Roof-dual Relaxation

unary
E ({x p })    p ( x p )

 pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0)
   pq ( x p , xq )
pairwise submodular
~
   pq ( x p , xq )   ~            ~             ~           ~
 pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0)
pairwise nonsubmodular

[Slide credit: Carsten Rother]
Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({x p })    p ( x p )    E ' ({x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )           pq ( x p , xq )   pq (1  x p ,1  xq )

2
~                      ~
~                         pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq )    
2

[Slide credit: Carsten Rother]
Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({x p })    p ( x p )                   E ' ({x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )                        pq ( x p , xq )   pq (1  x p ,1  xq )

2
~                        ~
~                                    pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq )                 
2

~
f pq ( x p , xq )   pq ( x p ,1  xq )

~            ~            ~            ~
 pq (0,0)   pq (1,1)   pq (0,1)   pq (1,0)               f pq (0,1)  f pq (1,0)  f pq (0,0)  f pq (1,1)
Non- submodular                                                  Submodular
Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({x p })    p ( x p )    E ' ({x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )           pq ( x p , xq )   pq (1  x p ,1  xq )

2
~                      ~
~                         pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq )    
2

Property of the problem:
•
•       is submodular ! Ignore x p  1  x p
(solvable using st-mincut)
Minimization using Roof-dual Relaxation
Double number of variables: x p  x p , x p ( x p  1  x p )

E ({x p })    p ( x p )    E ' ({x p },{x p })  
 p ( x p )   p (1  x p )
2

   pq ( x p , xq )           pq ( x p , xq )   pq (1  x p ,1  xq )

2
~                      ~
~                         pq ( x p ,1  xq )   pq (1  x p , xq )
   pq ( x p , xq )    
2

Property of the solution:

xp  1 xp                    x p is the optimal label
Recap

• Exact minimization of Submodular QBFs
using graph cuts.

• Obtaining partially optimal solutions of non-
submodular QBFs using graph cuts.
But ...
• Need higher order energy functions to
model image structure
– Field of experts [Roth and Black]
• Many problems in computer vision involve
multiple labels

E(x) =   ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)
i             i,j           c

x ϵ Labels L = {l1, l2, … , lk}   Clique c  V
Transforming problems in QBFs

Pseudoboolean     Pseudoboolean
Functions         Functions

Multi-label      Pseudoboolean
Functions          Functions
Transforming problems in QBFs

Pseudoboolean     Pseudoboolean
Functions         Functions

Multi-label      Pseudoboolean
Functions          Functions
• Simple Example using Auxiliary variables

f(x) =     {   0 if all xi = 0
C1 otherwise
x ϵ L = {0,1}n

min f(x)         =         min C1a + C1 ∑ ā xi
x                       x,a ϵ {0,1}
Submodular Function                        Function

∑xi = 0                a=0 (ā=1)               f(x) = 0
∑xi ≥ 1                a=1 (ā=0)               f(x) = C1

min f(x)             =               min C1a + C1 ∑ ā xi
x                                 x,a ϵ {0,1}
Submodular Function                                         Function

C1∑xi

C1

1       2     3
∑xi

min f(x)             =               min C1a + C1 ∑ ā xi
x                                 x,a ϵ {0,1}
Submodular Function                                          Function

C1∑xi

a=0                              a=1
Lower envelop
of concave
C1                                                      functions is
concave

1       2     3
∑xi

min f(x)
x
=               min f1 (x)a + f2(x)ā
x,a ϵ {0,1}
Submodular Function                                          Function

f2(x)
a=1
f1(x)                Lower envelop
of concave
functions is
concave

1       2     3
∑xi

min f(x)
x
=               min f1 (x)a + f2(x)ā
x,a ϵ {0,1}
Submodular Function                                          Function

f2(x)

a=0                              a=1
f1(x)                Lower envelop
of concave
functions is
concave

1       2     3
∑xi
Transforming problems in QBFs

Pseudoboolean     Pseudoboolean
Functions         Functions

Multi-label      Pseudoboolean
Functions          Functions
Multi-label to Pseudo-boolean
So what is the problem?
Em (y1,y2, ..., yn)                       Eb (x1,x2, ..., xm)

yi ϵ L = {l1, l2, … , lk}                     xi ϵ L = {0,1}

Multi-label Problem                       Binary label Problem
such that:
Let Y and X be the set of feasible solutions, then
1. For each binary solution x ϵ X with finite energy there
exists exactly one multi-label solution y ϵ Y
-> One-One encoding function T:X->Y

2. arg min Em(y) = T(arg min Eb(x))
Multi-label to Pseudo-boolean
• Popular encoding scheme
[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]
Multi-label to Pseudo-boolean
• Popular encoding scheme
[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]

Ishikawa’s result:

E(y) =     ∑ θi (yi) + ∑ θij (yi,yj)
i             i,j

y ϵ Labels L = {l1, l2, … , lk}

Convex
θij (yi,yj) = g(|yi-yj|)           Function

g(|yi-yj|)
|yi-yj|
Multi-label to Pseudo-boolean
• Popular encoding scheme
[Roy and Cox ’98, Ishikawa ’03, Schlesinger & Flach ’06]

Schlesinger & Flach ’06:

E(y) =       ∑ θi (yi) + ∑ θij (yi,yj)
i                i,j

y ϵ Labels L = {l1, l2, … , lk}

θij(li+1,lj) + θij (li,lj+1)  θij (li,lj) + θij (li+1,lj+1)

Covers all Submodular
multi-label functions
More general than Ishikawa
Multi-label to Pseudo-boolean
Problems

• Applicability
– Only solves restricted class of energy functions
– Cannot handle Potts model potentials

• Computational Cost
– Very high computational cost
– Problem size = |Variables| x |Labels|
– Gray level image denoising (1 Mpixel image)
(~2.5 x 108 graph nodes)
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut
and energy minimization?

What problems can we solve
using st-mincut?

st-mincut based Move algorithms

St-mincut based Move algorithms

E(x) =   ∑ θi (xi) + ∑ θij (xi,xj)
i          i,j

x ϵ Labels L = {l1, l2, … , lk}

• Commonly used for solving non-submodular
multi-label problems
• Extremely efficient and produce good
solutions
• Not Exact: Produce local optima
Energy   Move Making Algorithms

Solution Space
Move Making Algorithms
Current Solution
Search
Neighbourhood
Optimal Move
Energy

Solution Space
Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc
Key Property
Energy

(t)

Move Space

Solution Space

Bigger move              • Better solutions
space                 • Finding the optimal move hard
Moves using Graph Cuts
Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy

Current Solution
Search
Neighbourhood

Move Space (t) : 2N                  N   Number of
Variables
L   Number of
Space of Solutions (x) : LN                    Labels
Moves using Graph Cuts
Expansion and Swap move algorithms
[Boykov Veksler and Zabih, PAMI 2001]
• Makes a series of changes to the solution (moves)
• Each move results in a solution with smaller energy

Current Solution

Move to new         Construct a move           How to
solution              function            minimize move
functions?
Minimize move function
to get optimal move
General Binary Moves

x = t x1 + (1- t) x2
New           Current         Second
solution        Solution        solution

Em(t) = E(t x1 + (1- t) x2)
Minimize over move variables t to get the
optimal move

Move energy is a submodular QPBF
(Exact Minimization Possible)

Boykov, Veksler and Zabih, PAMI 2001
Swap Move
•   Variables labeled α, β can swap their labels

[Boykov, Veksler, Zabih]
Swap Move
•    Variables labeled α, β can swap their labels
Tree
Ground
Swap Sky, House                          House
Sky

[Boykov, Veksler, Zabih]
Swap Move
•   Variables labeled α, β can swap their labels

• Move energy is submodular if:
– Unary Potentials: Arbitrary
– Pairwise potentials: Semimetric

θij (la,lb) ≥ 0
θij (la,lb) = 0           a = b

Examples: Potts model, Truncated Convex

[Boykov, Veksler, Zabih]
Expansion Move
•   Variables take label a or retain current label

[Boykov, Veksler, Zabih]
Expansion Move
•   Variables take label a or retain current label
Tree
Ground
House
Status:   Initialize with Tree
House
Sky
Expand Ground                        Sky

[Boykov, Veksler, Zabih]
Expansion Move
•   Variables take label a or retain current label

• Move energy is submodular if:
– Unary Potentials: Arbitrary
– Pairwise potentials: Metric                    Semi metric
+
Triangle
θij (la,lb) + θij (lb,lc) ≥ θij (la,lc)    Inequality

Examples: Potts model, Truncated linear

[Boykov, Veksler, Zabih]
General Binary Moves

x = t x1 + (1-t) x2
New             First                    Second
solution         solution                  solution

Minimize over move variables t

Move Type       First           Second        Guarantee
Solution         Solution
Expansion     Old solution      All alpha      Metric
Fusion       Any solution     Any solution      

Move functions can be non-submodular!!
Solving Continuous Problems using
Fusion Move

x = t x1 + (1-t) x2
x1, x2 can be continuous              Optical Flow
Example

Solution
from
Method 2   x2

Solution
from                          F               Final
Method 1   x1                       x          Solution

(Lempitsky et al. CVPR08, Woodford et al. CVPR08)
Range Moves
• Move variables can be multi-label

x = (t ==1) x1 + (t==2) x2 +… +(t==k) xk

• Optimal move found out by using the Ishikawa
• Useful for minimizing energies with truncated convex
pairwise potentials

T
θij (yi,yj) = min(|yi-yj|,T)
θij (yi,yj)
|yi-yj|

O. Veksler, CVPR 2007
Move Algorithms for Solving Higher
Order Energies
E(x) =    ∑ θi (xi) + ∑ θij (xi,xj) + ∑ θc (xc)
i             i,j               c

x ϵ Labels L = {l1, l2, … , lk}         Clique c  V

• Higher order functions give rise to higher order move
energies

• Move energies for certain classes of higher order
energies can be transformed to QPBFs.

[Kohli, Kumar and Torr, CVPR07]             [Kohli, Ladicky and Torr, CVPR08]
Outline of the Tutorial

The st-mincut problem

Connection between st-mincut
and energy minimization?

What problems can we solve
using st-mincut?

st-mincut based Move algorithms

Solving Mixed Programming Problems
x – binary image segmentation (xi ∊ {0,1})
ω – non-local parameter (lives in some large set Ω)

E(x,ω) = C(ω) + ∑ θi (ω, xi) +        ∑ θij (ω,xi,xj)
i               i,j
unary            pairwise
constant
potentials        potentials
≥0

ω                                              θi (ω, xi)
Pose                                              Shape
Prior
Stickman        Rough Shape
Model             Prior
Open Problems

Characterization of Problems Solvable using
st-mincut

• What functions can be transformed to
submodular QBFs?

Submodular
Functions

st-mincut
Equivalent
Minimizing General Higher Order
Functions

• We saw how simple higher order potentials
can be solved

• How more sophisticated higher order
potentials can be solved?
Summary
Exact
Transformation

Pseudoboolean Function
Problem             Or Relaxed
transformation
(partially optimal)

st-mincut
S

Sub-problem

T
Move making algorithms
Thanks. Questions?
Use of Higher order Potentials
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =
{    0 if xi=xj
C otherwise

8
E(6,6,6) = 0 + 0 = 0
7
Disparity     6
Labels
5

P1              P2          P3
Pixels

Stereo - Woodford et al. CVPR 2008
Use of Higher order Potentials
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =
{    0 if xi=xj
C otherwise

8
E(6,6,6) = 0 + 0 = 0
7
Disparity     6
E(6,7,7) = 1 + 0 = 1
Labels
5

P1              P2          P3
Pixels

Stereo - Woodford et al. CVPR 2008
Use of Higher order Potentials
E(x1,x2,x3) = θ12 (x1,x2) + θ23 (x2,x3)

θij (xi,xj) =
{    0 if xi=xj
C otherwise

8
E(6,6,6) = 0 + 0 = 0
7
Disparity     6
E(6,7,7) = 1 + 0 = 1
Labels                                              E(6,7,8) = 1 + 1 = 2
5

Pairwise potential
P1              P2          P3
penalize slanted
Pixels                      planar surfaces

Stereo - Woodford et al. CVPR 2008
Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc                   Transformation
T   function
E(x)        (t)

x
T(xc, t) = xn = xc + t
Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc                    Transformation
T    function
E(x)        (t)
Em   Move Energy

x
T(xc, t) = xn = xc + t
Em(t) = E(T(xc, t))
Computing the Optimal Move
Current Solution
Search
Neighbourhood
Optimal Move
xc                          Transformation
T      function
E(x)        (t)
Em      Move Energy

x
T(xc, t) = xn = xc + t
minimize
Em(t) = E(T(xc, t))                    t*
Optimal Move

```
DOCUMENT INFO
Shared By:
Categories:
Stats:
 views: 12 posted: 4/18/2011 language: English pages: 134
How are you planning on using Docstoc?