# Primal-Dual Interior-Point Methods for Solving Basis Pursuit

Document Sample

Introduction
Interior-Point methods
Finite Termination
Computational results
Conclusions

Primal-Dual Interior-Point Methods for Solving
Basis Pursuit

Ewout van den Berg
Department of Computer Science, UBC

Supervisors: Michael P. Friedlander, Uri Ascher

April 12, 2007

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Context
Finite Termination
Formulation of Linear Program
Computational results
Conclusions

The Basis Pursuit problem

Matrix Φ, unit-norm columns

Given b = Φs0 and Φ, recover s0 ;
Underdetermined, inﬁnitely many solutions; ﬁnd the sparsest

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Context
Finite Termination
Formulation of Linear Program
Computational results
Conclusions

The Basis Pursuit problem

Matrix Φ, unit-norm columns

Given b = Φs0 and Φ, recover s0 ;
Underdetermined, inﬁnitely many solutions; ﬁnd the sparsest
Solve minimize              s   0   subject to Φs = b
s

Ewout van den Berg Department of Computer Science, UBC     Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Context
Finite Termination
Formulation of Linear Program
Computational results
Conclusions

The Basis Pursuit problem

Matrix Φ, unit-norm columns

Given b = Φs0 and Φ, recover s0 ;
Underdetermined, inﬁnitely many solutions; ﬁnd the sparsest
Solve minimize              s   0   subject to Φs = b
s
Solve minimize              s   1   subject to Φs = b, instead
s
Equivalence holds when s0 is sparse enough; eg. Donoho
Many applications, for example geophysics with huge problems

Ewout van den Berg Department of Computer Science, UBC     Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Context
Finite Termination
Formulation of Linear Program
Computational results
Conclusions

Linear Program for minimize s                                         1    subject to Φs = b

Split s = u − v with u, v ≥ 0 and
u
deﬁne x =        , and m × n matrix A = [Φ − Φ].
v
Rewrite s        1   = u        1   + v        1   =     ui +         vi = c T x.

Ewout van den Berg Department of Computer Science, UBC          Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Context
Finite Termination
Formulation of Linear Program
Computational results
Conclusions

Linear Program for minimize s                                         1    subject to Φs = b

Split s = u − v with u, v ≥ 0 and
u
deﬁne x =        , and m × n matrix A = [Φ − Φ].
v
Rewrite s        1   = u        1   + v        1   =     ui +         vi = c T x.
Linear Program:

minimize c T x subject to Ax = b, x ≥ 0
x

Dual problem

maximize b T y subject to AT y + s = c, s ≥ 0
y

Ewout van den Berg Department of Computer Science, UBC          Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Interior-Point Framework
Finite Termination
Stopping Criteria
Computational results
Conclusions

Framework

1: Given initial x, y and s
2: for k = 0, 1, . . . do
3:   Solve
0 AT I
                                
∆x         0
 A 0 0   ∆y  =           0      
S 0 X          ∆s     −XSe + σµe

4:   Determine line search parameter α
5:   Update x, y , and s
6:   Check convergence criteria
7: end for

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Interior-Point Framework
Finite Termination
Stopping Criteria
Computational results
Conclusions

Stopping Criteria

The following criteria are used to stop the algorithm
Limit the number of iterations
Relative progress in x, y , and s between successive iterations
Threshold the primal-dual gap µ = x T s which is zero at the
optimum
Condition number of the matrix; becomes unboundedly
ill-conditioned

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Finite Termination
Indicator functions
Computational results
Conclusions

Finite Termination

Suppose P is the set of indices i such that xi∗ = 0.
Denote by AP those columns of A whose index is in P.
∗
Then, AP xP = b, since xj = 0 for j ∈ P.

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Finite Termination
Indicator functions
Computational results
Conclusions

Finite Termination

Suppose P is the set of indices i such that xi∗ = 0.
Denote by AP those columns of A whose index is in P.
∗
Then, AP xP = b, since xj = 0 for j ∈ P.
Use an approximation I for P and solve
1
minimize           AI w − b 2 .
2
w         2
If the norm AI w ∗ − b is suﬃciently small and w ∗ ≥ 0, we
set xI = w ∗ and xI c = 0 and accept this as the solution to
the LP.

Ewout van den Berg Department of Computer Science, UBC     Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Finite Termination
Indicator functions
Computational results
Conclusions

Finite Termination

Iterates x(k)                                                 Iterates s(k)
2                                                           2
10                                                          10

0                                                           0
10                                                          10

−2                                                          −2
10                                                          10

−4                                                          −4
10                                                          10

−6                                                          −6
10                                                          10

−8                                                          −8
10                                                          10
0    50       100       150      200         250            0      50       100       150      200       250
Iteration                                                     Iteration

Ewout van den Berg Department of Computer Science, UBC           Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Finite Termination
Indicator functions
Computational results
Conclusions

Indicator functions

Method                  Year       Formulation
Primal                  1986       {i | xi ≥ τ }
Dual                    1991       {i | si < τ }
Ye                      1991       {i | si < xi }
Primal-dual             1991       {i | xi /si > τ }
Tapia primal            1991       {i | xik /xik−1 > τ }
Tapia dual              1991       {i | sik /sik−1 < τ }
Mehrotra-Ye             1993       {i | |xik − xik−1 |/xik−1 < |sik − sik−1 |/sik−1 }
s k−1 |x k − xik−1 |
                                ﬀ
Generalized M-Y            –         i | ik−1 · ik               <τ
xi        |si − sik−1 |
Gap                     2007       {i | si < τ (s)}

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Finite Termination
Indicator functions
Computational results
Conclusions

Indicator functions

Indicator functions should
not be too sensitive with respect to its parameters
be able to indicate the correct set before a stopping criterion
is reached
be conservative in their estimation
be easy to compute

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods    Comparison
Finite Termination    Exponentially decreasing entries
Computational results    Condition of signal
Conclusions

Comparison

Matrix of size m × 3, 000; FT = Finite Termination;

m     NNZ        Long-Step          Mehrotra.PC         SparseLab      CPLEX
—    FT            —       FT                        Barrier
300         18       10     6            5        4                  6         25
725         64       50    34           25       18                 31        125
1150        313      171   140           86       79                119        411
1575        521      281   231          163      136                  –        809
2000        796      423   332          230      187                323       1176
2425       1177      687   578          440      449                618       1978
2850       1813      975   849          625      667                868       2346
Runtime in seconds

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods    Comparison
Finite Termination    Exponentially decreasing entries
Computational results    Condition of signal
Conclusions

Comparison

Matrix of size m × 3, 000; FT = Finite Termination;

m     NNZ        Long-Step          Mehrotra.PC         SparseLab      CPLEX
—    FT            —       FT                        Barrier
300         18       10     6            5        4                  6         25
725         64       50    34           25       18                 31        125
1150        313      171   140           86       79                119        411
1575        521      281   231          163      136                  –        809
2000        796      423   332          230      187                323       1176
2425       1177      687   578          440      449                618       1978
2850       1813      975   849          625      667                868       2346
Runtime in seconds

Works quite well, but . . .

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods    Comparison
Finite Termination    Exponentially decreasing entries
Computational results    Condition of signal
Conclusions

Exponentially decreasing entries
When b = Ax0 with the entries of x0 given by x0 (j) = αe jβ for
1 ≤ j ≤ k, and zero otherwise, solving the LP gives us the
following trajectories.

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods    Comparison
Finite Termination    Exponentially decreasing entries
Computational results    Condition of signal
Conclusions

Exponentially decreasing entries
When b = Ax0 with the entries of x0 given by x0 (j) = αe jβ for
1 ≤ j ≤ k, and zero otherwise, solving the LP gives us the
following trajectories.
2                                                                  2
10                                                                 10

0                                                                  0
10                                                                 10

−2                                                                 −2
10                                                                 10

−4                                                                 −4
10                                                                 10
x

s
−6                                                                 −6
10                                                                 10

−8                                                                 −8
10                                                                 10

−10                                                                −10
10                                                                 10
0      100   200      300        400      500      600             0   100    200       300      400    500     600
Iteration                                                         Iteration

Ewout van den Berg Department of Computer Science, UBC             Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods        Comparison
Finite Termination        Exponentially decreasing entries
Computational results        Condition of signal
Conclusions

Condition of signal

600
Random Φ with unit-norm
500                                                        columns
400                                                        10,000 randomly generated s0
300                                                                    max{|s0 (i)|}
ρ=      min{|s0 (i)| | s0 (i)=0}
200
Plotted ρ versus number of
100                                                        iterations before ﬁnite
0 0             2             4                 6        termination succeeded
10            10            10             10

Ewout van den Berg Department of Computer Science, UBC        Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Computational results
Conclusions

Conclusions

From various tests, we found the Tapia Primal-Dual and
Mehrotra-Ye indicators to work very well;

[1][3][2]

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Computational results
Conclusions

Conclusions

From various tests, we found the Tapia Primal-Dual and
Mehrotra-Ye indicators to work very well;
Primal-dual interior-point methods inherently require many
iterations when the entries in the solution cover a wide range
of magnitudes;

[1][3][2]

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Computational results
Conclusions

Conclusions

From various tests, we found the Tapia Primal-Dual and
Mehrotra-Ye indicators to work very well;
Primal-dual interior-point methods inherently require many
iterations when the entries in the solution cover a wide range
of magnitudes;
In general the use of ﬁnite termination techniques improve
performance considerably;

[1][3][2]

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Computational results
Conclusions

Conclusions

From various tests, we found the Tapia Primal-Dual and
Mehrotra-Ye indicators to work very well;
Primal-dual interior-point methods inherently require many
iterations when the entries in the solution cover a wide range
of magnitudes;
In general the use of ﬁnite termination techniques improve
performance considerably;
Future work, consider the constraint Ax − b ≤ ε instead of
Ax = b.
[1][3][2]

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit
Introduction
Interior-Point methods
Finite Termination
Computational results
Conclusions

Bibliography

David L. Donoho and Michael Elad.
Optimally sparse representation in general (nonorthogonal)
dictionaries via 1 minimization.
PNAS, 100(5):2197–2202, March 2003.
A. S. El-Bakry, R. A. Tapia, and Y. Zhang.
A study of indicators for identifying zero variables in
interior-point methods.
SIAM Review, 36(1):45–72, 1994.
Stephen J. Wright.
Primal-Dual Interior-Point Methods.
SIAM, 1997.

Ewout van den Berg Department of Computer Science, UBC    Primal-Dual Interior-Point Methods for Solving Basis Pursuit

DOCUMENT INFO
Shared By:
Categories:
Stats:
 views: 13 posted: 2/9/2010 language: English pages: 23