# svm1028t.ppt

Document Sample

```					    The Most Important Concept in
Optimization (minimization)

 A point is said to be an optimal solution of a
unconstrained minimization if there exists no
decent direction

 A point is said to be an optimal solution of a
constrained minimization if there exists no
feasible decent direction
 There might exist decent direction but move
along this direction will leave out the feasible
region
Minimum Principle

Let f : R n ! R be a convex and differentiable function
F ò R n be the feasible region.

x ã 2 argmin f (x) ( ) r f (x ã)(x à x ã) > 0 8x 2 F
x2 F

Example:
min (x à 1) 2 s:t: a ô x ô b
x 1 + x 26 4
min x 2 + x 2
1     2
à x1 à x2 6 à 2
x2 R2                  x 1; x 2> 0
r f (x) = [2x 1; 2x 2]
r f (x ã) = [2; 2]

ï

ï   ï

ï
Minimization Problem
vs.
Kuhn-Tucker Stationary-point Problem

MP:   min f (x) such that
x2 Ò
g(x) 6 0

KTSP: Find x 2 Ò; ë 2 R m such that
r f (x) + ë 0r g(x) = 0
ë 0g(x) = 0
g(x) 6 0; ë > 0
Lagrangian Function
L (x; ë) = f (x) + ë 0g(x)

Let L (x; ë) = f (x) + ë 0g(x) and ë > 0
 If f ( x ) ; g( x ) are convex then L ( x; ë ) is convex.

 For a fixed ë > 0 , if x 2 arg minf L ( x; ë )j x 2 R n g
then           ì
@ (x;ë) ì
L
@x  ì = r f (x) + ë 0r g( x ) = 0
x= x

 Above result is a sufficient condition if L ( x; ë )
is convex.
KTSP with Equality Constraints?
(Assume h(x) = 0 are linear functions)

h(x) = 0 ,   h(x) 6 0 and à h(x) 6 0
KTSP: Find x 2 Ò; ë 2 R k; ì + ; ì          à   2 R m such that
r f (x) + ë 0 g(x) + (ì
r                +   à ì à ) 0 h(x) = 0
r
0                  0
ë g(x) = 0; (ì + ) h(x) = 0; (ì à ) 0 à h(x)) = 0
(
g(x) 6 0; h(x) = 0
ë> ; ì + ; ì   à
> 0
KTSP with Equality Constraints

KTSP: Find x 2 Ò; ë 2 R k; ì 2 R m such that
r f (x) + ë 0 g(x) + ì r h (x) = 0
r
ë 0g(x) = 0; g(x) 6 0; h(x) = 0
ë> 0

 If ì = ì   +à ì à and ì + ; ì   à
> 0 then
ì is free variable
Generalized Lagrangian Function
L (x; ë; ì ) = f (x) + ë 0g(x) + ì 0h(x)

Let L (x; ë; ì ) = f (x) + ë 0g(x) + ì 0h(x) and ë > 0
 If f ( x ) ; g( x ) are convex and h ( x ) is linear then
L ( x; ë; ì ) is convex.
 For fixed ë > 0; ì , if x 2 arg minf L ( x; ë; ì )j x 2 R n g
then
ì
@ (x;ë;ì ) ì
L
@x      ì          = r f (x) + ë 0r g( x ) + ì 0r h ( x ) = 0
x= x

 Above result is a sufficient condition if L ( x; ë; ì )
is convex.
Lagrangian Dual Problem

max min L (x; ë; ì )
ë;ì x 2 Ò
subject to   ë> 0
Lagrangian Dual Problem

max min L (x; ë; ì )
ë;ì x 2 Ò
subject to     ë> 0

max ò(ë; ì )
ë;ì
subject to   ë> 0
where   ò(ë; ì ) = inf L (x; ë; ì )
x2Ò
Weak Duality Theorem

Let x 2 Ò be a feasible solution of the primal
problem and (ë; ì ) a feasible solution of the
dual problem. Then f ( x ) > ò( ë; ì )

ò(ë; ì ) = inf L (x; ë; ì ) ô L (x ; ë; ì )
à
x2 Ò

Corollary: supf ò(ë; ì ) j ë > 0 g
6 inf f f (x)j g(x) 6 0; h(x) = 0g
Weak Duality Theorem

Corollary: If f (x ) = ò(ë ; ì ) where ë ã> 0
ã         ã   ã

and g(x ) 0; h(x ) = 0 , then x
ã 6         ã              ã and (ë ã; ì ã)

solve the primal and dual problem respectively.
In this case,
0 6 ë ? g(x) 6 0

Let x 2 Ò ; ë
ã          ã>
0; ì 2 R m satisfying
ã

L (x ã; ë; ì ) 6 L (x ã; ë ã; ì ã) 6 L (x; ë ã; ì ã);
8 x 2 Ò; ë      > 0: Then (x ã; ë ã; ì ã) is called
The saddle point of the Lagrangian function
Dual Problem of Linear Program

Primal LP          min       px  0
x2 R n
subject to Ax > b; x > 0

0
Dual LP          max b ë
ë2Rm
subject to      A ë 6 p; ë > 0
0

※ All duality theorems hold and work perfectly!
Application of LP Duality (I)
Farkas’ Lemma

For any matrix A 2 R m â n and any vector b 2 R m ;
either
Ax 6 0; b0x > 0 has a soluti on
or
0
A ë = b;    ë > 0 has a soluti on

but never both.
Application of LP Duality (II)
LSQ-Normal Equation Always Has a Solution

For any matrix A 2 R m â n and any vector b 2 R m ;
consider min jj Ax à bjj 2
2
x2 R n

x ã 2 arg minf jjAx à bjj 2g ,
2      A0 ã = A0
Ax     b

Claim: A 0      0 always has a solution.
Ax = A b
Dual Problem of Strictly
Primal QP       min      1 0       0
2x Qx   + px
x2 R n
subject to    Ax 6 b
With strictly convex assumption, we have
Dual QP
1 0        0   à1    0       0
max à       2(p +     ë A)Q (A ë + p) à ë b
subject to       ë> 0

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 3 posted: 4/28/2013 language: English pages: 17