Docstoc

svm1028t.ppt

Document Sample
svm1028t.ppt Powered By Docstoc
					    The Most Important Concept in
      Optimization (minimization)

 A point is said to be an optimal solution of a
  unconstrained minimization if there exists no
  decent direction

 A point is said to be an optimal solution of a
  constrained minimization if there exists no
  feasible decent direction
   There might exist decent direction but move
     along this direction will leave out the feasible
     region
                Minimum Principle


Let f : R n ! R be a convex and differentiable function
F ò R n be the feasible region.


x ã 2 argmin f (x) ( ) r f (x ã)(x à x ã) > 0 8x 2 F
         x2 F

Example:
           min (x à 1) 2 s:t: a ô x ô b
                      x 1 + x 26 4
min x 2 + x 2
      1     2
                    à x1 à x2 6 à 2
x2 R2                  x 1; x 2> 0
                        r f (x) = [2x 1; 2x 2]
                        r f (x ã) = [2; 2]



        ï


                ï   ï


                                         ï
      Minimization Problem
               vs.
Kuhn-Tucker Stationary-point Problem

 MP:   min f (x) such that
       x2 Ò
            g(x) 6 0


 KTSP: Find x 2 Ò; ë 2 R m such that
        r f (x) + ë 0r g(x) = 0
        ë 0g(x) = 0
        g(x) 6 0; ë > 0
              Lagrangian Function
               L (x; ë) = f (x) + ë 0g(x)

Let L (x; ë) = f (x) + ë 0g(x) and ë > 0
 If f ( x ) ; g( x ) are convex then L ( x; ë ) is convex.

 For a fixed ë > 0 , if x 2 arg minf L ( x; ë )j x 2 R n g
  then           ì
         @ (x;ë) ì
           L
             @x  ì = r f (x) + ë 0r g( x ) = 0
                   x= x

 Above result is a sufficient condition if L ( x; ë )
    is convex.
           KTSP with Equality Constraints?
         (Assume h(x) = 0 are linear functions)

         h(x) = 0 ,   h(x) 6 0 and à h(x) 6 0
KTSP: Find x 2 Ò; ë 2 R k; ì + ; ì          à   2 R m such that
         r f (x) + ë 0 g(x) + (ì
                     r                +   à ì à ) 0 h(x) = 0
                                                  r
     0                  0
   ë g(x) = 0; (ì + ) h(x) = 0; (ì à ) 0 à h(x)) = 0
                                       (
             g(x) 6 0; h(x) = 0
                   ë> ; ì + ; ì   à
                                      > 0
       KTSP with Equality Constraints

KTSP: Find x 2 Ò; ë 2 R k; ì 2 R m such that
       r f (x) + ë 0 g(x) + ì r h (x) = 0
                   r
       ë 0g(x) = 0; g(x) 6 0; h(x) = 0
                   ë> 0

 If ì = ì   +à ì à and ì + ; ì   à
                                      > 0 then
   ì is free variable
    Generalized Lagrangian Function
          L (x; ë; ì ) = f (x) + ë 0g(x) + ì 0h(x)

Let L (x; ë; ì ) = f (x) + ë 0g(x) + ì 0h(x) and ë > 0
 If f ( x ) ; g( x ) are convex and h ( x ) is linear then
  L ( x; ë; ì ) is convex.
 For fixed ë > 0; ì , if x 2 arg minf L ( x; ë; ì )j x 2 R n g
  then
                   ì
        @ (x;ë;ì ) ì
         L
           @x      ì          = r f (x) + ë 0r g( x ) + ì 0r h ( x ) = 0
                       x= x

 Above result is a sufficient condition if L ( x; ë; ì )
  is convex.
Lagrangian Dual Problem


 max min L (x; ë; ì )
   ë;ì x 2 Ò
  subject to   ë> 0
    Lagrangian Dual Problem

        max min L (x; ë; ì )
         ë;ì x 2 Ò
        subject to     ë> 0

          max ò(ë; ì )
           ë;ì
          subject to   ë> 0
where   ò(ë; ì ) = inf L (x; ë; ì )
                     x2Ò
         Weak Duality Theorem

Let x 2 Ò be a feasible solution of the primal
problem and (ë; ì ) a feasible solution of the
dual problem. Then f ( x ) > ò( ë; ì )

   ò(ë; ì ) = inf L (x; ë; ì ) ô L (x ; ë; ì )
                                    à
               x2 Ò

Corollary: supf ò(ë; ì ) j ë > 0 g
          6 inf f f (x)j g(x) 6 0; h(x) = 0g
         Weak Duality Theorem

Corollary: If f (x ) = ò(ë ; ì ) where ë ã> 0
                 ã         ã   ã

and g(x ) 0; h(x ) = 0 , then x
        ã 6         ã              ã and (ë ã; ì ã)


solve the primal and dual problem respectively.
In this case,
               0 6 ë ? g(x) 6 0
     Saddle Point of Lagrangian


Let x 2 Ò ; ë
     ã          ã>
                     0; ì 2 R m satisfying
                         ã



 L (x ã; ë; ì ) 6 L (x ã; ë ã; ì ã) 6 L (x; ë ã; ì ã);
 8 x 2 Ò; ë      > 0: Then (x ã; ë ã; ì ã) is called
The saddle point of the Lagrangian function
     Dual Problem of Linear Program

 Primal LP          min       px  0
                    x2 R n
              subject to Ax > b; x > 0

                              0
 Dual LP          max b ë
                 ë2Rm
             subject to      A ë 6 p; ë > 0
                              0


※ All duality theorems hold and work perfectly!
         Application of LP Duality (I)
                Farkas’ Lemma

For any matrix A 2 R m â n and any vector b 2 R m ;
either
         Ax 6 0; b0x > 0 has a soluti on
or
          0
         A ë = b;    ë > 0 has a soluti on

 but never both.
       Application of LP Duality (II)
      LSQ-Normal Equation Always Has a Solution

For any matrix A 2 R m â n and any vector b 2 R m ;
consider min jj Ax à bjj 2
                            2
           x2 R n

  x ã 2 arg minf jjAx à bjj 2g ,
                            2      A0 ã = A0
                                    Ax     b


 Claim: A 0      0 always has a solution.
           Ax = A b
      Dual Problem of Strictly
     Convex Quadratic Program
Primal QP       min      1 0       0
                         2x Qx   + px
                x2 R n
                subject to    Ax 6 b
With strictly convex assumption, we have
Dual QP
               1 0        0   à1    0       0
   max à       2(p +     ë A)Q (A ë + p) à ë b
  subject to       ë> 0

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:4/28/2013
language:English
pages:17