Docstoc

IanAndrew_subspace

Document Sample
IanAndrew_subspace Powered By Docstoc
					 Approximate Nearest Subspace Search
with Applications to Pattern Recognition

 Ronen Basri, Tal Hassner, Lihi Zelnik-Manor
        presented by Andrew Guillory and Ian Simon
                The Problem
• Given n linear subspaces Si:

              Z x0
               i
                T
                The Problem
• Given n linear subspaces Si:

              Z x0
               i
                T



• And a query point q:
                The Problem
• Given n linear subspaces Si:

              Z x0
                i
                 T



• And a query point q:



• Find the subspace Si that minimizes dist(Si,q).
                      Why?
• object appearance variation = subspace




  – fast queries on object database
                      Why?
• object appearance variation = subspace




  – fast queries on object database


• Other reasons?
                   Approach
• Solve by reduction to nearest neighbor.
  – point-to-point distances
                   Approach
• Solve by reduction to nearest neighbor.
  – point-to-point distances




                                   not actual reduction
                   Approach
• Solve by reduction to nearest neighbor.
  – point-to-point distances




                                   not actual reduction

• In higher-dimensional space.
            Point-Subspace Distance
• Use squared distance.


dist  x, S  
    2                        T   2
                         Z x
                  Z x  Z x 
                     T               T


                    x T ZZ T x
                   xx  ZZ 
                      T                  T


                 2hxx  hZZ 
                         T                   T
            Point-Subspace Distance
                                                                                 a11 
• Use squared distance.                                                          2
                                                                                a 
                                                                                 12 
                                                                                  
dist  x, S  
                                 2
                                                                       a1d    
    2                        T
                         Z x                        a11   a12    
                                                                             a1d 
                  Z x  Z x 
                     T               T
                                                    a12   a22       a2 d    a22 
                    x T ZZ T x
                                                 h
                                                                         
                                                                              2
                   xx  ZZ 
                      T                  T
                                                   a
                                                    1d    a2 d   
                                                                            
                                                                       add     
                 2hxx  hZZ 
                         T                   T
                                                                                 a2 d 
                                                                                 
                                                                                  
                                                                                 add 
                                                                                 2
                                                                                 
            Point-Subspace Distance
                                                                                 a11 
• Use squared distance.                                                          2
                                                                                a 
                                                                                 12 
                                                                                  
dist  x, S  
                                 2
                                                                       a1d    
    2                        T
                         Z x                        a11   a12    
                                                                             a1d 
                  Z x  Z x 
                     T               T
                                                    a12   a22       a2 d    a22 
                    x T ZZ T x
                                                 h
                                                                         
                                                                              2
                   xx  ZZ 
                      T                  T
                                                   a
                                                    1d    a2 d   
                                                                            
                                                                       add     
                 2hxx  hZZ 
                         T                   T
                                                                                 a2 d 
                                                                                 
                                                                                  
                                                                                 add 
• Squared point-subspace distance                                                2
                                                                                 
  can be represented as a dot product.
                        The Reduction
• Let:
                               Remember:
           
     u  h ZZ      T
                        
          
     v  h xx   T                                   
                                 dist 2 x, S   2h xx T  h ZZ T   
                            The Reduction
• Let:
                                           Remember:
     u  h ZZ         T
                            
     v  h xx     T                                           
                                             dist 2 x, S   2h xx T  h ZZ T   

• Then:

         dist 2 u, v           u  2u  v  v
                                       2           2


                             dist x, S   u  v
                                   2         2         2
                     The Reduction
dist u, v   dist x, S   u  v
    2            2           2        2
                                                 
                                          u  h ZZ T   
                                                
                                          v  h xx T
                     The Reduction
dist u, v   dist x, S   u  v
    2            2           2        2
                                                                       
                                                                u  h ZZ T   
                                          constant over query
                                                                      
                                                                v  h xx T
                     The Reduction
dist u, v   dist x, S   u  v
    2            2            2       2
                                                                       
                                                                u  h ZZ T   
                          ?               constant over query
                                                                      
                                                                v  h xx T
                         The Reduction
dist u, v   dist x, S   u  v
        2            2                   2   2
                                                                              
                                                                       u  h ZZ T   
                                     ?           constant over query
                                                                             
                                                                       v  h xx T
u
    2
                    
                    h ZZ  T     2


               hZZ  hZZ 
                     T               T


                  TrZZ ZZ 
                1
                        T           T
                                             ZTZ = I
                2
                     TrZZ 
                   1
                                T

                   2
                     TrZ Z 
                   1
                            T

                   2
            
                    1
                      d  k 
                    2
                         The Reduction
dist u, v   dist x, S   u  v
        2            2                   2    2
                                                                               
                                                                        u  h ZZ T   
                                     ?            constant over query
                                                                              
                                                                        v  h xx T
u
    2
                    
                    h ZZ  T     2


               hZZ  hZZ 
                     T               T


                  TrZZ ZZ 
                1
                        T           T
                                              ZTZ = I
                2
                     TrZZ 
                   1
                                T

                   2
                     TrZ Z 
                   1
                            T
                                             Z is d-by-(d-k), columns orthonormal.
                   2
            
                    1
                      d  k 
                    2
                         The Reduction
dist u, v   dist x, S   u  v
        2            2                   2    2
                                                                               
                                                                        u  h ZZ T   
                                     ?            constant over query
                                                                              
                                                                        v  h xx T
u
    2
                    
                    h ZZ  T     2


               hZZ  hZZ 
                     T               T


                  TrZZ ZZ 
                1
                        T           T
                                              ZTZ = I
                2
                     TrZZ 
                   1
                                T

                   2
                     TrZ Z 
                   1
                            T
                                             Z is d-by-(d-k), columns orthonormal.
                   2
            
                    1
                      d  k 
                    2
                        The Reduction
• For query point q:

  dist 2 u, v   dist 2 q, S  
                                      1
                                      2
                                       d k  q
                                                   4
                                                       
                        The Reduction
• For query point q:

  dist 2 u, v   dist 2 q, S  
                                      1
                                      2
                                       d k  q
                                                   4
                                                       


• Can we decrease the additive constant?
                    Observation 1
        
  Tr ZZ T  d  k


• All data points lie on a hyperplane.
                    Observation 1
          
  Tr ZZ T  d  k


• All data points lie on a hyperplane.

• Let:
                       d k             T q2 
         u  h ZZ T         I   v  h qq    I
                         d                   d 
                                                  

• Now the hyperplane contains the origin.
                     Observation 2
• After hyperplane
  projection:
                                   2
                T d k 
             h ZZ 
      2
  u                           I
                         d    
                   k d  k 
          
                     d 1


• All data points lie on a
  hypersphere.
                     Observation 2
• After hyperplane                     • Let:
  projection:
                                            1       k d  k  
                                                                T q2 
                                                                       I
                                   2
                T d k               v                     h qq 
             h ZZ 
      2
  u                           I            q
                                                2
                                                      d 1          d 
                         d                                           
                   k d  k 
          
                     d 1
                                       • Now the query point
                                         lies on the hypersphere.
• All data points lie on a
  hypersphere.
                     Observation 2
• After hyperplane                     • Let:
  projection:
                                            1       k d  k  
                                                                T q2 
                                                                       I
                                   2
                T d k               v                     h qq 
             h ZZ 
      2
  u                           I            q
                                                2
                                                      d 1          d 
                         d                                           
                   k d  k 
          
                     d 1
                                       • Now the query point
                                         lies on the hypersphere.
• All data points lie on a
  hypersphere.
         Reduction Geometry




• What is happening?
         Reduction Geometry




• What is happening?
                         Finally
            dist 2 u , v    dist 2 q, S   
                              1       k d  k 
             
                              q
                                  2
                                        d 1
                    k       k d  k  
                 1   k             
                    d        d 1    


• Additive constant depends only on dimension
  of points and subspaces.
• This applies to linear subspaces, all of the
  same dimension.
                    Extensions
• subspaces of different dimension
  – lines and planes, e.g.
  – Not all data points have the same norm.
     • Add extra dimension to fix this.
                    Extensions
• subspaces of different dimension
  – lines and planes, e.g.
  – Not all data points have the same norm.
     • Add extra dimension to fix this.


• affine subspaces
      Z iT x  b
  – Again, not all data points
    have the same norm.
               Approximate
          Nearest Neighbor Search
• Find point x with distance
  d(x, q) <= (1 + ε) mini d(xi,q)
• Tree based approaches: KD-trees, metric / ball
  trees, cover trees
• Locality sensitive hashing
• This paper uses multiple KD-Trees with
  (different) random projections
                 KD-Trees
• Decompose space into axis aligned rectangles




                                   Image from Dan Pelleg
          Random Projections
• Multiply data with a random matrix X with
  X(i,j) drawn from N(0,1)
• Several different justifications
  – Johnson-Lindenstrauss (data set that is small
    compared to dimensionality)
  – Compressed Sensing (data set that is sparse in
    some linear basis)
  – RP-Trees (data set that has small doubling
    dimension)
                    Results
• Two goals
  – show their method is fast
  – show nearest subspace is useful
• Four experiments
  – Synthetic Experiments
  – Image Approximation
  – Yale Faces
  – Yale Patches
Image Reconstruction
Yale Faces
           Questions / Issues
• Should random projections be applied before
  or after the reduction?
• Why does the effective distance error go down
  with the ambient dimensionality?
• The reduction tends to make query points far
  away from the points in the database. Are
  there better approximate nearest neighbor
  algorithms in this case?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:8/1/2012
language:
pages:39