# IanAndrew_subspace

Document Sample

```					 Approximate Nearest Subspace Search
with Applications to Pattern Recognition

Ronen Basri, Tal Hassner, Lihi Zelnik-Manor
presented by Andrew Guillory and Ian Simon
The Problem
• Given n linear subspaces Si:

Z x0
i
T
The Problem
• Given n linear subspaces Si:

Z x0
i
T

• And a query point q:
The Problem
• Given n linear subspaces Si:

Z x0
i
T

• And a query point q:

• Find the subspace Si that minimizes dist(Si,q).
Why?
• object appearance variation = subspace

– fast queries on object database
Why?
• object appearance variation = subspace

– fast queries on object database

• Other reasons?
Approach
• Solve by reduction to nearest neighbor.
– point-to-point distances
Approach
• Solve by reduction to nearest neighbor.
– point-to-point distances

not actual reduction
Approach
• Solve by reduction to nearest neighbor.
– point-to-point distances

not actual reduction

• In higher-dimensional space.
Point-Subspace Distance
• Use squared distance.

dist  x, S  
2                        T   2
Z x
     Z x  Z x 
T               T

       x T ZZ T x
      xx  ZZ 
T                  T

    2hxx  hZZ 
T                   T
Point-Subspace Distance
 a11 
• Use squared distance.                                                          2
a 
 12 
  
dist  x, S  
2
a1d    
2                        T
Z x                        a11   a12    
                           a1d 
     Z x  Z x 
T               T
  a12   a22       a2 d    a22 
       x T ZZ T x
h
                     
                            2
      xx  ZZ 
T                  T
 a
  1d    a2 d   

add     
    2hxx  hZZ 
T                   T
 a2 d 
 
  
 2
 
Point-Subspace Distance
 a11 
• Use squared distance.                                                          2
a 
 12 
  
dist  x, S  
2
a1d    
2                        T
Z x                        a11   a12    
                           a1d 
     Z x  Z x 
T               T
  a12   a22       a2 d    a22 
       x T ZZ T x
h
                     
                            2
      xx  ZZ 
T                  T
 a
  1d    a2 d   

add     
    2hxx  hZZ 
T                   T
 a2 d 
 
  
• Squared point-subspace distance                                                2
 
can be represented as a dot product.
The Reduction
• Let:
Remember:

u  h ZZ      T

 
v  h xx   T                                   
dist 2 x, S   2h xx T  h ZZ T   
The Reduction
• Let:
Remember:
u  h ZZ         T

v  h xx     T                                           
dist 2 x, S   2h xx T  h ZZ T   

• Then:

dist 2 u, v           u  2u  v  v
2           2

 dist x, S   u  v
2         2         2
The Reduction
dist u, v   dist x, S   u  v
2            2           2        2

u  h ZZ T   
 
v  h xx T
The Reduction
dist u, v   dist x, S   u  v
2            2           2        2

u  h ZZ T   
constant over query
 
v  h xx T
The Reduction
dist u, v   dist x, S   u  v
2            2            2       2

u  h ZZ T   
?               constant over query
 
v  h xx T
The Reduction
dist u, v   dist x, S   u  v
2            2                   2   2

u  h ZZ T   
?           constant over query
 
v  h xx T
u
2
        
h ZZ  T     2

   hZZ  hZZ 
T               T

TrZZ ZZ 
1
            T           T
ZTZ = I
2
TrZZ 
1
                    T

2
TrZ Z 
1
                T

2

1
d  k 
2
The Reduction
dist u, v   dist x, S   u  v
2            2                   2    2

u  h ZZ T   
?            constant over query
 
v  h xx T
u
2
        
h ZZ  T     2

   hZZ  hZZ 
T               T

TrZZ ZZ 
1
            T           T
ZTZ = I
2
TrZZ 
1
                    T

2
TrZ Z 
1
                T
Z is d-by-(d-k), columns orthonormal.
2

1
d  k 
2
The Reduction
dist u, v   dist x, S   u  v
2            2                   2    2

u  h ZZ T   
?            constant over query
 
v  h xx T
u
2
        
h ZZ  T     2

   hZZ  hZZ 
T               T

TrZZ ZZ 
1
            T           T
ZTZ = I
2
TrZZ 
1
                    T

2
TrZ Z 
1
                T
Z is d-by-(d-k), columns orthonormal.
2

1
d  k 
2
The Reduction
• For query point q:

dist 2 u, v   dist 2 q, S  
1
2
d k  q
4

The Reduction
• For query point q:

dist 2 u, v   dist 2 q, S  
1
2
d k  q
4


• Can we decrease the additive constant?
Observation 1
    
Tr ZZ T  d  k

• All data points lie on a hyperplane.
Observation 1
      
Tr ZZ T  d  k

• All data points lie on a hyperplane.

• Let:
        d k             T q2 
u  h ZZ T         I   v  h qq    I
          d                   d 
         

• Now the hyperplane contains the origin.
Observation 2
• After hyperplane
projection:
2
 T d k 
   h ZZ 
2
u                           I
          d    
k d  k 

d 1

• All data points lie on a
hypersphere.
Observation 2
• After hyperplane                     • Let:
projection:
1       k d  k  
 T q2 
I
2
 T d k               v                     h qq 
   h ZZ 
2
u                           I            q
2
d 1          d 
          d                                           
k d  k 

d 1
• Now the query point
lies on the hypersphere.
• All data points lie on a
hypersphere.
Observation 2
• After hyperplane                     • Let:
projection:
1       k d  k  
 T q2 
I
2
 T d k               v                     h qq 
   h ZZ 
2
u                           I            q
2
d 1          d 
          d                                           
k d  k 

d 1
• Now the query point
lies on the hypersphere.
• All data points lie on a
hypersphere.
Reduction Geometry

• What is happening?
Reduction Geometry

• What is happening?
Finally
dist 2 u , v    dist 2 q, S   
1       k d  k 
 
q
2
d 1
 k       k d  k  
     1   k             
 d        d 1    

• Additive constant depends only on dimension
of points and subspaces.
• This applies to linear subspaces, all of the
same dimension.
Extensions
• subspaces of different dimension
– lines and planes, e.g.
– Not all data points have the same norm.
• Add extra dimension to fix this.
Extensions
• subspaces of different dimension
– lines and planes, e.g.
– Not all data points have the same norm.
• Add extra dimension to fix this.

• affine subspaces
Z iT x  b
– Again, not all data points
have the same norm.
Approximate
Nearest Neighbor Search
• Find point x with distance
d(x, q) <= (1 + ε) mini d(xi,q)
• Tree based approaches: KD-trees, metric / ball
trees, cover trees
• Locality sensitive hashing
• This paper uses multiple KD-Trees with
(different) random projections
KD-Trees
• Decompose space into axis aligned rectangles

Image from Dan Pelleg
Random Projections
• Multiply data with a random matrix X with
X(i,j) drawn from N(0,1)
• Several different justifications
– Johnson-Lindenstrauss (data set that is small
compared to dimensionality)
– Compressed Sensing (data set that is sparse in
some linear basis)
– RP-Trees (data set that has small doubling
dimension)
Results
• Two goals
– show their method is fast
– show nearest subspace is useful
• Four experiments
– Synthetic Experiments
– Image Approximation
– Yale Faces
– Yale Patches
Image Reconstruction
Yale Faces
Questions / Issues
• Should random projections be applied before
or after the reduction?
• Why does the effective distance error go down
with the ambient dimensionality?
• The reduction tends to make query points far
away from the points in the database. Are
there better approximate nearest neighbor
algorithms in this case?

```
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
 views: 3 posted: 8/1/2012 language: pages: 39