PowerPoint Presentation by 8tD2aO

VIEWS: 4 PAGES: 24

									             Dimensionality Reduction
             and Feature Construction

• Principal components analysis (PCA)
   – Reading: L. I. Smith, A tutorial on principal
     components analysis (on class website)

   – PCA used to reduce dimensions of data without much
     loss of information.

   – Used in machine learning and in signal processing and
     image compression (among other things).
PCA is “an orthogonal linear transformation that transfers
the data to a new coordinate system such that the greatest
variance by any projection of the data comes to lie on the
first coordinate (first principal component), the second
greatest variance lies on the second coordinate (second
principal component), and so on.”
                Background for PCA
• Suppose attributes are A1 and A2, and we have n training
  examples. x’s denote values of A1 and y’s denote values of
  A2 over the training examples.

• Variance of an attribute:

                              n

                              ( xi  x ) 2
               var( A1 )    i 1
                                    (n  1)
• Covariance of two attributes:
                             n

                             ( x  x )( y
                                   i             i    y)
         cov( A1 , A2 )    i 1
                                       (n  1)



• If covariance is positive, both dimensions increase
  together. If negative, as one increases, the other decreases.
  Zero: independent of each other.
• Covariance matrix
   – Suppose we have n attributes, A1, ..., An.



   – Covariance matrix:

        C nn  (ci , j ), whereci , j  cov(Ai , Aj )
 cov(H , H ) cov(H , M ) 

 cov(M , H ) cov(M , M ) 
                          
                         


  var(H ) 104.5 
  104.5 var(M ) 
               
                

  47.7 104.5 
 104.5 370 
            
             

  Covariance matrix
• Eigenvectors:
   – Let M be an nn matrix.
       • v is an eigenvector of M if M  v = v
       •  is called the eigenvalue associated with v

    – For any eigenvector v of M and scalar a,
                      M  av  av

    – Thus you can always choose eigenvectors of length 1:

                         v1  ... vn  1
                           2         2



    – If M has any eigenvectors, it has n of them, and they are
      orthogonal to one another.

    – Thus eigenvectors can be used as a new basis for a n-dimensional
      vector space.
                            PCA

1. Given original data set S = {x1, ..., xk}, produce new set
   by subtracting the mean of attribute Ai from each xi.




        Mean: 1.81   1.91            Mean:   0     0
2. Calculate the covariance matrix:
                    x                  y
             x
             y



3. Calculate the (unit) eigenvectors and eigenvalues of the
   covariance matrix:
Eigenvector with largest
eigenvalue traces
linear pattern in data
4. Order eigenvectors by eigenvalue, highest to lowest.

                  .677873399
                  .735178956   1.28402771
           v1               
                             

                  .735178956
                 .677873399    .0490833989
           v2               
                             


   In general, you get n components. To reduce
   dimensionality to p, ignore np components at the bottom
   of the list.
Construct new feature vector.
Feature vector = (v1, v2, ...vp)


                     .677873399  .735178956
                     .735178956 .677873399 
  FeatureVector1                            
                                             

  or reduced dimension feature vector:


                      .677873399
                      .735178956
  FeatureVector 2               
                                 
  5. Derive the new data set.

      TransformedData = RowFeatureVector  RowDataAdjust

                              .677873399  .735178956
                              .735178956 .677873399 
        RowFeatureVector1                            
                                                      



        RowFeatureVector2   .677873399  .735178956


                 .69  1.31 .39 .09 1.29 .49 .19  .81  .31  .71 
                 .49  1.21 .99 .29 1.09 .79  .31  .81  .31  1.01
RowDataAdjust                                                       
                                                                     



          This gives original data in terms of chosen
          components (eigenvectors)—that is, along these axes.
    Reconstructing the original data
We did:
TransformedData = RowFeatureVector  RowDataAdjust

so we can do
RowDataAdjust = RowFeatureVector -1 
  TransformedData

= RowFeatureVector T  TransformedData

and
RowDataOriginal = RowDataAdjust + OriginalMean
   Example: Linear discrimination using PCA
             for face recognition

1. Preprocessing: “Normalize” faces

   •   Make images the same size

   •   Line up with respect to eyes

   •   Normalize intensities
2. Raw features are pixel intensity values (2061 features)

3. Each image is encoded as a vector i of these features

4. Compute “mean” face in training set:


                         M
                    1
                 
                    M
                        
                         i 1
                                i
•   Subtract the mean face from each face vector
                    i  i  

•   Compute the covariance matrix C

•   Compute the (unit) eigenvectors vi of C

•   Keep only the first K principal components (eigenvectors)
The eigenfaces encode the principal sources of variation
in the dataset (e.g., absence/presence of facial hair, skin tone,
glasses, etc.).

We can represent any face as a linear combination of these
“basis” faces.

Use this representation for:
   • Face recognition
       (e.g., Euclidean distance from known faces)

   • Linear discrimination
       (e.g., “glasses” versus “no glasses”,
       or “male” versus “female”)

								
To top