IDIAP-HeadPoseTracking-Clear07.p

Document Sample
IDIAP-HeadPoseTracking-Clear07.p Powered By Docstoc
					        Joint Head Tracking
        and Pose Estimation


       Jean-Marc Odobez, Sileye Ba
IDIAP Research Institute, Martigny, Switzerland
                                            Outline

      • System
               some of the tracking components


      • Results
              • AMI meeting task
              • CHIL lecture room task



                                                                        2
Joint Head Tracing and Pose Estimation   May 8, 2007   CLEAR Workshop
                      Joint Head Position and Pose Tracking
                                                       x t  ( xt , rt , lt )
      • Approach                                       translation,      roll        pose exemplar
                                                                      (in plane
            • joint optimization of position
                                                       scaling        rotation)

              and pose
            • Bayesian tracking with
              sampling approximation
            • mixed state (continuous/discrete)
            • exemplar appearance modeling
      • Observations
            • silhouette features                                 training data: Prima pointing

            • skin color masks
            • texture features

                                                                                                     3
Joint Head Tracing and Pose Estimation   May 8, 2007                              CLEAR Workshop
                            Sequential Monte-Carlo (SMC)
                                Head Pose Tracking
 • Goal: estimate the filtering distribution                             ( t
                                                                        p X | z: )
                                                                               1t

            object state                 Xt     and observation sequences   z1:t

 • Exact inference (Kalman Filter) not appropriate in vision
     (likelihood is highly non-linear)
   => use sampling approximation
                                   p ( X t | z1:t )




  => Particle Filter with Rao-Blackwelization
                                                                                                    4
Joint Head Tracing and Pose Estimation                May 8, 2007                  CLEAR Workshop
                                             State model
            State model :                X t  ( St , rt , kt )
                  2D transform                                           Out of plane head rotation
                          Translation+scaling                  roll   pose exemplar
                                                                      (index)




                                                                                                       5
Joint Head Tracing and Pose Estimation           May 8, 2007                          CLEAR Workshop
                                               Likelihood modeling
   • Generative head pose models
           • texture model (pose dependent)
           • skin model (pose dependent)
           • silhouette model (pose independent)
             => used to improve localization
   • Observation Likelihood:
           assuming conditional independence => product of likelihoods

                                                       t t t
                                                    X  S,r,k
                                                     t

                                                     ztext(St , r )
                                                                 t
                                                    zskin St , r )
                                                        ( t             ( textskinsil
                                                                      zz ,z ,z )
                               zsil(St , r )
                                          t

                                                    zsil(St , r )
                                                               t

                                                                                                   6
Joint Head Tracing and Pose Estimation              May 8, 2007                   CLEAR Workshop
                                    Proposal function

• Goal: sample new particles in
          high likelihood regions
   => proposal defined as mixture


                                 
             N
            1
   q1 (p1 
              d

   XX
    | 1t t  detz
   ( tzX p t (
   t
     i
       ) ( )
      ,
    X )|
       t
          i
                ( nt
                 |,
                 Xdet
                X ))
                t
            N
             n
             d1


      • state dynamics
        => preserves temporal continuity
      • output of a head detector
        => automatic (re)initialization and
           failure recovery


                                                                          7
Joint Head Tracing and Pose Estimation     May 8, 2007   CLEAR Workshop
                        Rao-Blackwellized particle filter

   • What: split the state component into two sets X t  ( St , rt , kt )
           • sample the first set (here 2D spatial components)
           • compute exact pdfs of other variables given sampled ones
             (here the pose values)
   • Why: several reasons
           • provides more accurate estimation with fewer particles

   X ti  ( S ti , rti , kti )
X ti  ( Sti , rti ,  ti (kt ))




                                                                                       8
Joint Head Tracing and Pose Estimation   May 8, 2007                  CLEAR Workshop
                                         Results: AMI task
 • Experiments                                                     15

    • Appearance models: learned from the
       Prima- POINTING database (15 people)
                                                                    0
 • Results          Pan (o)  Tilt (o) Roll (o)
                      mean error         8.8   9.4           9.8
                                                                   -15



 • Analysis                                                               15         0          -15


    • Good results (80% of pan errors <10o)
                                                                    15
    • larger errors on pan for side views
       (e.g. when people look at slides)
                                                                     0
    • tilt estimation less stable
    • large variations across individuals                           -15


                                                                          90         75           60

                                                                                                  9
Joint Head Tracing and Pose Estimation         May 8, 2007                     CLEAR Workshop
                                 Multi-view Head Pose Tracking

• Approach: tracking
   • apply head pose tracking method
     independently to each of the camera
   • obtain head poses w.r.t. room
     reference
• Approach: fusion
   • compute reliability for each camera
     (percentage of skin color pixels inside
     the estimated head bounding-box)
   • weighted average of the pose
     measurements of the two selected
     cameras


                                                                              10
Joint Head Tracing and Pose Estimation    May 8, 2007        CLEAR Workshop
                        Results in multi-view (CHIL data)
• IMPORTANT:                                     Experiment   pvec   pan    tilt       roll
 no ground-truth                                 1 camera     30     24.1   14         7.3
 bounding boxes were used,                       4 cameras    19.4   15     10         5.3
 only output of trackers
  (CHIL task ?)
• Results:
      • single camera tracking
        good results when face visible
         => large errors when head seen from the back
         => short time failure (presence of local distractors)

      • multi-view camera: fusion method
        much better results => camera switching according to
        reliability measure is efficient

                                                                                              11
Joint Head Tracing and Pose Estimation   May 8, 2007                        CLEAR Workshop
                                         Result example

• Color squares indicates camera selection
  (green: selected – red: unselected)

• Original views
  were cropped to
  allow better viewing

• Blue arrow:
  pointing vector

• Notice individual
  tracker errors


                                                                           12
Joint Head Tracing and Pose Estimation    May 8, 2007     CLEAR Workshop
                                         Conclusion
     • Happy to participate in the evaluation
        interesting, good to work on other data
        without having to worry about generating the annotation
     • Joint head tracking and pose estimation
         obtained good results in different conditions
          (meeting data, multi-view lecture room data)
     • Current/Future work
        • single head pose PF tracker (using 3D head position
          state space) should improve results (multi-view case)
        • combine with full/upper body
        • visual focus of attention estimation from head pose
                                                                         13
Joint Head Tracing and Pose Estimation    May 8, 2007   CLEAR Workshop

				
DOCUMENT INFO
Shared By:
Categories:
Stats:
views:3
posted:10/19/2010
language:English
pages:13