Diapositiva 1_1_ - Get as PowerPoint by pengtt


									Hamdi Dibeklioğlu

Ilkka Kosunen

Marcos Ortega

Albert Ali Salah

Petr Zuzánek

                       eNTERFACE ’10
         Amsterdam, July-August 2010
   Responsive photograph frame
    ◦ User interaction leads to different responses

   Modules of the project
    ◦ Video segmentation module
      Dictionary of responses
    ◦ Behaviour understanding
      Offline: Labelling dictionary
      Online: Cluster user action
    ◦ System logic
      Linking user actions to responses
                                                     Module 2:
                Module 4:

                                                     Visual Feature


 Automatic                           Program Logic                     Automatic
Segmentation                           (Learning)                     Segmentation
                   Segment Library                                                     Segment Library

                                       Module 3:
                                        System                          Module 5: Dual frame mode
External Data

Module 1: Offline segmentation
   5 video recordings (~1.5-2 min.)
    ◦ Same individual
    ◦ Different actions and expressions
   Manual annotation of videos
    ◦ ANVIL tool
    ◦ Annotated by different individuals
   Automatic segmentation
    ◦ Segmentation based on actions
    ◦ Optical flow: amount of activity over time
   Activity calculation based on feature tracking
    over the sequence
   Feature detection
    ◦ Shi-Tomasi corner detection algorithm
   Feature tracking
    ◦ Lucas-Kanade feature tracking algorithm
    ◦ Pyramidal implementation (Bouguet)
   Movement analysis
   To find a calm segment, just search for long
    period of frames with calculated optical flow
    below some treshold (we used 40% of average
    optical flow calculated from all frames)
   To find an active segment, search for frames
    with lot of optical flow, and then search
    forward and backward for the calm
   Face detection activates the system
    ◦ Viola-Jones face detector
   User’s behaviour can be monitored via
    ◦ Face detection
    ◦ Eye detection
         Valenti et al., isophote-curves based eye detection
    ◦ Optical flow energy
         OpenCV Lucas-Kanade algorithm
    ◦ Colour features
    ◦ Facial feature analysis
         The eMotion system
   Face and Eye detection: EyeAPI
            Face model: 16 surface patches
             embedded in Bezier volumes.
            Piecewise Bezier Volume Deformation
             (PBVD) tracker is used to trace the
             motion of the facial features.

* R. Valenti, N. Sebe, and T. Gevers.
  Facial expression recognition: A fully integrated approach.
  In ICIAPW, pages 125–130, 2007.
   12 motion units
   Naive Bayes (NB) classifier for
    categorizing expressions
   NB Advantage: the posterior probabilities
    allow a soft output of the system
Happiness   Surprise

 Disgust      Fear
   Linking user actions and system responses
   An action queue is maintained
    ◦ Different user inputs (transitions) lead to different
      responses (states)
   The responses (segments) are ‘unlocked’ one by
                                                 Period of

                       Face                                        input
                      detected         Wake-up           Neutral           Response
Before learning   After learning
   Currently two external programs are employed:
    ◦ SplitCam
    ◦ eMotion
   Glyphs are used to provide feedback to the user
   Glyph brightness is related to distance to
   Once a glyph is activated, the same user activity
    will elicit the same response
   Each user can have different behaviours activating
   Work on the learning module
   Testing the segmentation parameters
   The dual frame mode
   Speeding up the system
   Wizard of Oz study
   Usability studies
   SEMAINE integration?

To top