Hamdi Dibeklioğlu Ilkka Kosunen Marcos Ortega Albert Ali Salah Petr Zuzánek eNTERFACE ’10 Amsterdam, July-August 2010 Responsive photograph frame ◦ User interaction leads to different responses Modules of the project ◦ Video segmentation module Dictionary of responses ◦ Behaviour understanding Offline: Labelling dictionary Online: Cluster user action ◦ System logic Linking user actions to responses Module 2: Module 4: Real-time Interface Feature Analysis Visual Feature Analysis Segment Selection Automatic Program Logic Automatic Segmentation (Learning) Segmentation Segment Library Segment Library Module 3: System Module 5: Dual frame mode response External Data (Videos) Module 1: Offline segmentation 5 video recordings (~1.5-2 min.) ◦ Same individual ◦ Different actions and expressions Manual annotation of videos ◦ ANVIL tool ◦ Annotated by different individuals Automatic segmentation ◦ Segmentation based on actions ◦ Optical flow: amount of activity over time Activity calculation based on feature tracking over the sequence Feature detection ◦ Shi-Tomasi corner detection algorithm Feature tracking ◦ Lucas-Kanade feature tracking algorithm ◦ Pyramidal implementation (Bouguet) Movement analysis To find a calm segment, just search for long period of frames with calculated optical flow below some treshold (we used 40% of average optical flow calculated from all frames) To find an active segment, search for frames with lot of optical flow, and then search forward and backward for the calm segments. Face detection activates the system ◦ Viola-Jones face detector User’s behaviour can be monitored via ◦ Face detection ◦ Eye detection Valenti et al., isophote-curves based eye detection ◦ Optical flow energy OpenCV Lucas-Kanade algorithm ◦ Colour features ◦ Facial feature analysis The eMotion system Face and Eye detection: EyeAPI Face model: 16 surface patches embedded in Bezier volumes. Piecewise Bezier Volume Deformation (PBVD) tracker is used to trace the motion of the facial features. * R. Valenti, N. Sebe, and T. Gevers. Facial expression recognition: A fully integrated approach. In ICIAPW, pages 125–130, 2007. 12 motion units Naive Bayes (NB) classifier for categorizing expressions NB Advantage: the posterior probabilities allow a soft output of the system Happiness Surprise Anger Disgust Fear Sadness Linking user actions and system responses An action queue is maintained ◦ Different user inputs (transitions) lead to different responses (states) The responses (segments) are ‘unlocked’ one by one Period of inactivity Sleeping User Face input detected Wake-up Neutral Response Before learning After learning Currently two external programs are employed: ◦ SplitCam ◦ eMotion Glyphs are used to provide feedback to the user Glyph brightness is related to distance to activation Once a glyph is activated, the same user activity will elicit the same response Each user can have different behaviours activating glyphs Work on the learning module Testing the segmentation parameters The dual frame mode Speeding up the system Wizard of Oz study Usability studies SEMAINE integration?
Pages to are hidden for
"Diapositiva 1_1_ - Get as PowerPoint"Please download to view full document