Docstoc

3D Face Reconstruction and Expression Modelling

Document Sample
3D Face Reconstruction and Expression Modelling Powered By Docstoc
					The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                         2009




           Alexander Woodward
                                                            and Expression Modelling
                                                          3D Human Face Reconstruction
                                                                      Outline
   2009




                                                                Aim
PRESENTATION




                                                                System overview
                                                                Related work
  The University of Auckland | Computer Science | New Zealand




                                                                3D face reconstruction
                                                                Expression modelling
                                                                Contributions and future work
                                                                      Overview
   2009




                                                                Aim: Integrated system for 3D face reconstruction and
PRESENTATION




                                                                 expression modelling
                                                                       Vision based not graphics based
  The University of Auckland | Computer Science | New Zealand




                                                                       Low cost and self-contained


                                                                Results can be applied to:
                                                                   Biometrics and security        Film
                                                                   Biomedical visualisation       Teleconferencing
                                                                   Computer and video games       Human computer interaction



                                                                                                                           3
                                                                  System overview
   2009



                                                                3D reconstruction
PRESENTATION




                                                                            Static                                     Dynamic
                                                                Active & passive binocular stereo
                                                                                                                      3D video scanner
  The University of Auckland | Computer Science | New Zealand




                                                                   Active structured lighting

                                                                   Active photometric stereo




                                                                                                    3D data



                                                                Expression modelling
                                                                 Marker based motion capture                  Video based

                                                                                                                 Sequences from 3D video
                                                                   Muscle inverse kinematics                            scanner
   2009

                                                                           Related work
PRESENTATION




                                                                 Complete systems for face reconstruction and
                                                                  animation are uncommon
                                                                       High hardware requirements
  The University of Auckland | Computer Science | New Zealand




                                                                       Data acquisition, motion capture and animation systems are
                                                                        often provided as disparate packages or only as a service,
                                                                        cf. a stand-alone solution
                                                                 At least 9 prominent projects aimed toward complete
                                                                  systems
                                                                       Excluding in-house solutions
                                                                       Large body of work in 3D face research
                                                                                3D reconstruction, expressions, motion capture

                                                                02 June 2012                      Department of Computer Science   5
                                                                        Related work
   2009



                                                                Borshukov et al (2003 – 2007)
PRESENTATION




                                                                    Playable Universal Capture approach
                                                                        3D scanner, marker based tracking, optical flow, video texture
                                                                Ma et al (2007, 2008)
  The University of Auckland | Computer Science | New Zealand




                                                                    Capture face reflectance
                                                                       3D scanner, photometric stereo, motion capture
                                                                       Light stage – 156 LED lights over an icosahedron
                                                                Image Metrics Inc. & U Sth Carolina Graphics Lab (2008)
                                                                   Digital Emily project
                                                                        Light stage captures geometry and reflectance
                                                                        33 expressions captured; creates an animation rig
                                                                        Performance data mapped to the 3D face
                                                                                                                                  6
                                                                  3D reconstruction
   2009



                                                                                       3D reconstruction
PRESENTATION




                                                                            Static                                     Dynamic
                                                                Active & passive binocular stereo
                                                                                                                      3D video scanner
  The University of Auckland | Computer Science | New Zealand




                                                                   Active structured lighting

                                                                   Active photometric stereo




                                                                                                    3D data



                                                                                     Expression modelling
                                                                 Marker based motion capture                  Video based

                                                                                                                 Sequences from 3D video
                                                                   Muscle inverse kinematics                            scanner
   2009

                                                                      3D reconstruction requirements
                                                                Off-the-shelf hardware, no special properties
PRESENTATION




                                                                   Cameras, PC, projector
                                                                Low acquisition time – faces move, esp. children
  The University of Auckland | Computer Science | New Zealand




                                                                Controlled lighting
                                                                Vision based
                                                                   New algorithms
                                                                Useful for any type of object




                                                                                                                    8
                                                                  Static 3D reconstruction
   2009



                                                                                         3D reconstruction
PRESENTATION




                                                                            Static                                     Dynamic
                                                                Active & passive binocular stereo
                                                                                                                      3D video scanner
  The University of Auckland | Computer Science | New Zealand




                                                                   Active structured lighting

                                                                   Active photometric stereo




                                                                                                    3D data



                                                                                     Expression modelling
                                                                 Marker based motion capture                  Video based

                                                                                                                 Sequences from 3D video
                                                                   Muscle inverse kinematics                            scanner
   2009

                                                                        Static 3D reconstruction
                                                                 Evaluated approaches:
PRESENTATION




                                                                   1. Active & passive binocular stereo      3. Active photometric stereo
                                                                   2. Active structured lighting
  The University of Auckland | Computer Science | New Zealand




                                                                    Evaluate effectiveness
                                                                        Accuracy, time complexity
                                                                    Determine best approach for dynamic
                                                                    3D reconstruction system
                                                                    12 algorithms
                                                                    Database of 15 faces
                                                                         Alternative test set             Ground truth data: 3D scanner
                                                                         Focus on stereo algorithms
                                                                 Compared to Middlebury, algorithms rank differently for faces
                                                                 Projected patterns improve and level out performance

                                                                                                                                            10
   2009

                                                                                    Active binocular stereo
                                                                Strip colour pattern: much higher accuracy
PRESENTATION




                                                                   SAD correlation algorithm:
  The University of Auckland | Computer Science | New Zealand




                                                                    Strip pattern           SAD - without               SAD - with
                                                                                              pattern                  strip pattern

                                                                                                 80%                       92%

                                                                    Pattern colour should contrast strongly on skin
                                                                                                                                 11
                                                                       Statistical results
   2009
PRESENTATION




                                                                                                 + Grad. + Strip
                                                                BP     + Grad. + Strip     CM                         DPM          + Grad.    + Strip

                                                                73%      77%     89%       88%     89%        92%      79%          84%        92%
  The University of Auckland | Computer Science | New Zealand




                                                                GC     + Grad. + Strip     SAD   + Grad. + Strip      SDPS + Grad.           + Strip
                                                                77%      83%     92%       80%     85%        92%       89%         90%        93%




                                                                FCV   Four Path Shapelet          Gray code

                                                                69%     54%       71%                97%
                                                                                                                    Ground truth
                                                                                                                                                12
                                                                  Dynamic 3D reconstruction
   2009



                                                                3D reconstruction
PRESENTATION




                                                                            Static                                     Dynamic
                                                                Active & passive binocular stereo
                                                                                                                      3D video scanner
  The University of Auckland | Computer Science | New Zealand




                                                                   Active structured lighting

                                                                   Active photometric stereo




                                                                                                    3D data



                                                                Expression modelling
                                                                 Marker based motion capture                  Video based

                                                                                                                 Sequences from 3D video
                                                                   Muscle inverse kinematics                            scanner
                                                                        Dynamic 3D reconstruction
   2009



                                                                Reconstruction at video rates →3D video!
PRESENTATION




                                                                    From static reconstruction best results:
                                                                        ‘One shot’ active illumination + Symmetric Dynamic Programming (SDPS)
                                                                        Project pattern every other frame to get a clean texture
  The University of Auckland | Computer Science | New Zealand




                                                                                               (2)


                                                                                                     (3)


                                                                                  (1)



                                                                                                  Monochrome stereo pair of video cameras +
                                                                                                  3rd colour web camera obtains colour texture.

                                                                                                                                                 14
                                                                        Colour texture generation
   2009
PRESENTATION




                                                                                         +                →
  The University of Auckland | Computer Science | New Zealand




                                                                     Colour image            Monochrome       Final texture
                                                                (reprojected into same         image
                                                                   reference frame)

                                                                Low resolution colour information combined with high
                                                                 resolution luminance information
                                                                 Next step: colour video cameras
                                                                                                                              15
     The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                  2009

                                                                                  Videos




16
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                             2009




                Patternless reconstruction
   2009

                                                                  Marker based expression modelling
                                                                3D reconstruction
PRESENTATION




                                                                            Static                                     Dynamic
                                                                Active & passive binocular stereo
                                                                                                                      3D video scanner
  The University of Auckland | Computer Science | New Zealand




                                                                   Active structured lighting

                                                                   Active photometric stereo




                                                                                                    3D data



                                                                Expression modelling
                                                                Marker based motion capture                   Video based

                                                                                                                 Sequences from 3D video
                                                                   Muscle inverse kinematics                            scanner
   2009

                                                                      Marker based expression modelling
                                                                Data driven:
PRESENTATION




                                                                   Stereo web-cameras, face
                                                                   markers.
  The University of Auckland | Computer Science | New Zealand




                                                                   Head motion - rigid
                                                                   Expressions - non-rigid
                                                                Tracked 3D points
                                                                   Unique 3D face model
                                                                   mapping
                                                                  Virtual muscle animation
                                                                      17 active muscles
                                                                      Muscle inverse kinematics (IK) –
                                                                       Jacobian Transpose

                                                                                                          19
     The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                                              2009
                                                                                                                     Example videos




                                        Surprise
                                                                                  Happiness – easiest to reproduce




20
   2009

                                                                                   Anger – needs teeth!
PRESENTATION
  The University of Auckland | Computer Science | New Zealand




                                                                Disgust – pursing of mouth & closing of eyes not represented
   2009

                                                                  Video based expression modelling
                                                                3D Reconstruction
PRESENTATION




                                                                            Static                                     Dynamic
                                                                Active & passive binocular stereo
                                                                                                                      3D video scanner
  The University of Auckland | Computer Science | New Zealand




                                                                   Active structured lighting

                                                                   Active photometric stereo




                                                                                                    3D data



                                                                Expression modelling
                                                                 Marker based motion capture                  Video based

                                                                                                                 Sequences from 3D video
                                                                   Muscle inverse kinematics                            scanner
                                                                      3D video based expression
   2009


                                                                      modelling
                                                                Image blending
PRESENTATION




                                                                Novel face expressions from
                                                                 multiple video sequences
  The University of Auckland | Computer Science | New Zealand




                                                                   Interactive
                                                                   Low preparation
                                                                                                 Sub-region masks
                                                                   Not data driven
                                                                Dense depth data – cf. marker
                                                                 system
                                                                 Video based → realistic 3D
                                                                 movement and texture
                                                                Reconstruction data directly
                                                                 used for expression modelling      control points
                                                                                                 11 Control points
                                                                                                                     23
                                                                       Synthetic expression results
   2009


                                                                Sadness: lower face region, anger: right eye region, surprise: left eye region
PRESENTATION
  The University of Auckland | Computer Science | New Zealand




                                                                     Happiness: lower face region, surprise: left and right eye regions




                                                                                                                                           24
                                                                       Synthetic expression results
   2009


                                                                Fear: lower face region, happiness: right eye region, anger: left eye region
PRESENTATION
  The University of Auckland | Computer Science | New Zealand




                                                                       Disgust: lower face region, anger: left and right eye regions
   2009

                                                                      Contributions
                                                                3D face reconstruction and expression modelling
PRESENTATION




                                                                 system
                                                                   Unique tool-set
                                                                  Low-cost, off-the-shelf
  The University of Auckland | Computer Science | New Zealand




                                                                  Vision based
                                                                To 3D face reconstruction:
                                                                       Extensive reconstruction comparison
                                                                       Face database
                                                                       Dynamic reconstruction system for 3D video: SDPS + pattern
                                                                To expression modelling:
                                                                       Marker based performance capture system
                                                                          Muscle based IK animation system, unique mapping approach
                                                                       Video based expression system – realistic, less flexible

                                                                                                                                       26
                                                                       Future work and perspective
   2009



                                                                Many areas for future research
PRESENTATION




                                                                   Refine hardware - better reconstructions ( low-cost? )
                                                                   Markerless motion capture - face ( feature ) tracking
                                                                       Statistical analysis on video data
  The University of Auckland | Computer Science | New Zealand




                                                                       Active appearance model (AAM)
                                                                   New animation system (out of scope)
                                                                   Full body → complete character
                                                                Synergy of computer vision and computer graphics!
                                                                   Physical models for animation
                                                                   Computer vision tools
                                                                       Especially 3D video & markerless motion capture
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                  2009




                                                                     Questions?
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                             2009
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                              2009

                                                                             Timeline of Experiments
                                                                    Universal expressions
   2009

                                                                    Ekman - 1987
PRESENTATION
  The University of Auckland | Computer Science | New Zealand




                                                                          Sadness         Anger       Happiness




                                                                           Fear          Disgust         Surprise

                                                                Recognisable in every culture! Used as exemplar expressions to
                                                                                       judge my results
                                                                                                                                 31
   2009

                                                                        Types of binocular stereo algorithm
                                                                 Local vs global optimisation
PRESENTATION




                                                                 WTA
                                                                    SAD, SSD
                                                                 Chen-Medioni –
  The University of Auckland | Computer Science | New Zealand




                                                                    local method with explicit surface constraints
                                                                    Seed propagation approach
                                                                 Dynamic programming – 1D optimisation
                                                                    SDPS – markov chain
                                                                    DPM
                                                                 Cubic algorithms – 2D optimisation
                                                                    Markov random field
                                                                    Energy minimisation
                                                                    Graph-cut (KZ1, RoyCox), Belief Propagation,
                                                                      Types of photometric stereo algorithm
   2009




                                                                Experiment focused on integration methods
PRESENTATION




                                                                   Assumes C² continuity – i.e. a smooth second derivative
                                                                Local optimisation – based on curve integrals
  The University of Auckland | Computer Science | New Zealand




                                                                   Four path integration
                                                                   Shapelet
                                                                       Explicit summation of basis functions
                                                                Global optimisation
                                                                   FCV – Frankot Chellappa Variant
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                               2009

                                                                             Structured lighting techniques
                                                                           Body modelling and animation
   2009




                                                                Body: generic skinned animation
PRESENTATION




                                                                     Skeletal hierarchy, fully articulated
  The University of Auckland | Computer Science | New Zealand




                                                                                                                                       • The bones of the hand




                                                                                                         • Each bone of the
                                                                                                                                       • Movement of the forearm
                                                                   • The body model with underlying      skeleton has a region of
                                                                   skeleton                              influence, denoted in green



                                                                02 June 2012                          Department of Computer Science                             35
     The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                                                              2009




                                                               RBF mapping
                                                               Input photograph
                                                                                   Interactive personalised avatar creator




36
                                   The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                                                2009




02 June 2012
                                                                                                                Results




  Department of Computer Science
  37
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                             2009

                                                                             Results
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                             2009

                                                                             Results
                                                                       3D video based expression system
   2009


                                                                       overview
                                                                Acquire sequences of individual expressions using
PRESENTATION




                                                                 dynamic 3D face reconstruction system.
                                                                       Expression sequences start from a neutral state.
  The University of Auckland | Computer Science | New Zealand




                                                                       Test subject’s head remains in the same position for every sequence
                                                                       A reference texture and depth map are taken from the neutral
                                                                        expression and used as the base for all image regions
                                                                11 control points are manually annotated on video
                                                                 sequences.
                                                                       Future work to automate this process.
                                                                Six sub-regions manually defined on the face.
                                                                A sub-region’s texture and depth updated by dragging
                                                                    a control point residing in it and its currently chosen
                                                                    expression sequence.
                                                                02 June 2012               Department of Computer Science                 40
                                                                           System conclusions
   2009




                                                                Sinusoidal interpolation instead of a linear one. This
PRESENTATION




                                                                 roughly models the biphasic nature of skin
                                                                Realistic animations are created as motion is derived
  The University of Auckland | Computer Science | New Zealand




                                                                 from 3D video sequences of real-life test subjects.
                                                                           A user can create unnatural but interesting looking expressions that
                                                                            can convey a comical feel
                                                                Texture maps sourced from video sequences solves
                                                                 the loss of detail in the marker based approach
                                                                           However, apart from the control points that were manually specified,
                                                                            no points on the face surface are tracked
                                                                Results could be refined by improving the quality of 3D
                                                                 video reconstruction.
                                                                02 June 2012                    Department of Computer Science                     41
                                                                      Test subject placement
   2009




                                                                Subject can be placed with knowledge of required view
PRESENTATION




                                                                 area, sensor size, and camera lens:
  The University of Auckland | Computer Science | New Zealand
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                               2009

                                                                             Projector synchronisation
                                                                       RBF mapping approach
   2009




                                                                Radial Basis Functions
PRESENTATION




                                                                User specified point correspondences on generic
                                                                 model and 3D face data
  The University of Auckland | Computer Science | New Zealand




                                                                   Specify divergences between data
                                                                For each dimension (in 3D)
                                                                   Find RBF approximation of (1D) displacements within the 3D
                                                                    space of specified points.
                                                                Using this RBF approximation all 3D points from the
                                                                 generic model can be mapped to the 3D face data
                                                                 proportions
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                              2009

                                                                             Marker tracking
                                                                       Rigid and non-rigid motion
   2009
PRESENTATION




                                                                 Anchor markers:
  The University of Auckland | Computer Science | New Zealand




                                                                Rigid orientation:




                                                                Remove rigid motion by using transpose of orientation and
                                                                              centre of gravity of anchors
                                                                       Muscle inverse kinematics
   2009




                                                                Forward kinematics is the calculation of a new position g of
PRESENTATION




                                                                 an end effector by specifying updates to parameters of a
                                                                 kinematic chain
  The University of Auckland | Computer Science | New Zealand




                                                                Inverse kinematics is the calculation of parameters for a
                                                                 kinematic chain to meet a desired goal position g, when
                                                                 starting from an initial position e.
                                                                Kinematic chain consists of joints
                                                                Each joint has DOF’s – its animitable parameters,
                                                                   E.g. 3-DOF for position, 1-DOF for orientation around one axis
                                                                    (position of joint implied through kinematic chain transformation)
                                                                       Jacobian Transpose approach
   2009




                                                                FK:
PRESENTATION




                                                                                               e = current end      g = goal end
                                                                                               effector position   effector position
                                                                IK:
  The University of Auckland | Computer Science | New Zealand




                                                                  d = change in end effector
                                                                            position


                                                                 First order estimate in positional
                                                                              change:


                                                                       Change in parameters:



                                                                  Jacobian Transpose estimate:
   2009
PRESENTATION




                                                                  Estimate assured to move closer to
                                                                              the goal g:
  The University of Auckland | Computer Science | New Zealand




                                                                    Always moving in a direction less
                                                                        than 90 degrees from d




                                                                02 June 2012                  Department of Computer Science   49
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                             2009
   2009
                                                                            Interface between Raw Data and
                                                                            Generic Model
                                                                 User specifies a ‘minimal’ set of correspondences between
PRESENTATION




                                                                 raw and generic data
                                                                 Radial Basis Functions (RBF) used as the interpolant
  The University of Auckland | Computer Science | New Zealand




                                                                Model with
                                                                animation system




                                                                                       Correspondences                   Results in a custom
                                                                                       made and mapped                   face with animation
                                                                                       via RBF with a final              system in place
                                                                                       nearest point map
                                                                                       and texture            •Feature extraction as a goal
                                                                                       projection
                                                                Depth map
                                                                         Face Animation Model
   2009



                                                                 Research primarily based on Terzopoulos, Waters, Parke collective work in
PRESENTATION




                                                                  the field
                                                                 Physically based model for skin tissue
                                                                     Mass-Spring system
  The University of Auckland | Computer Science | New Zealand




                                                                     Epidermal – Fascial – Skull levels of tissue
                                                                     Forces are applied to the tissue to simulate muscle contractions
                                                                     Springs bring elasticity, allow forces to propagate-> stretches and pulls!
                                                                 Abstract muscle definitions
                                                                     Decoupled from model
                                                                     Warped via RBF also
                                                                     Two types
                                                                            Face Animation System Forces
   2009




                                                                Model the behaviour of the tissue
PRESENTATION




                                                                            Reactionary over the evolution of applied muscle forces:
  The University of Auckland | Computer Science | New Zealand




                                                                                                                      Skull Penetration Constraint
                                                                                  Spring Forces                                                   
                                                                g j  c j (l j - lj )s j , g i  g j
                                                                                                                                      
                                                                                                                            f in  n i n i when f i n  n i  0
                                                                                                                      si  
                                                                                                                           0                otherwise
                                                                cj     - biphasic spring constant

                                                                l j   - rest length of spring
                                                                                                                        Muscle Forces
                                                                lj     - current length of spring
                                                                                                                        Applied to fascia nodes based
                                                                                                                        on the abstract muscle
                                                                s j  (x j  xi ) / l j   - spring direction vector     definitions………..(explained
                                                                                                                        later)
                                                                             Face System Forces
   2009
PRESENTATION




                                                                              Volume Preservation Force


                                                                                    i       i
                                                                  qe  k1 V e  V e ne  k2 pe  pie
                                                                   i                                                   
  The University of Auckland | Computer Science | New Zealand




                                                                ne
                                                                 i           - epidermal normal for volume element ‘e’

                                                                p e , p ie   - current and rest nodal positions with
                                                                  i
                                                                             respect to center of mass of element ‘e’

                                                                k1 , k 2     - force scaling constants




                                                                These forces allows for tissue form restitution
                                                                       Linear Muscle
   2009




                                                                Linear Muscle:
PRESENTATION




                                                                   Applies forces to nodes inside it’s angular range
                                                                   Influence is weighted by angle and radius from muscle vector
  The University of Auckland | Computer Science | New Zealand




                                                                                                        Displacement formula:
                                                                                                                               pv 1
                                                                                                             p  p  akr
                                                                                                                               pv 1
                                                                                                                         cos( )
                                                                                                Where          a  1                   and
                                                                                                                         cos()
                                                                                                          D
                                                                                                  cos( 
                                                                                                      1         ); for p inside sector ( v 1p n p m p1 )
                                                                                                          Rs 2
                                                                                               r
                                                                                                  cos( D  Rs  ); for p inside sector (p n p r p s p m )
                                                                                                  
                                                                                                      R f  Rs 2
                                                                                               ‘k’ = muscle contraction increment.
                                                                       Ellipsoid Muscle
   2009




                                                                Ellipsoid Muscle:
PRESENTATION




                                                                   Acts like a string bag
                                                                   Application of force weighted by radius only
  The University of Auckland | Computer Science | New Zealand




                                                                   Defined by major and 2 minor axes
                                                                   Can generate puckering effects


                                                                    Displacement formula:
                                                                                     pv 1
                                                                       p  p  kr
                                                                                     pv 1
                                                                          Physical Simulation
   2009




                                                                 Layered Tissue Model is a physically based one
PRESENTATION




                                                                 Euler integration is used to run the simulation
  The University of Auckland | Computer Science | New Zealand




                                                                                                                      Velocity dependent
                                                                 Equations of motion                                  damping co-efficient.
                                                                                                                      Controls the rate of
                                                                 a 
                                                                  t
                                                                  i
                                                                     1 e
                                                                     mi
                                                                            
                                                                        f i   i v ti  ~it  qit  ~i t  hit
                                                                                         g ~ s
                                                                                                            ~
                                                                                                                     dissipation of kinetic
                                                                                                                      energy which eventually
                                                                                                                      brings the facial mesh to
                                                                                                   Acceleration       rest.


                                                                  v t  t  v t  ta t
                                                                    i          i       i           Velocity



                                                                  x ti  t  x ti  tv ti  t   Nodal position
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                              2009




 Fear
                                        Happiness




 Disgust
                                        Sadness
                                                                             6 pre-built expressions




 Anger
                                        Surprise
                                                                           General conclusions and future
   2009


                                                                           work (old version)
                                                                Investigated low-end and cost effective equipment to
PRESENTATION




                                                                 create self-contained tools that can run entirely on any
                                                                 end user system.
  The University of Auckland | Computer Science | New Zealand




                                                                     A unique solution has been proposed for 3D face
                                                                      reconstruction and expression modelling with appropriate
                                                                      hardware
                                                                Synchronised audio capture of speech sequences
                                                                 would greatly add to the realism.
                                                                The attachment of the face model to a body would
                                                                 complete the system, giving a fully realised virtual
                                                                 human.

                                                                02 June 2012              Department of Computer Science         59
   2009




                                                                Conclusions drawn from the static reconstruction
PRESENTATION




                                                                 experiment formed the basis of a dynamic 3D face
                                                                 reconstruction system
  The University of Auckland | Computer Science | New Zealand




                                                                3D face reconstructions have no notion of higher order
                                                                 surface structure and are just a collection of points.
                                                                           This structure was addressed in the second part of this thesis which
                                                                            investigated face expression modelling.
                                                                Marker motion was combined with a muscle inverse
                                                                 kinematics framework to drive the facial animation
                                                                 system.
                                                                           A static face texture impacts on the visual result, as illumination cues
                                                                            such as wrinkles and shadowing over the face are lost.

                                                                02 June 2012                    Department of Computer Science                    60
   2009




                                                                To supplement the work on 3D faces, a body model
PRESENTATION




                                                                 was created
                                                                Interactive 3D video expression creation system which
  The University of Auckland | Computer Science | New Zealand




                                                                 ties together 3D face reconstruction and expression
                                                                 modelling.
                                                                Main problems faced were dealing with hardware
                                                                 constraints
                                                                           But focus on low-cost and off-the-shelf solutions
                                                                Focused on the computer vision aspects of facial
                                                                 reconstruction and expressions as opposed to
                                                                 computer graphics
                                                                02 June 2012                    Department of Computer Science   61
   2009




                                                                A combined marker and dense 3D reconstruction
PRESENTATION




                                                                 system could be developed, to incorporate further
                                                                 information for a muscle inverse kinematics system
  The University of Auckland | Computer Science | New Zealand




                                                                Highly detailed face animation is best served by taking
                                                                 advantage of real world data in the form of digital
                                                                 images and computer vision processing
                                                                Advanced physical models of faces meets the tools
                                                                 and approaches investigated within this thesis




                                                                02 June 2012          Department of Computer Science   62
                                   The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                                                2009




02 June 2012
  Department of Computer Science
  63
   2009




                                                                Details in important areas of the face that are not
PRESENTATION




                                                                 currently modelled include the eyelids, lips, teeth and
                                                                 inner mouth.
  The University of Auckland | Computer Science | New Zealand




                                                                Loss of texture detail in the forehead where the
                                                                 wrinkles are lost
                                                                Fine tuning of the preset muscle locations and
                                                                 parameters when mapping a new face model was
                                                                 sometimes needed to improve results or correct
                                                                 muscles



                                                                02 June 2012          Department of Computer Science       64
                                                                           System analysis
   2009




                                                                Capable of reproducing facial expressions from marker
PRESENTATION




                                                                 motion.
                                                                     Low cost hardware, easy retargetting to other models.
  The University of Auckland | Computer Science | New Zealand




                                                                Many differences between test-subjects
                                                                     Expression articulation, muscle control, gross face
                                                                      movement
                                                                Difficulty in performing when no emotional tie involved.
                                                                           Easy to understand the need for directors in performance capture
                                                                            situations.
                                                                           Some user direction was needed to describing to a test subject how
                                                                            an expression should be created



                                                                02 June 2012                   Department of Computer Science                    65
   2009




                                                                Issue: potentially multiple solutions for a vertex
PRESENTATION




                                                                 position when influenced by multiple muscles
                                                                Illumination conditions affect coloured marker
  The University of Auckland | Computer Science | New Zealand




                                                                 detection
                                                                Reflectance properties of the skin surface are
                                                                 important visual cues.
                                                                     Missing in this system
                                                                     Addressed in next chapter.




                                                                02 June 2012             Department of Computer Science   66
                                                                           Body modelling and animation
   2009


                                                                           system
PRESENTATION




                                                                Skinned animation system was chosen for real-time
                                                                 capability and ease of creating new body poses
  The University of Auckland | Computer Science | New Zealand




                                                                     A posable skeleton is associated with a body model (skin
                                                                      surface description), usually in the form of geometric data
                                                                           Forward and inverse kinematics used for animation
                                                                           Skin surface under new pose is determined based on skeletal bone
                                                                            local coordinate systems and blending between adjacent bones.
                                                                Future work: combine with the face reconstructions
                                                                 and animation systems.


                                                                02 June 2012                   Department of Computer Science                  67
                                                                           Static reconstruction experiment
   2009


                                                                 Evaluated three computer vision approaches to 3-D face reconstruction.
                                                                     Binocular stereo: passive.
PRESENTATION




                                                                     Structured lighting: active.
                                                                     Photometric stereo: active.
                                                                 Two main aims:
                                                                     Determine their effectiveness for 3D facial reconstruction.
  The University of Auckland | Computer Science | New Zealand




                                                                           Accuracy, time complexity.
                                                                     Provide a new and alternative test set for evaluating algorithms.
                                                                           Database of faces.
                                                                           We focus on stereo vision algorithms.
                                                                 Integrated lab environment designed.
                                                                     12 algorithms tested in total.
                                                                     Results compared to ground truth data obtained from a commercial 3D scanner.
                                                                 Summary:
                                                                     Active illumination techniques are most accurate.
                                                                     Stereo algorithm rankings were different from that expected.
                                                                     ‘One shot’ active illumination coupled with a traditional stereo algorithm a strong
                                                                      choice.
                                                                02 June 2012                         Department of Computer Science                     68
   2009

                                                                           Photogrammetry Laboratory
                                                                Optical ‘Range’.
PRESENTATION




                                                                   Integrated.
                                                                   Multiple systems view a
                                                                     common scene.
  The University of Auckland | Computer Science | New Zealand




                                                                 Stereo bench.
                                                                     Sideways for face capture!
                                                                                                                                  Example Data:
                                                                 Projector for structured
                                                                  lighting.
                                                                 Light sources for
                                                                  photometric stereo.
                                                                 Commercial 3D scanner.
                                                                           Solutionix Rexcan 400.
                                                                                                                         Depth map      Perspective visualisation

                                                                02 June 2012                     Department of Computer Science                            69
   2009

                                                                           Calibration
                                                                System calibration:
PRESENTATION




                                                                     Estimates intrinsic and extrinsic
                                                                      camera parameters
                                                                           I.e. camera projection matrices
  The University of Auckland | Computer Science | New Zealand




                                                                           For cameras:
                                                                               A calibration cube - 63 markings
                                                                                  defines a world co-ordinate system
                                                                               Tsai calibration
                                                                           For the lights:
                                                                               A calibration sphere - estimates
                                                                                  directions to lights
                                                                               Simple analytic derivation,
                                                                                  inaccurate
                                                                           Could also calibrate the projector using
                                                                            Tsai’s algorithm

                                                                02 June 2012                    Department of Computer Science   70
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                               2009

                                                                             World to image co-ordinates
   2009



                                                                 Rectification:
PRESENTATION




                                                                           The camera calibration matrices were used to rectify images.
                                                                              The resultant image pairs meet the epipolar constraint.
  The University of Auckland | Computer Science | New Zealand




                                                                 Data processing:
                                                                           Data must be compared in a common co-ordinate frame.
                                                                           Alignment done using a semi-automatic process involving 3D object rigid
                                                                            transformations.
                                                                           Small number of manual correspondences made.
                                                                           Data projected into disparity space.

                                                                02 June 2012                      Department of Computer Science                      72
   2009




                                                                Database of 15
PRESENTATION




                                                                 people created
                                                                  Data acquired
  The University of Auckland | Computer Science | New Zealand




                                                                   from all systems
                                                                  Rexcan ground
                                                                   truth
                                                                Test-bed for new
                                                                 algorithms




                                                                                      73
                                                                           Binocular Stereo
   2009


                                                                 Approach 1: Binocular stereo (stereo
                                                                  vision).                                                             System Geometry (side view)
PRESENTATION




                                                                     Passive.
                                                                     Active research area in our department.
                                                                     Textureless regions cause problems.
  The University of Auckland | Computer Science | New Zealand




                                                                           Remedy via active illumination.
                                                                 Test a set of local and global algorithms.
                                                                                                                                    Use two Canon digital SLRs –
                                                                                  Tested algorithms:                                 6 Mpixels
                                                                                                                                            1536 x 1024
                                                                 Sum of Absolute Differences (SAD)                                            resolution.
                                                                 Dynamic Programming Method (DPM)
                                                                 Symmetric Dynamic Programming Stereo (SDPS)
                                                                 Graph Cut (GC)
                                                                 Belief–Propagation (BP)
                                                                 Chen and Medioni (CM) – seed based algorithm
                                                                02 June 2012                      Department of Computer Science                                     74
                                                                           Structured Lighting
   2009


                                                                 Approach 2: Structured Lighting.                                   System Geometry (side view)
                                                                     Active approach. Depth inferred in the
PRESENTATION




                                                                      same manner as stereo.
                                                                     Augment stereo system with a colour
                                                                      projector.
  The University of Auckland | Computer Science | New Zealand




                                                                           Add structure to scene -> break
                                                                            homogeneity.
                                                                                                                                     6 of the Gray code projections:
                                                                           Projects 800 x 600 pixel image.
                                                                              Acer PL111 LCD Projector.
                                                                 Interested in ‘one shot’ patterns over
                                                                  Gray code.
                                                                 Tested algorithms:
                                                                 Time-multiplexed structured lighting using Gray code
                                                                 Direct coding - ‘one shot’ colour gradation pattern.
                                                                                                                                      •Add texture to face.
                                                                 Direct coding - ‘one shot’ colour strip pattern.                     •Used with standard stereo
                                                                                                                                      algorithms.
                                                                02 June 2012                        Department of Computer Science                                 75
                                                                           Photometric Stereo
   2009



                                                                 Approach 3: Photometric Stereo (PSM).
PRESENTATION




                                                                      Face viewed under 3 different known lighting
                                                                       conditions.
                                                                      Depth by integrating recovered surface orientation
  The University of Auckland | Computer Science | New Zealand




                                                                       map.
                                                                           Albedo independent approach used.                             System Geometry (top-down view)
                                                                      Three 150W light sources.
                                                                 Analysed gradient field integration
                                                                  techniques.
                                                                 Tested algorithms:

                                                                 Frankot-Chellappa Variant (FCV)                        Fourier based integration.

                                                                 Four-Scan method                                       Local integration paths.

                                                                 Shapelets                                              Summation of correlated basis functions.

                                                                02 June 2012                       Department of Computer Science                                     76
                                                                           A Collection of Reconstructions
   2009


                                                                 Example depth maps:
PRESENTATION
  The University of Auckland | Computer Science | New Zealand




                                                                 Ground        Gray        FCV                SAD             SDPS        GC     CM
                                                                  truth        code


                                                                     Structured lighting                                      Binocular Stereo



                                                                                 Photometric Stereo




                                                                02 June 2012                 Department of Computer Science                      77
                                                                           Photometric Stereo Results
   2009



                                                                Reconstruction accuracy:
PRESENTATION




                                                                            17 test subjects.
                                                                                         Percentage of errors less than 2 disparity units
  The University of Auckland | Computer Science | New Zealand




                                                                Method           P <=2,%

                                                                                           69
                                                                               54
                                                                                             71




                                                                                                                              97

                                                                                       Gold standard result for accuracy

                                                                02 June 2012                 Department of Computer Science             78
                                                                           Passive Stereo Results
   2009



                                                                Reconstruction accuracy:
PRESENTATION




                                                                Method         P<=2,%
  The University of Auckland | Computer Science | New Zealand




                                                                                                                    89
                                                                                                  79
                                                                GC                               77
                                                                                                      80
                                                                                           73
                                                                                                                  88



                                                                                                                         97



                                                                02 June 2012            Department of Computer Science        79
                                                                           Stereo + Gradation Pattern
   2009



                                                                Reconstruction accuracy:
PRESENTATION




                                                                Method         P<=2,%
  The University of Auckland | Computer Science | New Zealand




                                                                                                                         90
                                                                                                            84
                                                                GC                                         83
                                                                                                              85
                                                                                                 77
                                                                                                                     89



                                                                                                                              97



                                                                02 June 2012            Department of Computer Science             80
                                                                           Stereo + Strip Pattern
   2009



                                                                Reconstruction accuracy:
PRESENTATION




                                                                Method         P<=2,%
  The University of Auckland | Computer Science | New Zealand




                                                                                                                          93
                                                                                                                         92
                                                                GC                                                       92
                                                                                                                          93
                                                                                                                     89
                                                                                                                          92



                                                                                                                               97



                                                                02 June 2012            Department of Computer Science              81
                                                                               Improvement to Stereo from Active
   2009


                                                                                        Illumination
                                                                Addition of the Strip colour pattern.
PRESENTATION




                                                                     SAD stereo algorithm:                                Depth map
  The University of Auckland | Computer Science | New Zealand




                                                                       Strip pattern              SAD - without                         SAD - with
                                                                                                    pattern                            strip pattern

                                                                                                    P<=2 = 80%                         P<=2 = 93%

                                                                      Pattern colour should avoid skin tones.
                                                                02 June 2012              Department of Computer Science                         82
The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                               2009


                                                                             example
                                                                             Error map
   2009



                                                                Gray code approach most accurate.
PRESENTATION




                                                                      Slower acquisition time.
                                                                Look to alternative ‘one shot’ approaches.
                                                                Photometric stereo least accurate.
  The University of Auckland | Computer Science | New Zealand




                                                                Our test set has high resolution images and large
                                                                 disparity ranges.
                                                                      O(n3) stereo algorithms – GC, BP – inappropriate.
                                                                            Long processing time.
                                                                               Parameter setting difficult.
                                                                Our results differ from the Middlebury rankings:
                                                                    http://cat.middlebury.edu/stereo/

                                                                02 June 2012                   Department of Computer Science   84
   2009




                                                                All results contain errors.
PRESENTATION




                                                                           Need post processing to clean up data.
                                                                           Even for the commercial 3D scanner.
  The University of Auckland | Computer Science | New Zealand




                                                                Faces have many unique properties posing a challenge
                                                                 for 3D reconstruction
                                                                     Human sensitivity to errors in reconstruction - we see faces
                                                                      all the time.
                                                                     For computer vision:
                                                                              Specularities.
                                                                              Anistropic reflectance of hair.
                                                                              Sub-surface scattering.
                                                                              Large homogenous regions.

                                                                02 June 2012                       Department of Computer Science    85
                                                                           Static reconstruction conclusion
   2009


                                                                           and analysis
                                                                 Framework and test-bench for active and passive 3-D
PRESENTATION




                                                                  acquisition systems designed.
                                                                 Three computer vision approaches tested.
                                                                      12 algorithms altogether.
  The University of Auckland | Computer Science | New Zealand




                                                                 Analysed accuracy of algorithms for 3D face reconstruction.
                                                                      Data compared to scanner benchmark.
                                                                 Provided new alternative test set to Middlebury for testing
                                                                  stereo algorithms
                                                                      High resolution images of faces.
                                                                 Passive stereo combined with active illumination a promising
                                                                  approach.
                                                                      Want a one shot approach for faces (moving object).
                                                                      SDPS + Strip pattern.
                                                                      Leads to real-time spatio-temporal acquisition.
                                                                           Acquire 3D face performance.

                                                                02 June 2012                    Department of Computer Science   86
                                   The University of Auckland | Computer Science | New Zealand   PRESENTATION
                                                                                                                2009




02 June 2012
  Department of Computer Science
  87
   2009



                                                                A generic face model with an abstract muscle animation
PRESENTATION




                                                                 system was designed during my Master’s thesis.
                                                                                 Refined for PhD thesis.
                                                                                Can be personalised with 3D data and texture information from the
                                                                                 static reconstruction experiment using a custom RBF mapping
  The University of Auckland | Computer Science | New Zealand




                                                                                 procedure.



                                                                                                                                       • Example of muscle contraction




                                                                   • Generic morphable face with linear
                                                                   and ellipsoid muscles                                               • A biomechanical tissue model


                                                                02 June 2012                          Department of Computer Science                                     88

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:45
posted:6/3/2012
language:English
pages:88