Curio, Bülthoff, Giese Preface Introduction Page 1 PREFACE By

Curio, Bülthoff, Giese: Preface/ Introduction Page 1 PREFACE By Cristóbal Curio1, Heinrich H. Bülthoff1 & Martin A. Giese2 1) Department of Human Perception, Cognition and Action, Max Planck Institute for Biological Cybernetics, Spemannstr. 38, 72076 Tübingen, Germany 2) Section for Computational Sensomotorics, Hertie Institute for Clinical Brain Sciences and Center for Integrative Neuroscience, University Clinic Tübingen, Germany Correspondence to be addressed to either: cristobal.curio@tuebingen.mpg.de or martin.giese@uni-tuebingen.de Curio, Bülthoff, Giese: Preface/ Introduction Page 2 The recognition of faces is a highly important visual function with central importance for social interaction and communication. Impairments of face perception, such as prosopagnosia (face blindness), can create serious social problems, since the recognition of facial identity and emotional and communicative facial expressions is crucial for the social interaction in human and non-human primates. Correspondingly, the recognition of faces and facial expressions has been a fundamental topic in neuroscience at least since two centuries (Darwin, 1872) The scientific interest in face processing has vastly increased over the last decade. However, a major part of the existing studies has focused on the processing of static pictures of faces. This is reflected by the fact that over the last ten years more than 8,000 studies on the perception of faces have been listed in the PubMed library of the U.S. National Library of Medicine and the National Institutes of Health, while only 300 listed studies treat the recognition of dynamic faces or the perception of faces from movies. The neural mechanisms of the processing of pictures of faces have been the topic of intense debates in psychology, neurophysiology and functional imaging. Relevant topics have been whether faces form a ‘special’ class of stimuli that is processed by specifically dedicated neural structures, and potentially also exploiting computational principles that are differing substantially from the ones underlying the recognition of other shapes. Most of these debates are still ongoing and no final conclusions have been drawn on many of these topics. Even less is known about the mechanisms underlying processing of dynamic faces, and systematic research on this topic has just begun. Are there special mechanisms dedicated to the processing of dynamic as opposed to static aspects of faces? Are these mechanisms anatomically strictly separate? In how far does the processing of dynamic faces exploit the same principles as the processing of static pictures of Curio, Bülthoff, Giese: Preface/ Introduction Page 3 faces? Finally, what are general computational principles of the processing of complex spatiotemporal patterns, such as facial movements, in the brain? The high biological relevance of faces makes the analysis and modeling of faces also an important problem for technology. The recognition of static and dynamic faces has become a quite mature topic in computer vision (Li & Jain, 2005), and since the proposal of the first face recognition systems in the 1960s a large number of technical solutions for this problem have been proposed. Modern computer vision distinguishes different problems related to the processing of faces, such as identity and expression recognition, face detection in images, or the tracking of faces or of the three-dimensional geometry of the face in video sequences. The recognition of faces and facial expressions has important applications, such as surveillance, biometrics and human-computer interfaces. Not only the analysis, but also the synthesis of pictures and movies of faces has become an important technical problem in computer graphics. Starting from very simple systems in the 1970s (Brennan, 1985) that generated line drawings of faces, meanwhile systems for the simulation of photo-realistic pictures of faces have been proposed. Modern systems of this type exploit data on the texture and three-dimensional structure of faces, which are obtained by special hardware systems, such as laser scanners. Highly realistic simulations have been obtained for static pictures of faces (e.g. (Blanz & Vetter, 1999)) and recently also of movies of faces (e.g. (Blanz, Basso, Poggio, & Vetter, 2003)). The simulation of realistic dynamic facial expressions by computer graphics has been fundamental for the realization of many recent movies, such as The Final Fantasy, The Polar Express, Beowulf, or the most recent production of Benjamin Button. Other application domains for the simulation of faces and facial expressions encompass facial surgery, and forensic applications. Finally, a very challenging problem, with high potential Curio, Bülthoff, Giese: Preface/ Introduction Page 4 future relevance, which scientists have started to address only recently is the simulation of facial movements for humanoid robots (Kobayashi & Hara, 1995). Dynamic faces represent thus an interesting and largely unexplored topic in neuroscience with substantial importance for technical applications. The goal of this book is to provide an overview of recent developments in the field of dynamic faces. It is an outgrowth of a workshop on dynamic faces we have organized in March 2008 at the COSYNE conference in Snowbird/ Utah. The book presents a collection of recent work within an interdisciplinary framework, written by experts in different domains including neuroscience, psychology, neurology, computational theory and computer science. The book tries to cover a broad range of relevant topics, including the psychophysics of dynamic face perception, results from electrophysiology and imaging, clinical deficits in patients with impairments of dynamic face processing, and computational models providing interesting insights about mechanisms of the processing of dynamic faces in the brain. The book tries to address equally researchers in biological science and neuroscience, and in computer science. In neuroscience, we hope that an overview of the state-of-the art of the knowledge about the processing of dynamic faces might be suitable as compendium a basis for the design of new experiments in psychology, psychophysics, neurophysiology, social and communication sciences, imaging and clinical neuroscience. At the same time, increasing knowledge about the mechanisms that underlie the processing of dynamic faces in the nervous system seems highly relevant for computer science. Such knowledge seems suitable for improving technical systems for the recognition and animation of dynamic faces taking into account the constraints and critical properties that are central for the processing of dynamic faces in biological systems. Second, principles used by the brain to recognize dynamic faces might inspire novel solutions for technical systems for the recognition and processing of dynamic faces Curio, Bülthoff, Giese: Preface/ Introduction Page 5 – similar to the inspiration of a variety of other technical solutions in computer vision by principles of biological vision. Finally, the covered experimental methods for the quantitative characterization of the perception of dynamic faces provide a basis for the validation of technical systems that analyze or synthesize dynamic facial expressions. This makes biological methods interesting for the development and optimization of technical systems in computer science. The book is organized in terms of three major Parts that cover different interdisciplinary aspects of the recognition and modeling of dynamic faces: I) Psychophysics, II) Physiology, and III) Computational approaches. Each part starts with an overview that provides a more general overview of the area from a renowned expert in this field followed by chapters discussing a spectrum of relevant approaches in more detail. In Part I of the book (Psychophysics) Alan Johnston introduces the general topic of the psychophysics of the human perception of dynamic faces (Chapter 2). He discusses both methodological and technical issues that are important to consider for experimentalists and modeler. In Chapter 3 Alice O’Toole and Dana Roark present novel insights on the role of dynamic information in face recognition. They discuss the findings in the context of what so far has defined the supplemental information and the representation enhancement hypothesis. This chapter serves also as a good introduction into these two theories. In Chapter 4 Natalie Butcher and Karen Lander present in detail a series of studies including own new experimental data revealing the dynamic characteristics that are important for face recognition. Curio, Bülthoff, Giese: Preface/ Introduction Page 6 In Chapter 5 Cristóbal Curio and his colleagues present work that exploited 3D computer graphics methods for the generation of close-to-reality facial expressions for the study of highlevel after-effects in the perception of dynamic faces. The generation of the applied dynamic facial expressions was based on an algorithm that provides low-dimensional parameterizations of facial movements by approximating them by superposition of facial action units. The study shows in particular that dynamic faces produce similar high-level after-effects as previously shown for static pictures of faces. In Chapter 6 Barbara Knappmeyer has particularly investigated the interaction between facial motion and facial form. A cluster-based animation approach allowed the exchange of characteristic motion signatures between different identities. The study shows that the perception of facial identity is modulated by the perception of individual specific motion. In Chapter 7 Harold Hill presents a detailed overview on studies that focus on facial speech perception. His thorough discussion on this topic provides insights on spatio-temporal aspects of the interplay of auditory and facial speech signals which is supported by novel data. The topic of Part II of this book is devoted to physiology and is introduced by David Leopold (Chapter 8). Besides providing a brief overview of this section he lays out his view on outstanding neural challenges. Given that facial expressions play a central role in social communication, he suggests that the neurophysiological basis of the perception and production of facial expressions, including vocalization and gaze, should be studied taking into account interative contexts. In Chapter 9 of this Book section Stephen Shepherd and Asif Ghazanfar cover neurophysiological aspects of gaze, attention and vocal signals during the perception of Curio, Bülthoff, Giese: Preface/ Introduction Page 7 expressions. They review behavioral and electrophysiological evidence that the perception of the facial dynamics and vocalization is linked In Chapter 10 Charles Schroeder and Aina Puce review human electrophysiological ERP experiments related to facial movement. Using a novel methodological approach they provide evidence that socially relevant signals, such as the gaze direction towards or away from the observer, modulates the amplitude of ERP responses. In Chapter 11 Patrick Vuilleumier and Ruthger Righart provide a more detalied review of factors that influence the ERP signal (N170) during the perception of dynamic faces. They discuss in detail evidence from their own and others work for the coupling of the perception and production of dynamic faces. In Chapter 12 Beatrice deGelder and J. Van den Stock review clinical observations relating to the dynamic information in faces. They complement the discussion on the processing of static and dynamic faces in normal subjects with insights from studies how movement affects the perception of faces in patients with various cognitive deficits, ranging from developmental prosopagnosia to brain lesions and autism spectrum disorder. Computational aspects on dynamic faces are introduced in Part III of this book by the overview chapter of Pawan Sinha (Chapter 13). He formulates a number of computational challenges that are associated with the processing of dynamic faces. The computational chapters of this section cover rather diverse topics ranging from neural modeling (Chapter 14), automatic behavior classification with applications (Chapter 15), a real-time interactive avatar system for closedloop behavior research (Chapter 16) to state-of-the-art 3D Computer Graphics providing novel 3D stimuli for challenging experiments on dynamic faces (Chapter 17). Curio, Bülthoff, Giese: Preface/ Introduction Page 8 In Chapter 14 Thomas Serre and Martin A. Giese present elements of neural theories for object, face and action recognition that might be central for the development of physiologically-inspired models for the recognition of dynamic faces. In Chapter 15 Marian S. Bartlett and her colleagues give on overview on their work on automatic expression measurements. They present a computer vision based facial expression recognition system that is based on facial action unit detectors. They review the usefulness of their tool in various interactive applications that require automatic analysis and validated their system to human performance. In Chapter 16 by Steven Boker and Jeffrey Cohn a computational approach is presented that permits to dissociate in real-time facial appearance and dynamics during natural conversation. Their analysis and animation system is one of the first approaches in behavior research that allows studying facial expressions with realistic looking avatars that can reproduce and manipulate participants’ facial action and vocal sound during interactive conversation. Part III of the book is concluded by a novel computer graphics approach suitable for the construction of 3D facial animations (Chapter 17). Christian Walder and his colleagues present a kernel-based approach for the dense three-dimensional tracking of facial movements, providing essential data that is required for realistic and controllable face animation and dynamic face space analyses. We thank many colleagues without whom the successful completion of this book would not have been impossible. We thank Andreas Bartels, Martin Breidt, Isabelle Bülthoff, Christoph D. Dahl and Johannes Schultz for reviewing and providing comments on individual chapters of the draft. Curio, Bülthoff, Giese: Preface/ Introduction Page 9 We thank Stefanie Reinhard for her support during editing the book. We thank the COSYNE conference 2008 workshop chairs Fritz Sommer and Jascha Sohl-Dickstein in their support to pursue our workshop on dynamic faces. Finally we want to express our gratitude to Bob Prior and Susan Buckley from MIT Press for their support and guidance during the completion of this book. Cristóbal Curio Max-Planck-Institute for Biological Cybernetics, Tübingen Heinrich H. Bülthoff Max-Planck-Institute for Biological Cybernetics, Tübingen Dept. of Brain and Cognitive Engineering, Korea University Martin A. Giese Section for Computational Sensomotorics, Hertie Institute for Clinical Brain Sciences and Center for Integrative Neuroscience, University Clinic Tübingen Tübingen 12.07.2009 Curio, Bülthoff, Giese: Preface/ Introduction Page 10 Literature Blanz, V., Basso, C., Poggio, T., & Vetter, T. (2003). Reanimating faces in images and video. Computer Graphics Forum, 22(3), 641-650. Blanz, V., & Vetter, T. (1999). A morphable model for the synthesis of 3D faces, In Proceedings of the 26th annual conference on Computer graphics and interactive techniques, SIGGRAPH (pp. 187-194): ACM Press/Addison-Wesley Publishing Co. Brennan, S. E. (1985). Caricature Generator: The Dynamic Exaggeration of Faces by Computer. Leonardo, 18(3), 170-178. Darwin, C. (1872). The expression of the emotions in man and animals. London,: J. Murray. Kobayashi, H., & Hara, F. (1995). A basic study on dynamic control of facial expressions for face robot. Ro-Man'95 Tokyo: 4th Ieee International Workshop on Robot and Human Communication, Proceedings, 275-280 386. Li, S. Z., & Jain, A. K. (2005). Handbook of face recognition. New York: Springer.