Diapositive by MikeJenny

VIEWS: 5 PAGES: 53

									          GEPPETO1:
A modeling approach to study the
 production of speech gestures
           Pascal Perrier (ICP – Grenoble)
                        with
            Stéphanie Buchaillard (PhD)
              Matthieu Chabanas (ICP)
                   Ma Liang (PhD),
           Yohan Payan (TIMC – Grenoble)

 1 GEstures shaped by the Physics and by a
 PErceptually oriented Targets Optimization
              Outline
• Introduction
• Current hypotheses implemented in
  GEPPETO
• Some results obtained with a 2D
  biomechanical tongue model
• New issues raised by the use of 3D
  biomechanical tongue model
          Basic issues
 in Speech Production Research
• Phonology/Phonetics Interface
  – Link between discrete representations and
    continuous physical signals
  – Nature of physical correlates of speech
    units
          Basic issues
 in Speech Production Research
• Control and Production of Speech
  Gestures
  – Control variables
  – Central representations of physical
    characteristics of the speech production
    apparatus
  – Interaction Perception-Action
         Basic issues
in Speech Production Research
• From Gestures to Speech Sounds
  – Nature of acoustic sources
  – Relations between motor commands and
    acoustics
  – Interaction between airflow and
    articulatory gestures.
       What is GEPPETO?
• An evolutive modeling framework to
  quantitatively test hypotheses about the
  control and the production of speech
  gestures.
• It includes
  – Hypotheses about the physical correlates of
    phonological units.
  – Models of motor control
  – Physical models of the speech production
    apparatus
      Current Hypotheses
• Phonology/Phonetic Interface
  – The smallest phonological unit is the
    phoneme
  – Phonemes are associated with target
    regions in the auditory domain
  – Larger phonological units are associated
    with speech sequences for which specific
    constraints exist for target optimization
    or for motor commands sequencing
      Current Hypotheses
• Control of speech gestures
  – Control variables: l commands (EP Hypothesis,
    Feldman, 1966)
  – No on line use of feedback going through the
    cortex.
  – Short-delay orosensory and proprioceptive
    feedbacks are taken into account.
  – Existence in the brain of internal
    representations of the speech apparatus
    (internal models).
       Current Hypotheses
• Control of speech gestures
  – Internal representations do not account for the
    whole physical complexity of the speech
    production apparatus
  – Kinematic characteristics are not directly
    controlled. They are the results of the
    interaction between motor control setups and
    physical phenomena of speech production
     • Which characteristics of speech signals are
       specifically controlled?
 Application to the generation
 of speech gestures with a 2 D
  biomechanical tongue model
• Implementation of the model of
  control
• Inversion from desired perceptual
  objectives to motor commands
• Generation of gestures
   2D Biomechanical Model
• Finite element structure
• Linear elasticity (small deformations)
• No account of the gravity
          2D Biomechanical Model




Posterior genioglossus   Anterior Genioglossus   Hyoglossus
     2D Biomechanical Model




Styloglossus   Verticalis   Inferior Longitudinalis
  Learning a static internal model
         From l commands to formants



Step 1:
- Uniform sampling of
the l commands space
-Generation of the
corresponding tongue
shapes.                 9000 simulations
  Learning a static internal model
         From l commands to formants
Step 2: Computation of the area function.
  Learning a static internal model
         From l commands to formants
Step 3: Formants computation for 2 lip apertures
     (red dots: spread lips; blue dots: rounded lips)
  Learning a static internal model
         From l commands to formants
Step 4: Learning and generalizing
        with radial basis functions
                1ere couche             2nd couche
                1st layer                 nd
                                        2 layer
         1
             W 11                  2
 X1                  +         F       W 11
                                       ...    +      Y1
 ...
                     +         F              ...    ...
 Xn      1W              ...                  +      Yp
               1m

         1
          W nm
                     +         F   2
                                   W mp

                m neurones             p neurones
                       Inversion
       From target regions to l commands

Target regions
•Dispersion ellipses
in the (F1, F2, F3)
space
• Currently defined
by Fc1, Fc2, Fc3 and
sF1, sF2, sF3


                       Target regions for some non rounded
                                French phonemes
                       Inversion
       From target regions to l commands

Target regions
•Dispersion ellipses
in the (F1, F2, F3)
space
• Currently defined
by Fc1, Fc2, Fc3 and
sF1, sF2, sF3


                       Target regions for some non rounded
                                French phonemes
                 Inversion
   From target regions to l commands
                  Optimization
 Cost minimization (Gradient descent technique)
   Cost for a sequence made of N phonemes


                                 +
       Speaker oriented              Listener oriented

with
          Inversion
From target regions to l commands
           Example 1
        Sequence [œ-e-k-i]
          Inversion
From target regions to l commands
           Example 2
        Sequence [œ-e-k-a]
Production of tongue movements
   from inferred l commands
          Serial command patterns
No difference between vowels and consonants
             [oe] [e]   [k]   [a]
Execution of tongue movements
  from inferred l commands
   Öhman’s model: Vowel-to-Vowel basis
Consonants are seen as perturbation of V-V
            [oe] [e]   [k]   [a]
Execution of tongue movements
  from inferred l commands
        Observed flesh point
Production of tongue movements
   from inferred l commands
      Serial command patterns
                                 [a]



                                 [i]
Production of tongue movements
   from inferred l commands
      Öhman’s command patterns
                                 [a]



                                 [i]
Interaction control / physics.
     Influence on the shapes
     of the articulatory paths
   Example: the Articulatory loops




   [aka]                     [ika]
           R. Houde (1969)
      Fluid-Wall Interaction


 Imposed
 pressure
difference                Forces
             Flow model              Mechanics of the tissues.
                                     Finite element model)


                              Deformation
          Interaction control / physics.
                          Influence on the shapes
                          of the articulatory paths
                    Example: the Articulatory loops
                                                  Deplacement X - Y

         120

         115

         110
Y - mm




         105

         100
                    +++ PS = 3000 Pa
                    ...... PS = 800 Pa
         95
                    -------No aerodynamics

         90


               40        50          60      70        80             90   100   110   120
                                                       X - mm


                                                  [aka]
 Interaction control / physics.
      Influence on the shapes
      of the articulatory paths
    Example: the Articulatory loops




No aerodynamics       With aerodynamics
                  [aka]
Interaction control / physics.
     Influence on the shapes
     of the articulatory paths
   Example: the Articulatory loops
                                        Deplacement X - Y
                            113


      ... PS = 1600 Pa 112
     ---- No aerodynamics
                            111
                   Y - mm




                            110


                            109


                            108


                            107
                              61   62    63      64    65   66   67
                                              X - mm




                                        [ika]
 Interaction control / physics.
      Influence on the shapes
      of the articulatory paths
    Example: the Articulatory loops




No aerodynamics           With aerodynamics
                  [ika]
A 3D biomechanical tongue model:
 For a better account of physics
 • Visible Human Project ® data
     (Wilhelms-Tricarico, 2003)
 • Finite Element Mesh made of Hexahedres
 • Adaptation of the mesh to a specific speaker (PB)




Wilhelms-Tricarico R.,1995          Gerard et al., ICP Grenoble
Inner muscle structure
    of the tongue



                 Genioglossus (posterior)
                 Genioglossus (medium)
                  Superiorlongitudinalis
                  Inferior longitudinalis
                       Styloglossus
                       Transversus
                       Hyoglossus
                        Geniohyoid
                        Mylohyoid
                         Verticalis
                                 (anterior)
Vocal tract structure




                  TONGUE’S BODY

                   HYOID BONE

                    MANDIBLE

                     PALATE

                  OTHER MUSCLES
Elastical properties of tongue muscles
                               Displacement




                           0                  Force

                          Linear         Non Linear
    Tongue   Indentator


• Hyperelastic material (2nd order Yeoh
  model) with large deformation
  hypothesis
Effect of gravity




                    [1s]
Dealing with gravity with the
       EP hypothesis




            [300ms]
   Dealing with gravity with the
          EP hypothesis
• Activation of GGp
  and MH
 Increase of reflex
 activity




                       [300ms]
Dealing with gravity with the
       EP hypothesis




         GGP activation
  Dealing with gravity with the
         EP hypothesis
Example of a good choice of control parameters




                  [300ms]
              Conclusions
• A model of control based on perceptual
  objectives specified in terms of formants
  target regions associated with l motor
  commands and on an optimization process
  using a static model of the motor-
  perception relations can generate realistic
  speech movements if it is applying to a
  realistic physical model of speech
  production.
            Conclusions
• It supports our hypothesis that there
  is not need to assume the existence
  of a central optimization process that
  would apply to the articulatory
  trajectories in their whole (i.e.
  minimum of jerk, minimum of torque…)
           Conclusions
• It gives an interesting account of
  coarticulation phenomena by
  separating the effects of planning
  and those of physics.
• It permits to test hypotheses about
  the phonological units (see serial
  model versus Öhman’s model).
            Conclusions
However
• a systematic comparison with data is
  required (currently in progress for
  French, German, Chinese, Japanese)
• No account for time control, or for
  hypo/hyperspeech
• No account for gravity
           Conclusions
• Necessity to work on a more complex
  internal representations that would
  integrate some aspects of
  articulatory dynamics.
    Influence of elasticity modeling
                                       Hyperelastic

Large defo.
  Linear
                                      Small defo.
                                        Linear




       Activation of the Hyoglossus (2N)
 EP Hypothesis
(Feldman, 1966)




Perrier, Ostry, Laboissière, 1996
  EP Hypothesis
 (Feldman, 1966)




Perrier, Ostry, Laboissière, 1996
            Static Internal Models

    Central Nervous System      1ere couche                       2nd couche




 Desired
                          1
                             W 11                           2
                   X1                +         F                W 11
                                                                ...       +     Y1


                                                                                                        ld   Peripheral motor
                   ...
                                     +

formants
                                               F                          ...   ...
                   Xn    1W              ...                              +     Yp


                                                                                                                  system
                               1m

                         1
                          W nm
                                     +         F            2
                                                             W mp

                                m neurones                      p neurones




            Inverse Model
                                                                                                        l

           yi(t)
                                                   1ere couche                       2nd couche
                                          1
                                             W 11                               2
                         X1                             +             F             W 11
                                                                                    ...    +      Y1
                         ...
                                                        +             F                    ...    ...



                                                                                                                         Formants
                         Xn              1W                 ...                            +      Yp
                                               1m

                                         1
                                           W nm
                                                        +             F         2
                                                                                W mp

                                                   m neurones                       p neurones




                         Direct Model
follow-
  up
Maltreatment groups did not
  differ on MAOA activity

MAOA activity moderates the        Maltreatment experienced < 11 years
 impact of early
 maltreatment on the                     severe maltreatment
 developement of violence                probable maltreatment

                                         no maltreatment
       Arguments to support this model
              Plan of the talk

• Theoritical arguments related to neural network
  modelisation

• The lack of direct genetic determinism contributing to
  externalizing disorders

• The contribution of environmental variables to
  externalizing disorders

• The contribution of animal models to understand
  consequences of stress on development

• Towards a probabilistic epigenetic model


                          Cohen, Neurosciences BioBehavioral Review, 2010
 TOXIC AND PER NATAL FACTORS that impact brain
 development either during pregnancy and/or infancy)

 MICRO-ENVIRONMENTAL VARIABLES that impact the child
 and his/her family in a proximal way
 Low socioeconomic status, Early separation, Single/disrupted parent (father
 absence), Large number of siblings, Individual handicap or poor social skills,
 Sexual and/or physical abuse, Family violence and/or alcoholism, Mentally ill
 parent, Parental use of punishment as opposed to reward

 MACRO-ENVIRONMENTAL VARIABLES that impact at a
 more general societal level
 Urban residency, TV exposure, Minority, Rejection from school, Inclusion in
 at-risk pro-social alternatives (peer grouping: e.g., ganging; drug abuse),
 Competitive and violent culture

environt

                     epigenesis                          Stress & non genomic
                                           Disorder
                                                        transmission of behaviors


Cohen, Neuroscience BioBehav Rev 2010                       Mealey L, BBS. 1995
Maltreatment and antisocial behaviors: e.g. MICRO

             5 years                                7 years




                  Jaffee et al, J Abnorm Psychol. 2004


   Twin study – E-risk study - N=1116 same sex twin pairs

     Physical maltreatment plays a causal role in the
      development of children’s antisocial behavior
   Maternal depression and antisocial behaviors: e.g.
                        MICRO

Kim-Cohen et al.
Arch Gen Psy 2005

Twin study – E-risk
study
N=1116 same sex twin
pairs

Post natal maternal
depression predicted
antisocial behaviors at
age 7:
-with a ”dose effect”
-after controlling for
mother’s and father’s
ASPD
-with additive effects
-with high heritability
Intergenerational transmission of childhood
      conduct problems: a twin study

D’Onofrio et al. 2007 Arch Gen Psy

High-risk sample – 889 twin families – N=2554
Australia

Using the number of conduct disorder symtoms, there was
  a significant intergenerational transmission for all
  offsprings (male>female)

In male, largely mediated by environmental
  variables

In female, not the same. A common genetic liability
   accounted for the intergenerational relations
      Affiliation to an antisocial group during
               adolescence: e.g. MACRO




                 1                       Getting inside the
                                         group of ASB

                                         Leaving the group
                                          of ASB




Two trajectories regarding admission to a group:
   Early-adolescence then mid adolescence Lacourse et al.
                                             Dev Psych 2006
       Arguments to support this model
              Plan of the talk

• Theoritical arguments related to neural network
  modelisation

• The lack of direct genetic determinism contributing to
  externalizing disorders

• The contribution of environmental variables to
  externalizing disorders

• The contribution of animal models to understand
  consequences of stress on development

• Towards a probabilistic epigenetic model


                          Cohen, Neurosciences BioBehavioral Review, 2010
     Ealy stress, inproper maternal care, stress during gestation
     influence development through non genomic transmissions
     of behaviors in animal models

     Victor Denenberg (1970) – Michael Meaney (2000)



environt
                    epigenesis                       Stress & non genomic
                                         Disorder
                                                    transmission of behaviors
    Early experiences and epigenetic programming


Early experiences have long-term effects on behavioral and
biological systems: e.g. handling (pup/mother separation)
Denenberg et al, Science, 1967; Liu et al., Science, 1997



Early experiences affect future generation providing a non-
genomic mechanism for the transmission of behavioral traits
Denenberg & Whimbey, Science, 1963; Francis et al., Science, 1999



The uterine environment affects development through
environmental factors rather than genetic ones
Denenberg et al, Neuroreport, 1998; Francis et al., Nat Neurosci, 2003
Maternal care impacts development through behavioral
 programming and responses to stress in adulthood…


Maternal care ( pup licking/grooming and arched back nursing)
influences:

- Hypothalamic-Pituitary-Adrenal responses to stress in offsprings
Liu et al. Science, 1997

- Hippocampal synaptogenesis, plasticity and spatial learning and
memory
Liu et al. Nat Neurosci, 2000; Mirescu et al. Nat Neurosci, 2004

- The offspring epigenome in the hippocampus; and reversal
occurs when (1) cross-fostering is proposed; (2) histone
deacetylase is infused in early post natal
Weaver et al. Nat Neurosci, 2004
  Rats models
    implicate
   epigenetic
 regulation of
 hippocampal
glucocorticoid
 R expression
in mediationg
 the effects of
    early life
experience on
adult behavior

How adversity
gets under the
     skin?

                  Hyman et al. Nat Neurosc 2009
     Family context, early interactions and antisocial
                 behaviors: e.g. MICRO

Caspi et al. Dev Psychopathol,
2004
Twin study – E-risk study
N=565 MZ twin pairs

The twin receiving more
maternal negative emotional
expression and less warmth at
age 5 had more antisocial
behaviors at age 7


Jaffee et al. Child Dev, 2003
Twin study – E-risk study – N=1116 twin pairs

The less time father with low antisocial behaviors lived with their
  children, the more conduct problems their children had receiving
When father engaged in high levels of antisocial behavior, the more
  time they lived with their children, the more conduct problems
  their children had
         Reversibility of adverse rearing condition
      consequences by restoring normal family rearing

Adoption study
N=144 Romanian (≤42 months) adoptees
reared in very depriving institutions
vs. 52 UK born (≤6 months) adoptees

At entry, cognitive impairment was
associated with institutional deprivation
and with its duration

At follow-up (6 years of age):
- There was a remarkable degree of
recovery after restoration of normal
family rearing
-Major deficits persisted in a
substantial minority: General Cognitive
Index was 25 points lower in those
who entered the UK after 2 years vs.
those who entered before 6 months
whatever the degree of malnutrition

                                            Rutter et al, Dev Psychol. 2004
Institutional Rearing and Psychiatric Disorders
        in Romanian Preschool Children




                                   Zeanah et al, Am J Psy. 2009
     Institutional Rearing and Psychiatric Disorders
             in Romanian Preschool Children


Children with any history of institutional rearing had
  more psychiatric disorders than children without
  such a history (53.2% vs. 22.0%)

Children removed from institutions and placed in foster
  families were less likely to have internalizing
  disorders than children who continued with care as
  usual (22.0% vs. 44.2%)

Boys were more symptomatic than girls regardless of
  their caregiving environment
Unlike girls, boys had no reduction in total psychiatric
  symptoms following foster placement

                                         Zeanah et al, Am J Psy. 2009
Timing of intervention affects brain electrical activity in
   children exposed to severe psychosocial neglect
Scalp topography of alpha power Mean alpha power across sites




 Children exposed to PS neglect (institution) and cared as usual, N=48
 Children exposed to PS neglect (institution) and placed in foster care
 after 24 months, N=28
 Children exposed to PS neglect (institution) and placed in foster care
 before 24 months, N=25
 Children never institutionalized, N=42
                                                  Vanderwert et al, PlosOne. 2010
Association of exposure to peer verbal abuse with
     elevated psychiatric symptom scores




                                    Teicher et al, Am J Psy. 2010
Association of exposure to peer verbal abuse with
         corpus callosum abnormalities

                                 N=63
                                 No history of sexual abuse
                                 No history of physical abuse
                                 Peer verbal abuse score > 30




                                 %




                                     Teicher et al, Am J Psy. 2010
       Arguments to support this model
              Plan of the talk

• Theoritical arguments related to neural network
  modelisation

• The lack of direct genetic determinism contributing to
  externalizing disorders

• The contribution of environmental variables to
  externalizing disorders

• The contribution of animal models to understand
  consequences of stress on development

• Towards a probabilistic epigenetic model


                          Cohen, Neurosciences BioBehavioral Review, 2010
 Developmental view of externalizing disorders

                                Family factors
Protective factors           Psychosocial factors
                                Peer grouping
                                 Drug abuse
                              School exclusion


                         ODD                   CD                     ASP
  Cultural factors
Psychosocial factors
 Hostile parenting                            Impulsivity
Early life adversities                     Aggressiveness
   Toxic factors                             Hyperactivity
  Genetic factors                  Empathy/Callous-unemotional traits
 Biological factors                     Narcissism/Self-esteem
                         ADHD            Insecure attachment
                                            Manic defense


                         Child             Adolescent             Young Adult

                                        Modified from Loeber et al, JAACAP. 2000
       Metatheoritical model of probabilistic epigenesis
                 [≠ predetermined epigenesis]

             ENVIRONMENT                        Individual development
 Physical – Social – Cultural




                                                                                  BI-DIRECTIONAL
   BEHAVIOR and CONDUCT

                  NEURAL LEVEL

                 GENETIC LEVEL
                                          Early                Adolescence
                                          interaction

In this view, neural (and other) structures begin to function before they are
fully mature, and this activity, whether intrinsically derived or extrinsically
stimulated plays a significant role in the developmental process.
Since the coordination of formative functional and structural influences within
and between all levels of analysis is not perfect, a probabilistic element is
introduced in all developing systems and their outcome.
Some tension may occur (e.g. early life, culture, CU traits)
                                            Modified from Goetlieb, Dev Science, 2007
           Rearing condition, 5HTT polymorphism and (i)
            response to stress or (ii) alcohol preference
from Caspi and Moffitt, Nat Rev Neurosci. 2006




                                                                     Mother-           Peer-
                                                                     reared           reared



  Carence                Modulation                  Axe            Consommation
  précoce                génétique               corticotrope          d’alcool

Barr et al, Biol Psy. 2004                                      Barr et al, Arch Gen Psy. 2004
Epigenetic regulation of the glucocorticoid R
         in human brain associates
     with childhood abuse and suicide




                           McGowan et al. Nat Neurosci 2009
Hippocampal glucocorticoid R expression
        (a) total GR; (b) GR 1F




                       Increased cytosine
                    methylation of the NR3C1
                         promoter in the
                     hippocampus of abused
                     individuals who suicide

                       McGowan et al. Nat Neurosci 2009
As clinician, let’s work within a developmental
framework, a dimensional assessment and an
               integrative approach




                           Psycho
            Organic        Social
                                       Familial

                            Psychodynamic

      SUBJET               Cognitive

                           Clinical

								
To top