Docstoc

entrainment of speech and gesture_Rusiewicz_ASHA2010

Document Sample
entrainment of speech and gesture_Rusiewicz_ASHA2010 Powered By Docstoc
					ENTRAINMENT OF SPEECH & GESTURE:
EFFECTS OF PERTURBATION & PROSODY
                 Heather Leavy Rusiewicz, Ph.D, CCC-SLP
                          Duquesne University
                 Susan Shaiman, Ph.D., CCC-SLP
                          University of Pittsburgh
                 Jana M. Iverson, Ph.D.
                          University of Pittsburgh
                 Neil Szuminsky, M.A.
                          University of Pittsburgh

        Technical Session 1767 Presented at the 2010 ASHA Convention
        Friday, November 19th, 2010
Background
The big questions….

Interaction
                Do speech and gesture interact?
Timing
                Do changes in spoken language affect
Mechanism
                 the temporal parameters of gesture?
                 What is the mechanism of this
                  interaction?
Theoretical Background
    Traditional Account: Speech and gesture are
     seamlessly integrated (McNeill, 1985; 1992).
    Phonological Account: Sketch Model (de Ruiter, 1998;
     2000)

      Interactionof speech and gesture occurs at the level
       of the conceptualizer
      Processes below the level of the conceptualizer, such
       as prosodic stress assignment, do not affect the
       timing of gesture
Mechanism of synchronization

     Even if gestures temporally co-occur with
      prosodically stressed syllables, it is not clear why
      this alignment exists.

     Motor account: Temporal entrainment of the
      oral and manual motor systems via the coupling
      of gesture movements to the rhythmic production
      of prominent syllables.
Entrainment of speech & gesture
   Two internal, coupled oscillators that are self-
    entrained
   Prominent syllables act as pulses that can then
    entrain other oscillators
   Entrainment results in temporal synchronization of
    speech and gesture
Speech
Movement




           Entrainment
           of prominent
              pulses



Gesture
Movement
Temporal Entrainment of
Oral and Manual Systems
     Coupling of rhythmical oral and manual movements
      in early development (Iverson & Thelen, 1999)

     Entrainment of prominent syllables and nonspeech
      movements (e.g., Tuite, 1993)

     Changes in amplitude & frequency of movement
      due to interactions of repetitive movements of the
      speech and finger tapping (Kelso, Tuller & Harris, 1981; Smith,
      McFarland, & Weber, 1986; Franz, Zelaznik, & Smith, 1992)


     Speech rhythm is a neurocognitive oscillator that
      produces pulses that affect the motor system by
      time-locking events across different modalities (Port,
      2003)
Speech-Gesture Synchrony
 Keyfindings regarding speech-gesture
 synchronization:

   Body     movements co-occur with prominent syllables
   (e.g., de Ruiter, 1998; Rochet-Cappellan, Laboissiére, Galvàn, & Scwartz, 2008)



            of gesture during the perturbation of speech
   Cessation
   and atypical dysfluencies (e.g., Mayberry &Jaques, 2000; McNeill, 1992)

   Gestures        precede or synchronize with their lexical
   affiliate (e.g, Morrel-Samuels & Krauss)
 Purpose

The purpose of the experiment was to explore the theoretical
temporal relationship of speech and gesture as a function of:

  Syllable Position        Prosodic Stress       Perturbation:

     First vs.           Neutral vs.          Non-Altered
      Second               Contrastive           Feedback vs.
      Position             Pitch Accent
                                                 Delayed
                                                 Auditory
                                                 Feedback
Predictions

 Syllable Position        Prosodic Stress        Perturbation

1.   > Synchrony     1.     > Synchrony     1.    > total gesture time
     of gestures            of gestures           for trials with DAF
     with first             with accented
                            syllables
     position
                                            2.    Synchronization of
     syllables
                                                  speech and gesture
                                                  = for trials with
                                                  and without DAF
Gestures

    Deictic (i.e., pointing)
     gestures were the focus
     of this investigation.




                                   Onset
                                   Offset
                                   Apex
    Syllable Position

   30 pairs of bisyllabic compound nouns
    represented by color illustrations

     15   of the pairs shared the first syllable
       lifeboat   /lifeguard

     15   of the pairs shared the second syllable
       keyboard    / surfboard
Prosodic Stress

   Each target syllable produced with contrastive stress
    and neutral stress
      lifeboat vs. LIFEboat

      surfboard vs. surfBOARD
    Perturbation

   Auditory delay of 200 ms
   DAF results in slowed speech
   Slowed speechslowed gestures
   Same responses produced with and without an
    auditory delay
Methodology
Participants and Experimental Design

    Within-participants repeated measures design
    15 right-handed monolingual American English
     speakers (M=25.1 years, SD=3.2 years)
                                                            Back-projection of
Headphones, lapel mic, &       Head mounted mic:            stimuli onto screen
Facilitator: amplification &   acoustic recording
delay
                                 Optical sensor:
                                 gesture on- & offset




                                                  Theremin:
                                                  gesture apex



              Equipment Setup
   Time of
   gesture
onset and                                       Gesture Apex
     offset
recorded
      with
    optical           Gesture                            Gesture
                       Onset                             Offset
 sensor at
  starting
  position.

              Time of gesture apex recorded using voltage trace
              generated by theremin
Procedures
Presentation of Stimuli cont.
   Computer-presented instructions, familiarization
    procedure and eight practice trials preceded the
    experimental trials.
   Text prompt provided
              Is the lifeboat above the square?
   Picture display shown
   Spoken response and deictic gesture to target
    picture
Trials
   240 trials
   30 in each condition
       1st syllable - neutral
       1st syllable - contrastive
       2nd syllable - neutral
       2nd syllable - contrastive
   All responses spoken both
    with and without DAF
                                           DAF
                                     NAF
Data Reduction & Analysis
Data Reduction
   Speech
     Vowel      onset and offset (Higgins & Hodge, 2002; Shriberg, Campbell, Karlsson, Brown,
      McSweeney, & Nadler, 2003).

     Time of vowel midpoint
     Sentence onset and offset

   Gesture
     Time of gesture apex
     Time of onset and offset of gesture

     Time of gesture launch midpoint
      Gesture Launch Midpoint-Vowel Midpoint
      (GLM-VM)

   Gesture Launch Midpoint
    vs. Gesture Apex               gesture             vowel midpoint (VM)
                                    launch
                                  midpoint
                                     (GLM)

   ↑synchrony:↓GLM-VM                       gesture
                                             apex




   Predictions revisited
      pitch accented syllables

      first position syllables

      Synchrony maintained
       for DAF trials
      Total Gesture Time


   Predictions revisited
      ↑ total gesture
       time for DAF trials


                             Gesture   Gesture
                              Onset    Offset
Results
Speech Perturbation and Utterance Duration

   All participants produced longer utterance times on
    average with DAF (difference of 120 to 2346 ms)
   On average, sentence duration times were 867 ms
    longer for DAF trials than NAF trials [F(1,14)=39.420, p<.0000]
                                                      3500
                                                                      3123.35
                                                      3000


                                                      2500
                                                             2256.8
                             Sentence duration (ms)




                                                      2000


                                                      1500
                                                                                NAF

                                                      1000                      DAF


                                                       500


                                                         0
                                                              NAF      DAF
   GLM-VM interval

      Significant main effects of:
        Syllable     position (> synchrony for 1st position)
          [F(1,14)=27.848, p<.000]

        Contrast      (> synchrony for pitch accented syllables)
          [F(1,14)=5.301, p<.037]

        Perturbation       (< synchrony for DAF condition)                                     [F(1,14)=32.932,
          p<.000]                                          2.7                  Control
                                                          2.65                  Experimental                        2.62*
      No significant interactions                         2.6   2.57*                             2.57*
                                                          2.55
                                                                         2.5*
                                                           2.5
                                                                                                            2.44*
                                                          2.45
                                        GLM-VM interval




                                                                                           2.4*
                                                           2.4
                                                          2.35
                                        (ms)




Control: neutral, 1st, NAF                                 2.3
                                                          2.25
Experimental: contrastive, 2nd, DAF                                Contrast                    Position     Perturbation
   Total Gesture Time
      Significant main effects of:
        Syllable   position [F(1,14)=6.433, p<.025]
        Contrast [F(1,14)=10.087, p<.007]

      Increased time to complete entire gesture for
       accented syllables and second position syllables
       NS increase for DAF
                                                                                                             Control
                                                        1800
                                                                                                            Experimental
                                                        1700              1632.26*                                          1609.4
       trials                                           1600

                                                        1500
                                                               1477.16*
                                                                                     1534.34*
                                                                                                  1575.09*
                                                                                                                   1500.03
                                        Total gesture




                                                        1400
                                        time (ms)




                                                        1300

                                                        1200

                                                        1100
Control: neutral, 1st, NAF
                                                        1000
Experimental: contrastive, 2nd, DAF                                   Contrast                  Position               Perturbation
    Total Gesture Time
   Significant contrast x position interaction [F(1,14)=23.004, p<.000]
     Post-hoc analysis: Total gesture time significantly different
      only contrastive pitch accent on 2nd [t(30)=17.50, p<.0000], not 1st
      syllables [t(30)=1.57, p<.063]
                                            1900


                                            1800


                                            1700
                                                                             1671.87
                                                           1592.64
                                            1600
                            Total gesture
                            time (ms)




                                            1500
                                                                             1478.3
                                                           1476.03
                                            1400


                                            1300
                                                                             Contrast
                                                                             Neutral
                                            1200
                                                   1st Position      2nd Position
Linking to Predictions

Position
                 Syllable Position: upheld predictions
                      Tighter synchrony for 1st position syllables (GLM-VM)
Prosody

                      Longer gestures for 2nd position syllables, especially
Perturbation
                      when stressed
                  Prosodic Stress: upheld predictions
                      Tighter synchrony for accented syllables (GLM-VM)
                      Longer gestures for accented syllables, especially in
                       the 2nd position
                  Speech Perturbation: counter to predictions
                      Decreased synchronization for DAF trials
                      Non-significant increase in gesture time
Discussion
Effects of Speech Perturbation
   Greater asynchrony of speech and gesture
     Comparable   to McNeill (1992)
     Gesture type
     Spoken response

   Non-significant lengthening of gestures
     Consistent duration changes
     Re-thinking effect size
Linking to Theory

Interaction
                Changes in spoken language affect temporal
                  parameters of gesture
Timing

                 Gestures aligned with speech signal, especially
Mechanism

                  for prominent syllables
                 Gestures were lengthened as a function of
                  contrastive stress and syllable position
                 Second position accented syllables were longest
                  in duration and the most influential in gestural
                  temporal changes
                 Coordination of pulses within speech and
                  gesture systems
    Second position accented syllables

Duration
                 Longest in duration
Effort
                 Increased motoric effort (Goffman & Malin, 1999;
Automaticity
                  Smith, McFarland, & Weber, 1986; Tuller, Harris, & Kelso, 1982)

                  Atypical stress assignment requires non-
                   automatic phonological encoding and
                   motor programming
                  Act as a pulse that is an attractor for
                   the pulse of the manual movement (Port,
                   2003)
Limitations and Implications
   Manual movement requirements
   Task and Stimuli
   Speech and/or Gesture Perturbation
   Appropriate identification and measure of pulses
Acknowledgements
 SHRS Development Fund
 Diane Williams, Ph.D. (Duquesne University)

 J. Scott Yaruss, Ph.D. (University of Pittsburgh)

 Chris Dollaghan, Ph.D.(University of Texas at
  Dallas)
 Tom Campbell, Ph.D. (University of Texas at
  Dallas)
 Elaine Rubenstein, Ph.D. (University of Pittsburgh)
Thank you…Questions?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:8
posted:5/1/2011
language:English
pages:39