VIEWS: 8 PAGES: 39 POSTED ON: 5/1/2011
ENTRAINMENT OF SPEECH & GESTURE: EFFECTS OF PERTURBATION & PROSODY Heather Leavy Rusiewicz, Ph.D, CCC-SLP Duquesne University Susan Shaiman, Ph.D., CCC-SLP University of Pittsburgh Jana M. Iverson, Ph.D. University of Pittsburgh Neil Szuminsky, M.A. University of Pittsburgh Technical Session 1767 Presented at the 2010 ASHA Convention Friday, November 19th, 2010 Background The big questions…. Interaction Do speech and gesture interact? Timing Do changes in spoken language affect Mechanism the temporal parameters of gesture? What is the mechanism of this interaction? Theoretical Background Traditional Account: Speech and gesture are seamlessly integrated (McNeill, 1985; 1992). Phonological Account: Sketch Model (de Ruiter, 1998; 2000) Interactionof speech and gesture occurs at the level of the conceptualizer Processes below the level of the conceptualizer, such as prosodic stress assignment, do not affect the timing of gesture Mechanism of synchronization Even if gestures temporally co-occur with prosodically stressed syllables, it is not clear why this alignment exists. Motor account: Temporal entrainment of the oral and manual motor systems via the coupling of gesture movements to the rhythmic production of prominent syllables. Entrainment of speech & gesture Two internal, coupled oscillators that are self- entrained Prominent syllables act as pulses that can then entrain other oscillators Entrainment results in temporal synchronization of speech and gesture Speech Movement Entrainment of prominent pulses Gesture Movement Temporal Entrainment of Oral and Manual Systems Coupling of rhythmical oral and manual movements in early development (Iverson & Thelen, 1999) Entrainment of prominent syllables and nonspeech movements (e.g., Tuite, 1993) Changes in amplitude & frequency of movement due to interactions of repetitive movements of the speech and finger tapping (Kelso, Tuller & Harris, 1981; Smith, McFarland, & Weber, 1986; Franz, Zelaznik, & Smith, 1992) Speech rhythm is a neurocognitive oscillator that produces pulses that affect the motor system by time-locking events across different modalities (Port, 2003) Speech-Gesture Synchrony Keyfindings regarding speech-gesture synchronization: Body movements co-occur with prominent syllables (e.g., de Ruiter, 1998; Rochet-Cappellan, Laboissiére, Galvàn, & Scwartz, 2008) of gesture during the perturbation of speech Cessation and atypical dysfluencies (e.g., Mayberry &Jaques, 2000; McNeill, 1992) Gestures precede or synchronize with their lexical affiliate (e.g, Morrel-Samuels & Krauss) Purpose The purpose of the experiment was to explore the theoretical temporal relationship of speech and gesture as a function of: Syllable Position Prosodic Stress Perturbation: First vs. Neutral vs. Non-Altered Second Contrastive Feedback vs. Position Pitch Accent Delayed Auditory Feedback Predictions Syllable Position Prosodic Stress Perturbation 1. > Synchrony 1. > Synchrony 1. > total gesture time of gestures of gestures for trials with DAF with first with accented syllables position 2. Synchronization of syllables speech and gesture = for trials with and without DAF Gestures Deictic (i.e., pointing) gestures were the focus of this investigation. Onset Offset Apex Syllable Position 30 pairs of bisyllabic compound nouns represented by color illustrations 15 of the pairs shared the first syllable lifeboat /lifeguard 15 of the pairs shared the second syllable keyboard / surfboard Prosodic Stress Each target syllable produced with contrastive stress and neutral stress lifeboat vs. LIFEboat surfboard vs. surfBOARD Perturbation Auditory delay of 200 ms DAF results in slowed speech Slowed speechslowed gestures Same responses produced with and without an auditory delay Methodology Participants and Experimental Design Within-participants repeated measures design 15 right-handed monolingual American English speakers (M=25.1 years, SD=3.2 years) Back-projection of Headphones, lapel mic, & Head mounted mic: stimuli onto screen Facilitator: amplification & acoustic recording delay Optical sensor: gesture on- & offset Theremin: gesture apex Equipment Setup Time of gesture onset and Gesture Apex offset recorded with optical Gesture Gesture Onset Offset sensor at starting position. Time of gesture apex recorded using voltage trace generated by theremin Procedures Presentation of Stimuli cont. Computer-presented instructions, familiarization procedure and eight practice trials preceded the experimental trials. Text prompt provided Is the lifeboat above the square? Picture display shown Spoken response and deictic gesture to target picture Trials 240 trials 30 in each condition 1st syllable - neutral 1st syllable - contrastive 2nd syllable - neutral 2nd syllable - contrastive All responses spoken both with and without DAF DAF NAF Data Reduction & Analysis Data Reduction Speech Vowel onset and offset (Higgins & Hodge, 2002; Shriberg, Campbell, Karlsson, Brown, McSweeney, & Nadler, 2003). Time of vowel midpoint Sentence onset and offset Gesture Time of gesture apex Time of onset and offset of gesture Time of gesture launch midpoint Gesture Launch Midpoint-Vowel Midpoint (GLM-VM) Gesture Launch Midpoint vs. Gesture Apex gesture vowel midpoint (VM) launch midpoint (GLM) ↑synchrony:↓GLM-VM gesture apex Predictions revisited pitch accented syllables first position syllables Synchrony maintained for DAF trials Total Gesture Time Predictions revisited ↑ total gesture time for DAF trials Gesture Gesture Onset Offset Results Speech Perturbation and Utterance Duration All participants produced longer utterance times on average with DAF (difference of 120 to 2346 ms) On average, sentence duration times were 867 ms longer for DAF trials than NAF trials [F(1,14)=39.420, p<.0000] 3500 3123.35 3000 2500 2256.8 Sentence duration (ms) 2000 1500 NAF 1000 DAF 500 0 NAF DAF GLM-VM interval Significant main effects of: Syllable position (> synchrony for 1st position) [F(1,14)=27.848, p<.000] Contrast (> synchrony for pitch accented syllables) [F(1,14)=5.301, p<.037] Perturbation (< synchrony for DAF condition) [F(1,14)=32.932, p<.000] 2.7 Control 2.65 Experimental 2.62* No significant interactions 2.6 2.57* 2.57* 2.55 2.5* 2.5 2.44* 2.45 GLM-VM interval 2.4* 2.4 2.35 (ms) Control: neutral, 1st, NAF 2.3 2.25 Experimental: contrastive, 2nd, DAF Contrast Position Perturbation Total Gesture Time Significant main effects of: Syllable position [F(1,14)=6.433, p<.025] Contrast [F(1,14)=10.087, p<.007] Increased time to complete entire gesture for accented syllables and second position syllables NS increase for DAF Control 1800 Experimental 1700 1632.26* 1609.4 trials 1600 1500 1477.16* 1534.34* 1575.09* 1500.03 Total gesture 1400 time (ms) 1300 1200 1100 Control: neutral, 1st, NAF 1000 Experimental: contrastive, 2nd, DAF Contrast Position Perturbation Total Gesture Time Significant contrast x position interaction [F(1,14)=23.004, p<.000] Post-hoc analysis: Total gesture time significantly different only contrastive pitch accent on 2nd [t(30)=17.50, p<.0000], not 1st syllables [t(30)=1.57, p<.063] 1900 1800 1700 1671.87 1592.64 1600 Total gesture time (ms) 1500 1478.3 1476.03 1400 1300 Contrast Neutral 1200 1st Position 2nd Position Linking to Predictions Position Syllable Position: upheld predictions Tighter synchrony for 1st position syllables (GLM-VM) Prosody Longer gestures for 2nd position syllables, especially Perturbation when stressed Prosodic Stress: upheld predictions Tighter synchrony for accented syllables (GLM-VM) Longer gestures for accented syllables, especially in the 2nd position Speech Perturbation: counter to predictions Decreased synchronization for DAF trials Non-significant increase in gesture time Discussion Effects of Speech Perturbation Greater asynchrony of speech and gesture Comparable to McNeill (1992) Gesture type Spoken response Non-significant lengthening of gestures Consistent duration changes Re-thinking effect size Linking to Theory Interaction Changes in spoken language affect temporal parameters of gesture Timing Gestures aligned with speech signal, especially Mechanism for prominent syllables Gestures were lengthened as a function of contrastive stress and syllable position Second position accented syllables were longest in duration and the most influential in gestural temporal changes Coordination of pulses within speech and gesture systems Second position accented syllables Duration Longest in duration Effort Increased motoric effort (Goffman & Malin, 1999; Automaticity Smith, McFarland, & Weber, 1986; Tuller, Harris, & Kelso, 1982) Atypical stress assignment requires non- automatic phonological encoding and motor programming Act as a pulse that is an attractor for the pulse of the manual movement (Port, 2003) Limitations and Implications Manual movement requirements Task and Stimuli Speech and/or Gesture Perturbation Appropriate identification and measure of pulses Acknowledgements SHRS Development Fund Diane Williams, Ph.D. (Duquesne University) J. Scott Yaruss, Ph.D. (University of Pittsburgh) Chris Dollaghan, Ph.D.(University of Texas at Dallas) Tom Campbell, Ph.D. (University of Texas at Dallas) Elaine Rubenstein, Ph.D. (University of Pittsburgh) Thank you…Questions?
Pages to are hidden for
"entrainment of speech and gesture_Rusiewicz_ASHA2010"Please download to view full document