Dialogue on Dialogues
Multidisciplinary Evaluation of
Interspeech2006 - ICSLP
Michael McTear Kristiina Jokinen Jim Larson
Workshop Theme: Advanced
Speech-based Interactive Systems
• Advanced in terms of characteristics of spoken
language interaction, such as:
speech segmentation, disfluencies, turn-taking,
emotions, and adaptation (see Shriberg,
• How to incorporate these aspects in advanced
speech-based interactive systems
• How to evaluate these systems
e.g. are they more effective if they incorporate
more advanced characteristics?
Theory and Practice
• Relationship between theoretical work on
dialogue modelling and the practices and
requirements of practical interactive speech
• Is academic research on dialogue relevant to
• Are the methods developed in industry to ensure
successful applications relevant to the academic
Empirical Methods v Theoretical
• How or whether statistical speech-based
research can be applied to dialogue modelling
• Dialogue modelling is concerned with advanced
architectures and functionalities
• Are the methods used in statistical based
approaches applicable to advanced dialogue
Workshop Goals and Methods
• Encourage dialogue on dialogues
– Five different discussion sessions
• Engage in multidisciplinary discussions
– Computer Science, Computational Linguistics,
Speech and Language Technology, HCI
• Work on evaluation criteria
– The scribes reports on the results of the discussion
1. Evaluation criteria
• Evaluating Real-time Responsiveness in Dialog by Nigel
• Development and evaluation of the DIHANA project
dialog systems by D. Griol, F. Torres, L. Hurtado, S.
Grau, F. Garcia, E. Sanchis, and E. Segarra
• Challenges in evaluating spoken dialog systems that
reason and learn by Gregory Aist, Phil Michalak, George
Ferguson, James Allen
– What metrics should be used to measure static dialogs?
– What metrics should be used to measure dialogs that learn?
2. Semi-automatic design of
• Dialog Studio: An Example-Based Spoken Dialog
System Development Workbench by Sangkeun Jung,
Cheongjae Lee, Gary Geunbae Lee
• Reinforcement Learning for Spoken Dialogue Systems:
Comparing Strengths and Weaknesses for Practical
Deployment by Tim Paek
– What techniques can apply to semi-automatic design of dialogs?
– How to evaluate these techniques?
3. Methodologies for improving
• A WOz Variant with Contrastive Conditions by
Ester Levin and Rebecca Passonneau
• Human-centered Development of Interactive
Systems: Improving Usability in Early Lifecycles
Stages by Zoraida Callejas, Ramón López-
– How to evaluate alternative methodologies for
improving dialog design?
– What metrics to use?
4. Modelling dialogs
• Activity-based dialogue analysis as evaluation
method by Bilyana Martinovska, Ashish Vaswani
• Unifying language modeling capabilities for
flexible interaction by Deryle Lonsdale and
• Two faces of spoken dialogue systems by Jens
Edlund, Mattias Heldner & Joakim Gustafson
– How to evaluate alternative modelling techniques
– What metrics to use?
5. Multimodal dialogs and visual
• DS-UCAT: A new Multimodal Dialogue System for an
Academic Application by Ramón López-Cózar, Zoraida
Callejas, Germán Montoro
• Computer vision, eye tracking, spoken dialog systems,
and evaluation: Challenges and opportunities by Gregory
– What are the advantages and disadvantages of the X+V
approach to implementing multimodal dialogs?
– What are alternative architectures to X+V?
– How does visual input affect dialogs?
– Can X+V be extended to support visual input?
How to disseminate the results of this workshop to the speech dialogue community?
– Paper targeted to speech dialogue practitioners for publication in Speech
Technology Magazine or similar magazine
– Paper targeted to the academics community for publication in a referred
journal (special issue)
– Proceedings of the workshop (Ramon)
– Edit minutes and post on workshop web site
– Wikipage/blog (Jens)
– ISCA webpage
– Pls link the website!!
– Next Interspeech in Antwerp
– Next SpeechTech in S.F
– MapTask-based evaluation challenge
• Who will author/edit result?
• Process for authoring/editing the result
• Other next steps?