Dialogue on Dialogues Multidisciplinary Evaluation of Advanced Speech-based Interactive Systems Interspeech2006 - ICSLP Satellite Workshop Pittsburgh Michael McTear Kristiina Jokinen Jim Larson Workshop Theme: Advanced Speech-based Interactive Systems • Advanced in terms of characteristics of spoken language interaction, such as: speech segmentation, disfluencies, turn-taking, emotions, and adaptation (see Shriberg, Interspeech 2005) • How to incorporate these aspects in advanced speech-based interactive systems • How to evaluate these systems e.g. are they more effective if they incorporate more advanced characteristics? Theory and Practice • Relationship between theoretical work on dialogue modelling and the practices and requirements of practical interactive speech applications. • Is academic research on dialogue relevant to industrial applications? • Are the methods developed in industry to ensure successful applications relevant to the academic community? Empirical Methods v Theoretical Approaches • How or whether statistical speech-based research can be applied to dialogue modelling • Dialogue modelling is concerned with advanced architectures and functionalities • Are the methods used in statistical based approaches applicable to advanced dialogue modelling? Workshop Goals and Methods • Encourage dialogue on dialogues – Five different discussion sessions • Engage in multidisciplinary discussions – Computer Science, Computational Linguistics, Speech and Language Technology, HCI • Work on evaluation criteria – The scribes reports on the results of the discussion 1. Evaluation criteria • Evaluating Real-time Responsiveness in Dialog by Nigel G. Ward • Development and evaluation of the DIHANA project dialog systems by D. Griol, F. Torres, L. Hurtado, S. Grau, F. Garcia, E. Sanchis, and E. Segarra • Challenges in evaluating spoken dialog systems that reason and learn by Gregory Aist, Phil Michalak, George Ferguson, James Allen • Discussion: – What metrics should be used to measure static dialogs? – What metrics should be used to measure dialogs that learn? • Scribe: 2. Semi-automatic design of dialogs • Dialog Studio: An Example-Based Spoken Dialog System Development Workbench by Sangkeun Jung, Cheongjae Lee, Gary Geunbae Lee • Reinforcement Learning for Spoken Dialogue Systems: Comparing Strengths and Weaknesses for Practical Deployment by Tim Paek • Discussion – What techniques can apply to semi-automatic design of dialogs? – How to evaluate these techniques? • Scribe: 3. Methodologies for improving dialog design • A WOz Variant with Contrastive Conditions by Ester Levin and Rebecca Passonneau • Human-centered Development of Interactive Systems: Improving Usability in Early Lifecycles Stages by Zoraida Callejas, Ramón López- Cózar • Discussion – How to evaluate alternative methodologies for improving dialog design? – What metrics to use? • Scribe: 4. Modelling dialogs • Activity-based dialogue analysis as evaluation method by Bilyana Martinovska, Ashish Vaswani • Unifying language modeling capabilities for flexible interaction by Deryle Lonsdale and Rebecca Madsen • Two faces of spoken dialogue systems by Jens Edlund, Mattias Heldner & Joakim Gustafson • Discussion – How to evaluate alternative modelling techniques – What metrics to use? • Scribe: 5. Multimodal dialogs and visual input • DS-UCAT: A new Multimodal Dialogue System for an Academic Application by Ramón López-Cózar, Zoraida Callejas, Germán Montoro • Computer vision, eye tracking, spoken dialog systems, and evaluation: Challenges and opportunities by Gregory Aist • Discussion – What are the advantages and disadvantages of the X+V approach to implementing multimodal dialogs? – What are alternative architectures to X+V? – How does visual input affect dialogs? – Can X+V be extended to support visual input? • Scribe: Workshop Outcomes How to disseminate the results of this workshop to the speech dialogue community? Possibilities include: – Paper targeted to speech dialogue practitioners for publication in Speech Technology Magazine or similar magazine – Paper targeted to the academics community for publication in a referred journal (special issue) – Proceedings of the workshop (Ramon) – Edit minutes and post on workshop web site – Wikipage/blog (Jens) – ISCA webpage – Pls link the website!! – Next Interspeech in Antwerp – Next SpeechTech in S.F – MapTask-based evaluation challenge • Who will author/edit result? • Process for authoring/editing the result • Other next steps?
Pages to are hidden for
"Dialogue on Dialogues Multidisciplinary Evaluation of Advanced "Please download to view full document