Dialogue on Dialogues Multidisciplinary Evaluation of Advanced by wanghonghx


									   Dialogue on Dialogues
Multidisciplinary Evaluation of
 Advanced Speech-based
     Interactive Systems
          Interspeech2006 - ICSLP
              Satellite Workshop

 Michael McTear Kristiina Jokinen Jim Larson
  Workshop Theme: Advanced
Speech-based Interactive Systems
• Advanced in terms of characteristics of spoken
  language interaction, such as:
  speech segmentation, disfluencies, turn-taking,
  emotions, and adaptation (see Shriberg,
  Interspeech 2005)
• How to incorporate these aspects in advanced
  speech-based interactive systems
• How to evaluate these systems
  e.g. are they more effective if they incorporate
  more advanced characteristics?
         Theory and Practice
• Relationship between theoretical work on
  dialogue modelling and the practices and
  requirements of practical interactive speech
• Is academic research on dialogue relevant to
  industrial applications?
• Are the methods developed in industry to ensure
  successful applications relevant to the academic
  Empirical Methods v Theoretical
• How or whether statistical speech-based
  research can be applied to dialogue modelling
• Dialogue modelling is concerned with advanced
  architectures and functionalities
• Are the methods used in statistical based
  approaches applicable to advanced dialogue
 Workshop Goals and Methods
• Encourage dialogue on dialogues
  – Five different discussion sessions
• Engage in multidisciplinary discussions
  – Computer Science, Computational Linguistics,
    Speech and Language Technology, HCI
• Work on evaluation criteria
  – The scribes reports on the results of the discussion
          1. Evaluation criteria
• Evaluating Real-time Responsiveness in Dialog by Nigel
  G. Ward
• Development and evaluation of the DIHANA project
  dialog systems by D. Griol, F. Torres, L. Hurtado, S.
  Grau, F. Garcia, E. Sanchis, and E. Segarra
• Challenges in evaluating spoken dialog systems that
  reason and learn by Gregory Aist, Phil Michalak, George
  Ferguson, James Allen
• Discussion:
   – What metrics should be used to measure static dialogs?
   – What metrics should be used to measure dialogs that learn?
• Scribe:
    2. Semi-automatic design of
• Dialog Studio: An Example-Based Spoken Dialog
  System Development Workbench by Sangkeun Jung,
  Cheongjae Lee, Gary Geunbae Lee
• Reinforcement Learning for Spoken Dialogue Systems:
  Comparing Strengths and Weaknesses for Practical
  Deployment by Tim Paek
• Discussion
   – What techniques can apply to semi-automatic design of dialogs?
   – How to evaluate these techniques?
• Scribe:
 3. Methodologies for improving
         dialog design
• A WOz Variant with Contrastive Conditions by
  Ester Levin and Rebecca Passonneau
• Human-centered Development of Interactive
  Systems: Improving Usability in Early Lifecycles
  Stages by Zoraida Callejas, Ramón López-
• Discussion
  – How to evaluate alternative methodologies for
    improving dialog design?
  – What metrics to use?
• Scribe:
            4. Modelling dialogs

• Activity-based dialogue analysis as evaluation
  method by Bilyana Martinovska, Ashish Vaswani
• Unifying language modeling capabilities for
  flexible interaction by Deryle Lonsdale and
  Rebecca Madsen
• Two faces of spoken dialogue systems by Jens
  Edlund, Mattias Heldner & Joakim Gustafson
• Discussion
  – How to evaluate alternative modelling techniques
  – What metrics to use?
• Scribe:
5. Multimodal dialogs and visual
• DS-UCAT: A new Multimodal Dialogue System for an
  Academic Application by Ramón López-Cózar, Zoraida
  Callejas, Germán Montoro
• Computer vision, eye tracking, spoken dialog systems,
  and evaluation: Challenges and opportunities by Gregory
• Discussion
   – What are the advantages and disadvantages of the X+V
     approach to implementing multimodal dialogs?
   – What are alternative architectures to X+V?
   – How does visual input affect dialogs?
   – Can X+V be extended to support visual input?
• Scribe:
               Workshop Outcomes
How to disseminate the results of this workshop to the speech dialogue community?
      Possibilities include:
    –     Paper targeted to speech dialogue practitioners for publication in Speech
          Technology Magazine or similar magazine
    –     Paper targeted to the academics community for publication in a referred
          journal (special issue)
    –     Proceedings of the workshop (Ramon)
    –     Edit minutes and post on workshop web site
    –     Wikipage/blog (Jens)
    –     ISCA webpage
    –     Pls link the website!!
    –     Next Interspeech in Antwerp
    –     Next SpeechTech in S.F
    –     MapTask-based evaluation challenge
•     Who will author/edit result?
•     Process for authoring/editing the result
•     Other next steps?

To top