Dialogue and grammar

							                                 Goteborg University
                                 Dialogue Systems Lab




Dialogue and grammar


Karin Cavallin, Robin Cooper
David Hjelm, Staffan Larsson

      Göteborg University
     Dialogue Systems Lab

 Language Technology in Sweden
          2004-05-12
                                   Goteborg University
                                   Dialogue Systems Lab


     Speech recognition and NL
     interpretation & generation
• Options for SR:
  – Statistical language model
  – Grammar model
• Options for NL interpretation:
  – Word/phrase spotting
  – Grammar & parsing
• Options for NL generation
  – Template-based
  – Grammar-based
• Can be combined in various ways
                                         Goteborg University
                                         Dialogue Systems Lab


 Are grammars useful for dialogue
           systems?
• Most working systems don’t use a
  grammar!
• Instead:
  – keyword or phrase-spotting techniques for
    interpretation
  – template-based generation
• Also, most working systems are built
  more or less from scratch
  – do not fully separate application-specific
    knowledge from general and reusable
    knowledge
                                                   Goteborg University
                                                   Dialogue Systems Lab




                          GoDiS

• A dialogue system built using TrindiKit and based on the
  information state approach
• Implements a theory of Issue-based Dialogue
  Management
• Some goals with GoDiS:
   – Provide a domain-independent theory covering several
     dialogue genres
   – Modularity and reusability of system components
   – Enable rapid prototyping of new applications
• Focus on dialogue management
• These goals have been achieved on the level of dialogue
  management...
   – ... but not for NL processing
                                   Goteborg University
                                   Dialogue Systems Lab




   SR and interpretation in GoDiS

• Started out as text-based system
  – Phrase spotting for interpretation;
    strings to dialogue moves
  – Templates for generation; dialogue
    moves to strings
• Added SR component
  – Did not have a corpus, so needed to
    write a speech recognition grammar
    by hand
                                               Goteborg University
                                               Dialogue Systems Lab




                     tvGoDiS

• GoDiS application for programming a
  VCR over the telephone
   – schedule recordings
   – remove planned recordings
   – TV card used for recording television
     programs as video files
• Small corpus collected of users calling the
  system
   – approx. 100 sessions (calls) and 1100 utterances
   – misrecognised utterances not yet transcribed
   – GUI enables immediate access to corpus
                            Goteborg University
                            Dialogue Systems Lab




    Example from tvGoDIS corpus


• TrindiGui displaying
  a tvGoDiS session
• Audio files linked
  with utterances
• Access to
  information state at
  different stages of
  the dialogue
                                Goteborg University
                                Dialogue Systems Lab




       Problems with GoDiS

• Mismatch between SR module and
  interpretation module
 – correctly recognised speech was
   misinterpreted
 – Leads to strange and unexpected
   behaviour (unexpected interactions
   with dialogue management)
                                                            Goteborg University
                                                            Dialogue Systems Lab




       ASR/interpretation mismatch
S> Välkommen till videon! För
  att få hjälp, säg hjälp.         •   jag vill spela in ett program
S> Låt oss se. Vad kan jag             –   Covered by SR grammar
  hjälpa dig med?                      –   Not covered by phrase spotting
                                           lexicon
U> jag vill spela in ett program       –   However spela in and ett are
  (0.6)                                    recognized by interpretation
                                           module
S> Vill du lägga till en
  inspelning?                      •   Correct interpretation:
                                       –   jag vill spela in ett program 
U> ja (0.98)                               request(add_rec_job)
S> Låt oss se. Menar du från       •   Actual interpretation:
  noll ett eller till noll ett?        –   spela in 
                                           request(add_rec_job
                                       –   ett 
                                           answer(time(1:00))
                                   •   System assumes ett specifies
                                       either when the recording should
                                       stop or end – thus the follow-up
                                       question
                                 Goteborg University
                                 Dialogue Systems Lab




        Problems with GoDiS

• Mismatch between interpretation
  module and generation module
  – System said something that led the
    user to believe they could say X
  – but the system does not understand
    X
                                                     Goteborg University
                                                     Dialogue Systems Lab




Interpretation/generation mismatch

S> Du kan lägga till en           • Help message suggests that
  inspelning, ta bort en            the user can say lista
  planerad inspelning, och          inspelningar
  lista planerade inspelningar.   • But help message is not up-
  För att börja om, säg börja       to-date, and neither is SR
  om. För att sluta, säg sluta.     grammar.
S> huvudmenyn. Vad kan jag        • Instead interpretation
  hjälpa dig med?                   module finds word
U> lista inspelningar (0.75)        inspelningar
S> Ockey. gå till inspelningar.   • The user is asked to choose
S> Låt oss se. Vill du lägga        between roughly the same
  till en inspelning , radera       alternatives that was
  dina inspelningar eller få        presented in the help
  information om dina               message
  inspelningar?
                                       Goteborg University
                                       Dialogue Systems Lab




• Bugs can of course be fixed...
  – would take lots of time
  – no guarantee that new bugs won’t be
    introduced
  – no guarantee that all bugs have shown up
• These solutions are not reusable
• Instead, we want a general and
  principled solution
  – enable reuse and rapid prototyping of
    applications, including grammars
                                                     Goteborg University
                                                     Dialogue Systems Lab


Solution, part 1: single grammar for
   SR, interpretation, generation
• No mismatches between the grammars
• Only one grammar needs to be written for each
  application
• However:
   – the different modules may require different kinds of
     grammars, in different formats
• So:
   – We need to be able to generate all 3 grammars from a
     single grammar
• However, we may not want the exact same coverage for
  SR, interpretation and generation grammars
   – May not want to understand everything that can be
     generated
                                              Goteborg University
                                              Dialogue Systems Lab




        Solution, part 2: use GF!

• GF is a powerful tool for mutilingual grammar
  development
• Can be used to generate grammars in various
  other formats
• Subgrammars can be extracted from larger
  grammars
  – SR, interpretation and generation subgrammars may
    have different coverage
• In addition, GF offers resource grammars in
  several languages
  – This promises to further decrease the time needed
    for writing a new application grammar
                                   Goteborg University
                                   Dialogue Systems Lab




         GF GoDiS Grammar

• Application grammars
  – User grammar
  – System grammar
  – Shared grammar, used by user grammar
    and system grammar
• Utility grammars
  – Weekday, Time, Channel
• Application grammars may use
  – Utility grammars
  – Resource grammars
                                       Goteborg University
                                       Dialogue Systems Lab




             GF VCR grammar

• One abstract syntax
  – GoDiS dialogue moves as GF abstract trees
• Three concrete syntaxes
  – Swedish
  – English
  – GoDiS dialogue moves in prolog notation
• Statistics:
  – Abstract syntax : ~420 lines of code
  – Concrete syntax :
     • Prolog: ~440 lines of code
     • Swedish: ~440 lines of code
                                               Goteborg University
                                               Dialogue Systems Lab




 Parsing Swedish to abstract syntax

> p -cat=DMove -lang=Swe "spela in från kvart över tolv"

request (vcr_add_rec_job_args1 (startTimeToStoreReq
   (startTimeToStore (time hour12 minute15))))
                                              Goteborg University
                                              Dialogue Systems Lab


Linearising abstract syntax to Prolog
              notation
> l -lang=Prolog request (vcr_add_rec_job_args1
   (startTimeToStoreReq (startTimeToStore (time hour12
   minute15))))

request(add_rec_job , answer(start_time_to_store(12 :
   15))
                                               Goteborg University
                                               Dialogue Systems Lab




        Translating Prolog to Swedish

> t -cat=DMove Prolog Swe "request(add_rec_job ,
    answer(start_time_to_store(12 : 15))"

spela   in   från   tolv femton
spela   in   från   tolv och femton
spela   in   från   femton över tolv
spela   in   från   femton minuter över tolv
spela   in   från   kvart över tolv
                                   Goteborg University
                                   Dialogue Systems Lab




            Time Grammar
• A multilingual grammar module
• Swedish, English, Prolog, started on
  French and Catalan.
                                              Goteborg University
                                              Dialogue Systems Lab


    Constructing concrete syntaxes for
    time expressions isn’t always trivial
•   Abstract: time hour12 Minute30
•   Prolog: 12 : 30
•   English: half past twelve / half twelve
•   Swedish: halv ett

•   klockan två
•   klockan två och femton
•   ?klockan fjorton
•   two o’clock
•   *two fifteen o’clock
•   *fourteen o’clock

Corpus studies needed
                           Goteborg University
                           Dialogue Systems Lab




         Another problem

• am or pm?
• på morgonen
• på eftermiddagen
• på kvällen
• på natten
                                  Goteborg University
                                  Dialogue Systems Lab


    Combining GF and statistical
      methods for robustness
• Use GF to generate a corpus to use as a
  baseline for SLM training
• Use GF-generated SR grammars in
  initial system and collect corpus to be
  used for training SLMs for speech
  recognition
• For recognized out-of-grammar string,
  try to find nearest in-grammar string;
  confirm this with user (Gorrell 2003)
                                     Goteborg University
                                     Dialogue Systems Lab




               Conclusion

• Keeping track of separate SR,
  interpretation and generation grammars
  in a dialogue system is time-consuming
• GF offers the possibility of rapid
  prototyping of grammars for use in
  dialogue system applications
  – Generate other grammar formats from GF
  – Resource grammars for several languages
• GF can be combined with statistical
  methods for robustness