Lecture_23_-_ECG

Document Sample
Lecture_23_-_ECG Powered By Docstoc
					Embodiment and Computation:
  Convergent Constraints
     on Language Use



    Nancy Chang
    nchang@icsi.berkeley.edu
    UC Berkeley / International Computer Science
    Institute
            What does language do?
 A sentence can evoke an imagined scene and resulting inferences:

“Harry walked to the               “Harry walked into the cafe.”
  cafe.”
                     CAFE                           CAFE




 – Goal of action = at cafe           – Goal of action = inside cafe
 – Source = away from cafe            – Source = outside cafe
 – cafe = point-like location         – cafe = containing location
           Embodied inferences

                                    WALL


The scientist walked into the              Bonk!!
 wall.

The hobo drifted into the
 house.

The smoke drifted into the house.
      Metaphorical inference

 France fell into recession.
  Germany pulled it out.

 The economy is moving along at the
  pace of a Clinton jog.

 The Indian Government is stumbling in
  implementing its liberalization plan.
      Embodied knowledge needed
 What things can serve as containers?
   – rooms but not walls (usually)
 How do different entities interact?
   – how people and gases interact with houses.
 How are different actions/states
  related?
   – stumbling / walking, falling / containment
 How can actions vary?
   – rate, direction, degree of force, etc.
… that is, more than predicate-argument structure!
      WALK(x), FALL(y), HIT(x,y), etc.
       Embodiment in language

 Perceptual and motor systems play a
  central role in language production
  and comprehension

 Theoretical proposals
  – Linguistics: Lakoff, Langacker, Talmy
  – Neuroscience: Damasio, Edelman
  – Cognitive psychology: Barsalou, Gibbs,
     Glenberg, MacWhinney
  – Computer science: Steels, Feldman
              Theory of
              Language
              Structure



Theory of                 Theory of
Language                  Language
Acquisition                 Use

Goal: computationally precise
    theories of language
              Theory of
              Language
              Structure



Theory of                 Theory of
Language                  Language
Acquisition                 Use
           Theory
              of
          Languag
              e
          Structure




Theory                 Theory
    of                   of
Langua                Languag
   ge                  e Use
Acquisi
  tion
        Simulation hypothesis

We understand utterances by mentally
simulating their content.

– Simulation exploits some of the
  same neural structures activated during
  performance, perception, imagining, memory…

– Linguistic structure parametrizes the
  simulation.
    Language gives us enough information to simulate
        Language understanding
         as simulative inference
                    “Harry walked to the cafe.”      Utterance
Linguistic
knowledge
                                           Analysis Process

General
Knowledge
                    Schema    Trajector     Goal     Simulation
                    walk      Harry         cafe
     Belief State                                    Specification


                                    Cafe
                                                   Simulation
1. Embodiment and Simulation


 “What is an idea?
     It is an image that paints itself in my brain.”
                                   — Voltaire
        Computational efficacy

   Embodied representations the
    norm in robotics!

   Computational representations for
    lexical semantics: have been
    developed for:
    –   Spatial relations (Regier 1996)
    –   Actions (Bailey 1997, Narayanan 1997)
    –   Objects / attributes (Roy 1998)

   Metaphor understanding system
    based on simulation (Narayanan 1997)
         Metaphor system architecture

Target
domain


Metaphor
 maps




 Source
 domain



                               (Narayanan 1997)
Metaphor understanding system
  Indian Government stumbling in
    implementing liberalization plan
           Missing link: grammar!

   Metaphor understanding system
    demonstrates that embodied inferences
    for difficult case are feasible.
    –   BUT: system has no grammar!
    –   How do we bridge the gap?

   Need a grammatical theory/formalism that
    can served as an interface between
    linguistic units and embodied, dynamic,
    encyclopedic, context-based information
    (i.e., that can support simulation).
         2. Embodied
      Construction Grammar

”It is not enough to say that the mind is
embodied; one must say how.”
                                    — Damasio
        What passes as grammar?
 “Syntactic investigation of a given language
  has as its goal the construction of a grammar
  that can be viewed as a device of some sort for
  producing the sentences of the language under
  analysis.”                   (Chomsky 1957)

 Inadequate notion of grammar
  – Meaning-free: syntax separate from meaning,
    function and processing; unanalyzable symbolic
    units
  – Inflexible: strict word order, strictly hierarchical,
    strictly compositional
          Who’s up to the task?
 Most theories of language are not
  explicitly and systematically tied to
  action and perception

 Promising exceptions
  – Cognitive Grammar / cognitive linguistics
  – Construction Grammar
  – Typically criticized for being informal / vague

 We borrow liberally from both and
  formalize.
        Cognitive Linguistics

“Language is an integral part of
cognition which reflects the interaction
of cultural, psychological,
communicative, and functional
considerations, and which can only be
understood in the context of a realistic
view of conceptualization and mental
processing.”
        International Cognitive Linguistics Association website
                (http://www.cognitivelinguistics.org/aims.shtml)
              Key borrowed ideas
 Conceptual structures are embodied.
  – Meaning is conceptualization
    (part of larger cognitive system).
  – Concepts are grounded in human experience as
    physical, psychological and social beings in the
    world.                   (Lakoff 1987, 1985; Langacker 1991, 1987)


 Basic symbolic unit at all levels
  is a form-meaning pair, or construction.
  – Syntax is not independent of semantics.
  – Phrasal/clausal constructions can contribute
    meaning independently of constituents.
           (Fillmore 1988, Kay & Fillmore 1999, Lakoff 1987, Goldberg 1995)
Traditional levels of analysis

           Pragmatics

                        U
           Semantics
                        T
                        T
             Syntax
                        E
                        R
           Morphology   A
                        N
           Phonology    C
                        E
            Phonetics
Form-meaning mappings for language
Linguistic knowledge consists of form-meaning mappings:
      Form                   Meaning
         phonological            event structure
           cues                  sensorimotor control
         word order              attention/perspective
         intonation              social goals...
         inflection
                                               Cafe
            Construction Grammar
A construction is a form-meaning pair whose properties may not be
strictly predictable from other constructions.
                                    (Construction Grammar, Goldberg 1995)

 Form                                            Meaning

  block


  walk



                                                    Trajector
   to                                   Source                      Goal
                                                      Path
         Constructions as maps between
                    relations
 Complex constructions are mappings between relations in form
 and relations in meaning.
    Form                                 Meaning

Mover + Motion                          MotionEvent
                                         mover(Motion, Mover)
 before(Mover, Motion)
“is” + Action + “ing”
                                        ProgressiveAction
 before(“is”, Action)
                                         aspect(Action, ongoing)
 suffix(Action, “ing”)

Mover + Motion + Direction              DirectedMotionEvent
  before(Motion, Direction)               direction(Motion, Direction)
  before(Mover, Motion)                   mover(Motion, Mover)
   More on Construction Grammar
                                                  (Goldberg 1995)

 Clause-level patterns correspond to
  basic events
  transitive: Agent Action Patient
  ditransitive (dative): Giver Action Recipient Gift
 Economical: no explosion of senses
        He pushed the ball.
        He pushed her the ball.

 Novel uses handled more robustly
         Mary pushed the tissue off the table.
        ?Mary sneezed the tissue off the table.
        *Mary slept the tissue off the table.
  Embodied Construction Grammar
                   (Bergen and Chang 2002)

 Embodied representations
  – active perceptual and motor schemas
     (image schemas, x-schemas, frames, etc.)
  – situational and discourse context

 Construction Grammar
  – Linguistic units relate form and meaning.
  – Both constituency and (lexical) dependencies
    allowed.

 Constraint-based
  – based on feature structure unification (as in
    HPSG)
  – Diverse factors can flexibly interact.
              ECG Structures

 Schemas
  – image schemas, force-dynamic schemas,
    executing schemas, frames…
 Constructions
  – lexical, grammatical, morphological,
    gestural…
 Maps
  – metaphor, metonymy, mental space maps…
 Spaces
  – discourse, hypothetical, counterfactual…
                Image schemas

 Trajector / Landmark (asymmetric)
  – The bike is near the house
                                               TR      LM
  – ? The house is near the bike
 Boundary / Bounded Region                           boundary

  – a bounded region has a closed boundarybounded region
 Topological Relations
  – Separation, Contact, Overlap, Inclusion, Surround
 Orientation
  – Vertical (up/down), Horizontal (left/right, front/back)
  – Absolute (E, S, W, N)
                Embodied schemas
                                schema name

   schema Source-Path-Goal                    schema Container
       roles                                      roles
            source              role name              interior
            path                                       exterior
            goal                                       portal
            trajector                                  boundary


                                        Boundary
                                                    Interior
           Trajector
Source                   Goal                                     Portal
             Path
                                                               Exterior


   These are abstractions over sensorimotor experiences.
                Embodied constructions
                                                       ECG Notation
    Form                        Meaning
                                                      construction HARRY
   Harry                                               form : /hEriy/
                                                       meaning : Harry




                                   CAFE                construction CAFE
   cafe                                                 form : /khaefej/
                                                        meaning : Cafe



Constructions have form and meaning poles that are subject to type constraints.
     Representing constructions: TO
           construction TO
            form
              selff.phon  /thuw/
            meaning
              evokes
                                                      local alias
                    Trajector-Landmark as tl
                    Source-Path-Goal as spg
              constraints:                           identification constraint
                    tl.trajector  spg.trajector
                    tl.landmark  spg.goal
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
           The INTO construction
                     construction INTO
                      form
                        selff.phon  /Inthuw/
                      meaning
                        evokes
TO vs. INTO:                  Trajector-Landmark as tl
  INTO adds a                 Source-Path-Goal as spg
  Container schema            Container as cont
  and appropriate       constraints:
  bindings.                   tl.trajector  spg.trajector
                              tl.landmark  cont
                              cont.interior  spg.goal
                              cont.exterior  spg.source
           Constructions with constituents:
           The SPATIAL-PHRASE construction
         construction SPATIAL-PHRASE
          constructional
            constituents
local             sp : Trajector-Landmark
alias             lm : Thing
          form
            spf before lmf                    order constraint
          meaning
            spm.landmark  lmm                 identification constraint




        Constructions may also specify constructional constituents and
        impose form and meaning constraints on them:
           –order constraints
           –identification constraints
An argument structure construction
construction DIRECTED-MOTION
 subcase of Pred-Expr
 constructional
   constituents
         a : Ref-Exp              schema Directed-Motion
         m: Pred-Exp                  roles
         p : Spatial-Phrase                agent : Entity
 form                                      motion : Motion
   af before mf                            path : SPG
   mf before pf
 meaning
   evokes Directed-Motion as dm
   selfm.scene dm
   dm.agent am
   dm.motion  mm
   dm.path  pm
The CAUSED-MOTION construction

 construction CAUSED-MOTION
   subcase of Pred-Expr
   constructional
     constituents
             agent : Entity
             action: Action
             patient: Entity
             path : SPG
   form
      agentf before actionf
      actionf before patientf
      actionf before pathf
   meaning
     evokes Caused-Motion as cm
     selfm.scene cm
     cm.agent agentm
     cm.action  actionm
     cm.patient patientm
     cm.path  pathm
   Language Understanding Process
 An utterance is perceived
 This activates the form pole of some constructions
 The Analysis process assembles the constructions, using
  construal where necessary, and binds together their forms
  and their meanings
 The product is a constructional analysis
 This yields a semspec -- parameterized schemas linked
  together in specified ways
 The semspec is input into the Simulation process, where the
  understander imagines the content
 Resulting inferences are propagated through the conceptual
  system.
                   Simulation-based language understanding
construction WALKED
   form                                         “Harry walked into the cafe.”    Utterance
       selff.phon  [wakt]
   meaning : Walk-Action
    constraints
       selfm.time before Context.speech-time
       selfm..aspect  encapsulated



                                         Constructions              Analysis Process


      General                                                                   Semantic
      Knowledge                                                                 Specification

                                Belief State


                                                             CAFE
                                                                          Simulation
         Simulation specification




A simulation specification consists of:
- schemas evoked by constructions
- bindings between schemas
Language Understanding Process
Constructional analysis
Semantic Specification
                 Basic Feature Structure
         A new rule for “I”                The corresponding fstruct

   Pronoun  I
      number SG                                number : SG 
      person  1st                              person : 1st 
                                                             
-The top part of the rule is the old
CFG rule.                               -This data structure is attached to the
                                        nonterminal during parsing so that the
                                        parser can use the information.
-The next two lines set the agreement
features.

                                        -The feature is on the lhs of the colon
-The  denotes assignment to the
                                        And the value is rhs of the colon.
feature listed on the lhs.
       Feature Structure Unification
 To check the compatibility of two fstructs
  – Two feature structures are compatible if they have the
    same value for every feature they have in common (or if
    one or both leave the value unspecified).
  – This process of checking compatibility is called
    unification.

 Unification
  – Is a recursive process that takes two feature structures and
    either returns the combined feature structure if they are
    compatible or it returns failure.
  – Base case: Two values unify if they are the same string.
  – Recursive Case: Two feature structures unify if for each feature
    they have in common, those values unify.
  – The resulting feature structure just adds the features they don’t
    have in common to the resulting structure.
Language Analysis and
Embodied Construction
      Grammar

          John Bryant
   jbryant@icsi.berkeley.edu
   Getting From the Utterance to the
               SemSpec

 Need a grammar formalism
  – Embodied Construction Grammar (Bergen & Chang
    2002)

 Need new models for language analysis
  – Traditional methods too limited
  – Traditional methods also don’t get enough leverage out of
    the semantics.
   Embodied Construction Grammar



 Semantic Freedom
  – Designed to be symbiotic with cognitive approaches
    to meaning
  – More expressive semantic operators than traditional
    grammar formalisms


 Form Freedom
  – Free word order, over-lapping constituency


 Precise enough to be implemented
Traditional Parsing Methods Fall Short


 PSG parsers too strict
  – Constructions not allowed to leave constituent
    order unspecified

 Traditional way of dealing with
  incomplete analyses is ad-hoc
  – Making sense of incomplete analyses is important when
    an application must deal with “ill-formed” input

 Traditional unification grammar can’t
  handle ECG’s deep semantic operators.
                Recognizer Example




Mary kicked the ball into the net.


This is the initial Constituent
Graph for caused-motion.                   Patient



           Agent                  Action


                                            Path
               Recognizer Example



                         Construct:
                       Caused-Motion



Constituent:    Constituent:     Constituent:      Constituent:
  Agent           Action           Patient            Path




     The initial constructional tree for the instance of
       Caused-Motion that we are trying to create.
                  Recognizer Example



caused  motion.m 
agent : 5          agent.m               patient.m            
                    ,  category :
                         1                  , 3category :       
 scene : 4                                                     
                     resolved  ref : resolved  ref :
                                                                  
 action : 6       
    caused  motion.cm        path.m             
    agent :  5             source :          
             1                                        action.m 
4 patient : {3}{cm1} ,{7} path :               , 2tense :      
                                                                   
    action : 26            goal :                  x  schema :
                                                                       
     path : {7}
                          
                               trajector : {cm1}
                                                   
                 Recognizer Example

processed



 Mary kicked the ball into the net.


  A node filled with gray is removed.
                                        Patient



            Agent              Action


                                         Path
             Recognizer Example


                      Construct:
                    Caused-Motion



   RefExp:    Constituent:   Constituent:   Constituent:
    Mary        Action         Patient         Path




Mary kicked the ball into the net.
                   Recognizer Example

caused  motion.m 
agent : 5          agent.m                     patient.m            
                    ,  category : Person
                         1                        , 3category :       
 scene : 4                                                           
                     resolved  ref : Mary  resolved  ref
                                                                       :
                                                                           
 action : 6       
    caused  motion.cm        path.m           
    agent :  5             source :        
             1                                      action.m 
4 patient : {3}{cm1} ,{7} path :             , 2tense :      
                                                                 
    action : 26            goal :                x  schema :
                                                                     
     path : {7}
                          
                               trajector : {cm1}
                                                 
               Recognizer Example

processed



 Mary kicked the ball into the net.



                               Patient



            Agent     Action


                                Path
             Recognizer Example


                    Construct:
                  Caused-Motion



   RefExp:      Verb:     Constituent:   Constituent:
    Mary       kicked       Patient         Path




Mary kicked the ball into the net.
                     Recognizer Example

caused  motion.m 
agent : 5          agent.m                     patient.m           
                    ,  category : Person
                         1                        , 3category :      
 scene : 4                                                          
                     resolved  ref : Mary  resolved  ref :
                                                                       
 action : 6       
    caused  motion.cm        path.m           
    agent :  5             source :        
             1                                      action.m           
4 patient : {3}{cm1} ,{7} path :             , 2tense : simpPast 
                                                                       
    action : 26            goal :                x  schema : kick 
                                                                           
     path : {7}
                          
                               trajector : {cm1}
                                                 
                 Recognizer Example

processed



 Mary kicked the ball into the net.

 According to the Constituent Graph,
 The next constituent can either be the
 Patient or the Path.                     Patient



            Agent              Action


                                           Path
               Recognizer Example

processed



 Mary kicked the ball into the net.



                               Patient



            Agent     Action


                                Path
             Recognizer Example


                    Construct:
                  Caused-Motion



   RefExp:      Verb:      RefExp:    Constituent:
    Mary       kicked      Det Noun      Path




                Det     Noun



Mary kicked the ball into the net.
                    Recognizer Example

caused  motion.m 
agent : 5          agent.m                     patient.m           
                    ,  category : Person
                         1                        , 3category : ball 
 scene : 4                                                          
                     resolved  ref : Mary  resolved  ref :
                                                                       
 action : 6       
    caused  motion.cm        path.m           
    agent :  5             source :        
             1                                      action.m           
4 patient : {3}{cm1} ,{7} path :             , 2tense : simpPast 
                                                                       
    action : 26            goal :                x  schema : kick 
                                                                           
     path : {7}
                          
                               trajector : {cm1}
                                                 
               Recognizer Example

processed



 Mary kicked the ball into the net.



                               Patient



            Agent     Action


                                Path
             Recognizer Example


                    Construct:
                  Caused-Motion



   RefExp:      Verb:      RefExp:       Spatial-Pred:
    Mary       kicked      Det Noun      Prep RefExp


                                              RefExp


                Det     Noun      Prep     Det     Noun



Mary kicked the ball into the net.
                   Recognizer Example

caused  motion.m 
agent : 5          agent.m                     patient.m           
                    ,  category : Person
                         1                        , 3category : ball 
 scene : 4                                                          
                     resolved  ref : Mary  resolved  ref :
                                                                       
 action : 6       
    caused  motion.cm        path.m           
    agent :  5             source :        
             1                                      action.m           
4 patient : {3}{cm1} ,{7} path :             , 2tense : simpPast 
                                                                       
    action : 26            goal : net            x  schema : kick 
                                                                           
     path : {7}
                          
                               trajector : {cm1}
                                                 
             Resulting SemSpec


After analyzing the sentence, the following identities
are asserted in the resulting SemSpec:


             Scene = Caused-Motion
             Agent = Mary
             Action = Kick
             Patient = Path.Trajector = The Ball
             Path = Into the net
             Path.Goal = The net
                   Summary


 By expanding traditional notions of parsing
  and unification grammar, it is possible to
  make a robust ECG-based language
  analyzer.

 Further work is necessary to better ground
  partial analysis/semantic density.
  – But they seem promising.
 Embodied Construction Grammar provides
formal tools for linguistic description and analysis
motivated largely by cognitive/functional concerns.

 A shared theory and formalism for
  different cognitive mechanisms
   –Constructions, metaphor, mental spaces, etc.

 Precise specifications of
  structures/processes involved in language
  understanding

 Bridge to detailed simulative inference
  using embodied representations
                Summary: ECG
 Linguistic constructions are tied to a model
  of simulated action and perception
 Embedded in a theory of language
  processing
  – Constrains theory to be usable
  – Frees structures to be just structures, used in
    processing
 Precise, computationally usable formalism
  – Practical computational applications, like MT and
    NLU
  – Testing of functionality, e.g. language learning
 A shared theory and formalism for different
  cognitive mechanisms
  – Constructions, metaphor, mental spaces, etc.
                     ECG applications
 Grammar
   – Spatial relations/events                   (Bergen & Chang 1999;
                                                Bretones et al. In press)
   – Verbal morphology            (Gurevich 2003, Bergen ms.)
   – Reference: measure phrases              (Dodge and Wright 2002),
               construal resolution          (Porzel & Bryant 2003),
               reflexive pronouns            (Sanders 2003)
 Semantic representations / inference
   – Aspectual inference                        (Narayanan 1997; Chang, Gildea &
                                                          Narayanan 1998)
   – Perspective / frames                       (Chang, Narayanan & Petruck 2002)
   – Metaphorical inference                     (Narayanan 1997, 1999)
   – Simulation semantics                       (Narayanan 1997, 1999)

 Language acquisition
   – Lexical acquisition             (Regier 1996, Bailey 1997)
   – Multi-word constructions                   (Chang 2004; Chang & Maia 2001)
3. Simulation-based inference
          Interpretation: simulation

                                           walker at goal
Constructions
 can:
 specify which
  schemas and                energy
  entities are
                       walker=Harry                         goal=home
  involved in an
  event, and how
  they are related
 profile particular            Harry is walking home.
  stages of an event
 set parameters of
  an event
            Simulation Semantics
 execution-based model of events/processes
  – tractable, distributed, concurrent, context-sensitive

 X-schemas provide natural model of
  – resource consumption/production
  – goals, preconditions, effects
  – hierarchical events (multiple granularities)
                  Simulation Semantics (2)
 Captures fine-grained distinctions needed for
  interpretation
   –   aspectual inferences [Narayanan 1997, 1999; Chang et al. 1998]
   –   metaphoric inferences [Narayanan 1997, 1999]
   –   perspectival inferences [Chang et al. 2002]
   –   inductive bias for language learning [Bailey 1997, Chang 2000]

 Captures essential features of neural
  computation [Feldman & Ballard 1982, Feldman 1989,
  Valiant 1994]

   – active, context-sensitive knowledge representation

   – same representational substrate for action, perception
        [Boccino et al. 2001, NBL01, CNS02]
               Simulation Semantics
 Inspired by biological control theory, Simulation
  Semantics models events as executing-, or x-
  schemas.
 An x-schema is a Petri net: a weighted graph
  consisting of places (circles) and transitions
  (rectangles) connected by directed input and output
  arcs.
 A state is defined by the placement of a token (a
  black dot or number) in a particular place.
 The real-time execution        semantics of Petri
  nets models the production and consumption of
  resources:
   – A transition is enabled when its input places are marked such that it
     can fire by movement of tokens from input to output.
   – Arcs include resource, enable and inhibitory arcs.
   – Actions have hierarchical structure, permitting embeddings.
Language is embodied:
it is learned and used by people
with bodies who inhabit a
physical, psychological and
social world.


                      Th eor y
                         of
                      La ngu ag
                         e
                      St r uc t ur
                         e




           Th eor y                  Th eor y
              of                        of
           La ngu a                  La ngu ag
             ge                       e Us e
                 s
           Ac qui i
              o
             ti n

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:11
posted:2/25/2011
language:English
pages:75