Document Sample
Session8 Powered By Docstoc
					CSCTR Session 8
 group at UC Berkeley
 & Uni of Hawaii
 – Nancy Chang
 – Benjamin Bergen
 – Jerome Feldman, …
 General assumption
 – Semantic relations could be extracted from
   language input

 “In its communicative function, language
  is a set of tools with which we attempt to
  guide another mind to create within itself
  a mental representation that
  approximates one we have.” (Delancey
            What does language do?
 A sentence can evoke an imagined scene and resulting inferences:

“Harry walked to the               “Harry walked into the cafe.”
                     CAFE                           CAFE

 – Goal of action = at cafe           – Goal of action = inside cafe
 – Source = away from cafe            – Source = outside cafe
 – cafe = point-like location         – cafe = containing location
           Embodied inferences


The scientist walked into the              Bonk!!

The hobo drifted into the

The smoke drifted into the house.
       Embodiment in language

 Perceptual and motor systems play a
  central role in language production
  and comprehension

 Theoretical proposals
  – Linguistics: Lakoff, Langacker, Talmy
  – Neuroscience: Damasio, Edelman
  – Cognitive psychology: Barsalou, Gibbs,
     Glenberg, MacWhinney
  – Computer science: Steels, Feldman
              Theory of

Theory of                 Theory of
Language                  Language
Acquisition                 Use

Goal: computationally precise
    theories of language
        Simulation hypothesis

We understand utterances by mentally
simulating their content.

– Simulation exploits some of the
  same neural structures activated during
  performance, perception, imagining, memory…

– Linguistic structure parametrizes the
    Language gives us enough information to simulate
        Language understanding
         as simulative inference
                    “Harry walked to the cafe.”      Utterance
                                           Analysis Process

                    Schema    Trajector     Goal     Simulation
                    walk      Harry         cafe
     Belief State                                    Specification

1. Embodiment and Simulation

 “What is an idea?
     It is an image that paints itself in my brain.”
                                   — Voltaire
    Neural evidence: Mirror neurons

 Gallese et al. (1996) found “mirror”
  neurons in the monkey motor cortex,
  activated when
  – an action was carried out
  – the same action (or a similar one) was seen.
 Mirror neurons found in humans (Porro et al.
 Mirror neurons activated when someone:
  – imagines an action being carried out (Wheeler et al.
  – watches an action being carried out (with and
    without object) (Buccino et al. 2000)
The Motor System is somatotopically organized
               The Mirror System
The mirror system, like the motor
                             Buccino et al.,
system, is somatotopically organized.2001

                                    humans watching
                                    videos of actions
                                    without objects

                                    humans watching
                                    same actions with

Foot actions    Hand actions   Mouth actions
      Mirror neurons for language?
 Mirror neurons for specific effectors
  activated during passive listening:
   – Sentences describing mouth/leg/hand
     motions activates corresponding part of pre-
     motor cortex
                                    (Tettamanti et al., forthcoming)

  – Verbs associated with particular effectors
    activates corresponding areas of motor
    cortex            (Pulvermuller et al. 2001, Hauk et al.
Movement vs. Actions
           Psycholinguistic evidence
 Embodied language impairs action/perception
  – Sentences with visual components to their meaning
    can interfere with performance of visual tasks
                                         (Richardson et al. 2003)
  – Sentences describing motion can interfere with
    performance of incompatible motor actions
                                         (Glenberg and Kashak 2002)
  – Sentences describing incompatible visual imagery
    impedes decision task (Zwaan et al. 2002)
 Simulation effects from fictive motion
  – Fictive motion sentences describing paths that require
    longer time, span a greater distance, or involve
    more obstacles impede decision task (Matlock 2000, Matlock
    et al. 2003)
        Computational efficacy

   Embodied representations the
    norm in robotics!

   Computational representations for
    lexical semantics: have been
    developed for:
    –   Spatial relations (Regier 1996)
    –   Actions (Bailey 1997, Narayanan 1997)
    –   Objects / attributes (Roy 1998)

   Metaphor understanding system
    based on simulation (Narayanan 1997)
           Missing link: grammar!

   Metaphor understanding system
    demonstrates that embodied inferences
    for difficult case are feasible.
    –   BUT: system has no grammar!
    –   How do we bridge the gap?

   Need a grammatical theory/formalism that
    can served as an interface between
    linguistic units and embodied, dynamic,
    encyclopedic, context-based information
    (i.e., that can support simulation).
         2. Embodied
      Construction Grammar

”It is not enough to say that the mind is
embodied; one must say how.”
                                    — Damasio
              Key borrowed ideas
 Conceptual structures are embodied.
  – Meaning is conceptualization
    (part of larger cognitive system).
  – Concepts are grounded in human experience as
    physical, psychological and social beings in the
    world.                   (Lakoff 1987, 1985; Langacker 1991, 1987)

 Basic symbolic unit at all levels
  is a form-meaning pair, or construction.
  – Syntax is not independent of semantics.
  – Phrasal/clausal constructions can contribute
    meaning independently of constituents.
           (Fillmore 1988, Kay & Fillmore 1999, Lakoff 1987, Goldberg 1995)
Form-meaning mappings for language
Linguistic knowledge consists of form-meaning mappings:
      Form                   Meaning
         phonological            event structure
           cues                  sensorimotor control
         word order              attention/perspective
         intonation              social goals...
            Construction Grammar
A construction is a form-meaning pair whose properties may not be
strictly predictable from other constructions.
                                    (Construction Grammar, Goldberg 1995)

 Form                                            Meaning



   to                                   Source                      Goal
  Embodied Construction Grammar
                   (Bergen and Chang 2002)

 Embodied representations
  – active perceptual and motor schemas
     (image schemas, x-schemas, frames, etc.)
  – situational and discourse context

 Construction Grammar
  – Linguistic units relate form and meaning.
  – Both constituency and (lexical) dependencies

 Constraint-based
  – based on feature structure unification (as in
  – Diverse factors can flexibly interact.
              ECG Structures

 Schemas
  – image schemas, force-dynamic schemas,
    executing schemas, frames…
 Constructions
  – lexical, grammatical, morphological,
 Maps
  – metaphor, metonymy, mental space maps…
 Spaces
  – discourse, hypothetical, counterfactual…
                Image schemas

 Trajector / Landmark (asymmetric)
  – The bike is near the house
                                               TR      LM
  – ? The house is near the bike
 Boundary / Bounded Region                           boundary

  – a bounded region has a closed boundarybounded region
 Topological Relations
  – Separation, Contact, Overlap, Inclusion, Surround
 Orientation
  – Vertical (up/down), Horizontal (left/right, front/back)
  – Absolute (E, S, W, N)
                Embodied schemas
                                schema name

   schema Source-Path-Goal                    schema Container
       roles                                      roles
            source              role name              interior
            path                                       exterior
            goal                                       portal
            trajector                                  boundary

Source                   Goal                                     Portal

   These are abstractions over sensorimotor experiences.
                Embodied constructions
                                                       ECG Notation
    Form                        Meaning
                                                      construction HARRY
   Harry                                               form : /hEriy/
                                                       meaning : Harry

                                   CAFE                construction CAFE
   cafe                                                 form : /khaefej/
                                                        meaning : Cafe

Constructions have form and meaning poles that are subject to type constraints.
     Representing constructions: TO
           construction TO
              selff.phon  /thuw/
                                                      local alias
                    Trajector-Landmark as tl
                    Source-Path-Goal as spg
              constraints:                           identification constraint
                    tl.trajector  spg.trajector
                    tl.landmark  spg.goal
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
           The INTO construction
                     construction INTO
                        selff.phon  /Inthuw/
TO vs. INTO:                  Trajector-Landmark as tl
  INTO adds a                 Source-Path-Goal as spg
  Container schema            Container as cont
  and appropriate       constraints:
  bindings.                   tl.trajector  spg.trajector
                              tl.landmark  cont
                              cont.interior  spg.goal
                              cont.exterior  spg.source
           Constructions with constituents:
           The SPATIAL-PHRASE construction
         construction SPATIAL-PHRASE
local             sp : Trajector-Landmark
alias             lm : Thing
            spf before lmf                    order constraint
            spm.landmark  lmm                 identification constraint

        Constructions may also specify constructional constituents and
        impose form and meaning constraints on them:
           –order constraints
           –identification constraints
An argument structure construction
construction DIRECTED-MOTION
 subcase of Pred-Expr
         a : Ref-Exp              schema Directed-Motion
         m: Pred-Exp                  roles
         p : Spatial-Phrase                agent : Entity
 form                                      motion : Motion
   af before mf                            path : SPG
   mf before pf
   evokes Directed-Motion as dm
   selfm.scene dm
   dm.agent am
   dm.motion  mm
   dm.path  pm
The CAUSED-MOTION construction

 construction CAUSED-MOTION
   subcase of Pred-Expr
             agent : Entity
             action: Action
             patient: Entity
             path : SPG
      agentf before actionf
      actionf before patientf
      actionf before pathf
     evokes Caused-Motion as cm
     selfm.scene cm
     cm.agent agentm
     cm.action  actionm
     cm.patient patientm
     cm.path  pathm
                   Simulation-based language understanding
construction WALKED
   form                                         “Harry walked into the cafe.”    Utterance
       selff.phon  [wakt]
   meaning : Walk-Action
       selfm.time before Context.speech-time
       selfm..aspect  encapsulated

                                         Constructions              Analysis Process

      General                                                                   Semantic
      Knowledge                                                                 Specification

                                Belief State

         Simulation specification

A simulation specification consists of:
- schemas evoked by constructions
- bindings between schemas
Language Understanding Process
Constructional analysis
Semantic Specification
Language Analysis and
Embodied Construction

          John Bryant
                     ECG applications
 Grammar
   – Spatial relations/events                   (Bergen & Chang 1999;
                                                Bretones et al. In press)
   – Verbal morphology            (Gurevich 2003, Bergen ms.)
   – Reference: measure phrases              (Dodge and Wright 2002),
               construal resolution          (Porzel & Bryant 2003),
               reflexive pronouns            (Sanders 2003)
 Semantic representations / inference
   – Aspectual inference                        (Narayanan 1997; Chang, Gildea &
                                                          Narayanan 1998)
   – Perspective / frames                       (Chang, Narayanan & Petruck 2002)
   – Metaphorical inference                     (Narayanan 1997, 1999)
   – Simulation semantics                       (Narayanan 1997, 1999)

 Language acquisition
   – Lexical acquisition             (Regier 1996, Bailey 1997)
   – Multi-word constructions                   (Chang 2004; Chang & Maia 2001)
3. Simulation-based inference
          Interpretation: simulation

                                           walker at goal
 specify which
  schemas and                energy
  entities are
                       walker=Harry                         goal=home
  involved in an
  event, and how
  they are related
 profile particular            Harry is walking home.
  stages of an event
 set parameters of
  an event
            Simulation Semantics
 execution-based model of events/processes
  – tractable, distributed, concurrent, context-sensitive

 X-schemas provide natural model of
  – resource consumption/production
  – goals, preconditions, effects
  – hierarchical events (multiple granularities)
               Simulation Semantics
 Inspired by biological control theory, Simulation
  Semantics models events as executing-, or x-
 An x-schema is a Petri net: a weighted graph
  consisting of places (circles) and transitions
  (rectangles) connected by directed input and output
 A state is defined by the placement of a token (a
  black dot or number) in a particular place.
 The real-time execution        semantics of Petri
  nets models the production and consumption of
   – A transition is enabled when its input places are marked such that it
     can fire by movement of tokens from input to output.
   – Arcs include resource, enable and inhibitory arcs.
   – Actions have hierarchical structure, permitting embeddings.

Shared By: