Session8

Document Sample
Session8 Powered By Docstoc
					CSCTR Session 8
 group at UC Berkeley
 & Uni of Hawaii
 – Nancy Chang
 – Benjamin Bergen
 – Jerome Feldman, …
 General assumption
 – Semantic relations could be extracted from
   language input

 “In its communicative function, language
  is a set of tools with which we attempt to
  guide another mind to create within itself
  a mental representation that
  approximates one we have.” (Delancey
            What does language do?
 A sentence can evoke an imagined scene and resulting inferences:

“Harry walked to the               “Harry walked into the cafe.”
  cafe.”
                     CAFE                           CAFE




 – Goal of action = at cafe           – Goal of action = inside cafe
 – Source = away from cafe            – Source = outside cafe
 – cafe = point-like location         – cafe = containing location
           Embodied inferences

                                    WALL


The scientist walked into the              Bonk!!
 wall.

The hobo drifted into the
 house.

The smoke drifted into the house.
       Embodiment in language

 Perceptual and motor systems play a
  central role in language production
  and comprehension

 Theoretical proposals
  – Linguistics: Lakoff, Langacker, Talmy
  – Neuroscience: Damasio, Edelman
  – Cognitive psychology: Barsalou, Gibbs,
     Glenberg, MacWhinney
  – Computer science: Steels, Feldman
              Theory of
              Language
              Structure



Theory of                 Theory of
Language                  Language
Acquisition                 Use

Goal: computationally precise
    theories of language
        Simulation hypothesis

We understand utterances by mentally
simulating their content.

– Simulation exploits some of the
  same neural structures activated during
  performance, perception, imagining, memory…

– Linguistic structure parametrizes the
  simulation.
    Language gives us enough information to simulate
        Language understanding
         as simulative inference
                    “Harry walked to the cafe.”      Utterance
Linguistic
knowledge
                                           Analysis Process

General
Knowledge
                    Schema    Trajector     Goal     Simulation
                    walk      Harry         cafe
     Belief State                                    Specification


                                    Cafe
                                                   Simulation
1. Embodiment and Simulation


 “What is an idea?
     It is an image that paints itself in my brain.”
                                   — Voltaire
    Neural evidence: Mirror neurons

 Gallese et al. (1996) found “mirror”
  neurons in the monkey motor cortex,
  activated when
  – an action was carried out
  – the same action (or a similar one) was seen.
 Mirror neurons found in humans (Porro et al.
 1996)
 Mirror neurons activated when someone:
  – imagines an action being carried out (Wheeler et al.
    2000)
  – watches an action being carried out (with and
    without object) (Buccino et al. 2000)
The Motor System is somatotopically organized
               The Mirror System
The mirror system, like the motor
                             Buccino et al.,
system, is somatotopically organized.2001

                                    humans watching
                                    videos of actions
                                    without objects

                                    humans watching
                                    same actions with
                                    objects


Foot actions    Hand actions   Mouth actions
      Mirror neurons for language?
 Mirror neurons for specific effectors
  activated during passive listening:
   – Sentences describing mouth/leg/hand
     motions activates corresponding part of pre-
     motor cortex
                                    (Tettamanti et al., forthcoming)



  – Verbs associated with particular effectors
    activates corresponding areas of motor
    cortex            (Pulvermuller et al. 2001, Hauk et al.
    2004)
Movement vs. Actions
           Psycholinguistic evidence
 Embodied language impairs action/perception
  – Sentences with visual components to their meaning
    can interfere with performance of visual tasks
                                         (Richardson et al. 2003)
  – Sentences describing motion can interfere with
    performance of incompatible motor actions
                                         (Glenberg and Kashak 2002)
  – Sentences describing incompatible visual imagery
    impedes decision task (Zwaan et al. 2002)
 Simulation effects from fictive motion
  sentences
  – Fictive motion sentences describing paths that require
    longer time, span a greater distance, or involve
    more obstacles impede decision task (Matlock 2000, Matlock
    et al. 2003)
        Computational efficacy

   Embodied representations the
    norm in robotics!

   Computational representations for
    lexical semantics: have been
    developed for:
    –   Spatial relations (Regier 1996)
    –   Actions (Bailey 1997, Narayanan 1997)
    –   Objects / attributes (Roy 1998)

   Metaphor understanding system
    based on simulation (Narayanan 1997)
           Missing link: grammar!

   Metaphor understanding system
    demonstrates that embodied inferences
    for difficult case are feasible.
    –   BUT: system has no grammar!
    –   How do we bridge the gap?

   Need a grammatical theory/formalism that
    can served as an interface between
    linguistic units and embodied, dynamic,
    encyclopedic, context-based information
    (i.e., that can support simulation).
         2. Embodied
      Construction Grammar

”It is not enough to say that the mind is
embodied; one must say how.”
                                    — Damasio
              Key borrowed ideas
 Conceptual structures are embodied.
  – Meaning is conceptualization
    (part of larger cognitive system).
  – Concepts are grounded in human experience as
    physical, psychological and social beings in the
    world.                   (Lakoff 1987, 1985; Langacker 1991, 1987)


 Basic symbolic unit at all levels
  is a form-meaning pair, or construction.
  – Syntax is not independent of semantics.
  – Phrasal/clausal constructions can contribute
    meaning independently of constituents.
           (Fillmore 1988, Kay & Fillmore 1999, Lakoff 1987, Goldberg 1995)
Form-meaning mappings for language
Linguistic knowledge consists of form-meaning mappings:
      Form                   Meaning
         phonological            event structure
           cues                  sensorimotor control
         word order              attention/perspective
         intonation              social goals...
         inflection
                                               Cafe
            Construction Grammar
A construction is a form-meaning pair whose properties may not be
strictly predictable from other constructions.
                                    (Construction Grammar, Goldberg 1995)

 Form                                            Meaning

  block


  walk



                                                    Trajector
   to                                   Source                      Goal
                                                      Path
  Embodied Construction Grammar
                   (Bergen and Chang 2002)

 Embodied representations
  – active perceptual and motor schemas
     (image schemas, x-schemas, frames, etc.)
  – situational and discourse context

 Construction Grammar
  – Linguistic units relate form and meaning.
  – Both constituency and (lexical) dependencies
    allowed.

 Constraint-based
  – based on feature structure unification (as in
    HPSG)
  – Diverse factors can flexibly interact.
              ECG Structures

 Schemas
  – image schemas, force-dynamic schemas,
    executing schemas, frames…
 Constructions
  – lexical, grammatical, morphological,
    gestural…
 Maps
  – metaphor, metonymy, mental space maps…
 Spaces
  – discourse, hypothetical, counterfactual…
                Image schemas

 Trajector / Landmark (asymmetric)
  – The bike is near the house
                                               TR      LM
  – ? The house is near the bike
 Boundary / Bounded Region                           boundary

  – a bounded region has a closed boundarybounded region
 Topological Relations
  – Separation, Contact, Overlap, Inclusion, Surround
 Orientation
  – Vertical (up/down), Horizontal (left/right, front/back)
  – Absolute (E, S, W, N)
                Embodied schemas
                                schema name

   schema Source-Path-Goal                    schema Container
       roles                                      roles
            source              role name              interior
            path                                       exterior
            goal                                       portal
            trajector                                  boundary


                                        Boundary
                                                    Interior
           Trajector
Source                   Goal                                     Portal
             Path
                                                               Exterior


   These are abstractions over sensorimotor experiences.
                Embodied constructions
                                                       ECG Notation
    Form                        Meaning
                                                      construction HARRY
   Harry                                               form : /hEriy/
                                                       meaning : Harry




                                   CAFE                construction CAFE
   cafe                                                 form : /khaefej/
                                                        meaning : Cafe



Constructions have form and meaning poles that are subject to type constraints.
     Representing constructions: TO
           construction TO
            form
              selff.phon  /thuw/
            meaning
              evokes
                                                      local alias
                    Trajector-Landmark as tl
                    Source-Path-Goal as spg
              constraints:                           identification constraint
                    tl.trajector  spg.trajector
                    tl.landmark  spg.goal
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
           The INTO construction
                     construction INTO
                      form
                        selff.phon  /Inthuw/
                      meaning
                        evokes
TO vs. INTO:                  Trajector-Landmark as tl
  INTO adds a                 Source-Path-Goal as spg
  Container schema            Container as cont
  and appropriate       constraints:
  bindings.                   tl.trajector  spg.trajector
                              tl.landmark  cont
                              cont.interior  spg.goal
                              cont.exterior  spg.source
           Constructions with constituents:
           The SPATIAL-PHRASE construction
         construction SPATIAL-PHRASE
          constructional
            constituents
local             sp : Trajector-Landmark
alias             lm : Thing
          form
            spf before lmf                    order constraint
          meaning
            spm.landmark  lmm                 identification constraint




        Constructions may also specify constructional constituents and
        impose form and meaning constraints on them:
           –order constraints
           –identification constraints
An argument structure construction
construction DIRECTED-MOTION
 subcase of Pred-Expr
 constructional
   constituents
         a : Ref-Exp              schema Directed-Motion
         m: Pred-Exp                  roles
         p : Spatial-Phrase                agent : Entity
 form                                      motion : Motion
   af before mf                            path : SPG
   mf before pf
 meaning
   evokes Directed-Motion as dm
   selfm.scene dm
   dm.agent am
   dm.motion  mm
   dm.path  pm
The CAUSED-MOTION construction

 construction CAUSED-MOTION
   subcase of Pred-Expr
   constructional
     constituents
             agent : Entity
             action: Action
             patient: Entity
             path : SPG
   form
      agentf before actionf
      actionf before patientf
      actionf before pathf
   meaning
     evokes Caused-Motion as cm
     selfm.scene cm
     cm.agent agentm
     cm.action  actionm
     cm.patient patientm
     cm.path  pathm
                   Simulation-based language understanding
construction WALKED
   form                                         “Harry walked into the cafe.”    Utterance
       selff.phon  [wakt]
   meaning : Walk-Action
    constraints
       selfm.time before Context.speech-time
       selfm..aspect  encapsulated



                                         Constructions              Analysis Process


      General                                                                   Semantic
      Knowledge                                                                 Specification

                                Belief State


                                                             CAFE
                                                                          Simulation
         Simulation specification




A simulation specification consists of:
- schemas evoked by constructions
- bindings between schemas
Language Understanding Process
Constructional analysis
Semantic Specification
Language Analysis and
Embodied Construction
      Grammar

          John Bryant
   jbryant@icsi.berkeley.edu
                     ECG applications
 Grammar
   – Spatial relations/events                   (Bergen & Chang 1999;
                                                Bretones et al. In press)
   – Verbal morphology            (Gurevich 2003, Bergen ms.)
   – Reference: measure phrases              (Dodge and Wright 2002),
               construal resolution          (Porzel & Bryant 2003),
               reflexive pronouns            (Sanders 2003)
 Semantic representations / inference
   – Aspectual inference                        (Narayanan 1997; Chang, Gildea &
                                                          Narayanan 1998)
   – Perspective / frames                       (Chang, Narayanan & Petruck 2002)
   – Metaphorical inference                     (Narayanan 1997, 1999)
   – Simulation semantics                       (Narayanan 1997, 1999)

 Language acquisition
   – Lexical acquisition             (Regier 1996, Bailey 1997)
   – Multi-word constructions                   (Chang 2004; Chang & Maia 2001)
3. Simulation-based inference
          Interpretation: simulation

                                           walker at goal
Constructions
 can:
 specify which
  schemas and                energy
  entities are
                       walker=Harry                         goal=home
  involved in an
  event, and how
  they are related
 profile particular            Harry is walking home.
  stages of an event
 set parameters of
  an event
            Simulation Semantics
 execution-based model of events/processes
  – tractable, distributed, concurrent, context-sensitive

 X-schemas provide natural model of
  – resource consumption/production
  – goals, preconditions, effects
  – hierarchical events (multiple granularities)
               Simulation Semantics
 Inspired by biological control theory, Simulation
  Semantics models events as executing-, or x-
  schemas.
 An x-schema is a Petri net: a weighted graph
  consisting of places (circles) and transitions
  (rectangles) connected by directed input and output
  arcs.
 A state is defined by the placement of a token (a
  black dot or number) in a particular place.
 The real-time execution        semantics of Petri
  nets models the production and consumption of
  resources:
   – A transition is enabled when its input places are marked such that it
     can fire by movement of tokens from input to output.
   – Arcs include resource, enable and inhibitory arcs.
   – Actions have hierarchical structure, permitting embeddings.

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:2/26/2012
language:
pages:42