CSCTR Session 8
group at UC Berkeley
& Uni of Hawaii
– Nancy Chang
– Benjamin Bergen
– Jerome Feldman, …
– Semantic relations could be extracted from
“In its communicative function, language
is a set of tools with which we attempt to
guide another mind to create within itself
a mental representation that
approximates one we have.” (Delancey
What does language do?
A sentence can evoke an imagined scene and resulting inferences:
“Harry walked to the “Harry walked into the cafe.”
– Goal of action = at cafe – Goal of action = inside cafe
– Source = away from cafe – Source = outside cafe
– cafe = point-like location – cafe = containing location
The scientist walked into the Bonk!!
The hobo drifted into the
The smoke drifted into the house.
Embodiment in language
Perceptual and motor systems play a
central role in language production
– Linguistics: Lakoff, Langacker, Talmy
– Neuroscience: Damasio, Edelman
– Cognitive psychology: Barsalou, Gibbs,
– Computer science: Steels, Feldman
Theory of Theory of
Goal: computationally precise
theories of language
We understand utterances by mentally
simulating their content.
– Simulation exploits some of the
same neural structures activated during
performance, perception, imagining, memory…
– Linguistic structure parametrizes the
Language gives us enough information to simulate
as simulative inference
“Harry walked to the cafe.” Utterance
Schema Trajector Goal Simulation
walk Harry cafe
Belief State Specification
1. Embodiment and Simulation
“What is an idea?
It is an image that paints itself in my brain.”
Neural evidence: Mirror neurons
Gallese et al. (1996) found “mirror”
neurons in the monkey motor cortex,
– an action was carried out
– the same action (or a similar one) was seen.
Mirror neurons found in humans (Porro et al.
Mirror neurons activated when someone:
– imagines an action being carried out (Wheeler et al.
– watches an action being carried out (with and
without object) (Buccino et al. 2000)
The Motor System is somatotopically organized
The Mirror System
The mirror system, like the motor
Buccino et al.,
system, is somatotopically organized.2001
videos of actions
same actions with
Foot actions Hand actions Mouth actions
Mirror neurons for language?
Mirror neurons for specific effectors
activated during passive listening:
– Sentences describing mouth/leg/hand
motions activates corresponding part of pre-
(Tettamanti et al., forthcoming)
– Verbs associated with particular effectors
activates corresponding areas of motor
cortex (Pulvermuller et al. 2001, Hauk et al.
Movement vs. Actions
Embodied language impairs action/perception
– Sentences with visual components to their meaning
can interfere with performance of visual tasks
(Richardson et al. 2003)
– Sentences describing motion can interfere with
performance of incompatible motor actions
(Glenberg and Kashak 2002)
– Sentences describing incompatible visual imagery
impedes decision task (Zwaan et al. 2002)
Simulation effects from fictive motion
– Fictive motion sentences describing paths that require
longer time, span a greater distance, or involve
more obstacles impede decision task (Matlock 2000, Matlock
et al. 2003)
Embodied representations the
norm in robotics!
Computational representations for
lexical semantics: have been
– Spatial relations (Regier 1996)
– Actions (Bailey 1997, Narayanan 1997)
– Objects / attributes (Roy 1998)
Metaphor understanding system
based on simulation (Narayanan 1997)
Missing link: grammar!
Metaphor understanding system
demonstrates that embodied inferences
for difficult case are feasible.
– BUT: system has no grammar!
– How do we bridge the gap?
Need a grammatical theory/formalism that
can served as an interface between
linguistic units and embodied, dynamic,
encyclopedic, context-based information
(i.e., that can support simulation).
”It is not enough to say that the mind is
embodied; one must say how.”
Key borrowed ideas
Conceptual structures are embodied.
– Meaning is conceptualization
(part of larger cognitive system).
– Concepts are grounded in human experience as
physical, psychological and social beings in the
world. (Lakoff 1987, 1985; Langacker 1991, 1987)
Basic symbolic unit at all levels
is a form-meaning pair, or construction.
– Syntax is not independent of semantics.
– Phrasal/clausal constructions can contribute
meaning independently of constituents.
(Fillmore 1988, Kay & Fillmore 1999, Lakoff 1987, Goldberg 1995)
Form-meaning mappings for language
Linguistic knowledge consists of form-meaning mappings:
phonological event structure
cues sensorimotor control
word order attention/perspective
intonation social goals...
A construction is a form-meaning pair whose properties may not be
strictly predictable from other constructions.
(Construction Grammar, Goldberg 1995)
to Source Goal
Embodied Construction Grammar
(Bergen and Chang 2002)
– active perceptual and motor schemas
(image schemas, x-schemas, frames, etc.)
– situational and discourse context
– Linguistic units relate form and meaning.
– Both constituency and (lexical) dependencies
– based on feature structure unification (as in
– Diverse factors can flexibly interact.
– image schemas, force-dynamic schemas,
executing schemas, frames…
– lexical, grammatical, morphological,
– metaphor, metonymy, mental space maps…
– discourse, hypothetical, counterfactual…
Trajector / Landmark (asymmetric)
– The bike is near the house
– ? The house is near the bike
Boundary / Bounded Region boundary
– a bounded region has a closed boundarybounded region
– Separation, Contact, Overlap, Inclusion, Surround
– Vertical (up/down), Horizontal (left/right, front/back)
– Absolute (E, S, W, N)
schema Source-Path-Goal schema Container
source role name interior
Source Goal Portal
These are abstractions over sensorimotor experiences.
Harry form : /hEriy/
meaning : Harry
CAFE construction CAFE
cafe form : /khaefej/
meaning : Cafe
Constructions have form and meaning poles that are subject to type constraints.
Representing constructions: TO
Trajector-Landmark as tl
Source-Path-Goal as spg
constraints: identification constraint
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
The INTO construction
TO vs. INTO: Trajector-Landmark as tl
INTO adds a Source-Path-Goal as spg
Container schema Container as cont
and appropriate constraints:
bindings. tl.trajector spg.trajector
Constructions with constituents:
The SPATIAL-PHRASE construction
local sp : Trajector-Landmark
alias lm : Thing
spf before lmf order constraint
spm.landmark lmm identification constraint
Constructions may also specify constructional constituents and
impose form and meaning constraints on them:
An argument structure construction
subcase of Pred-Expr
a : Ref-Exp schema Directed-Motion
m: Pred-Exp roles
p : Spatial-Phrase agent : Entity
form motion : Motion
af before mf path : SPG
mf before pf
evokes Directed-Motion as dm
The CAUSED-MOTION construction
subcase of Pred-Expr
agent : Entity
path : SPG
agentf before actionf
actionf before patientf
actionf before pathf
evokes Caused-Motion as cm
Simulation-based language understanding
form “Harry walked into the cafe.” Utterance
meaning : Walk-Action
selfm.time before Context.speech-time
Constructions Analysis Process
A simulation specification consists of:
- schemas evoked by constructions
- bindings between schemas
Language Understanding Process
Language Analysis and
– Spatial relations/events (Bergen & Chang 1999;
Bretones et al. In press)
– Verbal morphology (Gurevich 2003, Bergen ms.)
– Reference: measure phrases (Dodge and Wright 2002),
construal resolution (Porzel & Bryant 2003),
reflexive pronouns (Sanders 2003)
Semantic representations / inference
– Aspectual inference (Narayanan 1997; Chang, Gildea &
– Perspective / frames (Chang, Narayanan & Petruck 2002)
– Metaphorical inference (Narayanan 1997, 1999)
– Simulation semantics (Narayanan 1997, 1999)
– Lexical acquisition (Regier 1996, Bailey 1997)
– Multi-word constructions (Chang 2004; Chang & Maia 2001)
3. Simulation-based inference
walker at goal
schemas and energy
involved in an
event, and how
they are related
profile particular Harry is walking home.
stages of an event
set parameters of
execution-based model of events/processes
– tractable, distributed, concurrent, context-sensitive
X-schemas provide natural model of
– resource consumption/production
– goals, preconditions, effects
– hierarchical events (multiple granularities)
Inspired by biological control theory, Simulation
Semantics models events as executing-, or x-
An x-schema is a Petri net: a weighted graph
consisting of places (circles) and transitions
(rectangles) connected by directed input and output
A state is defined by the placement of a token (a
black dot or number) in a particular place.
The real-time execution semantics of Petri
nets models the production and consumption of
– A transition is enabled when its input places are marked such that it
can fire by movement of tokens from input to output.
– Arcs include resource, enable and inhibitory arcs.
– Actions have hierarchical structure, permitting embeddings.