Docstoc

Slide 1 - International Center for Computational Logic.ppt

Document Sample
Slide 1 - International Center for Computational Logic.ppt Powered By Docstoc
					  Language Understanding
and Unified Cognitive Science


            Jerome Feldman
      International Computer Science Institute
              U. California at Berkeley
                    Berkeley, CA
            jfeldman@icsi.berkeley.edu
         Unified Cognitive Science
                  Neurobiology
                  Psychology
                  Computer Science
                  Linguistics
                  Philosophy
                  Social Sciences
                 Experience

Take all the Findings and Constraints Seriously
                            Functionalism
 In fact, the belief that neurophysiology is even relevant to the
functioning of the mind is just a hypothesis. Who knows if we’re
looking at the right aspects of the brain at all. Maybe there are other
aspects of the brain that nobody has even dreamt of looking at yet.
That’s often happened in the history of science. When people say that
the mental is just the neurophysiological at a higher level, they’re being
radically unscientific. We know a lot about the mental from a scientific
point of view. We have explanatory theories that account for a lot of
things. The belief that neurophysiology is implicated in these things
could be true, but we have very little evidence for it. So, it’s just a kind
of hope; look around and you see neurons: maybe they’re implicated.


                               Noam Chomsky 1993, p.85
                Embodiment
   Of all of these fields, the learning of
  languages would be the most
  impressive, since it is the most human
  of these activities. This field,
  however, seems to depend rather too
  much on the sense organs and
  locomotion to be feasible.
  Alan Turing (Intelligent
  Machines,1948)
Continuity Principle of the American Pragmatists
                           Lectures
    I. Overview
    2. Simulation Semantics
    3. ECG and Best-fit Analysis
    4. Compositionality
    5. Simulation, Counterfactuals, and Inference


Utterance      Discourse & Situational
                      Context                Constructions



                     Analyzer:
                     incremental,
                  competition-based,
                    psychologically
                       plausible



                 Semantic Specification:
                                                      Simulation
                  image schemas, bindings,
                      action schemas
        Psycholinguistic evidence
• Embodied language impairs action/perception
   – Sentences with visual components to their meaning can
     interfere with performance of visual tasks
                                                   (Richardson et al. 2003)
   – Sentences describing motion can interfere with performance
     of incompatible motor actions
                                                   (Glenberg and Kashak 2002)
   – Sentences describing incompatible visual imagery impedes
     decision task (Zwaan et al. 2002)
• Simulation effects from fictive motion sentences
   – Fictive motion sentences describing paths that require
     longer time, span a greater distance, or involve more
     obstacles impede decision task (Matlock 2000, Matlock et al. 2003)
  Neural evidence: Mirror neurons
• Gallese et al. (1996) found “mirror” neurons in
  the monkey motor cortex, activated when
   – an action was carried out
   – the same action (or a similar one) was seen.
• Mirror neuron circuits found in humans (Porro et al.
  1996)

• Mirror neurons activated when someone:
   – imagines an action being carried out (Wheeler et al. 2000)
   – watches an action being carried out (with or without
     object) (Buccino et al. 2000)
               The Mirror System
The mirror system, like the motor system, is
somatotopically organized.         Buccino et al.,
                                          2001

                                     humans watching
                                     videos of actions
                                     without objects

                                     humans watching
                                     same actions with
                                     objects


 foot
Foot actions      hand
                 Hand actions    mouth
                                Mouth actions
Fast Brain ~ Slow Neurons

Mental Connections are Active
    Neural Connections

There is No Erasing in the Brain
Movement vs. Actions
   Pulvermueller Lab
                Brains ~ Computers
•   1000 operations/sec     •   1,000,000,000 ops/sec
•   100,000,000,000 units   •   1-100 processors
•   10,000 connections/     •   ~ 4 connections
•   graded, stochastic      •   binary, deterministic
•   embodied                •   abstract, disembodied
•   fault tolerant          •   crashes frequently
•   evolves                 •   explicitly designed
•   learns                  •   is programmed
        The ICSI/Berkeley
Neural Theory of Language Project

                                      ECG
                              Learning early
                              constructions
                              (Chang, Mok)
               Active representations
• Many inferences about actions derive from what
  we know about executing them
• Representation based on stochastic Petri nets
  captures dynamic, parameterized nature of actions

                     walker at goal


                                      Walking:
                                      bound to a specific walker with a
                                         direction or goal
      energy
                                      consumes resources (e.g., energy)
                                      may have termination condition
walker=Harry                             (e.g., walker at goal)
goal=home
                                      ongoing, iterative action
     Learning Verb Meanings
                  David Bailey


A model of children learning their first verbs.
Assumes parent labels child’s actions.
Child knows parameters of action, associates with word
Program learns well enough to:
  1) Label novel actions correctly
  2) Obey commands using new words (simulation)
System works across languages
Mechanisms are neurally plausible.
System Overview
 Learning Two Senses of PUSH




Model merging based on Bayesian MDL
        The ICSI/Berkeley
Neural Theory of Language Project

                                      ECG
                              Learning early
                              constructions
                              (Chang, Mok)
         The Binding Problem
 Massively Parallel Brain

 Unitary Conscious Experience

 Many Variations and Proposals

 Our focus: The Variable Binding Problem
                   SHRUTI
• SHRUTI does inference
  by connections between
  simple computation
  nodes
• Nodes are small groups
  of neurons
• Nodes firing in sync
  reference the same
  object
  Proposed Alternative Solution
• Indirect references
  – Pass short signatures, “fluents”
      • Functionally similar to SHRUTI's time slices
  – Central “binder” maps fluents to objects
      • In SHRUTI, the objects fired in that time slice
  – Connections need to be more complicated than
    in SHRUTI
      • Fluents are passed through at least 3 bits
      • But temporal synchrony is not required
                           Lectures
    I. Overview
    2. Simulation Semantics
    3. ECG and Best-fit Analysis
    4. Compositionality
    5. Simulation, Counterfactuals, and Inference


Utterance      Discourse & Situational
                      Context                Constructions



                     Analyzer:
                     incremental,
                  competition-based,
                    psychologically
                       plausible



                 Semantic Specification:
                                                      Simulation
                  image schemas, bindings,
                      action schemas
  Ideas from Cognitive Linguistics
• Embodied Semantics (Lakoff, Johnson, Sweetser, Talmy
• Radial categories    (Rosch 1973, 1978; Lakoff 1985)
    – mother: birth / adoptive / surrogate / genetic, …
• Profiling (Langacker 1989, 1991; cf. Fillmore XX)
    – hypotenuse, buy/sell (Commercial Event frame)
• Metaphor and metonymy                 (Lakoff & Johnson 1980, …)
    – ARGUMENT IS WAR, MORE IS UP
    – The ham sandwich wants his check.
• Mental spaces (Fauconnier 1994)
    – The girl with blue eyes in the painting really has green eyes.
• Conceptual blending (Fauconnier & Turner 2002, inter alia)
    – workaholic, information highway, fake guns
    – “Does the name Pavlov ring a bell?” (from a talk on ‘dognition’!)
                   Image schemas

• Trajector / Landmark (asymmetric)                      TR        LM
   – The bike is near the house
   – ? The house is near the bike                                 boundary

• Boundary / Bounded Region                                bounded region

   – a bounded region has a closed boundary
• Topological Relations
   – Separation, Contact, Overlap, Inclusion, Surround
• Orientation
   – Vertical (up/down), Horizontal (left/right, front/back)
   – Absolute (E, S, W, N)
               Schema Formalism
SCHEMA <name>
 SUBCASE OF <schema>
 EVOKES <schema> AS <local name>
 ROLES < self role name>: <role restriction>
        < self role name> <-> <role name>
 CONSTRAINTS <role name> <-        <value>
                <role name> <-> <role name>
           A Simple Example

SCHEMA hypotenuse
 SUBCASE OF line-segment
 EVOKES right-triangle AS rt
 ROLES Comment inherited from line-segment
 CONSTRAINTS
    SELF <-> rt.long-side
           Language understanding: analysis &
construction WALKED
                      simulation
   form                                         “Harry walked into the cafe.”    Utterance
       selff.phon  [wakt]
   meaning : Walk-Action
    constraints
       selfm.time before Context.speech-time
       selfm..aspect  encapsulated



                                         Constructions              Analysis Process


      General                                                                   Semantic
      Knowledge                                                                 Specification

                                Belief State


                                                             CAFE
                                                                          Simulation
          Semantic specification




The analysis process produces a semantic specification that
•includes image-schematic, motor control and conceptual structures
•provides parameters for a mental simulation
       Task: Interpret simple discourse
              fragments/ blurbs
France fell into recession. Pulled out by Germany
US Economy on the verge of falling back into recession after
  moving forward on an anemic recovery.
Indian Government stumbling in implementing Liberalization
  plan.
Moving forward on all fronts, we are going to be ongoing and
  relentless as we tighten the net of justice.
The Government is taking bold new steps. We are loosening
  the stranglehold on business, slashing tariffs and removing
  obstacles to international trade.
                             Results
• Model was implemented and tested on discourse fragments from a
  database of 50 newspaper stories in international economics from
  standard sources such as WSJ, NYT, and the Economist.
• Results show that motion terms are often the most effective method to
  provide the following types of information about abstract plans and
  actions.
   – Information about uncertain events and dynamic changes in goals
      and resources. (sluggish, fall, off-track, no steam)
   – Information about evaluations of policies and economic actors and
      communicative intent (strangle-hold, bleed).
   – Communicating complex, context-sensitive and dynamic economic
      scenarios (stumble, slide, slippery slope).
   – Commincating complex event structure and aspectual information
      (on the verge of, sidestep, giant leap, small steps, ready, set out,
      back on track).
• ALL THESE BINDINGS RESULT FROM REFLEX, AUTOMATIC
  INFERENCES PROVIDED BY X-SCHEMA BASED
  INFERENCES.
Embodied Construction Grammar

• Embodied representations
  – active perceptual and motor schemas
  – situational and discourse context


• Construction Grammar
  – Linguistic units relate form and meaning/function.
  – Both constituency and (lexical) dependencies allowed.


• Constraint-based (Unification)
  – based on feature structures (as in HPSG)
  – Diverse factors can flexibly interact.
     Embodied Construction Grammar
                 ECG
      (Formalizing Cognitive Linguistics)

1. Community Grammar and Core Concepts
2. Deep Grammatical Analysis
3. Computational Implementation
  a. Test Grammars
  b. Applied Projects – Question Answering
4. Map to Connectionist Models, Brain
5. Models of Grammar Acquisition
               Verb Constructions
Construction BITE1                   schema ForceApplication
 subcase of Verb                      subcase of MotorControl
 form: bite                           evokes ForceTransfer as FT
 meaning: ForceApplication             roles
   constraints:                          Actor ↔ FT.Supplier ↔ Protagonist
     Effector ← teeth                    Acted Upon ↔ FT.Recipient
                                         Effector
     Routine ← bite // close mouth
                                         Routine
                                         Effort ↔ FT.Force.amount
                Semantic Specification
                       He bit the apple
EventDescriptor
 eventtype
 ProfiledProcess       CauseEffect        ForceApplication
 ProfiledParticipant    causer             actor
                        affected           actedupon
                                           routine  bite
                                           effector  teeth

          RD27
          category      Person


                                     RD55
                       Apple
                                     category
       Modeling context for
language understanding and learning

• Linguistic structure reflects experiential
  structure
  – Discourse participants and entities
  – Embodied schemas:
     • action, perception, emotion, attention, perspective
  – Semantic and pragmatic relations:
     • spatial, social, ontological, causal

• ‘Contextual bootstrapping’ for grammar learning
         Constrained Best Fit in Nature
                    inanimate      animate
physics           lowest energy
                  state
chemistry         molecular
                  minima
biology                            fitness, MEU
                                  Neuroeconomics
vision                             threats,
                                   friends
language                           errors,
                                   NTL
   Two perspectives on language
            learning
Computational models                   Developmental evidence
• Grammatical induction               • Prior knowledge
   – language identification             –   concepts
   – context-free grammars,              –   event-based knowledge
     unification-based                   –   social cognition
     grammars                            –   lexical items
   – statistical NLP
• Word learning models                • Data-driven learning
   – semantic representations            – basic scenes
       • logical forms
                                         – lexically specific patterns
       • discrete representations
       • continuous representations      – usage-based learning
   – statistical models
            Language Acquisition
• Opulence of the substrate
  – Prelinguistic children already have rich
    sensorimotor representations and sophisticated
    social knowledge
  – intention inference, reference resolution
  – language-specific event conceptualizations
                             (Bloom 2000, Tomasello 1995,
                           Bowerman & Choi, Slobin, et al.)
• Children are sensitive to statistical information
  – Phonological transitional probabilities
  – Most frequent items in adult input learned earliest
                       (Saffran et al. 1998, Tomasello 2000)
  Experiment: learning verb islands
• Question:
  – Can the proposed construction learning model
    acquire English item-based motion constructions?
    (Tomasello 1992)



                               Form:            text : throw the ball
• Given: initial lexicon and                    intonation : falling
                               Participants :   Mother, Naomi, Ball
  ontology                     Scene :          Throw
                                                thrower : Naomi
• Data: child-directed                          throwee : Ball
  language annotated with      Discourse :      speaker :Mother
                                                addressee Naomi
  contextual information                        speech act : imperative
                                                activity : play
                                                joint attention : Ball
The intuition behind learning a new
form-meaning pairing from context

                           Put-Action
                              put-agent
        construction Put
                              put-theme
                              location
          before



       construction Coat       Coat

          before


       construction Here       Sofa
The learner learns a new lexically-specific
construction from the form-meaning pair

        construction Put-Coat-Here

          constituents
            v: Put
            o: Coat
            p: Here

          form
             vf before of before pf

          meaning: Caused-Motion-Scene
            selfm.means  vm
            selfm.mover  om
            selfm.path  pm
   Experiment: learning verb islands
Subset of the CHILDES database of parent-child
  interactions (MacWhinney 1991; Slobin )
• coded by developmental psychologists for
   – form: particles, deictics, pronouns, locative phrases, etc.
   – meaning: temporality, person, pragmatic function,
     type of motion (self-movement vs. caused movement;
     animate being vs. inanimate object, etc.)
• crosslinguistic (English, French, Italian, Spanish)
   – English motion utterances: 829 parent, 690 child utterances
   – English all utterances: 3160 adult, 5408 child
   – age span is 1;2 to 2;6
        A quantitative measure: coverage
• Goal: incrementally improving comprehension
   – At each stage in testing, use current grammar to analyze test set
• Coverage = % role bindings analyzed

• Example:
   – Grammar: throw-ball, throw-block, you-throw
   – Test sentence: throw the ball.
       • Bindings: scene=Throw, thrower=Nomi, throwee=ball
       • Parsed bindings: scene=Throw, throwee=ball
                         – Score test grammar on sentence: 2/3 = 66.7%
Learning to comprehend
          Usage-based learning,
       comprehension, and production
                              discourse & situational
                                     context


                                   world knowledge

utterance                                                                  comm. intent


                                     constructicon
                   reinforcement                      reinforcement
        analyze       (usage)                            (usage)
           &                                                           generate
        resolve

                                      hypothesize
      analysis                       constructions                    utterance
                                     & reorganize


      simulation     reinforcement                   reinformcent     response
                      (correction)                    (correction)
     Unified Cognitive Science
             Neurobiology
             Psychology
             Computer Science
             Linguistics
             Philosophy
             Social Sciences
             Experience


Take all the Findings and Constraints Seriously
                      The ICSI/Berkeley
              Neural Theory of Language Project

•   Principal investigators             Alumni
       Jerome Feldman (UCB,ICSI)         Terry Regier (UCB Ling,
       George Lakoff (UCB Ling)           CogSci)
       Srini Narayanan (UCB,ICSI)
                                          Johno Bryant (Ask)
       Lokendra Shastri (now India)
                                          David Bailey (Google)
• Affiliated faculty                      Leon Barrett (Google)
    Chuck Fillmore (ICSI)                Nancy Chang (Sony Paris)
    Eve Sweetser (UCB Ling)              Joe Makin (UCSF)
    Rich Ivry (UCB Psych)                Eva Mok (U. Chicago)
    Lisa Aziz-Zadeh (USC)
                                          Andreas Stolcke (ICSI, SRI)
 Graduate Students                       Dan Jurafsky (Stanford Ling)
       *Ellen Dodge (Ling)
                                          Olya Gurevich (Powerset)
       Michael Ellsworth (Ling)
       Joshua Marker (Ling)              Benjamin Bergen (UCSD)
       Shweta Narayan (Ling)             Carter Wendelken (UCB)
               Source-Path-Goal

SCHEMA: spg
ROLES:
 source: Place
 path: Directed Curve
 goal: Place
 trajector: Entity
          Translational Motion
SCHEMA translational motion
 SUBCASE OF motion
 EVOKES spg AS s
 ROLES
   mover <-> s.trajector
   source <-> s.source
   goal   <-> s.goal
 CONSTRAINTS
    before:: mover.location <-> source
    after::   mover.location <-> goal
         Event Structure for semantic QA
                               Srini Narayanan
• Reasoning about dynamics
    – Complex event structure
        • Multiple stages, interruptions, resources, framing
    – Evolving events
        • Conditional events, presuppositions.
    – Nested temporal and aspectual references
        • Past, future event references
    – Metaphoric references
        • Use of motion domain to describe complex events.
• Reasoning with Uncertainty
    – Combining Evidence from Multiple, unreliable sources
    – Non-monotonic inference
        • Retracting previous assertions
        • Conditioning on partial evidence
       Components of the System
• Object references
   – Fluents
   – Binder
• Short term storage
   – Predicate state
• Long term storage
   – Facts, mediators, what predicates exist
• Inference
   – Mediators
• Types
   – Ontology
Simulation-based language understanding
                           “Harry walked to the cafe.”        Utterance

Constructions
                                                   Analysis Process

     General
     Knowledge
                            Schema    Trajector     Goal     Simulation
                            walk      Harry         cafe
            Belief State                                     Specification


                                            Cafe
                                                           Simulation
  Representing constructions: TO
           construction TO
            form
              selff.phon  /thuw/
            meaning
              evokes
                                                      local alias
                    Trajector-Landmark as tl
                    Source-Path-Goal as spg
              constraints:                           identification constraint
                    tl.trajector  spg.trajector
                    tl.landmark  spg.goal
The meaning pole may evoke schemas (e.g., image schemas) with a
local alias. The meaning pole may include constraints on the schemas
(e.g., identification constraints ).
        The INTO construction
                     construction INTO
                      form
                        selff.phon  /Inthuw/
                      meaning
                        evokes
TO vs. INTO:                  Trajector-Landmark as tl
  INTO adds a                 Source-Path-Goal as spg
  Container schema            Container as cont
  and appropriate       constraints:
  bindings.                   tl.trajector  spg.trajector
                              tl.landmark  cont
                              cont.interior  spg.goal
                              cont.exterior  spg.source
An ECG analysis with THROW-TRANSITIVE

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:5/16/2012
language:English
pages:58
suchufp suchufp http://
About