Learning Center
Plans & pricing Sign in
Sign Out

Knowledge Representation --- Images and Propositions

VIEWS: 207 PAGES: 116

									Knowledge Representation
--- Images and Propositions

       Lecturer: Siyun Liu
Knowledge Structures
       Mental Representation of Knowledge        »
         declarative knowledge
            facts that can be stated
            knowing that
         procedural knowledge
            procedures that can be implemented
            knowing how
Knowledge Representations
   External Representations »

   Internal Representations »

External Representations
   Pictures »

   Words »

   The differences between picture
    representation and word
    representation »
    Picture Representations
   The picture is relatively analogous to the real-
    world object it represents;

   The picture shows concrete attributes (shape
    & relative size) that are similar to the features
    and spatial properties of the real-world object
    the picture represents.

   Pictorial representations convey all features
Picture Representation

Word Representation
   Words are symbolic representations

   Words capture abstract and
    categorical information

   Representations in words usually
    convey information sequentially

Words Are Symbolic
   拉丁符号:Ç Ì Ñ Ô Ø à ã Ë Đ
   希腊符号:Ύ ξλζέ Ϊ ΠΣΘ ς ΰ
   希伯来语:‫א ז כ ס ץ ף ל ה ב‬
   汉语:中国

    Pictures vs. Words
    Neither form of representation actually
     retains all of the characteristics of
     what is being represented

    Pictures and words represent
     relationships in different ways

Internal Representations
   Mental Images »

   Propositions »

Mental Images
   Definition »
   Various Forms »

Definition of Mental Images
       The mental representation of things
        that are not currently being sensed by
        the sense organs;
         It may represent things that have never
          been observed by your senses at any
         It may represent things that do not exist
          at all.
Various Forms
       Seeing ------ visual images »
         seeing in the absence of a visual stimulus
         not presently visible to the eyes
       Hearing
       Smell
       Taste
    Mental Imagery
   Memory Codes of Mental Imagery     »
   Empirical Investigations of Imagery »
   The nature of mental imagery »
       Questions about mental images
            What are images?
            What kinds of properties do images have?
            How are these like or unlike the properties that real pictures

   Neuropsychological Findings                »
Memory Codes of Mental Imagery

   The Dual-Coding Hypothesis   »

   The Relational-Organizational
    Hypothesis »

   The Dual-Coding Hypothesis
      Allan Paivio (1969, 1971)
          We can use either verbal or imagined codes or
           both for representing information.
               Verbal
                     Containing information about an item’s abstract, linguistic
               Imagined
                     Mental pictures of some sort that represent what the item
                      looks like
          Pictures and concrete words
               Both verbal labels and visual images
          abstract words
               Only verbal label
                                            Evidence1 for Dual-Coding
Home                                       Evidence2 for Dual-Coding
Evidence for Dual-Coding
   Paivio (1965)
       four lists of noun pairs
            CC: concrete word vs. concrete word
                 Book vs. Table
            CA: concrete word vs. abstract word
                 Chair vs. Justice
            AC: abstract word vs. concrete word
                 Freedom vs. Dress
            AA: abstract word vs. abstract word
                 Beauty vs. Truth

     Evidence for Dual-Coding
   Participants learnt one of four lists, then
    had a recall test

Evidence for Dual-Coding
   Interpretations for results (1969)
       When items are coded by both images and
        verbal labels, the chances of the learner’s
        retrieving them are obviously better.
            If the learner forgets the verbal label, he or she
             might still access the visual images, or vice
       Items coded only by verbal labels are
            If the verbal label is forgotten or “misplaced”,
             the learner has less to go on.
    Evidence for Dual-Coding
   Interpretations for results (1969)
       The first noun in a pair (“stimulus” noun) serves as
        a conceptual peg on which the second (“response”)
        noun is hooked.
       The stimulus noun serves as a “mental anchor”, a
        place to which the representation of the response
        noun can be attached.
       The imaginability of the first noun is particularly
        important in improving memorability
            This is why recall in the CA condition was significantly
             higher than in the AC condition

Evidence for Dual-Coding
   Interpretations for results
       Whenever possible, participants
        spontaneously formed visual images of the
        noun pairs
            The formation was easiest with concrete nouns
       Visual imagery, unlike verbal labelling,
        increases as a function of concreteness.
            The more concrete the noun, the richer the
             image and the more elaborated the internal
    Evidence for Dual-Coding
   Lee Brooks (1968)
       Provided evidence that images are distinct from
        verbal materials or at least use different processes
        from those used by verbal materials.
       Two types of tasks
            visual task:
                  briefly presented a picture, answer questions about it
            verbal task:
                  briefly presented a sentence, make judgment about it
       Two types of responses
            Visual: point to an answer
            Verbal: saying “yes” or “no”

Evidence for Dual-Coding
   Lee Brooks (1968)
       Material 1 used in the experiment
            Task for material 1
            Findings for task 1
       Material 2 used in the experiment
            Task for material 2
            Findings for task 2

    Evidence for Dual-Coding

                                A BIRD IN THE HAND
                                IS NOT IN THE BUSH.

Start at the corner marked      For each word in the sentence
with a dot, and indicate        above, indicate whether or
whether or not each corner is   not each word is a concrete
at the extreme top or bottom.   noun.
    Evidence for Dual-Coding
   Participants responded in different ways
       Verbal
            Participants say “yes” or “no”
       Spatial
            Participants were given a response sheet on
             which the letters Y and N were printed in an
             irregular pattern, and were told to point to
             either a Y or an N in each row to indicate their
Evidence for Dual-Coding
   Task 1:
       Participants took almost two and a half
        times longer when they responded by
        pointing than they did by responding
        verbally. Reason
   Task 2:
       Participants were faster to respond by
        pointing than they were to respond
        verbally. Reason
Evidence for Dual-Coding
   It requires the formation of a visual image of
    an F.
       The visual image probably has at least some
        picture-like qualities (spatial or visual), so a spatial
        or visually guided response (pointing) would be
        interfered with to a greater extend than would a
        verbal response.
       The visual image is more disruptive of, and
        disrupted by, another spatial or visual type of task
        (pointing) than by a verbal kind of task (talking).
Evidence for Dual-Coding
   Holding a sentence in memory (a verbal
    task) is easier to do with a concurrent
    visual/spatial task (such as pointing)
    than with another verbal task.

Evidence for Dual-Coding
   Conclusion
       Images and words use different kinds of
        internal codes (dual-coding hypothesis)
       Visual images and words are separate
        codes Try it
            Visual imagery interferes with other visual tasks
            Verbal tasks interfere with other verbal tasks
            Visual imagery does not interfere with verbal

Which of the following tasks is (are) the
hardest one(s)?

-Imagine an elephant and at the same time try
to draw a house.

-Imagine the definition of “elephant” and at the
same time write down a definition of short-
term memory.

- Draw an elephant and at the same time
provide a definition of short-term memory.
The Relational-Organizational Hypothesis

          Bower (1970b)
              The relational-organizational hypothesis
                   An alternative to the dual-coding hypothesis
                   Imagery improved memory by producing more
                    associations between the items to be recalled,
                    rather than being richer than verbal labels
                        Individuals create a number of links or hooks
                         between the information to remember and other
                         information in paired-associates learning.

Home                                                          Continue…
The Relational-Organizational Hypothesis

       Imagery facilitates the creation of a
        greater number of hooks that link the
        two to-be-remembered pieces of
           Bower (1970b) provided evidence to
            distinguish between the dual coding and
            the relational-organizational hypothesis

Evidence for relational-organizational
   Three groups of participants with
    different instructions for a paired-
    associates learning task
       G1: to rehearse aloud
       G2: to construct two images that did not
        interact and were “separated in imaginal
        space” see here
       G3: to construct an interactive scene of the
        two words in a pair see here
Piano vs. Cigar

Dog vs. Bathtub

Evidence for relational-organizational

   Results
       All participants recognized about 85% of
        the previously seen words, however,
            G1: recalled 30% of the paired associates
            G2: recalled 27% of the paired associates
            G3: recalled 53% of the paired associates

Evidence for relational-organizational

   If Dual-Coding hypothesis is correct
       G2 = G3
           Imagery simply led to more elaborated coding
            of the paired associates
   However
       G2 < G3
           It is not imagery per se that helps memory
           Rather, it is the way in which imagery is used
    Evidence for relational-organizational

   Interacting images presumably create or
    suggest more links between the target
    information and other information, making
    the target information easier to retrieve

   Evidence support for relational-
    organizational hypothesis

Empirical Investigations of Imagery

    Mental Rotation of Images
        Shepard & Metzler (1971)       »
        Cooper & Shepard        (1973) »
    Images Scanning
        Functional-equivalence hypothesis   »
        Image Scaling   »
        Image Scanning      »
Mental Rotation of Images

   Shepard & Metzler (1971)
       They showed participants perspectives line
        drawings of three dimensional objects.
       Participants were presented two drawings
            The drawings depicted the same object but with
             one rotated by some degree. See here
                 Rotation was in a picture plane
                 Rotation was in depth
            The drawings depicted mirror-image reversals
                 The objects were similar but not identical. See here
                 The mirror images were also sometimes rotated.
Mental Rotation of Images

Mental Rotation of Images

Mental Rotation of Images

Mental Rotation of Images
   The amount of time it took participants
    to decide if the two drawings depicted
    the same object or a mirror-image
    reversal was directly proportional to the
    angle of rotation between the drawings.

                          See Figure Here
    Mental Rotation of Images


            0   20    40   60   80 100 120 140 160 180

                     Angle of rotation (degree)      Return
Mental Rotation of Images
   Changes in manipulations
       Materials
            Rotation of letters
       Procedure
            Participants given a cue showing the
             orientation to which the test stimulus would be
             rotated, before the test stimulus appeared

Mental Rotation of Images

       Mental Rotation of Images
          Participants’ performances were the
           same for all angles of rotation if the
           cues were presented early enough
              1000 msec before the test stimulus
          Participants were able mentally to
           rotate their images either clockwise or
           counterclockwise, depending on which
           direction led to a lesser angle.
Home                                       More questions
Mental Rotation of Images
   Are participants in these experiments
    mentally rotating the whole stimulus, or
    are they looking only at certain parts?

   Lynn Cooper (1975)
       Used irregular polygons as materials ▶
Mental Rotation of Images

Mental Rotation of Images
   Two steps:
       Participants were first trained to
        discriminate between original and mirror-
        image reflections of the polygons
       Then they were shown either the original
        polygons or the reflections at different
        angles of rotation and were asked to
        determine whether the object depicted was
        the original or a reflection of the original.
Mental Rotation of Images

Mental Rotation of Images
   The reaction times again increased
    linearly with the angle of rotation
   The rate of rotation was the same for
    all the polygons, regardless of their
       Participants mentally rotated entire
            Treated the very simple polygons in exactly the
             same manner as they did the very complex
Functional-equivalence hypothesis

     Although we do not construct images
      that are exactly identical to percepts,
      we do construct images that are
      functionally equivalent to percepts
     These functionally equivalent images
      are analogous to the physical percepts
      they represent
Home                             Apply to …
Functional-equivalence hypothesis

            Red eye
            Eat carrots
            Short tail

    Image Scaling
   Seeing featural details of large objects is easier than
    seeing such details of small ones, and we respond more
    quickly to questions about large objects we observe
    than to questions about small ones we observe.
   If imaginal representation is functionally equivalent to
    perception, participants also should respond more
    quickly to questions about features of imaginally large
    objects than to questions about features of imaginally
    small ones. Example
       this prediction was confirmed
Image Scaling
1. The bigger
   the size, the
   more detail
2. Even with

 Image Scaling

are faster for
rabbits when
they are next
to smaller

   Image Scanning
      The visual images constructed and
       transformed by people are like real
       pictures in many ways.
          People’s scanning of their visual images is
           in some ways similar to their scanning of
           actual pictures.
               The greater the distance between two parts,
                the longer it takes to scan between them.

Home                               Evidence 1 for Image Scan
                                   Interesting wrinkles
    Image Scanning
   Kosslyn (1973)
       Procedure
            Participants memorized a picture of an object (e.g., boat)
            Then they created an image of that object in their mind
            Participants were asked to focus on one part, then asked
             to look for another part on the boat and press a key
       Rationale
            If imagery, like perception, is spatial (i.e., analog), then it
             should take longer for participants to find parts that are
             located further from the initial point of focus See here
       This was supported by the results
            See also Kosslyn et al (1978)
Image Scanning

Image Scanning

Image Scanning

Image Scanning
   Barbara Tversky (1981) ▶
       People’s maps are systematically distorted
        in orienting and anchoring oddly shaped
            Use different heuristics, or rule of thumb
            Follow the Principles of perceptual organization

   Chambers and Reisberg (1992) ▶

                                       Which is
    Image Scanning                   further west

   Draw a map of United States (China), put
    in several cities:
       Seattle; Portland, Oregon; Carson City; Los
        Angeles; San Diego; Chicago; Boston;
        Portland, Maine; Philadelphia; New York;
        Washington, D. C. ▶
       北京,天津,石家庄,昆明,西安,济南,成
        都,上海,南京,杭州 ▶
Image Scanning

    Image Scanning
   Show participants an ambiguous figure ▶

   Presented the actual drawing for only
    about 5 seconds
       Enough time to form an image but not
        enough time to “reverse” the figure

Image Scanning

     Two different instructions
         Some participants were told it was a duck
         Others were told it was a rabbit

Image Scanning
   Participants were presented with a pair
    of duck/rabbits


A          B               A            C

   To choose which had actually been
Image Scanning
   When participants thought they were
    imaging a duck, they were well above
    chance at detecting the difference
    between (A) and (B);
   Exactly the opposite pattern emerged
    for those who had formed an initial
    image of a rabbit.
Image Scanning
   People paid more attention to the
    region they took to be the creature’s
    “face” and less to the back of the
    creature’s head.
   People who form images of the same
    physical stimulus, but who give
    different construals or meanings to the
    stimulus, actually form different images.
Image Scanning
   Language hints in constructing images
       Front-back
       Furrier
       Cleaner-dirtier; fatter-thinner
       Better-worse; smarter-dumber
   Imagery created depends a lot on the
    nature of the task at hand

The nature of mental imagery
   Principles of Visual Imagery   »

   Critiques of Mental Imagery Research and
    Theory »

   The significance of knowing whether people
    use imagery as a means of coding
    information »
Principles of Visual Imagery
   Finke’s (1989) review
       Describe the fundamental nature and
        properties of visual images
       Five Principles
            Implicit Encoding ▶
            Perceptual Equivalence ▶
            Spatial Equivalence ▶
            Transformational Equivalence ▶
            Structural Equivalence ▶

Implicit Encoding
   Images are places from which some
    information can be obtained, even if
    that information was never intentionally
       Imagery can be used to answer questions
        for which you probably don’t have a
        directly stored answer
            Tracing an F in Brook’s (1968) task

Perceptual Equivalence
   Many of the same kinds of internal
    processes used in mental visualization
    are used in visual perception as well

       Study by Farah (1985) ▶

Perceptual Equivalence
   Participants were asked to form an image of
    a letter
       H or T
   When H was presented later at a low level of
       Participants were better at detecting the letter
        compared to the one participants had not form an
   Imagery can “prime” the visual pathway used
    in detecting an actual stimulus
       Visual imagery is as perceptual “anticipation”
            The visual system is getting ready to actually to see
    Spatial Equivalence
   The spatial arrangement of the elements
    of a mental image corresponds to the way
    objects or their parts are arranged on
    actual physical surfaces or in an actual
    physical space
       Study by Kosslyn et al
       Map scanning study by Nancy Kerr (1983) ▶
Spatial Equivalence
   Two groups of participants
       Congenitally blind participants
            Learned the map by feeling objects
       Sighted participants
   Task
       Imagined moving a dot from one object to
   Findings
       The greater the distance between objects, the
        longer it took both blind and sighted participants
        to scan
       Visual imagery has spatial properties
Transformational Equivalence
   Imagined transformations and physical
    transformations exhibit corresponding
    dynamic characteristics and are
    governed by the same laws of motion
       Example: mental rotation
       Other kinds of transformations will work
        with images in much the same way they
        work with real objects as well
Structural Equivalence
   Images are organized and assembled
       The structure of mental images corresponds to
        that of actual perceived objects, in the sense that
        the structure is coherent, well organized, and can
        be reorganized and reinterpreted.
            The larger the object, the more time it would take to
             look over or to draw
            Visual images are formed, not all at once, but in pieces
             that are assembled into a final rendition
       Study by Kosslyn et al (1983) ▶
Structural Equivalence
   Participants were asked to form images of
    pictures that differed in amount of detail ▶
       It took participants about one and a third times as
        long to form an image of the detailed pictures as it
        did other participants to form images of outline
   The greater the complexity of the conceived
    structure of the object, the longer it takes to
    assemble an image of it
Structural Equivalence

    Critiques of Mental Imagery
    Research and Theory
   Tacit knowledge and demand
    characteristics ▶

   The picture metaphor ▶

   Propositional theory ▶

    Tacit knowledge and demand
   Pylyshyn (1981)
       The results from many imagery studies reflect
        participants’ underlying and implicit, tacit
        knowledge and beliefs about the task rather
        than their construction and manipulation of
        visual images.
            Participants’ scanning time is proportional to
             distance scanned, because they know that the
             amount of time it takes to physically scan between
             two points in a visual display depends on distance
             and because they expect the experiment to
             demand this kind of performance.
                 Knowledge and expectation       Continue…
Tacit knowledge and demand
   Finke (1989)
       Move a cup from one end of a desk to
        another end
   Two options to do the task
       Slide the cup across the desk
       Pick up the cup and place it in the new

Tacit knowledge and demand
   Demand characteristics of tasks
       Some tasks make it obvious to participants
        how the participants ought to perform
       The instructions, the tasks themselves, or
        something else about the situation cues
        the person on how to behave

Tacit knowledge and demand
   Experimenter expectancy effects
       Participants try to please and may behave
        artificially just to perform in ways they
        believe will satisfy the experimenter
            Intons-Peterson (1983) conducted several
             imagery studies
                 Had some experimenters believe the results would
                  turn out in one way
                 Had other experimenters believe the opposite
                 Results: participants performed as experimenters
                  expected them to
The picture metaphor
   Much discussion of an analogy between
    pictures and images
   Pylyshyn (1973)
       Pictures and images differ in several ways
            You can physically look at a picture without
             first knowing what it’s a picture of
            You cannot “look” at an image unless you first
             know what it is
                 Images are internal constructions formed with some
                  intention in mind
The picture metaphor
   Pictures and images are disrupted, and
    disruptable, in different ways.
       You can cut a photograph in half, with the
        result that arbitrary parts of the objects
        depicted disappear.
       Images are organized more meaningfully,
        and when they fade, only the meaningful
        parts disappear.
The picture metaphor
   Images seem more easily distorted by
    the viewer’s interpretations than are
    pictures or photographs
       Carmichael et al (1932)
            Presented participants with patterns ▶
            Participants’ later reproductions of the patterns
             were distorted in accordance with the label
             initially provided

The picture metaphor
  Word                              Word
  list1         Stimulus Figures    list2

  Curtains in                      Diamond in
  a window                         a rectangle

  Bottle                           Stirrup

  Crescent                         Letter C

The picture metaphor
   Nickerson & Adams (1979)
       People make many errors when trying to
        reproduce their images of familiar objects
            Draw a penny
                 How much information is on your penny?

Propositional theory
   Propositional theorists believe there is a
    single code, neither visual nor verbal
    but propositional in nature, that is used
    to store and mentally represent all
       Propositions are a means of specifying
        relationships between different concepts

    Propositional theory
   Pylyshyn (1973)
       All information is mentally represented and
        stored by propositions.
            Participants in visual imagery experiments might
             look as if they were consulting or manipulating
             internal visual representations, but they would
             actually be using internal propositional
             representations, the same kind of representations
             that underlie their processing of verbal material.
                 Sentences or stories

Propositional theory
   Kosslyn (1976)
       Tested the association strength between animals
        and their physical attributes
            “claws” are more strongly associated with “cat” than
       When people did not use imagery, they were
        faster to verify that cats had claws
            High association value, small visual part of a cat
       The higher the association value, the more
        propositions relating the two items, then the faster
        the verification time
Propositional theory
   Kosslyn (1976)
       When participants reported having used
        imagery to do the task, their reaction times
        went in the opposite direction.
            They were faster to verify visually larger parts
             with low association values than visually
             smaller parts with higher association values
            Using imagery results in performance that
             propositional theory does not predict
The significance
   It is crucial to explaining how they carry
    out a variety of cognitive tasks

   Make good predictions about when they
    use what code

   Predict when they are likely to be able
    to do things easily, and when they
Neuropsychological Findings
   Lateralization of function »
       Patients with identified lesions
       Split-brain patients
   Recent findings »
   Other findings     »
Neuropsychological Findings
   Right hemisphere
       visual memory and visual perception
            Representing and manipulating knowledge of a
             visuospatial nature
   Left hemisphere
       verbal memory and verbal comprehension
            Representing and manipulating verbal and
             other symbol-based knowledge

Neuropsychological Findings
   Both animals and humans show that
    both hemispheres may be partially
    responsible for image task performance
       Brain activation of humans   »
Brain Activation in mental rotation

Neuropsychological Findings
   Roland & Friberg (1985)
       Participants performed three cognitive
        tasks while their cerebral blood flows were
        being monitored
       Tasks were mental arithmetic, memory
        scanning of an auditory stimulus, and
        visual imagery (visualizing a walk through
        a familiar neighbourhood)

    Neuropsychological Findings
   Massive activation in the parts of the
    brain important for visual processing of
    information during the imagery task
       Mostly in the occipital lobe and other
        posterior regions

   During the other two tasks, there were no
    such increases in cerebral blood flow to
    those parts                    Continue…
Neuropsychological Findings
   Farah (1988)
       Replicated the results using ERPs ▶
            Measuring electrical activity in the brain
   Same results were found by many other
       Kosslyn & Ochsner, 1994
       Miyashita, 1995
       Kosslyn, Thompson, Kim, & Alpert, 1995

Neuropsychological Findings

                 ERPs were
                 -Occipital cortex
                 -Temporal cortex

Neuropsychological Findings
   The specific area of the occipital lobe
    showing maximal activation differed
    depending on whether the image
    created was small, medium, or large

   More findings ▶

A double dissociation

Propositional Theory
   We do not store mental representations
    in the form of images; rather our
    mental representations more closely
    resemble the abstract form of a
   A proposition ▶
Propositional Theory
   The meaning underlying a particular
    relationship among concepts

   Both images and verbal statements are
    mentally represented in terms of their
    deep meanings (as propositions), not as
    specific images or statements.

Propositional Theory
   Look at the picture ▶
       Create a proposition
            [in (dog, bathtub)]
       Predicate calculus
            Expressing the underlying meaning of a
            Describe any kind of relationship
                 Actions, attributes, positions, class memberships

Synthesizing Images and Propositions
    Mental representations may take any of the three
     forms (Johnson-Laird’s Mental Models)
        Propositions
             fully abstracted representations of meaning
        Mental models
             knowledge structures that individuals construct to understand
              and explain their experiences
        Images
             much more specific representations, which retain many of the
              perceptual features of particular objects, viewed from a
              particular angle, with the particular details of a given
Synthesizing Images and Propositions

To top