Docstoc

Probabilistic Approach to Semantic Representation

Document Sample
Probabilistic Approach to Semantic Representation Powered By Docstoc
					A Probabilistic Approach to
 Semantic Representation


        Tom Griffiths
        Mark Steyvers
       Josh Tenenbaum
• How do we store the meanings of words?
  – question of representation
  – requires efficient abstraction
• How do we store the meanings of words?
  – question of representation
  – requires efficient abstraction


• Why do we store this information?
  – function of semantic memory
  – predictive structure
              Latent Semantic Analysis
                         (Landauer & Dumais, 1997)



           co-occurrence matrix                      high dimensional space

             Doc1       Doc2      Doc3 …
 words         34        0          3      SVD           words
   in          0         12         2
semantic       5         19         6                         in
                                                                   spaces
spaces         11        6          1
                                                        semantic
  …            …         …          …



                    X                                   UDVT
       Mechanistic Claim
Some component of word meaning can be
 extracted from co-occurrence statistics
            Mechanistic Claim
   Some component of word meaning can be
    extracted from co-occurrence statistics

But…
– Why should this be true?
– Is the SVD the best way to treat these data?
– What assumptions are we making about meaning?
     Mechanism and Function
 Some component of word meaning can be
  extracted from co-occurrence statistics

Semantic memory is structured to aid retrieval
      via context-specific prediction
              Functional Claim
  Semantic memory is structured to aid retrieval
       via context-specific prediction



– Motivates sensitivity to co-occurrence statistics
– Identifies how co-occurrence data should be used
– Allows the role of meaning to be specified exactly,
  and finds a meaningful decomposition of language
     A Probabilistic Approach
• The function of semantic memory
        – The psychological problem of meaning
        – One approach to meaning

• Solving the statistical problem of meaning
        – Maximum likelihood estimation
        – Bayesian statistics

• Comparisons with Latent Semantic Analysis
        – Quantitative
        – Qualitative
     A Probabilistic Approach
• The function of semantic memory
        – The psychological problem of meaning
        – One approach to meaning

• Solving the statistical problem of meaning
        – Maximum likelihood estimation
        – Bayesian statistics

• Comparisons with Latent Semantic Analysis
        – Quantitative
        – Qualitative
 The Function of Semantic Memory
• To predict what concepts are likely to be needed
  in a context, and thereby ease their retrieval

• Similar to rational accounts of categorization
  and memory (Anderson, 1990)

• Same principle appears in semantic networks
  (Collins & Quillian, 1969; Collins & Loftus, 1975)
The Psychological Problem of Meaning

• Simply memorizing whole word-document
  co-occurrence matrix does not help

• Generalization requires abstraction, and this
  abstraction identifies the nature of meaning

• Specifying a generative model for documents
  allows inference and generalization
    One Approach to Meaning
• Each document a mixture of topics
• Each word chosen from a single topic




•          from parameters
•          from parameters
    One Approach to Meaning
w     P(w|z = 1) = f (1)   w    P(w|z = 2) = f (2)
HEART              0.2     HEART             0.0
LOVE               0.2     LOVE              0.0
SOUL               0.2     SOUL              0.0
TEARS              0.2     TEARS             0.0
JOY                0.2     JOY               0.0
SCIENTIFIC         0.0     SCIENTIFIC        0.2
KNOWLEDGE          0.0     KNOWLEDGE         0.2
WORK               0.0     WORK              0.2
RESEARCH           0.0     RESEARCH          0.2
MATHEMATICS        0.0     MATHEMATICS       0.2
     topic 1                     topic 2
            One Approach to Meaning
 Choose mixture weights for each document, generate “bag of words”
q = {P(z = 1), P(z = 2)}
                           MATHEMATICS KNOWLEDGE RESEARCH WORK MATHEMATICS
        {0, 1}                 RESEARCH WORK SCIENTIFIC MATHEMATICS WORK

                               SCIENTIFIC KNOWLEDGE MATHEMATICS SCIENTIFIC
     {0.25, 0.75}                   HEART LOVE TEARS KNOWLEDGE HEART


                               MATHEMATICS HEART RESEARCH LOVE MATHEMATICS
      {0.5, 0.5}                    WORK TEARS SOUL KNOWLEDGE HEART


     {0.75, 0.25}                    WORK JOY SOUL TEARS MATHEMATICS
                                        TEARS LOVE LOVE LOVE SOUL


        {1, 0}               TEARS LOVE JOY SOUL LOVE TEARS SOUL SOUL TEARS JOY
       One Approach to Meaning
                                              q


                                              z
• Generative model for co-occurrence data
• Introduced by Blei, Ng, and Jordan (2002)
• Clarifies pLSI (Hofmann, 1999)              w
                Matrix Interpretation
        documents                     topics
                                                        documents




                                               topics
                              words
words




           C              =            F                  Q
       normalized                mixture                 mixture
   co-occurrence matrix        components                weights



        A form of non-negative matrix factorization
               Matrix Interpretation
        documents               topics
                                                      documents




                                         topics
                        words
words




           C        =            F                          Q

        documents           vectors
                                         vectors  vectors             documents



                                                            vectors
           C        =
words




                        words




                                 U                 D                    VT
    The Function of Semantic Memory
• Prediction of needed concepts aids retrieval

• Generalization aided by a generative model

• One generative model: mixtures of topics

• Gives non-negative, non-orthogonal factorization
  of word-document co-occurrence matrix
     A Probabilistic Approach
• The function of semantic memory
        – The psychological problem of meaning
        – One approach to meaning

• Solving the statistical problem of meaning
        – Maximum likelihood estimation
        – Bayesian statistics

• Comparisons with Latent Semantic Analysis
        – Quantitative
        – Qualitative
The Statistical Problem of Meaning

• Generating data from parameters easy

• Learning parameters from data is hard

• Two approaches to this problem
  – Maximum likelihood estimation
  – Bayesian statistics
   Inverting the Generative Model
• Maximum likelihood estimation
                                  WT + DT parameters

• Variational EM (Blei, Ng & Jordan, 2002)
                                   WT + T parameters

• Bayesian inference
                                         0 parameters
         Bayesian Inference




• Sum in the denominator over Tn terms

• Full posterior only tractable to a constant
   Markov Chain Monte Carlo
• Sample from a Markov chain which
  converges to target distribution

• Allows sampling from an unnormalized
  posterior distribution

• Can compute approximate statistics
  from intractable distributions
                                       (MacKay, 2002)
              Gibbs Sampling

For variables x1, x2, …, xn
  Draw xi(t) from P(xi|x-i)

    x-i = x1(t), x2(t),…, xi-1(t), xi+1(t-1), …, xn(t-1)
Gibbs Sampling




                 (MacKay, 2002)
             Gibbs Sampling
• Need full conditional distributions for variables
• Since we only sample z we need




            number of times word w assigned to topic j

            number of times topic j used in document d
                Gibbs Sampling
                         iteration
                         1
i         wi        di   zi
 1   MATHEMATICS    1    2
 2    KNOWLEDGE     1    2
 3     RESEARCH     1    1
 4       WORK       1    2
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        ?
 2    KNOWLEDGE     1    2
 3     RESEARCH     1    1
 4       WORK       1    2
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        ?
 2    KNOWLEDGE     1    2
 3     RESEARCH     1    1
 4       WORK       1    2
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        ?
 2    KNOWLEDGE     1    2
 3     RESEARCH     1    1
 4       WORK       1    2
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        2
 2    KNOWLEDGE     1    2        ?
 3     RESEARCH     1    1
 4       WORK       1    2
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        2
 2    KNOWLEDGE     1    2        1
 3     RESEARCH     1    1        ?
 4       WORK       1    2
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        2
 2    KNOWLEDGE     1    2        1
 3     RESEARCH     1    1        1
 4       WORK       1    2        ?
 5   MATHEMATICS    1    1
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2
i         wi        di   zi       zi
 1   MATHEMATICS    1    2        2
 2    KNOWLEDGE     1    2        1
 3     RESEARCH     1    1        1
 4       WORK       1    2        2
 5   MATHEMATICS    1    1        ?
 6     RESEARCH     1    2
 7       WORK       1    2
 8     SCIENTIFIC   1    1
 9   MATHEMATICS    1    2
10       WORK       1    1
11     SCIENTIFIC   2    1
12    KNOWLEDGE     2    1
 .          .       .    .
 .          .       .    .
 .          .       .    .
50        JOY       5    2
                Gibbs Sampling
                         iteration
                         1         2   …   1000
i         wi        di   zi       zi        zi
 1   MATHEMATICS    1    2        2          2
 2    KNOWLEDGE     1    2        1          2
 3     RESEARCH     1    1        1          2
 4       WORK       1    2        2          1
 5   MATHEMATICS    1    1        2          2
 6     RESEARCH     1    2        2          2
 7       WORK       1    2        2          2
 8     SCIENTIFIC   1    1        1    …     1
 9   MATHEMATICS    1    2        2          2
10       WORK       1    1        2          2
11     SCIENTIFIC   2    1        1          2
12    KNOWLEDGE     2    1        2          2
 .          .       .    .        .          .
 .          .       .    .        .          .
 .          .       .    .        .          .
50        JOY       5    2        1          1
       A Visual Example: Bars


                    sample each pixel from
                      a mixture of topics



 pixel = word
image = document
A Visual Example: Bars
From 1000 Images
        Interpretable Decomposition




• SVD gives a basis for the data, but not an interpretable one

• The true basis is not orthogonal, so rotation does no good
      Application to Corpus Data
• TASA corpus: text from first grade to college

• Vocabulary of 26414 words

• Set of 36999 documents

• Approximately 6 million words in corpus
                    A Selection of Topics
    THEORY          SPACE         ART     STUDENTS      BRAIN      CURRENT       NATURE       THIRD
  SCIENTISTS       EARTH         PAINT    TEACHER       NERVE    ELECTRICITY      WORLD        FIRST
 EXPERIMENT         MOON         ARTIST    STUDENT      SENSE      ELECTRIC       HUMAN      SECOND
OBSERVATIONS       PLANET      PAINTING   TEACHERS     SENSES       CIRCUIT    PHILOSOPHY     THREE
  SCIENTIFIC      ROCKET        PAINTED   TEACHING        ARE          IS         MORAL      FOURTH
EXPERIMENTS         MARS        ARTISTS      CLASS    NERVOUS     ELECTRICAL   KNOWLEDGE       FOUR
 HYPOTHESIS         ORBIT      MUSEUM    CLASSROOM     NERVES      VOLTAGE      THOUGHT       GRADE
    EXPLAIN    ASTRONAUTS        WORK       SCHOOL      BODY         FLOW        REASON        TWO
   SCIENTIST        FIRST     PAINTINGS   LEARNING      SMELL      BATTERY        SENSE       FIFTH
  OBSERVED     SPACECRAFT        STYLE       PUPILS     TASTE        WIRE           OUR     SEVENTH
EXPLANATION       JUPITER     PICTURES    CONTENT      TOUCH         WIRES        TRUTH       SIXTH
     BASED       SATELLITE       WORKS  INSTRUCTION   MESSAGES      SWITCH       NATURAL     EIGHTH
OBSERVATION     SATELLITES        OWN      TAUGHT     IMPULSES    CONNECTED     EXISTENCE      HALF
      IDEA     ATMOSPHERE    SCULPTURE       GROUP       CORD     ELECTRONS        BEING      SEVEN
   EVIDENCE     SPACESHIP       PAINTER     GRADE      ORGANS     RESISTANCE        LIFE        SIX
   THEORIES       SURFACE         ARTS     SHOULD      SPINAL       POWER          MIND      SINGLE
   BELIEVED     SCIENTISTS   BEAUTIFUL     GRADES       FIBERS   CONDUCTORS     ARISTOTLE     NINTH
 DISCOVERED    ASTRONAUT       DESIGNS     CLASSES    SENSORY      CIRCUITS     BELIEVED        END
   OBSERVE        SATURN      PORTRAIT        PUPIL      PAIN        TUBE      EXPERIENCE     TENTH
     FACTS          MILES      PAINTERS      GIVEN         IS      NEGATIVE      REALITY    ANOTHER
                     A Selection of Topics
     DISEASE                 MIND       STORY     FIELD     SCIENCE      BALL         JOB
                 WATER
    BACTERIA                WORLD     STORIES  MAGNETIC      STUDY       GAME        WORK
                   FISH
     DISEASES               DREAM        TELL   MAGNET    SCIENTISTS     TEAM        JOBS
                   SEA
      GERMS                 DREAMS   CHARACTER    WIRE    SCIENTIFIC FOOTBALL       CAREER
                  SWIM
       FEVER   SWIMMING    THOUGHT CHARACTERS    NEEDLE  KNOWLEDGE BASEBALL EXPERIENCE
       CAUSE      POOL   IMAGINATION  AUTHOR    CURRENT      WORK      PLAYERS EMPLOYMENT
     CAUSED                MOMENT       READ       COIL   RESEARCH       PLAY   OPPORTUNITIES
                  LIKE
      SPREAD              THOUGHTS       TOLD     POLES   CHEMISTRY      FIELD     WORKING
                 SHELL
     VIRUSES                  OWN     SETTING      IRON  TECHNOLOGY PLAYER         TRAINING
                 SHARK
    INFECTION                REAL       TALES   COMPASS      MANY    BASKETBALL     SKILLS
                  TANK
       VIRUS                  LIFE       PLOT     LINES  MATHEMATICS COACH         CAREERS
                SHELLS
MICROORGANISMS SHARKS      IMAGINE    TELLING     CORE     BIOLOGY     PLAYED     POSITIONS
      PERSON                 SENSE     SHORT    ELECTRIC     FIELD     PLAYING       FIND
                 DIVING
                                               DIRECTION    PHYSICS       HIT      POSITION
   INFECTIOUS  DOLPHINS CONSCIOUSNESS FICTION
     COMMON                STRANGE     ACTION    FORCE   LABORATORY     TENNIS       FIELD
                 SWAM
     CAUSING                FEELING      TRUE   MAGNETS     STUDIES     TEAMS    OCCUPATIONS
                  LONG
    SMALLPOX                WHOLE      EVENTS       BE      WORLD       GAMES      REQUIRE
                  SEAL
       BODY                  BEING      TELLS  MAGNETISM SCIENTIST      SPORTS   OPPORTUNITY
                  DIVE
   INFECTIONS                MIGHT       TALE      POLE   STUDYING        BAT        EARN
                DOLPHIN
     CERTAIN                 HOPE      NOVEL    INDUCED    SCIENCES     TERRY        ABLE
              UNDERWATER
                     A Selection of Topics
     DISEASE                 MIND       STORY     FIELD     SCIENCE      BALL          JOB
                 WATER
    BACTERIA                WORLD     STORIES  MAGNETIC      STUDY       GAME        WORK
                   FISH
     DISEASES               DREAM        TELL   MAGNET    SCIENTISTS     TEAM         JOBS
                   SEA
      GERMS                 DREAMS   CHARACTER    WIRE    SCIENTIFIC FOOTBALL       CAREER
                  SWIM
       FEVER   SWIMMING    THOUGHT CHARACTERS    NEEDLE  KNOWLEDGE BASEBALL EXPERIENCE
       CAUSE      POOL   IMAGINATION  AUTHOR    CURRENT      WORK      PLAYERS EMPLOYMENT
     CAUSED                MOMENT       READ       COIL   RESEARCH       PLAY   OPPORTUNITIES
                  LIKE
      SPREAD              THOUGHTS       TOLD     POLES   CHEMISTRY      FIELD     WORKING
                 SHELL
     VIRUSES                  OWN     SETTING      IRON  TECHNOLOGY PLAYER         TRAINING
                 SHARK
    INFECTION                REAL       TALES   COMPASS      MANY    BASKETBALL     SKILLS
                  TANK
       VIRUS                  LIFE       PLOT     LINES  MATHEMATICS COACH         CAREERS
                SHELLS
MICROORGANISMS SHARKS      IMAGINE    TELLING     CORE     BIOLOGY     PLAYED     POSITIONS
      PERSON                 SENSE     SHORT    ELECTRIC     FIELD     PLAYING        FIND
                 DIVING
                                               DIRECTION    PHYSICS       HIT      POSITION
   INFECTIOUS  DOLPHINS CONSCIOUSNESS FICTION
     COMMON                STRANGE     ACTION    FORCE   LABORATORY     TENNIS       FIELD
                 SWAM
     CAUSING                FEELING      TRUE   MAGNETS     STUDIES     TEAMS    OCCUPATIONS
                  LONG
    SMALLPOX                WHOLE      EVENTS       BE      WORLD       GAMES      REQUIRE
                  SEAL
       BODY                  BEING      TELLS  MAGNETISM SCIENTIST      SPORTS   OPPORTUNITY
                  DIVE
   INFECTIONS                MIGHT       TALE      POLE   STUDYING        BAT        EARN
                DOLPHIN
     CERTAIN                 HOPE      NOVEL    INDUCED    SCIENCES     TERRY        ABLE
              UNDERWATER
     A Probabilistic Approach
• The function of semantic memory
        – The psychological problem of meaning
        – One approach to meaning

• Solving the statistical problem of meaning
        – Maximum likelihood estimation
        – Bayesian statistics

• Comparisons with Latent Semantic Analysis
        – Quantitative
        – Qualitative
        Probabilistic Queries
•      can be computed in different ways

• Fixed topic assumption:

• Multiple samples:
     Quantitative Comparisons
• Two types of task
  – general semantic tasks: dictionary, thesaurus
  – prediction of memory data


• All tests use LSA with 400 vectors, and a
  probabilistic model with 100 samples each
  using 500 topics
            Fill in the Blank
• 12856 sentences extracted from WordNet
         his cold deprived him of his sense of _
         silence broken by dogs barking _
         a _ hybrid accent

• Overall performance
  – LSA gives median rank of 3393
  – Probabilistic model gives median rank of 3344
Fill in the Blank
                               Synonyms
    • 280 sets of five synonyms from WordNet,
      ordered by number of senses
BREAK (78)        EXPOSE (9)         DISCOVER (8)         DECLARE (7)       REVEAL (3)

CUT (72) REDUCE (19)        CONTRACT (12)        SHORTEN (5)       ABRIDGE (1)

RUN (53)          GO (34)            WORK (25)            FUNCTION (9)      OPERATE (7)

    • Two tasks:
           – Predict first synonym
           – Predict last synonym
    • Increasing number of synonyms
First Synonym
Last Synonym
Synonyms and Word Frequency
    Synonyms and Word Frequency

Probabilistic




   LSA
    Synonyms and Word Frequency

Probabilistic




   LSA
  Word Frequency and Filling Blanks
Probabilistic                  LSA
 Performance on Semantic Tasks
• Performance comparable, neither great

• Difference in effects of word frequency due
  to treatment of co-occurrence data

• Probabilistic approach useful in addressing
  psychological data: frequency important
         Intrusions in Free Recall
CHAIR
FOOD      • Intrusion rates from Deese (1959)
DESK
TOP
LEG       • Used average word vectors in LSA,
EAT         P(word|list) in probabilistic model
CLOTH
DISH
WOOD      • Favors LSA, since probabilistic
DINNER
MARBLE      combination can be multimodal
TENNIS
Intrusions in Free Recall
Intrusions in Free Recall




   models       word frequency
  Word Frequency is Not Enough

• An explanation needs to address two questions:
  – Why do these words intrude?
  – Why do other words not intrude?
  Word Frequency is Not Enough

• An explanation needs to address two questions:
  – Why do these words intrude?
  – Why do other words not intrude?


• Median word frequency rank: 1698.5
• Median rank in model: 21
              Word Association
• Word association norms from Nelson et al. (1998)
                             PLANETS

          associate number     people     model

                 1             EARTH      STARS
                 2             STARS       STAR
                 3             SPACE       SUN
                 4              SUN       EARTH
                 5             MARS       SPACE
                 6           UNIVERSE      SKY
                 7            SATURN     PLANET
                 8           GALAXY     UNIVERSE
Word Association
 Performance on Memory Tasks

• Outperforms LSA on simple memory tasks,
  both far better at predicting memory data

• Improvement due to role of word frequency

• Not a complete account, but can form a part
  of more complex memory models
        Qualitative Comparisons
• Naturally deals with complications for LSA
  – Polysemy
  – Asymmetry


• Respects natural statistics of language

• Easily extends to other models of meaning
    Beyond the Bag of Words

      q

z     z    z


w     w   w
    Beyond the Bag of Words

      q               q

z     z    z      z    z      z


w     w   w       w   w       w


                  s    s      s
                     Semantic categories
      FOOD       MAP        DOCTOR       BOOK      GOLD      BEHAVIOR      CELLS     PLANTS
     FOODS     NORTH         PATIENT     BOOKS     IRON         SELF        CELL      PLANT
      BODY     EARTH         HEALTH    READING    SILVER    INDIVIDUAL ORGANISMS     LEAVES
   NUTRIENTS   SOUTH        HOSPITAL INFORMATION COPPER   PERSONALITY     ALGAE       SEEDS
      DIET      POLE        MEDICAL    LIBRARY    METAL      RESPONSE   BACTERIA       SOIL
       FAT      MAPS          CARE      REPORT   METALS        SOCIAL  MICROSCOPE     ROOTS
     SUGAR    EQUATOR       PATIENTS      PAGE    STEEL     EMOTIONAL   MEMBRANE    FLOWERS
    ENERGY      WEST          NURSE      TITLE     CLAY      LEARNING   ORGANISM      WATER
      MILK      LINES       DOCTORS    SUBJECT     LEAD       FEELINGS     FOOD       FOOD
     EATING     EAST       MEDICINE      PAGES    ADAM   PSYCHOLOGISTS    LIVING      GREEN
     FRUITS  AUSTRALIA      NURSING      GUIDE      ORE    INDIVIDUALS     FUNGI       SEED
  VEGETABLES   GLOBE      TREATMENT     WORDS   ALUMINUM PSYCHOLOGICAL     MOLD       STEMS
     WEIGHT    POLES         NURSES    MATERIAL  MINERAL   EXPERIENCES MATERIALS     FLOWER
      FATS   HEMISPHERE    PHYSICIAN    ARTICLE    MINE   ENVIRONMENT    NUCLEUS       STEM
     NEEDS    LATITUDE     HOSPITALS   ARTICLES   STONE        HUMAN      CELLED       LEAF
CARBOHYDRATES PLACES            DR       WORD   MINERALS     RESPONSES STRUCTURES   ANIMALS
    VITAMINS    LAND           SICK      FACTS      POT      BEHAVIORS  MATERIAL      ROOT
   CALORIES    WORLD       ASSISTANT    AUTHOR   MINING      ATTITUDES STRUCTURE     POLLEN
    PROTEIN   COMPASS     EMERGENCY   REFERENCE  MINERS   PSYCHOLOGY      GREEN     GROWING
   MINERALS  CONTINENTS    PRACTICE       NOTE      TIN        PERSON     MOLDS       GROW
                    Syntactic categories
    SAID     THE       MORE         ON        GOOD        ONE         HE         BE
   ASKED      HIS       SUCH        AT       SMALL      SOME         YOU       MAKE
 THOUGHT    THEIR       LESS       INTO       NEW       MANY        THEY        GET
   TOLD     YOUR       MUCH       FROM    IMPORTANT      TWO           I       HAVE
    SAYS     HER      KNOWN        WITH      GREAT       EACH        SHE         GO
  MEANS       ITS       JUST    THROUGH      LITTLE       ALL        WE        TAKE
  CALLED      MY      BETTER       OVER      LARGE      MOST          IT         DO
   CRIED     OUR      RATHER     AROUND         *        ANY       PEOPLE      FIND
  SHOWS      THIS    GREATER    AGAINST        BIG      THREE     EVERYONE      USE
ANSWERED    THESE     HIGHER     ACROSS       LONG       THIS      OTHERS       SEE
   TELLS       A      LARGER       UPON       HIGH      EVERY    SCIENTISTS    HELP
  REPLIED     AN      LONGER     TOWARD    DIFFERENT   SEVERAL    SOMEONE      KEEP
 SHOUTED     THAT     FASTER      UNDER     SPECIAL      FOUR       WHO        GIVE
EXPLAINED    NEW     EXACTLY      ALONG        OLD       FIVE      NOBODY      LOOK
 LAUGHED    THOSE    SMALLER       NEAR     STRONG       BOTH        ONE       COME
  MEANT     EACH    SOMETHING    BEHIND      YOUNG        TEN    SOMETHING     WORK
  WROTE       MR      BIGGER        OFF     COMMON        SIX      ANYONE      MOVE
 SHOWED      ANY       FEWER      ABOVE      WHITE      MUCH     EVERYBODY      LIVE
 BELIEVED    MRS      LOWER       DOWN       SINGLE    TWENTY       SOME        EAT
WHISPERED     ALL    ALMOST      BEFORE     CERTAIN     EIGHT       THEN      BECOME
           Sentence generation
RESEARCH:
[S] THE CHIEF WICKED SELECTION OF RESEARCH IN THE BIG MONTHS
[S] EXPLANATIONS
[S] IN THE PHYSICISTS EXPERIMENTS
[S] HE MUST QUIT THE USE OF THE CONCLUSIONS
[S] ASTRONOMY PEERED UPON YOUR SCIENTISTS DOOR
[S] ANATOMY ESTABLISHED WITH PRINCIPLES EXPECTED IN BIOLOGY
[S] ONCE BUT KNOWLEDGE MAY GROW
[S] HE DECIDED THE MODERATE SCIENCE

LANGUAGE:
[S] RESEARCHERS GIVE THE SPEECH
[S] THE SOUND FEEL NO LISTENERS
[S] WHICH WAS TO BE MEANING
[S] HER VOCABULARIES STOPPED WORDS
[S] HE EXPRESSLY WANTED THAT BETTER VOWEL
            Sentence generation
LAW:
[S] BUT THE CRIME HAD BEEN SEVERELY POLITE OR CONFUSED
[S] CUSTODY ON ENFORCEMENT RIGHTS IS PLENTIFUL

CLOTHING:
[S] WEALTHY COTTON PORTFOLIO WAS OUT OF ALL SMALL SUITS
[S] HE IS CONNECTING SNEAKERS
[S] THUS CLOTHING ARE THOSE OF CORDUROY
[S] THE FIRST AMOUNTS OF FASHION IN THE SKIRT
[S] GET TIGHT TO GET THE EXTENT OF THE BELTS
[S] ANY WARDROBE CHOOSES TWO SHOES

THE ARTS:
[S] SHE INFURIATED THE MUSIC
[S] ACTORS WILL MANAGE FLOATING FOR JOY
[S] THEY ARE A SCENE AWAY WITH MY THINKER
[S] IT MEANS A CONCLUSION
                   Conclusion
Taking a probabilistic approach can clarify some
of the central issues in semantic representation



– Motivates sensitivity to co-occurrence statistics
– Identifies how co-occurrence data should be used
– Allows the role of meaning to be specified exactly,
  and finds a meaningful decomposition of language
 Probabilities and Inner Products
• Single word:            w



                              F
• List of words:
            Model Selection
• How many topics does a language contain?

• Major issue for parametric models

• Not so much for non-parametric models
  – Dirichlet process mixtures
  – Expect more topics than tractable
  – Choice of number is choice of scale
      Gibbs Sampling and EM
• How many topics does a language contain?

• EM finds fixed set of topics, single estimate

• Sampling allows for multiple sets of topics,
  and multimodal posterior distributions
           Natural Statistics
• Treating co-occurrence data as frequencies
  preserves the natural statistics of language

• Word frequency

• Zipf’s Law of Meaning
Natural Statistics
Natural Statistics
Natural Statistics
  Word Association
          CROWN

 people            model

  KING              KING
 JEWEL             TEETH
 QUEEN              HAIR
 HEAD              TOOTH
  HAT             ENGLAND
  TOP              MOUTH
 ROYAL             QUEEN
THRONE             PRINCE
   Word Association
            SANTA

  people               model

CHRISTMAS             MEXICO
  TOYS               SPANISH
   LIE              CALIFORNIA

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:6
posted:11/30/2011
language:English
pages:90