Documents
Resources
Learning Center
Upload
Plans & pricing Sign in
Sign Out

Pattern Recognition - PowerPoint

VIEWS: 72 PAGES: 84

									Reading in Speech and
Language Processing


         Esther Levin
  Dept of Computer Science
            CCNY
Credits and Acknowledgments
  Some of the materials used in this course were
 taken from
    Dr. Mazin Rahim, AT&T
    Dr. Massimo Poesio, The University of Essex, UK
    Dr. Michael McTear , University of Ulster, UK
    Dr. Larry Rabiner, Rutgers
    Dr. Julia Hirschberg, Columbia University
    Dr. Claire Cardie, Cornell University
    Dr. Eric Atwell's, University of Leeds, UK
    Dr. Anoop Sarkar, SFU
    Dr. Jason Eisner, JHU
    Dr. Mary P. Harper, Purdue University
    David G. Stork, Stanford University
                 Outline
Preliminaries
Introduction
   What is this course about
Background Material
 Probability theory
 Information theory
           WHY DID YOU TAKE THIS COURSE?
• Introduce yourself:
    Name
    Degree Program
    Research Interests
    Why did you take this course?
• Example:
   Esther Levin
   Ph.D. from Technion- Israely Institute of Technology in
    1988
   Neural Networks, Speech Recognition, Machine
    Learning, Spoken Language Understanding and
    Spoken Dialog Systems.
             ONLINE RESOURCES
The course website is available at:

http://www-cs.engr.ccny.cuny.edu/~esther/ReadingSNLP
Natural language and NLP
―natural‖ language
   Languages that people use to communicate with
    one another
Ultimate goal
   To build computer systems that perform as well at
    using natural language as humans do
Immediate goal
   To build computer systems that can process text
    and speech more intelligently
        Speech/language understanding
        Language/speech generation
           Goals of the field
Computers would be a lot more useful if they
 could handle our email, do our library
 research, talk to us …

But they are fazed by natural human language.

How can we tell computers about language?
 (Or help them learn it as kids do?)
        Goals of the course
  Introduce you to NLP problems &
  solutions
  Relation to linguistics & statistics
At the end you should:
   Agree that language is subtle & interesting
   Know about several applications of SNLP

   Understand research papers in the field
Natural Language Engineering in 2001:
         Where we should be
 Amer. Good afternoon, Hal. How's everything going?
 Hal. Good afternoon, Mr Amer. Everything is going extremely well.
 Amer. Hal, you have an enormous responsibility on this mission, in
 many ways perhaps the greatest responsibility of any single mission
 element. You are the brain and central nervous system of the ship, and
 your responsibilities include watching over the men in hibernation.
 Does this ever cause you any - lack of confidence?
 Hal. Let me put it this way, Mr Amer. The 9000 series is the most
 reliable computer ever made. No 9000 computer has ever made a
 mistake or distorted information. We are all, by any practical definition
 of the words, foolproof and incapable of error.
 Amer. Hal, despite your enormous intellect, are you ever frustrated by
 your dependence on people to carry out actions?
 Hal. Not in the slightest bit. I enjoy working with people. I have a
 stimulating relationship with Dr Poole and Dr Bowman. My mission
 responsibilities range over the entire operation of the ship, so I am
 constantly occupied. I am putting myself to the fullest possible use,
 which is all, I think, that any conscious entity can ever hope to do.
 How This Course Works?
The course will consist of
  lectures by the instructor
 talks by a few invited speakers from both
  academia and industry doing active
  research in Speech and Natural Language
  Processing
 reading presentations by students
     Student’s Presentation
Student presenter leads discussion of a set of
papers/book chapters on a given topic.
The reading material is partially assigned by the
instructor, and partially researched by the student.
The remaining students will email one or more
questions per paper to the presenter, by 9 am in the
morning of the class.
   These questions should be the kind of questions that you
    would ask if you heard the contents of the paper in a talk, or
    were reviewing the paper.
The presenter will email the collection of questions to
everyone, which you should read (and print out)
before class.
                Grading
In addition to leading one or more class
discussions, all students will be expected to
do all the readings, and send the email
questions as well as participate in the other
discussions. Attendance is compulsory and
absence will be penalized.
Grade Basis: email questions (20%), class
participation (20%), leading 1 or 2 classes
(60%),
                    Text books
"Foundations of Statistical Natural Language Processing" by Manning
& Schütze.

SPEECH and LANGUAGE PROCESSING: An Introduction to Natural
Language Processing, Computational Linguistics, and Speech
Recognition, by D. Jurafsky and J.H. Martin,

Spoken Language Processing - A Guide to Theory, Algorithm, and
System Development, by X. Huang, A. Acero, and H.W. Hon.

Spoken Dialog Technology: towards the conversational user interface,
by Michael F. McTear

Fundamentals of Speech Recognition, by L.R. Rabiner and B.W.
Juang,.
Topics
           1. Machine Translation
   •    M&S- 13; J&M- 21


Machine translation can use a method based on linguistic rules,
which means that words will be translated in a linguistic way — the
most suitable (orally speaking) words of the target language will
replace the ones in the source language.
It is often argued that the success of machine translation requires the
problem of natural language understanding to be solved first.
(wikpedia)
         1. Machine Translation
   •   M&S- 13; J&M- 21


Машинный перевод может использовать метод основанный на
лингвистических правилах, который намеревается что слова
будут переведены в лингвистической дороге - самые
целесообразные (устно говоря) слова целевого языка заменат
одни в исходном языке. Часто поспорено что успех машинного
перевода требует, что проблема вникания естественного языка
разрешена сперва. (Yahoo Babel fish English to Russian)
           1. Machine Translation
   •    M&S- 13; J&M- 21


Machine transfer can use the method based on the linguistic rules,
which will intend that word will be transferred in the linguistic road
- most expedient (orally speaking) words of purposeful language
zamenat some in the source language. Frequently posporeno that the
success of machine transfer requires, that the problem of the
understanding of natural language is solved first.
(Yahoo Babel fish Russian to English)
   2. Information Retrieval

M&S-15; J&M-17
      3. Text Categorization
M&S-16
Automatic Yahoo classification, etc.
 • Topic 1 sample: In the beginning God created …
 • Topic 2 sample: The history of all hitherto existing society is
   the history of class struggles. …
Input text: Matt’s Communist Homepage. Capitalism is unfair
and has been ruining the lives of millions of people around the
world. The profits from the workers’ labor …
Input text: And they have beat their swords to ploughshares,
And their spears to pruning-hooks. Nation doth not lift up sword
unto nation, neither do they learn war any more. …
            4. Collocations
M&S -5
techniques for identifying constructions and
multiword expressions in text. The typical
construction is a multiword unit like ―take a
hike; take five‖ where the meaning of the
multiword unit cannot be determined from the
meaning of the words which make it up.
   Go broke
   Kicked the bucket
5. Word Sense Disambiguation

 M&S-7
   Bank – river or financial institution
   keep
       He kept eating
       He kept calm
       He kept a record
 Important for machine translation,
 information retrieval and other apps
6. Named Entity Recognition
Subtask of Information Extraction
Locate and classify elements in text into predefined
categories
    names of persons, organizations, locations, expressions of
    times, quantities, monetary values, percentages, etc.
For example, a NER system producing MUC-style
output might tag the sentence,
   Jim bought 300 shares of Acme Corp. in 2006.
   <ENAMEX TYPE="PERSON">Jim</ENAMEX> bought
    <NUMEX TYPE="QUANTITY">300</NUMEX> shares of
    <ENAMEX TYPE="ORGANIZATION">Acme
    Corp.</ENAMEX> in <TIMEX
    TYPE="DATE">2006</TIMEX>.
7. Spoken Language Structure
HA&H -2
Phonemes, syllables, words
Syntax and semantics
8 (9?) Hidden Markov Models
    and Speech Recognition
(M&S-9; J&M –7)
Any given day, the weather
can be described as being
in one of three states:
   State 1: precipitation (rain,
    snow, hail, etc.)
   State 2: cloudy
   State 3: sunny
What is Speech Recognition?
Speech recognition is the process of
converting an acoustic signal, captured by a
microphone or a telephone, to a set of words.
The recognised words can be used as
    final results e.g. applications such as commands &
     control, data entry, and document preparation
    input to further linguistic processing in order to
     achieve speech understanding.
           9. Discourse
J&M –18
analysis of language in the context of
text, i.e., related group of sentences:
monologue, dialogue, HCI.
   Reference resolution.
   Text Coherence.
10. Dialogue and Conversational
              Agents
J&M-19
the fundamental structures and algorithms in
conversational agents, also known as
spoken dialogue systems, or spoken
language systems.
      AT&T Communicator
AT&T Communicator… How can I help you?
From Washington DC to Denver roundtrip.
Flying from Washington to Denver, roundtrip flight. O.k., and
what date do you want to leave?
Next Friday, in the morning.
Leaving April 27-th …There is an outbound flight from …
6: 35 am …, would you like me to hold that flight?
No, do you have a later flight?
O.k.… A later flight is with AA departing at 3:32 pm … and
arriving Denver at 8:02 pm. Would you like me to hold that
flight?
Sure.
O.k., and what date do you want to return?
…
     11. Spoken Language
          Understanding
HA&H – 17
Suppose recognizer transcribes
correctly acoustic signal into sequence
of words
Mapping between transcription and
what the system is supposed to do.
12. Spoken Dialog Management

 McTear – 4,13
 Functional Components of Dialog
 Systems, knowledge representation,
 learning optimal dialog strategies
         Other Topics:

Summarization
Question Answering,
Multi Modal HCI
…
Why is NLP such a difficult problem?
Ambiguity!!!! …at all levels of analysis


 Phonetics and phonology
     Concerns how words are related to the sounds that realize them
     Important for speech-based systems.
          » "I scream" vs. "ice cream"
          » "nominal egg"
     Moral is:
          It's very hard to recognize speech.
          It's very hard to wreck a nice beach.
 Morphology
     Concerns how words are constructed from sub-word units
          Unionized
          un-ionized in chemistry?
Why is NLP such a difficult problem?
Ambiguity!!!! …at all levels of analysis


 Syntax
      Concerns sentence structure
     Different syntactic structure implies different
      interpretation
          Squad helps dog bite victim.
                [np squad] [vp helps [np dog bite victim]
                [np squad] [vp helps [np dog] [inf-clause bite victim]]
          Helicopter powered by human flies.
          Visiting relatives can be trying.
Why is NLP such a difficult problem?
Ambiguity!!!! …at all levels of analysis



 Semantics
     Concerns what words mean and how these
      meanings combine to form sentence meanings.
          Jack invited Mary to the Halloween ball.
                dance vs. some big sphere with Halloween decorations?
          Visiting relatives can be trying.
          Visiting museums can be trying.
                Same set of possible syntactic structures for this sentence
                But the meaning of museums makes only one of them
                 plausible
Why is NLP such a difficult problem?
Ambiguity!!!! …at all levels of analysis



 Discourse
     – Concerns how the immediately preceding sentences affect
      the interpretation of the next sentence
          » Merck & Co. formed a joint venture with Ache Group, of
           Brazil. It will be called Prodome Ltd.
          » Merck & Co. formed a joint venture with Ache Group, of
           Brazil. It will own 50% of the new company to be called
           Prodome Ltd.
          » Merck & Co. formed a joint venture with Ache Group, of
           Brazil. It had previously teamed up with Merck in two
           unsuccessful pharmaceutical ventures.
Why is NLP such a difficult problem?
Ambiguity!!!! …at all levels of analysis



 Pragmatics
     – Concerns how sentences are used in different
      situations and how use affects the interpretation of
      the sentence.
          ``I just came from New York.'‘

          » Would you like to go to New York today?
          » Would you like to go to Boston today?
          » Why do you seem so out of it?
          » Boy, you look tired.
Ambiguity: Favorite Headlines
Iraqi Head Seeks Arms
Is There a Ring of Debris Around Uranus?
Juvenile Court to Try Shooting Defendant
Teacher Strikes Idle Kids
Stolen Painting Found by Tree
Kids Make Nutritious Snacks
Local HS Dropouts Cut in Half
Obesity Study Looks for Larger Test Group
Ambiguity: Favorite Headlines
British Left Waffles on Falkland Islands
Never Withhold Herpes Infection from
Loved One
Red Tape Holds Up New Bridges
Man Struck by Lightning Faces Battery
Charge
Clinton Wins on Budget, but More Lies
Ahead
Hospitals Are Sued by 7 Foot Doctors
      Levels of Language
Phonetics/phonology/morphology: what
words (or subwords) are we dealing
with?
Syntax: What phrases are we dealing
with? Which words modify one
another?
Semantics: What’s the literal meaning?
Pragmatics: What should you conclude
from the fact that I said something?
How should you react?
      Subtler Ambiguity
Q: Why does my high school give me a
suspension for skipping class?

A: Administrative error. They’re
supposed to give you a suspension for
auto shop, and a jump rope for skipping
class. (*rim shot*)
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

To get a donut (spare tire) for his car?
What’s hard about this story?
John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

store where donuts shop? or is run by
  donuts?
or looks like a big donut? or made of
  donut?
or has an emptiness at its core?
What’s hard about this story?
I stopped smoking freshman year, but
John stopped at the donut store on his
   way home from work. He thought a
   coffee was good every few hours. But it
   turned out to be too expensive there.
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

Describes where the store is? Or when
 he stopped?
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

Well, actually, he stopped there from
 hunger and exhaustion, not just from
 work.
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

At that moment, or habitually?
  (Similarly: Mozart composed music.)
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

That’s how often he thought it?
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

But actually, a coffee only stays good for
 about 10 minutes before it gets cold.
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

Similarly: In America a woman has a baby
  every 15 minutes. Our job is to find that
  woman and stop her.
What’s hard about this story?

John stopped at the donut store on his
  way home from work. He thought a
  coffee was good every few hours. But it
  turned out to be too expensive there.

the particular coffee that was good every
  few hours? the donut store? the
  situation?
What’s hard about this story?
 John stopped at the donut store on his
 way home from work. He thought a
 coffee was good every few hours. But it
 turned out to be too expensive there.

too expensive for what? what are we
  supposed to conclude about what John
  did?
how do we connect ―it‖ to ―expensive‖?
Difficulties in NLP (cont.)
   Ambiguity
        books: NOUN or VERB?
   you need many books vs. she books her flights online
   No left turn weekdays 4-6 pm except transit vehicles
    When may transit vehicles turn: Always? Never?
        Thank you for not smoking, drinking, eating or playing
         radios without earphones. (MTA bus)
        Thank you for not eating without earphones??
   Thank you for drinking?? …
        Fred’s hat was blown off by the wind. He tried to catch
         it.
        ...catch the wind or ...catch the hat ?
         Rules or Statistics?

   Preferences:
            context clues: she books  books is a verb
   rule: if an ambiguous word (verb/nonverb) is preceded by a
    matching personal pronoun  word is a verb
            pronoun reference:
   she/he/it often refers to the most recent noun or pronoun (but
    there are certainly exceptions)
            selectional restrictions:
   catching hat is better than catching wind (but not always)
            semantics:
            We thank people for doing helpful things or not doing annoying
             things
        More Examples
Fred saw the dog with his binoculars.
I saw the Golden Gate Bridge flying into
San Francisco. Avoid marking shoes
Can companies litter the environment
(spoken--no final punctuation)
Every man saw the boy with his
binoculars.
     Types of Knowledge

Acoustic/Phonetic Knowledge: How words
are related to their sounds.

Morphological Knowledge: How words are
constructed out of basic meaning units. un +
friend + ly  unfriendlylove + past tense 
lovedobject + oriented  object-oriented
 More Types of Knowledge
Lexical Knowledge (or Dictionary):
This should include information on parts
of speech, features (e.g., number,
case), typical usage, and word
meaning.
Syntactic Knowledge: How words are
put together to make legal sentences
(or constituents of sentences).
         Syntactic Analysis
Rules:                Lexicon:
   S  NP VP            Jack: Propernoun
   NP  Propernoun      saw, slept: Verb
   NP  Det Noun        on, with: Prep
   NP  NP PP           every, his, the: Det
   VP  Verb NP         binoculars, boy,
   VP  Verb PP          man, table: Noun
   VP  Verb NP PP
   PP  Prep NP
Parse tree for Jack slept on the table.




                                      63
Trees




        64
 More Types of Knowledge
Semantic Knowledge: Word meanings, how
words combine into sentence meaning, e.g.,
     Fred runs. maps to: (run r2 (inst r2 event) (agent
    r2 Fred)),
combining words into a sentence affects
sentence and word meaning.
    Examples:
        Fred broke the window with the block.
        Fred broke the window with Nadia.
Syntax-Directed Mapping to
        Semantics




                             66
More Types of Knowledge
   Pragmatic Knowledge: How context
    affects the interpretation of a sentence.
    Examples:
        Louise loves him.
             [Context 1:] Who loves Fred?
             [Context 2:] Louise has a cat.
        What time is it?
             [Context 1:] Fred is fidgeting and staring at his
              watch.
             [Context 2:] Louise has no watch.
 More Types of Knowledge
World Knowledge: How other people's
minds work, what a listener knows or
believes, the etiquette of language.
Examples:
 Will you pass the salt?
 I read an article about the war in the paper.
 Fred saw the bird with his binoculars.
 Tim was invited to Tom's birthday party. He
  went to the store to buy him a present.
      Types of Ambiguity
Lexical: cook in Fred will cook the chicken.
versus The cook chased the chicken.
Syntactic: Fred saw the bird in the nest with
the binoculars.
Semantic: Every man showed a boy his
paper.
Pragmatic: Situation affects the meaning of
every sentence!
Ambiguity of the Day(From "The
        Two Ronnies")
Ronnie Corbett: In the sketch that
follows, I'll be playing a man whose wife
leaves him for the garbage man.

Ronnie Barker: And I'll be playing the
garbage man who refuses to take him
                       Statistical NLP
• Imagine:
  – Each sentence W = { w1, w2, ..., wn } gets a probability
    P(W|X) in a context X (think of it in the intuitive sense
    for now)
  – For every possible context X, sort all the imaginable
    sentences W according to P(W|X):
  – Ideal situation:
             best sentence (most probable in context X)   NB: same for
                                                          interpretation
    P(W)                                 ―ungrammatical‖ sentences
                Real World Situation
• Unable to specify a set of grammatical sentences using fixed
  ―categorical‖ rules
• Use statistical ―model‖ based on REAL WORLD DATA and
  care about the best sentence only (disregarding the
  ―grammaticality‖ issue)

              best sentence
P(W)




   Wbest                                                    Wworst
                      Caveat
NLP has an AI-ish flavor to it.
     We’re often dealing with ill-defined problems
     We don’t often come up with perfect
      solutions/algorithms
     We can’t let either of those facts get in our way
  If this bothers you, you should drop.
Early Roots: 1940’s and 1950’s
Work on two foundational paradigms
   –Automaton
        Turing’s (1936) model of algorithmic computation
        Kleene’s (1951, 1956) finite automata and regular expressions

   Shannon (1948) applied probabilistic models of discrete
    Markov processes to automata for language
   Chomsky (1956)
        First considered finite-state machines as a way to characterize
         a grammar
        Led to the field of formal language theory
Early Roots: 1940’s and 1950’s

Work on two foundational paradigms
   Probabilistic or information-theoretic
    models for speech and language
    processing
      Shannon: the ―noisy channel‖ model
      Shannon: borrowing of ―entropy‖ from
       thermodynamics to measure the information
       content of a language
    Two Camps: 1957-1970

Symbolic paradigm
   Chomsky
     Formal language theory, generative syntax,
      parsing
     Linguists and computer scientists

     Earliest complete parsing systems
           Zelig Harris, UPenn
    Two Camps: 1957-1970

Symbolic paradigm
   Artificial intelligence
   Created in the summer of 1956
        Two-month workshop at Dartmouth
        Focus of the field initially was the work on reasoning and
         logic (Newell and Simon)
   Early natural language systems were built
        Worked in a single domain
        Used pattern matching and keyword search
    Two Camps: 1957-1970

Stochastic paradigm
 Took hold in statistics and EE
 Late 50’s: applied Bayesian methods to
  OCR
 Mosteller and Wallace (1964): applied
  Bayesian methods to the problem of
  authorship attribution for The Federalist
  papers.
    Additional Developments

1960’s
    First serious testable psychological models of human
    language processing
        Based on transformational grammar
   First on-line corpora
        The Brown corpus of American English
              1 million word collection
              Samples from 500 written texts
              Different genres (news, novels, non-fiction, academic,….)
              Assembled at Brown University (1963-64, Kucera and Francis)
   William Wang’s (1967) DOC (Dictionary on Computer)
        On-line Chinese dialect dictionary
                      1970-1983
Explosion of research
   Stochastic paradigm
        Developed speech recognition algorithms
             HMM’s
             Developed independently by Jelinek et al. at IBM and
              Baker at CMU
   Logic-based paradigm
        Prolog, definite-clause grammars (Pereira and Warren,
         1980)
        Functional grammar (Kay, 1979) and Lexical Functional
         Grammar
                   1970-1983
Explosion of research
   Natural language understanding
      SHRDLU (Winograd, 1972)
      The Yale School
            Focused on human conceptual knowledge and
             memory organization
 Logic-based LUNAR question-answering
  system (Woods, 1973)
 Discourse modeling paradigm
Revival of Empiricism and FSM’s
           1983-1993
 Finite-state models
    Phonology and morphology (Kaplan and Kay,
     1981)
    Syntax (Church, 1980)
 Return of empiricism
    Rise of probabilistic models in speech and
     language processing
    Largely influenced by work in speech recognition
     at IBM
 Considerable work on natural language
 generation
        A Reunion of a Sort…
            1994-present
Probabilistic and data-driven models had become
quite standard
Increases in speed and memory of computers
allowed commercial exploitation of speech and
language processing
   Phone – based dialog systems
   Spelling and grammar checking
Rise of the Web emphasized the need for language
based information retrieval and information extraction

								
To top