Semantic Annotation Evaluation and Utility.ppt

Document Sample
Semantic Annotation Evaluation and Utility.ppt Powered By Docstoc
					Semantic Annotation
Evaluation and Utility
       Bonnie Dorr
     Saif Mohammad
     David Yarowsky
        Keith Hall
                 Road Map
• Project Organization
• Semantic Annotation and Utility Evaluation
• Focus Area: Informal Input
  –   Belief/Opinion/Confidence (modality)
  –   Dialog Acts
  –   Complex Coreference (e.g., events)
  –   Temporal relations
• Interoperability
• Current and Future Work
              Project Organization
  CMU (Mitamura, Levin, Nyberg)
                                                                 BBN (Ramshaw, Habash)
                                                                      Temporal Annotation
      Entity relations
                                                                      Coreference (complex)
       Committed Belief

                                              Bonnie Dorr
                                              David Yarowsky
                                              Keith Hall
                                              Saif Mohammad

                                                                 UMBC (Nirenburg, McShane)
Columbia (Rambow, Passonneau)                                       Modality: polarity,
    Dialogic Content                                                epistemic, belief,
    Committed Belief                                                deontic, volitive,
                                                                    potential, permissive,
                                  Affiliated Efforts
                                         Ed Hovy
                                         Martha Palmer
                                         George Wilson (Mitre)
    Semantic Annotation & Utility
    Evaluation Meeting: Feb 14th
• Site presentations included an overview of the
  phenomena covered and utility-motivating
  examples, extracted from the target corpus.
• Collective assessment of what additional
  capabilities could be achieved if a machine
  could achieve near human-performance on
  annotation of these meaning layers relative to
  applications operating on text without such
  meaning layer analysis.
• Compatibility, Interoperability, integration into
  larger KB environment.
• How can we automate these processes?
•   Kathy Baker (DoD)          •   Sergei Nirenburg (UMBC)
•   Mona Diab (Columbia)       •   Eric Nyberg (CMU)
•   Bonnie Dorr (UMD)          •   Doug Oard (UMD)
•   Tim Finin (JHU/APL)        •   Boyan Onyshkevych (DoD)
•   Nizar Habash (Columbia)    •   Martha Palmer (Colorado)
•   Keith Hall (JHU)           •   Rebecca Passonneau (Columbia)
•   Eduard Hovy (USC/ISI)      •   Owen Rambow (Columbia)
•   Lori Levin (CMU)           •   Lance Ramshaw (BBN)
•   James Mayfield (JHU/APL)   •   Clare Voss (ARL)
•   Teruko Mitamura (CMU)      •   Ralph Weischedel (BBN)
•   Saif Mohammad (UMD)        •   George Wilson (Mitre)
•   Smaranda Muresan (UMD)     •   David Yarowsky (JHU)
         Analysis of Informal Input:
   Unifies Majority of Annotation Themes
• Four relevant representational Layers:
    –   Belief/Opinion/Confidence (modality)
    –   Dialog Acts
    –   Coreference (entities and events)
    –   Temporal relations
• Many relevant applications:
    –   KB population
    –   Social Network Analysis
    –   Sentiment analysis
    –   Deception detection
    –   Text mining
    –   Question answering
    –   Information retrieval
    –   Summarization
• Analysis of informal input is dynamic: a first analysis may be refined
  when subsequent informal input contributions are processed
Representational Layer 1: Committed Belief
• Committed belief: Speaker indicates in this utterance
  that Speaker believes the proposition
   – I know Afghanistan and Pakistan have provided the richest
     opportunity for Al Qaeda to take root.
• Non-committed belief: Speaker identifies the
  proposition as something which Speaker could believe,
  but Speaker happens not to have a strong belief in the
   – Afghanistan and Pakistan may have provided the richest
     opportunity for Al Qaeda to take root.
• No asserted belief: for Speaker, the proposition is not
  of type in which Speaker is expressing a belief, or could
  express a belief. Usually, this is because the proposition
  does not have a truth value in this world.
   – Did Afghanistan and Pakistan provide the richest opportunity for
     Al Qaeda to take root?
Committed Belief is not Factivity
           Fact                       Opinion
  CB       Smith was                  Smith was a nasty
           assassinated.              dictator.
  NA       Smith will be              Smith will become a
           assassinated.              nasty dictator (once he is
                                      in power).
  CB = committed belief, NA = No asserted belief
 • Committed-belief annotation and factivity annotation are
 • NA cases may lead to detection of current and future threats,
    sometimes conditional. Multiple modalities (opinion detection):
     – Potential: ―Smith might be assassinated — if he is in power.‖
     – Obligative: ―Smith should be assassinated.‖
  Committed Belief is not Tense
           Past               Future
   CB      Smith was          Smith will be
           assassinated.      assassinated tomorrow.
   NA      I hope Smith       I hope Smith will regret
           regretted his      his acts.
   CB = committed belief, NA = No asserted belief
• Special feature to indicate future tense on CB (committed
  belief) and NCB (non-committed belief)
           Why Is Recognizing
        Committed Belief Important?
• Committed-Belief Annotation Distinguishes
   – Propositions that are asserted as true (CB)
   – Propositions that are asserted but speculative (NCB)
   – Propositions that are not asserted at all (NA)
• Important whenever we need to identify facts
   – IR Query: show documents discussing instances of peasants
     being robbed of their land
       • Document found 1: The people robbing Iraqi peasants of their
         land should be punished         RELEVANT: YES
       • Document found 2: Robbing Iraqi peasants of their land would
         be bad.                         RELEVANT: NO
   – QA: Did the humanitarian crisis in Iraq end?
       • Text found 1: He arrived on Tuesday, bringing an end to the
         humanitarian crisis in Iraq.     ANS: YES.
       • Text found 2: He arrived on Tuesday, calling for an end to the
         humanitarian crisis in Iraq.     ANS: I DON’T KNOW
Representational Layer 2: Dialog Acts
    •   INFORM
    •   COMMIT
    •   ACCEPT
    •   REJECT
    •   PERFORM
Why is dialog analysis important?
• Understanding the outcome of an interaction
   – What is the outcome?
   – Who prevailed?
   – Why (status of interactants, priority of communicative
• Application of a common architecture to
  automatic analysis of interaction in email, blogs,
  phone conversations, . . .
• Social Network Analysis: Is the speaker/sender
  in an inferior position to the hearer/receiver?
   – How can we know? (e.g., REJECT a REQUEST)
      Representational Layer 3:
   Complex Coreference (e.g., events)
Annotate events beyond ACE coreference
  – ACE does not identify Events as coreferents when
    one mention refers only to a part of the other
  – In ACE, the plural event mention is not coreferent
    with mentions of the component individual events.
  – ACE does not annotate:
  ―Three people have been convicted…Smith and
    Jones were found guilty of selling guns…‖
  ―The gunman shot Smith and his son. ..The attack
    against Smith.‖
 Related Events (and sub-events)
• Events that happened
   ―Britain bombed Iraq last night.‖
• Events which did not happen
   ―Hall did not speak about the bombings.‖
• Planned events
    planned, expected to happen, agree to do…
   ―Hall planned to meet with Saddam.‖
• Sub-Event Examples:
  – ―drug war‖ (contains subevents: attacks, crackdowns,
  – ―attacks‖ (contains subevents: deaths, kidnappings,
    assassination, bombed…)
   Why is complex coreference
     resolution important?
• Complex Question Answering:
  – Event questions: Describe the drug war
    events in Latin America.
  – List questions: List the events related to
    attacks in the drug war.
  – Relationship questions: Who is attacking
            Representational Layer 4:
              Temporal Relations
Baghdad 11/28 -- Senator Hall arrived in Baghdad yesterday. He
told reporters that he “ will not be visiting Tehran” before he left
Washington. He will return next Monday.

TimeUnit    Type                  Relation      Parent
11/28     Specific.Date           After         arrived
arrived   Past.Event              Before        <writer>
yesterday Past.Date               Concurrent    arrived
told      Past.Say                Before        arrived
visiting Neg.Future.Event         After         told
left      Past.Event              After         told
return    Future.Event            After         <writer>
Monday Specific.Date              Concurrent    return
       Temporal Relation Parse

                 arrived                 return

told            yesterday       11/28    Monday


                       (not) visiting

        Temporal Relation Analysis:
         Inter-annotator Agreement
          Temporal Type             Matches      Clashes   Agreement
              410_nyt                    30            1       96.8%
              419_apw                    28            0      100.0%
              602CZ                      34            3       91.9%
              ENRON                      12            2       85.7%
              Total                     104            6       94.5%
           Parent Pointers          Matches      Clashes   Agreement
                 410_nyt                 27            4       87.1%
                 419_apw                 27            1       96.4%
                 602CZ                   26           11       70.3%
                 ENRON                   13            1       92.9%
                 Total                   93           17       84.5%

Temp Relations       Exact Match   Partial Mat      Clash Exact Agree Part Agree
     410_nyt                  23            3           1      85.2%      96.3%
     419_apw                  24            3           0      88.9%    100.0%
     602CZ                    23            2           1      88.5%      96.2%
     ENRON                    11            1           1      84.6%      92.3%
     Total                    81            9           3      87.1%      96.8%
      Why is Temporal Analysis
• Constructing activity schedules from text

• Question answering (temporal):
  did/does/will X happen
  before/after/same_time_with Y?

 where X,Y are events, states, dates or time
        Interoperability: Data
• Common data model
• Multiple implementations
  – based on the same underlying schema
    (formal object model)
  – meet different goals / requirements
• Implementation Criteria:
  – Support effective run-time annotation
  – Support effective user interface, query/update
  – Support on-the-fly schema extension
Example: UMBC Modality

    Ongoing and Future work
• Move to new genre—informal input.
• Establish compatibility across levels.
• Continue examining intra-site and cross-site
  annotation agreement rates
• Initial assessment of computational feasibility of
  machine learning approaches—―our annotations
  are supposed to be fodder for ML approaches.‖
• Implementation of framework for superimposing
  semantic ―layers‖ on existing objects (e.g., on
  top of ACE types).
• Move to multiple languages.

Shared By:
yanyan yan yanyan yan