Semantic Annotation Evaluation and Utility Bonnie Dorr Saif Mohammad David Yarowsky Keith Hall Road Map • Project Organization • Semantic Annotation and Utility Evaluation Workshop • Focus Area: Informal Input – Belief/Opinion/Confidence (modality) – Dialog Acts – Complex Coreference (e.g., events) – Temporal relations • Interoperability • Current and Future Work Project Organization CMU (Mitamura, Levin, Nyberg) BBN (Ramshaw, Habash) Coreference Temporal Annotation Entity relations Coreference (complex) Committed Belief Evaluation Bonnie Dorr David Yarowsky Keith Hall Saif Mohammad UMBC (Nirenburg, McShane) Columbia (Rambow, Passonneau) Modality: polarity, Dialogic Content epistemic, belief, Committed Belief deontic, volitive, potential, permissive, evaluative Affiliated Efforts Ed Hovy Martha Palmer George Wilson (Mitre) Semantic Annotation & Utility Evaluation Meeting: Feb 14th • Site presentations included an overview of the phenomena covered and utility-motivating examples, extracted from the target corpus. • Collective assessment of what additional capabilities could be achieved if a machine could achieve near human-performance on annotation of these meaning layers relative to applications operating on text without such meaning layer analysis. • Compatibility, Interoperability, integration into larger KB environment. • How can we automate these processes? Attendees • Kathy Baker (DoD) • Sergei Nirenburg (UMBC) • Mona Diab (Columbia) • Eric Nyberg (CMU) • Bonnie Dorr (UMD) • Doug Oard (UMD) • Tim Finin (JHU/APL) • Boyan Onyshkevych (DoD) • Nizar Habash (Columbia) • Martha Palmer (Colorado) • Keith Hall (JHU) • Rebecca Passonneau (Columbia) • Eduard Hovy (USC/ISI) • Owen Rambow (Columbia) • Lori Levin (CMU) • Lance Ramshaw (BBN) • James Mayfield (JHU/APL) • Clare Voss (ARL) • Teruko Mitamura (CMU) • Ralph Weischedel (BBN) • Saif Mohammad (UMD) • George Wilson (Mitre) • Smaranda Muresan (UMD) • David Yarowsky (JHU) Analysis of Informal Input: Unifies Majority of Annotation Themes • Four relevant representational Layers: – Belief/Opinion/Confidence (modality) – Dialog Acts – Coreference (entities and events) – Temporal relations • Many relevant applications: – KB population – Social Network Analysis – Sentiment analysis – Deception detection – Text mining – Question answering – Information retrieval – Summarization • Analysis of informal input is dynamic: a first analysis may be refined when subsequent informal input contributions are processed Representational Layer 1: Committed Belief • Committed belief: Speaker indicates in this utterance that Speaker believes the proposition – I know Afghanistan and Pakistan have provided the richest opportunity for Al Qaeda to take root. • Non-committed belief: Speaker identifies the proposition as something which Speaker could believe, but Speaker happens not to have a strong belief in the proposition – Afghanistan and Pakistan may have provided the richest opportunity for Al Qaeda to take root. • No asserted belief: for Speaker, the proposition is not of type in which Speaker is expressing a belief, or could express a belief. Usually, this is because the proposition does not have a truth value in this world. – Did Afghanistan and Pakistan provide the richest opportunity for Al Qaeda to take root? Committed Belief is not Factivity Fact Opinion CB Smith was Smith was a nasty assassinated. dictator. NA Smith will be Smith will become a assassinated. nasty dictator (once he is in power). CB = committed belief, NA = No asserted belief • Committed-belief annotation and factivity annotation are complementary • NA cases may lead to detection of current and future threats, sometimes conditional. Multiple modalities (opinion detection): – Potential: ―Smith might be assassinated — if he is in power.‖ – Obligative: ―Smith should be assassinated.‖ Committed Belief is not Tense Past Future CB Smith was Smith will be assassinated. assassinated tomorrow. NA I hope Smith I hope Smith will regret regretted his his acts. acts. CB = committed belief, NA = No asserted belief • Special feature to indicate future tense on CB (committed belief) and NCB (non-committed belief) Why Is Recognizing Committed Belief Important? • Committed-Belief Annotation Distinguishes – Propositions that are asserted as true (CB) – Propositions that are asserted but speculative (NCB) – Propositions that are not asserted at all (NA) • Important whenever we need to identify facts – IR Query: show documents discussing instances of peasants being robbed of their land • Document found 1: The people robbing Iraqi peasants of their land should be punished RELEVANT: YES • Document found 2: Robbing Iraqi peasants of their land would be bad. RELEVANT: NO – QA: Did the humanitarian crisis in Iraq end? • Text found 1: He arrived on Tuesday, bringing an end to the humanitarian crisis in Iraq. ANS: YES. • Text found 2: He arrived on Tuesday, calling for an end to the humanitarian crisis in Iraq. ANS: I DON’T KNOW Representational Layer 2: Dialog Acts • INFORM • REQUEST-INFORMATION • REQUEST-ACTION • COMMIT • ACCEPT • REJECT • BACKCHANNEL • PERFORM • CONVENTIONAL Why is dialog analysis important? • Understanding the outcome of an interaction – What is the outcome? – Who prevailed? – Why (status of interactants, priority of communicative action)? • Application of a common architecture to automatic analysis of interaction in email, blogs, phone conversations, . . . • Social Network Analysis: Is the speaker/sender in an inferior position to the hearer/receiver? – How can we know? (e.g., REJECT a REQUEST) Representational Layer 3: Complex Coreference (e.g., events) Annotate events beyond ACE coreference definition – ACE does not identify Events as coreferents when one mention refers only to a part of the other – In ACE, the plural event mention is not coreferent with mentions of the component individual events. – ACE does not annotate: ―Three people have been convicted…Smith and Jones were found guilty of selling guns…‖ ―The gunman shot Smith and his son. ..The attack against Smith.‖ Related Events (and sub-events) • Events that happened ―Britain bombed Iraq last night.‖ • Events which did not happen ―Hall did not speak about the bombings.‖ • Planned events planned, expected to happen, agree to do… ―Hall planned to meet with Saddam.‖ • Sub-Event Examples: – ―drug war‖ (contains subevents: attacks, crackdowns, bullying…) – ―attacks‖ (contains subevents: deaths, kidnappings, assassination, bombed…) Why is complex coreference resolution important? • Complex Question Answering: – Event questions: Describe the drug war events in Latin America. – List questions: List the events related to attacks in the drug war. – Relationship questions: Who is attacking who? Representational Layer 4: Temporal Relations Baghdad 11/28 -- Senator Hall arrived in Baghdad yesterday. He told reporters that he “ will not be visiting Tehran” before he left Washington. He will return next Monday. TimeUnit Type Relation Parent 11/28 Specific.Date After arrived arrived Past.Event Before <writer> yesterday Past.Date Concurrent arrived told Past.Say Before arrived visiting Neg.Future.Event After told left Past.Event After told return Future.Event After <writer> Monday Specific.Date Concurrent return Temporal Relation Parse <writer> arrived return told yesterday 11/28 Monday left (not) visiting TIME Temporal Relation Analysis: Inter-annotator Agreement Temporal Type Matches Clashes Agreement 410_nyt 30 1 96.8% 419_apw 28 0 100.0% 602CZ 34 3 91.9% ENRON 12 2 85.7% Total 104 6 94.5% Parent Pointers Matches Clashes Agreement 410_nyt 27 4 87.1% 419_apw 27 1 96.4% 602CZ 26 11 70.3% ENRON 13 1 92.9% Total 93 17 84.5% Temp Relations Exact Match Partial Mat Clash Exact Agree Part Agree 410_nyt 23 3 1 85.2% 96.3% 419_apw 24 3 0 88.9% 100.0% 602CZ 23 2 1 88.5% 96.2% ENRON 11 1 1 84.6% 92.3% Total 81 9 3 87.1% 96.8% Why is Temporal Analysis Important? • Constructing activity schedules from text • Question answering (temporal): did/does/will X happen before/after/same_time_with Y? where X,Y are events, states, dates or time ranges. Interoperability: Data • Common data model • Multiple implementations – based on the same underlying schema (formal object model) – meet different goals / requirements • Implementation Criteria: – Support effective run-time annotation – Support effective user interface, query/update – Support on-the-fly schema extension Example: UMBC Modality Annotations 21 Ongoing and Future work • Move to new genre—informal input. • Establish compatibility across levels. • Continue examining intra-site and cross-site annotation agreement rates • Initial assessment of computational feasibility of machine learning approaches—―our annotations are supposed to be fodder for ML approaches.‖ • Implementation of framework for superimposing semantic ―layers‖ on existing objects (e.g., on top of ACE types). • Move to multiple languages.
Pages to are hidden for
"Semantic Annotation Evaluation and Utility.ppt"Please download to view full document