Docstoc

slide

Document Sample
slide Powered By Docstoc
					Part 5:



The evaluation of Web-based Chinese
Question-Answering System

     Keywords:Question answering、QAS


                 Oscar Li Jen Hsu
                   2010/01/29

                                       1
Why Question-Answering System?
How can it be applied?
 A Chinese Semantic Search Engine
  includes Question-Answering System
  (QAS). A QAS is easily to verify the
  accuracy by benchmarks which likes
  TREC 2003.
 LILOG, a text-understanding system that
  operated on the domain of tourism
  information in a German city. (Wikipedia, 2010)

                                                2
 Unix Consultant (UC), a system that
  answered questions pertaining to the Unix
  operating system. (Wikipedia, 2010)
 Some of the early AI systems included
  question-answering abilities. Two of the most
  famous early systems are SHRDLU and
  ELIZA. (Wikipedia, 2010)
 SHRDLU simulated the operation of a robot
  in a toy world (the "blocks world"), and it
  offered the possibility to ask the robot
  questions about the state of the world.
 ELIZA simulated a conversation with a
  psychologist.
                                             3
Literature Review
   Figueroa, A., Neumann, G., & Atkinson, J. (2009).
    Searching for Definitional Answers on the Web Using
    Surface Patterns. [Article]. Computer, 42(4), 68.
   朱德熙(2007)。語法講義。香港:商務印書館。
   Cai, D., Cui, H., Miao, X., Zhao, C., & Ren, X. (2004). A
    Web-based Chinese automatic question answering system.
    Paper presented at the Computer and Information
    Technology, 2004. CIT '04. The Fourth International
    Conference on.




                                                            4
Literature Review (cont.)
   Ong, C. S., Day, M.Y., & Hsu, W. L. (2009).
    The measurement of user satisfaction with question
    answering systems. Information & Management, 46(7),
    397-403.
   Hildebrandt, W., Katz, B., & Lin, J. (2004). Answering
    Definition Questions Using Multiple Knowledge
    Sources. In HLT-NAACL (pp. 49--56).




                                                             5
Searching for Definitional Answers
on the Web Using Surface Patterns
   Figueroa , Neumann , and Atkinson (2009)
    propose a approach that employs query
    rewriting techniques to increase the
    probability of extracting the nuggets(正確答案)
    from Web snippets(片段知識) by matching
    “surface patterns”.
   This method takes advantage of corpus-
    based semantic analysis and sense
    disambiguation strategies for extracting
    words that describe different concepts on
    the Web.
                                              6
Definitional Web-based QAS
system architecture




                             7
    Surface Patterns
   A surface pattern π
    ◦ π:δ [is|are|has been|was|were] [a|the|an] η
    ◦ δ :A name in a question.
    ◦ η :A answer.
   Example:
    “Who is δ” → “Who is Tom Hanks?”
    π → “Tom Hanks is an Academy
    Award-winning actor.”

                                                    8
Surface Patterns in Chinese
 例. 蒸汽機是誰發明的?
 由“的”字結構組成的判斷句(朱德
    熙,2007)
    ◦ “是”字開頭的句子,「是瓦特發明的蒸汽
      機」,底線內容表答案。
    ◦ 名詞或代詞挪到句首,「蒸汽機是瓦特發
      明的」
    ◦ 省略“是”,「瓦特發明的蒸汽機」



                              9
The modules

   Definitional miner module
    ◦ Extracting sentences from the Web that are
      likely to contain a definition of δ.

   Definitional rule matcher module
    ◦ This module identify the definiendum δ and
      its definition nugget η within the sentence by
      surface patterns.

                                                  10
The modules (cont.)

   Context miner module
    ◦ The same name or word in a question can
      refer to several meanings. This module
      extracts the different senses of δ by observing
      the correlation of their neighbors in the
      reliable semantic space.




                                                 11
The modules (cont.)
   Sense disambiguator module
    ◦ This module resolves the problem of the
      different senses of δ by discovering a set of
      uncorrelated words.


   Definition ranker module
    ◦ This module produces an ordered sequence
      of extracted definitions.


                                                      12
A Web-based Chinese Automatic
Question Answering System
 Cai, Cui, Miao, Zhao, and Ren (2004)
  proposed a web-based Chinese question
  answering system.
 The system uses the Google Web APIs to
  retrieval knowledge from Google.




                                      13
A
Web-based
Chinese
Automatic
Question
Answering
System
architecture




               14
A Web-based Chinese Automatic
Question Answering System (cont.)
   The system classify the question based on
    the Chinese question pattern to infer the
    “anticipated answer type” which includes
    PERSON, ORGANIZATION, LOCATION,
    DATE, TIME.
   The “Search Result Pretreatment module”
    of the system filters the sentences of a
    search result to acquire a candidate
    answer of the “anticipated answer type”
                                             15
A Web-based Chinese Automatic
Question Answering System (cont.)
 The “Answer Extraction module”
  calculates similarity between the question
  and answers, then grabs the top 5
  sentences as possible answers and return
  them to the user.
 The similarity is decided by
    “keywords in the question and the answer”,
    “Length of the question and the answer” ,
    “Sequence of keywords”, and
    “Keywords distance”.
                                                 16
The measurement of user satisfaction
with question answering systems

  Ong, Day, & Hsu (2009) proposed an
   evaluation model to measure User
   Satisfaction with Question Answering
   Systems (USQAS).
  The model provides a framework for the
   design of QAS from the user’s
   perspective and that it could help increase
   user acceptance of QAS.
                                           17
The measurement of user satisfaction
with question answering systems

    The USQAS instrument in this study
     provided a high degree of confidence in
     the reliability and validity of the scales.




                                                   18
A comprehensive model for measuring user satisfaction
  with QAS (USQAS).                                   19
E1 : My interaction with the QAS is clear and understandable.
E2 : Learning to use the QAS is easy.
E3 : It is easy for me to be come skillful at using the QAS.
E4 : I find it easy to use the QAS to do what I want it to do.
E5 : I find the QAS easy to use.
U1 : Using the QAS would enhance my effectiveness on the
  job.
U2 : I would and the QAS useful in my job.
U3 : Using the QAS would improve my job performance.
U4 : Using the QAS in my job would increase my productivity.
U5 : Using the QAS would make it easier to do my job.



 A comprehensive model for measuring user
  satisfaction with QAS (USQAS).
                                         20
S1 : The QAS is dependable.
S2 : The QAS employees provide prompt service to
  users.
S3 : The QAS has up-to-date hardware and software.
S4 : The QAS employees have the knowledge to do their
  job well.
I1 : Information provided in the QAS is easy to
  understand.
I2 : Information provided in the QAS is relevant.
I3 : Information provided by the QAS is complete.
I4 : Information provided in the QAS is personalized.

A comprehensive model for measuring user
 satisfaction with QAS (USQAS).
                                        21
Answering Definition Questions
Using Multiple Knowledge Sources
   Hildebrandt, Katz, & Lin (2004) proposed a
    multi-strategy approach to answering
    “definition questions” using three techniques
    to retrieve relevant nuggets(重要的答案).
    1. To lookup in a database created from the
       AQUAINT corpus.
    2. To lookup in a Web dictionary followed by
       answer projection.
    3. To lookup directly in the AQUAINT corpus
       with an IR engine.
                                                   22
Answering Definition Questions
Using Multiple Knowledge Sources

   A Factoid Question which likes “How tall
    is the Taiwan 101 Building? ” can be
    answered by using three technology : a
    textual corpus, information retrieval , and
    named-entity extraction.




                                             23
Answering Definition Questions Using
Multiple Knowledge Sources (cont.)
    A Definition Question is in contrast to
     factoid question. The objective for a
     Definition Question is to produce as many
     useful nuggets as possible.

    “Who is Aaron Copland?”
     ◦   “American composer”
     ◦   “wrote ballets and symphonies”
     ◦   “born in Brooklyn, New York, in 1900”
     ◦   “son of a Jewish immigrant”
     ◦   “American communist”
     ◦   “civil rights advocate”
                                                 24
Q&A

      25

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:10/24/2011
language:English
pages:25