Part 5:
The evaluation of Web-based Chinese
Question-Answering System
Keywords：Question answering、QAS
Oscar Li Jen Hsu
2010/01/29
1
Why Question-Answering System?
How can it be applied?
A Chinese Semantic Search Engine
includes Question-Answering System
(QAS). A QAS is easily to verify the
accuracy by benchmarks which likes
TREC 2003.
LILOG, a text-understanding system that
operated on the domain of tourism
information in a German city. (Wikipedia, 2010)
2
Unix Consultant (UC), a system that
answered questions pertaining to the Unix
operating system. (Wikipedia, 2010)
Some of the early AI systems included
question-answering abilities. Two of the most
famous early systems are SHRDLU and
ELIZA. (Wikipedia, 2010)
SHRDLU simulated the operation of a robot
in a toy world (the "blocks world"), and it
offered the possibility to ask the robot
questions about the state of the world.
ELIZA simulated a conversation with a
psychologist.
3
Literature Review
Figueroa, A., Neumann, G., & Atkinson, J. (2009).
Searching for Definitional Answers on the Web Using
Surface Patterns. [Article]. Computer, 42(4), 68.
朱德熙(2007)。語法講義。香港：商務印書館。
Cai, D., Cui, H., Miao, X., Zhao, C., & Ren, X. (2004). A
Web-based Chinese automatic question answering system.
Paper presented at the Computer and Information
Technology, 2004. CIT '04. The Fourth International
Conference on.
4
Literature Review (cont.)
Ong, C. S., Day, M.Y., & Hsu, W. L. (2009).
The measurement of user satisfaction with question
answering systems. Information & Management, 46(7),
397-403.
Hildebrandt, W., Katz, B., & Lin, J. (2004). Answering
Definition Questions Using Multiple Knowledge
Sources. In HLT-NAACL (pp. 49--56).
5
Searching for Definitional Answers
on the Web Using Surface Patterns
Figueroa , Neumann , and Atkinson (2009)
propose a approach that employs query
rewriting techniques to increase the
probability of extracting the nuggets(正確答案)
from Web snippets(片段知識) by matching
“surface patterns”.
This method takes advantage of corpus-
based semantic analysis and sense
disambiguation strategies for extracting
words that describe different concepts on
the Web.
6
Definitional Web-based QAS
system architecture
7
Surface Patterns
A surface pattern π
◦ π：δ [is|are|has been|was|were] [a|the|an] η
◦ δ ：A name in a question.
◦ η ：A answer.
Example:
“Who is δ” → “Who is Tom Hanks?”
π → “Tom Hanks is an Academy
Award-winning actor.”
8
Surface Patterns in Chinese
例. 蒸汽機是誰發明的？
由“的”字結構組成的判斷句（朱德
熙,2007）
◦ “是”字開頭的句子，「是瓦特發明的蒸汽
機」，底線內容表答案。
◦ 名詞或代詞挪到句首，「蒸汽機是瓦特發
明的」
◦ 省略“是”，「瓦特發明的蒸汽機」
9
The modules
Definitional miner module
◦ Extracting sentences from the Web that are
likely to contain a definition of δ.
Definitional rule matcher module
◦ This module identify the definiendum δ and
its definition nugget η within the sentence by
surface patterns.
10
The modules (cont.)
Context miner module
◦ The same name or word in a question can
refer to several meanings. This module
extracts the different senses of δ by observing
the correlation of their neighbors in the
reliable semantic space.
11
The modules (cont.)
Sense disambiguator module
◦ This module resolves the problem of the
different senses of δ by discovering a set of
uncorrelated words.
Definition ranker module
◦ This module produces an ordered sequence
of extracted definitions.
12
A Web-based Chinese Automatic
Question Answering System
Cai, Cui, Miao, Zhao, and Ren (2004)
proposed a web-based Chinese question
answering system.
The system uses the Google Web APIs to
retrieval knowledge from Google.
13
A
Web-based
Chinese
Automatic
Question
Answering
System
architecture
14
A Web-based Chinese Automatic
Question Answering System (cont.)
The system classify the question based on
the Chinese question pattern to infer the
“anticipated answer type” which includes
PERSON, ORGANIZATION, LOCATION,
DATE, TIME.
The “Search Result Pretreatment module”
of the system filters the sentences of a
search result to acquire a candidate
answer of the “anticipated answer type”
15
A Web-based Chinese Automatic
Question Answering System (cont.)
The “Answer Extraction module”
calculates similarity between the question
and answers, then grabs the top 5
sentences as possible answers and return
them to the user.
The similarity is decided by
“keywords in the question and the answer”,
“Length of the question and the answer” ,
“Sequence of keywords”, and
“Keywords distance”.
16
The measurement of user satisfaction
with question answering systems
Ong, Day, & Hsu (2009) proposed an
evaluation model to measure User
Satisfaction with Question Answering
Systems (USQAS).
The model provides a framework for the
design of QAS from the user’s
perspective and that it could help increase
user acceptance of QAS.
17
The measurement of user satisfaction
with question answering systems
The USQAS instrument in this study
provided a high degree of confidence in
the reliability and validity of the scales.
18
A comprehensive model for measuring user satisfaction
with QAS (USQAS). 19
E1 : My interaction with the QAS is clear and understandable.
E2 : Learning to use the QAS is easy.
E3 : It is easy for me to be come skillful at using the QAS.
E4 : I find it easy to use the QAS to do what I want it to do.
E5 : I find the QAS easy to use.
U1 : Using the QAS would enhance my effectiveness on the
job.
U2 : I would and the QAS useful in my job.
U3 : Using the QAS would improve my job performance.
U4 : Using the QAS in my job would increase my productivity.
U5 : Using the QAS would make it easier to do my job.
A comprehensive model for measuring user
satisfaction with QAS (USQAS).
20
S1 : The QAS is dependable.
S2 : The QAS employees provide prompt service to
users.
S3 : The QAS has up-to-date hardware and software.
S4 : The QAS employees have the knowledge to do their
job well.
I1 : Information provided in the QAS is easy to
understand.
I2 : Information provided in the QAS is relevant.
I3 : Information provided by the QAS is complete.
I4 : Information provided in the QAS is personalized.
A comprehensive model for measuring user
satisfaction with QAS (USQAS).
21
Answering Definition Questions
Using Multiple Knowledge Sources
Hildebrandt, Katz, & Lin (2004) proposed a
multi-strategy approach to answering
“definition questions” using three techniques
to retrieve relevant nuggets(重要的答案).
1. To lookup in a database created from the
AQUAINT corpus.
2. To lookup in a Web dictionary followed by
answer projection.
3. To lookup directly in the AQUAINT corpus
with an IR engine.
22
Answering Definition Questions
Using Multiple Knowledge Sources
A Factoid Question which likes “How tall
is the Taiwan 101 Building? ” can be
answered by using three technology ： a
textual corpus, information retrieval , and
named-entity extraction.
23
Answering Definition Questions Using
Multiple Knowledge Sources (cont.)
A Definition Question is in contrast to
factoid question. The objective for a
Definition Question is to produce as many
useful nuggets as possible.
“Who is Aaron Copland?”
◦ “American composer”
◦ “wrote ballets and symphonies”
◦ “born in Brooklyn, New York, in 1900”
◦ “son of a Jewish immigrant”
◦ “American communist”
◦ “civil rights advocate”
24
Q&A
25