Title Three Stages of Evaluation for Syndromic Surveillance from

Document Sample
Title Three Stages of Evaluation for Syndromic Surveillance from Powered By Docstoc
					Title: Three Stages of Evaluation for Syndromic Surveillance from Chief
Complaint Classification

Authors: Wendy W Chapman PhD, John N Dowling PhD, Oleg Ivanov MD, MS,
Robert T Olszewski PhD, Michael M Wagner MD, PhD

Abstract:

Much of the electronic clinical data that could be harnessed for
syndromic surveillance is in free-text format. Natural language
processing (NLP) techniques may be useful to syndromic surveillance by
classifying and extracting information described in free-text sources.
Our approach to evaluating whether the NLP techniques are useful
involves three stages of evaluation.

First, we are evaluating the technical accuracy of the NLP techniques
to answer the question “How well can we classify patients into relevant
syndromic categories from text?” Second, we are evaluating the
diagnostic accuracy of the techniques to answer the question “How well
can we diagnose patients of interest using the NLP techniques?” Third,
we are evaluating the outcome efficacy of the techniques to answer the
question “How well can we detect outbreaks with an NLP-based syndromic
surveillance system?”

We present evaluation results for all three stages of evaluation,
quantifying our ability to perform syndromic surveillance from free-
text triage chief complaints. Evaluations of technical accuracy have
quantified our ability to automatically classify chief complaints into
any of eight syndromic categories: respiratory, gastrointestinal,
neurological, rash, hemorrhagic, botulinic, constitutional, and
febrile. The technical accuracy of chief complaint classifiers is high
but could be improved with synonym replacement, customized spell-
checking, and separation of multiple problems. Evaluations of
diagnostic accuracy have compared chief complaint classification to
actual syndromic diagnoses for the eight syndromes. We have generated
reference standards for syndromic diagnoses from ICD-9 discharge
diagnoses and from physician review of emergency department reports.
Diagnostic accuracy of chief complaint classification is fair to
moderate, suggesting that in spite of high technical accuracy, the
clinical information in chief complaints may only be useful for
detecting moderate to large outbreaks. Evaluations of outcome efficacy
have quantified our ability to detect seasonal respiratory and
gastrointestinal outbreaks. Outcome efficacy of chief complaint
classification is timely and accurate for the large outbreaks we
compared.

Three stages of evaluation are helpful in understanding performance of
syndromic surveillance from free-text chief complaints. We have
evaluated our ability to classify patients into relatively prevalent
syndromes, such as respiratory and gastrointestinal, and also into
syndromes that are rare and difficult to characterize, such as
hemorrhagic, botulinic, and constitutional. We conclude that NLP is
useful for syndromic surveillance from chief complaints but that to
increase sensitivity and specificity, we need to extend our techniques
from chief complaints to other clinical information sources, including
chest radiograph and emergency department reports.