Get documents about "Performance Characteristics of Control Chart Detection Methods
W
Shared by: fla18296
Categories
Tags
control limits, control charts, control chart, statistical process control, detection methods, detection algorithms, data sources, syndromic surveillance, performance characteristics, disease outbreak, special cause variation, clinical variables, statistical control, cusum charts, surveillance systems
-
Stats
- views:
- 11
- posted:
- 1/15/2010
- language:
- English
- pages:
- 20
Document Sample
Performance Characteristics of
Control Chart Detection Methods
October, 2007
Jerome Tokars, MD, MPH
Division of Emergency Preparedness and Response
Centers for Disease Control and Prevention
The findings and conclusions in this presentation are
those of the author(s) and do not necessarily represent
the views of the Centers for Disease Control and
Prevention
TM
Background
• A variety of statistical algorithms are used to
recognize data anomalies in time series
displays
• BioSense uses modified EARS C2 algorithm
• Stratify baseline days by weekday vs weekend
(W2)
• Standard method based on counts, “rate” method
that accounts for total visits
• These modifications improve fit (i.e.,
expected closer to observed), but the effect
on sensitivity/specificity not studied
• Objective: to evaluate the performance of
several variations of control chart methods
TM
Methods
• Datasets
• Department of Defense (DoD) outpatient
clinics final diagnoses
• Hospital emergency department (ED) chief
complaints
• Analysis at facility level
• Included facility/syndrome with mean count
≥0.5 per day
TM
Algorithms
• 12 variations tested
• Baseline days un-stratified (C2) vs
stratified by weekday/weekend (W2)
• 3 baseline durations: 7, 14, 28 days
• Count vs rate methods
• All use 2-day buffer between baseline days
and index day
• Standard: C2-7-count1
• BioSense: W2-7-count2, W2-7-rate2
1Hutwagner L, Thompson W, Seeman GM, et al. The Bioterrorism
Preparedness and Response Early Aberration Reporting System (EARS). J
Urban Hlth 2003;80(2, suppl 1):i89--i96.
2Available on request (jit1@cdc.gov)
TM
Statistical Methods
• Minimum standard deviation=0.5
• Test statistic=residual/standard deviation
• Empirical threshold for 1% alert rate for each
method and dataset (threshold range: 3.2-4.8
stds)
• Single-day injections of 8 counts to all days
• Sensitivity = days exceeding threshold/total days
• Multi-day injections of 30 counts over ~5.5
days in log-normal distribution
• Hospital ED, fever syndrome
• Sensitivity = signals detected/signals injected
TM
Results: Dataset Characteristics
Dept of Defense Hospital ED
outpatient diagnosis chief complaint
Study period 9/04-6/07 3/06-7/07
No. of facilities 312 331
Total facility/
syndrome/days 1,740,635 508,390
Syndrome 7.4 7.8
count, mean (1.0-23.4) (0.9-20.1)
(range by
syndrome)
No. of 10 8
syndromes
TM
Distribution of Syndrome Counts by
Day of Week
25%
Percent of Counts
20%
15% DoD
10% Hospital ED
5%
0%
Mon Tue Wed Thu Fri Sat Sun
Day of Week
TM
Single-Day Injection, DoD
8 counts added per day
Difference = 22.3%
70%
65%
Sensitivity, %
60%
Count
55%
Rate
50%
45%
40%
C2-7 C2-14 C2-28 W2-7 W2-14 W2-28
Un-stratified Stratified
TM
Single-Day Injection, Hospital ED
8 counts added per day
Difference = 12.5%
65%
60%
Sensitivity, %
55%
Count
50% Rate
45%
40%
C2-7 C2-14 C2-28 W2-7 W2-14 W2-28
Un-stratified Stratified
TM
Multi-Day: Hospital ED, Fever Syndrome
30 counts added over ~5.5 days
Difference = 9.8%
80%
75%
Sensitivity, %
70%
Count
65%
Rate
60%
55%
50%
C2-7 C2-14 C2-28 W2-7 W2-14 W2-28
Un-stratified Stratified
TM
Summary of Best Methods
Department of Hospital ED
Defense
Stratification
by weekday Stratified (W2)1 Un-stratified (C2)3
vs weekend
Baseline 14-28 days2 14-28 days2
duration
Count vs rate Rate1 (Rate)1
1Method produces lower residuals
2Longer baseline provides better estimate of standard deviation
3Stratificiationbaseline days further from index day
TM
Discussion
• We studied theoretical detection of artificially-
added signals
• In practice, signal detection has not been
highly successful
• Better coverage, more specific data, and
better detection algorithms may improve our
track record
• We studied only simple methods—worthwhile
to optimize simple less computer-intensive
algorithms
TM
Conclusions
• Simple modifications of standard algorithms,
especially a longer baseline, improve
sensitivity
• Best methods depend on data characteristics
(eg, day-of-week effect) and could be
selected automatically by software
• Further work planned to extend multi-day
signal injection; examine subgroups, other
data sources, and additional algorithms
TM
Acknowledgments
Coauthors
Jian Xing1, Howard Burkom2, John Copeland1, Steve
Bloom3, Lori Hutwagner1
Collaborator
Hwa-Gan Chang
Data sources
Department of Defense; state and local health
departments; hospitals/hospital systems
1 Centers for Disease Control and Prevention
2 The Johns Hopkins Applied Physics Laboratory
3 Science Applications Incorporated
TM
Extra Slides
TM
Sensitivity, Single-Day Injection,
Hospital ED, by Syndrome
100%
80%
Sensitivity, %
60% C2-7-Count
40% C2-28-Rate
20%
0%
em
p
h
r
ut
t
v
ot
eu
as
es
as
Fe
Lc
B
H
G
N
R
R
Syndrome
Alert rate=1% 8 additional counts injected per day
TM
Limitations/Strengths
• Limitations
• Restricted to data with mean count ≥ 0.5/day
• Studied selected patient types, data types: results
may not be generalizable
• Only facility-level aggregation tested
• Only simple control chart methods tested
• Strengths
• BioSense, as a system-of-systems, enables
testing in many facilities/jurisdictions
• Two major data sources examined
• Empirical methods to determine alerting
thresholds
TM
Percent of Visits Analyzed, by Syndrome
Syndrome DoD Hospital ED
Botulism-like 3.1 1.9
Hemorrhagic 5.3 13.6
Lymphadenopathy 4.3 0
Localized Cutaneous 11.0 12.7
Gastrointestinal 14.7 14.8
Respiratory 17.4 14.9
Severe Injury or Death 0 0
Neurological 11.1 14.5
Rash 11.4 13.1
Specific Infection 10.9 0
Fever 10.8 14.4
All Syndromes 100% 100%
TM
Stratified: W2-7 Count Calculations
Day:Day of Week Syndrome Count Total Visits
1: M 5 100
2: T 6 90
3: W 3 75
4: T 2 80
5: F 7 70
6: S 6 100
7: S 3 90
8: M 10 80
9: T 5 85
10: W 9 75
11: T 10 95
12: F (Index Day) 7 90
Expected value = mean (5,6,3,2,7,10,5)
TM
Stratified: W2-7 Rate Calculations
Day:Day of Week Syndrome Count Total Visits
1: M 5 100
2: T 6 90
3: W 3 75
4: T 2 80
5: F 7 70
6: S 6 100
7: S 3 90
8: M 10 80
9: T 5 85
10: W 9 75
11: T 10 95
12: F (Index Day) 7 90
N=sum (5,6,3,2,7,10,5) D=sum(100,90,75,80,70,80,85)
Expected value = 90*N/D
TM
Related docs
