Decision Support Systems
Ida Sim, MD, PhD
March 11, 2008
Division of General Internal Medicine, and
the Center for Clinical and Translational Informatics
UCSF
Copyright Ida Sim, 2008. All federal and state rights reserved for all original material presented in this course
through any medium, including lecture or print.
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Outline
• Decision support systems
– background, definition
– clinical versus research decision support
• How decision support systems “think”
– rule-based systems
– neural networks
• CDSS Effectiveness
• CDSS Adoption
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Big Picture of Health Informatics
PATIENT CARE /
WELLNES RESEARCH
Virtual Medical
Patient knowledge
Clinical Decision
Medical logic
Support Systems
Decision
support
Clinical
research
Transactions transactions
Raw
Raw data research
data
Workflow modeling and support, usability, cognitive support,
computer-supported cooperative work (CSCW), etc.
3
Clinical Decision Support
• Clinical decision support system (CDSS)
– software that is designed to be a direct aid to clinical decision-
making
– in which the characteristics of an individual patient are
matched to a computerized clinical knowledge base
– and patient-specific assessments or recommendations are
then presented to the clinician and/or the patient for a decision
(Sim et al, JAMIA, 2001)
• Examples of clinical decisions to be supported?
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Major Target Tasks of CDSSs
• Diagnostic support
– DxPlain, QMR
• Drug dosing
– aminoglycoside, theophylline, warfarin
• Preventive care
– reminders for vaccinations, mammograms
• Disease management
– diabetes, hypertension, AIDS, asthma
• Test ordering, drug prescription
– reducing daily CBCs in hospital, drug allergy checking
• Utilization
– referral management, clinic followup
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
What Isn’t a CDSS
• Medline
• UpToDate
• Static guideline repositories
– www.guideline.gov (National Guideline
Clearinghouse)
• Online laboratory data, test results, chart
notes
• Retrospective quality improvement reports
– how your vaccination rates compare to your
colleagues’
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Big Picture of Health Informatics
PATIENT CARE /
WELLNES RESEARCH
Virtual Medical
Patient knowledge
CTMS Decision
Medical logic
Support Systems
Decision
support
Clinical
research
Transactions transactions
Raw
Raw data research
data
Workflow modeling and support, usability, cognitive support,
computer-supported cooperative work (CSCW), etc.
7
CTMS Decision Support
• Clinical trial execution decision support system
– software that is designed to be a direct aid to decision-making
in clinical trial execution
– in which the characteristics of an individual subject are
matched to a computerized protocol
– and subject-specific assessments or recommendations are
then presented to the study-nurse, etc. for a decision
• Examples of CTMS decisions to be supported?
– determining eligibility
– protocol-defined procedures (e.g., if WBC PNEUMONIA
• if PNEUMONIA => GIVE-ANTIBIOTICS
• if GIVE-ANTIBIOTICS => CHECK-ALLERGIES
• if PNEUMONIA and GIVE-ANTIBIOTICS and NOT
(ALLERGIC-DOXYCYCLINE) => GIVE-DOXYCYCLINE
– use if sparse data
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Backward Chaining Rules
• Backward chaining/reasoning (goal-driven)
– start with “goal rule,” determine whether goal rule
is true by evaluating the truth of each necessary
premise
– example
• patient with lots of findings and symptoms
• is this SLE? => are 4 or more ACR criteria satisfied?
– malar rash?
– discoid rash?
– skin photosensitivity? etc
• if 4 or more ACR criteria true => SLE
– use if lots of data
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Rule Reasoning Problems
• Combinatorial explosion of rules
– need rule for each contingency
• if MOD-WBC and COUGH and FEVER and ABN-CXR =>
PNEUMONIA
• Rules may be contradictory
– if COUGH and ABN-CXR => INTERSTITIAL-LUNG-DZ
• Rules may be circular
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Representational Challenges
• Need to use standard vocabulary terms
– need to manage evolution of vocabularies (e.g., changing
terminologies in psychiatry (DSM-xx))
• Rules may involve complex semantic relationships
– if NEPHROPATHY caused-by DIABETES
• caused solely by? predominantly by?
– if SINUSITIS greater than 6 months
• representing temporal relationships requires 2nd order logic
• Need knowledge engineering and clinical expertise to
build and maintain the knowledge base over time
– need to keep rules up-to-date with latest evidence
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Sharing Rules
• Why not have libraries of rules?
• Reusable, central upkeep, evidence-based...
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Medical Logic Modules (MLMs)
• For sharing forward chaining rules
• Expressed in Arden Syntax (an ASTM standard)
• help_amp_for_pneumonia - • validation: testing;;
Ampicillin for Pneumonia • library:
• maintenance: – purpose: Recommend the use
– title: Ampicillin for of ampicillin for pneumonia.;;
Pneumonia;;
– explanation: If the patient has
– filename:
help_amp_for_pneumonia;; pneumonia, then suggest
– version: 1.00;; treatment with ampicillin unless
– institution: LDS Hospital;; there is a penicillin allergy.;;
– author: Peter Haug, M.D.; • keywords: pneumonia; penicillin;
George Hripcsak, M.D.;; ampicillin;;
– specialist: ;; • citations: 1. HELP Frame
– date: 1991-05-28;; Manual, version 1.6. LDS
Hospital, August 1989, p.81.;;
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Sharing of MLMs: No Success
• Work of reuse often greater than building from
scratch
– rules are often outdated: need to check evidence base
– context is under-specified
• is pneumonia rule inpatient or outpatient? in HIV patients?
– can be wrong for local context
• resistance patterns vary in different locales
– definitional problems
• your “pneumonia” is not my “pneumonia”
– curly braces problem
• if {K+} > 5.5 => alert MD
• need to interface to local clinical information system to access
value of K+, using interchange (HL7) and data standard (e.g.,
LOINC)
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Summary of Rule-Based Systems
• Deterministic, relatively simple reasoning
• Combinatorial explosion even for small
domains
• Requires extensive knowledg engineering
and clinical expertise
• Rules are difficult to share
• But remain most widely used method due to
simplicity for small problems
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Outline
• Decision support systems
– background, definition
– clinical versus research decision support
• How decision support systems “think”
– rule-based systems
– neural networks
• CDSS Effectiveness
• CDSS Adoption
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Neural Networks
• Finds a non-linear relationship between input parameters
and output state
• Structure of network
– usually input, output, and 1-2 hidden fully connected layers
– each connection has a “weight”
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
NN for MI Diagnosis
• Inputs (e.g., all patient characteristics in the EHR)
• EKG findings (ST elevation, old Q’s)
• rales (Yes, No)
• JVD (in cm)
• Outputs are the set of possible outcomes/diagnoses
EKG findings
Acute MI
Rales
JVD No Acute MI
Response to TNG
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Training the Neural Network
• Network gets “trained”
– give examples of known patients and diagnoses
• can handle missing data
– system iteratively adjusts connection weights to
find the network “pattern” that associates sets of
input variables (patients) with right output state (MI
or not)
• Test accuracy on another set of patients
• In Baxt’s MI neural network
– training set: 130 pts with MI, 120 without
– test set: 1070 UCSD ER patients with anterior
chest pain
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Baxt’s Acute MI Neural Net
• Evaluation results: prevalence of MI 7% (Lancet, 1996)
Sensitivity Specificity
Physicians 73.3% (63.3-83.3) 81.1% (78.7 – 83.5)
Neural Net 96.0% (91.2 – 100) 96.0% (94.8 – 97.2)
• Results were driven by non-standard predictors
– rales, jugular venous distention
• Why wasn’t this neural network used more widely?
– “black box” nature limits explanatory ability and lessens
acceptance
– users have to input the variables manually
• interfacing to EHRs would increase adoption
– need to define and code “rales” and other input terms
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Outline
• Decision support systems
– background, definition
– clinical versus research decision support
• How decision support systems “think”
– rule-based systems
– neural networks
• CDSS Effectiveness
• CDSS Adoption
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Is Decision Support Effective?
• Moderate benefit found in improving physician
behavior (Garg, 2005)
– diagnosis: 4/10 (40%) studies beneficial
– reminder systems: 16/21 (76%)
– disease management systems: 23/37 (62%)
– drug dosing: 19/29 (66%)
– few studies improved patient outcomes: 7/52 (13%)
• Counted the number of systems in each category that
were “effective” (p>0.05)
– but CDSS not all the same (apples and oranges)
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
CDSS Running Example
• Hypertension treatment Clinical Decision Support
System (CDSS)
– Clinic has an EHR
– During patient visit, CDSS notes that BP and trend is
too high.
– CDSS checks patient’s Cr, diabetes status, cardiac
status, current meds and allergies and recommends
drug therapy change according to JNC VII guidelines
and insurance coverage.
– Presents e-prescription for MD to verify. If verified,
order is sent directly to pharmacy and medication list
updated.
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
“Apples” HTN CDSS
• Clinical Decision Support Systems (CDSSs)
– software designed to directly aid clinical decision-making
• help clinician to prescribe anti-hypertensive
– in which the characteristics of an individual patient are
matched to a computerized knowledge base
• match EHR and other data to computable guideline
– and patient-specific assessments or recommendations are
presented to the clinician and/or patient for a decision
• recommends drug according to clinical, guideline, and insurance
information
• provides clinician with decision choice to prescribe or not
prescribe
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
“Oranges” HTN CDSS
• Clinical Decision Support Systems (CDSSs)
– software designed to directly aid clinical decision-making
• help clinician to prescribe anti-hypertensive
– in which the characteristics of an individual patient are
matched to a computerized knowledge base
• clerk routinely abstracts current BP, A1C, meds, allergies and
insurance status from paper chart into a database
• computer runs pt information against computerized guideline
• computer outputs a piece of paper with recommendation
– and patient-specific assessments or recommendations are
presented to the clinician and/or patient for a decision
• MD given piece of paper with individualized drug recommendation
• MD writes prescription in usual paper-based way
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Taxonomy of CDSSs
KNOWLEDGE/DATA SOURCE DECISION SUPPORT
Clinical knowledge source [ ] •Reasoning method
Patient data source [ ] •Clinical urgency
Data source intermediary [ ] •Recommendation
Degree of customization explicitness
Update mechanism •Logistical complexity
•Response requirement
CONTEXT INFORMATION DELIVERY
•Target decision maker •Delivery format
•Clinical setting •Delivery mode
•Clinical task •Action integration
•Unit of optimization •Delivery interactivity/explanation availability
•Relation to point of
care
OR
•Potential external
barriers to action
WORKFLOW
System user/ System Target
•Degree of workflow
Target decision user/Output decision
integration
maker intermediary [ ] maker
CDSS Characteristics
• Using taxonomy, reviewed and classified 42 RCT-
evaluated CDSSs
• Tremendous variation in decision-maker/context, how
recommendation delivered, staff needed to make
system run, complexity of recommended actions
– 45% targeted to clinician, 55% patient, 5% both
– 62% based on national guidelines or literature
– 69% “pushed” recommendation to decision maker
– 43% collected data directly from the EHR
• 45% required data input intermediary (11% MD)
– 26% required an output intermediary
• Generalizing successes from literature is difficult
(Berlin, Sim, 2006)
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
CDSS Effectiveness Summary
• Current data suggests CDSSs can improve the
process of care and perhaps clinical outcomes
– most effective at preventive care reminders
– modest at best for drug dosing and active care
– generally not helpful for improving diagnosis except with
trainees
• Findings limited by
– methodological problems and design type of studies
– insufficient appreciation of workflow component of CDSSs
– insufficient appreciation of heterogeneity of systems
• Bottom line: only moderate evidence of benefit
– limited generalizability of evidence
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Outline
• Decision support systems
– background, definition
– clinical versus research decision support
• How decision support systems “think”
– rule-based systems
– neural networks
• CDSS Effectiveness
• CDSS Adoption
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Low CDSS Adoption
• Adoption of CDSSs beyond simple reminders
– no play --> no gain
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Implications
• Clear trade-off between physician coding effort and
“smarter” decision support
• Don’t expect more decision support than coding allows
– generally successful decision support
• preventive care: age, last mammogram, etc.
• allergies: Yes/No on specific drugs
• drug dosing: weight, height, creatinine, age
– generally unsuccessful decision support
• diagnostic assistance
• complicated therapies (e.g., management of hypertension)
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206
Summary on Decision Support
• Most CDSSs are rule-based
• Moderate evidence of benefit
– workflow/organizational inputs underappreciated
• Fundamental trade-off between
– effort of coding data and quality of decision support
• Greater decision support adoption will require
– wider EHR use and better interoperability
• richer, usable, standard clinical vocabulary
• standard EHR format
• Need to be realistic on what decisions computers
can best support
March 11, 2008: I. Sim Decision Support Systems
Medical Informatics – Epi 206