“e-indicators” of Ambulatory Care
Quality: New Paradigms for
Measuring Clinical Performance
Using Electronic Health Records
Presented at the Academy Health, Washington DC June 10 2008
Jonathan P. Weiner, DrPH
Department of Health Policy & Management
Johns Hopkins Bloomberg School of Public Health
Topics to be discussed:
• Ideas and paradigms regarding health IT
(HIT) as applied to ambulatory / population
based quality performance measurement and
• Findings from a multi-faceted collaborative
project to develop electronic health record
(EHR) based “e-indicators.”
• Future challenges and opportunities of EHR
based quality measurement.
Embracing new HIT capabilities to measure quality in
the ambulatory care sector: The “e-indicator” project
Goal: To develop innovative measures of quality that take
advantage of new HIT capabilities and data sources.
Development effort based at Johns Hopkins and Park Nicollet Institute:
– Collaboration involving leading edge “wired” integrated delivery
systems: Park Nicollet (MN), Kaiser Permanente (OR), Geisinger
(PA), Health Partners (MN) Billings (MT), Dupont/Nemours (DE),
Boston CHC network (MA).
– Involves both an “adult” project focusing on chronic illness and a “child
project” focusing on screening and development
– Advisors include key quality groups, NCQA, NQF, AMA, Medicare,
VA, ONCHIT, AHRQ, medical specialty societies
– Funded by Commonwealth Fund, Robert Wood Johnson, US AHRQ,
– Methods include lit review, survey of experts & early adopters, case
study, development of starter-set of “e-indicators”.
Applications of HIT-based performance
• Quality improvement for organizations
– Real time (safety / care management)
– Retrospective evaluation
• “Pay for Performance” (P4P) incentives
• Community / regional reporting and strategic planning
HIT will transform performance measurement
• e-Indicators will be essential tool for:
– Provider / clinician teams /organizations
– Health plan / sponsors
– Government / communities / public health
– Outcomes researchers / scientists
Data sources and types of quality /
Type of Measure
Data Source: Denominator Process Outcome Pt-Cent. Cost
Electronic / HIT
PH records / registry X
Insurance files X X X X
EHR X X X X
CPOE (order entry) X
PHR / Pt. “web portal” X X X
CDSS (clinical support) X X
Paper medical record X X
Surveys X X
A proposed typology for
HIT based “e-indicators” of quality
1) Translational: Traditional (e.g., paper record and
claims) measures translated for use on HIT platforms
2) HIT-facilitated: Measures that while not conceptually
limited to HIT, would not otherwise be feasible.
3) HIT-enabled: Measures that generally would not be
possible outside of EMR/EHR context.
4) HIT system management: Measures needed to
implement, manage and evaluate HIT systems
5) “e-iatrogenesis”: Measures of patient harm caused
at least in part by application of HIT.
Examples of each type of e-indicator
- Computerization of existing HEDIS measures
- % of children > BMI of x receiving intervention
- % of patients with e-prescriptions who did not pick up their
Rx within x days
- % of clinicians reviewing and responding to abnormal lab
value with x hours
- % of consumer generated web-based shared care plans
accessed by both generalist & specialists within 6 months
- % of in scope care that is routed through CDSS supported
Examples of each type of e-indicator -
4) HIT system management:
- Attainment of EHR interoperability targets
- % of CDSS alerts ignored by clinicians
- % of allergy lists updated by patient annually
- % of e-prescriptions that result in wrong drug
- % of patients needlessly exposed to imaging
radiation due to inappropriate use of CDSS module
Broad observations re EHRs and quality
• The needs of quality measurement have not been
adequately addressed by most current EHR / HIT
• The most useful quality measures will be hard-wired
into HIT “workflow” or “streamed” automatically.
• Integration of real-time clinical decision support and
provider order entry (e.g., e-prescribing) represent a
major paradigm shift for quality measurement.
• To date, wired organizations have replicated paper
records and claims data methods and approaches.
• Outside of IDSs, e-indicators at person and
population level are not feasible without EHRs
Advantages of EHR based measures
• Data elements that are “streamed” or part of structured
e-workflows are likely to be most accurate.
• 100% electronic census will reduce bias associated with
sampled chart reviews and surveys.
• Abstraction errors due to poor handwriting will be
• There are rich new sources of information in the domains
of time and information flow.
Challenges of EHR based measures
• The process of entering, transferring, archiving, and
analyzing EHR data introduce numerous opportunities for
• There are new types of errors associated with provider
behaviors (e.g., cut and paste “plagiarizing.”)
• CDSS or workflow algorithms can introduce and
perpetuate measurement errors by systematizing them.
• For foreseeable future, analysis of free-text section of EHR
will be fraught with difficulty.
Some recommendations moving forward
• Quality measurement must be part of workflow and HIT
system design from the start.
• We need to develop a better understanding of how new
paradigms and data sources affect reliability and validity.
• For foreseeable future HIT “systems management”
indicators should be central part of IT CQI process.
• EHR / HIT systems offer great opportunities for population
and person based measures (moving beyond those that are
provider or patient based).
• Although there will likely be considerable positive benefit,
we must learn how to measure and monitor “e-iatrogenesis”
Further information / Acknowledgements
If you wish to receive copies of our project
manuscripts please give me your card or send
me an e-mail at firstname.lastname@example.org
I would like to acknowledge co-authors of this
– Jinnet Fowles
– Kitty Chan
– Betsy Kind