Early Evidence of the Leapfrog Group‟s Impact on Hospital Safety
2004 Academy Health Annual Research Meeting
Dennis Scanlon* Eric Ford
Penn State University
Jon Christianson
University of Minnesota * This project was supported by an Investigator Award in Health Policy Research from the Robert Wood Johnson Foundation
What is the Leapfrog Group and How is it Organized?
• Up to 98,000 Americans die each year from preventable medical mistakes they experience during hospitalizations, according to the Institute of Medicine
– Medical errors are a leading cause of death in America -- there are more deaths in hospitals each year from preventable medical mistakes than there are from vehicle accidents, breast cancer, or AIDS.
• Leapfrog, a coalition of more than 150 public and private organizations that provide health care benefits, is a voluntary program aimed at mobilizing large purchasers to alert the healthcare industry that big leaps in patient safety and customer value will be recognized and rewarded with preferential use and other intensified market reinforcements.
What is the Leapfrog Group and How is it Organized?
• Leapfrog works with medical experts to identify problems and propose solutions that will improve hospital systems. Leapfrog focuses on the quality of certain aspects of care relevant to urban area hospitals. Leapfrog‟s initial safety agenda included three areas:
– Computerized Physician Drug Order Entry Systems – Using Intensivists in hospital ICU‟s – Evidence based referrals for seven conditions and procedures
• Hospitals are invited to complete Leapfrog‟s web survey and share information with their communities about their efforts to reduce preventable medical mistakes. • Leapfrog has chosen a “Regional Rollout” strategy to advance its safety agenda in specific communities. To date, 24 RROs have been selected across four waves.
Objectives of the RRO Case Study Project
• Describe/Document RRO structure and activities • Obtain feedback from various stakeholders belonging to or affected by the RROs • Provide feedback to Leapfrog and the RRO leaders • Evaluate the impact of the RRO strategy for accomplishing Leapfrog‟s objectives – Realize the capabilities and limitations of voluntary collective health purchasing efforts more broadly
Study Design
• Multiple case studies where the RRO is the case • Data sources – Interviews with key stakeholders in each RRO – Secondary documents (e.g., RRO application, news articles) – Leapfrog hospital disclosure results – Secondary data sources (e.g., Interstudy and AHA data) • Developed (with peer review) five protocols for common key stakeholders – RRO leader – Hospital executive – Health plan executive – Employer/purchaser – Benefits consultant
Study Design
• „Logic Model‟ developed to guide the analysis (Sofaer et al.) • Selected 7 RROs and identified key stakeholders for interview • 2 non Leapfrog RRO „control‟ sites to be added • Generalizability • 4 researchers conducted interviews - two in each market – tape recorded and summary notes produced by each interviewer • Analysis of qualitative and secondary data by team of five researchers using standard approaches • Preliminary results provided to RRO leaders for feedback and accuracy verification • Produce final reports and products
Leapfrog Regional Rollout Logic Evaluation Model
Assumptions
•The Institute of Medicine‟s „Errors‟ and „Quality‟ reports has opened a window of opportunity to focus on health care safety and quality in the United States. Private and public sector purchasers can be catalysts for increased emphasis on patient safety by identifying a small number of safety leaps that evidence suggests will reduce the number of errors and deaths occurring in hospitals. Purchasers will also shift their focus from a pure dollar perspective to a „value‟ perspective. The Leapfrog Group is the formal entity that will organize purchasing activity around patient safety issues. The Leapfrog group will do this by identifying the relevant safety „leaps‟ and by enlisting and supporting purchasers in their efforts to influence/change hospital behaviors and decisions. Health care delivery is „local,‟ therefore a successful national safety initiative must recognize the localized nature of hospital care and health services delivery. The Leapfrog Group is intended to be a short-term rather than perpetual organization, with operations ceasing once patient safety has received significant attention by relevant stakeholders.
Leapfrog Central Activities
•Determines the patient safety leaps based on expert advice and scientific evidence. Translates and communicates the leaps to purchasers and hospitals by providing information and technical assistance/support. Develops (in conjunction with Medstat) a hospital survey to measure hospital progress on the recommended leaps. Organizes purchasing power by recruiting individual public and private sector purchasers and selecting regions and leaders willing to promote the Leapfrog standards in local communities. Engages various stakeholder groups (e.g., hospitals, employers, public sector purchasers, health plans, consumers, regulators) in feedback and discussion regarding patient safety generally and the chosen leaps specifically. Advocates for patient safety nationally through media exposure and by developing a reputation as the „authoritative watchdog‟ on hospital patient safety.
RRO Mission
•Agree to Leapfrog‟s purchasing principles and accept the role as Leapfrog‟s „agent‟ in the local market. Develop a work plan for Leapfrog activities including goals and objectives, staffing, resources and a timeline. Choose the approach (e.g., punitive or collaborative) to engaging and influencing hospitals‟ behavior. Identify and garner local support in the purchasing community for the Leapfrog Group nationally, and for the local Leapfrog initiative.
RRO Process Activities
•Hospital completion of Leapfrog survey indicating status regarding Leaps Number and participation of local employers recruited to support RRO effort Tangible and significant health plan support for RRO efforts Public announcement of Leapfrog imitative in region.
Intermediate Outcomes
Hospitals •Increased number of hospitals adopt CPOE Increased number of hospitals use intensivists Fewer EHR surgeries performed at low volume hospitals Hospital admissions increases at compliant hospitals and declines at non-compliant hospitals Employers •Improved employer understanding of patient safety and medical errors, and awareness of variation across local hospitals in performance regarding patient safety •Employers provide financial incentives that reward use of Leapfrog compliant hospitals Employers contract with health plans that support Leapfrog standards Employers and purchasers educating their employees on hospital safety. Health Plans •Health plans develop hospital contracting incentives linked to Leapfrog leap compliance.
Final Outcomes (Goals)
•Medical errors in community hospitals are reduced and patient risk of being harmed by adverse hospital events declines Hospital costs become more „efficient‟ as cost savings from error reduction exceed costs of implementing the leaps.
Leapfrog Regional Rollout Logic Evaluation Model
Assumptions
•The Institute of Medicine‟s „Errors‟ and „Quality‟ reports has opened a window of opportunity to focus on health care safety and quality in the United States. Private and public sector purchasers can be catalysts for increased emphasis on patient safety by identifying a small number of safety leaps that evidence suggests will reduce the number of errors and deaths occurring in hospitals. Purchasers will also shift their focus from a pure dollar perspective to a „value‟ perspective. The Leapfrog Group is the formal entity that will organize purchasing activity around patient safety issues. The Leapfrog group will do this by identifying the relevant safety „leaps‟ and by enlisting and supporting purchasers in their efforts to influence/change hospital behaviors and decisions. Health care delivery is „local,‟ therefore a successful national safety initiative must recognize the localized nature of hospital care and health services delivery. The Leapfrog Group is intended to be a short-term rather than perpetual organization, with operations ceasing once patient safety has received significant attention by relevant stakeholders.
Leapfrog Central Activities
•Determines the patient safety leaps based on expert advice and scientific evidence. Translates and communicates the leaps to purchasers and hospitals by providing information and technical assistance/support. Develops (in conjunction with Medstat) a hospital survey to measure hospital progress on the recommended leaps. Organizes purchasing power by recruiting individual public and private sector purchasers and selecting regions and leaders willing to promote the Leapfrog standards in local communities. Engages various stakeholder groups (e.g., hospitals, employers, public sector purchasers, health plans, consumers, regulators) in feedback and discussion regarding patient safety generally and the chosen leaps specifically. Advocates for patient safety nationally through media exposure and by developing a reputation as the „authoritative watchdog‟ on hospital patient safety.
Leapfrog Regional Rollout Logic Evaluation Model
RRO Mission
•Agree to Leapfrog‟s purchasing principles and accept the role as Leapfrog‟s „agent‟ in the local market. Develop a work plan for Leapfrog activities including goals and objectives, staffing, resources and a timeline. Choose the approach (e.g., punitive or collaborative) to engaging and influencing hospitals‟ behavior. Identify and garner local support in the purchasing community for the Leapfrog Group nationally, and for the local Leapfrog initiative.
RRO Process Activities
•Hospital completion of Leapfrog survey indicating status regarding Leaps Number and participation of local employers recruited to support RRO effort Tangible and significant health plan support for RRO efforts Public announcement of Leapfrog imitative in region.
Leapfrog Regional Rollout Logic Evaluation Model
Intermediate Outcomes
Hospitals •Increased number of hospitals adopt CPOE Increased number of hospitals use intensivists Fewer EHR surgeries performed at low volume hospitals Hospital admissions increases at compliant hospitals and declines at non-compliant hospitals Employers •Improved employer understanding of patient safety and medical errors, and awareness of variation across local hospitals in performance regarding patient safety •Employers provide financial incentives that reward use of Leapfrog compliant hospitals Employers contract with health plans that support Leapfrog standards Employers and purchasers educating their employees on hospital safety. Health Plans •Health plans develop hospital contracting incentives linked to Leapfrog leap compliance.
Final Outcomes (Goals)
•Medical errors in community hospitals are reduced and patient risk of being harmed by adverse hospital events declines Hospital costs become more „efficient‟ as cost savings from error reduction exceed costs of implementing the leaps.
RRO Case Study Sample
RRO Site Atlanta Dallas Rochester 1st Wave
Using 2nd Wave Employer Coalition Health Led Led Plan Led Incentives
Savannah
Seattle St. Louis
Wisconsin
Interviews Conducted
(* = multiple respondents in some interviews)
Respondent Type Rochester , NY Dallas – Ft. Worth, Texas Savannah, Georgia Seattle, Washington St. Louis, Missouri Atlanta, Georgia Wisconsin
Total
RRO Lead
1 3 1 1 3
1 5* 0 1 2
1 6* 0 1 1
1 3 3* 1* 5*
1* 4 1 1* 3
1 6 2 1 2
1 4 4* 1 1
7 31 11 7 17
Hospital Administrator Health Plan Administrator Benefits Consultant Employer
Other
3 12
2* 11
1 10
3* 16
2 12
4 16
2 13
17 90
Total
Atlanta Market Factors
Characteristics
Number of hospital systems # Percentage of hospital beds controlled by the three largest hospital systems # Percentage of hospitals# controlled by a for-profit system Percentage of staffed beds controlled by the three largest hospitals # Average hospital occupancy rate* # Average hospital occupancy rate at the three largest hospitals* # 7 50.8% 34.6% 23.0% 62.1% (24.0 – 86.0%) 77.8% (71% - 84%)
Number of employees of the three largest private Atlanta employers
Percentage of employees enrolled in PPOs
58,851
30%
Percentage of employees enrolled in HMOs
Sources: AHA Guide, 2002. Atlanta Business Chronicle, 2003. Atlanta Business Journal, 2002. * = Does not include newborns. # = Includes only Leapfrog urban and targeted hospitals
47%
Rochester RRO Organization
Rochester Health Commission
12 Initiatives • Community-wide clinical guidelines • Patient Safety • Healthcare worker shortage
Twelve Initiatives
• Monroe County Health Action • End-of-life / Palliative care • LEAPFROG • Addressing Chronic Diseases • Efficient and Effective Use of Resources • Increasing Administrative Efficiency
Leapfrog Initiative
Ed Black, MD / Cathy Kausch Excellus – RRO Lead
• Increasing Access to Quality Services • Benchmarking and Reporting Performance
Community-wide advisory committee • Two major health insurers • Key business groups • Three large urban hospital • Local employers systems • Monroe County Medical Society • Smaller rural hospitals
New Leaps Subcommittee Physician Education Subcommittee Consumer Education Subcommittee IPS Leap Subcommittee Communications Subcommittee Information Subcommittee
How Should Success be Measured???
• Increased collaboration among hospitals and payors in the community
– Awareness and assessment by various stakeholders
• % of targeted hospitals that disclose to the Leapfrog survey • % of hospitals compliant on the various leaps • % of hospitals making real progress towards leap adoption • % of admissions to hospitals meeting relevant leaps
– Degree to which purchasers/plans incent use of Leapfrog compliant hospitals
Hospital Comments about the Leapfrog Group (examples)
• “I have seen no appreciable change in the community as a whole regarding Leapfrog, but our hospital will purchase a CPOE system at least in part due to the impetus of Leapfrog” • “Leapfrog has really helped getting some of the leader‟s minds at least directed towards quality as an important issue to payers” • “It‟s instigated change in the provide community vis-à-vis physician order entry and increased focus on outcomes. It hasn‟t really changed purchaser behaviors, but I‟m not sure it was designed to.” • I applaud the payers demanding safety, but I still don‟t understand how they picked the leaps (e.g., esophageal surgery volume) relative to other viable candidates (e.g., smart pumps)
Hospital Comment about the RROs
“If their (the RRO) true goal is to improve quality, they did get people‟s attention. They also caused hospitals to form several committees that are sharing information. The most improvement I think it has spawned relates to ICU care. I believe they are measuring important things internally and seeing improvement in many hospitals. Of course, these measures are not part of Leapfrog, because they (Leapfrog) are not measuring things that are important relative to patient care, just what they can easily promote and reference some literature with. Interesting though, to fight where Leapfrog wanted to go, the hospitals and doctors have gotten together, looked at articles, and are busy trying on their own to improve quality. Frankly, I think going at this directly would have been more successful, but will hand it to them (the RRO and LF), they have found a way to hype their effort, and it has helped push some people along.”
Hospital Survey Reporting by RRO
RRO Criteria First Wave
(As of December 2001)
Second Wave
(As of December 2002)
Third Wave
(As of January 2004)
Seattle, Washington
Percentage Hospitals Responded Percentage of responding hospitals in compliance with one or more Leaps
92% (N = 25) NA NA NA 51.6% (N = 31)
100% (N = 25) 55% (N = 25) NA NA 73.3 (N = 30)
100% (N = 25) 52.0% (N = 25) 81.6% (N = 49) 32.5% (N = 40) 67.9% (N = 28)
Wisconsin
Percentage Hospitals Responded Percentage of responding hospitals in compliance with one or more Leaps
Atlanta, Georgia
Percentage Hospitals Responded
Percentage of responding hospitals in compliance with one or more Leaps
Savannah, Georgia Percentage Hospitals Responded Percentage of responding hospitals in compliance with one or more Leaps
Source: The Leapfrog Group, January 2004
NA
NA NA
55% (N = 22)
100% (N = 3) 100% (N = 3)
52.6% (N = 19)
60% (N = 5) 100% (N = 3)
Dallas Ft. Worth: Evidence Based Hospital Referral.
57 Hospitals Targeted 36 Hospitals Responded
Procedure (LeapFrog Standard) (Applies to N Hospitals)
Coronary Artery Bypass (450) (N=19)
Percutaneous Coronary Intervention (400) (N=21) Abdominal Aortic Aneurysm Repair (50) (N=21) Pancreatic Resection (11) (N=15) Esophagectomy (13) (N = 10) High Risk Deliveries & Neonatal ICUs (15) (N = 19)
Distribution Of Procedures / Census Among Hospitals
1 performs more than 500
11 perform more than 400
5 perform 449 – 300
3 perform 300 – 400
3 performs 200-300
4 perform 200-300
10 perform less than 200
3 perform less than 200
5 perform more than 50
2 perform more than 11 0 performs more than 13 9 have more than 15
2 performs 40-50
1 performs 9-11 0 perform 9-13 2 have 10-15
0 perform 30-40
0 perform 7-9 0 performs 7-9 1 has 8-10
14 perform less than 30
12 perform less than 7 10 perform less than 7 6 have less than 8
In your view, what has been the impact of the Leapfrog safety agenda in your market? Please mark the answer that best characterizes your view.
Category A very credible process having a significant and positive impact on safety. A credible process that has stimulated meaningful discussion and some impact on safety in our market. A credible process, but meeting some resistance in the marketplace and having little impact. The Leapfrog process has been strongly resisted and has no impact. Did not answer Number of respondents 12 27 % of respondents 25.5 57.5
7
14.9
1
9
2.1
16.1
RRO Performance Ratings By Location
Performance Rating Rochester, NY (N = 12) Dallas – Ft. Worth, Texas (N = 11) Savannah, Georgia (N = 10) Seattle, Washington (N = 16) St. Louis, Missouri (N = 12) Atlanta, Georgia (N = 16) Wisconsin (N = 13)
Very Successful Mostly Successful Moderately Successful
2(17%)
0 (0%)
0 (0%)
2 (13%)
0 (0%)
0 (0%)
1 (8%)
5 (42%)
3 (27%)
2 (20%)
1 (6%)
1 (8.3%)
0 (0%)
2 (15%)
4 (33%)
7 (64%)
4 (40%)
4(25%)
5 (42%)
8 (50%)
7 (54%)
Not Very Successful
Not Asked
0 (0%)
0 (0%)
1 (10%)
4 (25%)
4 (33%)
7 (44%)
2 (15%)
1 (8.3%)
1 (9%)
3 (30%)
5(31%)
2 (17%)
1 (6%)
1(8%)
Summary (preliminary)
• Leapfrog has captured hospitals‟ attention but impact on final outcomes is hard to discern (still to early to tell) – Different response to the three leaps • Lack of broad purchaser involvement in many markets. – Activity led by small number of individual corporate champions or coalitions – Small share of hospital admissions represented by RRO employers • RROs captured by hospitals in some markets? • Strict adherence to leaps vs. local development • Relationship to other patient safety efforts (crowding out or being crowded out) • Rural hospital concerns • Health plan response • Sentinel events
Future Work and Caveats
• Add control sites and additional RRO sites to test hypotheses • Assess impact with expanded leaps • Quantitative analysis of hospital reporting over time • A true assessment will take time!