Strategies for Researching the Impact of Pay for Performance Programs III

Evaluation of the NYS DOH Quality Incentive Program Stephen Zuckerman The Urban Institute 2007 AcademyHealth ARM Orlando, FL THE URBAN INSTITUTE Acknowledgements • Funded by The Commonwealth Fund • Qualitative component of project by Robert Berenson, Terri Coughlin, Brad Gray, Allison Cook and Mindy Cohen • Mindy Cohen also collaborated on the quantitative component and data development (today’s talk) • Sharon Long guided the evaluation design, but can’t be blamed for where we ended up THE URBAN INSTITUTE The QARR System • Quality Assurance Reporting Requirements • 1994 NYS DOH started collecting data on quality to members in commercial and MMC plans • Uses selected measures from HEDIS, some NYSspecific measures and CAHPS • All plans required to submit audited QARR results to DOH, which uses data for various purposes (e.g., public reporting since 1995) THE URBAN INSTITUTE The Quality Incentive Program • 2000 – auto-assignment partly based on QARR performance – complicated formula that included preference for prepaid health services plans (PHSPs) and QI performance (less important with most beneficiaries in plans) • 2001 – added premium bonuses; initially up 1% add-on, raised to 3% in 2004 • QI bonuses are based on composite scores on QARR and CAHPS THE URBAN INSTITUTE QI Methodology • 15 measures chosen each year – 10 from QARR and 5 from CAHPS • Some annual variation in QARR measures selected (which is not made until Sept of year being measured); consistency in CAHPS • QARR generally include preventive, prenatal, chronic care, children measures THE URBAN INSTITUTE QI Methodology (cont) • QARR scores from each plan compared to the 75th percentile of Statewide average of all participating plans from 2 years before. • Plan gets 10 points for each measure that meets or exceeds 75th percentile for QARR. (CAHPS slightly different – get 5 or 10 points). • Plans that meet a specific point threshold, set each year by DOH, can receive 25%, 50%, 75% or 100% premium bonus THE URBAN INSTITUTE Quantitative Questions • How did plan performance with respect to Medicaid enrollees change under the QI program? • How do trends in plan performance measures compare between Medicaid managed care enrollees and commercial managed care enrollees? • Controlling for other potential determinants of plan performance, is there evidence of an impact of the quality incentive program on Medicaid enrollees? THE URBAN INSTITUTE Quantitative Approach • Difference-in-differences framework: Medicaid versus commercial-only measures • A pre-post analysis that recognizes constraints imposed by number of plans and time period. • We allowed the Medicaid effect to vary with the share of plan enrollment from Medicaid • Models estimated with weighted least squares to produce robust standard errors THE URBAN INSTITUTE QARR Measures • Women’s Health Care: breast cancer screening (mammography), and postpartum care • Mental Health Care: ambulatory follow-up visits within 30 days of a hospitalization; effective antidepressant medication management (for 84 or 180 days) • Preventive Health Care: lead testing in children, visits to primary care physicians for children of different ages; and • Chronic Disease: diabetes HbA1c testing and poor control of diabetes THE URBAN INSTITUTE Data Development • NYS Quality Assurance Reporting Requirements • Area Resource File and Interstudy Data • NYS Enrollment Files found on website • NYS DOH did not document or store historical data in an easily accessible format, leading to a great deal of data prep THE URBAN INSTITUTE Findings • Medicaid worse than commercial in 8 out of 11 QARR measures in the pre-QI Period • Significant improvement for Medicaid in breast cancer screening and postpartum care • Adolescent access improved among plans with a high Medicaid share • But, Medicaid still lagged behind THE URBAN INSTITUTE Challenges (1) • Definitions of QARR measures changed over time - Well-child, immunizations, prenatal care • Small samples limited analytic questions (e.g. impact of NYC enrollees) – 28 Medicaid plans, 9 commercial only – HEDIS measures were being refined and rotated out in some years THE URBAN INSTITUTE Challenges (2) • Were there adequate controls for confounding policies? – MMC, SSI, public reporting, quality monitoring and improvement • Lack of enrollee or provider data from plans • QI program was evolving over time – Auto-assignment, 1% bonuses, 3% bonuses – Medicaid CAHPS started after QI program – Endogenity (avoided by looking at overall QI impact) THE URBAN INSTITUTE A P4P Evaluation Design – Unreal? • Create a payer-specific control group that does not get the incentive payment – Possibly, from another state • Keep the QARR/HEDIS measures defined consistently over time • Acquire more comprehensive plan-level data on enrollees and providers • Try P4P without other policies that could affect outcomes – Is this possible given market pressures? THE URBAN INSTITUTE