Overview of 2004-2005 External Quality Review (EQR) Activities by 0n9axIhb

VIEWS: 0 PAGES: 43

									      HSAG Performance Improvement
   Projects Using Data to Develop Interventions
      and Statistical Testing to Evaluate Results

                          Breakout Session #1
                     Florida EQR Quarterly Meeting
                              June 18, 2008




Presented by:
Donald Grostic, MS
Associate Director, Research and Analysis
Team
Yolanda Strozier, MBA
EQRO Project Manager
Intervention Cycle Framework
                                      Data Collection (CMS Protocol Activity VI)



 Statistical Testing and                                              Data Mining and
 Linking Intervention                                                 Causal/Barrier Analysis
 to Outcomes

                             Evaluate           Identify
                                                           CMS
                    CMS Activity                           Activity
                    VII,VIII, IX, X                        VII,VIII



                           Implement               Plan
   Three Tips                CMS                    CMS
                             Activity
                                                                       Steps for
                                                    Activity
                             VII, VIII              VIII               Intervention
What does the intervention cycle have to
      do with CMS PIP Activities?
                       Identify   Plan   Implement Evaluate

                        ☑         ☑        ☑         ☑
    Activity 7
     Assess the
    Improvement
      Strategy


                        ☑                  ☑         ☑
    Activity 8
     Review Data
      Analysis &
   Interpretation of
        Results


                                                     ☑
    Activity 9
      Reported
   Improvement is
       Real?


                                                     ☑
    Activity 10
     Sustained
   Improvement?
The ‘Identify’ Stage



           Identify
                  Data Mining
•    What is data mining?

Answer:

     Data mining is the process of sorting
      through large amounts of data and picking
      out relevant information.
        Data Mining (continued)
•    What is data mining used for?

Answers:

     Data mining is the statistical and logical analysis
      of large sets of data, looking for patterns of care,
      or service delivery that can aid decision making.

     To identify and determine areas of non-
      compliance that will be analyzed during the
      causal/barrier analysis.
Data Mining vs. Data Analysis Plan
 •    How does data mining differ from a data
      analysis plan?

 Answer:

      A data analysis plan includes calculating and
       comparing overall indicator rates between
       measurement periods using statistical testing.

      Data mining will include analysis that goes beyond
       just calculating and comparing indicator rates
       between measurements.
        Data Mining–Example
PIP topic (clinical):

Follow-up after acute care inpatient hospitalization.


Indicator:

The percentage of members with follow-up within 7 days
after acute care discharge for a mental health diagnosis.
Data Mining Example Step One
•   Group the population or sample.

First, group members by county or ZIP code. For our
     example, the population breaks into three counties:
    County A, County B, and County C.
Data Mining Example Step Two
•   Calculate compliance and noncompliance for
    each county.

The percentage compliant and noncompliant by county are
    presented in the following table.
                      Percentage    Percentage Non-
                      Compliant        Compliant
    County A             65%             35%
    County B             35%             65%
    County C             20%             80%



Question: Which county should you data mine further?
Data Mining Example Step Three
 •   Identify groups where the majority of members are
     noncompliant.
 Answer:
    First we need to know how many members of the population are
     in each county.
    Selecting County B will have the greatest effect on the compliance
     rate because it has the majority of the population and the second
     lowest compliance rate.

                 Percentage        Percentage           Number of
                 Compliant        Non-Compliant         Members
County A             65%                35%                 80
County B             35%                65%                 220
County C             20%                80%                 20
Data Mining Example Next Steps
•   Now that you have identified County B,
    what should you do next?

Answer:
   Continue the process of grouping and selecting to
    find the group that will have the greatest effect on
    compliance.
   For County B, you may consider grouping the data
    by PCP or facility next.
          Data Mining Caution!
Words of caution:

   Grouping and selecting can be taken to a point where the groups
    selected may be too small to make an impact.

   Always keep in mind the number of members affected in the
    selected group relative to the total population.

   If there is difficulty identifying noncompliant groups or non-
    compliance is equally distributed among groups, you may be
    dealing with a systemwide issue.

   Please keep in mind that data mining is a dynamic, iterative
    process that takes practice.

   The more you data mine the better you will become at selecting
    groups that yield the best effect on rates.
Questions and Answers
   What is a Causal/Barrier Analysis?

• A causal/barrier analysis is:
   – A systematic process for identifying the problem.
   – A method for determining what causes the barriers.
   – A way to identify what improvement opportunities are
     available.
• Causal/barrier analysis has also been called:
   – Root cause analysis
 How do I perform a causal/barrier analysis?

Determine why an event or condition occurs.
1. What is the problem?
     - Define the problem and explain why it’s a
       concern.
2. Determine the significance of the problem.
     - Look at the data and see how the problem
       impacts your members and/or health plan.
  How do I perform a causal/barrier analysis?
                   (cont.)

3. Identify the causes/barriers.
     - Conduct analysis of chart review data,
       surveys, focus groups.
     - Brainstorming at quality improvement
       committee meetings.
     - Literature review.
4. Develop/implement interventions based on
   identified barriers.
   Causal/Barrier Methods and Tools

• Methods:
   –   Quality improvement committees
   –   Develop an internal task force
   –   Focus groups
   –   Consensus expert panels
• Tools:
   –   Fishbone
   –   Control chart
   –   Flow chart (process mapping)
   –   Barrier/intervention table
Questions and Answers
The ‘Plan’ Stage




         Identify




          Plan
                 A Physical Health Example

                        Data                          Providers

                                   Outreach
                                                                  Knowledge
Data accuracy
                                      Demographic
                                      changes
                                                                     Compliance
    Data completeness


                                                                                      Low Well Visit
                                                                                         Rates
                                       Compliance
                                                                      Billing
        Demographics                                                  Well vs. Sick
                                         Medical
                                         Records
      Transportation


                                 Knowledge    Paper    EHRs


                       Members                        Systems


         What questions could be asked to drill down these causes?
         What data are needed to identify the most crucial cause?
         A Mental Health Example
• Discharge planning
   – Client
   – Communication
   – Transportation
   – Community involvement
• No follow-up appointment set at time of discharge
• Time lag/claim data
• Not client focused
• Provider access
• Culture change
• Demographic information
     What questions could be asked to drill down these causes?
     What data are needed to identify the most crucial cause?
         Interventions Checklist

 Analyze barriers (root causes)
 Choose and understand target audience
 Select interventions based on cost/benefit
 Implement interventions
 Track intermediate results (optional)
 Remeasure
 Modify interventions as needed
Questions and Answers
The ‘Implement’ Stage




                Identify




                Plan
    Implement
          The ‘Implement’ Stage

Three tips:
1. Observe and document whether the intervention
   is implemented as intended
2. Note any lesson(s) learned
3. Document any change(s) that may threaten the
   results between measurement periods
  –   Methodology (e.g. definition of indicators, sampling)
  –   Circumstances (e.g. merger, population, provider)
Questions and Answers
The ‘Evaluate’ Stage: Statistical Testing




                      Identify
           Evaluate




                       Plan
       Statistical Significance Testing


• What is statistical testing and why do we use it?

Answers:
    Statistical testing is calculating specific test statistics
     and associated p values to determine if an observed
     difference is a true difference and not due to chance
     alone.
    The CMS Protocols require that statistical testing be
     used to prove that any improvement in rates is real.
    Without statistical testing, a PIP would not meet the
     CMS Protocols.
        Statistical Significance Testing


• What type of statistical testing is appropriate for my
  PIP?


Answer:
 Fisher’s Exact Test or Chi-square test for rates or proportions.
 T test for means would be the appropriate statistical testing.
        Statistical Significance Testing


• What type of statistical testing is appropriate for this
  indicator?
Indicator A: The percentage of members with follow-up within
   7 days after acute care discharge for a mental health diagnosis.


Answer:
 Fisher’s Exact Test or Chi-square test for rates or proportions.
        Statistical Significance Testing


• What is the difference between
  Fisher’s Exact Test and a Chi-square test?
Answer:
 Fisher’s Exact Test will provide the exact p value while the Chi-
  square test is an approximation of the p value.
 As your numerators and denominators increase in size, the
  Chi-square test and Fisher’s Exact Test produce the same p
  value.
 If in doubt about which test to use, use Fisher’s Exact Test.
        Statistical Significance Testing


• What type of statistical testing is appropriate for this
  indicator?
Indicator B: The average response from a member satisfaction
   survey where answers range from 1=satisfied to 5=dissatisfied.


Answer:
T test for means would be the appropriate statistical testing.
       Statistical Significance Testing


• How do I report statistical significance testing results?

Answer:
  When using a Fisher’s Exact Test, Chi-square test or a t test,
  report the test used, its associated p value along with each
  indicator, and its numerator and denominator in tabular form.
                 Statistical Significance Testing

Indicator A: The percentage of members with follow-up within
7 days after acute care discharge for a mental health diagnosis.


  Time      Measurement Periods   Numerator    Denominator    Rate or     Industry   Statistical Testing and
 Periods                                                      Results    Benchmark        Significance


CY 2003    Baseline               20          41             48.8%      60%          N/A


CY 2004    Remeasurement 1        27          51             52.9%      60%          Fisher’s Exact Test
                                                                                     P value = 0.8340
                                                                                     Chi-square test
                                                                                     P value = 0.8517
                                                                                     NOT SIGNIFICANT
                                                                                     AT THE 95%
                                                                                     CONFIDENCE
                                                                                     LEVEL
                 Statistical Significance Testing

Indicator B: The average response from a member satisfaction
survey where answers range from 1=satisfied to 5=dissatisfied.


  Time      Measurement Periods   Numerator    Denominator    Rate or     Industry   Statistical Testing and
 Periods                                                      Results    Benchmark        Significance


CY 2008    Baseline               253         100            2.53       N/A          STD DEV = 1.298


CY 2009    Remeasurement 1        371         113            3.28       N/A          STD DEV = 1.561
                                                                                     T-test
                                                                                     P value = 0.0002
                                                                                     SIGNIFICANT AT
                                                                                     THE 95%
                                                                                     CONFIDENCE
                                                                                     LEVEL
       Statistical Significance Testing


• If I use the entire population for my study, do I still
  have to do statistical significance testing?

Answer:
   Yes. It is appropriate to do statistical
   testing on the entire eligible population.
 Reasons for Statistical Significance Testing on
              Entire Populations

• CMS is interested in performance over time.
• The population will continuously change over time.
• The members who are studied in one year may or may not
  appear in the following years.
• A population that is selected at one point in time is a sample
  from the true population that contains all members.
• The entire eligible population for a measure in one year is a
  sample population drawn from a universe of “all years” or “all
  populations” that could be selected.
• CMS has approved statistical testing on populations.
Questions and Answers
The ‘Evaluate’ Stage: Linking
  Intervention to Outcomes

                   Revise




       Evaluate    Identify




                   Plan
       Implement
        The ‘Evaluate’ Stage: Linking
          Intervention to Outcomes
                        Threats to internal/external validity: Any
                        environmental, organizational, methodological
                        changes between measurement periods?
                                  No                         Yes
Outcome: Improved       • Intervention seems to    • Cannot ascertain if the
                        be effective               improvement really is
                        • Consider standardizing   due to intervention
                        the intervention to        • Investigate the
                        subsequent                 relationship between the
                        measurement periods        change circumstances
Outcome: No Change or •Intervention does not       and the outcomes
Worsens               seem to be effective
                      • Consider revising the
                      intervention to
                      subsequent
                      measurement periods
The ‘Evaluate’ Stage: Linking
  Intervention to Outcomes

                    Revise




       Evaluate     Identify




        QI
                    Plan
        Implement
Questions and Answers

								
To top