Assessing study quality for a systematic review - Leuven

Document Sample
Assessing study quality for a systematic review - Leuven Powered By Docstoc
					Assessing study quality for a
     systematic review
                Trudy Bekkering, PhD
        Center of Evidence-Based Medicine &
    Belgian Branch of the Dutch Cochrane Center
   Centre for Methodology of Educational Research
       Katholieke Universiteit Leuven, Belgium
        Why is it so important?
• Meta-analysis aims to increase precision
• Meta-analysis of studies with bias in results
  gives very precise but wrong results
• Garbage in, garbage out
        Bias versus imprecision
Bias:
• A systematic error in the results or the
  inferences
• Methodological flaw
• Overestimation or underestimation
Bias versus imprecision


Ideal study   BIAS
        Bias versus imprecision
Imprecision:
• A random error in the results
• Sample variation
• Direction of error is random
Bias versus imprecision


Ideal study   IMPRECISION
Bias versus imprecision


BIAS + IMPRECISION
Bias versus imprecision


BIAS ? IMPRECISION?
     Risk of bias versus bias
• Clear empirical evidence that particular flaws
  in study design can lead to bias.
• Usually impossible to know to what extent
  biases have affected the results.
• Key consideration = should the results be
  believed
  Tools for assessing study quality

Scales: discouraged
Checklist: between 3 and 57 items
Cochrane tool: “domain based”

Depends on study design
      Tool for RCTs: Cochrane tool
Risk of bias on 6 domaines:
1.   Sequence generation
2.   Allocation concealment
3.   Blinding
4.   Incomplete outcome data
5.   Selective reporting
6.   Other sources of bias
       Risk of which biases?

                   Differences between baseline characteristics of
Selection bias     the groups compared

                   Differences in the care that is provided, or in
Performance bias   exposure to other factors than the intervention


Attrition bias     Differences in withdrawals from a study


Detection bias     Differences in how outcomes are determined


                   Differences between reported and unreported
Reporting bias     outcomes
How do you assess?
       Domaine                       Description                           Judgement
                          QUOTE: “patients were randomly allocated”
                          COMMENT: probably done, since earlier
Sequence generation       reports of this study describe use of random
                                                                         YES (low risk of bias)
                          sequences

                                                                         YES (low risk)
Allocation concealment                                                   NO (high risk)
                                                                         UNCLEAR (uncertain)

                                                                         YES
Blinding                                                                 NO
                                                                         UNCLEAR

                                                                         YES
Incomplete outcome data                                                  NO
                                                                         UNCLEAR

                                                                         YES
Selective outcome                                                        NO
reporting                                                                UNCLEAR

                                                                         YES
Other sources of bias                                                    NO
                                                                         UNCLEAR
           Sequence generation

       Adequate                  Not adequate                    Unclear
l Random number table       Generated by:                    If insufficient
                            l Odd/even date of birth         information about
l Computer generated list   l Date/day of admission          sequence
                                                             generation!
                            l Hospital record number
l Coin tossing
l Shuffling cards /         Allocation by:
  envelopes                 l Clinical judgement
l minimization              l Participant’s preference
                            l Result of lab test
                            l Availability of intervention
 Allocation concealment

      Adequate                 Not adequate               Unclear
l Central randomisation    l Open random allocation   If insufficient
  (telephone, web-based,     schedule                 information about
  pharmacy)                                           sequence
                           l Envelopes without        generation!
l Sequentially numbered      appropriate safeguards
  drug containers of                                  (e.g. “sealed
  identical appearance     l Quasi-randomisation      envelopes”)

l Sequentially numbered,
  opaque, sealed
  envelopes
        Blinding of intervention

• Participants (patients, clients)

• Care providers (doctors, nurses, teachers …)

• Outcome assessors
    Blinding of intervention

      Adequate                  Not adequate                Unclear
l No blinding, outcome      l No or incomplete           l insufficient
  (assessment) not            blinding and outcome         information
  likely to be influenced     (assessment) likely to
  by lack of blinding         be influenced by lack of   l Issue not
l Blinding ensured and        blinding                     addressed in
  unlikely to have been                                    the study
  broken                    l Blinding attempted but
l Either participants or      likely that could have
  some personnel              been broken
  unblinded, but unlikely
  to introduce bias +
  outcome assessment
  blinded
Incomplete outcome data


“Attrition” (drop-out): no data


       • Withdrawal
       • Do not attend follow-up appointment
       • Failure to complete questionnaire / diaries
       • Cannot be located (lost to follow-up)
       • Decision by investigator to cease follow-up
       • Data or records are lost
Incomplete outcome data


“Exclusion”:         data available, but excluded from analysis



   • Participants found to be ineligible after enrolment

   • An “as treated” (or per-protocol) analysis: participants
      are only included if they received the intended intervention in
      accordance with the protocol
 Assessing risk of bias

       Low risk of bias                        High risk of bias
                                       l Difference in proportion of
                                         incomplete data across groups
                                         and related to outcomes (e.g.
l Complete outcome data                  adverse effects in experimental
                                         group)
l Missing in both groups but reasons
  are reported and balanced across     l Differences in the reasons for
  groups                                 missing data (e.g. smoking
                                         cessation)
l Reason unlikely to be connected
  with outcome (moved away)            l “as treated” (per protocol)
                                         analysis
Selective outcome reporting

=     selection of a subset of the variables recorded for inclusion in publication,
      on the basis of the results


For example:
• Omission of non-significant outcomes
• Choice of data for an outcome (e.g. osteoporosis)
• Choice of analysis (e.g. blood pressure)
• Reporting of subsets of data (e.g. sepsis)
• Under-reporting of data (e.g. only “not significant”)
Selective outcome reporting


     Adequate                Not adequate             Unclear
l All pre-specified      l Not all pre-specified   l Insuficient
  outcomes are reported    outcomes are reported     information
  (protocol available)   l Primary outcome is
                           reported using analysis
l All expected outcomes    method or subset which
  are reported (protocol   was not pre-specified
  not available)         l Reported primary
                           outcomes were not pre-
                           specified
                         l Incomplete reporting
                         l Key outcome not
                           reported
Presentation in your review


“Risk of bias graph”
“Risk of bias summary”
Assessing risk of bias in NRS


 •Selection bias          (how was group allocation?)
 •Performance bias        (blinding, fidelity of interventions)
 •Attrition bias          (completeness of sample & follow-up)
 •Reporting bias          (selective outcome reporting)




 •Confounding and adjustment
                Confounding


Comparison intervention - control


     Intervention versus            ?   Difference in
           control                        outcome

                                    ?

        Imbalance in
      prognostic factors
                Confounding


Association between 2 factors


      Presence of risk                         Occurrence of
          factor                                 outcome




                          Confounding factor
Confounding & adjustment


• At the stage of protocol: list potential confounding factors

• Identify the factors the authors have considered and
  omitted

• Assess balance between groups at baseline

• What did authors do to control for confounders (matching,
  restricting to subgroups, stratification, regression modelling)
                  Tool for NRS
Downs and Black instrument
(J Epidemiol Community Health 1998;52:377-84)

27 items:
• Reporting (10)
• External validity (applicability) (3)
• Internal validity - bias (7)
• Internal validity – confounding (6)
• Power (1)
  Downs and Black
    instrument

 http://www.nccmt.ca/
registry/view/eng/9.html
                Tool for NRS
Newcastle-Ottawa Scale (NOS)
(Wells 2008)

8 items covering 3 perspectives:
•Selection of study groups
•Comparibility of groups
•Ascertainement of exposure (case-control) or
 outcome (cohort)
http://www.ohri.ca/programs/
clinical_epidemiology/oxford.asp
        Diagnostic studies


QUADAS tool
Whiting BMC Medical Research Methodology 2003;3:25


 Cochrane version: 11 items (out of 14 original)

 Diagnostic Test Accuracy Working Group: handbook

      http://srdta.cochrane.org/
      handbook-dta-reviews
 Other risk of bias assessment
              tools


SIGN (Scottish Intercollegiate Guidelines Network)

http://www.sign.ac.uk/methodology/
checklists.html
               In summary

• Risk of bias assessment is essential for systematic reviews

• For RCT: use the Cochrane tool
• For NRS:
   • Higher risk of bias (selection bias & reporting bias)
   • Use the appropriate tool to assess risk of bias
   • Consider how potential confounders are addressed

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:1
posted:9/13/2013
language:English
pages:38