Docstoc

Systematic Reviews and Meta-Analysis - University College Cork

Document Sample
Systematic Reviews and Meta-Analysis - University College Cork Powered By Docstoc
					   Systematic Reviews and
        Meta-Analysis

Methodologies for a new era summer school

 School of Applied Social Studies, University
                 College Cork
                20 June 2011

             Dr Paul Montgomery
 Aims

1) Discuss the advantages and main
 features of systematic reviews
2) Introduce basic principles of meta-
 analysis
Course feedback
The Problem

Millions of articles published in
 thousands of journals each year

Practitioners and researchers are busy

Subjective summaries may misrepresent
 research
Reviews
Systematic Reviews
     Aim to answer specific questions, reduce
      uncertainty, identify outstanding questions
     Common methods include narrative
      synthesis, meta-analysis (meta-regression)
Traditional ‘journalistic’ reviews
     Aim to persuade, draw attention to a topic,
      synthesise information, etc.
     Narrative synthesis most common
 Systematic Review:

“the application of scientific strategies
  that limit bias to the systematic
  assembly, critical appraisal, and
  synthesis of all relevant studies on a
  specific topic."

        Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic
     reviews of randomized contro trials in health care from the Potsdam Consultation
                  on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71
Systematic Reviews
Clear Question
     Define the population, problem, intervention,
      alternative interventions, and outcomes
Replicable Method
     Search strategy
     Inclusion criteria
     Analytical strategy
Transparent Process
Advantages
 Explicit methods limit bias in identifying and
  rejecting studies
 Information can be understood quickly
 Reduced delay between discoveries and
  implementation
 Results can be formally compared
 Heterogeneity can be identified and new
  hypotheses generated
 Quantitative reviews increase precision
 Producers

Cochrane
Campbell
EPPI
DARE
NICE
Interested practitioners/ academics
Cochrane Review Process
 Register titles and check for overlap
 Protocols developed and peer reviewed
 Searches performed widely on all main
  databases, grey literature searches, personal
  contacts
 Abstracts reviewed by two authors
 Data collected and trial quality assessed
 Data synthesis and analysis
 Write-up
 Reviewed by Cochrane/ Campbell editors, then
  peer reviewed
 Systematic reviews

Key components:
 1. Ask a good question
 2. Identify studies
 3. Extract data
 4. Synthesise data
 5. Interpreting the results
Who Should Review?

“Experts, who have been steeped in a subject for
  years and know what the answer ‘ought’ to be,
  are less able to produce an objective review of
  the literature in their subject than non-experts.
  This would be of little consequence if experts'
  opinions could be relied on to be congruent
  with the results of independent systematic
  reviews, but they cannot.”

                        (Trisha Greenhalgh)
PICO Mad-libs
    For
P   ____________ does
I    ____________ compared to
C    ____________ improve/reduce
O    ____________ ?
 Highly Sensitive Search

Electronic Searches
    Databases/Indexes
    Additional Electronic Searches
Hand Searches
Personal Contacts
    Electronic databases:
     PsycInfo                                Searching PsycLit and
    covers 73%                               Embase will cover 92%
     and has a                                   of the core 505
   psychological                             ‘psychiatric’ journals.
       focus
                         PsycInfo

                                    Embase
                                                       Embase covers
                                                      67% plus lots of
                         Medline                     European journals
 Medline covers 23%                                     that Medline
   of the core 505
                                    BA                     misses.
    ‘psychiatric’
journals, plus most of
the major biomedical                         Biological Abstracts covers
      journals.                                 48%, plus lots of life
                                                   sciences stuff.
Electronic Searches
 Sensitivity vs. Specificity
     Even if 2 terms and 3 databases return almost all
     literature on a subject, the goal of a systematic
     review is to find everything.
 Electronic Searches

Specific Authors
Reverse Citation
Agencies / Non-Profits
Funding Bodies
Academic Groups / Research Centers
Google
 Additional Searches

Previous Reviews
Bibliographies of Related Articles
Hand Search Journals (that aren’t
 indexed)
Conference Reports
 (many are electronically published)
Personal Communication

 Call or Email Authors
 Attend Conferences
 Write to:
      Agencies / Non-Profits
      Providers / Manufacturers / Distributors
      Funding Bodies
      Academic Groups / Research Centers
Questions to Ask
 Which programs will be studied?
 Compared to what?
 What study designs are acceptable?
 What must a study measure?
 How must it be measured?
 Must researchers be blind at allocation,
  during the trial, etc?
 How will dropouts be handled?
 What about missing data?
Inclusion and Exclusion
                Types of studies
              Types of participants
              Types of comparisons


                      Specify:
               Types of outcomes
   Multiplicity (time, comparisons, measures,
                     statistics)
Transparency

Be clear about all definitions, searches,
 inclusion and exclusion criteria, etc.
Report ongoing trials
List excluded studies, particularly if:
     The trials contain valuable information
     Exclusion was a close call
     You discovered something about a trial
Evaluating a Review

  Even if a review is
‘systematic’ it may not
  be well-conducted.
  How do we tell the
      difference?
Validity
1. Did the review address a clearly
   focussed question?
2. Were the right sort of studies
   selected?
3. Was the search strategy explicit and
   comprehensive?
4. Did the reviewers assess the quality
   of the identified studies?
 Importance:

1. Were the results similar from study to
   study?
2. What is the overall result of the
   review?
3. How precise are the results?
 Potential Sources of Bias

Describe aspects of study design that
 might have influenced the magnitude or
 direction of results
Use of rating scales with fixed cut-offs
 potentially misleading
Consider external validity
Juni P, Witschi A, Bloch R,
 Egger M. The hazards of
   scoring the quality of
  clinical trials for meta-
analysis. JAMA 1999; 282:
        1054-1060
 Tower of Babel

Studies that find a treatment effect are
 more likely to be published in English-
 language journals.
Opposing studies may be published in
 non-English-language journals.
     Gregoire G, Derderan F, Le Lorier J. Selecting the language of the publications
     included in a meta-analysis: is there a Tower of Babel Bias? J.Clin.Epidemiol.
                                    1995;48:159-163
 Publication Bias

“the tendency of investigators, reviewers
  and editors to differentially submit or
  accept manuscripts for publication on
  the direction or strength of the study
  findings.”

      Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should
   unpublished data be included in meta-analyses? Current convictions and controversies.
                               JAMA 1993; 269: 2749-2753
Unpublished data

Controversial
Unpublished data may not be a full or
 representative sample (Cook 1993)
Publication is no guarantee of scientific quality
 (Oxman 1991)
    Cook DJ, Guyatt GH, Ryan G, Clifton J, Buckingham L, Willan A et al. Should
 unpublished data be included in meta-analyses? Current convictions and controversies.
                             JAMA 1993; 269: 2749-2753
   Oxman AD, Guyatt GH, Singer J, Goldsmith CH, Hutchison BG, Milner RA et al.
   Agreement among reviewers of review articles. J.Clin.Epidemiol. 1991;44:91-98.
 Meta-analyses:

“A systematic review that employs
  statistical methods to combine and
  summarise the results of several
  studies.”


       Cook DJ, Sakett DL, Spitzer WO. Methodological guidelines for systematic
    reviews of randomized contro trials in health care from the Potsdam Consultation
                 on Meta-Analysis. J. Clin. Epidemiol. 1995;48:167-71
   Summarising trials




      Systematic reviews

                           Meta-analyses



Reviews
 Meta-analyses

 Mathematically combine the results of
  different studies
 For dichotomous or continuous outcomes
 From analytical (treatment) or observational
  (aetiology, diagnosis, prognosis) studies
 ‘Weighted’ by study size (usually 1/se2)
  and/or quality
Benefits of meta-analysis:

1. To increase statistical power for primary end
   points and for subgroups.
2. To improve estimates of effect size.
3. To resolve uncertainty when reports disagree
4. To answer questions not posed at the start of
   individual trials.

  Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of
           randomized controlled trials. N.Engl.J.Med. 1987;316:450-455
 Outcome Measures

Continuous / Dichotomous (/ Ordinal)

Objective / Subjective
  Meta-analysis

 Some outcomes are measured on scales –
  e.g. depression or continuously e.g. sleep
  minutes
 Continuous outcomes can be calculated using
  the scale on which they were measured
  (WMD)
 If changes in depression are measured on
  different scales it is still possible to combined
  them but on a standardised scale
 Meta-analysis

 Alternatively we might be interested in
  binary data - two mutually exclusive states
 Dead/alive; hospitalised/not hospitalised
 These data will be measured in a different
  way to continuous (scale) data
 Reported as ‘event rates’
 Meta-analysis

Central Tendency:
     Mean (Cohen’s d, Hedges’s g)
     Odds Ratio / Relative Risk / Rate Ratio
Variance (Confidence Interval)
Clinical Significance (NNT/NNH)
Heterogeneity (I2, Q, Chi2)
 Dichotomous Outcomes

Odds are calculated by dividing the
 number of events by non-events (ie
 clients experiencing the event divided
 by clients not experiencing an event)
Risk/Rate is more widely reported in
 reviews as it tends to be easier to
 communicate
 Weighting

Some studies contribute more weight to
 the ‘average’ result than do others
The more precise the effect estimate,
 the more weight is given
Wide variation is sometimes associated
 with small studies
  Weighting

Clinical trials are rarely conducted according to
 identical protocols
Severity of the problem, intensity of the
 intervention, duration, setting of trial, age may
 account for differences in response
Apples and oranges?
Sources of Heterogeneity:
     Study participants
     Comparisons
     Intervention design
     Delivery
     Duration of follow-up
     Outcome measures
     Methods
Heterogeneity

    Estimates from individual trials vary more than can
     be explained by the play of chance alone
    N.B. Meta-analysis should NOT overlook important
     material differences in subgroup response
Heterogeneity – approaches

     Qualitative v. quantitative
     Qualitative – reconsider pooling
     Does it makes sense to average effects
      from the studies?
     Fixed v. random effects
 Subgroup Analysis

If together there is excessive variation,
 when analysed separately there is a
 uniform response to treatment in each
 subgroup
Hypothesis generating
 Sensitivity analysis:

Sensitivity analyses investigate how the
 conclusions of a review change when one
 or more of the decisions or assumptions
 are altered.
 Testing for heterogeneity

Look at plots of results
Formal tests of homogeneity
     I2
     Q
     Chi2
Assess qualitative differences in study
 design or implementation
                     .
                                     (self-rated) (Self-Rated Symptoms) at Post-Treatment
                             Anxiety Anxiety at Post-Treatment compared to No-Treatment
Weeks   Study name           Comparison          Outcome            Statistics for each study     Sample size                   Hedges's g and 95% CI

                                                                  Hedges's    Lower     Upper                                                                         Relative
                                                                     g         limit     limit   Media   Comp                                                          weight

4       Hassan 1992          Wait List           Combined           4.30       2.55       6.05    10       8                                                              0.68
5       Bickel 2007          Wait List           Combined           0.81       -0.33      1.96     8       5                                                              1.40
4       Milne 1998           Wait List (TaU)     Combined           -0.33      -1.35      0.70     7       6                                                              1.67
8       Rosen 1976           Wait List           Combined           -0.73      -1.72      0.27    16       6                                                              1.74
1       Klein 2001           Wait List           Combined           0.66       -0.18      1.51    10      12                                                              2.20
8       Abramowitz 2009      Wait List           Combined           0.39       -0.45      1.22    11      10                                                              2.23
8       Lidren 1994          Wait List           Combined           0.91       0.09       1.74    12      12                                                              2.29
8       Richards 2006        Wait List           Combined           0.63       -0.14      1.41    23       9                                                              2.46
12      Fletcher 2005        Wait List           HADS - Anxiety     0.10       -0.65      0.86    11      15                                                              2.56
1       Heading 2001         Wait List           Combined           0.17       -0.58      0.92    13      13                                                              2.57
13      Kiely 2002           Wait List (TaU)     Combined           0.85       0.12       1.58    16      14                                                              2.66
8       Lewis 1978           Monitoring          Combined           0.58       -0.13      1.29    38      10                                                              2.77
4       Jones 2002           Wait List (TaU)     Combined           0.58       -0.06      1.21    19      20                                                              3.15
8       Grime 2004           Wait List           HADS - Anxiety     0.41       -0.22      1.04    16      23                                                              3.16
10      Carlbring 2001       Wait List           Combined           0.81       0.18       1.44    21      20                                                              3.18
8       Sorby 1991           No Int (Plus TaU)   Combined           0.62       -0.00      1.24    25      17                                                              3.22
6       Smith 1997           Attention           Combined           0.33       -0.29      0.94    30      15                                                              3.25
10      Titov 2009           Wait List           Combined           0.98       0.37       1.59    24      21                                                              3.27
11      Arpin-Cribbie 2007   No Int              Combined           0.81       0.24       1.38    29      22                                                              3.50
10      Berger 2009          Wait List           Combined           0.75       0.18       1.31    31      21                                                              3.55
10      Carlbring 2006       Wait List           Combined           1.19       0.64       1.74    30      30                                                              3.63
9       Carlbring 2007       Wait List           Combined           1.01       0.47       1.55    29      29                                                              3.70
14      Hazen 1996           Wait List           Combined           0.43       -0.10      0.97    27      27                                                              3.75
13      Zetterqvist 2003     Wait List           Combined           0.39       -0.07      0.84    37      45                                                              4.28
12      Van Boeijen 2005     Treatment as Usual Combined            -0.13      -0.59      0.32    53      28                                                              4.29
10      Titov 2008b          Wait List           Combined           0.79       0.34       1.24    41      40                                                              4.32
10      Titov 2008c          Wait List           Combined           0.45       0.02       0.87    61      34                                                              4.54
10      Titov 2008a          Wait List           Combined           0.83       0.42       1.23    50      49                                                              4.64
13      Mead 2005            Wait List           HADS               0.18       -0.20      0.56    50      53                                                              4.83
12      Rapee 2007           Wait List           Combined           0.38       0.00       0.76    56      52                                                              4.87
9       Proudfoot 2004       Treatment as Usual BAI                 0.38       0.10       0.66    99      98                                                              5.65
                                                                    0.55       0.40       0.70    903     764

                                                                                                                -2.00   -1.00           0.00            1.00   2.00
            Institutionalisation (RR<1 favours home visits)
Group by   Study name          Time (m)               Risk ratio and 95% CI                        Statistics for each study
Time
                                                                                       Relative   Risk    Lower       Upper
                                                                                        weight    ratio    limit       limit

12-23      Dalby (2000)           14                                                       1.20   0.35      0.01       8.41
12-23      Hogan (2001)           12                                                       2.14   2.05      0.19       22.17
12-23      Newbury (2001)         12                                                       3.21   0.94      0.13       6.55
12-23      Hall (1992)            12                                                       7.41   0.38      0.11       1.37
12-23      Hebert (2001)          12                                                       8.06   1.04      0.30       3.53
12-23      Kono (2004)            18                                                      10.83   0.64      0.22       1.83
12-23      Yamada (2003)          18                                                      17.26   1.30      0.56       3.01
12-23      Bernabei (1998)        12                                                      21.52   0.67      0.32       1.43
12-23      Gill (2002)            12                                                      28.37   0.72      0.38       1.39
12-23                                                                                             0.78      0.55       1.10
24-35      Hall (1992)            24                                                      42.30   0.16      0.04       0.67
24-35      Sorenson (1988)        30                                                      57.70   1.02      0.81       1.28
24-35                                                                                             0.46      0.08       2.80
36+        Hall (1992)            36                                                       9.53   0.16      0.04       0.67
36+        Van Rossum (1993)      36                                                      12.00   1.38      0.44       4.30
36+        Pathy (1992a)          36                                                      12.86   1.29      0.46       3.66
36+        Byles (2004)           36                                                      14.97   2.84      1.25       6.43
36+        Stuck (1995)           36                                                      15.52   0.42      0.20       0.90
36+        Pathy (1992b)          36                                                      16.48   0.56      0.29       1.09
36+        Stuck (2000)           36                                                      18.64   1.51      0.99       2.30
36+                                                                                               0.90      0.49       1.67

                                          0.1   0.2   0.5      1        2     5   10
                                 Funnel Plot of Standard Error by Log risk ratio
                 Mortality: Trim and Fill (missing studies shown, 1 trimmed)
                 0.0




                 0.5
Standard Error




                 1.0




                 1.5




                 2.0


                   -2.0   -1.5      -1.0     -0.5        0.0         0.5   1.0     1.5   2.0


                                                    Log risk ratio
                                                  Symptoms) at by Hedges's g
                            Anxiety (Self-Rated of Standard ErrorPost-Treatment
                                      Funnel Plot
                 0.0



                 0.2



                 0.4
Standard Error




                 0.6



                 0.8



                 1.0


                       -4       -3      -2      -1      0       1       2         3   4
                                         Regression of Ave Age on Log risk ratio
                                             Mortality by age: Meta-regression
                 2.00

                 1.60

                 1.20

                 0.80
Log risk ratio




                 0.40

                 0.00

                 -0.40

                 -0.80

                 -1.20

                 -1.60

                 -2.00

                         67.16   69.01     70.86   72.70   74.55    76.40    78.25   80.10   81.94   83.79   85.64

                                                                   Ave Age
                                Anxiety (Self-Rated Symptoms) at Post-Treatment:
                                Number of contacts with researchers and clinicians
                                                               Point
                                                             Estimate    SE         P                           Q         df     P
                                                  Slope        0.03     <0.01     <0.001          Model       12.93       1    <0.001
                                                 Intercept     0.27     0.08      <0.01           Residual    30.34       27    0.30
                         Regression of Number of Contacts on Hedges's g
                                                                   Total                                      43.27       28    0.03

             2.00

             1.72

             1.44

             1.16
Hedges's g




             0.88

             0.60

             0.32

             0.04

             -0.24

             -0.52

             -0.80
                 -1.35   1.47      4.29   7.11     9.93        12.75      15.57           18.39       21.21           24.03    26.85
Limitations

Junk-In, Junk-Out
The results of large trials sometimes
 differ
Chance Events: Aggregation and
 Disaggregation
Conclusion

Systematic reviews seek to reduce
 bias and improve the reliability and
 accuracy of the conclusions.
Meta-analysis is a powerful research
 tool, but it should be conducted only
 in the context of a systematic review,
 and it has important limitations.

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:3/21/2012
language:
pages:57