Introduction to systematic reviewing 3 May 2005 by Jv24D2A

VIEWS: 4 PAGES: 27

									SYSTEMATIC REVIEWS 1
ESRC Methods Festival, Oxford

  9.25-9.45 am 19th July 2006

 Introduction to research synthesis and the
ESRC Methods for Research Synthesis Node
of the National Centre for Research Methods
               David Gough

EPPI-Centre, Social Science Research Unit,
Institute of Education, University of London
           Today’s sessions




         • Morning: Systematic Reviews 1
                     – 4 approaches to systematic reviews
         • Afternoon: Systematic Reviews 2
                     – Implications of different approaches for stages
                       of a review



19th July 2006 (2)
   1. Social Science Research Unit


              Five streams of work:

                     – Childhood Studies
                     – Evaluation of Social Interventions
                     – Sexual Health, Reproduction and Social Exclusion
                     – Evidence for Policy and Practice Information and
                       Co-ordinating Centre
                     – Perspectives, Participation and Research



19th July 2006 (3)
           EPPI-Centre vision and mission
           statement




           Developing and promoting participatory and
           user-friendly systematic reviews addressing
           important questions in different domains of
           policy, practice and research in the public
           interest



19th July 2006 (4)
           Evidence for Policy and Practice Information
           and Co-ordinating (EPPI) Centre


                Conducting reviews since 1993 In health          Methods and tools to
                   promotion, education, transport.            Support all types of review
                                                              Question and primary studies
                 Supporting 25+ groups and 70+ reviews
                 in criminology, employment,speech and           EPPI-Reviewer software
                           language, social care


                                                   Methodological work, e.g.
                  Formal links               Methods for Research Synthesis Project
                 with Cochrane                      ESRC National Centre
                 and Campbell                        for Research Methods
                 Collaborations

                                                             Short courses and
                                                                Masters course
                                On-line libraries
                                                                in evidence for
                             of research evidence
                                                           public policy and practice
19th July 2006 (5)
           ESRC NCRM Methods for Research
           Synthesis node

              a) A question-driven framework
              b) Methods for integrating a breadth of
                 approaches
              c) Contribution from primary studies
              d) Linking synthesis with e-social science
              e) Training and capacity building
              f) Engagement with users in relation to
                 understanding and demand for SRS
19th July 2006 (6)
           Reviews: Building on what we know



         What do we want to know?
          What do we know already?
            How do we know it?
              What more do we want to know?
                How can we know it?


         Not just a linear process: many overlapping
          research questions and questions, concepts and
          methods will change
19th July 2006 (7)
           How do we do this?
              Methods of study



            • Primary study: explicit rigorous methods
              within accepted rules of evidence for that
              discipline

            • Secondary study: exactly the same!



19th July 2006 (8)
                Key decision-making stages in SRS

                  Form review team

                  Formulate review question and develop protocol
                  Define studies to be considered (inclusion criteria )

                  Search for and screen studies (search strategy)

                  Describe studies (systematic map of research)

                  Assess study quality (and relevance)

                  Synthesise findings (answering review question)

                  Communicate and engage
19th   July 2006 (9)
                                          QUESTIONS
                 USERS OF RESEARCH:       What do we
                  PERSPECTIVES AND       want to know?
                    PARTICIPATION                        (i) Research
                                                         (ii) User of service/ public
                                                         (iii) Practitioner
                                          TYPES OF       (iv) Policy community

           Interpretation                KNOWLEDGE       (v) Organisational

          and application
                                                         Effects?
                                          RESEARCH       How/processes?
                                          QUESTIONS      Nature/extent/frequency?
                                                         Perspectives/ concepts?
          Communication
                                                         Action
                              What has    RESEARCH       research
                                                         Survey
                               been      STUDIES AND     Case study
                               done?      METHODS        Experimental
                                                         Multi method

               What do we know?                          Effects?
              How do we know it?                         How/processes?
              What don’t we know?         RESEARCH
            What more do we want to       EVIDENCE       Perspectives/ concepts?
                    know?
             How could we know it?

19th July 2006 (10)
          Some reasons why research does
          not make a difference
       i. Research only one of many factors
       ii. Instrumental use / avoidance of research
       iii. Questions about the quality of research or
            the perspective being ‘sold’
       iv. Access to research
       v. Interpretation and use and relation to
            other knowledge and contexts
       vi. Relevance of the research

19th July 2006 (11)
          Approaches to reviewing and types
          of data and synthesis

    Nature of data and analysis in the review predominantly:
    • A priori specified method of review or iterative approach
      (for e.g review question, bases for assessment of quality of
      studies, and/or framework for synthesis emerging during
      the review)
    • Numerical or narrative data
    • ‘Empirical’ or ‘conceptual’ data
    • Relatively homogeneous or heterogeneous data
    • Numerical or narrative analysis of data in synthesis
    • Integrative (meta empirical) or interpretative (meta
      conceptual) synthesis of data

19th July 2006 (12)
               Key decision-making stages in SRS

                 Form review team

                 Formulate review question and develop protocol
                 Define studies to be considered (inclusion criteria )

                 Search for and screen studies (search strategy)

                 Describe studies (systematic map of research)

                 Assess study quality (and relevance)

                 Synthesise findings (answering review question)

                 Communicate and engage
19th   July 2006 (13)
          Synthesis


         • ‘The process or result of building up
           separate elements, especially ideas, into a
           connected whole, especially a theory or
           system’ (OED)

         • Not just a report of the findings of the
           individual studies in a review
         • Involves a transformation of the data from
           primary studies in some way

19th July 2006 (14)
          Approaches to reviewing and types
          of data and synthesis

    Nature of data and analysis in the review predominantly:
    • A priori specified method of review or iterative approach
      (for e.g review question, bases for assessment of quality of
      studies, and/or framework for synthesis emerging during
      the review)
    • Numerical or narrative data
    • ‘Empirical’ or ‘conceptual’ data
    • Relatively homogeneous or heterogeneous data
    • Numerical or narrative analysis of data in synthesis
    • Integrative (meta empirical) or interpretative (meta
      conceptual) synthesis of data

19th July 2006 (15)
          Approaches to reviewing : examples

       Statistical meta analysis1
       • A priori or iterative procedures of review
       • Numerical or narrative data
       • ‘Empirical’ or ‘conceptual’ data
       • Relatively homogeneous or heterogeneous data
       • Numerical or narrative analysis of data in synthesis
       • Integrative or interpretative synthesis of data
       1For    e.g. Many statistical meta analysis reviews addressing ‘what works’
              questions

19th July 2006 (16)
          Meta analysis example: Does sex
          education improve the use of
          contraception amongst young people?




             From DiCenso et al. (2002) Interventions to reduce unintended pregnancies amongst adolescents:
             a systematic review of randomised controlled trials. British Medical Journal 231: 1426-1434


19th July 2006 (17)
          Approaches to reviewing: examples


            Narrative reviews of quantitative studies1

            • A priori or iterative procedures of review

            • Numerical or narrative data
            • Empirical’ or ‘conceptual’ data
            • Relatively homogeneous or heterogeneous data

            • Numerical or narrative analysis of data in synthesis

            • Integrative or interpretative synthesis of data
            1For      e.g. Many narrative reviews addressing ‘what works’ questions

19th July 2006 (18)
          Approaches to reviewing: examples


               Thematic reviews of views studies and some meta
                 ethnography
               • A priori or iterative procedures of review
               • Numerical or narrative data
               • ‘Empirical’ or ‘conceptual’ data
               • Relatively homogeneous or heterogeneous data
               • Numerical or narrative analysis of data in synthesis
               • Integrative or interpretative synthesis of data

19th July 2006 (19)
          Approaches to reviewing: examples

   Some meta ethnography, critical interpretative
    synthesis, realist synthesis
   • A priori or iterative procedures of review
   • Numerical or narrative data
   • ‘Empirical’ or ‘conceptual’ data
   • Relatively homogeneous or heterogeneous data
   • Numerical or narrative analysis of data in synthesis
   • Integrative or interpretative synthesis of data
19th July 2006 (20)
          Approaches to reviewing: examples

   Some Bayesian meta analysis

   • A priori or iterative procedures of review

   • Numerical or narrative data

   • ‘Empirical’ or ‘conceptual’ data
   • Relatively homogeneous or heterogeneous data

   • Numerical or narrative analysis of data in synthesis

   • Integrative or interpretative synthesis of data
19th July 2006 (21)
          Mixed methods

                                        SCOPING AND MAPPING
                                        (of 272 reports of 193 studies)


                                              Review question
                      e.g. What is known about the barriers to, and facilitators of, fruit and
                               veg intake amongst children aged 4 to 10 years?


                       Trials (N=33)                                 ‘Views’ studies (N=8)
        1. Application of inclusion criteria                  1. Application of inclusion criteria
               2. Quality assessment                                2. Quality assessment
                  3. Data extraction                                   3. Data extraction
           4. Synthesis using statistical                           4. Thematic synthesis
                    meta-analysis

                                             Trials and ‘views’
                                         Mixed methods synthesis


19th July 2006 (22)
          Extensiveness of reviews


        Concerns related issues of (i) time and staff
          resources and (ii) breadth and depth of
          evidence

        (i) Quick overview
        1. Non systematic scoping reviews
        2. Systematic but limited maps or syntheses
        3. Are Rapid Evidence Assessments 2 or 3?
19th July 2006 (23)
          (ii) Breadth of evidence and
          evidence sources in reviews


       a. Broad or narrow range of evidence types.
       b. Broad or narrow evidence sources.
       c. Focus on secondary or primary sources:
          Inclusion of previous reviews, primary data
          or both
       In addition: reviews may have sub-questions
          focusing on different evidence (or different
          questions / methods on the same
          evidence)
19th July 2006 (24)
          Other aspects of extensiveness of
          systematic reviews
    Each stage of a review can vary in its relative
      breadth, depth, exhaustiveness, and analysis:
    • Question: breadth broad or narrow
    • Searching: full exhaustive or purposive or scoping
    • Map: analytic or descriptive
    • Synthesis breadth: broad or narrow
    • Extent of evidence: adequate unless specified
      minimal
    • Synthesis type: synthetic unless specified
      descriptive
19th July 2006 (25)
                                          Feed into decision making
                                         Feed into decision making




                        Communication of review
                      Communication of review findings
                        findings
19th July 2006 (26)
          Thank you



                      SSRU website: http://www.ioe.ac.uk/ssru/
                      SSRU's EPPI website: http://eppi.ioe.ac.uk

                      NRCM MRS website
                      http://www.ncrm.ac.uk/nodes/mrs/people.php

                      Email: d.gough@ioe.ac.uk


19th July 2006 (27)

								
To top