Learning Center
Plans & pricing Sign in
Sign Out

New methodological standards for Cochrane reviews first output



        New methodological standards for
              Cochrane reviews
              first output from the MECIR project

       Rachel Churchill                               Julian Higgins
Co-ordinating Editor representative          Methods Groups representative on
        on Steering Group                            Steering Group
 School of Social and Community              MRC Biostatistics Unit, Cambridge,
  Medicine, University of Bristol                            UK
                                                 & Centre for Reviews and
                                             Dissemination, University of York
• Background and purpose of MECIR

• Development of methodological standards for conduct of
  intervention reviews

• The first set of standards

• Some examples

• Next steps and discussions around implementation
• Coordinated by MARS (Methods Application and Review
   – Co-convened by Julian Higgins and Rachel Churchill to
     facilitate interaction between Methods Groups and CRGs;
     supports Editor in Chief and CEU

• Coordinating Group from MARS and the CEU (Rachel Churchill,
  Julian Higgins, Jackie Chandler, David Tovey and Toby
• Stands for
                 Methodological Expectations of
                 Cochrane Intervention Reviews

• Purpose is
   – to specify methodological expectations for Cochrane
     protocols, reviews, and updates of reviews on the effects of
   – to ensure that these methodological expectations are
     supported and implemented across the Collaboration
                      Development process
        MECIR                                                  Collaboration-wide
                                1. WGs submitted                   consultation
     Working Groups
                                   long list of items

1. Developing a question and
   deciding the scope of the                              • All registered entities
   review                                                   (16/6/11 – 8/8/11)
                                 Coordinating Group
2. Searching for studies
                                                          • 51% responded
3. Selecting studies and
                                                             - CRGs (35/53)
   collecting data
                                                             - MGs (8/15)
4. Assessing risk of bias in
                                                             - Centres (5/14)
   studies                     2. WGs asked to consider      - Centre branches (2/15)
5. Analysing data and
                                Mandatory items              - Fields (5/11)
   undertaking meta-analyses
6. Interpretation and          vs                            - CEU, TWG, Author rep
   presenting results          Highly Desirable
                                Good practice
                               Rationale for each
                Consultation process
• Further work by Coordinating Group, before extensive
  consultation and feedback from all Cochrane entities

• All registered entities (16 June ‘11 – 8 August ‘11)

• 51% entities responded
   – CRGs (35/53)
   – MGs (8/15)
   – Centres (5/14)
   – Centre branches (2/15)
   – Fields (5/11)
   – CEU, TWG, Author rep
   Consultation feedback (Aug – Oct 2011)

• High levels of support for most items

• Decision around status item description, expectation and
  rationale (merging or omitting as appropriate) and provided
  associated Handbook reference for each standard

• 100 standards, reduced to 80

• Provided responses to comments and final decision regarding
  status, description, and rationale for each item (see Annex 2)
           Summary of MECIR standards
• 80 standards for the conduct of CIRs structured
   – Setting research question to inform the scope of the
   – Setting the eligibility criteria for including studies in the
   – Selecting outcomes to be addressed
   – Planning the review methods
   – Searching for studies
   – Selecting studies into the review
   – Collecting data from included studies
   – Assessing risk of bias in included studies
   – Synthesizing the results of included studies
   – Summarizing the findings
   – Reaching conclusions
Some examples
Example of one we
    got right
      Pre-defining clear and unambiguous criteria for
        interventions and comparators (standard 7)

   Define in advance the eligible interventions and the
interventions against which these can be compared in the
                     included studies
•   CRGs:                 30 / 0
•   Centres:              7/0
•   Fields:               4/0
•   Methods Groups:       6/0

• Left unchanged
 Example of one
we had to work on
            Excluding studies without useable data
                         (standard 40)

    Do not exclude studies on the basis of there being no
    usable data (either because the data are inconvenient
       format or because they are not reported at all)

                                               [Draft] (Mandatory)
•    CRGs:                     23 / 5
•    Centres:                  6/1
•    Fields:                   4/0
•    Methods Groups:           6/0
•    Three suggestions to change to Highly Desirable
         Excluding studies without useable data
                      (standard 40)

                                     We will not implement
        Hugely important               this in our group

  RevMan will protest if a          somewhat confusing
  study is ‘included’ with
      no data at all
                                       agree if the outcomes of
Clarify that this relates to the     interest were measured as
formal “exclusion” of studies                an outcome
       from the review
                                   should be rephrased
      Excluding studies without useable data
                   (standard 40)

 Include studies in the review irrespective of whether
measured outcome data are reported in a ‘usable’ way.
Some key ones
            Some key conduct standards
•   Consider important potential adverse effects: Mandatory
•   Clarify role of outcomes: Mandatory
•   Search trials registers: Mandatory
•   Double (independent) data extraction: Mandatory

• Risk of bias tables: Mandatory
• Summary of findings tables: Highly desirable
• The five GRADE considerations (study limitations, consistency
  of effect, imprecision, directness and publication bias) to assess
  the quality of the body of evidence: Mandatory
The new stuff
             Standards not currently in
                  the Handbook
• Rerun or update searches for all relevant databases within six
  months before publication of the review or review update, and
  screen the results for potentially eligible studies: Mandatory,
• Incorporate fully any studies identified in the rerun or update
  of the search within six months before publication of the
  review or review update: Highly Desirable
• Compare magnitude and direction of effects reported by
  studies with how they are presented in the review, taking
  account of legitimate differences: Mandatory
• Consider in advance whether issues of equity and relevance of
  evidence to specific populations are important...: Highly
               Other MECIR outputs
• Guidance on implementation of standards for updates to be
   – to be produced
• Reporting standards (for protocols, reviews, updates)
   – in draft
   – Working Groups to be invited to input
   – consultation: Collaboration and external (users)
• Good practices and Common errors
   – in draft
   – will continually evolve
            Facilitating implementation
A mixture of
• Distribution to entities and individuals
• Handbook amendments
   – prominent incorporation of standards throughout
   – extension/clarification of guidance as appropriate
• Training materials (Training Working Group)
• Check lists for authors, editors, peer reviewers
   – building on existing author checklist (Editorial Resources
• Modifications to RevMan
• Baseline audit
                 Concluding remarks
• MECIR conduct standards are (almost all) clarifications of
  existing Handbook guidance
• Adopting the standards should ensure that Cochrane
  intervention reviews are consistently of high quality
• We hope you will support and implement them
   – and contribute to development of further expectations,
     and future evaluations of them all
• The standards will be updated over time
• For more, see
• Finally, THANK YOU to all individuals and entities who have
  contributed so helpfully so far to the development of the
              Particular thanks to Working Groups
Doug Altman                              Rachel Marshall
Mohammed Ansari (Methods lead, WG1)      Ruth Mitchell
Sally Bell-Syer                          Donal O’Mathuna
Patrick Bossuyt                          Anna Noel-Storr
Deborah Caldwell                         Georgia Salanti (Methods lead, WG5)
Christopher Cates                        Doug Salzwedel
Rachel Churchill (Co-Eds lead, WG4)      Margaret Sampson
Mike Clarke (Co-Eds co-lead, WG2)        Jelena Savovic
Jan Clarkson (Co-Eds co-lead, WG6)       Holger Schünemann (Methods lead, WG 6)
Philippa Davies                          Ian Shemilt
Marina Davoli (Co-Eds lead, WG1)         Nandi Siegfried
Ruth Foxlee                              Jonathan Sterne (Methods lead, WG4)
Chantelle Garritty                       Britta Tendal (Methods lead, WG3)
Davina Ghersi (Co-Eds co-lead, WG2)      David Tovey
Julie Glanville (Methods co-lead, WG2)   Peter Tugwell
Peter Herbison                           Lucy Turner
Julian Higgins                           Claire Vale
Sophie Hill (Co-Eds lead, WG3)           Julia Walters
Toby Lasserson                           Helen Worthington (Co-Eds lead, WG 5 and Co-Eds
Edith Leclercq                                 co-lead, WG6)
Carol Lefebvre (Methods co-lead, WG2)    Janelle Yorke
Jessie McGowan

To top