Critically appraising research
in pain management
Making sense of the evidence
M.Sc. The nature of pain and its’ management,
• Why do we need to appraise the
literature on pain ?
• What is the current state of the art ?
• How do we go about it ?
• What are good sources of information ?
The process of “weighing
up” the evidence to assess
how useful it is in making
decisions about clinical
Some “drivers” promoting critical review of
• Evidence based practice
Clinical effectiveness > guidelines, standards
• Continuing Professional Development (CPD)
• Modernisation Agenda eg. patient choice
• Research – asking the right questions
Types of Evidence
Generated from first hand experience
eg. diaries, letters, research reports
• Secondary (synthesis of others work)
eg. narrative and systematic literature
reviews, scholarly papers
c. 20.000 journals, 2 million papers p.a.
Pain Management – The Evidence
• Meta-analyses, RCT’s
• Systematic reviews
• Cohort studies
• Case studies
• Narrative reviews
• Reports, editorials, working papers, product
• Conference proceedings, abstracts
• Dissertations etc.
Traditional Hierarchy of Evidence
Which research has most “weight” ?
• Meta-analysis, systematic review
• High quality RCT
• Controlled study without randomisation
• Observational study eg. case control study
• Descriptive study
• Expert consensus
• Clinicians and clients/patients views
- The “sharp end” of critical appraisal
- Used for development of clinical guidelines
Many stages including:
• Development of a protocol
• Systematic selection of studies
• Data Extraction
• Quality assessment
• Statistical or other synthesis of findings
• Rating of the overall body of evidence
• Cox II selective inhibitors are not recommended
for routine (regular) use in patients with
rheumatoid arthritis or osteoarthritis (2001,
• Current evidence on the safety and efficacy of
percutaneous disc decompression using
coblation for lower back pain does not appear
adequate to support the use of this procedure
without special arrangements for consent and
for audit or research (Consultation Document,
25 May 2004)
• Adalimumab, etanercept and infliximab for the
treatment of ankylosing spondylitis (in progress,
due Feb. 2007)
Assessing the quality
a) Quality of individual publications
• Guidelines/reading guides
• Criterion based scoring tools
b) Quality of bodies of evidence
• Grading schemes
Quality Evaluation Tools
• Many “off-the-shelf” tools available for
different types of studies
eg. Jadad, Delphi, Maastricht, SIGN
• Debate on scales v scores v weighting
• Few have been properly validated and
the criteria for validation are unknown
• Move towards customised checklists
• Critical Appraisal Skills Programme
• Scottish Intercollegiate Guidelines Network
• Jadad et. al. (1996) Control Clin Trials 17:1–12
• Newcastle Ottawa Observational Study checklist
Quality in quantitative research
Emphasis on issues such as:
Random allocation of subjects, “blinding”
Similarity at baseline
Standard, reliable and valid outcome assessment
Intention to treat analysis (ITT)
Quality in qualitative research
• Dependability (reliability) eg.
- justification of methods
- audit trail
- providing information about the researcher
- member checking, inter-researcher comparisons, triangulation
• Credibility eg.
- providing raw quotes
- reference to accepted procedures for analysis
- discussion of how researcher may have effected outcomes
- clear distinction between data and interpretation
• Transferability eg.
- detail about participants and context
- identifying differences between individual participants and
participants and researcher
CASP – Qualitative Studies
Rigour: has a thorough and appropriate
approach been applied to key research
methods in the study?
Credibility: are the findings well
presented and meaningful?
Relevance: how useful are the findings to
you and your organisation?
Structure of a research paper
• Title, author
• Abstract: summary of what the paper is about
• Introduction: Background including previous
research, aims, research question/hypothesis,
• Methods: patients, methods, equipment, data
• Discussion (may include results in qualitative
• Conclusion, recommendations for clinical
practice and further research
General questions for any study
• Are the aims clearly stated ?
• Was it ethical ?
• Does the design match the aims ?
• Was the sample size justified ?
• Sufficient methodological detail eg. drop outs ?
Missing data ?
• Are the measurements/outcomes valid and
reliable ? (NB. study population)
• Are statistical methods/data analysis described ?
• Are the main findings explicit ?
• Are important effects considered ?
• Are results/findings compared with previous
• Are implications for practice discussed ?
Papers for review:
Effects of pool-based and land-based
aerobic exercise on women with fibromyalgia/
chronic widespread muscle pain
Jentoft et al
Arthritis Care and Research, 2001, 45:42-47
In the system: the lived experience of chronic
back pain from the perspectives of those
seeking help from pain clinics
Walker et al
Pain, 1999, 621-628