Introduction to Critical Appraisal by pptfiles

VIEWS: 21 PAGES: 39

									Critical Appraisal Extended Study Day
                                                                            Evidence in Practice
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                    Critical Appraisal Study Day


                                                                            John Blenkinsopp
                                                                      Clinical Effectiveness Advisor
                                                                     University Hospital of North Tees



                                                                   www.evidencebasedpractice.org.uk
Critical Appraisal Extended Study Day
              Women and Children
                                                                                      Welcome
                                        Leeds November 23rd 2010

                                                                   • Clinical Effectiveness Advisor
         Healthcare Governance




                                                                     – Training in Critical Appraisal/EBP
                                                                     – Implementation (NICE)

                                                                   • Evidence in Practice

                                                                   • BMA/ScHaRR/NICE
                                                                     – Critical Appraisal Training Days
                                                                        • Basic, Advanced and Facilitator
              Women and Children
         Healthcare Governance
Critical Appraisal Extended Study Day
    Leeds November 23rd 2010           attended today?
                                Who are you and why have you
Critical Appraisal Extended Study Day
              Women and Children
                                                                           Programme
                                        Leeds November 23rd 2010
                                                                   09.30   Introduction to the day
         Healthcare Governance




                                                                   09.45   Diagnostics and Screening
                                                                   10.30   Coffee
                                                                   10.45   Workshop 1 – Appraising a Diagnostic Study
                                                                   12.00   Lunch
                                                                   13.00   Introducing Qualitative Studies
                                                                   13.30   Workshop 2 - Appraising a Qualitative study
                                                                   14.30   Coffee
                                                                   14.45   Introduction to Facilitation
                                                                   15.15   Workshop 3 – Facilitation Skills
                                                                   16.15   Feedback and Future Planning
                                                                   16.30   Evaluations and Close
Critical Appraisal Extended Study Day
                                                                         Aim of this session
              Women and Children

                                        Leeds November 23rd 2010

                                                                   • The aim of today is to:
         Healthcare Governance




                                                                   • Develop your critical appraisal knowledge
                                                                     in a SIS environment:
                                                                     – Safe
                                                                     – Informal
                                                                     – Supportive
Critical Appraisal Extended Study Day
              Women and Children
                                                                    Learning Outcomes for Today
                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   By the end of today’s session you will know:

                                                                   a) How a diagnostic/screening test works
                                                                   b) How to appraise a qualitative study
                                                                   c) Some tips for facilitating a session
Critical Appraisal Extended Study Day
              Women and Children
                                                                            Plan for Today
                                        Leeds November 23rd 2010

                                                                   • To prove that by asking the right questions
         Healthcare Governance




                                                                     you can decide on the quality of a paper
                                                                     without:
                                                                     – knowing the subject
                                                                     – reading all of the paper
                                                                     – understanding all of the issues

                                                                   • But…some issues may be new to you – feel
                                                                     free to ask questions
Critical Appraisal Extended Study Day                                   What is Critical Appraisal?
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • “To weigh up the
                                                                     evidence critically to
                                                                     assess its validity and
                                                                     usefulness”
                                                                   • Validity
                                                                      – ‘Closeness to the
                                                                        Truth’
                                                                   • Usefulness                • Sackett and Haynes Evidence
                                                                      – ‘Clinical                Based Medicine 1995: 1 : 4-5
                                                                        Applicability’
                                                                            Evidence-Based Practice:
Critical Appraisal Extended Study Day

                                                                              The cycle of evidence
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                                           Clinical      Evaluate performance
                                                                                           decision

                                                                   Implement findings                         Information need

                                                                   Appraise for                                 Formulate
                                                                   validity and                                 answerable
                                                                   usefulness                                   question
                                                                                        Find evidence from
                                                                                        systematic research
Critical Appraisal Extended Study Day
              Women and Children
                                                                      Background to critical appraisal
                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • “Clinical Epidemiology “ McMaster
                                                                     University
                                                                   • Small group work with ‘consenting
                                                                     adults’
                                                                   • Problem Based Learning – Rather than
                                                                     fill students’ heads with facts - cultivate
                                                                     skills for lifelong learning
                                                                   • Developed User Guides to the Medical
                                                                     Literature
Critical Appraisal Extended Study Day
                                                                            Why is it important?
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • Clinicians would have
                                                                     to read 17 articles/day;
                                                                     365 days per year to
                                                                     keep up-to-date
                                                                     (internal medicine)
                                                                   • Research is of variable
                                                                     quality
                                                                   • An estimated 1% is
                                                                     judged clinically
                                                                     relevant
                                                                   • Which is the 1%?
Critical Appraisal Extended Study Day
              Women and Children
                                                                    How is Critical Appraisal done???
                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • Think about Critical Appraisal in practice
                                                                     in your Trust/organisation – do you know
                                                                     how is it done?

                                                                   • Discuss this with the person next to you
Critical Appraisal Extended Study Day                                 North Tees and Hartlepool
                                                                   2007 Journals/Lunch Club Survey
              Women and Children

                                        Leeds November 23rd 2010

                                                                   – ‘The Juniors or the MMC trainees are given a journal article to
         Healthcare Governance




                                                                     present. We call it critical appraisal, but really it is a PowerPoint
                                                                     summarising the paper’.

                                                                   – ‘We don’t do the numbers…most of that is taken on trust’

                                                                   – ‘A lot of the time the [pharmaceutical] company brings a paper
                                                                     and appraises it for us....the lunch is always nice’

                                                                   – ‘The lunch is provided by [various Pharmaceutical Companies] if
                                                                     we do their paper and they give us pens and key rings - …I got a
                                                                     pen drive last week’

                                                                   – ‘The lunches are always popular’

                                                                   – ‘We all try to attend for the lunches but some of us get bleeped
                                                                     during the presentation’
Critical Appraisal Extended Study Day                                            We are looking for..
              Women and Children

                                        Leeds November 23rd 2010
                                                                   Three most regularly criticised areas of research:
         Healthcare Governance




                                                                   Bias, lack of clarity and statistics

                                                                   • Bias is everywhere – appraising gives us an opportunity to
                                                                     pull the wool from the eyes

                                                                   • Clarity – am I thick? Well presented research is easier to
                                                                     implement

                                                                   • If the figures don’t add up, then I can’t implement the
                                                                     evidence
               Women and Children
          Healthcare Governance
 Critical Appraisal Extended Study Day
         Leeds November 23rd 2010




                    structure
                  • Can add a




  discussion
• …but may
                                • None today




  restrict open
                                               Checklists
Critical Appraisal Extended Study Day                                   Critical Appraisal - problems
                                                                   • Initially time-consuming
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • It does not always provide the reader
                                                                     with the ‘easy’ answer or the answer
                                                                     one might have hoped for

                                                                   • It can be dispiriting if it highlights a
                                                                     lack of good evidence –it may take
                                                                     determination to persist with an area of
                                                                     interest when access to good research in
                                                                     the area is limited.


                                                                   • Maths and
                                                                     Statistics!
              Women and Children
         Healthcare Governance
Critical Appraisal Extended Study Day
    Leeds November 23rd 2010        avoid using statistics today
                                Pie Chart to show whether you can
Critical Appraisal Extended Study Day
              Women and Children
                                                                                Fear of figures
                                        Leeds November 23rd 2010

                                                                   • You are a doctor who
         Healthcare Governance




                                                                     prescribes a patient 10
                                                                     tablets – 1 to be taken
                                                                     every half hour. How
                                                                     long in minutes until
                                                                     they have taken them
                                                                     all?
Critical Appraisal Extended Study Day
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • You are a doctor who prescribes a patient
                                                                     10 tablets – 1 to be taken every half hour.
                                                                     How long in minutes until they have taken
                                                                     them all?

                                                                   • 270 minutes (not 300) (4 ½ hours – not 5)
              Women and Children
         Healthcare Governance
Critical Appraisal Extended Study Day
    Leeds November 23rd 2010




              and Screening
            Diagnostic Testing
Critical Appraisal Extended Study Day
              Women and Children
                                                                          What is Screening?
                                        Leeds November 23rd 2010

                                                                   “The identification of disease or defect in
         Healthcare Governance




                                                                     asymptomatic patients by the application
                                                                     of diagnostic tests which can be applied
                                                                     RAPIDLY to sort out apparently well
                                                                     persons who PROBABLY have a disease
                                                                     from those who PROBABLY do not”
                                                                   • (Commission on Chronic Illness 1957)
Critical Appraisal Extended Study Day                                 Why Diagnostic Testing and
                                                                             Screening?
              Women and Children

                                        Leeds November 23rd 2010
                                                                   • Accurate and timely diagnosis is the key to good
         Healthcare Governance




                                                                     treatment

                                                                   • Health Services are constantly looking for new,
                                                                     convenient, clinically effective and cost effective
                                                                     ways to screen the population for major health
                                                                     issues

                                                                   • Diagnostic Tests can be the driver for the better
                                                                     use of evidence based practice - we can measure
                                                                     new tests to see how good they are
Critical Appraisal Extended Study Day                                 ‘The most difficult aspect of
                                                                     Evidence-Based Practice’ (Muir
              Women and Children

                                        Leeds November 23rd 2010

                                                                                Gray)
         Healthcare Governance




                                                                   • WHY? – Statistics and Language



                                                                   • Statistics
                                                                      – There are a lot of statistics required to ‘test the
                                                                        test’ (but we will work together in the group)
Critical Appraisal Extended Study Day                                     ‘The most difficult aspect of
                                                                       Evidence-Based Practice’ (Muir
              Women and Children

                                        Leeds November 23rd 2010

                                                                                    Gray)
         Healthcare Governance




                                                                   •   The Language

                                                                       – The result of a test may be positive or negative: this has nothing
                                                                         to do with a bad prognosis, but rather means that the test worked
                                                                         or not, and a certain parameter that was evaluated was present or
                                                                         not. We have true positives, false positives, true negatives and
                                                                         false negatives

                                                                       – Difficult to explain to patients

                                                                       – A negative screening test for breast cancer means that no sign of
                                                                         breast cancer could be found (..which is in fact very positive for
                                                                         the patient).
Critical Appraisal Extended Study Day                              Critically Appraising a Screen or a
                                                                              diagnostic test
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   Validity: Does the test provide a good indication of who
                                                                     does and does not have the disease?

                                                                   •     Sensitivity of the test (Ability to detect cases of the
                                                                         disease)

                                                                   •     Specificity of the test (Ability to exclude those
                                                                         without the disease)

                                                                   •     Reliability: (precision): Does the test give consistent
                                                                         and reliable results?
Critical Appraisal Extended Study Day
              Women and Children
                                                                   Sensitivity and Specificity
                                        Leeds November 23rd 2010

                                                                   • Validity of Screening Test (Accuracy)
         Healthcare Governance




                                                                   • Sensitivity: Is the test detecting true cases
                                                                     of disease? (Ideal is 100%: 100% of cases
                                                                     are detected)

                                                                   • Specificity: Is the test excluding those
                                                                     without disease? (Ideal is 100%)
Critical Appraisal Extended Study Day
              Women and Children
                                                                   Sensitivity and Specificity
                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • Smoke Alarm
                                                                   • ….or searching Medline

                                                                   • Highly Sensitive

                                                                   • …but not very specific
Critical Appraisal Extended Study Day                              Where do we set the cut-off for a
                                                                           screening test?
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                    • We have to consider:

                                                                    • The impact of a high number of false positives:
                                                                       – Anxiety for the patient, cost of further testing


                                                                    • Importance of not missing a case:
                                                                       – seriousness of disease
Critical Appraisal Extended Study Day                                        How do they work?
                                                                             Diagnosis Murder*
              Women and Children

                                        Leeds November 23rd 2010

                                                                   • Ten men are awaiting trial for murder. Only
         Healthcare Governance




                                                                     three of them actually committed a murder
                                                                     and the seven others are innocent of any
                                                                     crime.
                                                                   • A jury hears each case and finds six of the
                                                                     men guilty of murder.
                                                                   • Two of the convicted are true murderers.
                                                                     Four men are wrongly imprisoned. One
                                                                     murderer walks free
Critical Appraisal Extended Study Day
              Women and Children
                                                                              Diagnosis Murder
                                        Leeds November 23rd 2010

                                                                                True Criminal Status
         Healthcare Governance




                                                                   Jury       Murderer         Not Murderer
                                                                   Verdict

                                                                   Guilty     Rightly          Wrongly Convicted
                                                                              Convicted

                                                                   Innocent   Wrongly          Rightly acquitted
                                                                              Acquitted
Critical Appraisal Extended Study Day
              Women and Children
                                                                              Diagnosis Murder
                                        Leeds November 23rd 2010

                                                                                  True Criminal Status
         Healthcare Governance




                                                                   Jury Verdict   Murderer         Not Murderer

                                                                                  Rightly
                                                                   Guilty         Convicted        Wrongly
                                                                                  2 men            Convicted

                                                                   Innocent       Wrongly          Rightly
                                                                                  Acquitted        acquitted
Critical Appraisal Extended Study Day
              Women and Children
                                                                                  Diagnosis Murder
                                        Leeds November 23rd 2010

                                                                                   True Criminal Status
         Healthcare Governance




                                                                   Jury Verdict      Murderer         Not Murderer
                                                                                     Rightly
                                                                   Guilty            Convicted        Wrongly
                                                                                     2 men            Convicted
                                                                                                      Rightly
                                                                   Innocent          Wrongly          acquitted
                                                                                     Acquitted        3 men
Critical Appraisal Extended Study Day
              Women and Children
                                                                              Diagnosis Murder
                                        Leeds November 23rd 2010

                                                                                  True Criminal Status
         Healthcare Governance




                                                                   Jury Verdict      Murderer        Not Murderer

                                                                                     Rightly
                                                                   Guilty            Convicted       Wrongly
                                                                                     2 men           Convicted
                                                                                     Wrongly         Rightly
                                                                   Innocent          Acquitted       acquitted
                                                                                     1 man           3 men
Critical Appraisal Extended Study Day
              Women and Children
                                                                              Diagnosis Murder
                                        Leeds November 23rd 2010

                                                                                  True Criminal Status
         Healthcare Governance




                                                                   Jury Verdict   Murderer          Not Murderer

                                                                                  Rightly           Wrongly
                                                                   Guilty         Convicted         Convicted
                                                                                  2 men             4 men
                                                                                  Wrongly           Rightly acquitted
                                                                   Innocent       Acquitted 1 man   3 men
Critical Appraisal Extended Study Day
              Women and Children
                                                                    A Good Diagnostic Test
                                        Leeds November 23rd 2010

                                                                   • New tests should be validated by comparison
         Healthcare Governance




                                                                     against an established gold standard in an
                                                                     appropriate spectrum of subjects

                                                                   • Diagnostic tests are seldom 100% accurate (false
                                                                     positives and false negatives will occur)
                                                                      – …Meaning that some patients will get the wrong
                                                                        diagnosis no matter what we do…and we have to be
                                                                        prepared for that
                                                                           Critical Appraisal
Critical Appraisal Extended Study Day
              Women and Children

                                        Leeds November 23rd 2010
         Healthcare Governance




                                                                   • We are looking for..
                                                                     – An appropriate cohort of patients subjected to
                                                                       diagnostic test
                                                                     – They need to be recruited into the study at early
                                                                       stage in their disease
                                                                     – The new diagnostic test must be compared to a
                                                                       ‘gold standard’
                                                                     – There must be blinded assessment
                                                                     – The test must be ‘reproducible’
Critical Appraisal Extended Study Day                                 How Good does a Diagnostic Test
                                                                               have to be?
              Women and Children

                                        Leeds November 23rd 2010
                                                                   • It Varies
         Healthcare Governance




                                                                   • Depends upon
                                                                      – Seriousness of disease and the consequences of high false positivity
                                                                        rate:

                                                                   • WHO suggestions are that a rapid HIV test would need
                                                                     extremely high sensitivity 90% (rule in) and 95% specificity
                                                                     (rule out)

                                                                   • WHO also suggest that a screen for nearsighted children
                                                                     would not need approximately 80% sensitivity and >95%
                                                                     specificity
Critical Appraisal Extended Study Day
              Women and Children
                                                                   Decision to use a screening test
                                        Leeds November 23rd 2010

                                                                   • Following Critical Appraisal
         Healthcare Governance




                                                                     – Cost/Benefit Analysis Criteria


                                                                     – If the cost of screening all the individuals who
                                                                       might have a disease is outweighed by the
                                                                       benefits of early detection to the few that have
                                                                       the disease, then the screening test is used

                                                                     – That decision is always evidence-based
              Women and Children
         Healthcare Governance
Critical Appraisal Extended Study Day
    Leeds November 23rd 2010




               …is about screening for Down Syndrome
                                                       The paper we are about to Appraise

								
To top