MoCA vs

Document Sample
MoCA vs Powered By Docstoc
					           THE M A I N EV EN T

               M oC A
                  vs.
       SE
     MM

Regional Geriatric Program of Eastern Ontario
              City Wide Rounds

              Dr. Frank Molnar
               April 17, 2009
Objectives
1. To learn how to review validation studies (studies of
   sensitivity and specificity)
2. To use this new knowledge to review validation
   studies for the use of the MOCA in dementia / MCI
3. To discuss the copyright issues of the MMSE (and
   MOCA)
4. To discuss the optimal use of the MOCA
      For those with MMSE ≥ 26?
      For those in whom the MMSE is unrevealing?
      In place of the MMSE?
Disclosure
 I do not own stocks in nor have I been employed by
 any company with a financial interest in either the
 MMSE or MOCA.

 I have formal methodological training (i.e. MSc
 Thesis) in the derivation and validation of cognitive
 tests.

 I use both the MMSE and the MOCA in clinical
 practice

 I am not a copyright lawyer
Acknowledgements
 Ian MacDowell
 Malcolm Hing

 Judy Willoughby

 My patients

 Everyone else who has argued with me about
 these concepts
The Preliminary Event

   In order to truly understand the
   results of the studies to be reviewed
   we need to understand:
     The definitions of sensitivity and
     specificity
     How sensitivity and specificity are
     affected by:
        Cut-off values employed
        Overlap of cognitive scores
        Choice of test
Definitions

 Sensitivity
   % of diseased persons identified as
   diseased (score below cut-off)

 Specificity
   % of normal persons identified as
   normal (score above cut-off)
1. Sensitivity and specificity are
affected by the cut-off score
employed
Scores for persons                   MOCA or MMSE          Scores for persons
with normal                                                with dementia
cognition                                30
       Specificity = 25%     xxx x x
                                               xx x x     Sensitivity = 100%
                              xxxxx
                                               xx x
                             xx x x x
                             xxxxx              xxx x
                                               x
                                         20
                                               xx xx
                                               xx
                                                 xxx x
                                               x x xx
                                               xxx xx
                                        10     x xx x x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE          Scores for persons
with normal                                                with dementia
cognition                                30
                             xxx x x
                             xxxxx             xx x x x
       Specificity = 50%
                                               xxxx x     Sensitivity = 87.5%
                             xx x x x
                             xxxxx             x xxx x

                                         20    xxxxxx
                                               xx xxx
                                               xxxxx
                                                xxx x
                                        10     x x xx
1) Sensitivity = % with disease                x
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE         Scores for persons
with normal                                               with dementia
cognition                                30
                          xxx x x
                          xxxxx                xx x x x
                                               xxxx x
        Specificity = 75%
                          xx x x x
                          xxxxx                x xxx x    Sensitivity = 75%

                                         20    xxxxxx
                                               xx xxx
                                               xxxxx
                                                xxx x
                                        10     x x xx
1) Sensitivity = % with disease                x
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE         Scores for persons
with normal                                               with dementia
cognition                                30
                             xxx xx x
                             xxxxx             xx x x x
                                               xxxx x
                             xx x x x
                             xxxxx             x xxx x
       Specificity = 100%
                                         20    xxxxxx     Sensitivity = 62.5%
                                               xx xxx
                                               xxxxx
                                                xxx x
                                        10     x x xx
1) Sensitivity = % with disease                x
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE         Scores for persons
with normal                                               with dementia
cognition                                30
                             xxx xx x
                             xxxxx             xx x x x
                                               xxxx x
                             xx x x x
                             xxxxx             x xxx x

                                         20    xxxxxx
                                               xx xxx
       Specificity = 100%
                                               xxxxx      Sensitivity = 35%

                                                xxx x
                                        10     x x xx
1) Sensitivity = % with disease                x
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Take Home Message #1

 Sensitivity and Specificity for any given
 test are dependent on cut-off score
 studied
 For scales where high scores are good and
 low scores are bad (MMSE, MOCA)
   When cut-off is lowered
    • Sensitivity decreases
    • Specificity increases
   When cut-off is raise
    • Sensitivity increase
    • Specificity decreases
Sensitivity vs. Specificity

  100
   90
   80
   70
   60
                                  specificity
   50                             sensitivity
   40
   30
   20
   10
    0
        10        20         30
             MMSE and MoCA
2. Sensitivity and specificity are
affected by the population in
which the test is being used
     - Overlap of cognitive scores
       (spectrum of disease)
Scores for persons                   MOCA or MMSE        Scores for persons
with normal                                              with dementia
cognition                                30
                          xxx x x             xx x x x
                          xxxxx               xxxx x

        Specificity = 75%
                          xx x x x            x xxx x
                          xxxxx               xxxxxx     Sensitivity = 62%

                                         20   xx xxx
                                              xxxxx
                                               xxx x
                                              x x xx
                                        10
                                              x
1) Sensitivity = % with disease
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE         Scores for persons
with normal                                               with dementia
cognition                                30
                          xxx x x
                          xxxxx                xx x x x
                                               xxxx x
        Specificity = 75%
                          xx x x x
                          xxxxx                x xxx x    Sensitivity = 75%

                                         20    xxxxxx
                                               xx xxx
                                               xxxxx
                                                xxx x
                                        10     x x xx
1) Sensitivity = % with disease                x
who are identified as diseased by
test (i.e. % of diseased that fall
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE        Scores for persons
with normal                                              with dementia
cognition                                30
                          xxx x x
                          xxxxx
                          xx x x x            xx x x x
        Specificity = 75%
                          xxxxx                xxxx x    Sensitivity = 87.5%

                                         20   x xxx x
                                              xxxxxx
                                              xx xxx
                                              xxxxx
                                        10     xxx x
1) Sensitivity = % with disease
who are identified as diseased by
                                              x x xx
test (i.e. % of diseased that fall            x
below cut-off score)
2) Specificity = % of normals
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE        Scores for persons
with normal                                              with dementia
cognition                                30
                          xxx x x
                          xxxxx
        Specificity = 75%
                          xx x x x
                          xxxxx                          Sensitivity = 100%
                                              xx x x x
                                         20   xxxx x
                                              x xxx x
                                              xxxxxx
                                              xx xxx
                                        10    xxxxx
1) Sensitivity = % with disease
who are identified as diseased by              xxx x
test (i.e. % of diseased that fall
below cut-off score)                          x x xx
2) Specificity = % of normals                 x
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Scores for persons                   MOCA or MMSE        Scores for persons
with normal                                              with dementia
cognition                                30
                          xxxx x x x
                          x x x xxx x
       Specificity = 100%
                          xx x xxx
                                                         Sensitivity = 100%
                                              xx x x x
                                         20   xxxx x
                                              x xxx x
                                              xxxxxx
                                              xx xxx
                                        10    xxxxx
1) Sensitivity = % with disease
who are identified as diseased by              xxx x
test (i.e. % of diseased that fall
below cut-off score)                          x x xx
2) Specificity = % of normals                 x
who are identified as normal by          0
test (i.e. % of normals that
score above cut-off)
Less overlap – higher combined
sensitivity and specificity




  0        10      20        30




Greater overlap – lower combined
sensitivity and specificity




   0       10       20       30
Correct population distribution




   0          10         20            30



Incorrect distribution resulting in exaggerated
sensitivity and specificity




    0          10         20           30
Take Home Message #2
 The sensitivity and specificity depend on the
 amount of test score overlap between
 normal and diseased
   Sensitivity and specificity depend on
   sample / population
 Since the populations we take care of
 clinically are different from those in studies
   The Sensitivity and Specificity of a test in
   clinical practice will likely not match that in
   studies (we cannot know if it does)
   We can still compare the tests within 1 study
3 Take Home Messages

1.   Sensitivity and Specificity for any given
     test are dependent on cut-off score
2.   Sensitivity and Specificity depend on
     sample / population
        Since the populations we take care of
        clinically are different from those in studies
        the Sensitivity and Specificity of a test in
        clinical practice will likely not match that in
        studies
3.   Sensitivity and Specificity are dependent
     on the test employed
3. Sensitivity and specificity
are affected by the test
employed
THE MAIN EVENT

            MoCA
             In this
             corner…




MMSE
MOCA validation process
 Developed based on clinical intuition of main author
 (ZN)
 Iterative modification based on 5 years of clinical use
 Tested on 46 MCI / AD with MMSE > 24 vs. 46
 normal
    5 items replaced & weighting adjusted
 Clinical distribution

 We are now in the stage of validation
    Ongoing process
    Main dementia / MCI articles to be
    reviewed.
Search Strategy
 Emailed the author of the first article
 (Ziad Nasreddine)
    MoCA validation studies kept up-to-date on
    MoCA's website reference section at
    www.mocatest.org


 RGPEO Librarian searched the literature
    Did not find any studies not found on above
    website
3 MOCA Validation Studies in area of Dementia

1. Nasreddine et al. The Montreal Cognitive
   Assessment, MOCA: A brief Screening Tool For
   Mild Cognitive Impairment. Journal of the American
   Geriatrics Society 2005; 53: 695-699
2. Smith et al. The Montreal Cognitive Assessment:
   validity and Utility in a Memory Clinic Setting. The
   Canadian Journal of psychiatry 2007; 52; 329-332
3. Luis et al. Cross validation of the Montreal Cognitive
   Assessment in community dwelling older adults
   residing in the Southern US. International Journal of
   Geriatric Psychiatry 2008
Nasreddine et al - Design
 94 MCI & 93 AD (90 MMSE ≥17),90 NC
 at 2 Quebec MDCs
   Clinical diagnoses
   MCI – portions of Wechsler Memory Scale
 French and English MMSEs / MOCAs
   Note; language-based (verbal) tests must
   be separately validated in each language
     I am not certain this was done
Nasreddine et al - Results
 MOCA (cut-off 25/26)
   90% SENS to detect MCI
   100% SENS to detect AD
 MMSE (cut-off 25/26)
   18% SENS to detect MCI
   78% SENS to detect AD

 MOCA seems to win on SENS
 (particularly for MCI)
Nasreddine et al - Results
 SPEC = % Normals ≥ 26 (correctly
 identified as normal
 MOCA (cut-off 25/26)
   87% SPEC to normals
     Mislabelled 13% as impaired
 MMSE (cut-off 25/26)
   100% SPEC to normals
 MMSE seems to win on SPECS
Nasreddine – Results (my interpretation)

  The results only describe part of
  the story
    If you lowered the MOCA cut-off, its specificity
    would improve and sensitivity will drop
    If you raise the MMSE cut-off, its sensitivity would
    improve and specificity will drop
    SENS / SPEC are very dependent on cut-
    offs.
  In any case, we do not rely solely on
  cut-off scores but look at which parts of
  the test were failed (disaggregate
  analysis)
Nasreddine et al – recommendations

 If patients have cognitive complaints
 and functional impairment then likely
 dementia
   MMSE first
   MOCA if MMSE ≥ 26 (MCI, Mild dementia)
 If patients have cognitive complaints
 but no functional impairment then likely
 normal or MCI
   MOCA first
Smith et al.- Design

 6 month prospective study of 67 patients
 with MMSE > 24 at UK MDC (mean MMSE
 27.4). I will only look at initial
 assessment – not 6 month F/U
    32 dementia (ICD-10)
       18 AD, 13 VaD, 1 PDD
    23 MCI (Petersen Criteria)
    12 MCC (normal memory or clearly identifiable
    psychiatric disease causing subjective memory
    complaint)
Smith et al.- Results (Table 2)

 MOCA (cut-off 25/26)
    83% SENS to detect MCI
    94% SENS to detect Dementia
 MMSE (cut-off 25/26)
    17% SENS to detect MCI
    25% SENS to detect Dementia
 MOCA seems to win on SENS (for
 both MCI and Dementia) using
 25/26 cut-offs
Smith et al.- Results (Table 2)

 SPEC = % MCC ≥ 26 (correctly
 identified as not MCI / Dementia)
 MOCA (cut-off 25/26)
    50% SPEC to MCC
      Mislabelled 50% as MCI / Dementia
 MMSE (cut-off 25/26)
    100% SPEC to MCC
 MMSE seems to win on SPEC at
 25/26 cut-offs
Smith et al.- Results (my interpretation)

 The results only describe part of
 the story
    If you   lowered the MOCA cut-off, its specificity
    would    improve and sensitivity will drop
    If you   raise the MMSE cut-off, its sensitivity
    would    improve and specificity will drop
    SENS     / SPEC are very dependent on cut-
    offs.
 In any case, we do not rely solely
 on cut-off scores but look at which
 parts of the test were failed
 (disaggregate analysis)
Smith et al.- Conclusions

 Findings support conclusions of
 Nasreddine et al
 Low specificity of MOCA may be
 because used MCC rather than
 normal controls
 MOCA is a useful screen in a
 memory disorder clinic for people
 who score > 25 on the MMSE
Luis et al - design

   118 community-dwelling (Florida)
   subjects involved in a prospective study
     20 probable AD (NINCDS-ADRDA)
     24 amnestic MCI (Petersen)
     74 cognitively normal
   looked at sensitivity vs. specificity over a
   range of cut-offs. This represents a
   major improvement in analytic
   methodology over the 2 previous studies.
Luis et al – Results (Table 2)
   MOCA (cut-off 26/27 – higher than prior studies)
      100% SENS & 35% SPEC to detect MCI
      97% SENS & 35% SPEC to detect MCI/AD
   MOCA (cut-off 23/24 – lower than prior studies)
      96% SENS & 95% SPEC to detect MCI
          This is a suspect finding:
              too good – never see these values in cognitive tests
              see next slide
   MMSE (cut-off 27/28 – higher than prior studies)
      58% SENS & 84% SPEC to detect MCI
   MMSE (cut-off 24/25 – lower than prior studies)
      17% SENS & 96% SPEC to detect MCI
      36% SENS & 96% SPEC to detect MCI/AD
      Sensitivity vs. Specificity



100
 90
 80
 70
 60
                                    specificity
 50                                 sensitivity
 40
 30
 20
 10
  0
      10        20         30
           MMSE and MoCA
Possible explanation

       Incorrect distribution resulting in exaggerated
       sensitivity and specificity
Luis et al – My Conclusions

   The results are questionable - likely due
   to patient selection factors
      The sample size (20 AD) was likely too small
      resulting in a non-representative sample with
      spectrum bias
      I believe the results indicate that 96% of MCI
      participants had a MOCA ≤ 23. Does this make
      sense clinically or do you see MCI patients with
      MOCA ≥ 24?
   The analytic approach (examining ROC curves to find the best
   cut-offs) should be the standard – I used this approach for my
   MSc thesis in the 1990s and it was standard back then.
MY SUMMARY of MOCA
studies
•   First 2 validation studies did not employ best methodology
•   3rd validation study likely has spectrum bias.
•   In all tests, when sensitivity increases, specificity decreases (the
    reverse is also true). This trade-off of sensitivity vs. specificity
    means that the increased sensitivity of the MOCA comes
    with the expected price – lower specificity resulting in more
    normal people being labelled as impaired (higher false
    positives).
•   At a cut-off of 25/26 MOCA better at picking up AD than MMSE
    (with cut-off of 25/26)
     – This may not be true if MMSE cut-offs are altered
        Despite the limitations with these studies, I believe the
        MOCA better than MMSE at detecting MCI / early
        dementia
     – Likely a ceiling effect of the MMSE that increasing cut-off will
        probably not compensate for.
       Tests may have differential sensitivity in
         different ranges of cognitive decline
                                                 MOCA   MMSE
                                        Normal   30     30
                                                 25
                                  MCI
                                                 20

                  Mild dementia
                                                        25
                                                 15
    Moderate dementia
                                                        20
                                                 10
                                                        15
                                                        10
Severe dementia
                                                 5       5

                                                 0       0
NOW FOR SOMETHING
COMPLETELY DIFFERENT



 That was the science

 Now we move to the social /
 legal environment in which the
 MMSE and MOCA operate
http://www3.parinc.com/uploads/pdfs/MMSE_Copyright_PermReq.pdf


MMSE™ Copyright Permission Request Form

Permission is necessary if you want to use a modified version of the MMSE, including a modified format or
translation. If you plan to use the MMSE in its entirety and in its published form, you must purchase the
 number of test protocols you will need for your purposes. To purchase the published form please
contact PAR Customer Support at 1.800.331.8378, or if outside the US and Canada at 813.968.3003.
• PAR will not grant permission to include the entire MMSE Test Form or scale in any publication, including
dissertations and theses. However, inclusion of up to four sample items may be approved.
• A per-copy royalty fee will be charged for all permissions granted for copies of translations of the MMSE or
modified versions of the test form. This fee does not include the cost of the User’s Guide, which is
necessary if you plan to obtain permission to use a translation or modified version of the test form.
There will be a permission fee charged for reproduction of up to four sample items from the MMSE Test Form to
be included in publications.
• Payment can be made in U.S. funds via check, money order, VISA, MasterCard, American Express, or
Discover. We also accept official institutional purchase orders from U.S. and Canadian residents.
Please complete the remainder of this form. Required sections are noted and highlighted. Please complete all
sections that are relevant to your request. If all of the required sections are not completed, we will not be
able to process your request. Once the form is completed, please send the form to the address below:

Copyright Permissions Department
16204 North Florida Avenue, Lutz, Florida 33549
Fax: Toll free 1.800.727.9329 (US & Canada). All other locations, fax 813.968.2598
E-mail: copyright@parinc.com
Upon submission of a completed form, your request will be reviewed and we will contact you
Is this copyright valid?
 Can one copyright something that has been
 in the public domain for decades?
   Similar to ‘established right of way argument’
 Can one copyright something that has
 appeared in numerous publications?
   Each publication holds copyright on its content
Does this copyright effect us (will it be
enforced if found to be valid)?
 Clinically?
    Are we individuals or a program / institution?
 Research?
    Can one publish research that involves the
    MMSE?
 Will they go after ‘little fish’ like us?
    Is it worth the cost of litigation for our relatively
    small scale use?
    Who wants to be the test case?
MOCA – also copyrighted
 Copyright not enforced (no demands for
 payment).
   Will this be permanent?
   Note the concept of ‘stealth copyrighting’
     Allow a product to gain widespread use before
     enforcing copyright as occurred with the MMSE
   Is any cognitive test immune from this process
   in the future?
COST:
How to read studies & select tests:
Cut-off:
     Sensitivity and Specificity for any given test are dependent on cut-off score
Objective:
    - screen for MCI & dementia in community (high cut-off)
     - screen for dementia (not MCI) in community (lower cut-off)
     - NOT for diagnosis
     - on inpatient setting can only screen for cognitive impairment (delirium,
     depression, MCI, dementia)
Sample:
     Sensitivity and Specificity depend on sample / population. Since the
     populations we take care of clinically are different from those in studies the
     Sensitivity and Specificity of a test in clinical practice will likely not match that
     in studies
Test Characteristics:
     Sensitivity and Specificity are dependent on the test employed. MOCA has
     high sensitivity but low specificity (relative to MMSE)
Given all of the above - How
do I suggest using the MOCA?
•   If it was used before to compare measures
•   If seeing someone who is high functioning and likely in the normal /
    MCI / very early dementia range
•   No / few/ borderline functional problems
•   If MMSE > 26
•   If MMSE results are unrevealing

•   If history suggests a need for a better measure of attention / frontal
    executive function (e.g. Lewy Body Disease, Frontal Lobe dementia)
    and language than MMSE provides
•   Look at areas where points were lost and decide how important this is.

•   If using in research or for large numbers of people I would consider the
    MOCA given the MMSE enforced copyright (or would consider paying
    to use the MMSE)

				
DOCUMENT INFO