Docstoc
EXCLUSIVE OFFER FOR DOCSTOC USERS
Try the all-new QuickBooks Online for FREE.  No credit card required.

ASSESSING THE USEFULNESS OF THE MEDICATION APPROPRIATENESS INDEX

Document Sample
ASSESSING THE USEFULNESS OF THE MEDICATION APPROPRIATENESS INDEX Powered By Docstoc
					                    ASSESSING THE USEFULNESS OF
               THE MEDICATION APPROPRIATENESS INDEX
                      IN A COMMUNITY SETTING
             Isabelle Côté1, Karen Farris2,3, Kari Olson4, Cheryl Wiens3, Sherry Dieleman3




                                                           Working Paper 03-01



   1   Manager, Health Economics, Bayer, Inc., 77 Belfield Road, Toronto, ON, M9W 1G6

   2   University of Iowa, College of Pharmacy, Iowa City, Iowa, United States

   3 Faculty of Pharmacy and Pharmaceutical Sciences, University of Alberta, Edmonton,
     Alberta, Canada

   4 Kaiser Permanente, Lakewood, Colorado, United States



   Dr. Côté was Research Fellow at the Institute of Health Economics, Edmonton, Alberta,
   Canada at the time of this work.




Legal Deposit
National Library Canada
ISSN 1481-3823                                                                  January 2003
                                               TABLE OF CONTENTS

SUMMARY.......................................................................................................................... 1
  Rationale ......................................................................................................................... 1
  Objective......................................................................................................................... 1
  Methods .......................................................................................................................... 1
  Findings .......................................................................................................................... 2
  Discussion of Findings .................................................................................................... 2

INTRODUCTION ................................................................................................................ 4
   Introduction Of The Medication Appropriateness Index .................................................. 4
   Literature Review: Measurement Properties of the MAI .................................................. 4
   Limitations of the MAI.................................................................................................... 5
   Objectives of the Study ................................................................................................... 6

METHODS........................................................................................................................... 6
  Study Design................................................................................................................... 6
  Sample for the Present Study........................................................................................... 6
  Data Collection ............................................................................................................... 7
  Statistical Analysis .......................................................................................................... 8

RESULTS........................................................................................................................... 10

DISCUSSION..................................................................................................................... 11

CONCLUSION................................................................................................................... 14

BIBLIOGRAPHY............................................................................................................... 15




  Institute of Health Economics Working Paper 03-01                                                                            ii
                                                      LIST OF TABLES


Table 1. Intra-rater agreement between score at time 2a and score at time 2b on a 3 and 5-
          point scale for each criterion according to the rater (N=257 drugs) .....................................17
Table 2. Proportion of appropriate and inappropriate ratings for which ratings in time 2a
          and ratings in time 2b were in agreement on a 3-point scale according to the
          rater (N=257 drugs)............................................................................................................18
Table 3. MAI mean score difference between score at time 2a and score at time 2b on a 3
         and 5-point scale according to the rater (N=257 drugs).......................................................19
Table 4. Inter-rater agreement between the two raters at time 1, time 2, time 3 and time 4
          on a 5-point scale for each criterion (28 patients at time 1, 2a and 2b; 10 patients
          at time 2c) ..........................................................................................................................20
Table 5. Inter-rater agreement between two raters at time 1, 2, 3 and 4 on a 3-point scale
          for each criterion (28 participants at time 1, 2a and 2b; n=10 patients at time 2c)................21
Table 6. Proportion of appropriate and inappropriate ratings for which the two raters were
          in agreement on a 3-point scale according to times (28 patients at time 1, 2a and
          2b; 10 patients at time 2c) ..................................................................................................22
Table 7. MAI mean overall score difference (± 1 standard deviation) between raters on a
         5-point scale at time 1, time 2, time 3 and time 4................................................................23
Table 8. MAI mean overall score difference (± 1 standard deviation) between raters on a
         3-point scale at time 1, time 2a, time 2b and time 2c...........................................................24
Table 9. Description of the drugs for which a major disagreement (difference of at least 3
          points in scores) was observed between raters at TIME 1 on a 5-point scale
          according the criteria composing the MAI ..........................................................................25
Table 10. Description of the drugs for which a major disagreement (difference of at least
           3 in scores) was observed between raters at TIME 2a on a 5-point scale
           according to the criteria composing the MAI....................................................................28
Table 11. Description of the drugs for which a major disagreement (difference of at least
           3 in scores) occurred between raters at TIME 2b on a 5-point scale according
           to criteria composing the MAI..........................................................................................32
Table 12. Inter-rater agreement between the two raters at time 1 and at time 2c on a 5-
           point scale for each criterion for 10 patients .....................................................................36
Table 13. Inter-rater agreement between the two raters at time 1 and at time 2c on a 3-
           point scale for each criterion for 10 patients .....................................................................37
Table 14. Mean MAI weighted scores at time 1 and at time 2b on a 5-point scale
          according to the rater (28 patients) ...................................................................................38




Institute of Health Economics Working Paper 03-01                                                                           iii
SUMMARY
Rationale

   Inappropriate use of medication among the community-dwelling population is a common problem
that often leads to increases in hospitalizations, mortality and morbidity. In 1993, Hanlon et al.
developed a Medication Appropriateness Index (MAI) to assist physicians and pharmacists in assessing
the appropriateness of a medication for a given patient. The MAI requires clinicians to rate ten explicit
criteria to determine whether a given medication is appropriate for an individual. The MAI has shown
good reliability in settings where patients’ medical data were easily accessible from medical charts.
However, it is unclear if similar results can be found in the community.

Objective

   The objective of the present study was to assess the inter- and intra-rater reliability of the MAI
among community-dwelling patients.

Methods

   Data from 28 patients out of 199 participating in a larger project examining the effect of multi-
disciplinary primary healthcare teams were used in this study. Pharmacists were provided with medical
diagnosis and medication lists with complete medication history to complete the MAI. The MAI was
completed at baseline (time 1) and at 4 months (time 2a). At 5 months (time 2b), the MAI was
reassessed using data used at 4 months. There were a total of 223 medications assessed at baseline and
257 medications at times 2a and 2b.

   For 3- and 5-point scoring schemes, percent of agreement and Kappa were calculated to assess
inter- and intra-raters reliability (intra rater reliability being calculated between times 2a and 2b). To
calculate the weighted MAI score, we used the weights suggested by Samsa et al (8) for eight of the ten
criteria and for two other criteria (practicality and drug-drug interaction), we used the weights
suggested by Volume et al. The weighted MAI score ranged from 20 to 100 on the 5-point scale and
from 20 to 60 on the 3-point scale. We also calculated an overall MAI score per patient for each
scoring scheme. A paired t-test was used to test the difference in overall MAI score between raters and
between times.




Institute of Health Economics Working Paper 03-01                                             1
Findings

   The intra-rater agreement on appropriate ratings between time 2a and 2b ranged from 59% to 90%
for rater 1 and from 27% to 94% for rater 2. Kappa values ranged from fair to almost perfect
agreement. For the intra-rater difference on the MAI overall score, scores at time 2a were in general
significantly higher than scores at time 2b on both scales for both raters.

   Related to the inter-rater agreement, kappa values generally increased over time from slight to fair.
Related to the difference on the MAI overall score, there was a statistically significant difference
between raters at times 2a and 2b on the 5-point scale, while on the 3-point scale there was no
significant difference.

   In general, criterion related to drug-disease interactions and duration of therapy were the criteria
generating the highest level of disagreement between time 2a and time 2b.                 Criterion related to
practicality of directions was the one generating the highest level of agreement.

   Considering the MAI weighted scores on a 5-point scale, the average MAI score ranged from 40.0
± 9.7 to 42.1 ± 6.1 at time 1 and from 40.9 ± 7.0 to 43.1 ± 6.7 at time 2b. In regards to the new
criterion, a statistically significant difference was observed over time for one rater.

Discussion Of Findings

   Intra-rater reliability was in general better than inter-rater reliability. Difference in clinical training
and experience between raters might explain why inter-rater reliability was lower than intra-rater
reliability. The percent of agreement and kappa values were in general higher on a 3-point scale than
on the 5-point scale. The drug-disease interactions criterion was one of the two criteria generating the
high disagreement between raters. This disagreement on drug-disease interactions might be due to the
medication review form. For instance, the space available for the description of the medical conditions
on that form was limited to five medical conditions. Consequently, patients with more than five
medical conditions could have had missing data. The duration of the therapy was the other criterion
generating high disagreement between the raters. This disagreement was probably due to the fact that
there was variability on the availability of this information.

   After four months in the study, the appropriateness of medication remained unchanged according to

the overall MAI score. The overall score of the MAI may not be as useful as the score obtain from

   2                                           Institute of Health Economics Working Paper 03-01
each criterion to look at changes over time. As the importance of each criterion (measured by the

weights) is meaningful, then the scores for each criterion should also be used in addition to the overall

score when doing research.




Institute of Health Economics Working Paper 03-01                                            3
INTRODUCTION
   Inappropriate use of medication among the community-dwelling population is a common problem
that often leads to increases in hospitalizations, mortality and morbidity. These problems are especially
evident in the elderly. It has been estimated that between 14% and 23% of elderly patients are
prescribed at least one inappropriate medication (1;2) and up to 40% of patients do not take their
medications as prescribed (3). Inappropriate medication prescribing or usage accounts for at least five
percent of hospital admissions among the elderly (4;5). Assessing medication appropriateness in the
community-setting is difficult to evaluate due to the numerous healthcare providers and practice-
settings involved.    However, medication appropriateness has received significant attention as an
indicator of the quality of processes of care.

Introduction Of The Medication Appropriateness Index

   Hanlon et al. (6) developed a Medication Appropriateness Index (MAI) to assist physicians and
pharmacists in assessing the appropriateness of a medication for a given patient. The MAI requires
clinicians to rate ten explicit criteria to determine whether a given medication is appropriate for an
individual. The ten explicit criteria are indication, effectiveness, dosage, correct directions, practical
directions, drug-drug interactions, drug-disease interactions, duplication, duration and expense (6).
Furthermore, in order to determine a single summated score for each drug in addition to an overall
score for the patient, a weighting scheme was developed. A weight of three was given for indication
and effectiveness. A weight of two was assigned to dosage, correct directions, drug-drug interactions
and drug-disease interactions.     A weight of one was assigned to practical directions, expense,
duplication and duration.

Literature Review: Measurement Properties Of The MAI

   The MAI has been evaluated for validity and reliability (6). In one of the earliest studies evaluating
the reliability of the MAI, Hanlon et al. (1992) enrolled a sample of ten elderly patients from a
Veterans Administration (VA) hospital who were taking five or more regularly scheduled drugs (n=60
drugs) to determine the inter- and intra- rater reliability among two practicing clinicians. The two
clinicians independently evaluated all medications at baseline and two to four months later. The
clinicians were provided with a medical problem list, a two year medication history, outpatient and
inpatient physician notes and the results of both laboratory and diagnostic tests. When all drugs were
considered, intra-rater agreement between ratings at baseline and two to four months later was 97%

   4                                             Institute of Health Economics Working Paper 03-01
(kappa: 0.92) for the two raters. The inter-rater agreement was 93% (kappa: 0.83) when all drugs were
considered.

    Additionally, in order to evaluate the generalizability of the index, two separate clinical pharmacists
were provided a second random sample of 10 patients who were taking a total of 105 drugs and asked
to complete MAI for each medication at one time point. The inter-rater reliability for all drugs
considered was 89% (kappa: 0.59).

    In order to assess the psychometric properties of the MAI, Hanlon at al. went on to report the
summated MAI score for 208 VA elderly patients receiving a combined total of 1644 regularly
scheduled medications. Across 1644 medications assessed, the summated MAI scores ranged from 0
to 10, although the scale ranges from 0 to 18. The average MAI score was 2.2 (standard deviation =
2.1).

    Additionally, in order to determine the inter- and intra-observer agreement on summated MAI
score, the MAI was assessed by two clinical pharmacists in a random sample of ten patients consuming
a total of 105 medications at baseline and 4 months later.         The summated MAI scores showed
“acceptable” reliability with an intra-class correlation coefficient of 0.74. A paired t-test comparing
scores obtained from observer one to scores obtained from observer two showed a statistically
significant difference (p=0.01), although the mean difference score was small (0.03) and the raters
agreed on 59% of the medications. The intra-observer comparison of time one to time two did not show
statistically significant differences.

Limitations Of The MAI

    Although the MAI is an important contribution to health services researchers, it has limitations.
First, the MAI does not consider the importance the population valued criteria. It could be argued that
to increase medication appropriateness, both healthcare provider and patient need to be involved in the
decision making process. For instance, Volume et al. reported that older adults rated drug-drug
interactions and directions as the highly or most important of the ten criteria (7). They rated disease
interactions, effectiveness, indication, practical directions and duplication as moderately important.
Volume et al. also triangulated data from rankings, discussion and clinician ratings (8) to conclude that
drug-drug interactions should be weighted as three and practical directions should be weighted two (7)
when assessed in older adults. Second, the MAI does not capture the situation in which the patient has
a medical indication for therapy but is not receiving the treatment. Third, while the MAI has shown

Institute of Health Economics Working Paper 03-01                                              5
good reliability in settings where the patients’ complete medical data were accessible from charts, i.e.
Veteran’s Affairs (VA) hospitals, it is unclear if similar results would be found in the community using
information from community pharmacies and office-based general practice settings (9-13). Fourth,
with the exception of Taylor et al, (14) all reported studies discussing the psychometric properties of
the MAI were performed by the same individual who originally developed the MAI. Consequently, the
generalizability of the instrument as used by other investigators is unknown.

Objectives Of The Study

   The objective of the present study was to assess the inter- and intra-rater reliability of the MAI

among community-dwelling patients. The MAI was assessed by two practicing clinical pharmacists

not involved in the development of the MAI in a setting in which access to medical charts was not

readily available. A new criterion was also developed and added to the MAI in order to capture

instances where a patient may have a disease state but is not receiving treatment for that condition.


METHODS
Study Design

   The MAI is an important contribution. The MAI has shown good reliability in settings where
patients’ medical data were easily accessible from medical charts. However, it is unclear if similar
results can be found in the community (9-13). In 1999, we conducted a project to develop and evaluate
an innovative multi-disciplinary team approach to primary health care. The objectives of this project
was to improve or maintain the health of a group of high-risk individuals living in the community by
enhancing the quality of their health care, including improved medication used, enhanced compliance,
and systematic follow-up. The project used the MAI to evaluate the medication appropriateness.

Sample For The Present Study

   Data from patients participating in a larger project examining the effect of multi-disciplinary
primary healthcare teams were used in this study. Participants were eligible for the larger project if
they were at “high risk” for medication problems, i.e. (1) were taking three or more routine medications
per day, (2) had one or more chronic disease states that were poorly controlled, (3) had one or more
chronic disease states that were not currently being treated, (4) had a dosage regimen that had changed

   6                                          Institute of Health Economics Working Paper 03-01
more than four times in the previous 12 months, (5) had medications that had a narrow therapeutic
index, (6) had an identified drug-related problem or the potential for one, (7) had a history of
noncompliance or (8) had experienced a recent decline in health status.

   From the 199 participants in the larger project, a mean number of 4.51 (standard deviation: ± 1.43)
routine medications were prescribed for each participant. In order to maximize the probability of
identifying medication inappropriateness and an adequate sample size to identify agreement and
disagreement, we selected a sub-sample of participants that met the following criteria: 1) were on six or
more routine prescription medications or 2) were on three to five routine prescription medications and
on a medication with a narrow therapeutic index (on warfarin or digoxin).                Sixty-nine (35%)
participants met these criteria. From the 69 participants, 35 (51%) were randomly selected According
to our power calculations, 200 drugs was sufficient to detect statistically significant difference in scores
over time, assuming we wanted to detect a five percent change (0.9 on a 0-18 scale) on the original
MAI, which reported a standard deviation of 2.1 (6). From the 35 patients, five patients were chosen to
train the pharmacists in the MAI scoring and two patients were lost to follow-up. Therefore, the main
results are based on MAI ratings for 28 participants.

Data Collection

   The MAI was assessed by two clinical pharmacists with advanced training (Pharm.D degree). One
pharmacist had 5 years of clinical practice of clinical training in cardiology with a focus on
cardiovascular risk reduction. The other pharmacist had seven years of clinical experience with a
clinical training in geriatrics.   Neither of them was involved with the development or original
evaluations of the MAI. The pharmacists were provided with medical diagnosis and medication lists
with complete medication history in order to complete the MAI. The medical diagnosis list was
provided by the patient’s primary care provider. The medication list was provided by the pharmacy
from which the patient received their prescription medications and included both prescription and non-
prescription drugs. Medication histories were collected by pharmacists trained (other than the two
pharmacists who rated the MAI) in the delivery of comprehensive pharmaceutical care during an in-
home interview with the patient using standard data collection forms.

   General instructions described by Hanlon et al (6) were followed by the two clinical pharmacists to
complete the MAI. In addition to this, the two clinical pharmacists used data from five patients to



Institute of Health Economics Working Paper 03-01                                               7
practice scoring the MAI. After completing the MAI, the pharmacists met to address any ambiguity in
scoring and interpretation of the data.

   The MAI was completed at baseline (time 1) and at 4 months (time 2a) for 28 patients. At 5
months (time 2b), the MAI was reassessed using data used at 4 months. There were a total of 223
medications assessed at baseline and 257 medications at times 2a and 2b.          Preliminary analyses
indicated a lack of perfect inter-rater agreement. Therefore in an effort to enhance agreement, we
added an additional component to the original design study.         This special exercise implied the
reassessment of the MAI for 10 patients (time 2c) using data used at four months. There were a total of
84 medications assessed.

   In the effort to enhance inter-rater agreement, three patients for whom there were large
discrepancies in overall scores between raters at time 2b were identified. We described each drug for
which a major disagreement, i.e. a minimum difference of 3 points between raters on the 5-point scale,
was observed at any point of time (time 1 or time 2a or time 2b) for each criterion. The two raters met
to discuss their ratings on the medications for these three patients. The discussion was audio recorded.
Following that discussion, the two clinical pharmacists reassessed the MAI for 10 patients randomly
selected (time 2c) from the patients not included in the discussion. The pharmacists were provided
with the same information as for time 1 thru 2b.

Statistical Analysis

   For the purpose of this study, the original scoring developed by Hanlon et al. was modified to
create two scoring schemes.       Each criterion was rated for each drug on a 6-point Likert scale
(1=strongly agree, 2=agree; 3=neither agree or disagree, 4=disagree; 5=strongly disagree, 6=do not
know). We modified the original scheme by recoding response number six to a value of three to obtain
a 5-point scale. We also modified the original scheme by aggregating the 6-point scale into a 3-point
scale. Responses 1 and 2 were grouped together and scored 1 (agree), responses 3 and 6 were grouped
and scored 3 (neither agree or disagree) and responses 4 and 5 were grouped and scored 2 (disagree).
In regard to the new criterion, we also used two schemes: a 5-point scale (1=No, 2=low degree of
suspicion, 3= Moderate degree of suspicion, 4= High degree of suspicion, 5=Yes) and a 3-point scale
by aggregating the 5-point scale into a 3-point scale. Responses 1 and 2 were grouped together and
scored as 1 (indicated “no need for a new drug”), response 3 (“moderate suspicion about the need for a



   8                                         Institute of Health Economics Working Paper 03-01
new drug”) was not changed and responses 4 and 5 were grouped and scored as 2 (indicated a “need for
a new drug”).

   For both 3- and 5-point scoring schemes, percent of agreement and Kappa were calculated to assess
inter- and intra-raters reliability (intra rater reliability being calculated between times 2a and 2b). On
the 5-point scoring scheme, Kappa statistic was preferred to intra-class correlation coefficient because
the score distribution of each criterion was heavily skewed in most cases. On the 3-point scale, Kappa
statistics were supplemented with the proportion of drugs rated as appropriate (Ppos) and inappropriate
(Pneg) according to each criterion for which the two raters were in agreement. For example, for the
criterion related to indication, if 20 drugs were rated as appropriate (score of 1=agree that the drug is
appropriate) by one of the two raters and 5 of these drugs were rated as appropriate by both of the two
raters, the Ppos would be 25% (5/20). The proportion of drugs rated as appropriate (Ppos) and
inappropriate (Pneg) according to each criterion for which ratings at time 2 and 3 were in agreement
were also calculated for both raters. We compared the percent of agreement between times for the
sample of ten patients using a Chi-square test. To identify discrepancies between the raters, we
described the drugs for which a major disagreement (difference of 3 points or more on the 5-point
scale) was observed for at least eight criteria at time 1, time 2a and time 2b.

   To interpret the Kappa coefficients reported for each criterion including the new criterion, we used
the guidelines of Landis et al. (15) who considered a coefficient lower than 0 to be a poor agreement,
between 0.00 and 0.20 to be a slight agreement, between 0.21 and 0.40 to be fair, between 0.41 and
0.60 to be moderate, between 0.61 and 0.80 to be substantial and higher than 0.81 to be almost perfect.

   In order to calculate the weighted MAI score, we used the weights suggested by Samsa et al (8) for
eight of the ten criteria, i.e. a weight of three for the correct drug indications and effectiveness, a
weight of two for dosage, directions and drug-disease interaction and a weight of one for duplication,
duration of the therapy and costs were used. For two of the eight criteria, we used the weights
suggested by Volume et al (7), i.e. a weight of two for practicality and a weight of three for drug-drug
interaction.    The weights were multiplied by the rating and summed across all criteria for each
medication. The weighted MAI score ranged from 20 to 100 on the 5-point scale and from 20 to 60 on
the 3-point scale.




Institute of Health Economics Working Paper 03-01                                             9
   We also calculated an overall MAI score per patient for each scoring scheme. A paired t-test was
used to test the difference in overall MAI score between raters and between times. Differences
between scores were considered statistically significant at a p-value ≤ 0.05.

   Difference between scores was considered clinically important for a difference equal or greater than
1.3 for the MAI overall score and for a difference equal or greater than 0.18 for each criterion score
included in the MAI. Given the lack of published standard for a clinically important difference (CID)
in MAI, we used a crude approximation based on suggested guidelines for interpreting effect sizes
(Cohen regarded effect sizes of 0.20 as small, 0.50 as moderate and 0.80 as large). CID was calculated
using a small effect size and standard deviations of the MAI scores were those calculated on the 5-point
scale on the average score of the two raters. All analyses were conducted using SPSS 10.1.0 for
Windows.

RESULTS
   The intra-rater agreement between scores at time 2a and time 2b is presented in Tables 1 and 2.
Results are presented for the two raters separately because the percent of agreement between the raters
was low (in general lower than 50% at time 2a and 2b). In Table 1, for the 10 criteria included in the
MAI, the percent of agreement as well as the kappa values were higher on the 3-point scale than on the
5-point scale for both raters. The percent of agreement was over 50% for all criteria (except for
criterion related to drug-disease interactions for rater 1 on the 5-point scale) on both the 3-point and 5-
point scales. Kappa values ranged from fair to almost perfect agreement. In general, criteria related to
drug-disease interactions (for rater 1) and duration (for rater 2) were those criteria generating the
highest level of disagreement between time 2a and time 2b. Criteria related to practicality of directions
were the one generating the highest level of agreement (for rater 2). In regards to the new criterion
(omission), the percentage of agreement was in general lower than the percentage of agreement using
the standard (original) criteria for the MAI.

   Intra-rater difference on the MAI overall score is presented in Table 3. In general, scores at time 2a
were significantly higher than scores at time 2b on both scales for both raters.

   The inter-rater agreement is presented in Tables 4, 5 and 6. On the 5-point scale (Table 4), the
percent of agreement as well as the kappa values generally increased over time from slight to fair. In
general, criterion related to drug-disease interactions was the criterion generating the highest level of
disagreement between raters while criterion related to the practicality of directions was the one

   10                                           Institute of Health Economics Working Paper 03-01
generating the highest level of agreement. In regards to the new criterion, the percentage of agreement
was lower than the percentage of agreement for the original criteria composing the MAI. On the 3-
point scale (Table 5), like on the 5-point scale, the percent of agreement and kappa values generally
increased over time.

   The inter-rater difference on the MAI overall score is presented in Tables 7 and 8. On the 5-point
scale (Table 7), there was a statistically significant difference between raters at times 2a and 2b, while
on the 3-point scale (Table 8) there was no significant difference.

   The drugs for which a major disagreement between raters was observed on at least eight criteria on
the 5-point scale are described in Tables 9 to 11. At time 1 (Table 9), four drugs led to major
disagreement between raters (Doxepin, Etidronate/calcium, Metformin and Pantoprazole). At time 2a,
three drugs (Calcium, Estrogen and Levothyroxine) and at time 2b, four drugs (Calcium, Fluticasone,
Pantoprazole and Vitamin D) led to major disagreement.

   The inter-rater agreement for the sample of ten patients is presented in Tables 12 and 13. On the 5-
point scale (Table 12), the percent of agreement between raters increased over time for most of the
criteria. Between time 2b and 2c, there is no trend. On the 3-point scale (Table 13), there was no
tendency over time in regard to the percent of agreement between raters and the kappa statistics.

   Mean MAI weighted scores on a 5-point scale are shown in Table 14. When all criteria were
summed together to obtain an average MAI score, no significant difference was observed between
raters at both time 1 and time 2b. The average MAI score ranged from 40.0 ± 9.7 to 42.1 ± 6.1 at time
1 and from 40.9 ± 7.0 to 43.1 ± 6.7 at time 2b.

DISCUSSION
   Intra-rater reliability was in general better than inter-rater reliability. Inter-rater reliability reached
moderate agreement while intra-rater reliability reached almost perfect agreement. The percent of
agreement and kappa values were in general higher on a 3-point scale than on the 5-point scale.
Criterion related to practicality generated the highest level of agreement. Criteria related to drug-
disease interactions and duration generated the highest level of disagreement. After four months in the
study, the appropriateness of medication remained unchanged according to the overall MAI score.

   Differences in clinical training and experience between raters might explain why the inter-rater
reliability was lower than that reported in previous studies. For instance, the clinical pharmacist with

Institute of Health Economics Working Paper 03-01                                               11
clinical experience in geriatrics may be more sensitive to drug-related problems among the elderly
population than the pharmacists with clinical experience in cardiology or vice versa. Physician raters
were not used in this study, but they would contribute positively to such a review in future studies.
Previous studies using the MAI showed excellent agreement between physician and clinical pharmacist
raters [6]. Using clinical pharmacists as raters is generally more economical and efficient.

   The percent of agreement and kappa values varied according to the scale used to measure the MAI.
It is not surprising to see higher agreement when using a 3-point rather than using a 5-point scale
because the 3-point scale is an aggregation of scores from the 5-point scale. Considering this, the
magnitude of changes in scores has to be interpreted differently whether using a 3-point scale or a 5-
point scale. For instance, a one-point score change measured on a 5-point scale corresponds to a
change of 20% (1/5) on the scale whereas a one-point score measured on a 3-point scale corresponds to
a change of 33% (1/3).

   The high level of agreement between raters on the criterion related to the practicality of directions
is not surprising because dosing of medications is usually once or twice daily. For this criterion, all the
information is available to the raters from the prescription. No additional medical information is really
needed to assess this criterion.

   Criteria related to drug-disease interaction and duration of the therapy had “low” inter-rater
reliability. The drug-disease interaction criterion was one of the two criteria generating the highest
disagreement between raters. This disagreement on drug-disease interactions might be due to the
medication review form. For instance, the space available for the description of the medical conditions
on that form was limited to five medical conditions. Consequently, patients with more than five
medical conditions could have had missing data. That is, the potential for incomplete information on
medical diagnosis may have complicated the ratings by the pharmacists on drug-disease interactions
giving more room for interpretation and variation in scoring. The duration of the therapy was the other
criterion generating high disagreement between the raters. This disagreement was probably due to the
fact that there was variability on the availability of this information.

   The information that the pharmacists received before filling out the MAI may also explain the
“low” inter-rater reliability but only at Time 2b. According to the discussion taped between the two
raters, the main reason why the two raters had major disagreements at time 2b was related to the



   12                                           Institute of Health Economics Working Paper 03-01
information they used to evaluate the MAI. One rater stated that she relied on pharmacy files and did
not use the information gathered during interviews (between pharmacists and patients).

   Inter-rater reliability depends on whether we look at the percent of agreement or the kappa
statistics. For instance, reliability measured with the percent of agreement was always higher than
reliability measured with the kappa statistics. This result reflects differences in the two methods. The
percent of agreement reflects the observed agreement while kappa statistics is an index which
compares the agreement against that which might be expected by chance. Kappa statistics are the
chance-corrected proportional agreement.

   The explanation of why certain drugs generated high disagreement between raters is difficult to
establish. The indication and effectiveness criteria were rated inappropriate in eight of the nine
instances where disagreement was highest. Calcium was the medication where raters agreed it was
indicated, but the remaining criteria created disagreement.

   The objective of the study was to examine the usefulness of the overall score of the MAI. We did
not calculate this score identical to previous studies because we wanted to explore different scoring
procedures (6,8). What is clear is that the overall score of the MAI may not be as useful as the score
obtained from each criterion to look at changes over time. For example, if the objective of using the
MAI is to determine the impact of an intervention on medication appropriateness over a 6-month
period, the scores obtained from each criterion should be shown in order to identify what has been
improved or not. By using the overall score only, improvement in score of a specific criterion might be
obscured by a decrement for another criterion. As the importance of each criterion (measured by the
weights) is meaningful, then the scores for each criterion should also be used in addition to the overall
score when doing research.

   Comparing this experience with previous uses of the MAI, we conclude that clinicians who rate the
MAI should work together to improve reliability. The latter characteristic is especially important in
research projects that are seeking to quantify change over time in medication appropriateness.
Standardizing the information requirements for each criterion would also be helpful for future studies.
As well, the minimal clinically important difference is an area where research is warranted.




Institute of Health Economics Working Paper 03-01                                           13
CONCLUSION
   The reliability of the MAI was not adequate in this study. Factors impacting the ratings included
raters with different clinical experience, lack of clinical information and, perhaps, lack of adequate
instruction to the raters. While the MAI has ten explicit criteria for rating medication appropriateness,
the ratings have an implicit component to them. For the MAI to be reliable in wide variety of settings,
information requirements and the minimum clinical experience of raters should be specified.




   14                                        Institute of Health Economics Working Paper 03-01
BIBLIOGRAPHY

1.     Aparasu RR, Mort JR. Inappropriate prescribing for the elderly: beers criteria-based review.
       Ann Pharmacother 2000; 34(3):338-346.

2.     Stuck AE, Beers MH, Steiner A, Aronow HU, Rubenstein LZ, Beck JC. Inappropriate
       medication use in community-residing older persons. Arch Intern Med 1994; 154(19):2195-
       2200.

3.     Bond WS, Hussar DA. Detection methods and strategies for improving medication compliance.
       Am J Hosp Pharm 1991; 48(9):1978-1988.

4.     Roughhead EE, Gilbert AL, Primrose JG, Lloyd NS. Drug-related hospital admission: a review
       of the Australian studies published 1988-1996. Med J Australia 1998; 168:405-408.

5.     Sullivan SD, Kreling DH, Hazlet TK. Noncompliance with medication regimens and
       subsequent hospitalizations: a literature analysis and cost of hospitalization estimate. J Res
       Pharmac Econom 1990; 2:19-33.

6.     Hanlon JT, Schmader KE, Samsa GP, Weinberger M, Uttech KM, Lewis IK et al. A method for
       assessing drug therapy appropriateness. J Clin Epidemiol 1992; 45(10):1045-1051.

7.     Volume CI, Burback LM, Farris KB. Reassessing the MAI: elderly people's opinions about
       medication appropriateness. Int J Pharm Pract 1999; 7:129-137.

8.     Samsa GP, Hanlon JT, Schmader KE, Weinberger M, Clipp EC, Uttech KM et al. A summated
       score for the medication appropriateness index: development and assessment of clinimetric
       properties including content validity. J Clin Epidemiol 1994; 47(8):891-896.

9.     Fitzgerald LS, Hanlon JT, Shelton PS, Landsman PB, Schmader KE, Pulliam CC et al.
       Reliability of a modified medication appropriateness index in ambulatory older persons. Ann
       Pharmacother 1997; 31(5):543-548.

10.    Hanlon JT, Weinberger M, Samsa GP, Schmader KE, Uttech KM, Lewis IK et al. A
       randomized, controlled trial of a clinical pharmacist intervention to improve inappropriate
       prescribing in elderly outpatients with polypharmacy. Am J Med 1996; 100(4):428-437.

11.    Schmader K, Hanlon JT, Weinberger M, Landsman PB, Samsa GP, Lewis I et al.
       Appropriateness of medication prescribing in ambulatory elderly patients. J Am Geriatr Soc
       1994; 42(12):1241-1247.

12.    Schmader KE, Hanlon JT, Landsman PB, Samsa GP, Lewis IK, Weinberger M. Inappropriate
       prescribing and health outcomes in elderly veteran outpatients. Ann Pharmacother 1997;
       31(5):529-533.

13.    Shelton PS, Hanlon JT, Landsman PB, Scott MA, Lewis IK, Schmader KE et al. Reliability of
       drug utilization evaluation as an assessment of medication appropriateness. Ann Pharmacother
       1997; 31(5):533-542.


Institute of Health Economics Working Paper 03-01                                            15
14.        Taylor CT, Stewart LM, Byrd DC, Church CO. Reliability of an instrument for evaluating
           antimicrobial appropriateness in hospitalized patients
           1. Am J Health Syst Pharm 2001; 58(3):242-246.

15.        Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics
           1977; 33(1):159-174.




      16                                       Institute of Health Economics Working Paper 03-01
Table 1. Intra-Rater Agreement Between Score At Time 2a And Score At Time 2b On A 3 And 5-Point Scale For Each Criterion
         According To The Rater (N=257 Drugs)
                                              Rater 1                                                       Rater 2
Criterion                  5-point scale                    3-point scale                 5-point scale                  3-point scale
                           %                                %                           %                                 %
                                        ĸa                                  ĸa                        ĸa                             ĸa
                       agreement                        agreement                   agreement                         agreement

Indication               0.69          NA                 0.86         0.51            0.60          0.43                0.77       0.53
Effectiveness            0.66          NA                 0.91         0.60            0.56          0.39                0.76       0.50
Dosage                   0.60          0.45               0.74         0.58            0.58          0.41                0.64       0.44
Correct directions       0.59          0.41               0.81         0.62            0.63          0.39                0.73       0.46
Practical directions     0.66          0.40               0.87         0.50            0.78          0.44                0.86       0.54
Drug-drug                0.56          0.36               0.80         0.54            0.66          0.45                0.81       0.46
interactions
Drug-disease             0.49          0.27               0.78         0.46            0.77          0.44                0.83       0.48
interactions
Duplication              0.56          0.37               0.77         0.54            0.67          0.42                0.72       0.47
Duration                 0.69          0.53               0.75         0.59            0.55          0.30                0.65       0.38
Expense                  0.69          NA                 0.86         0.56            0.57          0.30                0.72       0.40
Omission                  0.57          NA                0.86           NA             0.32         0.05                 0.43      -0.19
                b
(New criterion)
a
  ĸ: KAPPA statistics; b The unit of analysis was the patient (28 patients). NA: Not available because at least one category is empty for one
rater
Table 2. Proportion Of Appropriate And Inappropriate Ratings For Which Ratings In
         Time 2a And Ratings In Time 2b Were In Agreement On A 3-Point Scale
         According To The Rater (N=257 Drugs)
                                            Rater 1                         Rater 2
Criterion
                                    Pposa             Pnegb         Pposa             Pnegb
Indication                          0.87              0.51          0.80               0.44

Effectiveness                       0.90              0.53          0.78              0.43

Dosage                              0.75              0.44          0.94              0.62

Correct directions                  0.73              0.66          0.65              0.52

Practical directions                0.86              0.43          0.85              0.45

Drug-drug interactions              0.81              0.51          0.78              0.42

Drug-disease interactions           0.78              0.44          0.82              0.41

Duplication                         0.78              0.44          0.69              0.43
Duration                            0.59              0.51          0.27              0.46
Expense                             0.87              0.43          0.70              0.43
Omission (New criterion)c           0.33              0.85          0.32              0.46
a
 Ppos: the proportion of appropriate ratings for which the time 2a and time 2b were in agreement. b Pneg:
the proportion of inappropriate ratings for which time 2a and time 2b were in agreement. c The unit of
analysis was the 28 patients.




    18                                         Institute of Health Economics Working Paper 03-01
Table 3. MAI Mean Score Difference Between Score At Time 2a And Score At Time 2b On A 3 And 5-Point Scale According To
         The Rater (N=257 Drugs)

                                              Rater 1                                                                Rater 2

                           5-point scale                     3-point scale                      5-point scale                   3-point scale

                      Mean score                        Mean score                      Mean score                         Mean score
                      differencea                  b    differencea             b       differencea                  b     differencea
                                           P value                     P value                               P value                     P valueb
                         (SD)                              (SD)                               (SD)                             (SD)
Overall                   2.58              0.01           -0.59         0.47                -4.43            0.03             -1.89       0.04
score                    (5.20)                            (4.19)                           (10.16)                            (4.57)
a                                                                                       b
    Positive difference means that score at time 2b was higher than score at time 2a.       Paired t-test.
Table 4. Inter-Rater Agreement Between The Two Raters At Time 1, Time 2, Time 3 And Time 4 On A 5-Point Scale For
         Each Criterion (28 Patients At Time 1, 2a And 2b; 10 Patients At Time 2c)
                                         Time 1                    Time 2a                   Time 2b                   Time 2c
Criterion                            (n=223 drugs)              (n=257 drugs)             (n=257 drugs)              (n=84 drugs)
                                      %            ĸa             %           ĸa           %            ĸa           %            ĸa
                                  agreement                   agreement                agreement                 agreement
Indication                           0.46         0.14           0.53        0.27          0.55        NA            0.68        NA
Effectiveness                        0.43         0.14           0.44        0.17          0.49        NA            0.61        0.36
Dosage                               0.42         0.19           0.46        0.25          0.44        0.23          0.58        0.25
Correct directions                   0.30         0.04           0.44        0.20          0.55        0.28          0.38        NA
Practical directions                 0.42         0.07           0.58        0.25          0.71        0.36          0.74        NA
Drug-drug interactions               0.27         -0.02          0.34        0.13          0.50        0.27          0.46        NA
Drug-disease interactions            0.22         0.01           0.30        0.12          0.41        0.13          0.49        NA
Duplication                          0.42         0.13           0.51        0.26          0.51        0.28          0.51        0.21
Duration                             0.46         0.25           0.46        0.12          0.41        0.15          0.82        NA
Expense                              0.32         0.05           0.43        NA            0.48        0.22          0.60        NA
Omission (New criterion)b            0.21         0.03           0.11        NA            0.25        NA            0.00        NA
a
    ĸ: kappa s-tatistics. b The unit of analysis was the patient. NA: Not available because at least one category is empty for one rater.
Table 5. Inter-Rater Agreement Between Two Raters At Time 1, 2, 3 And 4 On A 3-Point Scale For Each Criterion (28
         Participants At Time 1, 2a And 2b; N=10 Patients At Time 2c)
                                     Time 1                    Time 2a                     Time 2b                    Time 2c
Criterion                         (n=223 drugs)             (n=257 drugs)               (n=257 drugs)              (n=84 drugs)
                                    %          ĸa           %             ĸa             %            ĸa           %             ĸa
                                agreement               agreement                    agreement                 agreement
Indication                        0.76        0.18         0.76          0.37           0.72         0.32          0.81         NA
Effectiveness                     0.74        0.19         0.74          0.31           0.72         0.30          0.81         0.44
Dosage                            0.57        0.31         0.55          0.30           0.51         0.25          0.64         0.31
Correct directions                0.57        0.15         0.65          0.33           0.66         0.32          0.52         0.22
Practical directions              0.75        0.10         0.81          0.37           0.84         0.42          0.81         0.40
Drug-drug interactions            0.65        0.01         0.69          0.26           0.76         0.37          0.71         NA
Drug-disease interactions         0.61        0.09         0.73          0.27           0.76         0.34          0.71         0.28
Duplication                       0.67        0.23         0.69          0.38           0.69         0.43          0.58         0.23
Duration                          0.61        0.36         0.52          0.17           0.47         0.18          0.85         NA
Expense                           0.64        0.06         0.73          0.32           0.70         0.27          0.76         0.41
                            b
Omission (New criterion)         0.43        0.16            0.82      -0.094           0.68        NA            0.20          NA
a                     b
  ĸ: kappa statistics. The unit of analysis was the patient. NA: Not available because one category is empty for one rater.
Table 6. Proportion Of Appropriate And Inappropriate Ratings For Which The Two Raters Were In Agreement On A 3-Point Scale
         According To Times (28 Patients At Time 1, 2a And 2b; 10 Patients At Time 2c)
                                            Time 1                  Time 2a                  Time 2b                   Time 2c
Criterion
                                            (n=223)                 (n=257)                  (n=257)                       (n=84)
                                        Ppos a       Pneg b     Ppos a     Pneg b        Ppos a     Pneg b        Ppos a        Pneg b
Indication                              0.79          0.07      0.76       0.31          0.72       0.32          0.82              0.00

Effectiveness                           0.76          0.07      0.75       0.32          0.71       0.33          0.78              0.39

Dosage                                  0.46          0.15      0.42       0.22          0.26       0.27          0.32              0.60

Correct directions                      0.52          0.26      0.57       0.42          0.57       0.43          0.44              0.30

Practical directions                    0.74          0.08      0.80       0.32          0.82       0.36          0.79              0.38

Drug-drug interactions                  0.64          0.11      0.68       0.30          0.73       0.37          0.68              0.31

Drug-disease interactions               0.61          0.17      0.71       0.31          0.75       0.34          0.69              0.29

Duplication                             0.66          0.18      0.72       0.28          0.66       0.37          0.59              0.27
Duration                                0.50          0.15      0.25       0.27          0.15       0.33          0.00              0.83
Expense                                 0.65          0.09      0.72       0.30          0.69       0.29          0.76              0.11
Omission (New criterion)c                 0.20      0.33          0.19        0.28       0.11       0.37          0.00       0.00
a                                                                                        b
  Ppos: the proportion of appropriate ratings for which the two raters were in agreement. Pneg: the proportion of inappropriate ratings for
which the two raters were in agreement. c The unit of analysis was the patient.
Table 7 MAI Mean Overall Score Difference (± 1 Standard Deviation) Between Raters On A 5-Point Scale At Time 1, Time 2,
        Time 3 And Time 4
                         Time 1                     Time 2a                           Time 2b                           Time 2c
                  (n=28 patients)                (n=28 patients)                (n=28 patients)                    (n=10 patients)
              Mean score      P valueb      Mean score      P valueb       Mean score       P valueb          Mean score         P valueb
              differencea                   differencea                    differencea                        differencea
                  (SD)                          (SD)                           (SD)                              (SD)
Overall           2.10            0.25          3.75          0.04             -3.27            0.00             -0.65            0.66
score             (9.53)                       (9.08)                         (5.34)                             (4.52)
a                                                                                                              b
    Positive difference means that score reported by the rater 1 was higher than score reported by the rater 2. Paired t-test.
Table 8 MAI Mean Overall Score Difference (± 1 Standard Deviation) Between Raters On A 3-Point Scale At Time 1, Time 2a,
        Time 2b And Time 2c
                       Time 1                       Time 2a                           Time 2b                           Time 2c
                (n=28 participants)            (n=28 participants)            (n=28 participants)               (n=10 participants)
              Mean score      P valueb      Mean score      P valueb       Mean score       P valueb         Mean score           P valueb
              differencea                   differencea                    differencea                       differencea
                  (SD)                          (SD)                          (SD)                               (SD)
Overall          -0.33          0.66            -0.09         0.91            -1.39             0.12            -1.36              0.22
score             (3.94)                       (4.24)                        (4.53)                              (3.23)
a                                                                                                              b
    Positive difference means that score reported by the rater 1 was higher than score reported by the rater 2. Paired t-test.
Table 9 Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 Points In Scores) Was Observed
        Between Raters At Time 1 On A 5-Point Scale According The Criteria Composing The Mai

Drug                                             Criterion number composing the MAIa                           TOTAL
(Generic name)                 1      2      3       4        5       6      7         8      9       10     (Number of
                                                                                                                 X)
Acarbose                                             X                                                           1
Acetylcysteine                                       X                                                           1
Alendronate                                          X                                                           1
Amitriptyline                  X                             X                       X        X       X          5
ASA                                                  X               X                                           2
Aspirin                                              X                                                           1
Azathioprine                                         X                       X                                   2
Budesonide                                           X               X       X                        X          4
Bupropion                                            X               X               X                X          4
Calcium                                                                              X                           1
Celecoxib                                            X                                                           1
Chloral-hydrate                                      X       X       X       X       X                X          6
Citalopram                                                   X                                        X          2
Clonazepam                                                           X               X                           2
Cyproterone                                          X                                                           1
Digoxin                                                      X       X               X                X          4
Diltiazem                             X              X                                                           2
Docusate                                                                     X       X                           2
Domperidone                                                                                   X       X          2
Doxepin                        X      X      X       X       X       X       X       X        X       X          10
Efavirenz                                                                    X                                   1
Enalapril                                                    X                                                   1
Estrogen                                                                                              X          1
Table 9 Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 Points In Scores) Was Observed
        Between Raters At Time 1 On A 5-Point Scale According The Criteria Composing The Mai

Drug                                             Criterion number composing the MAIa                           TOTAL
(Generic name)                 1      2      3       4        5       6      7         8      9       10     (Number of
                                                                                                                 X)
Etidronate/calcium             X      X      X               X               X       X        X       X          8
Fluconazole                                                                                           X          1
Furosemide                            X                                              X                X          3
Glucosamine                                  X                                                                   1
Hydrochlorothiazide/
                                                     X                                                            1
triamterene
Imipramine                                                           X                                            1
Insulin                                              X                                                            1
Ipatropium                                           X                                                            1
Labetalol                      X      X              X       X       X                        X       X           7
Lansoprazole                                 X                                                                    1
Levothyroxine                  X                                                     X                            2
Lodoxamide                                   X       X                                        X                   3
Lorazepam                                                                                             X           1
Metformin                      X      X              X       X       X       X       X        X       X           9
Methotrimeprazine                                                    X                                            1
Metoclopramide                                               X                       X                X           3
Metoprolol                                                   X               X                                    2
Naproxen                                                                                              X           1
Nifedipine                            X              X       X                                X       X           5
Nitrazepam                     X                             X                       X                X           4
Olanzapine                                                                                            X           1
Omeprazole                                                           X       X       X                X           4
 Table 9 Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 Points In Scores) Was Observed
         Between Raters At Time 1 On A 5-Point Scale According The Criteria Composing The Mai

 Drug                                                   Criterion number composing the MAIa                                TOTAL
 (Generic name)                     1       2      3        4        5       6       7        8         9        10      (Number of
                                                                                                                             X)
  Oxazepam                                                                   X                                                1
  Pantoprazole                     X       X       X        X        X                        X         X        X            8
  Perindopril                                               X                                                                 1
  Phenytoin                                                                                                      X            1
  Potassium                                X                X        X                                           X            4
  Prednisone                                                X                        X                                        2
  Ranitidine                                                         X                                                        1
  Senna                                    X                X        X                        X                  X            5
  Sertraline                                                                 X                                                1
  Sucralfate                               X                                                                                  1
  Theophylline                                              X                X                                                2
  Trazodone                                                                                   X                  X            2
  Triazolam                                                                  X                                                1
  Verapamil                                                                                             X                     1
  Vit B12                                                                                     X                  X            2
  Vit B6                                                                                      X                  X            2
  Vit C                                                              X       X       X        X                               4
  Vit E                                                              X       X       X        X         X        X            6
  Warfarin                         X       X                         X       X       X        X                  X            7
  Zafirlukast                                                                X                                                1
  Zopiclone                                                                                   X                               1
a
  Criterion 1: indication; criterion 2: effectiveness; criterion 3: Dosage; criterion 4: correct directions; criterion 5: practical directions;
criterion 6: drug-drug interactions; criterion 7: drug-disease interactions; criterion 8: duplication; criterion 9: duration; criterion 10:
expense.
Table 10. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Was
          Observed Between Raters At TIME 2a On A 5-Point Scale According To The Criteria Composing The MAI
                                          Criterion number composing the MAIa                      Total
Drug                      1     2     3      4      5      6      7       8      9       10     (Number of
                                                                                                   X)
Acarbose                                     X                                                      1
Acetaminophen                                X                            X                         2
Alendronate               X     X     X                                   X      X                  5
Amlodipine                                   X      X                                               2
Amytriptyline                   X            X      X                                               3
ASA                                   X      X             X      X                      X          5
Atorvastatin                                 X      X      X      X       X              X          6
Azathioprine                                        X                                               1
Azithromycin                                                              X                         1
Budesonide                                                        X                                 1
Bupropion                                    X                                           X          2
Calcium                   X     X            X      X      X      X       X              X          8
Cefuroxime                X                                                                         1
Celecoxib                                    X             X              X              X          4
Cerivastatin                                 X                                                      1
Chloral-hydrate                                                                                     0
Chromium                                                   X      X       X                         3
Citalopram                                          X                     X              X          3
Clonazepam                                                                                          0
Clopidogrel                                  X                                                      1
Cyclobenzaprine                                                   X                                 1
Table 10. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Was
          Observed Between Raters At TIME 2a On A 5-Point Scale According To The Criteria Composing The MAI
                                          Criterion number composing the MAIa                      Total
Drug                      1     2     3      4      5      6      7       8      9       10     (Number of
                                                                                                   X)
Cyproterone                           X                                                             1
Digoxin                                      X      X                                    X          3
Diltiazem                       X            X      X      X                             X          5
Docusate                              X      X      X             X                                 4
Domperidone                                                                              X          1
Efavirenz                                                                                           0
Enalapril                                                                                           0
Estrogen                  X     X            X      X      X      X       X              X          8
Etidronate                            X                    X                                        2
Etidronate/calcium                           X      X                                               2
Fluconazole                                                       X                      X          2
Fluticasone                                         X                                               1
Folic acid                                                                       X                  1
Furosemide                                                 X                                        1
Glucosamine                                                       X                                 1
Glyburide                                    X                                                      1
Hydrochlorothiazide/
                                             X                                                      1
triamterene
Ibuprofen                                           X                                    X          2
Indapamide                                                        X                                 1
Insulin                                      X                                                      1
Ipratropium                                  X                                                      1
Labetolol                                                                                           0
Table 10. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Was
          Observed Between Raters At TIME 2a On A 5-Point Scale According To The Criteria Composing The MAI
                                          Criterion number composing the MAIa                      Total
Drug                      1     2     3      4      5      6      7       8      9       10     (Number of
                                                                                                   X)
Levothyroxine             X     X            X      X      X      X       X              X          8
Lodoxamide                                                                                          0
Lorazepam                                                                                X          1
Medroxyprogesterone       X     X                          X      X       X              X          6
Metformin                                    X      X      X      X                                 4
Methotrimeprazine               X            X      X             X                                 4
Metolazone                X                                                                         1
Multi Vit.                                   X                                           X          2
Nabumetone                                                        X                                 1
Nadolol                                      X      X                     X                         3
Naproxen                                                                                 X          1
Nifedipine                                                                                          0
Nitrazepam                                                 X                                        1
Olanzapine                                                                                          0
Omeprazole                                   X             X                                        2
Pantoprazole                    X     X      X      X      X      X              X                  7
Perindopril                                                                                         0
Prednisone                                                 X      X                                 2
Psyllium                                     X                                                      1
Ranitidine                                                                                          0
Rofecoxib                                           X                                               1
Salbutamol                                          X                                               1
Senna                                        X      X                                    X          3
 Table 10. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Was
           Observed Between Raters At TIME 2a On A 5-Point Scale According To The Criteria Composing The MAI
                                                Criterion number composing the MAIa                             Total
 Drug                          1     2      3       4       5      6       7        8        9       10      (Number of
                                                                                                                  X)
  Sertraline                                                                                                       0
  Sucralfate                                                                                                       0
  Terazosin                    X     X              X               X     X                          X             6
  Theophylline                                                                                                     0
  Trazodone                                 X                                                        X             2
  Triazolam                          X                      X                                                      2
  Tryptophan                                                X       X     X                                        3
  Venlafaxine                                       X                                                X             2
  Verapamil                    X                    X       X                                                      3
  Vit B12                                                           X     X        X                 X             4
  Vit B6                                                                                                           0
  Vit C                                             X       X       X     X        X                 X             6
  Vit D                                     X                                                        X             2
  Vit E                                             X       X       X     X        X                               5
  Warfarin                                                          X                                              1
  Zafirlukast                                                                                                      0
  Zopiclone                                                                                                        0
a
  Criterion 1: indication; criterion 2: effectiveness; criterion 3: Dosage; criterion 4: correct directions; criterion 5: practical
directions; criterion 6: drug-drug interactions; criterion 7: drug-disease interactions; criterion 8: duplication; criterion 9:
duration; criterion 10: expense.
Table 11. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Occurred Between
          Raters At TIME 2b On A 5-Point Scale According To Criteria Composing The MAI
                                                      Criterion number composing the MAIa                          Total
Drug                             1       2        3        4       5       6       7        8     9      10     (Number of
                                                                                                                   X)
Acetaminophen                                              X       X               X        X            X          5
Amitriptyline                                                              X       X                                2
Amlodipine                       X       X                 X       X               X              X                 6
ASA                                                        X               X                             X          3
Azithromycin                                     X         X       X       X                                        4
Budesonide                                                 X                       X                                2
Bupropion                                X                 X       X       X                X     X                 6
Calcium                                  X                 X       X       X       X        X     X      X          8
Cefuroxime                                                 X       X                                                2
Celecoxib                                                  X                                                        1
Cerivastatin                                               X                                                        1
Cetalopram                                                                 X       X        X     X                 4
Clopidogrel                                                                        X                                1
Cyclobenzaprine                                                    X               X        X            X          4
Digoxin                                                    X       X                                     X          3
Diltiazem                                                  X       X       X                                        3
Docosate                                         X         X       X       X                             X          5
Domperidone                                                                                              X          1
Doxazosin                                                  X                                                        1
Effavirenz                                                 X                                                        1
Etidronate                       X       X       X                 X               X        X            X          7
Table 11. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Occurred Between
          Raters At TIME 2b On A 5-Point Scale According To Criteria Composing The MAI
                                                      Criterion number composing the MAIa                          Total
Drug                             1       2        3        4       5       6       7        8     9      10     (Number of
                                                                                                                   X)
Fluconazole                                                                                              X          1
Fluticasone                      X       X                         X       X       X        X     X      X          8
Fluticasone nasal                X       X                         X       X       X        X     X                 7
Folic acid                                                 X       X       X       X              X      X          6
Hydrochlorothiazide/
                                                           X       X                                                 2
triamterene
Imipramine                                                                                  X                        1
Indapamide                                                 X                                                         1
Insulin                                                    X                                                         1
Irbesartan                                                                                               X           1
Iron                                                               X       X       X        X                        4
Labetalol                                                                          X                                 1
Levothyroxine                                              X               X                                         2
Lorazepam                                                                                   X                        1
Losartan                                                                                                 X           1
Magnesium hydroxide +                    X                                                                           1
mineral oil
Metformin                                                          X                                     X           2
Methotrimeprazine                                                          X                                         1
Metoclopramide                           X                         X                        X            X           4
Metoprolol                                                                 X       X                                 2
Multi vit                                                                                   X            X           2
Nadolol                                                                            X                                 1
Table 11. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Occurred Between
          Raters At TIME 2b On A 5-Point Scale According To Criteria Composing The MAI
                                                      Criterion number composing the MAIa                          Total
Drug                             1       2        3        4       5       6       7        8     9      10     (Number of
                                                                                                                   X)
Nifedipine                       X       X                         X                                                3
Nitroglycerin                                              X                                                        1
Omeprazole                                                         X       X       X        X                       4
Orlistat                                                                                                 X          1
Oxazepam                                                                   X                                        1
Pantoprazole                     X       X       X         X       X       X       X        X            X          9
Perindopril                      X                                                                                  1
Potassium                                                  X                                                        1
Refecoxib                        X       X                 X       X                              X                 5
Salbutamol                                                         X                        X                       2
Senna                                    X       X         X       X       X                X                       6
Sertraline                                                                 X                X                       2
Terazosin                                                  X       X                              X                 3
Terbutaline                                                                        X                     X          2
Triazolam                                X                         X                                                2
Tryptophan                                                         X               X                                2
Venlafaxine                                                X                                                        1
Verapamil                                                  X                       X                     X          3
Vit B12                          X       X                 X               X       X        X                       6
Vit C                                                      X       X       X       X        X                       5
Vit D                            X       X       X                         X       X        X     X      X          8
Vit E                                    X                         X                        X            X          4
Warfarin                                                   X       X               X        X            X          5
 Table 11. Description Of The Drugs For Which A Major Disagreement (Difference Of At Least 3 In Scores) Occurred Between
           Raters At TIME 2b On A 5-Point Scale According To Criteria Composing The MAI
                                                            Criterion number composing the MAIa                                  Total
 Drug                                 1        2        3        4        5        6        7        8        9      10      (Number of
                                                                                                                                  X)
  Zopiclone                                                      X                                                                 1
a
  Criterion 1: indication; criterion 2: effectiveness; criterion 3: Dosage; criterion 4: correct directions; criterion 5: practical directions;
criterion 6: drug-drug interactions; criterion 7: drug-disease interactions; criterion 8: duplication; criterion 9: duration; criterion 10:
expense.
Table 12. Inter-Rater Agreement Between The Two Raters At Time 1 And At Time 2c On A 5-Point Scale For Each Criterion
           For 10 Patients
                                             Time 1                           Time 2b                          Time 2c
Criterion                                (n=87 drugs)                       (n=92 drugs)                     (n=84 drugs)
                                  % agreement            ĸa           % agreement           ĸa         % agreement           ĸa

Indication                            0.38              NA                0.52b            0.27            0.68             NA
Effectiveness                         0.33              NA                0.53 b           NA              0.61             0.36
Dosage                                0.47              NA                0.52             NA              0.58             0.25
Correct directions                    0.26              NA                0.53             NA              0.38             NA
Practical directions                  0.40              NA                0.77             NA              0.74             NA
Drug-drug interactions                0.40              NA                0.50             NA              0.46             NA
Drug-disease interactions             0.33              NA                0.47             NA              0.49             NA
Duplication                           0.29              0.00              0.52             0.30            0.51             0.21
Duration                              0.61              0.38              0.39 b           NA              0.82             NA
Expense                               0.38              0.05              0.48 b           NA              0.60c            NA
                            d                                                    e
Omission (New criterion)              0.20              NA                0.40             NA              0.00             NA
a                      b
 ĸ: kappa statistics. Percent of agreement between time 2b and time 2c is statistically significant at 0.05 level (Chi-square test was
performed on 77 drugs). c Percent of agreement between time 1 and time 2c is statistically significant at 0.05 level (Chi-square test was
performed on 69 drugs). d The unit of analysis was the patient (n=10 participants). e Chi-square test was not computed to compare the
percent of agreement between time 2b and time 2c because at time 4 the percent of agreement between raters was 0% (a constant instead
of a variable). NOTE: The number of drugs between Time 1, Time 2b and Time 2c differs due to changes in drug treatment of patients.
Table 13. Inter-Rater Agreement Between The Two Raters At Time 1 And At Time 2c On A 3-Point Scale For Each Criterion For
           10 Patients
                                            Time 1                            Time 2b                            Time 2c
Criterion                              (n=87 drugs)                      (n=92 drugs)                       (n=84 drugs)
                                 % agreement           ĸa         % agreement            ĸa           % agreement           ĸa

Indication                           0.74             0.14            0.72b             0.30             0.81 c            NA
Effectiveness                        0.71             0.22            0.72 b            NA               0.81 c            0.44
Dosage                               0.60             0.35             0.57             0.30              0.64             0.31
Correct directions                   0.58             0.18            0.64 b            NA               0.52 c            0.22
Practical directions                 0.77             0.16             0.87             0.50              0.81             0.40
Drug-drug interactions               0.63             NA               0.77             0.33              0.71             NA
Drug-disease interactions            0.59             NA               0.78             NA                0.71             0.28
Duplication                          0.67             0.15            0.69 b            0.39             0.58 c            0.23
Duration                             0.71             0.50            0.44 b            0.12              0.85             NA
Expense                              0.67             0.10            0.72 b            0.30              0.76c            0.41
Omission (New criterion)d                                                  e                                  e
                                      0.20           NA                0.40            NA                 0.20              NA
a
 ĸ: kappa statistics. b Percent of agreement between time 2b and time 2c is statistically significant at 0.05 level (Chi-square test was
performed on 77 drugs). c Percent of agreement between time 1 and time 2c is statistically significant at 0.05 level (Chi-square test was
performed on 69 drugs). d The unit of analysis was the patient. e Chi-square test was not computed to compare the percent of agreement
between times because at time 2c the percent of agreement between raters was 0% (a constant instead of a variable).
Table 14. Mean MAI Weighted Scores At Time 1 And At Time 2b On A 5-Point Scale According To The Rater (28 Patients)

                                                                           Mean score (± standard deviation)
                                                             TIME 1                                               TIME 2b
Criterion (range of scores)
                                                           (n=223 drugs)                                        (n=257 drugs)
                                          Reported by rater 1       Reported by rater 2        Reported by rater 1      Reported by rater 2
Indication (3-15)                            4.98 (1.36)               5.40 (1.64)                5.11 (1.11)                6.34 (1.22)a

Effectiveness (3-15)                         4.75 (1.27)               6.02 (1.74)               4.99 (1.03)                 6.57 (1.14)

Dosage (2-10)                                4.80 (1.13)               5.07 (1.16)               5.17 (0.94)                 5.99 (0.52)a

Correct directions (2-10)                    5.09 (1.51)               4.32 (1.25)               4.89 (1.07)a                4.71 (0.99)

Practical directions (2-10)                  3.66 (0.88)               3.29 (1.20)               3.24 (0.64)                 3.32 (0.74)

Drug-drug interactions (3-15)                7.17 (1.49)               5.65 (1.72)               6.78 (1.42)a                6.18 (1.67)

Drug-disease interactions (2-10)             4.85 (1.00)               3.67 (1.49)               4.23 (0.83)                 3.35 (0.87)

Duplication (1-5)                            1.92 (0.66)               1.93 (0.60)               2.22 (0.83)                 2.17 (0.60)

Duration (1-5)                               2.51 (0.59)               2.34 (0.76)               2.54 (1.24)                 3.04 (0.36)a
Expense (1-5)                                2.01 (0.42)               2.08 (0.61)               2.04 (0.87)                 2.08 (0.43)

ALL 10 criteria (20-100)                    42.05 (6.08)              39.95 (9.69)              43.09 (6.65)                40.92 (7.04)
Omission (3-15) (New criterion)b            4.87 (1.63)                3.71 (0.92)              4.91 (1.22)               3.66 (0.96)a
a
  Difference in score between time 1 and time 2b is statistically significant at a 0.05 level according to the rater. b The unit of analysis was
the patient. NOTE: the number of drugs between Time 1 and Time 2b differs due to changes in drug treatment of patients. The lower the
score, the more appropriate is the drug.
\
Institute of Health Economics Working Paper 03-01   39

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:47
posted:3/18/2011
language:English
pages:43