Learning Center
Plans & pricing Sign in
Sign Out





The University of Manchester, UK Mancunian Community Health NHS Trust, UK

CPLOL Lisbon Congress

1st - 4th May 1997

INTRODUCTION The ability to measure the outcome of health care services is of great importance to the UK National Health Service in its pursuit of quality and cost-effectiveness. With a remit to buy effective clinical services, healthcare purchasers are seeking out information to help them in their decisions and it is important that those decisions are based, not on historical practice, professional muscle-flexing or simply relative cost, but on accurate demonstrations of the effect of intervention. The importance of the outcomes issue can be seen in the establishment, by the Department of Health, of the UK Clearing House for Information on the Assessment of Health Outcomes, intended as a repository of information and advice on the development and application of outcome measures. Aphasia therapy specifically has been the focus of outcomes investigations longer than most areas, following the publication around 15 years ago of a series of group studies comparing professional intervention by speech and language therapists to voluntary support and to no intervention. Since then there has followed a series of well-designed case studies and replications. Unfortunately, these case studies which, on the whole, show positive outcomes of therapy, are not particularly accessible to purchasers. This is partly due to their scattered publication in a variety of journals and partly because their content is often in such clinical and theoretical detail: hours of reading are necessary to ascertain the outcome of therapy for a handful of patients. The papers are of enormous value to SLTs, but not appropriate for use by purchasers who are likely to look to more simply-presented and apparently generalisable information. Recent medical papers on stroke rehabilitation unfortunately continue to rely on the older group studies of aphasia treatment to inform their opinion of speech and language therapy in this area. They say that the evidence for speech therapy after stroke at currently provided levels is conflicting and wonder whether therapists' time could be better used in ways other than direct therapy, or our input restricted to assessment and advice only. There is a danger that purchasers might follow this suggestion, contracting speech and language therapy for a limited range of services only, unless they are provided with adequate information by SLTs themselves. Given the above, it is obviously vital to our profession that we continue to measure and disseminate the results of good practice wherever appropriate. The increasing requirement for outcome measures is therefore timely and therapists should seize the opportunity to develop sound and efficient measurement tools with which to evaluate services. Nevertheless, it will be difficult to construct tools which demonstrate clearly (yet quickly) the effects of our service. In 1995, we distributed a questionnaire to services throughout England and Wales, including factual and open questions about outcome measurement for aphasia therapy and requesting examples of measures being used. We received 115 replies, and 34 examples of outcome measures. Approximately 50% of respondents were already using outcome measures, and the majority of the remainder were in the process of devising them. I will summarise the main types of approach used, and what we thought were the advantages and disadvantages of each.


TYPES OF MEASURES We received 34 examples of outcome measures from respondents and, within this number, some patterns of approaches were observed. Measurement techniques included: rating scales, 'content free' measures, general rehabilitation scales questionnaires discharge codes computer based approaches. They are much too varied to give details of all them here but we can talk about the main types of approaches we found (the first three listed above) and our impressions of their likely advantages and disadvantages.


Rating scale approaches

The most common approach was to use rating scales, either using the Enderby/WHO headings (Impairment, Disability, Handicap, Distress), or similar. Examples of different rating scales are shown on the slide.

Impairment Disability

Impairment Function Effect on patient

Handicap Coping Distress Distress Understanding of condition

Pam Enderby’s Frenchay rating scale is an attempt to devise an outcome measure which can be used for any SLT client group and which covers all aspects of intervention. The impairment scales are specifically written for each major client group, but disability, handicap and distress scales are identical or very similar across all conditions. Each was originally a six-point rating with a written description at each stage, though the use of mid-points between each descriptor has now been introduced, giving an 11-point scale. The 'distress' scale has also now been re-titled 'well-being'. Others had developed their own rating scales using a varying number and title of divisions but mostly conforming to a six-point rating scale with written descriptors for each point. Some had


further subdivided the impairment scale into a series of different subskills (for example, naming, gesture, repetition) each of which had its own 6-point scale with descriptors. The main advantages of rating scales are in their simplicity. speed of completion (when familiar with the system) easy to report (number of points gained) easy to summarise - purchasers are presented with figures which are in a consistent format across patients and which are easily interpretable by non-SLTs. potential comparability - (across services / client groups / districts) Disadvantages: patients original profile, or pattern of improvement may not fit descriptors very well they usually assume a similar level of impairment over all language modalities. This can make either the initial position or the progress difficult to plot in a patient whose main problem and focus of therapy has been in one modality only. comparability: they are ordinal, not equal interval scales. This means that a move from a score from 1 to 2 on a scale may not be of the same 'value' as an increase from 3 to 4, or the same as an increase from 1 to 2 on a different scale (eg fluency). Reliability: is potentially a limitation with entirely subjective judgements being made by interested parties. It is here that the opinion of patients or carers may be useful as a comparative opinion on outcome.

2 'Content-free' measures We used this term for approaches which had no pre-written descriptors and, often, no pre-defined guidelines on setting aims, or recording progress. Some of these were very minimalist, eg as in the slide: NAME.............................................




Outcome and comments

An unstructured 'outcome' or 'comments' column will inevitably lead to a wide variety of responses which might be impossible to summarise and report. A common way of adding structure was therefore to standardise how outcome was reported, for example coding goals as achieved, partly achieved, unachieved; or patients described as unchanged, slightly improved,


significantly improved. (on slide) Advantages: can write relevant, specific aims for each client, therefore can make the measure sensitive to small amounts of change, It would be possible to write aims related to any aspects of a therapist's work, for example information-giving, or alteration of a carer's communication strategies; aspects which were not easily reflected in most rating scales. Disadvantages: can be difficult to summarise where no output coding specified, even where outcome is in a consistent format, comparability is low. Percentage goals achieved says nothing about the relative size or clinical significance of the goals. In fact, this method of summarising could encourage the setting of inappropriately small, easily achievable but non-functional goals, in order to improve the figures.

3 General rehabilitation scales Scales reflecting mobility and activities of daily living have a long history of use in rehabilitation settings. Modern versions such as the Functional Independence Measure include sections for communication and cognitive skills, and therefore have potential for use by SLTs. But the examples sent in show that there remains a mismatch between the detail available for recording communication progress and that for physical skills. (see slide) Mobility related 5 or 6-point rating scales for:

sitting transfers standing stepping walking ULF sitting proximal ULF distal ULF stairs

Communication related 4 & 5-point rating scales for:

understanding expression

The mobility and communication scales above are quite different in sensitivity and give the SLT less opportunity to show progress. Taking as an indication of the number of different sub-scales multiplied by the number of points on each scale, one example showed a mobility range of 48 points (over 9 sub-scales) and a communication related spread of 9 points (over 2 sub-scales). So the physiotherapist would be able to look separately at skills such as sitting, transfers, standing, stepping, walking (etc). A similar level of detail would enable the SLT to rate, for example, single word comprehension, syntactic comprehension, word finding, sentence production, use of gesture and so on rather than, as at present, reducing the complexity of communication into just two scales for understanding and expression of language.


These scales generally reach a higher level of psychometric credibility than others; the FIM for instance is an equal-interval scale with established reliability and validity and yet their sensitivity, in the SLT area at least, is limited. Sensitivity is often overlooked, but extremely important. An instrument which is unable to detect relevant changes is useless for outcome measurement. SLTs should be wary of involvement in an outcomes project where their intervention is likely to show up less favourably than that of other professionals, due to limitations of the measurement tool.

FURTHER ISSUES Measuring outcome against prediction The simple reporting of outcomes can give the impression that a small amount of change was a bad result implying a relatively ineffective service. The measurement of outcome against prediction can be used to demonstrate that a small amount of progress was, in some cases, the expected and appropriate result. For examples some of the measures we saw required prediction of eventual outcome, or outcome after a specified period. The outcome may be expressed in terms of a 'disability score'; often a sum of the score on several rating scales. Approaches varied in the time periods over which goals had to be set and achieved: some required an indicated timescale for each goal, whereas another specified particular times and asked the therapist to set relevant goals for each. We feel that both a measurement of progress (such as a test or rating scale) and the comparison of outcome against prediction are needed in order to interpret outcome data appropriately. Inclusion of patients' / carers' opinions Patients' and /or carers' opinions about the effects of intervention are an appropriate part of the outcome measure and most respondents were attempting to include them. They may take the form of a rating of change in ability, knowledge, adaptation or behaviour and could be a useful check on the reliability of the therapist’s scoring. But, since therapists require training to achieve reliability on scoring, it isn’t clear how reliable clients and carers would be. Also they may find it difficult to separate out their satisfaction with the outcome from satisfaction with other aspects of the service (eg length of waiting list, whether they liked the therapist.) Attributability The issue of attributability is important, but is hard to establish. People will naturally view a patient's situation holistically, and satisfaction with outcome and functional performance may reflect the interdependence of a range of skills. Thus a therapist may not be judged only on the merits of her particular intervention, but on the success of rehabilitation overall. This is an issue wherever multi-disciplinary functional assessment is taking place. Spontaneous recovery is also an issue in attributability. Aphasia therapists are frequently working within the commonly accepted limits of spontaneous recovery, which could of course be partly responsible for any measured change.



Purpose: There was great variety among respondents in their expectations of the information that outcome measures could deliver. One distinction was between those who viewed the measures as potential research tools in the demonstration of effective therapy approaches and those who saw them as an audit-based screen of services. To us, outcome measures do not seem the best way to conduct rigorous experimental research into therapy: it is too much to demand of our outcome measures that they both prove efficacy and measure service-wide outcome. We need to proceed on two fronts, using outcome measures as an audit tool for the monitoring of the outcome of therapy services, assuming (perhaps prematurely) that those services are of value, while continuing to carry out experimentally rigorous research studies to establish effective treatment approaches for aphasia. Content: Content of the measures differed greatly. In choosing measure, we should consider the reasons for doing them. If we require our outcomes to demonstrate positive change, it is important to have comprehensive measures that reflect what we actually do in therapy. We cannot focus purely on language impairment using standardised test scores pre and post intervention. We need to include the wider aspects of our therapy with the potential for change in functional communication, psychosocial status and patient/carer attitude and anxiety. Unfortunately, the attempt to consider such a wide range of factors in every aphasic client may also make outcome measurement unwieldy both for therapists to complete and for purchasers to interpret. Compromises: Many respondents expressed the need to have measures which are quick to administer, and yet the same therapists were anxious to have psychometrically sound assessment. This raises the question of whether these two aspects are compatible. Our review suggests that so far clinical feasibility has, not surprisingly, overridden psychometric considerations. Sensitivity, reliability and attributability will continue to be problems within a quick and broadly applicable measurement tool. It is clear that we are not yet anywhere near a nationally agreed approach to outcome measurement, and it is questionable, of course, whether such an agreement is desirable. It is also clear that in every case a compromise must be achieved between sensitivity on the one hand and speed and 'conformity' on the other. As one side increases, the other will inevitably suffer. Our paper provides no simple answer, but highlights the complexity of designing outcome measures which are both meaningful and do-able. Nevertheless, sharing of information about attempts to measure outcome should reduce duplication of effort and help in the dissemination of successful solutions which should benefit the profession of speech and language therapy as a whole.


To top