OUTCOMES

Measuring up
READ THIS IF YOU WANT TO COMPARE AND CONTRAST WHAT COLLEAGUES ARE DOING IDENTIFY GAPS IN RESOURCES DEMONSTRATE YOU ARE MAKING A DIFFERENCE

Given the many reasons in favour of outcome measurement it is important we know what we need to measure and how we are going to do it. Dianne Webster reports on the questions arising from her preliminary investigation of outcome measurement by therapists working with people with aphasia in Sheffield.
eral months and years after. Therapists differ in their experience and there is also variation in available resources, both materials and staffing. Speech and language therapists in Sheffield have a number of tools which could be used to measure outcome. These include: formal, standardised assessments (such as TROG, Boston Naming Test, PALPA, Comprehensive Aphasia Test) or informal checklists which are used where therapy has targeted a specific element of language processing in an impairment-based approach. measures of mood and self-esteem (for example VASES, SAD-Q or DISCS (Turner-Stokes et al., 2005)) or informal rating scales if therapy has targeted psychosocial issues. TOMS or informal observational checklists, measuring therapy outcomes that have targeted access to community activities and participation in society. For the project audit I used criteria based on guidance specific to the profession, including RCSLT Clinical Guidelines (2005). I formulated standards (table 1) based on my own clinical experience and informal discussion with colleagues. To find out if these standards were being met I sent a questionnaire (based on Simmons-Mackie et al.,
Table 1 Standards

ver recent years we have become increasingly aware of the need to measure outcomes, so that we know our intervention is having an impact on clients lives. As a recently qualified therapist I was keen to get to grips with the outcome measurement tools used in my team and the wider service. I wanted to find out whether or not therapists were measuring outcome, what they were measuring, the main factors affecting outcome measurement and perceived barriers. With the support of colleagues I undertook this as a project, carrying out a literature review followed by a baseline audit and a preliminary investigation of local practices to assess the impact and outcomes of clinical service (RCSLT, 2006) at an individual therapist level. The literature review in April 2005 revealed few articles that had investigated the use of outcome measures in everyday practice. Of these, studies such as those by Simmons-Mackie et al. (2005) and Hesketh & Hopcutt (1997) revealed diversity in the range of outcome measurement tools used by clinicians. Tools were formal and informal; functional and impairment based. Account for variability Attempts to define outcome measurement reflect the wide range of variables involved. Simmons-Mackie et al. (2005, p.2) aim to account for this variability rather than trying to pin down a definition: What rehabilitation professionals measure in outcome assessment, as well as when or how outcomes are measured, depends on a range of variables such as the philosophical and experiential orientation of the person doing the assessment, the intended audience, the time and resources available, and the specific purpose of the assessment. This is certainly a helpful approach when we are considering outcome measurement in a real world setting. Aphasia therapy in Sheffield takes place in a variety of settings at different points along the stroke pathway; from the acute phase right through to sev-

2005) to a group of Sheffield speech and language therapists working primarily with adults with aphasia following stroke. The questionnaire (available in full via the Extras group on the members area at www.speechmag.com) gathers information about the setting(s) the respondent works in and the general stage they see clients at following a stroke, along with their level of experience and continuing professional development. It also asks how much time they can offer a client with aphasia and whether computer based interventions are included in their service. Respondents are invited to define outcome measurement and explain the theoretical or philosophical models or frameworks that currently influence your approach to outcome measurement in aphasia. They then answer questions about whether, when and how they measure change before moving on to sections about what influences decision-making (for example aphasia severity / evidence of spontaneous recovery / motivation of client) and whether there are significant barriers to conducting outcome assessment in your setting / with individual clients. Finally, respondents are given a list of 26 outcome tools suitable for use with people with aphasia and asked to indicate whether they use them and, if so, why (for example to determine what to do in therapy / track client progress).
Results 89% respondents measure therapy outcome 100% of those who use tools do so in 2 or more areas

Criterion Outcomes of therapy should be routinely measured (RCSLT, 2005, p.17)

Standard All therapists will measure outcomes of therapy

Measurement will reflect the range of 80% of therapists will use tools in interventions delivered and the aims agreed 2 or more of the following areas: for therapy (RCSLT, 2005, p.17) impairment, activity, participation, well-being (WHO, 2002) Speech and language therapists must 80% of therapists will be informed of have available a wide variety of potential developments in theory / practice of approaches to therapy, including awareness aphasia therapy of the latest techniques, and be informed of developments in the theory and practice of aphasia therapy (RCSLT, 1996, p.164). This will then inform outcome measurement in aphasia.

89% reported involvement in a specific aphasia-related activity

SPEECH & LANGUAGE THERAPY IN PRACTICE Winter 2007

OUTCOMES

Nine of the ten questionnaires were returned. I was interested to note that: 1. Eight of the nine responding therapists reported that they measured outcomes. (The other therapist worked in the acute setting and our definition of outcome measurement did not include measurement of communication to staff and family or communication facilitation.) 2. Time constraints were reported most frequently as a barrier to outcome measurement. Reported time taken to measure outcome ranged from 30 minutes to 3 hours and related to time available for therapy. Therapy intensity differs according to setting and resources (such as assistants), with more typically available in intermediate care and inpatient rehabilitation settings. 3. All 8 therapists reported using tools that involved a range of the four categories: body functions and structures (impairment), activity, participation and contextual factors (well-being). As this mirrored the therapists reported influences to outcome measurement, with all respondents mentioning a range of philosophies / models, this suggests that therapists are attempting to address and target clinical areas at a range of levels. 4. Despite therapists working at a range of levels, the majority of outcome tools used were at an impairment level (such as the Comprehensive Aphasia Test, Pyramids and Palm Trees), with only 26 per cent at an activity / participation level (for example informal functional checklists, observation checklists) and a mere 9 per cent for well-being (such as VASES). As all therapists included well-being in their definitions and use of tools, this suggests to me that more tools are needed to assess directly the impact of therapy at the levels of activity, participation and well-being.Enderby & Emerson (1995) comment that most studies investigating aphasia therapy use standardised tests to measure aphasia outcome, yet these tests may be measuring little or nothing that has been targeted in therapy. The development and availability of valid tools that measure the impact of intervention at the levels of activity, participation and well-being would allow clinicians to capture the breadth of intervention and subsequent impact on the individual. This

can then provide information to service providers and commissioners on the full range and value of speech and language therapy and its relevance to service users. 5. Respondents definitions of outcome assessment varied, with disagreement about whether this should be broad or linked to a specific area of treatment. More consensus is needed on how intervention is measured. If clarity is gained on the factors that should be considered when defining measurement, tools can then be carefully chosen to capture these parameters. 6. Many therapists reported involvement in an aphasia-specific activity. We can assume that, in addition to a specific interest in aphasia therapy, these therapists have access to research and best practice findings, and that this should inform outcome assessment and enhance good practice.

NHS, the Payment by Results system (DoH, 2002) encourages efficiency, and so measuring outcome is imperative. Commissioners also want to know that services offer value for money and meet the needs of a given population. Given the importance of this subject, further discussion and reflection on the remaining questions is needed and will form the basis of future developments on outcome measurement within speech and language therapy in Sheffield. Dianne Webster is a Specialist Speech and Language Therapist with Sheffield Primary Care Trust, e-mail d.webster@nhs.net, telephone 0114 2264034.

Acknowledgements

Questions raised

One of the main outcomes of this project was the many questions it raised: How is outcome measurement viewed? Is it seen as an integral part of a therapy block or viewed as an additional activity? How can more time be made available to measure outcome? How is outcome measurement defined and what influences this decision? Can we gather more examples of therapy currently provided at the levels of Impairment, Activity, Participation and Well-being? How can the impact of our therapy on the individual be measured appropriately, across the breadth of our input? What limitations / barriers do the care pathway and particular models of service delivery place on individual accessibility to therapy services, subsequent therapy and outcome assessment? How is the client and carers perceived impact of therapy being measured? How is this being used to inform service planning and therapy focus at different stages of the care pathway? Outcome measurement for aphasia was reported by Sheffield speech and language therapists to take place routinely. This is reassuring as it is widely accepted that measuring outcome of therapy is good practice (RCSLT, 2005; RCP, 2004). Within the

I am grateful for the time, advice and support provided by my colleagues and managers within the Sheffield Speech & Language Therapy Service. With particular thanks to Caroline Haw, Clinical Supervisor and Dr Caroline Pickstone, Research Lead. SLTP

REFLECTIONS WHEN I WANT TO FIND SOMETHING OUT, DO I START WITH BASIC TOOLS SUCH AS A LITERATURE REVIEW AND AUDIT? DO I GET INVOLVED IN GROUPS WHERE I CAN DEVELOP MY KNOWLEDGE AND SKILLS? ARE THE OUTCOMES TOOLS I USE APPROPRIATE TO THE CHANGES I WANT TO CAPTURE? How has this article been helpful to you? What are you doing to measure the outcome of your intervention? Let us know via the Winter 07 forum on the www.speechmag.com members area.

References
Department of Health (2002) Reforming NHS Financial Flows. Introducing Payment by Results. Crown Copyright. Enderby, P . & Emerson, J. (1995) Does Speech and Language Therapy Work?: A Review of the Literature. London: Whurr Publishers Ltd. Hesketh, A., & Hopcutt, B. (1997) Outcome measures for aphasia therapy: Its not what you do, its the way you measure it, European Journal of Disorders of Communication 32 (3), pp.198-203. Royal College of Speech and Language Therapists (2005) Clinical Guidelines. Bicester: Speechmark. Royal College of Physicians (2004) National clinical guidelines for stroke. 2nd edn.. London: RCP . Royal College of Speech and Language Therapists (2006) Communicating Quality 3. London: RCSLT. Royal College of Speech and Language Therapists (1996) Communicating Quality 2. London: RCSLT. Simmons-Mackie,N., Threats, T.T. and Kagan, A. (2005) Outcome assessment in aphasia: a survey, Journal of Communication Disorders 38, pp.1-27. Turner-Stokes, L. Kalamus, M., Hirani, D. & Clegg, F. (2005) The depression intensity scale circles (DISCs): a first evaluation of a simple assessment tool for depression in the context of brain injury, Journal of Neurology, Neurosurgery and Psychiatry 76, pp. 1273-1278. World Health Organisation (2002) Towards a Common Language for Functioning, Disability and Health. Geneva: WHO.

Resources (availability)
Armstrong Naming Test Whurr Publishers - out of print Boston Naming Test http://www3.parinc.com Comprehensive Aphasia Test www.languagedisordersarena.com The Pyramids and Palm Trees Test www.harcourtassessment.com Psycholinguistic Assessments of Language Processing in Aphasia (PALPA) www.languagedisordersarena.com SAD-Q (Stroke Aphasic Depression Questionnaire) freely available on www.nottingham.ac.uk/iwho/general/links.php TOMS (Therapy Outcome Measures for Rehabilitation Professionals) www.wiley.com Trog-2 (Test for Reception of Grammar) www.harcourt-uk.com VASES (Visual Analogue Self-Esteem Scale) www.speechmark.net

SPEECH & LANGUAGE THERAPY IN PRACTICE Winter 2007

reprinted from www.speechmag.com

Pilot Questionnaire: Sheffield Outcome Measurement Aphasia Project

Name (optional) Email Address (optional) Telephone Number (optional)

What type of setting do you work in? (Please tick all that apply): Acute hospital Outpatient Nursing Home Intermediate Care Unit / Step-down facility Intermediate Care: Residential Home Other (please state): Inpatient Rehabilitation Domiciliary Setting Residential Home Intermediate Care: Nursing Home Intermediate Care: Domiciliary

How many years of experience do you have working with adults with aphasia?
Less than 3 years 3 to 5 years >5 to 10 years >10 to 15 years >15 years

On average, at what stage after their stroke are the people you see? (Please tick all that apply)
Less than 1 month post stroke > 6 months to 1 year >18 months to 2 years 1 to 3 months > 1 year to 18 months > 2 years >3 to 6 months

On average, how much time per week can you offer a client with aphasia please state: i) ii) Whether programmes are carried out by yourselves or by therapy assistants. How much time is spent in face-to-face contact with the client.

Do you use technologically/computer based interventions in your therapy? If yes, please specify what these are and whether they are controlled remotely by the therapist, whether they require SLT input for each session etc

Yes No

Do you: (Please tick relevant box)

reprinted from www.speechmag.com

i) ii) iii) iv)

Hold any postgraduate level qualifications specifically relating to aphasia? Yes No Attend aphasia-specific Special Interest Groups, discussion groups? Yes No Hold BAS membership Yes No Attend journal club/ reviews relating to aphasia? Yes No

How would you define outcome assessment?

If there are particular theoretical or philosophical models or frameworks that currently influence your approach to outcome measurement in aphasia, please state these below:

Do you measure change in aphasia therapy? Yes No (If No, go to Q,12)

At what intervals do you administer tools that help determine the outcome of therapy for aphasia? (Please tick relevant boxes)
i) At intervals during therapy ii) At the end of an episode of care iii) After an episode of care i.e. at a specified review point Yes No Yes No Yes No

If yes to iii), how long after the episode of care has ended would you measure change: i) Within one month after iii) > 3 to 6 months v) > 9 to 12 months ii) >1 to 3 months iv) > 6 to 9 months vi) > 12 months

What is your estimate on the average time spent on each outcome assessment for each individual client with aphasia? < hour > 3 hours > hour to 1 hour > 1 to 2 hours > 2 to 3 hours

reprinted from www.speechmag.com

For the following factors, please tick whether they would influence your decision regarding: i) Factor type of therapy to offer ii) choice of outcome assessment Would it affect therapy focus? Yes No Would it affect choice of outcome measure? Yes No

Side of CVA Aphasia severity Handedness Whether clients complete homework exercises Intensity of therapy available Co-existence of depression Presence of apraxia Presence of swallowing difficulties Whether problem-solving skills are preserved Presence of dementia Evidence of spontaneous recovery Motivation of client Home setting SLT attitude toward brain plasticity Please state any additional factors that you feel are important in determining the type of therapy and type of outcome assessment.

Are there any significant barriers to conducting outcome assessment in your setting/with individual clients (e.g. time constraints, funding etc.) Yes No If Yes, what are these barriers:

reprinted from www.speechmag.com

14. In the table below, please indicate with a tick which tools you use with people with aphasia and their purpose:
Outcome Assessment / Tool Determine what to do in therapy Track client progress Show outcome for referral Other (please specify)

Arizona Battery for Communication Disorders of Dementia (ABCD) Armstrong Naming Test Boston Naming Test (BNT) Care Aims Cognitive Linguistic Quick Test (CLQT) Comprehensive Aphasia Test (CAT) Conversation Analysis Profile for People with Aphasia (CAPPA) Discourse/Spontaneous speech analysis

(please specify.)
Functional Assessment

(please specify)
Functional Communication Checklist (FCC) Functional Communication Profile Informal aphasia assessments

(please specify )
Informal functional measure

(please specify.)
Interview (please specify ..) Measure of Cognitive Linguistic Ability (MCLA) Mt Wilga High Level Language Assessment Mini Mental Status Exam (MMSE) Observation (please specify .) Psycholinguistic Assessments of Language Processing in Aphasia (PALPA) Pyramids and Palm Trees Reading Comprehension Battery for Aphasia (RCBA) Therapy Outcome Measures (TOMS) Tree of Life Test For Reception Of Grammar (TROG) Visual Analogue Self Esteem Scale (VASES) Western Aphasia Battery (WAB)

Please note below any additional information that you feel might be helpful in this project.