This document outlines the key principles of an outcomes based approach to project evaluation. It aims to provide information for Youth Musics funded partners that will be of use when thinking about how you evidence what changes your work brings about. While the focus is on outcomes, the activities and resources that influence these outcomes should also be critically considered. This is the only way that we will be able to learn from what works well or not so well in our funded projects, and be able to share those lessons with others working in the sector. This document however is about outcomes and what we consider a useful approach to being able to capture them.
Our approach is based on that of the Charities Evaluation Services (CES) 2 . We would recommend that you consult their website for further information on their approach to evaluation (www.ces-vol.org.uk). You may also be interested in their training programme. In writing this document we have also drawn on the Big Lottery Funds outcomes approach, as laid out in their 2006 guidance document 3 (which is also based on that of the CES). We would encourage partners to consult this more detailed report.
We consider this to be a working document that will be added to over time as we and our partners develop more ways of working effectively with this approach.
2. What does the document cover? There are eight more sections in this document:
3. Why monitor and evaluate your work? 4. What are outcomes? 5. Youth Music as an outcomes funder 6. Taking an outcomes approach 7. Outcome indicators 8. Collecting information on outcomes
1 This document does not address the critical topics of codes of practice and ethical guidelines to follow when carrying out research. As a start, you may want to consult the United Kingdom Evaluation services guidelines http://www.evaluation.org.uk/resources/guidelines.aspx on these topics. 2 For the purpose of this document we have in particular drawn on Practical Monitoring and Evaluation: A guide for voluntary organsiations, (CES 2005) 3 Burns, S. & MacKeith (October 2006) Explaining the Difference your Project Makes: A BIG guide to using an outcomes approach 2 9. Making sense of the information 10. Developing this document and related resources
Throughout the document we use an example project Making Sounds to illustrate the principles we are discussing. This is not a real Youth Music project and the examples should be taken as illustrative rather than definitive samples of best practice.
3. Why monitor and evaluate your work? Before we move on to talk about outcomes specifically, we thought it would be useful to explain why Youth Music considers evaluation to be a core component of our and our partners work. Effective evaluation helps you to assess how well you are doing and be more effective in your practices. Obviously delivering projects is everyones priority, but it is critical that you allocate resources to evaluate your work. It will enable you to:
reflect on your practice (both during and after a project) and improve the quality of projects you deliver assess whether you are reaching your target participants or whether you need to amend the project to do so feel more confident that you are delivering an effective project identify both intended and unintended outcomes of your work make a strong case when making future funding applications, by having evidence of your achievements and lessons learnt.
In addition, by submitting the evidence to Youth Music this will help us to: develop a body of evidence of effective practice in the sector to be made available to all stakeholders advocate on behalf of the sector on the wide variety of outcomes music-making projects can achieve identify areas of need, where achieving outcomes may be more challenging and require innovation or extra resources evidence the impact of the work of Youth Music and its funded partners.
4. What are outcomes? Outcomes are the changes, benefits, learning and other effects that you can attribute to your projects activities- theyre the difference your project makes (CES, 2005, p.53). While these will probably focus on the children and young people taking part in your music-making project, they may also relate to a parent or carer, your staff or your organisation as a whole (see examples in section 6.2 below).
5. Youth Music as an outcomes funder Youth Music is clear about the changes we want to bring about through our activities and those of the partners we fund. If you have applied to Youth Music for funding under a particular programme then it should be clear what that programme is trying to achieve. You will be expected to be able to relate at least some of the intended outcomes of your project to those of the programme. As part of the conditions of funding you will be expected to report to us on your progress toward achieving those outcomes.
3
6. Taking an outcomes approach When taking an outcomes approach to your project design it is helpful to think about it on three levels (see Diagram 1 below): 1. Its overall aim 2. Its intended outcomes 3. Its activities
Diagram 1: Charities Evaluation Services Planning Triangle (adapted by Burns & MacKeith, 2006 on behalf of BIG 4 )
6.1. The overall aim of your project What is your project setting out to do? What is the overall change or difference you want it to make? Fundamentally this explains why your project exists.
Making Sounds 5 - Overall aim
Project description A project for young people in contact with a Youth Offending Team (YOT) in London. This is a 12 month project where young people will learn a vocal based skill- singing or beatboxing. They will attend workshops and will be linked to a peer mentor from a local youth arts club, who will also attend the workshops.
Overall Aim To improve the life chances of young people in contact with a London-based YOT, through music-making activity.
4 Burns, S. & MacKeith (October 2006) Explaining the Difference your Project Makes: A BIG guide to using an outcomes approach, p.8 5 Please note: This is NOT an actual project and does not necessarily reflect a project design that Youth Music would/has funded.
The overall aim of your project The differences you intend to make or the changes you aim to bring about for your beneficiaries, the community or the environment The main services and activities you plan to carry out what those working on your project will actually do Overall Aim Intended Outcomes Activities 4 6.2. The intended outcomes of your project Your intended outcomes are the changes that you are aiming to achieve because of your project (i.e. what do you expect to change through your project? What difference will it make, for example, for the young people taking part?). The language you use to articulate these should involve words that reflect change, for example to: increase, reduce, expand, enable develop, improve, etc. (CES, p18).
Your intended outcomes should be linked to the activities you will do (i.e. why are you doing the stated activities? What do you hope to achieve through delivering them?) The changes that result from your activities are your outcomes.
They should also be specific, measurable, achievable, realistic and timely (SMART). Think about the time and resources you have available and the type of activities you are providing, this will help you to devise outcomes (and relevant indicators, see 7 below) that are SMART. They must relate closely to the activities (specific), be counted or described thoroughly (measurable), and be achievable and realistic within the time and resources dedicated to the activities.
Making Sounds 4 - Intended Outcomes
By the end of the 12 month project: To have increased the musical skills of participants. To have improved participants knowledge of music-making opportunities in their locality. To have improved young peoples social and psychological well-being. To have increased young peoples motivation to engage in education and training. To have developed a more positive attitude to young peoples music-making among parents/carers. To have increased the skills and knowledge of the music leaders in working with young people in contact with a YOT.
6.3. The activities involved What are you going to do that will bring about these changes? As mentioned above, these are the activities you are going to deliver through your project. While you use change language for your outcomes (e.g. an increase in musical ability), you should use doing words for your activities (e.g. by providing 12 workshops).
You may find it helpful to think about these within the CES Planning Triangle structure (see diagram 1 for an adapted version of this). This allows you to think about each level and how they relate to each other. Will the overall aim of your project be met through achieving the intended outcomes? Will the activities you have planned realistically enable you to bring about the changes outlined in your intended outcomes? While some activities will help deliver on more than one outcome, it is critical that each activity is linked to at least one outcome, and that each outcome has at least one activity that will ensure your project delivers on it.
5 Making Sounds 4 - Activities
To run weekly workshops for participants during school term time. To employ a skilled music leader to deliver high quality music-making activities. To provide pastoral care to young people over the course of the project, with the support of youth workers from a local arts centre. To provide an information pack to young people and follow up with 1:1 sessions with the music leader on progression routes and further local opportunities for music making. To hold 2 performances to which parents/carers will be invited and to send them an information pack every three months about the project and key achievements of the group. To hold regular meetings between relevant YOT staff, the music leader and youth workers to discuss the projects progress and any issues arising.
Once you have looked at your project within this structure you should ask yourself the following questions developed by BIG for their partners:
When you have filled in a triangle, check the following: Have you used words of change in the top and middle and doing words in the bottom of the triangle? Does the middle of the triangle describe what the changes you intend for your projects participants, their parents/carers, your staff etc? Does the bottom of the triangle describe what those working on your project are going to do?
Now you can look at the triangle as a whole to check that the plans for your project are realistic: Look at each of your intended outcomes in the middle level. Will the activities you have listed at the bottom realistically help you to achieve one or more of your intended outcomes? Now look at the activities. Does each one link directly to one or more of the intended outcomes? If not, why is the activity included? You may want to consider whether the activity really is important. It may well be that the activity will help you to achieve another change. If this is the case you may want to include another outcome. It may help to do this if you number the outcomes and put the number(s) against each activity. From Burns & Mac Keith, 2006 (p. 10)
7. Outcome indicators Once you have decided on your intended outcomes you will need to identify the indicators you will use to evidence your progress in achieving these outcomes. What would you see, hear, or read about project participants that would show you had made progress toward your outcome? What would participants be doing differently? So, before your project starts you will need to decide what information you want to collect to evidence you progress to achieving your outcomes (outcome indicators) and how to collect the information needed. As with setting your outcomes, it is important that youre realistic about the number of indicators you set out to monitor and what kind of information you will be able to collect (see section 8 below).
6 Making Sounds 4 - Outcome Indicators
Outcomes Possible outcome indicators Increased musical skills
Young peoples own assessment of musical skills
Music leaders rating of young peoples musical skills
Improved knowledge of music- making opportunities in the locality
Level of understanding of own strengths and weaknesses
Whether participants can think of several ways of finding out about different local music- making opportunities. Improved young peoples social and psychological well-being
Whether participants feel: able to express themselves in front of others able to make decisions that they feel are good for them listened to by other people like what they say and do will make a difference to their lives and that they have: been able to turn up to the workshops on time learned to work well with other people
Increased motivation to engage in education and training
Level of perceived motivation to engage in education/training
Attendance levels at education/training A more positive attitude to young peoples music-making among parents/carers
Level of parent/carer attendance at performances
Parents/carers views on the value of music making for their young people and young people more generally
Increased skills and knowledge of the music leader in working with young people in contact with a YOT
Whether music leader feels well-equipped to meet the needs of young people in contact with a YOT
Whether music leader can identify a range of sources of support upon which he/she can draw to be able to effectively deliver the programme of work with this group of young people.
Some of the outcomes you set out to achieve may be difficult to measure and count. The CES (2005, p 23) give the example of the outcome to change young peoples attitudes about social issues, which cannot be measured easily, so you would need to use indicators that assess the change approximately (proxies). They suggest that using time keeping and attendance levels, alongside other evidence may be an indication of an increased sense of commitment and responsibility, which relates to the intended outcome. 7
8. Collecting information on outcomes As mentioned in 7 above, before starting your project you will need to think about how you will collect the information needed to demonstrate the progress you have made in achieving your outcomes. Maybe you already collect certain kinds of information as part of your monitoring systems that you could use? Maybe you could tweak the monitoring system you already use by rewording or adding questions? However, you may find that you need to find a new way of collecting the information. If you are looking at change you will need to think about collecting information at two points in time (at/towards the start and at the end of the project). It is beyond the scope of this document to provide a detailed account of tools/methods that you may want to use, as these will vary from project to project and will depend on the outcomes chosen 6 . For example, information on the outcomes for young people could include: Observation of their level of engagement/skills Self-assessment questionnaires Outcome scales Diaries they keep as part of the project Notes kept by project leads Interviews Focus groups Samples of work Etc.
As with all elements of this process it is important that you are realistic about what data collection you can do within the given resources.
Making Sounds 4 - Data collection tools for increased music skills outcome
Outcome indicator Data collection tool Young peoples own assessment of musical skills Young people complete at the beginning and again at the end of the project a scale that asks them to; Please mark your responses to the following questions on the record, where 1 is low and 8 is high
How do you rate your musical ability at the moment? 1 being that you dont feel you have any musical ability, 8 being that you think youre at the top of your game!
Music leaders rating of young peoples musical skills Music leader completes a log at the end of each workshop, noting progress made by each participant.
6 Practical Monitoring and Evaluation: A guide for voluntary organsiations, (CES 2005) provide some examples of data collection tools and some initial guidance on data analysis. 8 9. Making sense of the information The way you make sense of the data you collect on your outcomes will depend on the kind of information you have collected and the way in which you have collected it. For example, you will need to use a different approach to analysing numbers to that adopted if you have done interviews with young people. As with the tools for collecting data, it is beyond the scope of this document to provide an in-depth account of how to analyse the data you collect 6 .
Making Sounds 4 - Making sense of the information for increased music skills outcome*
Young people varied in the extent to which they felt their musical skills had increased over the course of the project. On average, participants reported a 4 point increase (on an 8 point scale) in how they rated their musical skills. None of the participants reported a decrease in their skills. Four felt they had only improved by 1 point. However, the music leaders observations noted in the session log show that three of these participants had initially ranked themselves toward the top end of the scale. Once they began to learn more about their chosen vocal skill their self-awareness had improved and they realised that they had more to learn to be top of their game. For the other participant in this group of four, both he and the music leader felt his musical skills had improved little over the course of the project. *This example only deals with reporting on the actual outcome and does not take account of, the activities and resources that influence these outcomes which should also be critically considered.
10. Developing this document and related resources Data collection tools If you have developed any tools (such as questionnaires for young people) to collect information on the outcomes of your project, and would be happy to share them with others working in the area we would greatly appreciate it if you could send them through to us. We would like to develop a repository of tools that could be accessed by funded partners.
Let us know what you think As mentioned in section 1, we consider this to be a working document and would greatly appreciate your feedback on how we can improve it. We also plan to develop it further by drawing on examples from our partners work in this area. Please forward any comments to research@youthmusic.org.uk
Other useful sources: http://www.biglotteryfund.org.uk/er_eval_explaining_the_difference.pdf Big Lottery Fund- Explaining the difference your project makes: A BIG guide to using an outcomes approach http://www.biglotteryfund.org.uk/er_eval_self_evaluation_uk.pdf Big Lottery Fund- Self Evaluation: A handy guide to sources
www.ces-vol.org.uk Charities Evaluation Services
http://www.evaluation.org.uk United Kingdom Evaluation services