Interpretación Australia

A Practical Guide to Evaluating Natural and Cultural Heritage Interpretation
Exit Interviews Observation Methods Focus Group Discussions
Gillian Savage Jane James
2001
The Authors
Gillian B Savage, BA Gillian is a Director of the Centre for Visitor Studies which specialises in conducting audience research and evaluation for natural and cultural heritage places. Her tertiary training encompasses education, literature and the social sciences, and she has worked extensively conducting evaluation research in places such as museums, galleries, botanic gardens, zoos, cultural attractions and national parks. With a depth of hands on experience, Gillian is active in exploring the theoretical background of research methods and in enhancing industry practice. In particular, she runs the website of the Evaluation and Visitor Research Special Interest Group (EVR SIG) of Museums Australia. This valuable resource for Australian practitioners can be found at: www.amol.org.au/evrsig/
Dr Jane A James, BSc (Hons), PhD Jane is a Senior Lecturer at Flinders University in South Australia and Coordinator of the Cultural Tourism degree programs. She has worked in tourism education for the past 11 years and has expertise in conducting training and interpretation programs for Government agencies, tourism operators and special interest groups. She has extensive consulting experience in user demand surveys and analysis at Festivals and Events, and in Regional Tourism and at Heritage Sites in South Australia. She has conducted visitor satisfaction surveys in partnership with the South Australian Tourism Commission and Australian Major Events at a number of festivals, including Tasting Australia and Opera In the Outback. She also has experience in interpretive planning, design and management. Jane has worked as an international consultant, educator and researcher in the UK, China, Europe and Canada.
Locked Bag 2116 North Sydney NSW 2059
GPO Box 2100 Adelaide SA 5001
Version 03.1 Gillian Savage and Jane James, 2001.
Practical Guide to Evaluating Natural and Cultural Heritage Interpretation
Contents
Foreword Introduction 1. Interpretation 2. Evaluation 3. Exit Survey 4. Focus Group Discussions 5. Observation Appendix
4 5 6 7 11 17 23
Foreword
ACTION This Practical Guide presents a skills-based program. In a skills program, what you learn depends on what you do. Research shows that unless you use a new idea within 24 hours, you will probably forget what you have learned. So make a phone call, write a note, tie a string on your finger do something that will turn what you have learnt into action. New skills are acquired and old ones improved by practice. So, run through it a few times, repeat it in different situations, try it out alone or with friends or colleagues.
ACTIVELY ADAPT Relate what you learn here to your own situation and needs. Keep your current problems and issues foremost in your mind and be thinking of ways that you can adapt the examples in this Guide. Ask yourself: How can I apply this to [... situation ...]? Make these skills into a part of your armoury, dont let them remain abstract knowledge.
ACT STRAIGHT AWAY

Echo Recount Review
DO IT SEVERAL TIMES
Repeat Rehearse Replay
Adapt . Amend . Ponder . Consider . Consult
SET GOALS To get the most out of the exercises here, be clear on what you intend to gain. Ask yourself: What do I hope to change as a result of studying this Guide? You are much more likely to follow through if you set goals now.
NETWORK If you are using this Guide in a Workshop situation, plan to introduce yourself to other participants and discover ways that you can help each other. Exchange contact details so you can follow-up later.
Introduction
This Guide is structured around action tasks. It aims to give practical advice on how to carry out Audience Evaluation using three methods: Exit Interviews Focus Group Discussions Observation Methods Users will learn some foundation concepts in the field of evaluation as applied to the interpretation of natural and cultural heritage. The material and examples in the book assume that users have experience of natural and cultural heritage interpretation, but assume no prior knowledge about evaluation methods. The three research modules can be explored in any order. However, those who are interested in interviewing techniques will find material in both the Exit Interview section and the Focus Group Section.
We include examples from both indoor and outdoor settings and also cover guided tours as well as static displays. However, the settings we have chosen are only examples and we encourage users to apply these research methods to their own situations according to their information needs. For example, guided tours could also be evaluated by short exit interviews or by an observation study as well as focus groups. Each module presents background information about the research technique and then walks through a typical project , step by step. We emphasise practical usefulness over theoretical discussion.
Field research methods that are practical, hands on and do-able
1. Interpretation
People who work in the area of interpretation, subscribe to a number of formal definitions about what they do professionally, but they have one thing in common. They want to share the things that are special about their place a museum, a park, a scenic view, an object, a gallery, a town with others. Moreover they want to share it in such a way that passes on some of their passion to the visitor. They want the visitor to experience the same goose bumps, the same thrill, the same enthusiasm that they feel, at least in part, for their special place. To do this they rely on interpretation an involvement in the story of the place.
Features Much interpretative teaching and methodology has flowed under the bridge since Tildens initial forays into interpretation, but the basics of the definition remain essentially unchanged. The principles are sometimes simplified, as with Sam Ham (1992) who proposes that interpretation should: be pleasurable or fun be relevant be organised have a theme
Definition Tilden in 1957 was the first person to formally define interpretation as: An educational activity which aims to reveal meanings and relationships through the use of original objects, by first hand experience, and by illustrative media, rather than simply to communicate factual information.
Principles He also defined six principles of interpretation: make the interpretation relevant to the visitor the core of interpretation is revelation based on information interpretation is an art (Ham (1992) refers to it as a craft) interpretation is more like provocation than instruction interpretation presents a whole rather than a part recognise the needs of different groups of people, specifically the children
Engaging the visitor
Purpose The purpose of interpretation remains the desire to share something special about a place, object or location with someone who visits it. So, assuming that both professional and wouldbe professional interpreters are plying their craft (Ham), or practising their art (Tilden), how do you know if you have been successful in getting across your message, your passion, your information? The answer lies in evaluation.
2. Evaluation
This book aims to outline the basics of evaluation. It will demonstrate the value of evaluation, when to do it, why to do it, how to do it and perhaps most importantly of all, what to do with the results of the evaluation. The what to do with it is perhaps the most empowering skill of all it could range from self congratulations for getting it right to a need for an overhaul (major or minor). Evaluation will provide valuable tools for planning for further interpretation, for resources, for personnel management, and for funding or sponsorship applications. Above all, evaluation will provide irrefutable evidence of whether your interpretation is working, or whether what you are planning will work. If it is then thats brilliant!! Be proud, tell your manager, tell us all! If it doesnt, evaluation will, most likely, indicate why not, thereby indicating, at least in part, that there is a problem. It may also suggest a way to rectify that problem. Dont despair, come up with a plan an interpretative plan based on the principles of interpretation then tell you manager you have the evidence (from the evaluation) of a problem, as well as, the beginnings of the solution. Surely a win-win situation?
Fig 1: Role of Evaluation in Interpretation Planning Define

On-going evaluation + feedback
Evaluate
Develop
Deliver Support Source: Best Practice in Park Interpretation and Education. ANZECC Working Group
www.biodiversity.environment.gov.au/protecte/anzecc/index1.htm
The Model outlines a planning process which: Describes Objectives Defines Key Performance Indicators Discovers Critical Success Factors
The objectives in an Interpretation Plan can be: Management objectives - what you want to achieve as managers including promoting the site to attract more visitors, increasing repeat visits or increasing visitor spending at the site Interpretive or Communications objectives by creating specific visitor experiences or influencing visitors perceptions including: Learning objectives - what your visitors learn as result of your interpretation Emotional objectives - what your visitors feel as result of your interpretation Behavioural objectives - what your visitors do, where they go
When to do it The Business Model recommended by the ANZECC Working Group provides a benchmark for best practice. This model recommends that Evaluation be included in two ways. It should be a separate stage which sums up the effectiveness and value of the project, and it should be incorporated in the planning process as a feedback loop at every stage. These roles are summarised in the following diagram.
2. Evaluation
All of these objectives can be evaluated and you can prepare for this by the way you define your objectives. When writing objectives, keep the SMART 1 process in mind. Planning which is SMART is more effective. SMART = Specific, Measurable, Attainable, Results-focused and Timetabled.
Evaluation will be possible if your Interpretation Objectives are SMART.
Specific : you say exactly what you

want to happen in concrete detail
Measurable : you know how you will

measure it numeric or descriptive
Attainable : feasible in scope and in

your control
Results-focused: expressed as
outputs, products or services
Timetabled: you know when you want

it to happen, including interim steps
1. See training session materials from the American Library Association's Annual Conference 2000 in Chicago, Illinois. Online at http://darkwing.uoregon.edu/~smichel/ istraining/index.html.
2. Evaluation
This section outlines the process of Evaluation and introduces core concepts of Visitor Research.
2.2 Visitor Research The assessment methods which are the focus of this workbook are visitor research methods. Visitor research aims to provide data which can give direction to interpretative planning or to allow judgements to be made about the effectiveness of displays or programs. Visitor research methods can be
Direct Interviews or questionnaires Indirect Unobtrusive observation
2.1 The process The evaluation process checks interpretation products and services regularly and systematically for: Effectiveness in achieving their stated objectives; Degree to which Key Performance Standards are being met; Level to which Key Performance Indicators have been achieved; Continuing relevance of their objectives and design features; Wider anticipated or unanticipated impacts; Value for money. Evaluation involves these steps:
Quantitative Numerical data in the form of counts, time spent, etc., which can be analysed statistically. Small amount of information from a large number of people.
Qualitative Data in the form of opinions, attitudes, perceptions, feelings or descriptions of behaviour. Large amount of information from a small number of people.
1. Review objectives. 2. Set Key Performance Standard [e.g. the

display will convey three key messages].
3. Determine a performance measure [e.g. a

visitor survey].
Evaluation is carried out at different stages of project development:

Beginning Front end Exploratory Focus groups, interviews, questionnaires Middle Formative Specific questions Observations, questionnaires End Summative Sums up Observations, questionnaires
4. Determine a Key Performance Indicator [KPI]

that relates to the measure [e.g. 80% visitors know the three key messages or 100% visitors learnt one of the key messages].
5. Gather the data. 6. Apply the data and draw conclusions.
Critical Success Factors
These are the high priority factors which underpin success. KPIs are the tools used to assess how well Critical Success Factors have been achieved. Evaluation can be time consuming, so concentrate your efforts on the factors that really matter.
In the earlier stages, exploratory research can describe visitor knowledge, needs and interests to give direction to the interpretation. Concepts and plans for interpretative ideas can be shown
2. Evaluation
to visitors and their responses noted. An open ended approach is useful here because it allows the research to uncover things that were not necessarily anticipated. Qualitative methods are very appropriate. In the middle stages, research can be used to pilot test specific aspects of the interpretation. Small scale field trials involving some observation or short interviews will be sufficient to test whether the right messages are getting across. In the final stages, visitor research can be used to gather data about the effectiveness of the interpretation in meeting its communication aims.
2.4 Drawing conclusions Once the data is collected, then evaluative judgements need to be made. Where KPIs have been defined, the measures give a simple indication of whether or not the goal has been met [e.g. more than 80% of visitors know the three key messages]. In other cases, the evaluative judgements are less clearly defined. It may be that evidence from a range of sources such as exit interviews, observations and staff comments might point to a certain conclusion. A simple process for drawing conclusions is:
2.3 Data Collection Process So, you have reviewed the objectives of the project which are to be evaluated using visitor research. You have thought about Key Performance Standards, KPIs and Critical Success Factors. Now you want to plan a data collection study using either a survey, some focus groups or observation methods. Here are some steps that all data collection projects follow.
1. Summarise the findings from all sources. 2. List some implications. 3. List appropriate responses. 4. Prioritise the responses in terms of need and
practicality.
Checklist Define your information needs
1. Write down all the things you want to know

about the topic, including both the general and the specific.
1 . List your information needs and against each

item write how you will use the data (see p11).
2. Invite colleagues to think of things they

would like to find out.
2 . Write a task list for the project. 3 . Plan your resources in terms of budget,
equipment, skills and time availability.
3. Sort through all the ideas, weed out

duplicates and group similar things together.
4. For each item on the list, write down how that

information would be applied.
4 . Decide what tasks to carry out in-house and

what tasks to outsource. Be clear about who is responsible for the overall project.
5. Prioritise the information on the list. Three

categories are useful here: must know; useful to know; nice to know. You will probably discard the nice to know items.
5 . Manage the distribution of tasks and their

completion.
6. Highlight all the items that can be answered

using survey/focus groups/observation.
6 . Apply the data to the question in hand and

look at the wider implications of it.
10
3. Exit Survey
This module presents a small scale exit interview project as a demonstration of the process of carrying out on-site interviews with visitors. The example is carried out in a museum or visitor centre with an interpretative display, but it can be adapted to a wide range of situations. We go through the process step-by-step.
For example
Information Demographic profile Main message Use Know your visitors better, and shape future programs appropriately. Check that the intended message is getting across, check for unintended messages. Verify the effectiveness of communication. Benchmark against other displays and programs Refine and modify
3.1 Information needs Firstly you need to be clear about what you want to find out and how you will use the information. It may be tempting to use this opportunity to ask all the things you have been wondering about, but the project will be much more useful, and do-able, if you keep a tight focus. Some things you might want to know might be: demographic profile of visitors their understanding of the main message their understanding of specific messages overall rating suggestions for improvement Make sure that most of the information on your list relates to the original goals and objectives for the project. Now you will see the value of writing performance indicators in ways that are measurable! Remember the SMART objectives outlined in Section 2. It is good practice to list these information needs and to note for each one how you will use the results. It helps if others in the organisation have some input, as this prepares the ground for wide-spread acceptance and use of the information in your organisation.
Specific messages Overall rating Suggestions
3.2 Plan your resources Resources include skills, equipment, time and money a shortage in any of these areas may compromise the project. You will either find the necessary skills within your organisation or you will outsource some elements of the work. To some extent, you will shape the project to fit the resources that are available. The key skills you will need for an exit interview project are: interviewing skills; clerical skills for tallying the data or doing data entry; writing skills to summarise the findings in a report. These are discussed in the following paragraphs.
11
3. Exit Survey
Interviewing is best carried out as a separate
exercise from other visitor services. The person delivering the service is not usually regarded as an independent evaluator, so guides should not conduct interviews about their own tours. Also, interviewing requires a particular set of skills. Interviews can be carried out by: staff who are a little separate from the service students on placement volunteers contracted interviewers. Contracted interviewers are ready-trained in procedures and skills, but others will need some training. This should be factored into the project. Depending on expertise, funds or equipment, the tallying or data entry can be done in three ways: pencil and paper tallying requires only time, patience and accuracy. It will give simple counts of the number who made each response and will probably be adequate for most purposes; spreadsheet tallying requires computer, software and a user who knows the basics of spreadsheet packages. It will give simple counts and allow some comparisons to be made. It also allows the data to be kept and used in flexible ways in the future (e.g. several surveys could be added together and seasonal differences noted). data entry into a statistics package requires the computer, software and a user who knows the software. This software is more specialist than a spreadsheet and unless you will be doing this regularly, it is better to outsource to a specialist agency. For a modest fee, the agency will turn your pile of questionnaires into neatly tabulated data. Like spreadsheet data, the data files can be kept and used in flexible ways and, indeed, data files are compatible with speadsheets allowing for transfer of data between software packages. Higher level statistical analysis can be carried out on the data.
The report writer needs to be able to write clearly about the purpose of the project, the steps followed and the findings. It helps if they know how to present data in charts and diagrams. They also need to have a good understanding of how the information can be applied in the organisation. The project manager is often the person who writes the report, or puts the final touches on a basic report prepared by an assistant.
Plain English uses everyday language uses shorter sentences uses active rather than passive forms avoids jargon and technical terms uses headings, diagrams, lists and easy to read fonts
3.3 Survey schedule To give an accurate report of visitor views, interviews need to be planned systematically. If you want the survey to include all kinds of visitors, you will need to conduct interviews at different times of the day, across all days of the week, and perhaps taking care to include school holidays as well as school terms, or to cover both winter and summer. To make the project more manageable, it is wise to restrict the interviewing to limited time periods (e.g. summer holiday only). When you do this, take care to report the findings as representing the views of summer holiday visitors and not necessarily representing the views of all visitors. You will probably need to balance convenience with coverage. But you should be careful not to make large claims about data based on a limited sample of visitors.
12
3. Exit Survey
3.4 Questionnaire design First, you will need to decide whether to carry out face to face interviews or whether to ask visitors to fill out a self-completion survey. Some pros and cons are listed here.
Face to face More expensive More reliable sample of visitors Better quality information - more specific and more detail Self-completion Cheaper Stronger volunteer effect in the sample Simple tick box information Structure of interview
The interview is a social exchange and must conform to social rules. It should have a clear beginning, middle and conclusion.
Introduce self and purpose of survey Open with simple questions Main flow of questions Open ended question Demographic details Thank and close
If you can afford it, use interviewers to carry out interviews. Self-completion is fine if you need to collect only a small amount of simple information. See examples of self-completion questionnaires in the Appendix. Question types include: categorical (e.g. time of arrival, things seen and done. demographics, etc.) yes/no rating scales (agree/disagree or rating on excellence, etc.) open-ended (write in the answer).
Start the interview with a couple of really simple questions that everyone can answer, such as time of arrival and who they are here with. Structure the main body of the interview then ask demographic questions at the end. If you really need to ask income level, make it the very last question. It is good to include at least one open ended question in all surveys because it gives visitors a chance to express themselves in their own words.
GOLDEN RULES 1. Try to plan the questionnaire so that as many questions as possible can be
answered by ticking a box or circling a number. This reduces the workload on the visitor and the interviewer and makes it much simpler (and cheaper) to collate and analyse.
2. Try to frame questions in specific behavioural terms. Ask people to report actual
behaviours rather than averages, tendencies or generalisations.
13
3. Exit Survey
Questionnaire design process
Use Refine Test Draft

Once you have drafted your questionnaire, test it with a couple of colleagues by asking them to play the role of the visitor. If possible, try to pilot it with some visitors. You might regard the first day of interviewing as a pilot. In this case, you would print only enough questionnaires for the first day. At the end of the day, review the questionnaire with your interviewers, make the final changes then print. Resist the urge to keep making adjustments during the course of fieldwork. See Appendix for example questionnaires.
about the kind of information you hope to gather, e.g. practical examples of what people actually do; or clear descriptions of what they found out. The project will go more smoothly if you show an active interest by dropping by to see how things are going. Staying close to the project like this can also help you to nip potential problems in the bud. Collecting 200 questionnaires with a key error is very disheartening!
3.6 Data summary and analysis Three methods of handling data were outlined in Section 3.2: tallying, spreadsheet and computer program. Whichever method is used, the questionnaires will need to be edited and coded.
Editing involves looking through each
3.5 Interviewer Briefing Your interviewers may be experienced market research interviewers, staff, students or volunteers. When using inexperienced interviewers, you will need to run a training session where you explain some of the basic skills and rules of effective interviewing. See the Appendix for a set of Briefing Notes suitable for teaching some of these basic points. All interviewers will need a Briefing Session which covers the following topics: purpose and scope of the survey interviewing schedule and quotas a run-through of the questionnaire, question by question orientation to the site (if necessary) logistics such as contacts, security and other matters In the Briefing Session, you can be quite clear
questionnaire and checking that all the questions have been answered properly. If the surveys are to be entered on computer then the editor should make sure that all answers are very visible so that data entry is smooth and swift.
Tallies are simple counts of the number of
people who said different things. You might choose to divide the questionnaires into categories [e.g. male and female] and to tally within those categories before making an overall total. Pre-coded questions can be tallied directly from the questionnaire, but open-ended questions will need to be coded first.
Coding open-ended questions This involves
dividing all the comments into categories and allocating a code number to each category. The codes numbers are written on the questionnaires and then they are tallied like the other questions.
14
3. Exit Survey
Here is the process in more detail.

1. Draw up a coding frame. The coding
frame is a list of numbers with comments beside them. All possible comments given by respondents should be fitted into one of the numbered categories. This list should be about 10-15 items long, and should include a dont know category and an other comment category. Every open ended question will have its own coding frame. To draw up a coding frame for a question take a sample of about 20-30 questionnaires and write down all of the comments made. Where two people make the same comment, record this with a tally mark. Then look through the long list of comments and start to group them according to meaningful ideas. Look past the surface language to the underlying idea. Try to reduce the list to 10-15 items. Give a number to each item.
2. Use the coding frame to code the question. Now you go through all the
age sex with children or not first visit or not geographic origin See the Appendix for an example of a table showing data cross analysed using a banner. Once you have the data in the form of tables, you can write your report. Summarising some of the data in graphs, charts or simple tables is the last step in data analysis.
3.7 Report Plan for your report to make sense to: co-workers senior management somebody who picks it up in five years time Your report should include the following sections: Introduction: what is the thing being evaluated, how will the information be used. Research Objectives: specific questions the research is trying to answer. Method: enough detail to allow the study to be repeated. Include the questionnaire in an Appendix. Executive Summary: outline of purpose and key findings. Findings: detailed findings. Implications (optional): suggestions for applying the results of the research. If you have a project team or a work group, it is very good to get them involved before the report is written. Circulate the data to the team, then
questionnaires writing the code numbers against the question. Some questionnaires may be given several numbers for a particular question because the visitor said a number of things. Every questionnaire should have a code people who said nothing would have the code for dont know or you may choose to add a code for no comment. Once the data is coded, you can either tally it by hand, enter it in a spreadsheet or send it to a data processing house. In the first two cases, you can carry out each tallying task individually. In the case of data processing, you will be asked to specify how you would like the tabulated data to look. The simplest and most powerful option is to ask to have all the questions cross analysed by a single banner. Some useful things to put in the banner:
15
3. Exit Survey
call a meeting to discuss what the results mean and how they can be applied. This is very effective in getting staff to support the results of the evaluation and also contributes insights that can be included in the report.
3.8 Circulate the results Each research project adds to the knowledge of the organisation as well as adding to the knowledge of the project team. This new knowledge can be spread through the organisation in a number of ways: present the findings at existing staff forums such as staff meetings, planning meetings, professional development meetings, etc; circulate a summary of the key findings to senior staff; prepare a summary or abstract suitable for Board Reports or Annual Reports; put the report on the organisation intranet; represent the organisation in professional forums such as conferences by talking about the project, including the evaluation.
16
4. Focus Group Discussions
4. Focus Group Discussions Qualitative Research includes focus group discussions and indepth interviews. This Guide describes a do it yourself focus group project, which can be adapted to a project which uses indepth interviews. A focus group discussion is a special type of meeting process. It is not the same as a workshop, a staff meeting, a consultative meeting, a public meeting, a forum, seminar or or other kinds of gatherings and get-togethers. Some key features of a focus group discussion: The group comprises about eight people with similar characteristics. A facilitator or moderator coordinates the meeting which is run as a free flowing discussion. The facilitator/moderator uses active listening techniques and does not reveal their own views. The results go beyond what people said and take account of body language and the context. Results are often inferred from a range of contributing elements. While most people are capable of convening a meeting and conducting it, focus group discussions draw on special skills. This workshop covers some of these skills at an introductory level, however expertise is only developed via practice under expert guidance.
6. Collate, analyse and interpret the data. 7. Write report and apply the findings.
The following pages describe how to carry out each of these steps.
4.1 Define information needs Follow the process outlined on p11 to define your information needs for the project.
4.2 Specify who to talk to Focus groups work better if you have a fairly homogeneous group, that is partly why they are called focus groups because they focus on particular types of people and on particular topics. Generally it is better to conduct several groups amongst different kinds of people than to mix a wide range of people together. For example, if you want to find out what visitors think of a new boardwalk that you have installed, you will find that a focus group of parents of young children will work better than if people of all ages are mixed together. People talk more freely when they are in a group where they have a lot in common. Once you have decided who to talk to, draw up a list of specifications for each group. Here is an example for an evaluation of a new boardwalk where you are planning to add more interpretive material.
Group 1 Parents of children under 10 years
Project stages
1. Define what information you need and say

how you will use it.
2. Specify who you will talk to. 3. Organise recruitment, schedule and venue for
groups.
who have been on the boardwalk.

Group 2 People under 25 without children who
have been on the boardwalk.

Group 3 People aged 45+ who have no children
4. Write discussion guide and prepare any

materials you want to show.
5. Conduct the group discussions.
under 18 and have been on the boardwalk.
17
You may also consider conducting a group or indepth interviews with teachers who have taken school groups and, perhaps, indepth interviews with people with disabilities. It is rarely practical to conduct focus groups amongst tourists, and, if you want their feedback, you may decide to conduct some onsite interviews with tourists. There are never enough resources to talk to everyone you are interested in, so some tough decisions need to be made about who to leave out.
relevance of your findings because the respondents will not be representative of your likely visitor profile. Focus groups can be conducted onsite if there is a suitable meeting room [education room, training room, etc.] however this often involves convoluted catering and security procedures. When the meeting is onsite, respondents are cued by the surroundings and less reliant on their personal memories of the experience. Offsite meetings usually incur a fee for hire of the venue but the advantage is that there is less energy consumed in the logistics of meet and greet and catering, etc. Focus groups work well with 6-10 people. The more people you have in the room, the less each one gets to say.
4.3 Recruit and schedule Ways to find the participants use a specialist recruiting agency. Most large cities have agencies that specialise in doing this. They will charge about $45-$55 per person. You will need to write detailed specifications for each group. have a recruiter onsite to explain the project and invite people to come to one of the research sessions. Using a professional recruiter, you can ask for 8 people and be fairly certain that they will turn up at the scheduled time. Our experience in recruiting directly onsite is that a high proportion say they will come but then dont turn up. So you may need to confirm 12 attendees in order to be sure that 8 will turn up. Respondents are paid a disbursement to attend. The current 1 rate is around $40-$50. This payment is an important element in ensuring you get a good sample of people. Without the payment, you tend to get only those people who have plenty of spare time or who are especially interested in the subject. This will limit the
1. January 2001. If you pay more than $50 there are GST implications.
4.4 Discussion guide and materials To prepare for the discussion, you will need to write a Discussion Guide. This will ensure that you cover all the topics and that each of the groups is taken through a similar process. The first items on any Discussion Guide are the welcome, housekeeping and ground rules for the discussion [see the example in the Appendix]. Plan your Discussion Guide as a series of topics [or headings] starting from broad areas and moving through to specific things. Against each topic, write down the things you want to find out. Dont write specific questions. A focus group discussion is not a question and answer session, it is a freely flowing discussion. If you are a beginner, you might want to script the phrasing of some of your introductions and questions, but be prepared to ad lib rather than read from a page. Think about different ways of getting the information you are after. Sometimes it is useful
18
to give respondents some activities to do as individuals or as a group. For example, you might write the names of places on cards and ask people to sort them according to some quality you are interested in, e.g. environmental responsibility. A group activity like this is very illuminating as people share their knowledge and views with each other when they complete the task as a group. Your kit for the day of the discussions will include: Discussion Guide Any visual material you want to show Materials for tasks you want to use Notebook for yourself Paper and pencils for participants Tape recorder and tapes Name tags Money for respondents [cash] Forms for participants to sign for cash
then moving on to another topic. It is vital that the moderator accept every comment that is offered. Respondents should not feel that some views are more acceptable than others. All views are welcome. Take care to affirm and acknowledge what is said without showing approval or disapproval of the content. Eye contact and body posture are important in creating a relaxed and open environment for discussion. If the moderator is tense, participants will find it harder to relax and open up. That said, a balance needs to be achieved so that individuals who are very socially dominant are toned down sufficiently to leave space for others to contribute.
Reflective listening This is a great way of
indicating that you have heard what someone has said without showing approval or disapproval. For example, if someone has described at some length their difficulties in getting around with a stroller. You might nod and say, It sounds like you had problems with that, then move the discussion along. Uhhuh is a good confirming line.
Open ended questions Questions which are
4.5 Conduct the group discussions The researcher needs to establish rapport with respondents and encourage the group to talk amongst themselves. The discussion will start with a brief introduction and some housekeeping information, then each respondent can be asked to say a little about themself. This ensures that each person speaks and gives everyone an idea of who the others are. The first topic that is opened will act as a warm up in which the group establishes a momentum of exchange, it also provides opportunities to learn general things about the activities and values of the participants. The moderator should be clear about introducing a topic, encouraging exchange on it, open ended are valuable in opening discussion. An open ended question is one which cant be answered with yes or no, but requires some comment. For example, What prompted you to go to the Park? invites an explanation, whereas Did you go because of the open day? invites a yes-no response.
Leading questions are those which suggest
what kind of response is considered desirable. Try to use phrasing which allows all kinds of answers to be acceptable.
Simple facts Asking people to recount simple
facts is a good way of encouraging participation and also gives an opening for further questions.
19
For example, asking What kinds of things do you do when you are at the Park? gives a window onto their preferred activities and allows exploration of why they do those things, why they dont do other things that are offered and what things they would like to do, if they were available.
Probing High quality information is specific
liking a picnic area and some walking trails You said that you liked the picnic area, what did you like about it? You like the picnic area because.....? [with an enquiring look] The picnic area... can you say a bit more about that? What is the picnic area like? Is the picnic area popular? Why do you think that is?
Confirming questions Sometimes it is helpful
rather than general and is well rounded. To get high quality information, you will need to probe. Probe for: more material when only a little information has been provided. For example, if you want to get an idea how much people know about the subject of the interpretation you are evaluating, you will need to probe to find out ALL that they know. Keep enquiring gently until you are clear about the boundaries of their knowledge. Listen to them telling each other things, or asking each other. Be careful NOT to answer content questions that they may direct at you. You may want to confirm whether they know specific facts and this is a good way to check. specific examples when people are waffling or speaking generalities. For instance, if someone says that they know all the National Parks in the area, you might ask which one they visited last and when that was. Some specific information goes a long way towards helping you understand how well they know the Parks. If they havent been to some for 10 years you will know that their knowledge is not up to date. to clarify when you are unsure what they mean by something. When clarifying, it is important not to rephrase what they said. You might ask, Im not quite clear what you mean, could you say a bit more about that? Here are some examples of probing for more more material and for specific information when someone has made a general comment about
to confirm what you hear or observe the group to be saying. Dont do this more than 2-3 times as it throws people into an analytical frame of mind which inhibits other forms of expression. Just confirm those things that you are really not sure of. Heres an example It sounds like you are unsure about what happens there?
Group vs individual Try to encourage people
to discuss, share, and exchange with each other and not just with you. This allows you to observe and note views and attitudes. Some ways to encourage this, Who else had experiences like this? Have others experienced different things? Is that a good idea? What else do you know about this? I see some of you nodding, do all of you agree with this?
Some things to watch for On behalf of others Listen for when people
talk on behalf of others. If locals start telling you what tourists think or do, you will need to ask them to tell you about themselves and what they think and do.
20
People often project their own views onto others. For example, it is common for them to talk about what children need [usually hands on learning with low text loads]. When you think that people may be expressing their own needs by talking about others, you need to find a way to confirm that they, too, might like these characteristics.
Social dominance You may need to damp
lawyer may be very persuasive about the reasons for going to a museum, but if he has not been to one in the past 5 years, we have to discount his views as second hand. On the other hand, a group of teenage boys who respond to a suggestion with smiles, eye contact and shifts in body posture are indicating genuine interest even if their only verbal expression is Yeah. Dont rely on transcripts alone for interpreting qualitative material.
down individuals who are more talkative, here are some suggestions: avoid eye contact with them select others for comment invite alternative views if necessary, say Wed like to hear the views of some of the others.
Researcher as observer If you succeed in
Data recording Tape recordings or video recordings are the best ways of capturing the exchange that takes place in a focus group discussion. If the researcher is new to the process, the tapes should be transcribed verbatim if possible. If recording is not possible then a note taker can be used. Preferably, the note taker should be in the room with the respondents and will need some training in recording in an unbiased way. One way to minimise bias is for the note taker to simply write down everything that is said, using the exact language of the respondents. They must NOT filter out material that they regard to be unimportant or less relevant. A notetaker can be helpful when you are thinking about the meaning and implications of the findings someone to verify perceptions and bounce ideas off.
getting the group members talking amongst themselves, it gives you a chance to observe and process the information so that new ideas can occur to you in the course of the discussion. These can be verified on the spot.
4.6 Analyse and interpret the data

What IS the data?
Analysis of qualitative data requires that we pay attention to what is said and how it is said, and also to what is avoided or left unsaid. In the effort to understand participant meanings, we pay attention to all strands of communication, including words spoken, the rhythm of discussion and exchange, types of discourse (e.g. debate or persuasion) as well as body language and facial expression. We place varying degrees of importance on the material presented. For example, an articulate
Analysis Analysis of qualitative material can use the following processes: counting and comparing how many people used the word discover, is that higher or lower than when we studied XYZ?; categorising this is a wayfinding issue, but that is a comprehension issue; noting patterns and themes e.g. older people seem to emphasise this over that,
21
someone in every group mentioned the importance of ABC; seeing plausibility it makes sense that parents would want to know that children could reach it; making metaphors they are all at sea and cant quite grasp the central message; assuming particulars into general they gave four different examples of stories they want to know about so there appears to be interest in narrative forms of presentation; noting relationships the people who have been to XYZ seem to have higher expectations; finding coherence what evidence hangs together and which elements appear to be contradictory? In this process we develop ideas and explanations and check the data for support. Some useful questions that guide our collation and analysis of qualitative data are: What are the broad-brush threads so far? What is puzzling, strange or unexpected in the data? What expectations are confirmed so far? This is interesting, but what are the implications? How confident am I that this is what it means? What else could it mean?
museology.
Two report styles
A fully detailed, narrative report, supported by examples and quotations from the fieldwork will provide strong evidence for the conclusions drawn from the study. A summary action report is an effective way of communicating the main findings.
Applying the data
The following grid provides a convenient framework for applying research findings to project development. The main findings from the research are listed under the four headings as shown in the following example. When used as a cooperative exercise, perhaps using a whiteboard, this task can help the project team synthesise and apply the research findings.
Audibility is a key issue.
Confirma tions
People are happy for the tours to teach values as well as facts.
di scoveries
Unex pected
4.7 Reporting Rather than just reporting everything people said, the written report should address issues and questions of specific interest to the project. It should report the findings of the study, and can comment on connections between the data and relevant theories from disciplines such as psychology, education, communication and
Be careful not to be too strident about valuebased messages.
Cautions
Some tours could include quite a lot of material that is not directly related to plants.
Opportuni ties
22
5. Observation
5. Observational Evaluation This module outlines the concept and methodology of observational evaluation, which uses unobtrusive methods to gain information about visitors and interpretation. Observation includes the following activities: tracking counting and timing watching, listening and noting To be effective, observational data needs to be: collected systematically, so it is not just a record of anecdotal information or opinions unobtrusive, so it records natural behaviour Observation studies are generally small scale, i.e. you may observe 10-20 encounters rather than 100s.
shows what actually happens in terms of movement, emotions, interest levels distinguishes between Streakers, Strollers and Students, i.e. people who spend varying amounts of time at the site or exhibit achievable in short time frames able to produce reliable, quality data for analysis it complements other forms of evaluation, notably visitor questionnaires.
When do you do it?
when the site is operational to check effectiveness and plan remedies during development to check that ideas work by making mock ups and drafts Observation can show you: how visitors use the site/exhibition how the elements attract visitors attention how the elements hold visitors attention how the traffic flow pattern works group interaction
Why evaluate by observation?
It is: quick and relatively simple cost effective useful for getting a basic understanding of physical and behavioural characteristics of visitors repeated over time, it can indicate trends
Before you begin
Some things to keep in mind decide what you want to find out
What we do, or what we think we do? When we evaluated the John Curtin exhibition at Parliament House we asked people how long they stayed in the exhibition. We also timed how long people spent there. On average, visitors estimates of how long they spent were about twice the times we recorded for actual visits. Sometimes, visitor perceptions are quite wrong. Observation gives more accurate information about actual behaviour than visitors estimates of what they have done.
23
5. Observation
do a pilot study and dont try to discover too much find a balance between knowing a little about a lot, and a lot about a little dont just copy another study modify it to suit your situation the ethical requirements of your organisation may limit the kinds of observation you do (e.g. use of video material). Some places require the display of a notice advising that observation is taking place
happen how physical aspects of a site affect the way visitors move or use an area (e.g. lighting, width of walkways, slope etc) what visitors stop and look at what they show to each other and relationships between particular groups e.g. adult showing child and vice versa the level of use of a particular facility (e.g. touch screen), by whom and for how long (e.g. < 1 minute, 1-5 minutes, > 5 minutes).
Observers should
be discrete act invisible keep a distance avoid contact with visitors, but if approached they should answer basic functional questions (e.g. where are the toilets?) then get back to their task not tell visitors what they are doing if asked, they should offer a vague response like Im just monitoring how things are working. carry out definite tasks of short duration (e.g. 60 minutes) as they will need to maintain high levels of attentiveness while they are observing accuracy will be greater for three short tasks than for a single longer one
How to do tracking
Tracking is most often carried out by having trained observers follow visitors discreetly plotting their path on a map (see exercise 1 in section 5.6). Galleries and exhibitions often have security videos, and the film may be used to track visitor movement (ethical considerations may apply).
5.1 Tracking where people go Tracking is used to track the movement of people around a site or exhibition, to see where people go and what they are interested in. It is time consuming but valuable, and can show you: how easily visitors orientate themselves and get to where they want to go how visitors choose what they will investigate based on curiosity visitor flow diagrams of how they move through a site what actually happens at a site or in a gallery, as opposed to perceptions of what should
5.2 Counting and Timing Counting is used to determine the number of people at a site or exhibition and timing is used to see how long they spend there. Counting and timing will show you a variety of things, including: the number of visitors in a particular location or at a specific site at particular times how many people/group size/age/gender
24
5. Observation
what is the longest/shortest/average visit length who does what and with whom (e.g. adult/ children) when (time and day) are the busy or quiet times whether particular groups focus on different things the attraction power and holding power of a site or exhibit (see formulas in section 5.4)
level and quality of interaction between a guide and visitors attracting power of a site or exhibit holding power of a site or exhibit observers may listen for comments which provide clues to how visitors think or feel level and nature of exchange within groups (e.g. an interaction where a grandparent explains a wind up telephone to a child and, further on in the display, the child explains the working of a digital mobile phone to the grandparent) the content of questions visitors ask Observation tasks should be structured to achieve one or two specific watching goals rather than multiple goals.
How to do Counting and Timing
Resources you will need: a stop watch or a watch with a second hand clip board and pen a form that allows you to record the relevant data (see exercise 2 in section 5.6) trained observers who understand the importance of being unobtrusive
How do you do Watching?
As with all forms of (observational) evaluation, you need to be prepared. 5.3 Watching who does what and how The purpose of Watching is to record the various behaviour patterns of visitors. Observers may watch for: who stops where who points to what and shows what to whom interaction with the displays and with other group members visible levels of comfort and interest reactions to noise, to the space attentiveness, especially with tour groups effectiveness of interpretation for people with disabilities, people with small children, people who are non-English speaking etc design and prepare a form on which you will record your data (see Appendix for sample form). use a check-off form with a section for comments decide what visitor behaviour you want to watch for dont just wander and try to record everything select a sample of typical subjects to observe (age, gender, group composition) plan to watch 10 people of a particular type (e.g. families with primary-age children), review the results then add more observations until little new material is emerging (law of diminishing return) this way you can be confident that you have uncovered the main findings and confirmed them observe at various times of the day and week,
25
5. Observation
school holidays and non-school holidays, with and without special events, different seasons.
5.5 Reporting the data For maximum usefulness, write a brief report, covering:
5.4 Attraction Power + Holding Power Two useful indices of exhibit or site performance are derived from simple observations of visitors and from measurements of time spent. The Attraction Power of an exhibit or site can be calculated. The number of people who stop at an exhibit or site, divided by the total number that pass it (including those who stop), expressed as a percentage, will provide a relatively reliable indication of the attractiveness or attention grabbing power. This is a simple measure of how successfully an exhibit catches peoples attention. e.g suppose that 60 people walk past an exhibit and 15 of them stop to look at it. The Attraction Power is 15/60 x 100 = 25% The Holding Power of a site or exhibit can also be calculated. The average time that people spend at an exhibit is divided by the actual time that is required to read or review it, the result is expressed as a percentage. This gives a measure of how well an exhibit holds peoples interest once they have stopped at it. e.g. suppose you need 10 minutes to read all the text on a panel and to operate a small model attached to it. If the average time people spend at the panel is 2 minutes, its Holding Power is 2/10 x 100 = 20% The higher the percentage in both Attraction and Holding Power, the better the site or exhibit.
1. Aim of project 2. Procedure used (methodology) 3. Main findings, for example

compile responses e.g. plot all visitor movement patterns calculate results e.g. length of time that people stay for, Attraction and Holding Power identify patterns of behaviour, of movement, of comments collate observations who does what, with whom. Collate both positive observations (e.g. popularity of particular sites or exhibits) and negative observations (e.g. some displays not seen by anyone) summarise new information as well as information which confirms things you already knew
4. Implications and recommendations for action

Disseminate the results to interviewers, management, front of house, guides, planners and other relevant stakeholders. Use the results to improve things plan changes, modify interpretation, congratulate those things/people that work well for visitors, apply for funding or sponsorship.
26
5. Observation
5.6 Observation Exercises
Exercise 1: Tracking, Watching and Listening
Part 2 After the observation
note major pathways through the site, and note alternatives that were also used collate observations of behaviour, note major patterns (e.g. people which children tended to let the children lead) collate comments heard interpret these!
Resources you will need: a gallery or site with specific boundaries map(s) of the location with the positions of displays, exhibits and signs marked tracking, watching and listening form (see Appendix) Purpose of Exercise: to trial observation evaluation utilising tracking, watching and listening to record visitor flows through a site or gallery to give a brief description of common behaviours
Part 1 Onsite, observers should:
select a visitor and follow them from a distance record start time describe them on the form in space provided next to Visitor A, Visitor B etc draw their path on the map as they go mark with numbers 1, 2, 3, 4, 5 where they stop mark the number on the map so that it corresponds with the same number column on the form on the form record how long they stop at each stop record type of behaviour (if possible) record comments overheard (if possible) record end time
27
5. Observation
Exercise 2: Counting and Timing
females visited compile a demographic profile browse through the tally sheets and see what patterns emerge e.g. do people with children stay longer calculate what percentage of people stayed for various timeframes (e.g. < 3 minutes, 4-10 minutes, > 10 minutes) calculate the average time spent summarise common behaviours observed calculate Attraction Power using the formula provided in section 5.4, p26. calculate Holding Power using the formula provided in section 5.4, p26
This is what you need a gallery or site with specific boundaries map(s) of the location with the positions of displays, exhibits and signs marked counting and timing observation form (see appendix) Purposes of Exercise: to measure length of time visitors spend in the gallery or at the site to give a demographic profile of visitors to give a brief description of common visitor behaviours to count the number of people who do things/do not do things (this is the basis for the Attraction Power and Holding Power calculations)
Part 1 Prior to survey:
plan observation tasks, design the recording form plan observation positions and times distribute stop watches or use watches with second hands
Part 2 Onsite, observers position themselves
and record the information on the form following the accompanying directions (see Counting + Timing Form in Appendix).
Part 3 After the survey
calculate how many people visited in the time, you might arrive at a figure for persons per hour on weekdays using entry and exit time, calculate how long people stayed do a gender breakdown - how many males/
28
References
References ANZECC Working Group. Best Practice in Park Interpretation and Education.
www.biodiversity.environment.gov.au/protecte/anzecc/ index1.htm
Rennie, L. J. and McClafferty T., (1996) Handbook for formative evaluation of interactive exhibits. Questacon, Canberra. Tilden, F., (1957), Interpreting Our Heritage, University of North Carolina Press. University of Oregon, Writing Measurable Objectives.
http://darkwing.uoregon.edu/~smichel/istraining/ index.html
Askin, M, (1999), A Revised Edition - Troy, Keskin Color Kartpostalcilik Ltd., Sti. Matbaasi, Turkey. Borun, M., Korn, R. and Adams, R., (1999), Introduction To Museum Evaluation , American Association of Museums, Washington Carter, J., (1997), A Sense of Place An Interpretive Planning Handbook, Tourism and Environment Initiative, Scotland and online at
www.scotinterpnet.org.uk
Useful websites
Diamond, Judy, (1999), Practical Evaluation Guide : Tools for Museums and Other Informal Educational Settings , Walnut Creek, California: AltaMira Press Hall, C.M. and McArthur, S., (1996), Heritage management in Australia and New Zealand The Human Dimension , Oxford University Press, Australia. Ham, S. H., (1992), Environmental Interpretation A Practical Guide for People with Big Ideas and Small Budgets , North American Press, Colorado Kruger, R.A., (1994), Focus Groups: A Practical Guide for Applied Research, Second Edition. California: Sage Publications, Inc. Leask, A. and Yeoman, I., (1999), Heritage Visitor Attractions An Operations Management Perspective, Cassell, London.
Evaluation and Visitor Research Special Interest Group of Museums Australia (EVR SIG).
www.amol.org.au/evrsig/
Australasian Evaluation Society

www.aes.asn.au/
29
Appendix
Examples and master sheets
1. Self completion questionnaire sample (p.13) 2. Interview questionnaire sample (p.14) 3. Briefing notes for interviewers (p.14) 4. Data cross analysis using a banner (p. 15) 5. Discussion guide for focus group (p.18) 6. Tracking and watching recording form (p.27) 7. Counting and timing recording form (p. 28)
30
1. Self completion questionnaire sample
(Referred to on p.13) This is an example questionnaire in selfcompletion format. It has been adapted from a questionnaire used to evaluate a number of different kinds of events at a cultural institution. You could adapt it for your situation by omitting questions, changing items or modifying questions.
Self completion questionnaire
6. Date: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Event:. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Before arriving today, what did you expect this event to be like? .................................................................................... .................................................................................... .................................................................................... ....................................................................................
1.
What was your main reason for visiting the [ place ] today? Circle ONE answer only To see the [place] in general To see/go on/do [name particular event] To spend time with family or friends To go to the caf or shop To [name particular thing] To [name particular thing] Other (specify) 1 2 3 4 5 6 7 7.
Were these expectations met? Yes, mostly . . . . . . . . . . . .1 Yes, partially . . . . . . . . . . .2 No . . . . . . . . . . . . . . . . . . .3 If no how was it different? .................................................................................... ....................................................................................
2.
Including this visit, how many times have you visited the [ place ] in the last 2 years? ....................................................................................
.................................................................................... .................................................................................... ....................................................................................
3.
Have you been to other events at [ place ] in the past? Yes . . . . . . . . . . . . . . . . . . 1 No . . . . . . . . . . . . . . . . . . . 2 8. How would you rate this event overall?
Low
<< ============= >>
High
4.
What was your main reason for coming to this event today? .................................................................................... .................................................................................... .................................................................................... 9.
How would you rate each of the following aspects of the event? (Circle rating number)
Low << ======= >> High
5.
Where have you heard about this event? Circle as many as apply Information at the [ place ] Talking to someone Newspaper or magazine Radio Brochure or flyer The internet Cant remember where Other (specify)................................................. 1 2 3 4 5 6 7 8
Organisation of the event Amount of information provided Relevance of information given Interest generated in the topic Quality of the presentation Suitability of the venue (room/space)
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 4 4 4 4 4
5 5 5 5 5 5
Self completion questionnaire
10. Please suggest ways to improve similar events here at [ place ] .................................................................................... .................................................................................... .................................................................................... .................................................................................... ....................................................................................
Finally, there are some questions about you . . . 15. Were you born in Australia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Overseas in English-speaking country . . . . . . . . . 2 Overseas in non-English speaking country. . . . . . 3 16. Which of these age groups do you come into? Under 18. . . . . . . . . . .1 18-25 . . . . . . . . . . . . . 2 26-35 . . . . . . . . . . . . . 3 35-55 . . . . . . . . . . . . . 4 Over 55. . . . . . . . . . . . 5 17. Gender Male. . . . . . . . . . . . . . .1 Female . . . . . . . . . . . .2 18. What is your main occupation? Skilled work/Trade/Unskilled. . . . . . . . . . . . . . . . 1 Clerical/Administration/Sales . . . . . . . . . . . . . . . 2 Professional . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Looking for work . . . . . . . . . . . . . . . . . . . . . . . . . 4 Home duties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Student . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 Retired. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Other . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 19. Where do you live? Local area. . . . . . . . . . . . . 1 Other Australia . . . . . . . . . 2 Overseas. . . . . . . . . . . . . .3 Postcode . . . . . . . . . . . . Postcode . . . . . . . . . . . Country . . . . . . . . . . . . .
11. Would you recommend an event like this to other people? Yes . . . . . . . . . . . . . . . 1 No . . . . . . . . . . . . . . . . 2 .................................................................................... .................................................................................... 12. Do you have any suggestions for subjects for talks, tours or events at [ place ]? .................................................................................... .................................................................................... .................................................................................... ....................................................................................
13. Who did you come to the event with? By myself . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 Other family adult . . . . . . . . . . . . . . . . . . . . . 2 A child/children . . . . . . . . . . . . . . . . . . . . . . .3 Non-family adult . . . . . . . . . . . . . . . . . . . . . 4 Part of a tour group . . . . . . . . . . . . . . . . . . . . 5
20. Have you attended public events at any of the following places in the last three months? ( Circle all that apply) List some local attractions. . . . . . . . . . . . . . . . . . 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
14. Are you a member of the Friends of [ place ]? Yes . . . . . . . . . . . . . . . 1 No . . . . . . . . . . . . . . . . 2
23.
Have you used any of the following facilities at the [ place ] today? How would you rate the ones you have used?
Used Not used Intend to use Very Poor Poor Average Good Very good
Cafe Shop Public toilets Public telephones Carpark Visitor Information Desk
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
1 1 1 1 1 1
2 2 2 2 2 2
3 3 3 3 3 3
4 4 4 4 4 4
5 5 5 5 5 5
2. Interview questionnaire sample
(Referred to on p.14) This is an example of a questionnaire that would be used by an interviewer. It has been adapted from a questionnaire used to evaluate an exhibition. You could adapt it for your situation by omitting questions, changing items or modifying questions.
Onsite questionnaire
Hello, my name is and I am doing a survey of visitors to the Botanic Gardens. May I ask you a few questions? [It will take about 10 minutes.] 1. Is this your first visit to the Botanic Gardens? Yes . . . . . . . . . . . . . . . . . . 1 No . . . . . . . . . . . . . . . . . . . 2
2.
What was your main reason for visiting the Gardens today? Dont read out. Circle ONE number only. To see the Gardens in general . . . . . . . . . . . . . . . . . . . . .1 To see a particular part of the Gardens . . . . . . . . . . . . . . 2 To visit the shop/cafe . . . . . . . . . . . . . . . . . . . . . . . . . . . .3 Just passing through . . . . . . . . . . . . . . . . . . . . . . . . . . . .4 Other (specify) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .5 ..............................................
3.
Did you come by yourself or with other people?
By self. . . . . . . . . . . . . . . .1 With others . . . . . . . . . . . . 2
skip to 6
4.
Including yourself how many people are in your group today? Record number . . . . . . . . . . . . . . . . . . . . .
5.
Which of the following parts of the Garden have/will you Thinking now about [particular part of the Gardens]
6.
Which of the following statements applies to you? Circle one only I have looked closely at most things . . . . . . . . . . . . . . . . . . . . . . . . . . .1 I have looked at quite a lot, but skipped past some things . . . . . . . . . .2 I had a general look but didnt look closely at many things . . . . . . . . . .3
7.
What are the main things this part of the Gardens is trying to show? Probe for content ............................................................................................................................................................................................. ............................................................................................................................................................................................. ............................................................................................................................................................................................. .............................................................................................................................................................................................
8.
And on a scale of 1 to 10, where 1 is terrible and 10 is excellent, what score would you give it for
Score Comments
Layout Amount of information Ability to hold your interest Beauty
Onsite questionnaire
9.
How would you rate this part of the Gardens overall? Would you say it was
Very good . . . . . . . . . . . . .1 Good. . . . . . . . . . . . . . . . . 2 Poor . . . . . . . . . . . . . . . . . 3 Very poor . . . . . . . . . . . . .4 DK. . . . . . . . . . . . . . . . . . . 5
10.
Agree strongly
Agree mildly
Disagree mildly
Disagree strongly
Dont know
The display added a lot to what I know about this topic The display presented information in an interesting way
4 4 4
3 3 3 3 3
2 2 2 2 2
1 1 1 1 1
0 0 0 0 0
The display ...
4 4
11. Was there anything that you LIKED in particular about this garden? Probe ............................................................................................................................................................................................. ............................................................................................................................................................................................. ............................................................................................................................................................................................. ............................................................................................................................................................................................. 12. And was there anything that you DISLIKED in particular? Probe ............................................................................................................................................................................................. ............................................................................................................................................................................................. ............................................................................................................................................................................................. .............................................................................................................................................................................................
Now, I have a couple of questions about you 13. What is the postcode of your home address? Postcode . . . . . . . . . . . . . . . . . . . . Country . . . . . . . . . . . . . . . . . . . . 14. What year were you born in? Thank you for your help 15. Record gender Male . . . . . . . . . . . . . . . . . . . . . . . 1 Female . . . . . . . . . . . . . . . . . . . . .2 Time of interview: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Date: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .............................................. ............................
Name of interviewer
3. Briefing notes for interviewers
(Referred to on p.14) All inexperienced interviewers should be given a training session based on these notes.
Practical Guide to Evaluating Natural and Cultural Heritage
Interviewer Notes 1
What is Market Research?
Market research is the business of gathering facts, ideas or opinions. Organisations are continually developing new concepts and products which rely on effective communication to reach their markets, consumers or audiences. To be successful over long periods, organisations realise they must keep in touch with their customers, users, visitors or audiences. Market research provides
A means by which people providing goods or services can find out what members of the
public think about the products being offered for sale, the advertising, the quality of the services offered and so on. This can apply to both the private sector (businesses) and the public sector (government).
An opportunity for the members of the public to give their opinions and ideas back to the
suppliers of these goods and services.
Survey Research
There are different types of Market Research. The project we are conducting is called Survey Research. Survey Research involves collecting facts, ideas or opinions from a large number of people. When is survey research used? Surveys are conducted when information needed cannot be collected more accurately or less expensively by any other method. For Example: An airline wants to know how many people use their planes to fly between Melbourne and Sydney. It does not need to do a survey to find out. A simple count of tickets sold will give the answer. If the same airline wants to know what air travellers think about in-flight service between Melbourne and Sydney then a survey is needed. We would talk to a group of people who had made the trip recently and ask for their opinions and suggestions. To record their opinions and suggestions we would use a standard form, a list of questions, the questionnaire. The specific group of people we talk to for that survey would be referred to as the sample . By talking to a relatively small group of people we get results that represent the opinions of the total population under study. Just as you can know what a pot of soup tastes like by sampling a few spoonfuls, so you can find out what users think by interviewing a sample of them.
1. Source: Adapted from Market Research Society of Australia Interviewer Training Materials
All surveys have specific purposes, for example to give answers to problems, to measure the size of a market, to measure awareness of advertising, to establish attitudes to products, advertising or social issues, or to measure the effectiveness of programs. While survey objectives may vary, the basic methods of collecting the information are standardised.
Standard Interviewing Techniques
Today we will take you through some of the standard procedures of interviewing, and outline the skills you will develop in becoming a professionally competent interviewer. Where do you fit in? The Interviewer is vital to the success of Market Research as a profession. Every survey is only as good as the interviewing that is done to collect the information. This information can be coded, punched into the newest computers, run on very sophisticated computer programs and the results looked at by the best brains in the country, but if the interviewing is not of the highest standard, then the whole study is wasted. Your central role means that the way you do each interview is vitally important.
Reliability
You must interview the right people You must get complete and accurate information You must work to a schedule. Your interviews will be checked to ensure company standards are met.
The interviewing role
As interviewers, you will play three roles:
The first role is that of the technician' who applies standard techniques and uses the same
questionnaire for each interview.
The second role is that of a human being who builds up a suitable relationship with each
respondent. This enables you to 'record' and act as a tape-recorder to relay information to the survey clients and so help them provide better services.
The third role is as a caring person. You should conduct the interview in such a way that
the respondents feel good about the encounter. You can help them with any queries they may have.
Basic Rules of Interviewing
Use a separate questionnaire for each respondent. Do not interview people who you know personally, or people who have been present
while someone else was interviewed.
Approach people with a confident, smiling attitude. Create a pleasant, friendly rapport. If people refuse to be interviewed, simply thank them and look for someone else. Don't let
a refusal affect your positive approach to the next respondent.
Be careful not to suggest answers to respondents. We want their views, not yours. If they
ask you a question, avoid answering it. At the end of the questionnaire you can have an exchange of views with the respondent if you like.
Do not show the questionnaire to respondents, position yourself so that it is not easy for
them to read it.
Wear your identification badge while you are approaching respondents and interviewing.
The questionnaire
As far as possible, you should read the questions exactly as written. As far as possible, all
interviews should be the same, so all interviewers should follow the questionnaire in the same way. Ask every question (except when following 'skip' instructions), follow the order of the questionnaire.
If a respondent does not understand a question, repeat it exactly as it is written on the

questionnaire.
There is no point in conducting poor quality interviews. We need high quality information,
and you are the person who must be careful and thorough to get the best information.
The Approach
Approach everyone who you think has looked through the exhibition. Approach all age groups and types of people. Let them see you approach them, make eye contact, smile and use a suitable greeting, introduce yourself. For longer questionnaires you will need to ask whether the respondent has the time to answer your questions. If they have, ask the screening questions then sit down with them and begin. Try to interview your respondent separately from their companions.
Types of questions Introduction
You introduce yourself to the respondent.

Screening questions
Check you are speaking to the right person. Ask these before you sit down.
Pre-coded questions
These are questions where the answers are written on the questionnaire. You circle the code number beside the answer which the respondent gives. If you need to change an answer, cross it out clearly and circle the right number. These questions can be either 'single response' where you circle one answer only, or 'multiple response' where you can circle more than one number.
Open-ended questions
For these questions, you need to record word for word what the respondent says, for example, I like to go swimming every day. Use abbreviations if necessary but be prepared to make the writing clearer after the interview. Be sure the respondent is actually answering the question. If the question is What kind of art do we have here? the respondent might say I dont really mind what kind of art, I like all art. While this might be interesting to know, it does not answer the question. You will need to ask the question again, using the same words. You might affirm that you have heard what the respondent has said, by saying You like all art and then repeat the question. Probing All open ended questions need probing, that is getting more information without influencing the respondent in any way. It is important not to suggest answers, or to lead respondents towards an answer in any way. Probing is the most challenging, important and interesting part of interviewing. The quality of an interview depends a great deal on your ability to probe successfully. Have patience, listen and be interested in what the respondent has to say. There are three types of probing:
to help respondents enlarge on what they have said to help respondents clarify what they have said to help respondents explain the reasons behind what they have said.
To enlarge Where else do you know of? Do you know of any other places? You keep asking these kinds of questions until you have gathered all the places they know of, not just the places they thought of first. To clarify Particularly when they give general answers, like 'souvenirs' as a response to a question like, What things do you want to buy in Australia? Here you would probe like this: Q: What kind of souvenirs would you like to buy? A: Small things, Australian things. Q: What do you mean 'small things'? A: Well, things like scarves and ties, or maybe macadamia nuts. Q: And what do you mean by 'Australian things'? A: Macadamia nuts, or t-shirts, or well, anything with Australia, or Sydney written on it I suppose.
Other probes might be What do you mean by .............? This kind of questioning will take you from the general to the specific. Specific information is high-quality information.
To explain when we want to know why they say something, a probe might be Why do you feel unsafe in the streets at night? Are there any other reasons that you feel unsafe?
'Skip' instructions
Where there are no 'skip' instructions always go to the next question. 'Skip' instructions allow respondents to skip questions which do not apply to them.
Show cards
Show cards are sometimes used to give respondents a range of possible answers. They are presented in a book or folder which you might like to think of as an answer book. Of course, all the answers in this book are right, there are no wrong answers. You may let the respondent read each page, or help them by reading out the longer ones.
Rating Scales
These are used to measure the strength of a respondent's reply. For example:
Agree strongly Agree a little Not sure Disagree a little Disagree strongly
If a respondent says 'disagree', ask: Would that be 'disagree a little' or 'disagree strongly' ? and circle the appropriate answer.
Demographic data
It is vital that these questions be answered correctly. They are used to cross-analyse the other questions on the survey to look for differences in response between various groups of people, men compared with women, for example.
Clerical Procedures
Fill in questionnaires accurately. Write quickly in pencil while the interview is underway.
At the end of the interview, quickly check through the questionnaire to make sure you have recorded everything accurately and clearly.
Your health
Prepare to be out in the sun for long periods of time. To avoid sunburn, wear a hat, long sleeves and trousers, and take a 15+ sunscreen lotion with you.
Public Relations
It is our experience that people are happy to be interviewed about the things they like to do. People in a strange place often need information. Have a city map with you, and know where the nearest toilets and information centre are. Be prepared to be helpful to the tourists you encounter (without spending too much time).
Contacts
Please feel free to call your supervisor on the following phone numbers, if you have any queries, or if you want to clarify anything.
6
4. Data cross analysis using a banner
(Referred to on p. 15) This is an example of a table of data which has been cross-analysed by a standard banner. In this case, the banner includes: sex, age, area, number of adults in household and no. of children in household. It is taken from the LeisureScope survey of Environmetrics Centre for Visitor Studies. We see that 6.5% of the total population had been to the Powerhouse Museum in the previous 6 months, compared with 9.7% of people aged 35-49 years. A higher proportion of people with children under 12 years had been (7.3%) compared with people with teenagers (4.4%).
5. Discussion guide for focus group
(Referred to on p.18) This is an example of a discussion guide used to evaluate a guided tour in a botanic garden.
Example discussion guide

Introduce self and general topic. One person at a time, quieter speak up, your views are important, no right or wrong answers. [Mention recording/mirror if applicable.] Ask each person to introduce themselves: name, where they live, family circumstances, hobbies and interests. Previous experience of guided walks. Why did you decide to take this guided walk? You went on two different walks, can you describe for us what your walk was about? What were the main things they were telling you? What were the highlights of the walk for you? What kind of person would you recommend this walk for? Task: Order of importance for guide skills. Discuss your guide with respect to these skills. Suggestions for improving the walk you went on. Suggestions for other kinds of walks that they could offer.
6. Tracking and watching recording form
(Referred to on p.27) When doing a tracking study, you use a fresh form for each party you are following. You record their start and finish times and trace their path on the floorplan. There is space to record details of the group composition and observations about behaviours and activities.
7. Counting and timing recording form
(Referred to on p. 28) When counting and timing people at a particular activity, you use this form. It allows for 20 different observations. For each person you observe, record a brief description of them, note the time they spend at the activity and note any specific behaviours.

Interpretación Australia

Diunggah oleh

Informasi Dokumen

Deskripsi Asli:

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Interpretación Australia

Diunggah oleh

Hak Cipta:

Format Tersedia

A Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Exit Interviews Observation Methods Focus Group Discussions

Gillian Savage Jane James

Locked Bag 2116 North Sydney NSW 2059

GPO Box 2100 Adelaide SA 5001

Version 03.1 Gillian Savage and Jane James, 2001.

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

ACT STRAIGHT AWAY

Adapt . Amend . Ponder . Consider . Consult

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Field research methods that are practical, hands on and do-able

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Fig 1: Role of Evaluation in Interpretation Planning Define

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Evaluation will be possible if your Interpretation Objectives are SMART.

Specific : you say exactly what you

Measurable : you know how you will

Attainable : feasible in scope and in

Timetabled: you know when you want

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

1. Review objectives. 2. Set Key Performance Standard [e.g. the

3. Determine a performance measure [e.g. a

Evaluation is carried out at different stages of project development:

4. Determine a Key Performance Indicator [KPI]

5. Gather the data. 6. Apply the data and draw conclusions.

Critical Success Factors

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Checklist Define your information needs

1. Write down all the things you want to know

1 . List your information needs and against each

2. Invite colleagues to think of things they

3. Sort through all the ideas, weed out

4. For each item on the list, write down how that

4 . Decide what tasks to carry out in-house and

5. Prioritise the information on the list. Three

5 . Manage the distribution of tasks and their

6. Highlight all the items that can be answered

6 . Apply the data to the question in hand and

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Specific messages Overall rating Suggestions

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Interviewing is best carried out as a separate

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Questionnaire design process

Use Refine Test Draft

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Here is the process in more detail.

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

4. Focus Group Discussions

1. Define what information you need and say

who have been on the boardwalk.

have been on the boardwalk.

4. Write discussion guide and prepare any

5. Conduct the group discussions.

under 18 and have been on the boardwalk.

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

4. Focus Group Discussions

Practical Guide to Evaluating Natural and Cultural Heritage Interpretation

4. Focus Group Discussions