Figure out what you are trying to learn and have a clear image in your mind of the
research objectives. This will help you determine whom you will survey, formulate the
questionnaire and decide on the kind of statistical analysis you need to perform. If your
goals are unclear, the results will probably be unclear.
Usually, the population will be too large for you to attempt to survey all its members. A
small, but carefully chosen sample can be used to represent the population. Sampling
is used to cut costs and effort while still obtaining information from a representative
sample of the target population. It is essential that the number of individuals
participating in the survey be large enough to produce results that are reliable and valid
and represent the target population with the minimum statistical error. If unsure, there
are some websites that help you calculate your sample size and your error (e.g.,
http://www.raosoft.com/samplesize.html, http://www.surveysystem.com/sscalc.htm,
http://www.researchinfo.com/docs/calculators/samplesize.cfm,
http://www.macorr.com/ss_calculator.htm, etc)
The main questions in selecting your sampling design are (apart from your budget of
course):
- Which are the demographics?
- What is the size of your target population?
- Are you interested in any sub-groups of the population? It is sometimes useful to
establish quotas to ensure that your sample accurately reflects relevant sub-groups
in your target population.
- How many will be included?
Sampling methods can be either probability (each member of the population has an
equal probability of being surveyed) or non probability (members are selected from the
population in some non-random way). The advantage of probability sampling is that
1
http://mechanisms.energychange.info
sampling error can be calculated. In non probability sampling, the degree to which the
sample differs from the population remains unknown.
Probability methods
Random sampling: the purest form of probability sampling. Each member of the
population has an equal and known chance of being selected. When there are very
large populations, it is often difficult or impossible to identify every member of the
population, so the pool of available subjects becomes biased.
Systematic sampling: After the sample size has been calculated, every Nth record is
selected from a list of population members. As long as the list does not contain any
hidden order, this sampling method is as good as the random sampling method.
Stratified sampling: The population is divided into groups of individuals that are similar
in some respect. After the population has been divided into the strata, a random
selection of a proportion of individuals from each stratum is made.
Cluster sampling: used if the target population is spread over a large geographic area.
The survey area (such as a territory) is divided into clusters. A random sample of these
clusters is drawn and individuals within the cluster participate in the survey.
2
http://mechanisms.energychange.info
3. Decide on your survey method
The methodology will affect response rate, the kind of questions that can be asked and
how they are asked. When choosing the most appropriate method you must consider
the best way to communicate with potential respondents, skills of staff / interviewers,
the availability of internal resources and time available. The key is to choose a method
that will:
- give you the highest response rate with your particular target group
- be the most convenient for them
- fit your time line
- fit your budget
- fit your staff and resources
The primary methods for obtaining survey research are the following:
b) Telephone interviews
Advantages
- It is possible to achieve high response rates
- The amount of non-response to questionnaire items can be minimized
- Quick results
- Less costly than face to face interviews
- You can dial random telephone numbers when you do not have the actual
telephone numbers of potential respondents
Disadvantages
- Sometimes difficult to reach a selected resident of a household
- Long or complex questions should be avoided
- Can be confused with telemarketing sales, which drops the response rate
- No working person is home during the day. This limits calling time to a "window" of
about 18:00-21:00.
- You cannot show visual material for assessment
3
http://mechanisms.energychange.info
c) Mail interviews (self completed)
Advantages
- Social desirability bias is minimized
- Cost effective
- The questionnaire can include pictures
- They are not intrusive
Disadvantages
- You need to ask simple, easy to answer questions, as it is self completed
- Some questions may not be complete on returned questionnaires
- Too much time is needed until all questionnaires are filled and sent back to you
- You can not be sure of who exactly has filled in the questionnaire
- You cannot reach populations of lower educational and literacy levels, or
immigrants
- Low response rate. One way of improving response rates is to mail a letter telling
your sample to watch for a questionnaire in the next week or two. Another is to
follow up a questionnaire mailing after a couple of weeks with a card asking people
to return the questionnaire. Another way is to use an incentive. Another possibility is
to include the people who return completed surveys in a drawing for a prize. A third
is to offer a copy of the top line results to those who complete the questionnaire.
4
http://mechanisms.energychange.info
Advantages
- Fast results
- Economical
- You can show pictures, video and play sound
- The novelty element often stimulates higher response levels
- You can use complex question skipping logic and randomizations
- Some people will give more honest answers to questions about sensitive topics
- On average, people give longer answers to open-ended questions on internet
questionnaires than they do on other kinds of self-administered surveys.
Disadvantages
- Internet surveys do not reflect the population as a whole in cases with low internet
usage penetration, or practical exclusion of older target groups
- You need to ask simple, easy to answer questions, as it is self completed
- People can easily quit in the middle of a questionnaire. They are not as likely to
complete a long questionnaire on the web as they would be if talking with a good
interviewer.
- Depending on your software, there is often no control over people responding
multiple times to bias the results.
5
http://mechanisms.energychange.info
Qualities of a Good Question
- Avoids emotionally charged or leading words and phrases that point towards a
certain answer.
- Asks for an answer on only one dimension. Do not put two questions into one. For
example if you ask “To what extent did the X initiative influence you to unplug your
TV when not in use and change your energy saving settings in your PC?”, the
respondent will not know to which of the two to answer. A good question asks for
only one "bit" of information.
- Can accommodate all possible answers so that you do not confuse and frustrate
the respondent. Add all the necessary response categories and allow multiple
responses.
- If you want only one answer from each person, ensure that the options are mutually
exclusive. A good question leaves no ambiguity in the mind of the respondent.
There should be only one appropriate choice for the respondent to make.
- Follows comfortably from the previous question. Transitions between questions
should be smooth.
Question Types
a) Open ended vs. closed questions
- In closed questions, response categories are provided and interviewers /
respondents simply have to circle or choose an option
- Open-ended questions are asked without specific response alternatives;
respondents need to write / say their own answer. They are good for depth and
unbiased opinions
- Open-ended questions are best used when having multiple response options may
be too leading and result in biased answers
b) Multiple-choice
- In such questions, respondents choose one or multiple response answers
- Make sure you include all the relevant alternatives as answer choices.
c) Scales (e.g., rating, ranking or agreement)
- Scales is an issue on which researchers differ; whether to use a scale with an odd
or even number of points. Some like to force people to give an answer that is
clearly positive or negative. Others feel it is important to offer a neutral, middle
option.
- Ranking example: Of the following 3 services, which one would you feel is most
important? Which one is second-most important? And which one is the third-most
important?
- Agreement example: On a scale of 1 to 5 where 5 is strongly agree and 1 is
strongly disagree how would you rate your agreement or disagreement with the
following statement…
If you need help designing the questionnaire, you can download a free questionnaire
wizard (e.g., http://www.dobney.com/Technology/QWizard.zip).
6
http://mechanisms.energychange.info
5. Pre-test the questionnaire – conduct pilot interviews
The last step in questionnaire design is to test it with a small number of interviews
before conducting your main survey. Ideally, you should test it on the same kinds of
people you will include in the main study. If that is not feasible, have a few people,
other than the question writer, try the questionnaire. Piloting can reveal unanticipated
problems with questionnaire length, wording, comprehension, question sequence,
instructions to skip questions, inadequate response alternatives in closed questions,
etc. It can help you see if the interviewees understand your questions and give useful
answers. After the pilots, you can revise the questionnaire, accordingly, if needed.
A good interviewer follows a few simple guidelines which ensure detailed, accurate,
and unbiased data. Some tips follow:
- Read the questions as written, do not change the wording
- Do not suggest responses (with verbal or non verbal cues); allow the respondent to
answer without input from you.
- If respondents get off topic in open ended questions, you should remind them of the
question. Sometimes repeating the question will help. If the respondent has
difficulty thinking of a response, resist the temptation to “help”.
- Record answers neatly and thoroughly
- Occasionally, a respondent will want to terminate before completion. If this is the
case, you should make sure the interview gets done. This can be accomplished by
telling the respondent how important his/her opinions are and working together to
get through the interview quickly. If it is impossible to complete the interview,
arrange a time to call back at the respondent’s convenience to complete the
interview.
- For telephone interviews
o Do not simply hang up when someone refuses to participate. Politely ask
the respondent why they do not wish to take part. If they are busy you can
arrange a more convenient time to call back. If they are concerned about
the nature of the study, you can answer their questions. If they are nervous
about offering their opinions, you can reassure them that all information is
strictly confidential and there are no right and wrong answers.
o Try calling back 3 to 5 times before assuming you cannot reach the person
at that number.
o You should record on the sample list what happens every time you attempt
to do an interview with a person (refusal, absence, wrong number, ineligible
respondent, scheduled call-back, busy line, etc).
- For face to face interviews
o The most appropriate location to conduct a face to face interview is a place
where your sample frequents and is comfortable for them to participate. If
7
http://mechanisms.energychange.info
you are randomly selecting respondents, it is important to utilise more than
one location in order to ensure a better representation.
o While at the location, select every Nth passer by to ensure randomisation
Unless you have been entering responses directly into a computer programme,
processing the data involves preparing and translating the data for analysis. It involves
taking the completed questionnaires and putting them into a format that can be
summarised and interpreted.
Coding: separate answers to open ended questions into groups that can share a code
list. When no new responses are found, rewrite codes and assign a number to each
code.
Data entry and tabulation: data entry and tabulation can be easily done with computer
software packages. Write the corresponding codes beside each question on each
questionnaire. Then you can enter the data in a programme such as SPSS or Excel.
Most statistical applications have data entry capabilities.
Data analysis: once the data has been entered into your statistical package, the
analyses required to answer your research questions can be performed. For most
surveys simple descriptive statistics (frequencies, means, medians, ranges, etc) may
be all that is needed to be able to interpret your results. More complex analyses may
be required when comparisons are needed between subgroups or for measurements
taken at different times. Confidence intervals, t-tests (to compare results for continuous
data), Chi-square (to compare results for categorical data), Factor analysis, Cluster
analysis, and Correspondence analysis are some of the most common analysis
performed.
8
http://mechanisms.energychange.info
Checklist for quantitative surveys
9
http://mechanisms.energychange.info