Questionnaire Design Methodology

Contents
1. 2. 3. 4. 5. 6.

What are questionnaires? Why use them? Properties of a questionnaire Procedures for questionnaire design Principles of questionnaire design Questionnaire piloting Conclusions

Various definitions of questionnaire

A set of questions arranged in a certain structure to collect information from targeted group to answer one or several of research questions A set of questions particularly used to address some statistically typical issues by gathering information in a survey. And others
3

Why questionnaire?
Questionnaire is used to Assess the need for investment and development, and the differences among project targeted areas. To measure outcomes of project and intervention programmes Assess the beneficiaries satisfaction Assess the project impacts
4

The importance of Questionnaire

Questionnaires are the most important tool of a social investigation Questionnaires reflect the research issues The quality of a questionnaire determines the quality of a survey Designing a high quality questionnaire helps:

Collect most accurate data Increase response rate Reliable and valuable Save time and money

Low quality questionnaires

Non response may affect the questionnaire validity just as same as missing data (or obtaining inaccurate data).

Low response rates

Reduce the effectiveness of sample size

Reduce the research power

Reduce the accuracy of final results

Procedures for questionnaire design

3 major steps: 1. Deciding on what to ask 2. Defining types of question, wording for each type of question 3. Structuring and formatting the questions

Deciding on what to ask

Define mid-level steps between the research question(s) and the final question(s) Research question(s) Relevant matters Stay focused Questions

Defining types of question

Yes/No questions >< Open questions Single choice >< Multiple choice Categorical questions Ordinal questions Linkert-scale questions Single-pole >< Double-pole

Wording for questions

Clear meaning, not vague Avoid multi-meaning questions Avoid negative questions Avoided guiding questions Stay focused Questions

Structuring the questions

Title Introduction (enclosed with intro letter) Contact Minimal types of question Avoid horizontal & vertical structure Avoid overuse of boxes, tables, and lines

Structuring the questions (cont.)

Clear, complete and attractive structure Two styles of question order:

From easy to difficult, from general to specific From most interesting to less interesting

No matter what style to be chosen, follow the logic, i.e. questions in each part are relevant Keep the rhythm of the questionnaire Avoid overusing of skips and turns

The Art of asking questions

Ask the right question Respondents must understand the question Respondents must know the answer Respondents must be willing and able to give the answer

13

Types of Survey Questions

1. 2.

3.

Questions about activities, facts/events Questions about psychological status or attitudes Questions about knowledge/perception

14

Questions with a specific time

Bad question: Over the past 6 months, what new household appliances have you purchased? Better question: Now Im going to read a list of household devices. As I read each, please confirm whether you bought that appliance during the last 6 months. How about a refrigerator? a kitchen device? a microwave? 15

Specifying
Who, what, when, where and how? Ex. : Whose income? What are included? Over what period of time? Ex.: In 2002, what was your total household income? Before tax? Please provide the incomes of each member of your family, including wages, social welfare, and public aids
16

Questions with single-meaning words

Not clear: How do you compare the distance between you and your family at the moment and in the past? Do you think you live closer to your family? Or still the same? Clearer: Compared with your previous house, do you live closer now? Or is it further? Or the same?

17

Open vs. Closed Questions

General rules: for quantitative research, closed questions are often better than open questions, because:
Easier for respondents Easier for data entry, Not necessary to recode data Easier for analysts to categorize respondents

18

Responses to open & closed questions

Open questions are those left open for respondents to freely answer.
How do you feel about your health status today?

Closed questions provide predetermined answers

Do you feel well today? Yes No How much do you agree with the following statement: Today my health is great: Agree Neither agree nor disagree Disagree
19

Closed format

Closed questions are often useful as it is

Easy and quick to fill in Easy to code, record and analyse

Closed questions are often not useful as...

It does not cover all possible responses (comprehensiveness) It is not able to obtain deep information
20

Disadvantages of Closed Questions

Answers may lead the respondents It may be too easy to answer without much thinking Should not be used when:

Asking for the frequency of sensitive behaviors There are many different possible answers

21

Open format

Open questions useful when

Answers are unknown Answers are too many/complicated to be coded searching for further explanations of the answers

Open questions are not useful when

Theres a need to code possible errors It is difficult to analyse and synthesize
22

Type of measuring scales

Nominal scale
Ex. Male/female; Provinces: Bac Kan/Ha Tinh/Tra Vinh/Tuyen Quang

Ordinal scale
Ex. Very poor poor average rich very rich; like normal dislike

Interval scale
Ex. range of income, range of age.

Ratio scale
Ex. Percentage of male/female, ethnic groups 23

Format of response categories

Circling or tick off your answers Skip & leave blank

24

Survey Intro/Cover Letter

Introduction should include:
who is conducting the survey the topics covered in the survey Statement of confidentiality Statement that the time length of the survey depends on the research mode, topics, population

25

Check the format of questionnaire

Number all the questions sequentially Use large, clear font type; List groups of answers in vertical sequence Avoid double/triple-choice answers Be consistent with guidelines and orders of answer groups.

26

Check the format of questionnaire

Dont split question across pages. If necessary, move the question and answers to next page. Put specific instructions of questionnaire as needed, then the questions Distinguish instructions from questions Pre-code the questionnaire using check boxes

27

Principles for questionnaire design

Ask what you want to attain Ask the way you want to gain the answer: use of words Understand the mind of respondents and the context of the interview Other rules

28

Use of words
Wording should be accurate: a small change could make great differences Words in a question affect the answer. The more specific the question, the more it influences the answer.

29

Context of the interview

Many cases are encountered:

The interviewer may even chitchat various topics with an enthusiastic respondent In some cases, the interviewee may refuse to answer.

Notes for a new interviewer (see next page)

30

Rules for new interviewers

Avoid jumping to the questions until you really understand the research question Once raising a question, ask yourself why I need to know this matter? Use available questions from different sources.

31

Use available questions

Ready-made questions from previous successful investigations are encouraged to use, so that:
To save time and money (short-cut) To compare with the previous results To improve the validity of the answers of research with similar sample size, context and without changes For investigations with different time and with changes, use the same questions to measure trend.
32

Use available questions (cont.)

When using questions from other surveys, be concerned with:

Copyright Setting of question (as the answers for some questions are particularly used for a certain situation)

33

Other principles
Guarantee clear codes for main and subquestions (ex. 1, 2a, 3, 4b) Start with simpler questions (not sensitive) Use proper logic. If the questionnaire is long, start with the keys Put questions on anthropology at the end Put questions for past experiences prior to those for current status. Put general questions first, then specific

34

Other principles (cont.)

Apply the funnel-shaped rule (from wide to narrow questions) Use the rule of filtering Use equivalent terms (avoid technical ones) Avoid unclear terms/short terms Avoid multi-meaning words/terms Keep sentences short (if possible)

35

Other principles (cont.)

Avoid suggesting the answers (by proposing a hypothesis) Avoid leading questions Be cautious using hypothetical questions Avoid bias (see next)

36

Piloting: Testing the Questionnaire

Preferable to test the questionnaire with those similar to targeted subjects of the research population Conduct pilot using the same mode for the major study Consider cognitive pretesting

37

Piloting: Testing & Evaluation

1. Develop & test the research tools

Piloting

2. Assessing the feasibility of a survey against sample size

3. Identify logistical problems of the questionnaire

38

Piloting: Testing & Evaluation

Prepare first draft and pre-test

Distribute questionnaire to small sample

Evaluate responses & re-draft

Checklist items: (adapted by Bowling 2002) The questions measure what is intended wordings are understood by and means a similar thing to all Instructions are easily understood/followed Do the responses cover all relevant issues? Any questions systematically/frequently missed? Basically test for bugs

39

Complete the Questionnaire:

Questionnaire Design - Practical approach
1.

2.
3. 4. 5. 6. 7.

Decide what information is needed Search for existing questions Focus groups Draft new questions/revise existing ones Sequence the questions Get peer evaluation Revise and self-test or peer test

Complete the Questionnaire:

Questionnaire Design - Practical approach 8. Think-aloud interviews 9. Revise/eliminate questions 10. Prepare interviewer instructions for pilot test 11. Pilot test (10-20 cases) 12. Revise eliminate questions based on respondent & interviewer comments

Complete the Questionnaire:

Questionnaire Design - Practical approach
13. Pilot

again, if necessary 14. Prepare final interviewer instructions 15. Modify questionnaires if interviewers raise questions in training 16. After the interview completes, debrief interviewers for potential problems 17. Use experience from a study for planning a study in the future

Conclusions

Not general recommendations, just issues to be aware of, consider and balance out Consider the aims of study, population and resources available
Principal objective is to collect data that is RELIABLE, VALID & UNBIASED
43

Key References To Get Started

Dillman, Don. Mail and Internet Surveys: The Tailored Design Method. New York: John, Wiley & Sons, Inc, 2000. Bradburn, N, Sudman, S. and Wansink, B.. Asking Questions: The Definitive Guide to Questionnaire design for Market Research, Political Polls, and Social and Health Questionnaires. San Francisco: Jossey Bass, 2004. Oppenheim AN. Questionnaire design, interviewing and attitude measurement. 2nd ed. London: Pinter; 1992.

See also questionnaires chapter in the handbook of survey

44