Air University Sampling and Surveying Handbook

Guidelines for planning, organizing, and conducting surveys

The views and opinions e pressed represent the personal views of the author and editor only and should not in any way be construed to reflect any endorsement or confirmation by the !epartment of !efense, the !epartment of the Air "orce, or any other agency of the United States Government#

$evised %&!"' (dition) *ay +,,+ &revious (ditions) -../, -..0, -.11, -.12 This handbook contains guidelines for planning, organizing, and conducting surveys# 3t should be useful to anyone embarking on a pro4ect re5uiring the gathering of data through the medium of the 5uestionnaire# The te t is designed to be easily readable, even for someone with a limited background on the sub4ect# The book is the product of the efforts of several people# *a4or 6eith 7# $oss did the ma4ority of the work in fulfillment of his research re5uirements while a student at the Air 7ommand and Staff 7ollege, Air University, *a well Air "orce 8ase, Alabama in -.99# :t 7ol :awrence !# 7lark was the research advisor for the pro4ect, and *a4or Thomas 7# &adgett did the final editing and assembling on the original edition# !r# Thomas $# $enckly, Air

University 7urriculum 7oordinator, edited the -.11 reprint of the first edition, the -..0 second edition, and the -../ 3nternet edition, in addition to providing supplemental information on bias in survey research %7hapter 2' and common statistical analysis errors %7hapter /'# 3f you have any 5uestions about the book, !r# $enckly can be reached at) H; AU<7"A22 :e*ay &laza South*a well A"8, A: 0/--+=/002%00>' .20=+.1. or !S? >.0=+.1.thomas#renckly@ma well#af#mil To say that peopleAs opinions and attitudes are more important now than ever before is very nearly an understatement# *ore and more we are seeing individuals and groups relying heavily on the opinions and attitudes of customers, constituents, concerned citizens, focus groups, etc#, to provide information for decision making# 3t is also obvious even to the casual observer that surveys %including paper=based 5uestionnaires, personal interviews, and telephone polls' play a crucial role in gathering these opinions and attitudes# Surveys are also used as evaluation and control devices# They can be used to measure the effectiveness of an ongoing pro4ect, such as an information program, for e ample# 8y surveying the participants in a program, the effectiveness of the program can be determined# Also, management can use surveying as an aid to control, by finding new problem areas and insuring that old problem areas have been corrected %which is, for instance, one of the fundamental premises of total 5uality management'# The need for accurate information to fuel the decision=making process e ists at all levels of management# This has created a trend for surveys to be generated at lower management levels by staff officers, many of whom are not e perienced in survey development or administration# The growing necessity to survey and the relative lack of knowledge on surveying methodology leads to a significant demand for information on the sub4ect# The primary purpose of this guide is to supply this information in simple, non=technical language# An e5ually important purpose of this guide is to identify problems that may arise during development of a survey and to provide techni5ues and guidance for solving these problems# The procedures presented in this guide are designed to help you develop valid and useful surveys# The steps in surveying are varied and comple # Therefore, this guide only highlights the ma4or information, techni5ues, and procedures available to the surveyor# $eferences offering more detailed treatments of these sub4ects are provided in the bibliography and appendices# Since the surveyor fre5uently is unable to reach the entire group in which he is interested, this guide e plains sampling techni5ues# To use these techni5ues necessitates only a rudimentary knowledge of statistics# "inally, although many of the techni5ues and procedures covered here apply e5ually well to the personal or telephone interview survey, the

primary focus is on the self=administered and group=administered surveys# Bne word of caution) 8ecause the steps in survey preparation are closely interrelated, you should study this entire guide before beginning an initial survey effort#

"B$(CB$! ############################################################################################################## ######### i &$("A7(#################################################################################################################### #######ii 7HA&T($ #######################################################################################################################3?T$B!U7T3B? TB SU$D(E3?G ######################################################################################## -Types of Surveys########################################################################################################## Advantages F !isadvantages of Two Types of Surveys############################################# To Survey or ?ot to Survey ######################################################################################### 0 Guide Butline ############################################################################################################### > 7HA&T($ + #######################################################################################################################9 TH( A3$ "B$7( &($SB??(: SU$D(E &$BG$A*############################################################## 9 7HA&T($ 0 #######################################################################################################################. !(D(:B&3?G TH( &U$&BS(, HE&BTH(S3S, F SU$D(E &:A?############################################ .The &urpose################################################################################################################ ## . The Hypothesis, Bb4ective, or $esearch ;uestion##################################################### -, The Survey &lan ######################################################################################################### -The !ata 7ollection

&lan ########################################################################################### -The !ata $eduction and $eformatting &lan ############################################################## -+ Bpen= and 7losed=(nded ;uestions ########################################################################## -+ The Analysis &lan ###################################################################################################### -0 7HA&T($ > #####################################################################################################################

-2

SA*&:3?G T(7H?3;U(S A?! $(:AT(! STAT3ST37A: 7B?7(&TS ################################### -2Sampling *ethods###################################################################################################### -2 Simple $andom Sampling########################################################################################## -. Stratified $andom Sampling ###################################################################################### +, Systematic Sampling################################################################################################## ++ Gudgment Sampling#################################################################################################### +0 &urposive Sampling#################################################################################################### +0 $epresentative Sampling############################################################################################ +> "actors 3nfluencing Sample Size################################################################################ +> 7onfidence :evel and &recision################################################################################# +2 !etermining the Size of the Sample########################################################################### +2 7HA&T($ 2 ##################################################################################################################### +. TH( ;U(ST3B??A3$(###################################################################################################### +.&ros and 7ons############################################################################################################# +. The 7ontents ############################################################################################################## 0Types of ;uestions##################################################################################################### 00

;uestionnaire 7onstruction######################################################################################## 00 3ntensity ;uestions and the :ikert Scale#################################################################### 09 8ias and How to 7ombat 3t######################################################################################## 0. 8ias 3nvolving Dolunteer Samples############################################################################# >7onclusions Carranting *a imum 7onfidence##################################################### >+7onclusions Carranting 7onsiderable 7onfidence ############################################### >+7onclusions Carranting Some 7onfidence ############################################################ >07onclusions Carranting *inimum 7onfidence ##################################################### >>Summary ########################################################################################################### ######### >2 7HA&T($ / ##################################################################################################################### >9 7B**B? STAT3ST37A: A?A:ES3S ($$B$S #################################################################### >9 A&&(?!3H A################################################################################################################### 2ST(&S 3? SU$D(E3?G###################################################################################################### 2A&&(?!3H 8 ################################################################################################################### 20 SU$D(E !(D(:B&*(?T T3*(TA8:(S ############################################################################ 20 A&&(?!3H 7 ################################################################################################################### 22 SU$D(E !ATA SBU$7(S ################################################################################################ 22 A&&(?!3H !################################################################################################################### 29 HBC TB US( A $A?!B* ?U*8($ TA8:(###################################################################### 29TA8:( B" $A?!B* ?U*8($S######################################################################################## 21 A&&(?!3H ( ################################################################################################################### 2.

TA8:( B" I DA:U(S ###################################################################################################### 2. A&&(?!3H "#################################################################################################################### /SA*&:(S ################################################################################################################### ##### /-7over :etter ############################################################################################################### /&rivacy Act Statement################################################################################################ /+ Survey 3nstructions##################################################################################################### /0

A&&(?!3H G################################################################################################################### /2 $AT3?G B$ 3?T(?S3TE S7A:(S####################################################################################### /2TA8:( D333=(=- ==A77(&TA83:3TE &H$AS(S ############################################################## /2TA8:( D333=(=+ ==!(G$((S B" (H7(::(?7() "3$ST S(T ########################################## /9TA8:( D333=(=0 ==!(G$((S B" (H7(::(?7() S(7B?! S(T ###################################### /1TA8:( D333=(=> ==!(G$((S B" :36( A?! !3S:36( ##################################################### /.TA8:( D333=(=2 ==!(G$((S B" GBB! A?! &BB$ ####################################################### 9,TA8:( D333=(=/ ==!(G$((S B" GBB! A?! 8A! ######################################################### 9-TA8:( D333=(=9 ==!(G$((S B" AG$(( A?! !3SAG$((############################################### 9+TA8:( D333=(=1 ==!(G$((S B" *B$( A?! :(SS ######################################################## 90TA8:( D333=(=. ==!(G$((S B" A!(;UAT( A?! 3?A!(;UAT( ################################### 9>TA8:( D333=(=-, ==!(G$((S B" A77(&TA8:( A?! U?A77(&TA8:(########################## 92TA8:( D333=(=-- == 7B*&A$3SB? &H$AS(S################################################################## 9/TA8:( D333=(=-+ ==!(G$((S B" SAT3S"A7TB$E A?! U?SAT3S"A7TB$E ################### 99TA8:( D333=(=-0 ==!(G$((S B" U?SAT3S"A7TB$E#################################################### 99TA8:( D333=(=-> ==!(G$((S B" &:(ASA?T################################################################ 99TA8:( D333=(=-2 ==!(G$((S B" AG$((A8:( ############################################################# 91TA8:( D333=(=-/ ==!(G$((S B" !(S3$A8:( ############################################################## 91TA8:( D333=(=-9 ==!(G$((S B" ?37(######################################################################### 9.TA8:( D333=(=-1 ==!(G$((S B" A!(;UAT(############################################################### 9.TA8:( D333=(=-. ==!(G$((S B" B$!3?A$E ############################################################### 9.TA8:( D333=(=+, ==!(G$((S B" AD($AG(################################################################# 1-TA8:( D333=(=+- ==!(G$((S B" H(S3TAT3B? ############################################################# 1-TA8:( D333=(=++ ==!(G$((S B" 3?"($3B$################################################################## 1+TA8:( D333=(=+0 ==!(G$((S B" &BB$######################################################################## 1+TA8:( D333=(=+> ==!(S7$3&T3D( &H$AS(S################################################################# 10 A&&(?!3H H################################################################################################################### 10

SA*&:( S(TS B" $(S&B?S( A:T($?AT3D(S ################################################################## 10 Table D333="=- ==&hrases at :east Bne Standard !eviation Apart with &arallel Cording################################################################################################################ ###### 1> Table D333="=+ ==3ntervals 8etween &hrases Are as ?early (5ual as &ossible####### 1/ Table D333="=0 ==Scale Dalue 3ntervals 8etween &hrases Are as ?early (5ual as &ossible ############################################################################################################### ####### 19 Table D333="=> ==Sets Using Brder=of=*erit :ists of !escriptors ########################## 11 838:3BG$A&HE ############################################################################################################ # 1.

3ntroduction to Surveying
Cebster defines a survey as Jthe action of ascertaining facts regarding conditions or the condition of something to provide e act information especially to persons responsible or interestedK and as Ja systematic collection and analysis of data on some aspect of an area or group#K A survey, then, is much more than the mere compiling of data# The data must be analyzed, interpreted, and evaluated# Bnly after this processing can data become information# The Le actnessL of the information is determined by the surveyorAs methods# Unless he makes a systematic collection of data, followed by a careful analysis and evaluation with predefined ob4ectives, his collection of data cannot become Je actK information# TE&(S B" SU$D(ES Surveys can be divided into two general categories on the basis of their e tensiveness# A complete survey is called a Jcensus#K 3t involves contacting the entire group you are interested in == the total population or Juniverse#K The other category is more commonM it is a sample survey# A sample is a representative part of a whole group %universe'# Thus a sample survey involves e amining only a portion of the total group in which you are interested, and from it, inferring information about the group as a whole# A!DA?TAG(S F !3SA!DA?TAG(S B" TH( TCB TE&(S B" SU$D(ES Bne of the decisions to be made in surveying is whether or not to

sample# &arten %-.2,, p -,.' presents a list of advantages and disadvantages of the sample survey# %These, in turn, imply the advantages and disadvantages of the census#' The three most important considerations for the surveyor are) speed, low cost, and increased accuracy and analysis of the data# 8y sampling only a small portion of a large population, it is possible to collect data in far less time than would be re5uired to survey the entire group# ?ot only is data collection 5uicker, but data processing and analysis also re5uire less time because fewer pieces of data need to be handled# $apidly changing conditions and the short turn=around time imposed in many surveys make the efficient use of time a critical variable# 3f an accurate snapshot of the attitudes of a particular group is desired, currency is of paramount importance# &rofessional political pollsters make their living by providing 5uick snapshots of the Jpolitical climate#K $esults of such polls lose their accuracy very 5uickly %sometimes in as little as +> hours==particularly in the days preceding a ma4or election'# So, for these pollsters, time is truly of the essence# 3tAs probably a safe bet that those of you reading this guide will not need that degree of speed# ?evertheless, speed is essential to ensure the data are Lfresh,L especially when it comes to assessing public opinion in a volatile or contentious area before they change appreciably# The smaller amount of data gathered by sampling as opposed to surveying an entire population can mean large cost savings# 8y limiting the group to be surveyed, less time, hence less cost, are involved in collecting, formatting, and analyzing the data# 3n addition, if automated data processing %A!&' e5uipment is being used to analyze data, your overall time investment will be even less, as will be the overall cost# Sampling allows you to do a credible 4ob for a smaller investment of time and money# &arten %-.2,' also notes that sampling enables the surveyor Jto give more attention to each return received and to make certain that the data are as accurate as possibleK %p --,'# This attention may lead to more precise information than would a less careful collection of data from the entire population# ?othing more than a rudimentary 5uality control is possible for the great volume of raw data gathered in a census# The more data collected, the greater the potential for making JaccountingK errors# The disadvantages of sampling are few, but important# The main disadvantages stem from risk, lack of representativeness, and insufficient sample size, each of which can cause errors# 3nattention to any of these potential flaws will invalidate survey results# 3t is important to realize that using a sample from a population to infer something about the entire population involves a risk# The risk results from dealing with partial information# 3f risk is not acceptable in seeking the solution to a problem or the answer to a 5uestion, then a complete survey or census, rather than a sample survey, must be conducted# !etermining the representativeness of the sample is the surveyorAs

greatest problem when sampling# 8y definition, LsampleL means a representative part of an entire group# To avoid the charge of using Jbiased data,K it is necessary to obtain a sample that meets the re5uirement of representativeness, and this is not an easy task# Cithout a representative sample, a survey will, at best, produce results that are misleading and potentially dangerous# &rocedures for minimizing the possibility of using a nonrepresentative sample are covered in 7hapter ># The final ma4or problem in sampling is to determine the size of the sample# The size of the sample you need for a valid survey depends on many variables including the risk you are willing to accept and the characteristics of the population itself# The determination of sample size is discussed in 7hapter ># Here, it is sufficient to say that if sampling becomes too complicated, or the re5uired sample size becomes too large, the easiest solution may be to survey the entire population# The decision as to whether to survey the entire population or only a sample of it is not based on the above advantages and disadvantages alone# 3t is affected by many other variables that are covered later in this guide# TB SU$D(E B$ ?BT TB SU$D(E 8efore attempting a survey, you should investigate some basic facts and answer some pertinent 5uestions# The result of this investigation will be a greater realization of the work involved in producing a survey# &erhaps it will lead to a decision not to survey# Surveys demand time%maybe more time than you have available# The e act amount of time varies greatly from survey to survey depending on the number of people to be surveyed and the content of the survey# A survey of a few 5uestions administered to the people in your office may take only a day or so, whereas a larger survey administered to a great number of people located worldwide can take over three months from the time the survey is delivered to the printer %see Appendi 8'# And this does not include the time needed to design the survey and construct the 5uestionnaire# *oreover, coordination with officials and the customers of the survey takes additional time# 3f your estimate of the time needed to produce the survey e ceeds your deadline date, you are likely to decide you do not have the time to conduct a survey# A hurried survey wastes both your time and that of your respondents# The results of a hurried effort are 5uestionable at best# Surveys are e pensive to produce# The solution to the problem or the answer to the 5uestion may not be worth the cost to produce it# (ven if it would be worth the price, you may not be able to obtain the needed funds, either from your own pocket or from your organizational budget# Although

no standard estimates of survey cost are available, some of the items of e pense can be e amined# The primary e pense is in time and effortM the time you spend producing the survey could be spent on other tasks# 3f other personnel are needed, they will have to be paid# Access to typewriters, word processors, and calculating machines %or computer resources' is a must# 3f you e pect to gather a great deal of data, the cost of renting A!& time and of purchasing A!& scanner sheets should be e amined# Surveys of more than -2,=+,, respondents cannot feasibly be tabulated by hand# The same is true for groups of less than -2, respondents if the survey 5uestionnaire is lengthy# The final cost involves supplies# At a minimum you will need paper and envelopes# Eou may also have to pay either the cost of printing the survey 5uestionnaire or the postage or both# (ach of the above costs that applies to your survey should be estimated and the total cost measured against the survey re5uirement# Since surveys are being used more and more, the information you want may have already been gathered# A search of some of the survey data sources listed in Appendi 7 might yield a solution to your problem or at least provide e amples of how others have approached similar problems# So before you undertake a survey, first make sure the answer to your problem does not already e ist# ?e t, evaluate the time you will need and determine the cost involved to produce the survey results, and then weigh these findings against the importance of the survey# Undertake a survey only if it is worth the time, effort, and cost to make it a good one#

GU3!( BUT:3?( The remaining chapters of this guide cover the various steps in surveying# 7hapter + outlines the official policies and procedures within the Air "orce for conducting surveys# 7hapter 0 covers the determination of the purpose, hypotheses, and survey plan# 7hapter > deals with the design of a sample survey and the techni5ue for determining the re5uired sample size# The concepts presented in 7hapter > will not apply to you, of course, if you are conducting a full census rather than a sample survey# 7hapter 2 outlines the construction of the survey 5uestionnaire# 7hapter / discusses some of the more common statistical errors committed by novice researchers and ways to avoid them# The various appendices contain checklists and data sources useful in surveying# The bibliography lists informative references on surveying#

The Air "orce Survey &rogram

&ersonnel

The purpose of the Air "orce &ersonnel Survey &rogram is to foster the development of compatible and effective surveys, and to minimize e posure of Air "orce personnel to repeated or unwarranted survey solicitations %Air "orce 3nstruction %A"3' 0/=+/,-, -..0M p -'# This instruction also describes the survey policy responsibilities and e plains how the survey program is conducted# This chapter will highlight some of the important points covered in A"3 0/=+/,-, but Air "orce personnel who plan to conduct official surveys within the Air "orce should become familiar with the entire instruction# The instruction designates the *ilitary &ersonnel Survey 8ranch, Air "orce *ilitary &ersonnel 7enter %A"*&7<!&*EAS', $andolph A"8, Te as 91-2,, as the controlling and approving agency for Air "orce military personnel surveys# Any member of the Air "orce wanting to conduct a survey covered by this instruction must submit a written re5uest through channels to A"*&7<!&*EAS for approval# JAny survey of Air "orce civilian personnel must conform to the Air "orce :abor $elations &rogram described in A"3 0/=9,-#K %A"3 0/=0/,-, paragraph 9#0'# Surveys going to non=!B! civilians %e#g#, dependents of military personnel, government contractors, general public, etc#' are a special concern# These must be approved through the Bffice of *anagement and 8udget %B*8', specifically the Bffice of the Administrative Assistant for the Secretary of the Air "orce, 3nformation *anagement &olicy !ivision %SA"<AA3A', -/,, Air "orce &entagon, Cashington, !7 +,00,=-/,,# The types of surveys re5uiring approval are defined in detail in paragraphs + and 0 of A"3 0/=+/,-# 3n general, surveys covered by this instruction must be personnel surveys and not occupational surveys# The latter type, referred to as a 4ob inventory, is used to identify the duties and tasks that comprise an Air "orce career field# 3t is especially important to note that this instruction does not apply to surveys that concern aspects of base activities that the base commander is authorized to change %e#g#, surveys of base conditions, commissary or hospital services, etc#'# 3n such cases, the survey does not re5uire approval under A"3 0/= +/,- %paragraph 9#1'# These are the ma4or e clusions, but paragraphs 9#1 and 9#. identify other less common e clusions# &aragraph 9#+ of A"3 0/=+/,- specifies in detail the information that must accompany the re5uest for approval to conduct a survey# Eou should be sure that) N N N N available information is inade5uate to satisfy your needs currently programmed surveys cannot produce the re5uired information the need for the data 4ustifies the cost to obtain it a survey will produce the best data with the minimum inconvenience to the respondents#

The Air "orce &ersonnel Survey &rogram specifies that all data collected must be treated as privileged information and that respondents will in no way suffer adverse actions as a result of their participation %or non=participation'# The

introductory paragraphs of A"3 0/=+/,- specify that all surveys sub4ect to the provisions of A"3 09=-0+ %Air "orce &rivacy Act &rogram' must contain a &rivacy Act Statement# This re5uires all respondents be advised of) N the "ederal statute or e ecutive order that authorizes the solicitation of the information N the principal purpose%s' for which the data are to be used N the routine uses to be made of it N whether furnishing the information is mandatory or voluntary N the effects %if any' on the individual of not providing all or part of the re5uested information# "inally, A"3 0/=+/,-, paragraph ., specifies conditions under which release of survey results must be coordinated with H; A"*&7<!&*EAS# (very member of the Air "orce who administers a survey should be familiar with and follow the guidelines established by this instruction and appropriate command<unit operating and implementing instructions#

!eveloping the &urpose, Hypotheses, F Survey &lan

The first steps in producing a survey are the most important# They determine where you are going %the purpose', how you will know when your are there = or what you e pect to find %the hypotheses, ob4ectives, or research 5uestions', and by what route you will go %the survey plan'# 3f these steps are not well planned, all the remaining steps will be wasted effort# TH( &U$&BS( The first step in producing a survey is to define the purpose or ob4ective of the survey# JA clear statement of purpose is necessary not only as a 4ustification<e planation of the pro4ect, but also as a guideline to determine if future actions in the pro4ect are in support of the original purposeK A Guide for !evelopment###, -.9>M p +'# Cithout knowledge of the e act nature of the problem %ob4ective', you cannot decide e actly what kind of data to collect or what to do with it once you have it# Usually a staff officer is given a problem or ob4ectiveM it seldom originates with him# 8ut this does not relieve the individual of responsibility for insuring that) N the problem is well stated

N the surveyor understands e actly what the problem is N the stated problem is the real problem The survey should be designed to answer only the stated problem# Adding additional interesting ob4ectives will lengthen and complicate the survey while clouding the real issue# TH( HE&BTH(S3S, B8G(7T3D(, B$ $(S(A$7H ;U(ST3B? Bnce the problem has been clearly stated, the ne t step is to form one or more hypotheses# The hypothesis is actually your educated guess about the answer to the problem# 3t should not be a capricious guess, however# 3t ought to be based on your prior e perience related to the problem, or perhaps any knowledge you may have of previous research done on the topic# Cithout such a framework in which to make an educated guess, you really have no basis for making a guess at all# 3f you do not have a clear basis for formulating an hypothesis, you should instead develop one or more ob4ectives or 5uestions to frame the scope of your 5uestionnaire# "or e ample, if a problem is identified on the base as declining use of the BfficersA 7lub, an immediately obvious 5uestion comes to mind) JAre the officers on this base satisfied with the BfficersA 7lub facilitiesOK This would be suitable as a research 5uestion# 3t is possible, though doubtful, if you could come up with a supportable hypothesis, or educated guess, as to the answer to the problem# Eou may, for instance, have gathered some anecdotal evidence %overhearing colleagues talking' of dissatisfaction with the clubAs facilities# 8ut, this may not be sufficient for making an educated guess that this is the real reason for the decline in club use# The problem could be seasonalM it could be related to a decline in the officer population on the baseM or a number of other possibilities# The point is that without some credible evidence to support an hypothesis, you should probably not formulate one# 3f you formulated an hypothesis for the current e ample on the basis of the anecdotal evidence available to you, you would naturally construct a 5uestionnaire to survey the opinions of officers regarding their use, or lack thereof, of the BfficersA 7lub and the reasons for it# Eou might never think to gather data from the base military personnel office to see if the officer population is lower now than usual or if there are seasonal %cyclic' trends in the size of the officer population on the base# 3n other words, establishing the hypothesis may blind you to collecting data on other possible causes of the problem# This is why all researchers are cautioned not to formulate hypotheses unless they have a solid base in theory or previously gathered evidence that suggests the hypothesis is, in fact, probable# Hypotheses must be carefully written# They should not contain moral 4udgments or biased statements such as JAll pilots are good leaders#K There are many ideas on what constitutes a good leader and your idea may not be the same as those of the people you will contact# Avoid words like should,

best, good, bad, and ought# Hypotheses should be as specific as possible# Avoid words such as most and some# 3f by most you mean a ma4ority, then say ma4ority# A survey can more easily be designed to test whether Jmore than 92 percent approveK than whether Jmost approve#K A well=formulated hypothesis, ob4ective, or research 5uestion translates the purpose into a statement that can be investigated scientifically# The level of difficulty you will face in producing a valid survey will increase dramatically if they are not well formulated# Take care in doing this step, and it will save you much effort later in the survey development process# TH( SU$D(E &:A? The ne t step after determining the purpose and hypotheses is constructing the survey plan# The purpose of the survey plan is to ensure that the survey results will provide sufficient data to provide an answer %solution' to the problem you are investigating# The survey plan is comprised of three different parts) N data collection plan N data reduction and reformatting plan N analysis plan ?one of these plans stands on its own# !ecisions you make on how you will analyze your data will affect your data collection plan# The type of data reduction you do will affect not only the types of analyses you can do, but also the amount and types of data you need to collect# 8ecause these plans are closely interrelated, they should be developed concurrently# TH( !ATA 7B::(7T3B? &:A? The purpose of the data collection plan is to ensure that proper data are collected in the right amounts# Eour hypothesis and your data analysis plan determine the appropriateness of the data# "or e ample, if you plan to analyze your results by age group to test a hypothesis, then you must collect data from each age group whose opinions you want to know# The right amount applies to sample data# As pointed out earlier, the use of sample data involves risk, and the amount of that risk is determined by the size of your sample# The amount of risk you are willing or able to accept should be stated in your analysis plan# &roper and right come together when your analysis plan involves both sampling and analyzing data by groups# Eou not only have to collect data from some members of each group you plan to analyze, but you also have to see that each group provides a response rate

that is high enough to ensure your meeting your minimum risk level# The concept of the proper sample size is covered in greater detail in 7hapter ># TH( !ATA $(!U7T3B? A?! $("B$*ATT3?G &:A? The purpose of the data reduction and reformatting plan is to identify up front and to decrease as much as possible the amount of data handling %reduction and reformatting' you will have to do# This plan is highly dependent on the other two plans# As previously mentioned, if your collection plan calls for a great deal of data, you should plan to use a computer to analyze the data# 3f A!& scanner sheets are to be used to record respondentsA answers, include the sheets with the 5uestionnaire so the respondent can fill out the scanner sheet# This will save a great deal of time that you would have to spend if you transferred the survey data to the scanner sheets yourself# 3t also eliminates the possibility of your making errors in transferring data# Eou should coordinate in advance with the A!& personnel to make sure they will be able to scan your answer sheets and, if necessary, analyze your data within your timeframe# A!& shops are busy places# The prudent surveyor will JbookK the scanning and analysis 4obs well in advance with A!& personnel to ensure their resources are available when needed# A strong potential for error and tedious corrective work lies in data reduction and reformatting# &roper care in developing this plan can save a great deal of time later and preclude error# B&(?= A?! 7:BS(!=(?!(! ;U(ST3B?S The use of Automatic !ata &rocessing %A!&' necessitates the use of closed=end 5uestions == a type of 5uestion you should consider even if you are hand=tabulating your data# A closed=end 5uestion lists possible answers from which the respondent picks the one he<she likes best# An e ample is the common multiple=choice 5uestion# The open=end 5uestion is one to which the respondents write the answer out in their own words# At first glance, the open=end 5uestion seems superior since respondents supply their answers rather than ones from your list of answers# 8ut the wide variety of answers respondents generally provide to open=end 5uestions turns out to be a great handicap later# "or every open=end 5uestion, there are virtually an infinite number of possible answers# Since you cannot analyze an infinite number of answers, you must devise some means of categorizing this diversity of answers into a smaller, more manageable group# Eou will find yourself spending a tremendous amount of time reading, comparing, categorizing, and recording each answer# *uch of this time can be saved if you use care in developing the 5uestionnaire and constructing your own categories in advance# 7onstruct each 5uestion so that every possible ma4or category of response is contained in the answer

list# Then, later, all the computer will have to do is count the number of answers in each category# 8y having the survey respondent, not you, categorize the answer, you will collect data that is more valid, reliably, and accurate than if you did the categorizing yourself# Additional information on closed=end 5uestions is provided in 7hapter 2# TH( A?A:ES3S &:A? "inally, an analysis plan ensures that the information produced by the analysis will ade5uately address the originally stated hypotheses, ob4ectives, or 5uestions# 3t also ensures an analysis that is compatible with the data collected during the survey# 3n the analysis plan, you determine which statistics you will use and how much risk you can take in stating your conclusions# (ach of these decisions will affect the amount and type of data you collect and how you will reduce it# ?ovice researchers often misuse statistical analyses out of ignorance of the assumptions on which the statistics are based# The most often committed error in statistical analysis by novices is using a statistical techni5ue with inappropriate data# The results of such analyses appear to be legitimate, but are actually impossible to interpret correctly# Ce will discuss some of these common errors and how to avoid them in 7hapter /# 7B?7:U!3?G THBUGHTS Bppenheim %-.//' suggests that to make sure all these parts of the survey plan are correctly interlocked, you can simply approach the natural se5uence of survey operations in reverse order# "irst determine what conclusions you are interested inM then decide what statistics and results will be needed to draw these conclusions# "rom this, the type of 5uestions needed and the nature of the sample can be determined# A conscientious survey plan will help you produce a well=designed survey# The proper data will be processed correctly and efficiently to produce the information re5uired to shed light on, and hopefully provide a solution to, the original problem#

Sampling Techni5ues and $elated Statistical 7oncepts

7hapter - identified some of the problems associated with sampling) N acceptance of a risk of error N choice of a representative sample N determination of the size of the sample This chapter outlines procedures for dealing with these challenges# "irst, different techni5ues designed to produce a representative sample from different types of populations are e plained# ?e t, the relationship between risk and sample size is investigated# "inally, techni5ues are discussed for 5uantifying the amount of risk present in your results and for determining the sample size necessary to achieve the confidence and reliability specified in your analysis plan# SA*&:3?G *(THB!S Eour overarching goal in doing a survey is to determine what some group thinks or feels about some issue# 3f money, time, or other resources were not a concern, the most accurate data you could get would come from surveying the entire population of interest# Since limited resources are a reality we all have to deal with, however, we are often forced to survey the views of only a few members of the population# 8ut never lose sight of the fact that the real purpose is to discover the views of the entire population# Bbviously, then, we want to be able to say with as much confidence as possible that the views of the group we surveyed represents the views of the entire population# Using a combination of powerful statistical tools, known as inferential statistics, and unbiased sampling techni5ues, any surveyor can collect data that actually represent the views of the entire population from which the sample was taken# Two things are absolutely necessary, however, to ensure a high level of confidence that the sample represents the population) N an unbiased sample N a sufficiently large sample 8ias as a statistical term means error# To say that you want an unbiased sample may sound like youAre trying to get a sample that is error=free# As appealing as this notion may be, it is impossible to achieveP (rror always occurs == even when using the most unbiased sampling techni5ues# Bne source of error is caused by the act of sampling itself# To understand it, consider the following e ample# :etAs say you have a bowl containing ten slips of paper# Bn each slip is printed a number, one through ten# This is your Jpopulation#K ?ow you are going to select a sample# Ce will use a random method for drawing the sample, which can be done easily by closing your eyes and reaching into the bowl and choosing one slip of paper# After choosing it, check the number on it and place it in the

sample pile# ?ow to determine if the sample is representative of the population, we must know what attribute%s' we wish to make representative# Since there are an infinite number of human attributes, we must precisely determine the one%s' we are interested in before choosing the sample# 3n our e ample, the attribute of interest will be the average numerical value on the slips of paper# Since the JpopulationK contained ten slips numbered consecutively from one to ten, the average numerical value in the population is) -+ 0>2/91.-, QQQQQQQQQ R22# -, As you can see, no matter what slip of paper we draw as our first sample selection, itAs value will be either lower or higher than the population average# :etAs say the slip we choose first has a . on it# The difference between our sample %.' and the population %2#2' averages is Q0#2 %plus signifies the sample average is larger than the population average'# The difference between the sample average and the population average is known as sampling error# That is, the sample mean %average' plus %or minus' the total amount of sampling error e5uals the population mean# Bn our second pick, we choose a slip that has a - on it# ?ow the average of sample values is) .Q R 2,# + The sampling error has shrunk from its previous value of Q 0#2 to its new value of = ,#2 %minus signifies the sample mean is now smaller than the population mean'# (ach time we choose a slip from the population to include in the sample, one of three mutually e clusive things can occur == the sample mean will become) N larger than the population mean N smaller than the population mean N e5ual to the population mean Bn average, each sampling brings the sample mean a bit closer to the population mean# Ultimately, if we sampled everyone from the population, the

sample mean and the population mean would be e5ual# This is why a complete census is completely accurate = there is no sampling error# Eet, if we are forced to use only a sample from the population, the larger the sample the less sampling error we will have, generally speaking# (5ually important to the size of the sample is the determination of the type of sampling to be done# 3n our e ample, we randomly %blindly' chose from the population# $andom sampling always produces the smallest possible sampling error# 3n a very real sense, the size of the sampling error in a random sample is affected only by random chance# The two most useful random sampling techni5ues are simple random and stratified random sampling methods# These will discussed shortly# 8ecause a random sample contains the least amount of sampling error, we may say that it is an unbiased sample# ?ote that we are not saying the sample contains no error, but rather the minimum possible amount of error# ?onrandom sampling techni5ues also e ist, and are used more fre5uently than you might imagine# As you can probably guess from our previous discussion, nonrandom sampling techni5ues will always produce larger sampling errors %for the same sample size' than random techni5ues# The reason for this is that nonrandom techni5ues generate the e pected random sampling error on each selection plus additional error related to the nonrandom nature of the selection process# To e plain this, letAs e tend our sampling e ample from above# :etAs say we want to sample from a JpopulationK of -,,, consecutively numbered slips of paper# 8ecause numbering these slips is time consuming, we have -, people each number -,, slips and place all -,, of them into our bowl when they finish# :etAs also say that the last person to finish has slips numbered from .,- to -,,,, and these are laid on top of all the other slips in the bowl# ?ow we are ready to select them# 3f we wanted to make this a truly random sampling process, we would have to mi the slips in the bowl thoroughly before selecting# "urthermore, we would want to reach into the bowl to different depths on subse5uent picks to make sure every slip had a fair chance of being picked# 8ut, let us say in this e ample that we forget to mi the slips in the bowl# :etAs also say we only pick from the top layer of slips# 3t should be obvious what will occur# 8ecause the top layer of slips is numbered .,- through -,,,, the mean of any sample %of -,, or less' we select will hover around .2,#2 %the true mean of the numbers .,- through -,,,'# 7learly, this is not even close to the true population mean %2,,#2 == the mean of the numbers from - to -,,,'# Sampling error amounts to the difference between the true population mean and the sample mean# 3n this e ample, the sampling error can as large as >2, %.2,#2 = 2,,#2'# This was a simple, and somewhat absurd, e ample of nonrandom sampling#

8ut, it makes the point# ?onrandom sampling methods usually do not produce samples that are representative of the general population from which they are drawn# The greatest error occurs when the surveyor attempts to generalize the results of the survey obtained from the sample to the entire population# Such an error is insidious because it is not at all obvious from merely looking at the data, or even from looking at the sample# The easiest way to recognize whether a sample is representative or not is to determine if the sample was selected randomly# To be a random sampling method, two conditions must be met# 3f both are met, the resulting sample is random# 3f not, it is a nonrandom sampling techni5ue) N every member in the population must have an e5ual opportunity of being selected, N the selection of any member of the population must have no influence on the selection of any other member All nonrandom sampling methods violate one or both of these criteria# The most commonly used nonrandom methods are) N systematic sampling %selecting every nth person from a group' N cluster sampling %selecting groups of members rather than single members' N convenience or incidental sampling %selecting only readily available members' N 4udgment or purposive sampling %selecting members who are 4udged to be appropriate for the study' S3*&:( $A?!B* SA*&:3?G A simple random sample is one in which each member %person' in the total population has an e5ual chance of being picked for the sample# 3n addition, the selection of one member should in no way influence the selection of another# Simple random sampling should be used with a homogeneous population, that is, one composed of members who all possess the same attribute you are interested in measuring# 3n identifying the population to be surveyed, homogeneity can be determined by asking the 5uestion, JChat is %are' the common characteristic%s' that are of interestOK These may include such characteristics as age, se , rank<grade, position, income, religious or political affiliation, etc# == whatever you are interested in measuring# The best way to choose a simple random sample is to use a random number table %or let a computer generate a series of random numbers automatically'# 3n either case, you would assign each member of the population a uni5ue number %or perhaps use a number already assigned to them such as SSA?, telephone number, zip code, etc#'# The members of the population chosen for the sample will be those whose numbers are identical to the ones e tracted from the random number table %or computer' in succession until the desired sample size is reached# An e ample of a random number table and instructions for its use appear in Appendi !#

*any statistical te ts or mathematical tables treat random number generation# A less rigorous procedure for determining randomness is to write the name of each member of the population on a separate card, and with continuous mi ing, draw out cards until the sample size is reached# The simple random sample re5uires less knowledge about the population than other techni5ues, but it does have two ma4or drawbacks# Bne is if the population is large, a great deal of time must be spent listing and numbering the members# The other is the fact that a simple random sample will not ade5uately represent many population attributes %characteristics' unless the sample is relatively large# That is, if you are interested if choosing a sample to be representative of a population on the basis of the distribution in the population of gender, age, and economic status, a simple random sample will need to be very large to ensure all these distributions are e5uivalent to %or representative of' the population# To obtain a representative sample across multiple population attributes, you should use the techni5ue of stratified random sampling# Ce made this point earlier in this chapter, but itAs such an important concept that it bears repeating# To determine if the sampling method you use is random or not, remember that true random sampling methods must meet two criteria) N (very member in the population must have an e5ual opportunity of being chosen for the sample %e5uality' N The selection of one member is not affected by the selection of previous members %independence' 8oth simple random and stratified random sampling methods meet these two criteria# ?onrandom sampling methods lack one or both of these criteria# Ce discuss stratified random sampling ne t# ST$AT3"3(! $A?!B* SA*&:3?G This method is used when the population is heterogeneous rather than homogeneous %or as discussed above, when you want to obtain a representative sample across many population attributes'# A heterogeneous population is composed of unlike elementsM such as, officers of different ranks, civilians and military personnel, or the patrons of a discount store %differing by gender or age'# A stratified random sample is defined as a combination of independent samples selected in proper proportions from homogeneous groups within a heterogeneous population# The procedure calls for categorizing the heterogeneous population into groups that are homogeneous in themselves# 3f one group is proportionally larger than another, its sample size should also be proportionally larger# The number of groups to be considered is determined by the characteristics of the

population# *any times the survey plan will determine some or all of the groups# "or e ample, if you are comparing enlisted and officer segments on your base, each of these will be a separate group# After dividing the population into groups, you then sample each homogeneous group# !ifferent sampling techni5ues can be used in each of the different groups, but keep in mind that random techni5ues produce the minimum amount of sampling error# "inally, you should calculate the sample statistics for each group to determine how many members you need from each subgroup# Ce will discuss the calculations involved in determining the size of your sample later in this chapter# These calculations are designed to determine the size of a simple random sample# Since the stratified sampling techni5ue re5uires you to create simple, homogeneous subgroups from a large heterogeneous group, think of the calculations for a stratified sample as a series of simple random sample size calculations for each homogeneous subgroup# The only other information you must know is the proportion of the population possessing the attribute contained in each homogeneous subgroup# "or e ample, letAs say we want to draw a random sample from a population of military personnel to assess their opinions on some issue# 3n addition, we would like to determine if the opinions differ by officer=enlisted affiliation and gender of the individuals surveyed# Ce recognize that the population we want to draw our sample from is heterogeneous with respect to the two attributes of interest to us# So, we have to create homogeneous subgroups %four to be e act') N N N N (nlisted, male (nlisted, female Bfficer, male Bfficer, female

?ow, each group is homogeneous on both attributes# To ensure each subgroup in the sample will represent its counterpart subgroup in the population, we must ensure each subgroup is represented in the sample in the same proportion to the other subgroups as they are in the population# :etAs assume that we know %or can estimate' the population of Air "orce military personnel to be distributed as follows) 9, percent male, 0, percent female and /2 percent enlisted, 02 percent officer# Cith that, we can determine the appro imate proportions of our four homogeneous subgroups in the population) N N N N (nlisted, male #/2 #9, R #>22 (nlisted, female #/2 #0, R #-.2 Bfficer, male #02 #9, R #+>2 Bfficer, female #02 #0, R #-,2 Thus, a representative sample of the Air "orce population %by race and enlisted=officer affiliation' would be composed of >2#2 percent enlisted males, -.#2 percent enlisted females, +>#2 percent officer males, and -,#2

percent officer females# (ach percentage should be multiplied by the total sample size needed to arrive at that actual number of personnel re5uired from each subgroup or stratum# As this e ample illustrates, stratified random sampling re5uires a detailed knowledge of the distribution of attributes or characteristics of interest in the population to determine the homogeneous groups that lie within it# A stratified random sample is superior to a simple random sample since the population is divided into smaller homogeneous groups before sampling, and this yields less variation within the sample# This makes possible the desired degree of accuracy with a smaller sample size# 8ut, if you cannot accurately identify the homogeneous groups, you are better off using the simple random sample since improper stratification can lead to serious error# SEST(*AT37 SA*&:3?G Sometimes it is more e peditious to collect a sample of survey participants systematically# This is fre5uently done, for instance, in e it polling of voters or store customers# 3t is a nonrandom sampling techni5ue, but is used primarily for its ease and speed of identifying participants# To use the systematic approach, simply choose every 6 th member in the population where 6 is e5ual to the population size divided by the re5uired sample size# 3f this 5uotient has a remainder, ignore it %round down'# "or e ample, if you need -,, members in your sample and the population consists of -,,, people, you need to sample every -,,,<-,, %or -,th' member of the population# Chen using this method, some suggest you should choose your starting point at random by choosing a random number from - to 6# 3f you recall the characteristic re5uirements for a random sample discussed above %e5uality and independence', you can see that systematic sampling methods lack both characteristics# (very member from the population does not have a e5ual chance of being selected, and the selection of members for the sample depends on the initial selection# $egardless of how you select your starting point, once selected, every subse5uent member of the sample is automatically determined# This method is clearly nonrandom# Some suggest that by mi ing the population well you can turn this into a random sampling techni5ue# They are wrong# $egardless of how much you mi the population before selecting a starting point, the fact remains that once that point is chosen, further selection of members for the sample is nonrandom %no independence'# $ecognize the limitation of this type of sampling# Since it is

nonrandom, the resulting sample will not necessarily be representative of the population from which it was drawn# This will affect your ability to confidently generalize results of the survey since you may not be sure to which segment of the population the results will apply# As a word of advice, unless you have e perience in systematic sampling techni5ues, and have full knowledge of the population to be sampled, you should avoid using this method# GU!G*(?T SA*&:3?G Gudgment sampling involves asking an e pert on an issue being investigated to define the members that should comprise the sample# The representativeness of the sample is determined solely by the 4udgment of the researcher# Since each member in the population does not have an e5ual chance of being chosen, a 4udgment sample is also a nonrandom sampling method# Since the sample does not meet the criterion of randomness = the basis for many statistical sampling applications % a 4udgment sample should never be used in a statistical evaluation effort# &U$&BS3D( SA*&:3?G As the name implies, purposive sampling involves selecting members from the population to comprise a sample because they specifically meet some prescribed purpose of possess specific attributes of interest that address the purpose of a particular research problem under investigation# &urposive sampling is used primarily in causal=comparative %e post facto' research where the researcher is interested in finding a possible cause=and= effect link between two variables, one of which has already occurred# The researcher intentionally selects the samples in such a way that one possesses the causal %independent' variable and one does not# The purpose of the research governs the selection of the sample and, thus, e cludes members of the population who do not contribute to that purpose# $(&$(S(?TAT3D( SA*&:3?G The types of sampling methods discussed above are only a few of the many available# Eou will find others in the references listed in the bibliography# (ach type is designed to obtain the most representative sample possible from different kinds of populations# 8efore using any sampling method yourself, first think about the population to which you want to generalize the results of your survey %which population do you want to represent'# 3f generalizing results is not your aim, any sampling method will do# 3f generalizing results is important, use a sampling method that will ensure your sample is representative of the population from which you draw it# $andom sampling methods typically ensure a high degree of confidence that the results do, in fact, represent those of the whole population#

"A7TB$S 3?":U(?73?G SA*&:( S3I( As pointed out in 7hapter -, when you sample you are dealing with only partial information# And you must accept a risk of being wrong when inferring something about the population on the basis of sample information# 3n the analysis portion of your survey plan, you identify the amount of risk you are willing %or allowed' to take# This amount of risk relates directly to the size of your sample# Simply stated, the less risk you are willing to take, the larger your sample must be# 3f you cannot accept any risk, you should survey the entire population %take a census' and you need not study this chapter any further# Chen determining your risk level, keep in mind the time and cost involved in obtaining the sample size sufficient to achieve the risk level you can accept# Eou may find it impossible to produce a sample large enough to meet that risk level# Another factor bearing on sample size is also obtained from your analysis plan# 3t is the number of groups you are planning to e amine within the population# "or e ample, if you are planning to compare two groups %enlisted and officer' on a base %your population', each of the groups must be sampled and each of the samples must be large enough to ensure satisfying your risk level# 7B?"3!(?7( :(D(: A?! &$(73S3B? $isk, as it relates to sample size determination, is specified by two interrelated factors) N the confidence level N the precision %or reliability' range# To minimize risk, you should have a high confidence %say .2 percent' that the true value you seek %the actual value in the population' lies somewhere within a small interval %say Q or = 2 percent' around your sample value %your precision'# Sawyer %-.9-M p >.' uses a baseball game analogy to e plain confidence level, precision range, and their relationship# A baseball pitcher may feel that he can get very few of his pitches %perhaps -, percent' over the e act center %small precision range' of home plate# 8ut since home plate is -9 inches wide, he may feel that he can get .2 percent of his pitches over the center of the plate with a precision of plus or minus 1 -<+ inches %a .2 percent confidence level'# 3f the plate is widened to 0, inches, he may feel .. percent confident# So when we widen the range of precision %or reliability', we increase our confidence level# :ikewise, if we reduce the range, we reduce our confidence level# *ost surveying organizations use a .2 percent confidence level and a S 2 percent precision level as the absolute minimum#

!(T($*3?3?G TH( S3I( B" TH( SA*&:( Bnce you determine your desired degree of precision and your confidence level, there are several formulas you can use to determine sample size depending on how you plan to report the results of your study# CeAll discuss three of them here# 3f you will be reporting results as percentages %proportions' of the sample responding, use the following formula)

3f you will report results as means %averages' of the sample responding, use the following formula)

3f you plan to report results in a variety of ways, or if you have difficulty estimating percentage or standard deviation of the attribute of interest, the following formula may be more suitable for use)

Ce illustrate this formula with the following e ample# 3f the total population %?' is -,,,,,, and you wish a .2T confidence level and S 2 percent precision level %d R #,2, I R -#./ from Appendi (', then)

So, a representative sample of 09, %0/.#.1 rounded up' would be sufficient to satisfy your risk level# 3nspection of the formula shows that the re5uired sample size will increase most rapidly if) N the confidence level %I factor' is increased, or N the precision level %d' is made smaller# 3f you have stratified your population into more than one group, the size of

each group will be its proportion %percentage' in the population times the total sample size as computed above# To illustrate, recall our earlier e ample of four stratified groups# Using the n of 09, calculated above, each of these strata should have the following sample sizes) N N N N (nlisted, male 09, #>22 R -/1#02 R -/1 (nlisted, female 09, #-.2 R 9+#-2 R 9+ Bfficer, male 09, #+>2 R .,#/2 R .Bfficer, female 09, #-,2 R 01#12 R 0. "inally, you should ad4ust the computed sample size %n' by dividing n by the e pected response rate# "or instance, if you e pect 92 n percent response rate, you should make your sample size e5ual # 3f ,92 # you canAt anticipate a response rate, assume a 2, percent response rate %i#e#, double the n value'# This sort of ad4ustment should ensure you get a sufficient number of responses regardless of return rate#

The ;uestionnaire
The final step in preparing the survey is developing the data collection instrument# The most common means of collecting data are the interview and the self= or group=administered 5uestionnaire# 3n the past, the interview has been the most popular data=collecting instrument# $ecently, the 5uestionnaire has surpassed the interview in popularity, especially in the military# !ue to this popularity, this chapter concentrates on the development of the 5uestionnaire# TH( ;U(ST3B??A3$( &$BS A?! 7B?S 3t is important to understand the advantages and disadvantages of the 5uestionnaire as opposed to the personal interview# This knowledge will allow you to ma imize the strengths of the 5uestionnaire while minimizing its weaknesses# The advantages of administering a 5uestionnaire instead of conducting an interview are) N lower costs N better samples N standardization

N respondent privacy %anonymity' The primary advantage is lower cost, in time as well as money# ?ot having to train interviewers eliminates a lengthy and e pensive re5uirement of interviewing# The 5uestionnaire can be administered simultaneously to large groups whereas an interview re5uires each individual to be 5uestioned separately# This allows the 5uestions to reach a given number of respondents more efficiently than is possible with the interview# "inally, the cost of postage should be less than that of travel or telephone e penses# $ecent developments in the science of surveying have led to incorporating computers into the interview process, yielding what is commonly known as computer automated telephone interview %or 7AT3' surveys# Advances in using this survey techni5ue have dramatically reshaped our traditional views on the time=intensive nature and inherent unreliability of the interview techni5ue# Eet, despite resurgence in the viability of survey interviews, instruction in the development and use of the 7AT3 techni5ue is well beyond the scope of this handbook# *any surveys are constrained by a limited budget# Since a typical 5uestionnaire usually has a lower cost per respondent, it can reach more people within a given budget %or time' limit# This can enhance the conduct of a larger and more representative sample# The 5uestionnaire provides a standardized data=gathering procedure# Using a well=constructed 5uestionnaire can minimize the effects of potential human errors %for e ample, altering the pattern of 5uestion asking, calling at inconvenient times, and biasing by Je plainingK# The use of a 5uestionnaire also eliminates any bias introduced by the feelings of the respondents towards the interviewer %or vice versa'# Although the point is debatable, most surveyors believe the respondent will answer a 5uestionnaire more frankly than he would answer an interviewer, because of a greater feeling of anonymity# The respondent has no one to impress with his<her answers and need have no fear of anyone hearing them# To ma imize this feeling of privacy, it is important to guard, and emphasize, the respondentAs anonymity# The primary disadvantages of the 5uestionnaire are nonreturns, misinterpretation, and validity problems# ?onreturns are 5uestionnaires or individual 5uestions that are not answered by the people to whom they were sent# Bppenheim %-.//' emphasizes that Jthe important point about these low response rates is not the reduced size of the sample, which could easily be overcome by sending out more 5uestionnaires, but the possibility of bias# ?onresponse is not a random processM it has its own determinants, which vary from survey to surveyK %p 0>'# "or e ample, you may be surveying to determine the attitude of a group about a new policy# Some of those opposed to it might be afraid to speak out, and

they might comprise the ma4ority of the nonreturns# This would introduce non= random %or systematic' bias into your survey results, especially if you found only a small number of the returns were in favor of the policy# ?onreturns cannot be overcome entirely# Chat we can do is try to minimize them# Techni5ues to accomplish this are covered later in this chapter# *isinterpretation occurs when the respondent does not understand either the survey instructions or the survey 5uestions# 3f respondents become confused, they will either give up on the survey %becoming a nonreturn' or answer 5uestions in terms of the way they understand it, but not necessarily the way you meant it# Some view the latter problem as a more dangerous occurrence than merely nonresponding# The 5uestionnaire instructions and 5uestions must be able to stand on their own and must use terms that have commonly understood meanings throughout the population under study# 3f novel terms must be used, be sure to define them so all respondents understand your meaning# The third disadvantage of using a 5uestionnaire is inability to check on the validity of the answer# !id the person you wanted to survey give the 5uestionnaire to a friend or complete it personallyO !id the individual respond indiscriminatelyO !id the respondent deliberately choose answers to mislead the surveyorO Cithout observing the respondentAs reactions %as would be the case with an interview' while completing the 5uestionnaire, you have no way of knowing the true answers to these 5uestions# The secret in preparing a survey 5uestionnaire is to take advantage of the strengths of 5uestionnaires %lower costs, more representative samples, standardization, and privacy' while minimizing the number of nonreturns, misinterpretations, and validity problems# This is not always as easy as it sounds# 8ut an inventive surveyor can very often find legitimate ways of overcoming the disadvantages# Ce provide some suggestions below to help# TH( 7B?T(?TS The key to minimizing the disadvantages of the survey 5uestionnaire lies in the construction of the 5uestionnaire itself# A poorly developed 5uestionnaire contains the seeds of its own destruction# (ach of the three portions of the 5uestionnaire = the cover letter, the instructions, and the 5uestions = must work together to have a positive impact on the success of the survey# The cover letter should e plain to the respondent the purpose of the survey and motivate him to reply truthfully and 5uickly# 3f possible, it should e plain why the survey is important to him, how he was chosen to participate, and who is sponsoring the survey %the higher the level of sponsorship the better'# Also the confidentiality of the results should be strongly stressed# A well written cover letter can help minimize both nonreturn and validity problems# An e ample is

given in Appendi "# 3n support of the statement above regarding level of sponsorship, the signature block on the letter should be as high level as you can get commensurate with the topic being investigated# "or instance, a survey about Air "orce medical issues or policy should be signed by the Air "orce Surgeon General or higher, a survey on religious issues by the Air "orce 7hief of 7haplains, etc# Another tip that seems to help improve response rate is to identify the survey as official# (ven though the letter is on government stationery and is signed by an military official, it may help to mark the survey itself with an B""373A: stamp of some sort# 3n general, the more official the survey appears, the less likely it is to be disregarded# The cover letter should be followed by a clear set of instructions e plaining how to complete the survey and where to return it# 3f the respondents do not understand the mechanical procedures necessary to respond to the 5uestions, their answers will be meaningless# The instructions substitute for your presence, so you must anticipate any 5uestions or problems that may arise and attempt to prevent them from occurring# 3f you are using A!& scanner sheets, e plain how you want the respondent to fill it in = what portions to use and what portions to leave blank# $emember anonymityP 3f you do not want respondents to provide their names or SSA?s, say so e plicitly in the instructions, and tell them to leave the ?A*( and SSA? portions of the scan sheets blank# 3f you need respondentsA SSA? and<or name included on the survey for tracking or analysis purposes, you will need to put a &rivacy Act Statement somewhere on the survey %refer to 7hapter +'# The L3nstructionsL page is usually a good place for this statement# 3t places it in a prominent place where all respondents will see it, but does not clutter the instrument itself or the cover letter# The third and final part of the 5uestionnaire is the set of 5uestions# Since the 5uestions are the means by which you are going to collect your data, they should be consistent with your survey plan# They should not be ambiguous or encourage feelings of frustration or anger that will lead to nonreturns or validity problems# TE&(S B" ;U(ST3B?S 8efore investigating the art of 5uestion writing, it will be useful to e amine the various types of 5uestions# 7antelou %-./>M p 29' identifies four types of 5uestions used in surveying# The classifier or background 5uestion is used to obtain demographic characteristics of the group being studied, such as age, se , grade, level of assignment, and so forth# This information is used when you are categorizing your results by various subdivisions such as age or grade# Therefore, these 5uestions should be consistent with your data analysis plan# The second and most common type of 5uestion is the multiple choice or closed=end 5uestion# 3t is used to determine feelings or opinions on certain issues by allowing the respondent to choose an answer from a list you have provided %see 7hapter 0'# The intensity

5uestion, a special form of the multiple=choice 5uestion, is used to measure the intensity of the respondentAs feelings on a sub4ect# These 5uestions provide answers that cover a range of feelings# The intensity 5uestion is covered in greater detail later in this chapter# The final type of 5uestion is the free response or open=end 5uestion# This type re5uires respondents to answer the 5uestion in their own words %see 7hapter 0'# 3t can be used to gather opinions or to measure the intensity of feelings# *ultiple=choice 5uestions are the most fre5uently used types of 5uestions in surveying today# 3t is prudent, therefore, to concentrate primarily on factors relating to their application# ;U(ST3B??A3$( 7B?ST$U7T3B? *any researchers have investigated the comple art of 5uestion writing# "rom their e periences, they offer valuable advice# 8elow are some helpful hints typical of those that appear most often in te ts on 5uestion construction# - 6eep the language simple# Analyze your audience and write on their level# &arten %-.2,M p +,-' suggests that writing at the si th=grade level may be appropriate# Avoid the use of technical terms or 4argon# An appropriate corollary to *urphyAs :aw in this case would be) 3f someone can misunderstand something, they will# + 6eep the 5uestions short# :ong 5uestions tend to become ambiguous and confusing# A respondent, in trying to comprehend a long 5uestion, may leave out a clause and thus change the meaning of the 5uestion# 0 6eep the number of 5uestions to a minimum# There is no commonly agreed on ma imum number of 5uestions that should be asked, but research suggests higher return rates correlate highly with shorter surveys# Ask only 5uestions that will contribute to your survey# Apply the JSo whatOK and JCho caresOK tests to each 5uestion# J?ice=to=knowK 5uestions only add to the size of the 5uestionnaire# Having said this, keep in mind that you should not leave out 5uestions that would yield necessary data simply because it will shorten your survey# 3f the information is necessary, ask the 5uestion# Cith the availability of desktop publishing %!T&' software, it is often possible to give the perception of a smaller survey %using smaller point<pitch typefaces, etc#' even though many 5uestions are asked# A three=page type written survey can easily be reduced to a single page using !T& techni5ues# > :imit each 5uestion to one idea or concept# A 5uestion consisting of more than one idea may confuse the respondent and lead to a meaningless answer# 7onsider this 5uestion) JAre you in favor of raising pay and lowering benefitsOK Chat would a yes %or no' answer meanO 2 !o not ask leading 5uestions# These 5uestions are worded in a manner that suggests an answer# Some respondents may give the answer you are looking for whether or not they think it is right# Such 5uestions can alienate the respondent and may open your 5uestionnaire to criticism# A properly

worded 5uestion gives no clue as to which answer you may believe to be the correct one# Use sub4ective terms such as good, fair, and bad sparingly, if at all# These terms mean different things to different people# Bne personAs JfairK may be another personAs Jbad#K How much is JoftenK and how little is JseldomOK - 9# Allow for all possible answers# $espondents who cannot find their answer among your list will be forced to give an invalid reply or, possibly, become frustrated and refuse to complete the survey# Cording the 5uestion to reduce the number of possible answers is the first step# Avoid dichotomous %two=answer' 5uestions %e cept for obvious demographic 5uestions such as gender'# 3f you cannot avoid them, add a third option, such as no opinion, donAt know, or other# These may not + get the answers you need but they will minimize the number of invalid responses# A great number of JdonAt knowK answers to a 5uestion in a fact=finding survey can be a useful piece of information# 8ut a ma4ority of JotherK answers may mean you have a poor 5uestion, and perhaps should be cautious when analyzing the results# Avoid emotional or morally charged 5uestions# The respondent may feel your survey is getting a bit too personalP - .# Understand the should=would 5uestion# Selltiz, et al# %-./0, p +2-' note that respondents answer JshouldK 5uestions + from a social or moral point of view while answering JwouldK 5uestions in terms of personal preference# "ormulate your 5uestions and answers to obtain e act information and to minimize confusion# "or e ample, does JHow old are youOK mean on your last or your nearest birthdayO !oes JChat is your %military' gradeOK mean permanent or temporary gradeO As of what dateO 8y including instructions like JAnswer all 5uestions as of %a certain date'K, you can alleviate many such conflicts# %$efer to hint -0 below#' 3nclude a few 5uestions that can serve as checks on the accuracy and consistency of the answers as a whole# Have some 5uestions that are worded differently, but are soliciting the same information, in different parts of the 5uestionnaire# These 5uestions should be designed to identify the respondents who are 4ust marking answers randomly or who are trying to game the survey %giving answers they think you want to hear'# 3f you find a respondent who answers these 5uestions differently, you have reason to doubt the validity of their entire set of responses# "or this reason, you may decide to e clude their response sheet%s' from the analysis# Brganize the pattern of the 5uestions) &lace demographic 5uestions at the end of the 5uestionnaire# Have your opening 5uestions arouse interest# Ask easier 5uestions first# To minimize conditioning, have general 5uestions precede specific ones# Group similar 5uestions together# 3f you must use personal or emotional 5uestions, place them at the end of the 5uestionnaire#

-+# N N N N N N

- &retest %pilot test' the 5uestionnaire# This is the most important step in preparing your 5uestionnaire# The purpose of the pretest is to see 4ust how well your cover letter motivates your respondents and how clear your instructions, 5uestions, and answers are# Eou should choose a small group of people %from three to ten should be sufficient' you feel are representative of the group you plan to survey# After e plaining the purpose of the pretest, let them read and answer the 5uestions without interruption# Chen they are through, ask them to criti5ue the cover letter, instructions, and each of the 5uestions and answers# !onAt be satisfied with learning only what confused or alienated them# ;uestion them to make sure that what they thought something meant was really what you intended it to mean# Use the above -+ hints as a checklist, and go through them with your pilot test group to get their reactions on how well the 5uestionnaire satisfies these points# "inally, redo any parts of the 5uestionnaire that are weak# + Have your 5uestionnaire neatly produced on 5uality paper# A professional looking product will increase your return rate# As mentioned earlier, desktop publishing software can be used to add a very professional touch to your 5uestionnaire and improve the likelihood of its being completed# 8ut always remember the adage JEou canAt make a silk purse out of a sowAs ear#K A poorly designed survey that contains poorly written 5uestions will yield useless data regardless of how JprettyK it looks# -2# "inally, make your survey interestingP 3?T(?S3TE ;U(ST3B?S A?! TH( :36($T S7A:( As mentioned previously, the intensity 5uestion is used to measure the strength of a respondentAs feeling or attitude on a particular topic# Such 5uestions allow you to obtain more 5uantitative information about the survey sub4ect# 3nstead of a finding that 1, percent of the respondents favor a particular proposal or issue, you can obtain results that show 2 percent of them are strongly in favor whereas 92 percent are mildly in favor# These findings are similar, but the second type of response supplies more useful information# The most common and easily used intensity %or scaled' 5uestion involves the use of the :ikert=type answer scale# 3t allows the respondent to choose one of several %usually five' degrees of feeling about a statement from strong approval to strong disapproval# The J5uestionsK are in the form of statements that seem either definitely favorable or definitely unfavorable toward the matter under consideration# The answers are given scores %or weights' ranging from one to the number of available answers, with the highest weight going to the answer showing the most favorable attitude toward the sub4ect of the survey# The following 5uestions from the *innesota Survey of Bpinions designed to measure the amount of Janti=US

lawK feelings illustrate this procedure) - Almost anything can be fi ed up in the courts if you have + Bn the whole, 4udges are honest# enough money#

The weights %shown by the numbers below the answers' are not shown on the actual 5uestionnaire and, therefore, are not seen by the respondents# A person who feels that US laws are un4ust would score lower than one who feels that they are 4ust# The stronger the feeling, the higher %or lower' the score# The scoring is consistent with the attitude being measured# Chether JagreeK or JdisagreeK gets the higher weight actually makes no difference# 8ut for ease in interpreting the results of the 5uestionnaire, the weighting scheme should remain consistent throughout the survey# Bne procedure for constructing :ikert=type 5uestions is as follows %adapted from Selltiz, et al#, -./0M pp 0/9=0/1') - The investigator collects a large number of definitive statements relevant to the attitude being investigated# + 7onduct and score a pretest of your survey# The most favorable response to the attitude gets the highest score for each 5uestion# The respondentAs total score is the sum of the scores on all 5uestions# 0 3f you are investigating more than one attitude on your survey, intermi the 5uestions for each attitude# 3n this manner, the respondent will be less able to guess what you are doing and thus more likely to answer honestly# > $andomly select some 5uestions and flip=flop the Strongly Agree = =Strongly !isagree scale to prevent the respondent from getting into a pattern of answering %often called a response set'# The intensity 5uestion, with its scaled answers and average scores, can supply 5uantitative information about your respondentsA attitudes toward the sub4ect of your survey# The interested reader is encouraged to learn and use other scales, such as the Thurstone, Guttman, and Semantic !ifferential scales, by studying some of the references in the bibliography# A number of studies have been conducted over the years attempting to determine the limits of a personAs ability to discriminate between words typically found on rating or intensity scales# The results of this research can be of considerable value when trying to decide on the right set of phrases to use in your rating or intensity scale# Chen selecting phrases for a >=, 2=, 9=, or .=point :ikert scale, you should choose phrases that are far enough apart from one another to be easily discriminated, while, at the same time, keeping them close enough that you donAt lose potential information# Eou should also try to gauge whether the

phrases you are using are commonly understood so that different respondents will interpret the meaning of the phrases in the same way# An obvious e ample is shown with the following 0 phrases) Strongly Agree, ?eutral, Strongly !isagree These are easily discriminated, but the gap between each choice is very large# How would a person respond on this three=point scale if they only agreed with the 5uestion being askedO There is no middle ground between Strongly Agree and ?eutral# The same thing is true for someone who wants to respond with a mere disagree# Eour scales must have enough choices to allow respondents to e press a reasonable range of attitudes on the topic in 5uestion, but there must not be so many choices that most respondents will be unable to consistently discriminate between them# Appendi H provides several tables containing lists of phrases commonly used in opinion surveys with associated Jscale valuesK and standard deviations %or inter=5uartile range values'# Also provided is a short introduction describing how these lists can be used in selecting response alternatives for your opinion surveys# The information in that appendi is derived from research done for the U#S# Army $esearch 3nstitute for the 8ehavioral and Social Services at "ort Hood, Te as# 83AS A?! HBC TB 7B*8AT 3T :ike any scientist or e perimenter, surveyors must be aware of ways their surveys might become biased and of the available means for combating bias# The main sources of bias in a 5uestionnaire are) N N N N a nonrepresentative sample leading 5uestions 5uestion misinterpretation untruthful answers Surveyors can e pose themselves to possible nonrepresentative sample bias in two ways# The first is to actually choose a nonrepresentative sample# This bias can be eliminated by careful choice of the sample as discussed earlier in 7hapter ># The second way is to have a large number of nonreturns# The nonreturn bias %also called non=respondent bias' can affect both the sample survey and the complete survey# The bias stems from the fact that the returned 5uestionnaires are not necessarily evenly distributed throughout the sample# The opinions or attitudes e pressed by those who returned the survey may or may not represent the attitudes or opinions of those who did not return the survey# 3t is impossible to determine which is true since the non=respondents remain an unknown 5uantity# Say, for e ample, a survey shows that /, percent of those returning 5uestionnaires favor a certain policy# 3f the survey had a 9, percent response rate %a fairly high rate as voluntary surveys go', then the favorable replies are actually

only >+ percent of those 5uestioned %/, percent of the 9, percent who replied', which is less than 2, percentP a minority response in terms of the whole sample# Since little can be done to estimate the feelings of the nonreturnees, especially in a confidential survey, the only solution is to minimize the number of nonreturns# *iller %-.9,M p 1-' and Selltiz et al# %-./0M p +>-' offer the following techni5ues to get people to reply to surveys# Some of these have already been mentioned in earlier sections of this chapter# - Use follow=up letters# These letters are sent to the non=respondents after a period of a couple of weeks asking them again to fill out and return the 5uestionnaire# The content of this letter is similar to that of the cover letter# 3f you are conducting a volunteer survey, you should anticipate the need for following up with non=respondents and code the survey in some unobtrusive way to tell who has and who has not yet responded# 3f you donAt do that, but still need to get in touch with non=respondents, consider placing ads in local papers or base bulletins, announcements at commanderAs call, or notices posted in public places# 3f at all possible, provide a fresh copy of the survey with the follow= up letter# This often increases return rate over simply sending out a letter alone# + Use high=level sponsorship# This hint was mentioned in an earlier section# &eople tend to reply to surveys sponsored by organizations they know or respect# 3f you are running a military survey, obtain the highest=ranking sponsorship you can# (ffort spent in doing this will result in a higher percentage of returns# 3f possible, use the letterhead of the sponsor on your cover letter# 0 *ake your 5uestionnaire attractive, simple to fill out, and easy to read# A professional product usually gets professional results# > 6eep the 5uestionnaire as short as possible# Eou are asking for a personAs time, so make your re5uest as small as possible# 2 Use your cover letter to motivate the person to return the 5uestionnaire# Bne form of motivation is the have the letter signed by an individual known to be respected by the target audience for your 5uestionnaire# 3n addition, make sure the individual will be perceived by the audience as having a vested interest in the information needed# / Use inducements to encourage a reply# These can range from a small amount of money attached to the survey to an enclosed stamped envelope# A promise to report the results to each respondent can be helpful# 3f you do promise a report, be sure to send it# &roper use of these techni5ues can lower the nonreturn rate to acceptable levels# 6eep in mind, though, that no matter what you do, there will always be non=respondents to your surveys# *ake sure the effort and resources you spend are in proportion with the return you e pect to get# The second source of bias is misinterpretations of 5uestions# Ce have

seen that these can be limited by clear instructions, well=constructed 5uestions, and through 4udicious pilot testing of the survey# 8iased 5uestions can also be eliminated by constructing the 5uestions properly and by using a pilot test# "inally, internal checks and a good motivational cover letter can control bias introduced by untruthful answers# Although bias cannot be eliminated totally, proper construction of the 5uestionnaire, a well=chosen sample, follow= up letters, and inducements can help control it# 83AS 3? DB:U?T(($ SA*&:(S This section illustrates the many diverse, and sometimes powerful factors that influence survey findings as a result of using volunteers in a survey# The conclusions e pressed here regarding volunteer samples are provided to make the surveyor aware of the often profound effects of non= respondent bias on survey data# The e clusive use of volunteers in survey research represents another ma4or source of bias to the surveyor == especially the novice# Although it may not be immediately evident, it is nonetheless empirically true that volunteers, as a group, possess characteristics 5uite different from those who do not generally volunteer# Unless the surveyor takes these differences into consideration before choosing to use an e clusively volunteer sample, the bias introduced into the data may be so great that the surveyor can no longer confidently generalize the surveyAs findings to the population at large, which is usually the goal of the survey# "ortunately, research findings e ist which describe several uni5ue characteristics of the volunteer sub4ect# 8y using these characteristics appropriately, the surveyor may avoid inadvertent biases and pitfalls usually associated with using and interpreting results from volunteer samples# The following list provides ++ conclusions about uni5ue characteristics of the volunteer# The list is subdivided into categories representing the level of confidence to be placed in the findings# Cithin each category, the conclusions are listed in order starting with those having the strongest evidence supporting them# %from $osenthall and $osnow, The Dolunteer Sub4ect, -.92M pp -.2=-./') 7onclusions Carranting *a imum 7onfidence - Dolunteers tend to be better educated than nonvolunteers, especially when personal contact between investigator and respondent is not re5uired# + Dolunteers tend to have higher social=class status than nonvolunteers, especially when social class is defined by respondentsA own status rather than by parental status# 0 Dolunteers tend to be more intelligent than nonvolunteers when volunteering is for research in general, but not when volunteering is for somewhat less typical types of research such as hypnosis, sensory isolation,

se research, small=group and personality research# > Dolunteers tend to be higher in need for social approval than nonvolunteers# 2 Dolunteers tend to be more sociable than nonvolunteers# + 0 7onclusions Carranting 7onsiderable 7onfidence Dolunteers tend to be more arousal seeking than nonvolunteers, especially when volunteering is for studies of stress, sensory isolation, and hypnosis# Dolunteers tend to be more unconventional than nonvolunteers, especially when volunteering is for studies of se behavior# "emales are more likely than males to volunteer for research in general, more likely than males to volunteer for physically and emotionally stressful research %e#g#, electric shock, high temperature, sensory deprivation, interviews about se behavior'# Dolunteers tend to be less authoritarian than nonvolunteers# Gews are more likely to volunteer than &rotestants, and &rotestants are more likely to volunteer than $oman 7atholics# Dolunteers tend to be less conforming than nonvolunteers when volunteering is for research in general, but not when sub4ects are female and the task is relatively JclinicalK %e#g#, hypnosis, sleep, or counseling research'# 7onclusions Carranting Some 7onfidence - Dolunteers tend to be from smaller towns than nonvolunteers, especially when volunteering is for 5uestionnaire studies# + Dolunteers tend to be more interested in religion than nonvolunteers, especially when volunteering is for 5uestionnaire studies# 0 Dolunteers tend to be more altruistic than nonvolunteers# > Dolunteers tend to be more self=disclosing than nonvolunteers# 2 Dolunteers tend to be more malad4usted than nonvolunteers, especially when volunteering is for potentially unusual situations %e#g#, drugs, hypnosis, high temperature, or vaguely described e periments' or for medical research employing clinical rather than psychometric definitions of psychopathology# / Dolunteers tend to be younger than nonvolunteers, especially when volunteering is for laboratory research and especially if they are female# 7onclusions Carranting *inimum 7onfidence - Dolunteers tend to be higher in need for achievement than non=volunteers, especially among American samples# + Dolunteers are more likely to be married than nonvolunteers, especially when volunteering is for studies re5uiring no personal contact between investigator and respondent# 0 "irstborns are more likely than laterborns to volunteer, especially when recruitment is personal and when the research re5uires group interaction and a low level of stress# > Dolunteers tend to be more an ious than nonvolunteers, especially when

> 2 /

volunteering is for standard, nonstressful tasks and especially if they are college students# 2 Dolunteers tend to be more e troverted than nonvolunteers when interaction with others is re5uired by the nature of the research# 8org and Gall %-.9.' have suggested how surveyors might use this listing to combat the effects of bias in survey research# "or e ample, they suggest that) The degree to which these characteristics of volunteer samples affect research results depends on the specific nature of the investigation# "or e ample, a study of the level of intelligence of successful workers in different occupations would probably yield spuriously high results if volunteer sub4ects were studied, since volunteers tend to be more intelligent than nonvolunteers# Bn the other hand, in a study concerned with the cooperative behavior of adults in work=group situations, the tendency for volunteers to be more intelligent may have no effect on the results, but the tendency for volunteers to be more sociable could have a significant effect# 3t is apparent that the use of volunteers in research greatly complicates the interpretation of research results and their generalizability to the target population, which includes many individuals who would not volunteer# %pp -.,=-.-' SU**A$E The 5uestionnaire is the means for collecting your survey data# 3t should be designed with your data collection plan in mind# (ach of its three parts should take advantage of the strengths of 5uestionnaires while minimizing their weaknesses# (ach of the different kinds of 5uestions is useful for eliciting different types of data, but each should be constructed carefully with well= developed construction guidelines in mind# &roperly constructed 5uestions and well=followed survey procedures will allow you to obtain the data needed to check your hypothesis and, at the same time, minimize the chance that one of the many types of bias will invalidate your survey results#

7ommon Analysis (rrors

Statistical

8y far the most common analysis error committed by novices is the use of the wrong type of statistical tests with survey data# ?ovice surveyors most

fre5uently use intensity scale 5uestions that make use of a :ikert=type scale# 3n the section of 7hapter 2 entitled 3ntensity ;uestions and the :ikert Scale, we discussed the fact that these scales are typically assigned numerical weights to each ad4ective in the response set# &rofessional surveying organizations empirically anchor their instruments# This is done to ensure that each ad4ective in the response set is an e5uivalent distance from its ad4acent neighbors in the set# Anchoring is a labor intensive and complicated mathematical process whose e planation is beyond the scope of this handbook# Suffice it to say that most surveyors do not follow the process with the surveys they develop# The advantage of anchoring is that it creates a weighted scale along the entire response set of ad4ectives in which each ad4ective is a %mathematically' uniform distance from its neighboring ad4ectives# This creates what is known as an interval weighted scale# Cithout anchoring, one cannot be sure of the distance between responses in the set# :ikert, or any type of multiple= choice scale, that is not anchored will produce either nominal or ordinal data# "or e ample, consider the following response set)

3s the distance between Dery 3mportant and Somewhat 3mportant the same as that between ?ot 3mportant and ;uite UnimportantO How about the distance between any other pair of neighboring ad4ectives# Bf course, you could claim that the distances are e5ual, but how do you know all or even most respondents will see them as e5ual# All responses will be based their perceptions of the distances between each ad4ective# So, you can see that the numerical weights you assign to each ad4ective is arbitrary# Eou could 4ust as well weight them 2, >, 0, +, -M or +, -, ,, =-, =+M or -,,, 2,, +2, ,# (ach scale is as arbitrary as the ne t# The reason this is important is because whether the data generated by the survey are on an interval scale or not determines the particular kind of statistical tests you should use to analyze your data# *ost surveyors use descriptive statistics to provide general analysis of the response data# The most common descriptive statistics are the mean, variance, standard deviation, range, fre5uency counts, and percentage distribution# Some of these %such as the mean, variance, and standard deviation' re5uire interval data be used to make correct interpretations of results# &ercentages and fre5uency counts will work on any type of data %nominal, ordinal, or interval'# The mathematical computations for these statistics will work regardless of the kind of data you input, however# So, by using nominal or ordinal data in computing the mean will yield a result, but it may not necessarily be a true, or meaningful result# 7onsider the following e ample# Eou send out a survey containing one 5uestion to -,, people# The 5uestion is)

All -,, people respond to the 5uestion# "ifty say they like it a lot and 2, say they hate it# So, we weight each of the responses accordingly and get a total weight of) %2, 0' Q %2, -' R -2, Q 2, R +,,#

!ividing this total weight by the number of respondents %-,,' yields the average or mean response for the survey) +,, R +,# == %e5uivalent to a neutral rating'# -,, Ce interpret this to say that on average, people we surveyed are neutral toward the &residentAs economic policy# Bbviously, this is an erroneous interpretation of the actual results# ?ot one of the respondents was neutral to the 5uestion, yet the average response is neutral# The underlying problem is that the original data were not based on an interval scale# That is, our one=5uestion survey did not have an anchored scale# Anchored scales display the desirable 5uality of having e5ual intervals between each point along the scale# Cithout anchoring, you cannot be sure that scale points are e5uidistant from each other# 3n our current e ample, interval data were not generated because the scale was not anchored# 7onse5uently, we should not have computed the mean in the first place, because we could not reliably interpret it# Bf course, if the survey contained more 5uestions and<or a diverse set of response scales, the problem would only have been magnified# The proper descriptive analysis for nominal or ordinal data is to report fre5uencies %or percentages' of responses per category# 3n our e ample, it would have been most correct to simply report that 2, percent of those responding indicated they like the &residentAs economic policy and 2, percent indicated they hate it# Such a report is very easy to interpret, and provides accurate, useful data for decision=makers# Some surveyors are also interested in determining if responses from different groups of respondents are statistically different or not# Similarly, some are interested to know if respondentsA answers to certain 5uestions of the survey are related somehow either to their answers to other 5uestions on the same survey or to some demographic characteristic %their gender, rank, age, race, etc#'# To answer these types of 5uestions, surveyors must use a class of statistics known as inferential statistics# As with the descriptive statistics discussed above, there are different inferential statistics for use with interval data and with nominal or ordinal data# The former are called parametric statistics, while the latter are called non=parametric statistics# 3t is enough here to mention the names of the most basic statistical tests used to answer 5uestions about differences between respondent groups and relationships between responses# Bn the parametric side, the t=test is a common

test to determine if a statistically significant difference e ists between two %and only two' groups of respondents# To test for significant differences between three or more groups, the most common parametric test used is Analysis of Dariance %A?BDA'# Bn the non=parametric side, one should use a 7hi=S5uare test if the data are in the form of fre5uencies or counts within categories, or a *ann=Chitney %U' test if the data are in the form of ranks# The 7hi=S5uare %H+' test works regardless of how many groups %categories' there are# To determine relationships between responses, a useful parametric test is the &earson &roduct *oment 7orrelation 7oefficient %also known as the &earson correlation coefficient or, simply, the &earson r', while on the non=parametric side, there is the 7ontingency 7oefficient %7'# A book written by 8runing and 6intz %-.90', entitled 7omputational Handbook of Statistics, provides step=by=step procedures for manually calculating these and many other useful statistics with the use of 4ust a hand calculator# Their handbook is highly recommended as a basic resource te t# Eou should be able to obtain a copy through your local library, or purchase one through a local bookstore# Steps in Surveying

- !efine the purpose# 8e specificP + $eview e isting data# 3s a survey neededO 0 $ead applicable regulations# > !efine the hypothesis# 2 !efine the population# / !evelop the survey %F sampling' plan# 9 !evelop cover letter, instructions, F &rivacy Act Statement# 1 !evelop survey 5uestions# . &retest instrument# -, (dit and revise 5uestionnaire# -- Bbtain approvals as re5uired# -+ Survey %gather data'# -0 ;uality control<data reduction# -> Analysis and interpretation of results# -2 &repare report for customer%s'#

Survey !evelopment Timetables

Timetable for *a4or Surveys 7alendar !ay

-# !eliver to printer +# &rinting completed 0# $eceipt of 5uestionnaire by local surveying activities ># $eceipt of survey 5uestionnaires by respondents 2# Si =week administration period ends /# Answer sheets or completed 5uestionnaires returned by respondents 9# $eceive completed answer sheets by data reduction activity 1# 3nitial results available Timetable for Surveys Cith ( pedited &rinting and !irect *ailing to $espondents 7alendar !ay -# &rinting completed +# $eceipt of survey 5uestionnaires by respondents 0# "irst of 5uestionnaires returned to surveyor ># "ollow=up letters sent 2# "inal set of 5uestionnaires returned to surveyor

-> +> 0, 92 1,

., -, ,

-9 +9 09 2,

Survey !ata Sources

N Air University H; AU<7"A 22 :e*ay &laza South *a well A"8 A: 0/--+=/002

Attn) Survey 7ontrol Bfficer Approval source for all Air University surveys#

N ?ational Technical 3nformation Service %?T3S' 2+12 &ort $oyal $oad Springfield DA ++-/-

US !epartment of 7ommerce

Good source of published reports of Government agencies#

How to Use a $andom ?umber Table

+ 0 > ?umber each member of the population# !etermine population size %?'# !etermine sample size %*'# !etermine starting point in table by randomly picking a page and dropping your finger on the page with your eyes closed# 2 7hoose a direction in which to read %up to down, left to right, or right to left'# / Select the first * numbers read from the table whose last H digits are between , and ?# %3f ? is a two digit number, then H would be +M if it is a four digit number, H would be >M etc#'# - Bnce a number is chosen, do not use it again# + 3f you reach the end of the table before obtaining your * numbers, pick another starting point, read in a different direction, use the first H digits, and continue until done# ( ample) ? R 0,,M * R 2,M starting point is column 0, row + on $andom ?umber Table %ne t page'M read down# Eou would select population numbers >0, -0, -++, -/., etc#, until you had 2, uni5ue numbers# 2.>/1 ../.. ->,>0 -2,-0 -+/,, 00-++ .>-/. etc###### TA8:( of $A?!B* ?U*8($S

Table of I Dalues

7onfidence :evel ..#. ..#9 ..#2 ..#, .1#, .2#2 .2#, .,#, 12#, 1,#,

I "actor 0#+.,2 0#,,,, +#1,9, +#2921 +#0+/0 +#,,,, -#./,, -#/>>. -#>0.2 -#+1-/

Sample 7over :etter, &rivacy Act Statement, And 3nstruction Sheet

SA*&:( 7BD($ :(TT($ %:etterhead' "$B*) (HB %*a4or $oss, 0=+,>>' U!ateV SU8G) Survey Bn=8ase "acilities TB) - &eriodically, this command surveys its personnel to determine the effectiveness of base facilities and the desires of members concerning the re5uirements for any additional facilities# The attached 5uestionnaire was developed to obtain this information# $esults from this survey will be used

to improve current facilities and to help plan for new ones# + This is your chance to e press your opinions on current base facilities and to identify the need for additional ones# &lease answer the 5uestions as candidly as possible to provide us a valid assessment regarding facility improvements and additions# &articipation in this survey is voluntary, and no attempt will be made to attribute the answers with specific respondents# 3 solicit your prompt cooperation in this pro4ect and thank you for your time# GBH? G# GB?(S, 7ol, USA" 8ase 7ommander Attachment) ;uestionnaire SA*&:( &$3DA7E A7T STAT(*(?T

&$3DA7E A7T STAT(*(?T "B$ USA"=S7? HH=HHH

3n accordance with A"$ -+=02, paragraph 0,, the following information is provided as re5uired by the &rivacy Act of -.9>) - a# Authority) + %-' 2 US7 0,-, !epartmental $egulationsM and<or 0 %+' -, US7 1,-+, Secretary of the Air "orce, &owers, !uties, !elegation by 7ompensation# b# &rincipal &urpose) To sample Air "orce officer opinion and attitudes concerning base facilities# c# $outine Uses) To provide data as part of a base facilities study# d# &articipation in this survey is voluntary and respondents will not be identified# e# ?o adverse action of any kind may be taken against any individual who elects not to participate in any or all parts of this survey# SA*&:( SU$D(E 3?ST$U7T3B?S %!esigned for A!& Scanner Sheets'

+ 0 > 2

SU$D(E 3?ST$U7T3B?S

- !o not write your name or social security account number %SSA?' on the answer sheet# + There are no right or wrong answers to the 5uestions on this survey# Select

0 > 2 / 9 1

the or most appropriate response for each 5uestion# Use a ?o# + pencil when marking your answers on the answer sheet# !B ?BT use pen or marker# 8e sure your answer marks blacken the entire rectangle on the answer sheet# 8e sure to mark your answers carefully so that you enter them opposite the same answer sheet number as survey 5uestion number# Upon completion, please place your answer sheet in the attached envelope and place the envelope in base distribution# To help us ensure we meet our suspenses, please try to return your completed answer sheet by %return date'# Thank you for your time and cooperation#

$ating or 3ntensity Scales

Selection of $esponse Alternatives Using Scale Dalues and Standard !eviations Using scale values and standard deviations to select response alternatives will give a more refined set of phrases than using an order of merit list# 3n general, response alternatives selected from lists of phrases with scale values should usually have the following characteristics) N The scale values of the terms should be as far apart as possible# N The scale values of the terms should be as e5ually distant as possible# N The terms should have small variability %small standard deviations or inter5uartile ranges'# N Bther things being e5ual, the terms should have parallel wording# Table D333=(=- ==Acceptability &hrases %from) U#S# Army, -.90' Table D333=(=+ ==!egrees of ( cellence) "irst Set %from) *yers and Carner, -./1' ' Table D333=(=0 ==!egrees of ( cellence) Second Set %from) Gones and Thurstone, -.22' Table D333= (=> ==!egrees of :ike and !islike %from) Gones and Thurstone, -.22' Table D333=(=2 ==!egrees of Good and &oor %from) *yers and Carner, -./1' Table D333=(=/ ==!egrees of Good and 8ad %from) 7liff, -.2.' Table D333=(=9 ==!egrees of Agree and !isagree %from) Altemeyer, -.9,' Table D333=(=1 ==!egrees of *ore and :ess %from) !odd and Gerberick, -./,'

&hrase ( cellent &erfect in every respect ( tremely good Dery good Unusually good Dery good in most respects Good *oderately good 7ould use some minor changes ?ot good enough for e treme conditions ?ot good for rough use ?ot very good ?eeds ma4or changes 8arely acceptable ?ot good enough for general use 8etter than nothing &oor Dery poor ( tremely poor &hrase Superior "antastic Tremendous Superb ( cellent Terrific Butstanding Conderful !elightful "ine Good &leasant ?ice Acceptable Average All right B6 ?eutral "air *ediocre Unpleasant

Average /#+9 /#++ 2#9> 2#-. 2#,0 >#/+ >#+2 0#21 0#+1 0#-, +#9+ +#-, -#.9 -#9. -#9/ -#++ -#,/ ,#9/ ,#0/

Std# !ev# ,#2> ,#1/ ,#1,#92 ,#.1 ,#9+ ,#., ,#99 -#,. -#0, -#-2 ,#.2 -#-+ ,#., -#+-#,1 -#-,#.2 ,#9/ Std# !ev# -#-9 ,#10 -#0-#-. -#90 +#>2 -#.. +#0, -#12 +#-+ +#,1 +#,/ +#-> +#2. -#22 -#>+ -#/9 -#2, +#,/ -#1, +#1+

Scale Dalue +,#-+ +,#-+ -.#1> -.#1, -.#>, -.#,, -1#./ -9#0+ -/#.+ ->#1, ->#0+ -0#>> -+#2/ --#-+ -,#1> -,#9/ -,#+1 .#1, .#2+ .#>> 2#,>

8ad Dery 8ad Unacceptable Awful Terrible Horrible &hrase 8est of all ( cellent Conderful *ighty fine (specially good Dery good Good &leasing B6 "air Bnly fair ?ot pleasing &oor 8ad Dery 8ad Terrible &hrase :ike e tremely :ike intensely Strongly like :ike very much :ike very well :ike 5uite a bit :ike fairly well :ike :ike moderately *ildly like :ike slightly ?eutral :ike not so well :ike not so much !islike slightly *ildly dislike !islike moderately !islike !onWt like

0#11 0#+, +#/> -#.+ -#9/ -#>1 Scale Dalue /#-2 0#90#2+#11 +#1/ +#22 -#.-#21 ,#19 ,#91 ,#9=,#10 =-#22 =+#,+ =+#20 =0#,. Scale Dalue >#-/ >#,2 +#./ +#.+#/, +#0+ -#2-#02 -#-+ ,#12 ,#/. ,#,+ =,#0, =,#>=,#2. =,#9> =-#+, =-#21 =-#1-

+#-. +#-. +#,> -#2, ,#99 ,#19 Std# !ev# +#>0 -#,,#.9 ,#/9 ,#1+ ,#19 ,#9/ ,#/2 -#+> ,#12 ,#/> ,#/9 ,#19 ,#1, ,#/> ,#.1 Std# !ev# -#/+ -#2. ,#/. ,#/, ,#91 ,#2+ ,#2. ,#99 ,#/,#>9 ,#0+ ,#-1 -#,9 ,#.> ,#+9 ,#02 ,#>,#.> ,#.9

Strongly dislike !islike very much !islike intensely !islike e tremely &hrase ( ceptionally good ( tremely good Unusually good $emarkably good Dery good ;uite good Good *oderately good $easonably good "airly good Slightly good So=so ?ot very much *oderately poor $easonably poor Slightly poor &oor "airly poor ;uite poor Unusually poor Dery poor $emarkably poor ( ceptionally poor ( tremely poor &hrase ( tremely good Dery good Unusually good !ecidedly good ;uite good $ather good Good &retty good Somewhat good Slightly good Slightly bad Somewhat bad $ather bad

=+#09 =+#>. =0#00 =>#0+ Scale Dalue -1#22 -1#>> -9#,1 -/#/1 -2#>> ->#>> ->#0+ -0#>> -+#.+ --#./ --#1> -,#,1 /#9+ /#>> /#0+ 2#.+ 2#9+ 2#/> >#1, 0#+, 0#-+ +#11 +#2+ +#,1

,#20 ,#/> -#0. -#1/ Std# !ev# +#0/ -#/+#>0 +#-. +#92 +#9/ +#,1 +#+0 +#.0 +#>+ +#-. -#19 +#1+ -#/> +#>/ -#0/ +#,. -#/1 -#>> -#>> -#-9 -#9> -#-. -#-. Scale Dalue 0#>>. 0#+2, 0#+>0 0#,+> +#11, +#922 +#9-+ +#/++ +#>/+ +#>-9 -#>.9 -#0+0 -#+0+

8ad &retty bad ;uite bad !ecidedly bad Unusually bad Dery bad ( tremely bad &hrase !ecidedly agree ;uite agree 7onsiderably agree Substantially agree *oderately agree Somewhat agree Slightly agree &erhaps agree &erhaps disagree Slightly disagree Somewhat disagree *oderately disagree ;uite disagree Substantially disagree 7onsiderably disagree !ecidedly disagree &hrase Dery much more *uch more A lot more A good deal more *ore Somewhat more A little more Slightly more Slightly less A little less :ess *uch less A good deal less A lot less Dery much less *ean +#99 +#09 +#++#-, -#>9 #.> #/9 #2+ =#>0 =#/> =#.1 =-#02 =+#-/ =+#-9 =+#-9 =+#9/ Scale Dalue 1#,+ 9#/9 9#2, 9#+. /#00 /#+2 /#,, 2#.. 0#.9 0#./ 0#/> +#22 +#>> +#0/ -#./

-#,+> -#,-1 ,#.+> ,#9.9 ,#//+ ,#/0. ,#>9, Std# !ev# #>#>. #>+ #2, #>#>#0/ #>/ #>2 #01 #>9 #>+ #29 #2#>2 #>0 3nter5uartile $ange ,#/-#,> -#,/ ,#.1 -#,,#.1 ,#21 ,#29 ,#2/ ,#2> -#,> -#,/ -#--#,0 ,#2+
a

*inimum R ,#,2 Table D333=(=. ==!egrees of Ade5uate and 3nade5uate %from) *atthews, Cright, and Eudowitch, -.92' Table ( D333=(=-, ==!egrees of Acceptable and Unacceptable %from) *atthews, Cright, and Eudowitch %-.92'

&hrase Totally ade5uate Absolutely ade5uate 7ompletely ade5uate ( tremely ade5uate ( ceptionally ade5uate (ntirely ade5uate Cholly ade5uate "ully ade5uate Dery, very ade5uate &erfectly ade5uate Highly ade5uate *ost ade5uate Dery ade5uate !ecidedly ade5uate 7onsiderably ade5uate ;uite ade5uate :argely ade5uate Substantially ade5uate $easonably ade5uate &retty

*ean >#/+ , >#2> , >#>. , >#>+ >#01 , >#0> , >#0> >#+. > >#,/ 0 0#.+ + 0#1> 0 0#1> 0 0#>+ , 0#-> , 0#,+ , +#.1 , +#1/ 0 +#/, 1 +#>+

Std# !ev# #1>/ #.+- # 1+2 # 9-. # 1/. # 1/0

-#,01 # .-> #19/ -#,+/ # /,/

#.91 #12-#20/ # 19> # .9.

#..-#,0, # 99- # 1/+

ade5uate

+#0, / -#92 2 -#29 -#0+ 9 -#+, , ,#/+ 9 *ean ,#,,, =,#,+, =-#-29 =-#020 =-#01, =-#11+ =+#-,+ =+#-29 =+#+-/ =+#0>9 =0#/,, =0#902 =0#91, =0#.1, =>#-./ =>#>/, =>#/,1 =>#//9 =>#/1, =>#91> =>#9.+ =>#1,, =>#1,, #1.0 #/9, #9.0 # 2//

$ather ade5uate *ildly ade5uate Somewhat ade5uate Slightly ade5uate 8arely ade5uate &hrase ?eutral 8orderline 8arely inade5uate *ildly inade5uate Slightly inade5uate Somewhat inade5uate $ather inade5uate *oderately inade5uate "airly inade5uate &retty inade5uate 7onsiderably inade5uate Dery inade5uate !ecidedly inade5uate *ost inade5uate Highly inade5uate Dery, very inade5uate ( tremely inade5uate "ully inade5uate ( ceptionally inade5uate Cholly inade5uate (ntirely inade5uate 7ompletely inade5uate Absolutely

#.+1 Std# !ev# #,,, #0-/ #/01 #/+#99+ #90+ #.9> -#,-9 #1,, #.2. #/1, #999 #.>> -#2>2 #9>#209 #2+9 #/9/ #2,1 #>.1 #/>> #2+. #>0-

inade5uate Totally inade5uate &hrase Cholly acceptable 7ompletely acceptable "ully acceptable ( tremely acceptable *ost acceptable Dery, very acceptable Highly acceptable ;uite acceptable :argely acceptable Acceptable $easonably acceptable *oderately acceptable &retty acceptable $ather acceptable "airly acceptable *ildly acceptable Somewhat acceptable 8arely acceptable Slightly acceptable Sort of acceptable 8orderline

=>#.,, *ean >#9+ 2 >#/1 / >#>+ >#0. + >#-2 9 >#-2 9 >#,> , 0#+/ 0#-0 9 +#0. + +#+. > +#+1 , +#,, , -#.0 . -#1> , -#/1 / -#>2 1 -#,9 1 -#,0 . ,#.> , ,#,,

#>-+ Std# !ev# #2/0 #/-, #1/9 #9-/ #.-2 #1+2 #/0#.2/ #..-

-#>2/ #9++ #9++ -#-+2 #1-1 #.+> #9,, -#+>#2-1 #2++ #/>2 #+,,

?eutral *arginal 8arely unacceptable

, ,#,, , = ,#-+ , = -#-, ,

,#,,, #2-2 #0,,

&hrase Slightly unacceptable Somewhat unacceptable $ather unacceptable "airly unacceptable *oderately unacceptable &retty unacceptable $easonably unacceptable Unacceptable Substantially unacceptable ;uite unacceptable :argely unacceptable 7onsiderably unacceptable ?otably unacceptable !ecidedly unacceptable Highly unacceptable *ost unacceptable Dery, very unacceptable ( ceptionally unacceptable ( tremely unacceptable 7ompletely unacceptable (ntirely unacceptable Cholly unacceptable

*ean

Std# !ev#

= #21. -#+22 = #/9> -#9/2 =+#,+, #10/ =+#-/, #11, =+#0>, =+#>-+ =+#>>, =+#//9 =0#+02 =0#011 =0#0.+ =0#>>, =0#2,, =0#109 =>#+.> =>#>+, =>#>., =>#2>, =>#/1/ =>#.,, =>#.,, =>#.++ #/1#//+ #920 -#01#1.. -#,// #1-1 #99. -#,>> -#,-9 #202 #9+> #2,, #/,9 #>/> #0/#0/#+/.

Absolutely unacceptable Totally unacceptable

=>#.++ =>#.>-

#00> #+02

Table D333=(=-- ==7omparison &hrases%from *atthews, Cright, and Eudowitch, -.92'

&hrase 8est of all Absolutely best Truly best Undoubtedly best !ecidedly best 8est Absolutely better ( tremely better Substantially better !ecidedly better 7onspicuously better *oderately better Somewhat better $ather better Slightly better 8arely better Absolutely alike Alike The same ?eutral &hrase 8orderline

*ean >#1./ >#1>0 >#/,, t>#2/. >#090 >#+-/ >#,/, 0#.++ 0#9,, 0#>-+ 0#,2. +#+22 -#10> -#1-/ -#-29 ,#./,#201 ,#+-/ ,#-29 ,#,,, *ean = ,#,/-

Std# !ev# #2-, #>2. #9+#1+0 #10. -#>2. #.11 #11+ #.++ #.00 #1,+ #909 #1,#9-. #99/ #/2/ -#/+0 #1>9 #1,,#,,, Std# !ev# #0->

*arginal 8arely worse Slightly worse Somewhat worse *oderately worse ?oticeably worse Corse ?otably worse :argely worse 7onsiderably worse 7onspicuously worse *uch worse Substantially worse !ecidedly worse Dery much worse Absolutely worse !ecidedly worst Undoubtedly worst Absolutely worst Corst of all

= ,#-1> = -#,0. = -#+-/ = +#,91 = +#++, = +#2+. = +#//9 = 0#,+, = 0#+-/ = 0#+92 = 0#+92 = 0#+1/ = 0#>/, = 0#9/, = 0#.>= >#>0= >#>0= >#2-, = >#/1/ = >#99/

#.-. #1-/ #>.1 #1/, #.>> -#,0, -#>+0 -#,01 -#-,1 -#+,/ #119 #1,1 #1.. #.,9 #92+ #1+0 #9>1 #19+ -#+.-#+.1

Table D33=(=-+ ==!egrees of Satisfactory and Unsatisfactory %from) U#S# Army, -.90' &hrase ;uite satisfactory Satisfactory Scale Dalue >#02 0#/. Std# !ev# #.2 #19

?ot very satisfactory Unsatisfactory but usable Dery unsatisfactory

+#-+#,, ,#/.

#9/ #19 -#0+

Table D333=(=-0 ==!egrees of Unsatisfactory %from) *osier, -.>-' &hrase Unsatisfactory ;uite unsatisfactory Dery unsatisfactory Unusually unsatisfactory Highly unsatisfactory Dery, very unsatisfactory ( tremely unsatisfactory 7ompletely unsatisfactory Scale Dalue -#>9 -#,, ,#92 ,#92 ,#9,#+2 ,#-, ,#,,

Table D333=(=-> ==!egrees of &leasant %from) 7liff, -.2.' Table D333=(=-2 ==!egrees of Agreeable %from) *osier, -.>-' &hrase ( tremely pleasant Dery pleasant Unusually pleasant !ecidedly pleasant ;uite &leasant &leasant $ather pleasant &retty pleasant Somewhat pleasant Slightly pleasant &hrase Dery, very agreeable ( tremely agreeable Highly agreeable 7ompletely agreeable Unusually agreeable Dery agreeable ;uite agreeable Agreeable Scale Dalue 0#>., 0#-9> 0#-,9 0#,+1 +#1>. +#99, +#9>0 +#901 +#2,2 +#>>, Scale Dalue 2#0> 2#-, 2#,+ >#./ >#1/ >#1+ >#>2 >#-.

Table D333=(=-/ ==!egrees of !esirable %from) *osier, -.>-' Table D333=(=-9 ==!egrees of ?ice

&hrase Dery, very desirable ( tremely desirable 7ompletely desirable Unusually desirable Highly desirable Dery desirable ;uite desirable !esirable &hrase ( tremely nice Unusually nice Dery nice !ecidedly nice &retty nice ;uite nice ?ice $ather nice Somewhat nice Slightly nice

Scale Dalue 2#// 2#>+ 2#01 2#+0 2#+2 >#./ >#9/ >#2, Scale Dalue 0#020#-22 0#,-/ +#./. +#9/9 +#901 +#/0/ +#2/1 +#>11 +#+1/

Table D333=(=-1 ==!egrees of Ade5uate %from) U#S# Army, -.90' &hrase *ore than ade5uate Ade5uate ?ot 5uite ade5uate 8arely ade5uate ?ot ade5uate Scale Dalue >#-0 0#0. +#>, +#-, -#10 Std# !ev# -#-#19 #12 #1> #.1

Table D333=(=-. ==!egrees of Brdinary %from) 7liff, -.2.' Table D333=(=+, ==!egrees of Average %from) 7liff, -.2.' &hrase Brdinary Dery ordinary Somewhat ordinary $ather ordinary &retty ordinary Slightly ordinary !ecidedly ordinary ( tremely ordinary Unusually ordinary Scale Dalue +#,9> +#,90 +#,01 +#.0> +#,+/ -#.1, -#.>. -#.0/ -#192

&hrase $ather average Average ;uite average &retty average Somewhat average Unusually average ( tremely average Dery average Slightly average !ecidedly average

Scale Dalue +#-9+ +#->2 +#-,+#,.> +#,1, +#,/+ +#,2+ +#,0. +#,+0 +#,+,

Table D333=(=+- ==!egrees of Hesitation %from) !odd and Gerberick, -./,' &hrase Cithout hesitation Cith little hesitation Hesitant Cith some hesitation Cith considerable hesitation Cith much hesitation Cith great hesitation Scale Dalue 9#2, 2#10 >#99 >#01 0#+. 0#+, +#>a

3nter5uartile $ange /#2> 0#>, -#,/ -#/, 0#0. 2#+2 /#,,

*inimum R ,#2 Table D333=(=++ ==!egrees of 3nferior %from) 7liff, -.2.' &hrase Slightly inferior Somewhat inferior 3nferior $ather inferior &retty inferior ;uite inferior !ecidedly inferior Unusually inferior Dery inferior ( tremely inferior Scale Dalue -#2+, -#2-/ -#0+0 -#+.2 -#-1, -#-+9 -#,-0 ,#./0 ,#.+9 ,#9,2

Table D333=(=+0 ==!egrees of &oor %from) *osier, -.>-' Table D333=(=+> ==!escriptive &hrases %from) !odd and Gerberick, -./,' &hrase Scale Dalue

&oor ;uite poor Dery poor Unusually poor ( tremely poor 7ompletely poor Dery, very poor &hrase 7omplete ( tremely vital Dery certain Dery strongly Dery critical Dery important Dery sure Almost complete Bf great importance Dery urgent "eel strongly toward (ssential Dery vital 7ertain Strongly 3mportant Good Urgent 7rucial Sure Dital *oderately ?ow As at present &hrase Scale Dalue 1#12 1#9. 1#22 1#>, 1#+. 1#++ 1#-2 1#,/ 1#,2 1#,, 9#1, 9#21 9#22 9#-0 9#,9 /#10 /#9+ /#>/#0. 2#.0 2#.+ 2#+> 2#,0 2#,, Scale Dalue 3nter5uart il 3nter5uarti le $ange a #/2 #1> -#,2 -#,> -#-+ -#-/ #.2 #21 #.#., -#/, -#12 -#,2 -#>> #/9 -#-> -#+, -#20 -#90 -#19 -#/0 #.. #20 #2,

-#/, -#0, -#-1 ,#.2 ,#.2 ,#.+ ,#22

"air !onAt know Undecided !onAt care Somewhat 3ndifferent Bb4ect strongly to ?ot important Unimportant 8ad Uncertain !oesnAt make any difference ?ot sure ?ot certain ?on=essential !oesnAt mean anything 3nsignificant Dery little Almost none Dery unimportant Bnly as a last resort Dery bad ?one

>#./ >#1+ >#90 >#/0 0#9. 0#9, 0#2, 0#,. +#.> +#1. +#10 +#10 +#1+ +#/> +#21 +#2, +#-+ +#,1 +#,> -#92 -#9, -#2, -#--

e $ange #99 #1+ -#,/ +#,, #.> +#+, /#,9 -#00 -#>+ #.0 +#2, 0#-0 -#+> +#/+ -#/9 +#9-#-> #/> #29 -#+2 9#0, -#-0 #2.

*inimum R ,#2

Sample Sets of $esponse Alternatives

3t is sometimes valuable and is a time saver to have lists of response alternatives available to use# The tables in this section give some e amples of response alternatives that have been selected on different bases# These sets do

not e haust all possibilities# The sets of response alternatives that appear in Table D333="=- were selected so that the phrases in each set would have means at least one standard deviation away from each other and have parallel wording# Some of the sets of response alternatives have e treme end pointsM some do not# The sets of response alternatives shown in Table D333="=+ were selected so that the phrases in each set would be as nearly e5ually distant from each other as possible without regard to parallel wording# Table D333="=0 contains sets of response alternatives selected from lists of descriptors with only scale values given# The phrases were selected on the bases of e5ual appearing intervals# Table D333="=> has sets of response alternatives selected from order of merit lists of descriptors# Table D333="=- Sets Selected So &hrases Are at :east Bne Standard !eviation Apart and Have &arallel Cording %-1 Sets' Set $esponse Alternatives X
7ompletely acceptable $easonably acceptable 8arely acceptable 8orderline 8arely unacceptable $easonably unacceptable 7ompletely unacceptable Cholly acceptable :argely acceptable 8orderline :argely unacceptable Cholly unacceptable 0 :argely acceptable 8arely acceptable 8orderline 8arely unacceptable :argely unacceptable > $easonably acceptable Slightly acceptable 8orderline Slightly unacceptable $easonably unacceptable

2 Totally ade5uate Dery ade5uate 8arely ade5uate 8orderline 8arely inade5uate Dery inade5uate Totally inade5uate / 7ompletely ade5uate 7onsiderably ade5uate 8orderline 7onsiderably inade5uate 7ompletely inade5uate

Set $esponse Alternatives X

9 Dery ade5uate Slightly ade5uate 8orderline Slightly inade5uate Dery inade5uate 1 Highly ade5uate *ildly ade5uate 8orderline *ildly inade5uate Highly inade5uate . !ecidedly agree Substantially agree Slightly agree Slightly disagree Substantially disagree !ecidedly disagree -, *oderately agree &erhaps agree ?eutral &erhaps disagree *oderately disagree -Undoubtedly best 7onspicuously better *oderately better Alike *oderately worse 7onspicuously worse Undoubtedly worst

-+ *oderately better 8arely better The same 8arely worse *oderately worse

Table D333="=- %7ontWd' Sets Selected So &hrases Are at :east Bne Standard !eviation Apart and Have &arallel Cording %-1 Sets'

Set $esponse Alternatives X

-0

( tremely good $emarkably good Good So=so &oor $emarkably poor ( tremely poor -> ( ceptionally good $easonably good So=so $easonably poor ( ceptionally poor -2 Dery important 3mportant ?ot important Dery unimportant

Set $esponse Alternatives X

-/ :ike e tremely :ike moderately ?eutral !islike moderately !islike e tremely Strongly like :ike ?eutral !onAt like Strongly dislike

-9

-1

Dery much more A good deal more A little more A little less A good deal less Dery much less

Table D333="=+ Sets Selected So That 3ntervals 8etween &hrases Are as ?early (5ual as &ossible %-2 Sets'

Set X -

>

$esponse Alternatives 7ompletely acceptable $easonably acceptable 8orderline *oderately unacceptable ( tremely unacceptable Totally ade5uate &retty ade5uate 8orderline &retty inade5uate ( tremely inade5uate Highly ade5uate $ather ade5uate 8orderline Somewhat inade5uate !ecidedly inade5uate ;uite agree *oderately agree &erhaps agree &erhaps disagree *oderately disagree Substantially disagree Undoubtedly best *oderately better 8orderline ?oticeably worse Undoubtedly worst $esponse Alternatives

Set X

/ "antastic !elightful ?ice *ediocre Unpleasant Horrible 9 &erfect in every respect Dery good Good 7ould use minor changes ?ot very good 8etter than nothing ( tremely poor 1 ( cellent Good Bnly fair &oor Terrible . ( tremely good ;uite good So=so Slightly poor ( tremely poor -, $emarkably good *oderately good So=so ?ot very good Unusually poor $esponse Alternatives Cithout hesitation Cith little hesitation Cith some hesitation Cith great hesitation Strongly like :ike 5uite a bit :ike ?eutral *ildly dislike !islike very much !islike e tremely

Set X --

-+

-0 :ike 5uite a bit :ike :ike slightly 8orderline !islike !islike moderately !onAt like -> :ike 5uite a bit :ike fairly well 8orderline !islike moderately !islike very much -2 Dery much more A little more Slightly less Dery much less

Table D333="=0 Sets Selected "rom :ists Giving Scale Dalues Bnly So That 3ntervals 8etween &hrases Are as ?early (5ual as &ossible %-, Sets'

Set $esponse Alternatives X Dery, very agreeable Usually agreeable ;uite agreeable Agreeable + $ather average ;uite average Unusually average !ecidedly average Dery, very desirable 7ompletely desirable Dery desirable !esirable > ( tremely good Somewhat good Slightly good ( tremely bad

2 Slightly inferior $ather inferior Unusually inferior ( tremely inferior Set $esponse Alternatives X / ( tremely nice !ecidedly nice ?ice Slightly nice 9 Brdinary Slightly ordinary Unusually ordinary ( tremely pleasant !ecidedly pleasant Somewhat pleasant &oor Dery poor Dery, very poor Dery, very agreeable ( tremely agreeable Dery agreeable ;uite agreeable Agreeable Table D333="=> Sets Selected Using Brder=of=*erit :ists of !escriptor Terms %> Sets'

. -,

Set $esponse Alternatives X Dery good Good 8orderline &oor

Dery poor + Dery satisfactory Satisfactory 8orderline Unsatisfactory Dery unsatisfactory Dery superior Superior 8orderline &oor Dery poor ( tremely useful Bf considerable use Bf use ?ot very useful Bf no use

>

Air "orce 3nstruction 0/=+/,-# The Air "orce Sample Survey &rogram# Cashington, !#7#) !epartment of the Air "orce, &ublication &ending %-..0'# Altemeyer, $# A# %-.9,'# Adverbs and 3ntervals) A Study of :ikert Scales# American &sychological Association, &roceedings of the Annual 7onvention of -.9,, S %pt# -' pp 0.9=0.1# 8org, Calter $# and Gall, *eredith !amien %-.9.'# (ducational $esearch# Third (dition# ?ew Eork) :ongman, 3nc# 8runing, Games :# and 6intz, 8# :# %-.99'# 7omputational Handbook of Statistics# Second (dition# Glenview, 3:) Scott, "oresman and 7ompany# 7antelou, :ouis C#, Gr %-./>'# Bpinion $esearch as a *anagerial Tool for 3nternal &ublic $elations) The USA" Sample Survey and 3nternal 3nformation &rogram# Unpublished masterWs thesis, School of &ublic $elations and 7ommunications, 8oston, *ass#) 8oston University &ress# 7liff, ?# %-.2.'# Adverbs As *ultipliers# &sychological $eview, //, pp +9=>># 7ochran, C# G# %-./0' Sampling Techni5ues# ?ew Eork) Ciley and Sons, 3nc# !eming, Cilliam (# %-.2,' Some Theory of Sampling# ?ew Eork) Gohn Ciley and

Sons, 3nc# !epartment of the Air "orce# %Bctober -.9>'# A Guide for the !evelopment of the Attitude Bpinion Survey# Cashington, !#7# !odd, S# 7#, F Gerberick, T# $# %-./,'# Cord Scales for !egrees of Bpinion# :anguage and Speech, 0, pp -1=0-# !yer, $obert#, *atthews, G# G#, Stulac, G# "#, Cright, 7# (#, Eudowitch, 6enneth %-.9/'# ;uestionnaire 7onstruction *anual# &alo Alto) Bperations $esearch Associates %prepared for the U#S# Army $esearch 3nstitute for the 8ehavioral and Social Sciences, "ort Hood, Te as under 7ontract !AH7-.=9>=7=,,0+' Huff, !# %-.2>'# How to :ie with Statistics# ?ew Eork) C# C# ?orton and 7ompany, 3nc# Gones, :# D#, F Thurstone, :# :# %-.22'# The &sychophysics of Semantics) An ( perimental 3nvestigation# Gournal of Applied &sychology, 0., pp 0-=0/# 6ish, :eslie# %-./2' Survey Sampling# ?ew Eork) Gohn Ciley and Sons, 3nc# *atthews, G# G#, Cright, 7# (#, F Eudowitch, 6# :# %-.92' Analysis of the $esults of the Administration of Three Sets of !escriptive Ad4ective &hrases# &alo Alto) Bperations $esearch Associates %prepared for the U#S# Army $esearch 3nstitute for the 8ehavioral and Social Sciences, "ort Hood, Te as under 7ontract !AH7-.9>=7=,,0+' *iller, !elbert 7# %-.9,'# Handbook of $esearch !esign and Social *easurement# ?ew Eork) !avid *c6ay 7o#, 3nc# *osier, 7# 3# %-.>-'# A &sychometric Study of *eaning# Gournal of Social &sychology, -0, pp -+0=->,# *yers, G# H#, F Carner, C# G# %-./1'# Semantic &roperties of Selected (valuation Ad4ectives# Gournal of *arketing $esearch, 2, pp >,. >-+# Bppenheim, A# ?# %-.//'# ;uestionnaire !esign and Attitude *easurement# ?ew Eork) 8asic 8ooks, 3nc# &arten, *# %-.2,'# Surveys, &olls, and Samples) &ractical &rocedures# ?ew Eork) Harper and 8rothers# $osenthal, $obert and $osnow, $alph, :# %-.92'# The Dolunteer Sub4ect# ?ew Eork) Gohn Ciley# Sawyer, :awrence# %?ovember=!ecember -.9-' Statistics 7onfuse *e, Grandfather, 3nternal Auditor, Dol# +1, ?o# /, pp >.=2+#

Selltiz, 7laire, *arie Gahoda, *orton !eutsch, and Stuart C# 7ook %-./0'# $esearch *ethods in Social $elations# ?ew Eork) Holt,$inehart and Cinston, 3nc# U#S# Army Test and (valuation 7ommand %-.90'# !evelopment of a Guide and 7hecklist for Human "actors (valuation of Army (5uipment and Systems, U#S# Army Test and (valuation 7ommand %T(7B*', 7ontract !AA!B2=90=7=,011# Eoung, &auline D# %-.2.' Scientific Social Surveys and $esearch# &rentice=Hall, 3nc# ?ew Eork)