Research Design

Research Design
• “The program that guides the investigator in the process of
collecting, analyzing, and interpreting observations. It is a
logical model of proof that allows the researcher to draw
inferences concerning causal relations among the variables
under investigation” (Nachmias and Nachmias).
Research Design and Causality
• Relationships between variables: Two variables are related to
one another (i.e. are correlated) if one or more values of one
variable tend to be associated with one or more values of the
other variable.

• Causal relationship: A relationship in which one variable

directly causes/explains the other variable.
 3 Criteria for
Establishing Causality

• X is correlated with Y
• X precedes Y in time
• The observed relationship between X and Y is not
spurious
• Spurious Relationship = An observed relationship between X
and Y is said to be spurious (or partly spurious) if there exists a
third variable Z, which is both a cause of Y AND is correlated
with X.
Example of Spuriousness
Experimental Designs

A bunch of
people
Experimental Designs

Treatment
Group

A bunch of
people

Control
Group
Experimental Designs

Random
Assignment Treatment
Group

A bunch of
people

Random Control
Assignment
Group
Experimental Designs

Measure
the DV

Random
Assignment Treatment
Group

A bunch of
people Measure
The DV

Random Control
Assignment
Group
Experimental Designs

Measure
the DV

Random
Assignment Treatment
Group

A bunch of
people Measure
The DV

Random Control
Assignment
Group

(PRE-TEST)
Experimental Designs

Measure Introduce
the DV the IV
Random
Assignment Treatment “Stimulus”
Group

A bunch of
people Measure
The DV

Random Control
Assignment “Placebo”
Group

(PRE-TEST)
Experimental Designs

Measure Introduce Measure

the DV the IV the DV
Random
Assignment Treatment “Stimulus” Treatment
Group Group

A bunch of
people Measure
Measure
The DV
the DV
Random Control
Assignment
Group “Placebo” Control
Group

(PRE-TEST)
Experimental Designs

Measure Introduce Measure

the DV the IV the DV
Random
Assignment Treatment “Stimulus” Treatment
Group Group

A bunch of
people Measure Measure
The DV the DV

Random Control
Assignment
Group “Placebo” Control
Group

(PRE-TEST) (POST-TEST)
Experimental Designs
1. Select a sample
2. Randomly assign subjects into 2 or more groups.
3. Observe (measure) DV for all groups (if design includes
pretest)
4. Introduce the stimulus (IV)
5. Observe (measure) DV for each group
6. If the change in the value of the dependent variable varies
significantly across groups, then we conclude that X  Y
Experiments and Causality
• Correlation?
Experiments and Causality
• Correlation?
• Comparison of two or more groups (on dependent variable)
experiencing different levels of exposure to the causal
(explanatory) variable (X). This establishes correlation.
Experiments and Causality
• Temporal Precedence?
Experiments and Causality
• Temporal Precedence?
• The introduction of the independent variable (“stimulus”) is
manipulated by the researcher to insure temporal precedence.
Experiments and Causality
• Spuriousness?
Experiments and Causality
• Spuriousness?
• Random assignment insures that rival hypotheses are ruled
out, thus eliminating the threat of spuriousness. (How?)
Simple Experimental Designs
2-Group Pretest - Posttest Design (Classical or “Simple”
Experiment)

• R Mexp1 X Mexp2
• R Mcontrol1 Mcontrol2
Simple Experimental Designs
• 2-Group Posttest Only Design
• R X M1exp
• R M1control
OR
• R XA M1A
• R XB M1B
Simple Experimental Designs
2-Group Pretest - Posttest Design (Classical or “Simple”
Experiment)

• If two different treatments (and no pure control group):

• R M1A XA M2A
• R M1B XB M2B
Experimental Designs
• Key distinguishing feature of the
experimental design:

• Randomization (random assignment of

subjects to groups)
An Example: Rosenberg and McCafferty

• Hypothesis: Candidate presentation (appearance) is related to

candidate image (and therefore vote choice).
• Sample: University students
• Dependent Variable: Vote Choice (simulated ballot)
• Independent Variable: Candidate Appearance (picture in
campaign flyer)
Research Design (R&M)
• 2-group Posttest Only

• Generally:
• R XA M1A
• R XB M1B

• R XA (Good/Bad picture) M1A (Vote)

• R XB (Bad/Good picture) M1B (Vote)
The Independent Variable (“Stimulus” or
“Treatment”)
The Independent Variable (“Stimulus” or
“Treatment”)
Results
Other Types of Experimental Designs

• Multigroup designs – more than two

groups
• Multiple Group Pretest - Posttest Design
• Multiple Group Posttest Only Design
• Factorial Design
Methods of Experimentation
• Laboratory Experiments: experimentation among groups in a
laboratory setting

• Survey Experiments: manipulation of survey questions that

are randomly distributed amongst a group (more on surveys
later)

• Field Experiments: experiments that occur outside the

laboratory setting (in the real world)

• Note: all of these methods are experiments, because they

involve random assignment followed by the introduction of
a stimulus.
Assessing Validity of a
Research Design

Internal Validity

vs.

External Validity
Evaluating Research Designs:
Internal Validity
• Internal Validity - the degree to which we can be sure
that the independent variable caused the dependent
variable within the current sample
Evaluating Research Designs:
Internal Validity

• Experimental designs - randomization of subjects/units

across values of the independent variable greatly
reduces (eliminates?) the potential for spuriousness to
threaten internal validity
Evaluating Experimental Research
Designs: External Validity
• External Validity - the degree to which the results of the
analysis can be generalized beyond the current
sample/study. Can be maximized by:
• Using subjects (units) that are representative of the
population to which one’s theory applies
• Using a “laboratory” that is as close to “real life” conditions
as possible
• Field experiments
Nonexperimental Designs
• Deviate in some important way(s) from true experimental
design
• All nonexperimental designs lack random assignment of
subjects to groups
• But some nonexperimental designs may lack other features
too
Quasi-Experiments
• Lack random assignment, but otherwise similar to a true
experiment (have experimental and control groups)
• Example:
Quasi-Experiments

• M1A XA M2A
• M1B XB M2B
Quasi-Experiments

• M1A RetiredA M2A

• M1B ReturnedB M2B
Nonexperimental Designs
• Cross-Sectional Designs
• No manipulation of IV by researcher
• Observations for IV and DV recorded at
the same time
Example: Wine and Health
• Hypothesis: Drinking wine causes individuals to be healthier
(esp. heart)
• Existing studies: compared the health of wine drinkers to the
health of those who do not drink wine:
Research design
XA (Wine drinkers) M1A (Health)
XB (Non-drinkers) M1B (Health)
Spurious Results?
Controlling for Affluence
Research design:
XA (Affluent Wine drinkers) M1A (Health)
XB (Affluent Non-drinkers) M1B (Health)

XC (Poor Wine drinkers) M1C (Health)

XD (Poor Non-drinkers) M1D (Health)
Cross-Tabulation Exercise:
Control Variables
Example 1: No Control Variable
Sample Size = 80
  Good Health Poor health

Drink Wine 30 10
  Good Health Poor health
Drink Wine 20 20
Don’t Drink Wine 20 20
Don’t Drink Wine 4 36
Cross-Tabulation Exercise:
Control Variables

  Good Health Poor health

Drink Wine 20 20

Don’t Drink Wine 20 20

Cross-Tabulation Exercise:
Control Variables
Example 2: One Control Variable
Sample Size = 80

Affluent Good Health Poor Health

Drink Wine 18 2

Don’t Drink Wine 18 2

Not Affluent Good Health Poor Health

Drink Wine 2 18
Don’t Drink Wine 2 18
Cross-Tabulation Exercise:
Control Variables
Affluent Good Health Poor HHhHHHHHHEEIEIealth

Drink Wine 16 4

Don’t Drink Wine 10 10

Not Affluent Good Health Poor Health

Drink Wine 12 8

Don’t Drink Wine 4 16

Other Nonexperimental
Designs - Logitudinal
Time Series Design – repeated observations for X and Y for a
single unit

Panel Time Series Design – repeated observations for X and Y for

a group
Qualitative Case Studies
X O1
Inferences made by examining one case.

X 01
X 02
Inferences made by comparing cases.

Note: Remember Mill’s Method of Difference (JRM pg. 148-155)

Evaluating Internal Validity –
Experimental Designs
History
• Events other than explanatory variable at issue affect the dependent
variable.
Maturation
• Changes in individual during course of study.
Experimental mortality
• People die, move away, or drop out of study.
Instrument Decay
• Changes in instrument used to measure the dependent variable
Selection Bias
• Potential problem if no randomization/random assignment
Evaluating Nonexperimental Research
Designs: External Validity
• External Validity - the degree to which the results of the
analysis can be generalized beyond the current
sample/study. Can be maximized by:
• Nonexperimental studies can often avoid this problem
Class Exercise: What Design?
• Do education levels impact voting behavior?

• Does attorney experience improve the ability to win cases

before the U.S. Supreme Court?

• Does news viewing impact support for a war?

• Does difficulty of registration requirements impact voter

turnout?

• Does the number of police officers on the streets influence

crime rates?