Anda di halaman 1dari 38

1

Alternative Research Designs

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
2

(VI) Alternative Research Designs

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
3

Alternative Research Designs &


Threats to Validity

In their 1963 work, and in extensions of the original


paradigm, Campbell, Stanley, Cook and others
present various research designs as exemplars of
effective and ineffective ways to deal with various
threats that jeopardized the validity of a research
study

In this section we will focus on various of these


designs indicating by way of example how each
either exposes or protects the study from to internal
and external validity.

Threats to statistical conclusion validity and


construct validity will not be indicated since the
threats go beyond the mere architecture of the
research design.

The designs that will be presented in this section are


as follows.

Not all the designs

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
4

Pre-experimental designs

One-shot case study

One-group pretest-posttest design

Static-group design

True experimental designs

Pretest-posttest control group design

Solomon four-group design

Posttest-only control group design

Quasi-experimental designs

Time series design

Equivalent time-samples design

Equivalent materials design

Nonequivalent control group design

Counterbalanced design

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
5

Separate-sample pretest-posttest control group


design

Multiple time-series design

Recurrent institutional cycle design

Regression-discontinuity analysis

The following symbols will be used in presenting the


architecture of each design.

Symbol Interpretation

X The treatment or independent variable.

O Observation or measurement of the


outcome or dependent variable.

R Signifies that the sample subjects have


been randomly assigned to comparison
groups, experimental and control
groups.

(X) ???????

Mk ???????

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
6

------- This line separating comparison groups


indicates that there is no evidence that
the groups were equivalent in the first
instance.

One-Shot Case Study:


A Pre-Experimental Design

The chief of a 48-officer police department arranges


with a local health club to provide discounted
membership to all officers. He also convinces the
city manager to pay for the membership to improve
the general health and morale of the department.

42 of the officers take advantage of their health


club membership (X), and …

Six months later the chief concludes that both


the general morale (O) of the department is
better than he remembers it being six months
ago.

The Research Design

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
7

One-Shot Case Study Design

X O

Validity Problems

This study suffers from several threats to validity.

There is no pre-measure of morale other than


the chief’s memory which is non-empirical and
subject to bias.

Events external to the study may have


influenced the morale of the officer independent
of their health club participation, e.g. a pay raise,
promotions, changes in personnel (History).

The morale of the officers might have been


changing over time anyway (maturation).

The morale of the department may have


improved by the mere act of the chief doing
something about it independent of the

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
8

arrangement for free health club membership


(interaction of selection & X).

The table below summarizes the potential threats to


the validity of a one-shot case study. Consistent with
Campbell and Stanley’s 1963 notation, …

A minus sign (-) indicates the potential presence


of the threat, …

A plus sign (+) indicates that the design protects


against the threat, …

NA indicates that the threat is irrelevant to the


design, and …

A question mark (?) indicates a potential for the


threat dependent upon other considerations.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
9

Potential Threats to Validity in a One-Shot Case Study

Potential Threats to Validity Effect On


The Design

Internal Validity

History 
Maturation 
Testing NA
Instrumentation NA
Regression NA
Selection 
Mortality 
Interaction: selection & maturation NA

External Validity

Interaction: testing & X NA


Interaction: selection & X 
Reactive arrangements NA
Multiple –X interference NA

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
10

One-Group Pretest-Postest Design:


A Pre-Experimental Design

Working with the police department’s crime


prevention unit, …

The city manager installs high intensity street


lighting in a downtown area with an increasing
incidence of street crime.

When the instillation is completed, the cumulative


incidence of crime in the three months proceeding
and the three months following are compared.

Crime fell by 14%.

The Research Design

One-Group Pretest-Posttest Design

O X O

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
11

Validity Problems

There are several problems with this study.

Any number of extraneous factors external to


the study might have been operant that
depressed the incidence of crime other than the
lighting program, e.g. changes in seasons, fewer
people frequenting the area due to increasing
crime (History).

Crime might have been declining anyway


(maturation).

The table below summarizes the potential threats


inherent in a one-group pretest-posttest design.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
12

Potential Threats to Validity in a One-Group


Pretest-Posttest Design

Potential Threats to Validity Effect On


The Design

Internal Validity

History 
Maturation 
Testing 
Instrumentation 
Regression ?
Selection +
Mortality +
Interaction: selection & maturation 

External Validity

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
13

Interaction: testing & X 


Interaction: selection & X 
Reactive arrangements ?
Multiple –X interference NA

Static-Group Comparison Design:


A Pre-Experimental Design

A civil judge, tired of granting continuances to


unprepared lawyers and seeing her docket growing
out of control, …

Adopts a near zero-tolerance policy – “If you


want a continuance you better have a pretty
good reason.”

After six months of holding to this policy, she


compares the number of cases on her docket with
the number on the dockets of the other civil judge in
the courthouse.

Her docket is 12% smaller.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
14

The Research Design

The architecture of this static-group comparison is


shown below.

Static-Group Comparison Design

X O
--------------
O

Validity Problems

The essential problem with this study is that there is


no empirical evidence that the dockets of the two
judges were the same at the time the zero-tolerance
policy was initiated.

If two groups are to compared to determine the

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
15

effect of the independent variable on the dependent


variable, …

Some strategy must be employed to assure that


the groups are comparable on all variables that
affect the dependent variable other than the
independent variable.

This can be accomplished in several ways


including …

Random assignment of cases to groups,

Matching the groups on extraneous


variables,

Using the extraneous variables as blocking


factors or covariates.

The table below identifies the potential threats to the


validity of static-group comparison designs.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
16

Potential Threats to Validity in a Static-Group


Comparison Design

Potential Threats to Validity Effect On


The Design

Internal Validity

History +
Maturation ?
Testing +
Instrumentation +
Regression +
Selection 
Mortality 
Interaction: selection & maturation 

External Validity

Interaction: testing & X NA


Interaction: selection & X 
Reactive arrangements NA
Multiple –X interference NA

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
17

Pretest-Posttest Control Group Design:


A True Experimental Design

The state’s narcotic control agency initiated a test to


determine the utility of drug-dogs in increasing both
the number of drug-busts and the quantity of drugs
sized.

The state has 32 local interagency drug task forces,


which were randomly divided into two groups.

A coin flip was used to determine which of the


two groups would participate in the experiment.

The 16 task forces selected for the study were


then randomly divided into two more groups and

Another flip of the coin was used to determine


which 8 task forces group would receive grants
to buy drug-dogs.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
18

When the results were analyzed a year later, …

The differences in both the mean number of


arrests and the mean amount of drugs seized
were significant, indicating the effectiveness of
the canine program

The Research Design

Pretest-Posttest Control Group Design

R O X O

R O O

Validity Problems

This is a powerful design in [protecting from threats


to validity as indicated in the table below.

If the subjects in such a design know that they are


being measured pre, however, …

The knowledge may interact with the treatment

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
19

variable (X) limiting the generalizations from the


study to a pretest sensitized population
(Interaction of testing & X).

This may be different from the population the


study was designed to test.

Potential Threats to Validity in a Pretest-Posttest


Control-Group Design

Potential Threats to Validity Effect On


The Design

Internal Validity

History +
Maturation +
Testing +
Instrumentation +
Regression +
Selection +
Mortality +
Interaction: selection & maturation +

External Validity

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
20

Interaction: testing & X 


Interaction: selection & X ?
Reactive arrangements ?
Multiple –X interference NA

Solomon Four-Group Design:


A True Experimental Design

The state’s prison was found libel in five sexual


harassment suits in 16-months, and as a result, has
been directed by the governor to initiate a training
program for all personnel.

Based upon a call to the National Institute of


Corrections, the director finds a curriculum (X) of
what is reported to be an exemplary program, but …

Rather than implementing it system-wide, he


wants to test it efficacy in changing attitudes
about gender issues in the workplace (O).

The test was conducted by taking a random sample


of 200 employees and randomly dividing them into

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
21

four groups of 50 each.

Two of the groups would take a gender issues


attitudinal survey both one week before and one
week after the implementation of the training
program, but …

Bnly one group would actually participate on the


program

R O X O

R O O

One of the other two groups would participate in the


program but would only take the survey one week
after completing the program, while the other group
would not participate in the program but take the
survey one week later

R X O
R O

The Research Design

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
22

Solomon Four-Group Design

R O X O
R O O
R X O
R O

As indicated in the table below, this is a powerful


design in controlling for the various threats to
internal and external validity.

Notice that this design cobbles together both a …

Pretest-posttest control-group design (blue), and


a…

Posttest-only control group design (yellow), …

Which taken together greatly enhance its power


to control for threats to validity.

R O X O
R O O

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
23

R X O
R O

The threats to validity controlled by this design are


presented below.

This may be different from the population the


study was designed to test.

Potential Threats to Validity in a


Solomon Four-Group Design

Potential Threats to Validity Effect On


The Design

Internal Validity

History +
Maturation +
Testing +
Instrumentation +

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
24

Regression +
Selection +
Mortality +
Interaction: selection & maturation +

External Validity

Interaction: testing & X +


Interaction: selection & X ?
Reactive arrangements ?
Multiple –X interference NA

Posttest-Only Control Group Design:


A True Experimental Design

In the previous example on gender issues in the


workplace, the Solomon four-group design included
design included a posttest-only control group design.

By itself, it is a powerful design for controlling threats


to validity, and is more efficient than the Solomon
design since it involves …

Half the number of comparison groups, and only

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
25

half the number of observations on the


dependent variable.

The Research Design

Posttest-Only Control Group Design

R X O

R O

Validity Problems

The table below indicates the threats that this design


controls.

Potential Threats to Validity in a


Posttest-Only Control Group Design

Potential Threats to Validity Effect On


The Design

Internal Validity

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
26

History +
Maturation +
Testing +
Instrumentation +
Regression +
Selection +
Mortality +
Interaction: selection & maturation +

External Validity

Interaction: testing & X +


Interaction: selection & X ?
Reactive arrangements ?
Multiple –X interference NA

Time Series Design:


A Quasi-Experimental Design
The southeastern ward of the city is a transitional area
notorious for crack houses, open prostitution, gambling,
vandalism, burglary and more than its fair share of
robberies and barroom assaults.

After considerable pressure from the councilman who


represents the ward, the police organize a task force to

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
27

clean the area out.

Over a four moth period, undercover officers work the


area gathering intelligence and …

Over a five day period sweep through the ward


making dozens of arrests and bulldozing crack
houses.

The graph below shows the variation in reported calls for


service in the area four months before and four months
after the task force swept through the area.

CALLS Trend plot


900

850

800

750

700

650

600
Task force sweep
550

500
0 1 2 3 4 5 6 7 8 9

MONTH

Calls for service over an eight month period.

The Research Design

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
28

Time Series Design

O1 v O1 O1 X O1 O1 O1 O1

Validity Problems

There are two major problems with this design.

History – We have no control group to indicate


whether some external extraneous factor could
account for the down turn in the number of calls
for service.

The interaction of testing with X – In this type of


design there is always the risk that the pre-
measure of the dependent variable might
interact with the treatment variable and …

Make the subjects more or less receptive to


the treatment.

This is not a problem in this task force


example since the pre-measures are

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
29

unobtrusive.

The table below indicates how this design controls


for various threats to validity.

This may be different from the population the


study was designed to test.

Potential Threats to Validity in a Time Series Design

Potential Threats to Validity Effect On


The Design

Internal Validity

History —
Maturation +

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
30

Testing +
Instrumentation ?
Regression +
Selection +
Mortality +
Interaction: selection & maturation +

External Validity

Interaction: testing & X —


Interaction: selection & X ?
Reactive arrangements ?
Multiple –X interference NA

Equivalent Time Series Design:


A Quasi-Experimental Design

Jails are noisy places with the clanging of steel


doors, the shuffling sound of inmates being brought
in and out of cell black, officers calling out inmates,
and of course no soft sound deadening materials like
carpet.

All this noise creates a tense environment for

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
31

both inmates and staff.

A jail administrator decided to test the effect of easy


listening music on the incidence of inmate
infractions, assaults and the like.

Over a one-month period on random days varying


morning and afternoons, he had music piped into the
cells blocks and barracks.

Analysis of the data indicated that there was a


small yet significant decrease in the incidence of
incidents on the days that music was played (X1
O) compared with days when it was not played
(X0 O).

The Research Design

Equivalent Time Series Design

X1 O X0 O X1 O X0 O X1 O … X0 O

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
32

X1 = days music played

X0 = days music not played

Validity Problems

Two potential problems with this design are:

Interaction of testing with X, and …

Multiple treatment interference

Testing & X – If the subjects know that they are


being observed, tested, then the pretest sensitization
may affect or interact with the independent variable
(X). This of course is not a problem in the jailhouse
music example.

Multiple treatment interference – The repeated


treatments in this design could have an additive or
even multiplicative effect on the outcome measure
and not be generalizable to applications where there
will be one treatment and one measure of outcome.

The potential threat to validity in this design are


presented in the table below.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
33

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
34

Potential Threats to Validity in an


Equivalent Time Series Design

Potential Threats to Validity Effect On


The Design

Internal Validity

History —
Maturation +
Testing +
Instrumentation ?
Regression +
Selection +
Mortality +
Interaction: selection & maturation +

External Validity

Interaction: testing & X —


Interaction: selection & X ?
Reactive arrangements ?
Multiple –X interference —

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
35

Equivalent Materials Samples Design:


A Quasi-Experimental Design

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
36

Nonequivalent Control Group Design:


A Quasi-Experimental Design

Two juvenile probation officers decide to experiment


with the effect of a “scared straight” program (X) on
the reduction of recidivism (O) among first time
offenders.

One decides to serve as the control group, …

The other agrees to take all of her first-time


probationers to both the state prison and the
state’s juvenile reformatory within the first two
months on probation, there to be “scared
straight.”

The caseloads of the two officers are not exactly the


same since …

Cases are not assigned randomly, rather …

They are assigned on an as-needed basis to


assure that all officers are carrying the same
number of cases.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
37

Needless to say that the two officers are not


equivalent either, one is male, the other is female,
and the former has six more years of experience.

The Research Design

O X O
-----------------
O O

Relative to internal validity, …

The primary limitation of this design is that there


is no empirical evidence that the two groups are
equivalent to start with, …

As would be the case if the subjects were


randomly assigned to comparison groups.

None the less, it can be a useful design since


frequently in applied settings, the comparison groups
are pre-existing and randomization to groups is not
possible.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University
38

Validity Problems

As with other designs that do not have equivalent


comparison groups, this design is potentially liable
for an interaction between selection and maturation,
although this is not likely in the “scared straight”
example.

In addition,the pretests on the outcome variable may


sensitize the
The assets and liabilities of this design are shown in
the table below.

Assessing the Goodness of a Research Study: Charles M. Friel PhD, College of Criminal Justice, Sam Houston State University