COMMONWEALTH OF AUSTRALIA

Copyright Regulations 1969

WARNING

This material has been reproduced and communicated to you by or on behalf of

Griffith University pursuant to Part VB of the Copyright Act 1968 (the Act).

The material in this communication may be subject to copyright under the Act. Any further
reproduction or communication of this material by you may be the subject of copyright
protection under the Act.
Do not remove this notice

Course of Study:
(CCJ19) Doing Criminology

Title of work:
The science game; an introduction to research in the behavioural and social
sciences, 7th ed. (2007)

Section:
The Science Game (Chapter 4, pp. 76-79) Before-and-After Method pp. 76--79

Author/editor of work:
Agnew, Neil McK.; Pyke, Sandra W.

Author of section:
Sandra W. Pyke, Neil McK Agnew

Name of Publisher:
Oxford University Press
Before-and- fter ethod

In the previous chapter we observed that the after-the-fact method,

requiring relatively little mental effort, involves speculating about anteced-
ents (suspects) in a huge search space which stretches back in time. We also
observed that one important way of reducing the search space was to focus
in on a limited time period. The before-and-after study capitalizes on this
idea and, although still error-prone, it is a more powerful method than the
after-the-fact method.
The before-and-after method applies to the kinds of questions that
start out with a statement such as "I wonder what will happen if. ... " That
is, you start with 0 1-for example, the onset of Susan's rash-and then
introduce X, a new skin lotion-Hornet Honey. After the prescribed
three-month treatment, you make another observation (0 2 ) and report
"complexion improved." But did the Hornet Honey hel~r was it one or
more of the many otherX's in the stream of events taking place between 0 1
and 0 2 that did the trick? Was it one or more of a host of other independent
variables?
C-an you think of some other suspects? In that given period many other
suspected reasons could account for the rash's clearing up: she may have
finished exams, started holidays, gotten a new boyfriend, washed her face

,----~\ (,- ... ~\

'0 ... , X21 ... 1-----?> \, 02 J
,___ '+ '
I {,Xn
\. IJ ?

+
Onset Water Improved
of softener complexion
rash installed

76
 Before-and-Mter Method 77

with water treated in the family's new water softener, stopped eating so
much juhk between meals, finished some biochemical growing up, been
exposed to more sunshine, or gotten used to the rash so that even a small
improvement looked great. But Hornet Honey would be only too happy to
take credit for the improvement, and probably in such a situation, most
of us would be quite content to give the salve a testimonial, even though, as
we have just indicated, it is only one of many suspects. Many medical
treatments parade as cures, but are likewise one of many suspects. It
requires a comprehensive series of experiments of the kind to be discussed
shortly to obtain durable information about the adequacy of the supposed
treatment.
This before-and-after method pretends to be scientific but would not
be regarded so by most scientists because too many suspects always remain,
even after we have ruled out many others. Suspects always remain, so
whether a method passes as scientific is a relative question. Four main types
of suspects, or rogues, challenge us in our attempts to sort fact from fancy,
in our attempts to locate or manufacture stable packages of information
(Campbell & Stanley, 1966).
The following medical example helps us identify these four trouble-
makers. After exposing them we will consider various tactics we can use to
minimize the mischief they do.

FOUR MISLEADING SUSPECTS

A physician has been given the job of evaluating a new drug for treating
depressed patients in a hospital. He examines the patients before the
treatment (0 1) and after the treatment (0 2 ). The treatment (Xd) consists of
a six-week period during which two of the new pills are taken three times
a day. After the treatment period the doctor decides that most of the
patients have improved.

_t / ' '
~ /--- '
~ .....
,..--
(a.': ,\Xt,} . , X2, . . . (XJ,} ... , X7, ... , Xn 1,
I
02;'
\ ..... _,/ '.....,_ .... ' ,_, '

All patients New drug

i 75% of
in group tre-atment patients
are depressed improved

The physician is very pleased and writes an article for a medical journal. A
scientist reads the article and writes to the physician, stating that there are
at least four other explanations why the patients may have improved, none
of which were considered by the physician-four classes of events that
could have accounted for the shift in the dependent variable, in the change
' ... -' to 16
from ,'()~', , __2',, The four suspect types are outlined in Table 4-l.
78 Before-and-After Method

TABLE 4-1 Common Suspect Types

TYPE EXAMPLES
Historical (in-the-gap) Exam
Stock market crash
Catching cold
Death of mother
Hangover
Maturational (time-tied) Hungry
Tired
Older
Rested
Menstruating
Male midlife crisis
Instrument-decay (elastic-ruler) Boredom, fatigue, or mood of researcher
Instrument wear or breakdown
Bias of researcher, practice
Testing (on-stage) Recall
Putting best foot forward
Lying

Historical (In-the-Gap) Suspects

"Were there any changes in ward routine or any ward staff intro-
duced during the treatment period that may have contributed to patient
welfare?" A new cook or a new ward supervisor may have been more
influential than the drug in bringing about patient improvement. Remem-
ber that, after careful study, many wonde;r drugs turn out to be duds. Since
the physician was looking at the drug and not at other possibilities, the drug
got credit, and other possible suspects went unnoticed.
In other words, the history of the period between 0 1 and 0 2 is filled
with suspects in addition to the drug. Thus,-anything that occurs between 0 1
and 0 2 is an in-the-gap suspect or rogue, and may influence 0 2 We typically
Jocus on our prize, or "treatment" suspect, forgetting all the other suspects
that, singly or in combination, might have increased or decreased 0 2 .
That's one of the reasons why the speculations we make and the conclu-
sions we draw using the before-and-after method are prone to error.

Maturational (Time-Tied) Suspects

The researcher asks another question. "Would 50 percent of the
patients have improved in a six-week period even without treatment?"
Given time alone, some illnesses cure themselves-that is, there are
variables, such as natural recovery time in the case we just mentioned, or
maturity in the case of Susan's complexion changes during adolescence. In
many instances of the simple before-and-after design, time alone, rather
~han the pet treatment of a given investigator, deserves the credit for the
rmprovement. Thus, if time alone can produce changes in 0 2 without any specific
':~~t;aPment," we are dealing with a maturational or time-tied suspect or rogue.
~,~<!r<l.use we tend to focus on our prize or "treatment" suspect, we tend to
 Before-and-After Method 79

forget that time alone may account for changes in 0 2 So this is a second
reason why conclusions drawn using the before-and-after method are
error-prone.

Instrument-Decay (Elastic-Ruler) Suspects

The researcher mentions a third explanation apart from the drug-
namely, the doctor's ability to measure depression may have changed. We
cannot measure depression with a precise ruler; rather we measure it with
a person's judgment, which, as we all know, fluctuates from time to time
like an elastic ruler. For example, doctors want drugs to work, and this
could affect their judgments so that they actually imagine improvement,
whereas an unbiased observer would not. Thus, measuring instruments, human
or otherwise, can change or deteriorate between 0 1 and 0 2 : when that happens, we
are dealing with elastic-ruler suspects or rogues. Because we focus our limited
attention and analytic capacity on our prize or "treatment" suspect, we tend
to forget that the change in 0 2 may be attributable to a change in the
yardstick or judge. This is a third reason why decisions based on the
befor_e-and-after method are prone to error. In other words, the elasticity
of :(?.0 warrants particular attention in the before-and-after design.

Testing (On-Stage) Suspects

Finally, the researcher suggests yet another factor (other than the
drug) that could explain the patients' improvement. The very act of
interviewing the patients at the beginning to see how sick they were may
have influenced them, particularly if they knew what was happening. Some
people respond for a while to almost any new treatment. On the second
interview they might well want to appear better so as to be able to go home
or to help the nice young doctor. Thus, some organisms (including humans)
change between 0 1 and 0 2 merely because they are "on display" or being studied;
when this happens we are dealing with on-stage suspects or rogues. Because such
suspects typically remain outside our realm of attention, we tend to ignore
them, and may erroneously give credit for the change to the "treatment."
This, then, becomes a fourth reason for questioning speculations or
conclusions based on the before-and-after method.
These four types of suspects plague every before-and-after study.
Unless we separate out their effects from the effects of the treatment we
want to study, we are seldom sure whether the treatment is effective or not.
The control-group method, to be examined in the next chapter, goes a long
way toward bringing these four rogue suspects under control.