Lecture 2:

Experimental Design
Dr. Ghazala Azmat
Monday 19th January 2015
ECN374: Behavioural Economics
1

Todays Lecture
First Principles for Experimental Design
Experimental Design
Phases of an Experiment
Important Nuisances

First Principles

Creating an Economy in the Lab

A microeconomics system is a complete, self-contained
economy
It consists of a set of agents and the institutions through which
they interact
E.g., buyers and sellers operating in a particular market
This general description applies equally well to:
Theoretical models
Naturally occurring economies
Artificial economies in the lab

The Agent
The agents are the individual participants in the economy
Each agent has his or her own characteristics
Type (e.g., buyer)
Endowment of resources (e.g., time, goods, cash)
Information (e.g., regarding others endowment)
Technology (e.g., production function)
Preferences over outcome

The Institution
The institution specifies which interactions are allowed among
the agents
The institution consists of:
a message space (or choice set) for each agent type
E.g., a range of allowable bid prices in an auction or a
simple matrix game
An outcome function, given the agents choice
E.g., the winner and price at auction, or payoff matrix
for a simple game

Predictions
Typically we assume optimization and equilibrium
Often we use a theoretical structure to give a unique prediction
of what each agent will do and what the overall outcome will
be
We advance economic knowledge by testing the predictions
against observations in the field
We refine the model description (of agent characteristics or the
institution) when significant discrepancies are found

Replicability
With proper implementation we achieve replicability
This is the hallmark of controlled laboratory work (in any
science)
Replicability means that any competent investigator can
produce functionally similar data

Practicality
So how do we do it?
To implement the experiment you must first recruit human
subjects to fill the role of agents
If appropriate, you construct computerized agents (robots)
To control the institution, you give the agents the desired
message space and enforce the outcome function.
We will discuss this in more detail later.

Induced Value Theory

IVD is the key methodological innovation for experimental

economics
It is based on the idea that proper use of a reward medium
allows the experimenter to induce pre-specified
characteristics in the subjects
Such that their innate characteristics become irrelevant
Three conditions are sufficient:
1. Monotonicity
2. Salience
3. Dominance
10

1. Monotonicity
In a suitable reward medium, more is always better (or,
alternatively, less is always better)
E.g., we can safely assume that every human subject prefers
More cash earning to less
More grade points to less
Prefers less tedious work to more
Later we will discuss the practical advantages of using cash
payments to other reward mediums

11

2. Salience
For each agent, the reward component corresponds to a clear
outcome function, e.g., utility or profit
And the subject understands this
Salience connects the outcome in the microeconomic system
to a reward medium that the subject cares about
The connection cannot work properly unless the subjects are
fully aware of it

12

3. Dominance
The reward increments are much more important than other
components of subjects utility that are affected by the
experiment
Subjects may have rivalrous or altruistic motives towards
other subjects or towards the experimenter
But these motives cannot upset dominance if the subjects do
not know how their own actions affect others payoffs or the
experimenters goal

13

Experimental Design

14

Experimental Design
Your purpose determines the appropriate design for the experiment
It defines focus variables: those whose effects you want to
understand
But nuisance variables may also have an effect and you need to
account for them
The whole point of experimental design is to deal appropriately
with both kinds of variables
There are two basic devices to separate the effects of the focus
from nuisance variables: Control and Randomization
These help achieve independence among the variables affecting
the outcomes
15

Control I
You, the experimenter, can freely choose the values of many
variables, e.g., two different auction formats and the sorts of
cost to induce on the seller
The deliberate choice, or control, of key variables is what
distinguishes experimental data from happenstance data
You have two options for controlling a variable:
Hold it constant keeping it at the same level throughout
the experiment
Vary it between two or more levels this is called a
treatment variable
16

Control II
As you hold more variables constant, the experiment becomes
simpler but you can also learn about direct effects and
interactions
It sometimes takes serious thinking about and careful work
through the theory before you can decide on the right control
variables

17

Independence
Treatment variables are independent if knowing the value of
one variable does not give any information about the level of
the other variables
We want to vary the treatment variable independently because:
If two variables are dependent then their effects are harder
to separate
An experiment could help
How do we make control variables independent?
This is easy for variables you can OBSERVE or are aware
about, as you can control for them as constants
18

But
We are not out of the woods yet!
Other factors may affect independence
May have an effect on the subjects behaviour
We need randomization

19

Randomization I
Some potential nuisance variables are not controllable, so
independence seems problematic
The lack of control is especially serious when the nuisance
variable is not even observable and may interact with the focus
variable
E.g., some subjects are more altruistic than others and you
cant observe it
What happens if you assign the more altruistic subjects to the
first set of instruction conditions?
Your conclusions will be wrong!!
20

Randomization II
There is a simple solution
If you assign the conditions in a random order, your
treatments will become independent of all unobserved
variables (observable or not)
E.g., do not assign the first half of the subjects to arrive to the
encouraging instructions; they may be more (less) altruistic
Instead use a random devise to choose which instructions to
use for each subject
Randomization ensures independence as the number of
subjects increases
21

Phases of an Experiment

22

Phases of an Economic Experiment

1.
2.
3.
4.
5.
6.
7.
8.

Research Question
Experimental Design
Recruitment of Subjects
Explanation of the Rules
Play
Payments
Analysis
Writing up

23

1. Research Question
You should always start with a research question.
Think clearly what you want to study.
Research question always before observing data.
New questions?
New designs?
Replications are valid and needed.
Ask yourself which methodology is best to study your
question at hand:
Theory? Empirical Data? Experiments?
If experiment, why?
24

Why experiments?
Control/Precise measurement.
True exogeneity such that causes may be isolated.
This can be difficult with happenstance data because
explanatory variables may be highly correlated.
Relatively cheap
Legal issues
E.g., Welfare-to-work experiments would probably not be
feasible in some countries because of equal-treatment acts.

25

2. Experimental Design
Think of the best experimental design for your research
question.
Inspiration from others is valid, but think twice before setting
on a design.
Typically, you want to conduct an experiment to find out about
the effect of just a few variables.
Focus variables? Nuisance variables?
What is focus and what is nuisance may depend on the
purpose of your study.

26

Good Design
A good experimental design helps you:
to sharpen the effects of your focus variable.
to disentangle the effects of different focus variables.
to minimize the effects of nuisance variables.

27

Direct control:
Constants and Treatments
In a lab you can directly control for many variables.
Controlling these variables makes the difference between
experimental and happenstance data.
You can keep these variables constant or vary them.
The ones you vary are your treatment variables.
The more treatment variables you have, the more you might learn
about these variables but at the same time your experiment will
be more expensive.
28

Validity of laboratory data

Two issues:
Internal validity: Do the data permit correct causal
inferences?
Thats a matter of good design.

External validity: Can we generalize from lab to field?

The experimentalists traditional answer to this is: We should.
Thats the idea of parallelism.

29

Avoid confounded treatments

Suppose you are interested in the revenue of two auction
formats with different numbers of bidders.
Two treatments:
1) Auction type I with 4 bidders.
2) Auction type II with 10 bidders.
Whats wrong here?
Effects will be confounded. You only learn about the
interaction; nothing about the two variables on their own.
Never change two variables at the same time!
Vary all treatment variables independently.
30

Within-Subject and Between-Subject

Design
In a within-subject design, a single subject is observed in
different treatments
The subject serves as her own control group
In a between-subject design the norm in experimental
economics different subjects are tested in treatments A and B
Statistical variation across the subjects then muddies the water
of what is observed by comparing A and B
Within-subject designs are statistically powerful than betweensubject designs because they automatically control for
individual differences
These are often a large source of variation
31

3. Recruitment of subjects
Recruit without false promises
Subject database is extremely precious resource.
Record subjects history!!! (In particular, if you do more than
one session or related studies at a later point in time).
The problem of no-shows
Be NICE to your subjects. Pay show-up fee if you have over
recruited. Think airlines.
32

Who Should Your Subjects be?

Students vs. Non-students
Advantages of students:
accessibility
convenient recruiting
low opportunity costs
steep learning curves
lack of exposure to confounding material.
Disadvantages of students:
something to prove
contamination
low income.
33

Professionals
The use of business professionals may solve some problems
but create others
Behave often worse.
Harder to motivate, confuse abstract situations with real life.
Not much evidence in favour.
But also not much evidence at all.

34

Alternative subject pools

Experiments in developing countries
Experiments with representative samples
Experiments with survey participants

35

Where to Conduct Experiments?

Classroom experiments are extremely convenient for one-shot
But students may feel more empathy/sympathy towards class
mates than to others they would normally interact with
Anonymity extremely important
Grades as reward medium
In any case, dont teach them the topic before (except if you
plan this as a treatment).

36

How many subjects? I

There are two issues:
How many per game?
How many altogether?
How many observations? But what is an observation?
The first depends on the theory/model you look at.
Simple in game experiments
More difficult if you test competitive markets/models with
continuum of traders

37

4. Explanation of the rules

Instructions are KEY to your experiment.
Clarity is of utmost importance. Test them!!
Examples? Can be dangerous
Statement of purpose?
General rules: No talking etc.
Read them aloud?
38

Instructions
Instructions tell subjects what they need to know
Useful to have a clear instructional script that enables
precise replication
Reading instructions out load is common practice to establish
public knowledge
As close as we can get to common knowledge
assumption in game theory
Common practice to explain how actions lead to payoffs

39

5. Play
Pen and Paper vs. Computers?
Pen and paper experiments allow a great deal of freedom in
changing your design, treatment, parameters, and procedures
with little effort and delay
Computer experiments often require the software to be written
and rewritten
And computer facilities are expensive in equipment,
maintenance, and support staff time
But there are advantages to computer experiments.

40

Computerization
You can exercise tighter control on the flow of information in
the computer network e.g., eye contact, body language, and
voice inflection
Fast and accurate data
May play role in the experiment
Computers permit you to have less interaction with the
subjects

41

Matching Protocols
Experiments are usually designed with several periods of play
so subjects can learn from experience
But if a pair of subjects play together several times, the
possibility of reputation building can affect the prediction
that game theory makes
i.e., may be game-theoretical equilibria for the repeated
game that differ from the one-shot, stage-game equilibrium
Unless that is what we care about, we should avoid having
subjects play with each other more than once in experiment
session
42

Lab log
It is good practice to keep a lab log, i.e., record all experiments
in a bound log book by date, purpose, experiments, software,
parameter values
Also to note any unusual events
often we forget what we did, when and why.
Experiments should be replicable!

43

Pilot experiment
If you are running a new experiment, then it is sensible to
conduct a pilot
This will help you to see ambiguities in instructions, missing
information, too much (little) time, problems with equipment
etc

44

Lab setup & Registration

Dont underestimate time for preparations.
In particular, when you have to book space.
Registration should be done to keep track of subjects history
in your data base.
Who should conduct your experiment? (There is no final
advice. But do NOT try to influence your subjects; do NOT
stare over their shoulders, etc.)
Subject monitors (to make processes believable)
45

How to handle queries

Different approaches
Often subjects are not allowed to say/ask anything allowed
because this might spoil others.
E.g., think of questions like Doesnt that mean that I
should always play X?
If there is anything of general interest, repeat for all.

46

6. Payments
Monotonicity: Subjects must prefer more reward medium over
less and not become satiated.
Salience: The reward must depend on choices of participants.
Reward (payoff) function implements institutions (rule of the
game).
Dominance: Subjects utility depends predominantly on the
reward medium and other influences are negligible.

47

Rewards
Vermon Smith (1962) reported the earliest experiments
comparing behaviour of subjects who were rewarded in points
with those rewarded with dollars
He found that those paid only in points tended to approach
competitive equilibrium more erratically and seemed to grow
bored with the experiment faster than those who were paid
money
In other words, by inducing value using money payment, the
experimenter need rely only on the assumption that everybody
likes having more money and nobody gets tired of having
more of it
48

7. Analysis
Experimentrics are econometric techniques customized to
experimental applications
Good knowledge of Statistics and Econometrics will give you
a comparative advantage
Butdont use fancy techniques for the sake of being fancy!
Use the technique (graphs, tests, descriptive statistics,
econometrics) which best answers to your research question.
We will talk more specifically about techniques soon

49

8. Writing up
It should be the easiest part if:
Your research question was clear.
Your design is most appropriate for such question
Your analysis is correct and insightful.

50

Structure of every experimental paper:

1. Introduction:
Motivation (research Question)
Related Literature (Highlight your novelties!)
2. Experimental design and Procedures.
Treatments
Recruitment
Incentives

51

Structure of every experimental paper:

3. Results:
Descriptive Statistics
Graphs
Tests and other Econometrics
4. Conclusion:
How your goal was achieved.
Suggest further research (and experiments?).

52

Important Nuisances

53

Important Nuisances I

In choosing your design it is worth thinking through what

nuisance variables are likely to be important
And how to deal with them
Here are some:
1. Learning
Subjects behaviour usually changes over time as their
understanding of game deepens in the session
You can control for it by keeping it constant
Use only the last few periods or runs

54

Important Nuisances II
2. Experience
Similar to 1 but occurs across sessions
Keep a database to track which subjects already came and
played in a particular experiment
The easiest solution is to use inexperienced subjects
3. Boredom and Fatigue
Keep sessions to no more than 2 hours and the shorter the
better
You may save money and time by having them longer but you
will compromise on salience and dominance when data
comes from tired/bored subjects
55

Important Nuisances III

4. Extracurricular contact
Prevent any uncontrolled communication among subjects
during session
If they talk they may collude!
If you cant monitor them, change the parameters if there is a
break
5. Self-Selection
Have a long list from which you can choose your subjects
You should actively choose balanced subject pools

56

Important Nuisances IV
6. Idiosyncrasies of individual subjects
Subjects (or groups) with a particular background may lead to
unrepresentative behaviour
Replicate with different subject pools

57