Randomization Part 1: Why and When

Rema Hanna
Associate Professor of Public Policy
Evidence for Policy Design Harvard Kennedy School
August 28, 2012

Todays Agenda
Discuss randomized evaluations, which are considered the gold standard of evaluation methods Overview of what it is, what is required to do one, and how it can help inform policy
Will not able to cover all details of experimental methods, rather an overview If you are interested in these methods, we can recommend additional reading material and put you in touch with researchers working on policy questions you are interested in

Randomized control trials

Also known as: RCTs Randomized evaluations Randomized experiments

Social experiments

Randomization Part I
Theoretical Overview of Randomized Evaluations Basic Structure of a Randomized Evaluation When should you conduct a Randomized Evaluation Choosing sample and sample size

Randomization Part II
Randomization Methods Level/Unit of Randomization Constraints

Green Bank example

Threats Attrition Spillovers

Randomization Part I
Theoretical Overview of Randomized Evaluations Basic Structure of a Randomized Evaluation When should you conduct a Randomized Evaluation Choosing sample and sample size

Key Goal of Evaluation

To understand what would happen in the absence of a program or policy (the counterfactual) Hard to do! Why? Counterfactual cannot be easily observed
Example: What would happen in absence of a training program if the individuals did not receive the training? Cannot directly observe since we can only observe one state of the world

Approximating the Counterfactual

If you cannot observe what happens to someone in the absence of the program. Can you compare their outcomes to someone who did not receive the program? Lets think about the training program: Can you compare the outcomes of those who got the training with those that did not? Why or why not?

Approximating the Counterfactual

Cannot just compare those who received the program to those who did not Selection Bias: those in the program may be different than those who are not

Two Forms of Selection Bias

Observable Characteristics:
Example: Years of Experience, Gender Use statistical techniques to control for these. But then, how many variables do you control for? What do you control for?

Unobservable Characteristics:
Many important variables that you cannot easily define or control for, e.g. motivation to engage in the program in the first place

RCT Helps Solves This Problem

In an RCT, the program or policy is randomly assigned
Have a group of households, schools, villages, etc. under consideration (sample)
Randomly divide sample into treatment (receives the program) and control groups (does not) Hardcoded randomization Softcoded randomization

RCT Helps Solves This Problem

This means that probability of receiving the program is the same for everyone in sample Flip of a coin Since probability of treatment is the same, those in treatment are similar to those in control in every way other than the fact that they are selected for the treatment Note that they are similar on both observable and unobservable characteristics

Compare outcomes for treatment and control group to get program impact

Designing and Running a RCT

The theory behind an RCT is very simple: if done right, provides the true program impact (one that is very simple to understand) But as in any evaluation methodology, there are design choices and threats to the validity of the experiment We will cover some of these issues today, but this will just be a brief overview

Randomization Part I
Theoretical Overview of Randomized Evaluations Basic Structure of a Randomized Evaluation When should you conduct a Randomized Evaluation Choosing sample and sample size

Timeline of Evaluation
* Design Study **Chose Sample * Baseline (if necessary)

* Randomize
*Intervention * Monitor Process to Reduce Evaluation Threat
* Conduct Follow-up Surveys

* Data Entry * Data Analysis and Report Writing

Randomization Part I
Theoretical Overview of Randomized Evaluations Basic Structure of a Randomized Evaluation When should you conduct a Randomized Evaluation Choosing sample and sample size

When Should you Conduct an RCT?

Unfortunately, there are no simple rules to determine when you should conduct an RCT General guidelines of when it can be feasible and useful
Some of these guidelines are more general to when should you evaluate
Some of these are more specific to RCT

When Should You Conduct an RCT?

Do an RCT: When the knowledge gained is important enough
Dont do an RCT: The program/policy will not have a chance of being scaled, no new general information will be learned

Do an RCT: When it is ethically feasible (piloting new programs, resources are scare or will be phased in, program impact is unclear)
Dont do an RCT: Crisis, Disasters, etc.

When Should You Conduct an RCT?

Do an RCT: If you are at the start of the program, or before a large scale up
Dont do an RCT: When the program has already occurred

Do an RCT: Program is representative of how it would be implemented at scale

Dont do an RCT: When the program is receiving extra attention or resources that it would not receive when at scale

When Should You Conduct an RCT?

Do an RCT: When you can maintain the intervention for a reasonable period
I Dont do an RCT: If not possible to preserve randomization, or program funds will run out

Do an RCT: When you are fairly confident that the program can be implemented logistically
Dont do an RCT: When the logistics still have kinks (often times, we pre-pilot with logistics first)

When Should You Conduct an RCT?

Do an RCT: When it is possible to randomize (piloting, phasing in of program)
Dont do an RCT: If the program is being scaled nationally or to everyone at once

Do an RCT: When you have a sufficient sample size

Dont do an RCT: When the program is being implement for very few households or villages

When Should You Conduct an RCT?

Finally, any impact evaluation takes time, resources and expertise to do it properly! RCTs are no exception.. Only do an RCT when you have:
Political and administrative support for evaluation The right expertise on board to design a high quality randomized evaluation

Randomization Part I
Theoretical Overview of Randomized Evaluations Basic Structure of a Randomized Evaluation When should you conduct a Randomized Evaluation Choosing sample and sample size

Statistics is a Guessing Game

If we wanted to know the right answer to a question, we would interview everyone in the population
However, this is very expensive

Statistics tells us that if we choose a representative sample of the population, we can get very close to the right answer
But, what is the right sample? And, how large does the sample need to be for us to get the right answer?

Choosing a sample
Want a sample representative of the population

For example, consider the case of the training program

If you chose the best, most motivated regions to implement the program in, what would happen? If you chose the poorest regions to implement the program in, what would happen?

In general: best way to get a representative sample is to choose a random sample of the population

Important Concepts to Note

Random sample is not random assignment
Choose a random sample to have a representative sample of population Conduct random assignment of treatment within the sample to generate similar groups that allows you to estimate the causal effect of the program

How Big Should the Sample Be?

Once you have determined who your sample should be, the next question is: How large should that sample be? Lets think about this intuitively? Why does this matter?

Randomize Across Two Houses

Randomize Across Two Houses

Treatment Group

Control Group

Are these two groups comparable?

Solution: Need a Larger Sample

Randomize Across Two Villages

Village 1: Treatment Group Village 2: Control Group

Randomize Across Two Villages

Village 1: Treatment Group Village 2: Control Group

Randomize Across Many Units!

Village 1

Village 2

Technically
When we randomize, account for two aspects: Sample size is large enough in terms of people, houses, etc. Random Assignment is not clustered into a few units
Often times, need to do random assignment of clusters (e.g. schools, village, etc.)

In this case, need a large enough sample size of clusters to randomize over, and we will often take a random sample of households within each village to obtain info on each cluster

I still have not said what is large

This depends on two factors: 1) Treatment Effect Size
(size of the expected effect of the intervention)

2) Variance of Treatment Effects

(variation in the treatment effect across the sample of intervention participants/areas)

Lets intuitively see what this means.

Before we do so, remember that:

To obtain treatment effect:

Mean (treatment) - Mean (control) = Effect (size) We then test whether this is the real effect of just an artifact of this particular sample: this test of statistical significance relies on how messy the data is or the variance

If the treatment effect is small

Difference in Means

Hard to Measure Difference In Outcomes Between Treatment and Control Groups

If the treatment effect is large

Difference in Means

If the variance is large.

Same effect size but small variance

To summarize
Smaller sample size:
Large treatment effect and small variance

Larger sample size:

Small treatment effect and larger variance

To calculate what is small and large, we use a technique called a power calculation
At minimum: general rule of at least 50 per group or it is not worth doing But often times, requires larger than that

Conclusion
Randomization provides best estimate of counterfactual and the true program effect Does this by eliminating initial selection bias:
creates treatment and control groups similar on observables and non-observables

A large sample size is necessary to have similar treatment and control groups.
The definition of large depends on the effect size and variance of the outcome variables, so need to conduct power calculations to get right size.

Next Lecture
Will continue discussing the details of how to conduct a randomized evaluation