EC455E Essay

Candidate ID 12265

Word count: 3023

Executive MPA Student

London School of Economics and Political Science, Institute of Public Affairs
Houghton St, London WC2A 2AE

The Leader of the Opposition, the Rt. Hon. Ed Miliband MP

The Shadow Secreteray of State for Work and Pensions, Ms Rachel Reeves MP

The Labour Party

Labour Central, Kings Manor
Newcastle upon Tyne NE1 6PA
London, January 17th 2014

EC455E Essay

Candidate ID 12265

Dear Ed, Dear Rachel,

RE Evaluation proposal for the Shadow Work and Pensions Secretary

We received your advance draft of Ms Rachel Reeves upcoming speech to the Institute for
Public Policy Research. We appreciate your confidence. My colleagues charged me with
drafting a proposal for the evaluation on your policy initiative for conditioning the job seekers
allowance (JSA) to training among certain 18-24 year-old claimants.
As I understand it, the initiative is two-fold:
assess young claimants with a standardised series of test for numeracy, literacy and
information technology (IT) skills, as part of the process of gaining access to the JSA;
condition the JSA, for those scoring below an equivalent of level 2 qualification (GCSE
passing grade), mandating attendance of a basic skills course for up to 16 hours a week
as the requirement.
Please find below a comprehensive evaluation proposal including a detailed outline of a
possible randomized control trial (RCT), as the chosen method to ascertain the true, causal
impact of the initiative. A brief outline of a cost benefit analysis is included at the end.
At this stage, however, I have very limited knowledge about your initiative.
Yours sincerely,
Executive MPA Student

EC455E Essay

Candidate ID 12265

THE CLAIM OF YOUR CHOSEN POLICY:

conditioning JSA to

basic skills assessment and training increases qualifications, enabling better access to the labour
market.
Starting with an implied causal chain from your initiative, I illustrate it simply in Figure 1.

Macroeconomic
factors

Employed

Job
applicant

Skilled

Educated

Job
seeker

job seekers allowance

ability

Natural
level

steady
employment

reliability

Matching
jobs

affinity
income
level

applicable

standards

Matching
skills

transparent
qualification

mastering

Qualified

learning
by doing
effective
teaching

Baseline
assessment

Training
Basic skills

FIGURE 1 SIMPLIFIED CAUSAL CHAIN

3

EC455E Essay

Candidate ID 12265

Deliberately presented, basic skills are a fundamental component of a virtuous chain leading
back to steady employment, which only macroeconomic factors prevent from feeding into the
natural level of unemployment. Although JobCentrePlus may be delivering on jobs matching
services, skills assessment and training, providing financial and other support, it still relies on
basic skills being held by jobseekers. More critically it relies on self-assessment for the highest
level of qualification held by the candidate.
Labours initiative offers two remedies. First, accurate assessment of fundamental skills has a
causal relationship to employment as successful next stage training (far right arrow on the
Figure 1). Jobseekers with basic skills are able to progress on average to higher paying jobs
and to employment better than those without. In particular, securing basic skills has a causal
relationship to future employment as they underpin any training. Second, the negative
encouragement of penalties has a causal positive average effect on the undertaking of basic
skills training for those jobseekers requiring it (close to center arrow on Figure 1).

THE CONTEXT OF YOUR CHOSEN POLICY: 18-24 year-olds

who are eligible for JSA and do not meet Level2 qualification lack basic skills to access the
jobs ladder.
According to the ONS, 54% of people not in employment, education or training (NEET) aged
16-24 are available for work or about 600 thousand, however a smaller part only is on JSA
about 315 thousand, all but exceptions are 18-24 years old. Of those, currently 58 thousand are
claiming over twelve months.1 There is no current information on the number of claimants by
qualification. However, with the cooperation of the ONS, and the Department of Work and

ONS, A01: Summary of labour market statistics. Date of Publication:22 January 2014. Excel workbook,
Available online at http://www.ons.gov.uk/ons/rel/lms/labour-market-statistics/january-2014/table-a01.xls

EC455E Essay

Candidate ID 12265

Pensions, extensive data can most likely be obtained on recorded information about JSA
claimants. This data can play a vital role in providing descriptive covariates for young
claimants under the policy. Using them, we can significantly increase the power of our
evaluation and detect smaller effects from smaller samples.

RCT EVALUATION
The most significant challenge to our evaluation is the dense recent history of welfare to work
programmes, which have attempted to counter recessionary unemployment pressures. In this
context, randomization will ensure that the effects of policies past and present are on average
zero. We can therefore focus on collecting information on the initiative itself. A simplified
theory of change illustrates how the policy proposed would ideally effect an impact. (Figure 2)
In order to link the theory of change to the causal chain, I would need further information on
the tools available at JobCentrePlus and the process undertaken by a job seeker. Most
importantly, the stated objective being ultimately higher paying and stable employment, this
evaluation could be the opportunity to gather further evidence on the steps job seekers take to
return to employment. Specifically, a large scale and longer iteration of the RCT evaluation by
a Labour government could provide welfare policy with important information, using a follow
up of additional treatment groups from the basic skills stage on to further training, both in and
out of work. (Figure 2)
For clarity however, our starting point is your initiative. The two treatments are: first, assess
basic skills among all 18-24 new claimants; second, condition JSA on a basic skills training
course.

EC455E Essay
No fundamental
qualification
below Level 2
essential hinderance for
access to the labour
market
Lack of access to
education (NEET)
inadequate training on
offer
inability to take up
available training
past exit of educative
sytem
Failed self- assessment
no referential of
outcomes
experience of failure

Need

Candidate ID 12265

Appropriate
assessment
baseline
assessment
identified basic
skills shortage
Encouragement
negative: penalty
up to loss of JSA

Input
Adult education Ofsted
standardised programs
curriculum
deliverables
testing
Institutional support
Job Seeker's Allowance
(JSA)
Advice and follow up

Job seekers
fail test
attend
keep JSA

Output

Outcome
Appropriate learning
fundamentals
numeracy
literacy
IT
improve score on
baseline
Better self-assessment
responsibility
identifiable next step
Access to training
in-job
job seeker

Increased
productivity
Better use of
resources
Employment
opportunities?

Impact

Exiting the system

risk of harm
6
FIGURE 2 SIMPLIFIED THEORY OF CHANGE

EC455E Essay

TRIAL DESIGN:

Candidate ID 12265

in this proposed RCT, young job seekers are assigned to three

separate groups per Table 1.

Groups are allocated per treatment, and one receives no treatment: all evolve in the context of
existing practices and policies. The second treatment (conditioning to training) cannot be
applied without the first (assessment). Therefore there is a treatment group receiving only the
first treatment, a treatment group receiving both, and a control group receiving neither.

Assessment of basic skills

First group

Second group

Conditioning allowance to
training

Control group
Table 1
Such a design needs to be further refined. Aspects of note include the treatment differences and
the random assignment itself. Your proposed initiative covers all 18-24 claimants with at least
the first treatment, therefore a declining population of about 300 thousand.
I am conceiving this RCT evaluation of a sample of people to be valid for at least the entire
population of concern, 18-24 year-olds claiming JSA. For this, a random sample is the best
achievable sample. Because of concerns from experimental effects, preparing for a random
assignment of JobCentrePlus offices, the sample is made of randomly selected offices. We
should then check for excess imbalances in the sample, which are in contrast with the
population.

THE TREATMENTS have to be practical: there is no sense in presenting job seekers with
an assessment and pretending the next logical step is not basic skills training. Just as the first
group must have access to training, there have to be no strings attached (we can include
7

EC455E Essay

Candidate ID 12265

questions in the end survey to help determine whether the job seeker drew the conclusion
themselves or the JobCenterPlus employee included it as part of their obligations). A clear
difference has to be drawn with a potential policy application: while the implementation would
most likely not require the basic skills course from high scoring claimants, the RCT design
requires that all assigned to second group be mandated to basic training, however superfluous,
focusing progress on the end survey. Only this way can the evaluation not be subject to
selection bias from within.
The second group poses a further difficulty, however: we need to define the specific
circumstances of the penalties. A tempting digression here would be to debate the path leading
to penalties. While disregarding it would set up the treatment to fail from the start, it is an
important part of negative encouragement that they lead to unintended effects. My focus would
be to monitor and measure such effects. Monitoring the RCT and measuring key indicators is
part of the data we need (see dedicated section below).
A basic skills training will ensure minimal variation among providers, while contributing to
external validity. The penalties may be ineffective if it is understood that they will simply not
be carried out; private organisations may want to provide risk mitigation and JobCentrePlus
may want to shield individuals at increased risk. An idea would be to personalise
communications through mobile text messages with personal greeting, communicating
effectively about penalties.
Applying the penalties, up to loss of allowance, can create avoidance it would be
counterproductive to debate possible ways in which job seekers would attempt and be helped
by support staff and courses to avoid penalties for a plethora of reasons. Just as
counterproductive would be to create a complicated schedule of penalties disconnected from
the objectives of training, which they are meant to encourage. A clear benefit from an RCT is

EC455E Essay

Candidate ID 12265

to detect effects such as pitfalls and cheats in implementation. If support staff are afforded the
opportunity of deferring penalties, we should measure this as best we can. If training
organisations report attendance, we should also monitor this step. However, it would be a
diversion to focus part of the design in preventing the by-products of a policy in the first RCT.
Should the effect of treatment prove worthwhile, additional RCTs can address ways to limit
unintended effects, especially as we use small samples of JobCentrePlus offices.

RANDOM ASSIGNMENT of individuals is the main tool of RCT. As opposed to a

simple RCT, we can assign individuals to groups in clusters. The main objective in deviating
from a simple and clear RCT design (individual random assignment) is preventing
experimental effects. A simple RCT, which is always preferable to any complicating
derivatives, would probably create many of the common experimental effects (see section on
Threats to RCT).
Random assignment of individuals is simple but the policy partner JobCentrePlus is not an
appropriate agent for assignment. Support staff could not effectively assign job seekers
randomly, especially as their mandate includes giving consideration to all individual
idiosyncrasies. Even charging support staff with tracking the assignment of job seekers to
groups could create experimental effects. With this objective, I suggest random assignment of
offices rather than individuals (a higher level of randomization).
To perform a random assignment at a JobCentrePlus office or cluster level, we need to check
for balance and ideally include characteristics such as regional imbalances of unemployment,
age structure, etc. Using propensity score matching, we could group clusters on observable
characteristics; with group clusters complete, we randomly assign from each group half to
treatment and half to control. This requires a costly process because the UK enjoys a diverse
administrative geography, and JobCentrePlus offices have been reallocated progressively, with

EC455E Essay

Candidate ID 12265

varying coverage of population and territory. This question is important as it establishes the
valid counterfactual that has to be engineered. We can only use the control group average
outcomes as a valid measure for what the average outcomes would have been for the treatment
group had they not received treatment.
It is worthwhile asserting that without an RCT designed and implemented properly, variables
measured by the study will only generate correlations including omitted variable bias for
unobserved variables, and have no valid interpretation usable to evaluate the policy. This is the
main reason why monitoring the RCT and preventing threats as much as possible is essential:
reintroducing selection bias would sabotage the evaluation.

THREATS TO RCT
A higher level of randomization and separation of baseline survey from running the programme
ensures a reduced threat from experimental effects
Keeping choice within treatment and control group creates a fundamental bias. On the other
hand, we cannot assign treatment and control and not allow for control to drop out of JSA,
certainly if claimants find a job, any influence to complete training outside treatment
parameters is a problem. However, the attractiveness of training in the context of competing
offers of employment is an important real world variable that should be captured. Drop-outs
who are surveyed throughout the RCT are non-compliers, and we can use a statistical
instrument to interpret data generated in this way. Complete drop-outs form the attrition effect.
Here, attrition, usually a threat to control group is actually greater for the second treatment
group in the context of penalties. This could seriously harm the RCT and make this initiative
all the more challenging. The other main threats experienced by RCTs are deformation effects
from the essence of observing people in a controlled setting. The best known are specific to

10

EC455E Essay

Candidate ID 12265

treatment and control groups. Control group participants experience some effects knowing they
do not receive the treatment (the compensation part is known as John Henry effect).
Attrition is still possible with all groups from finding employment and not cooperating with a
follow-up and end surveys. With foreseeable attrition of such size, we should consider a budget
for follow-up of participants well beyond job seeker status, including incentives. The baseline
survey should include as much contact information as possible and regular contact may need
to be introduced. Another well-known effect is the distorted experience from knowledge of
receiving a treatment. The Hawthorne effect can be prevented just as the John Henry, by
isolating groups from each other, as I attempt to do by the higher level of randomization.
Externalities, defined as the effect on the population from running the experiment are unlikely
to arise. However, we should use the level of randomization to monitor for remotely possible
ones such as drops or surges in the number of claimants. Finally, as there is a considerable risk
from human error in any UK wide, spread out trial that can only be mitigated by deploying
costly resources, a double blind process could limit the potential for such, as well as the twolevel randomization. Of course, this will always be limited by the abundance of information
and speed of its propagation with virtual social networks.

11

EC455E Essay

Candidate ID 12265

MONITORING THE TRIAL:

COLLECTING DATA on the trial as it progresses is both useful to monitoring the trial
itself and enhances the design. An idea to design the monitoring process is to play out with
relevant actors from real world participants the processes in advance after randomly assigning
the clusters, doing so in a non-trial setting. We are ready to find a volunteer 18 year old non
English speaker, European Union citizen to go through the motions and report on a job seekers
experience up to date (who will refund the JSA).
Data collection will be shared among different actors, JobCentrePlus, the training organisation
and the RCT partner. Beyond the simple indicators available in a baseline survey from the steps
identified in the Need section, each step of the theory should have an indicator at least. For
example, to establish the standardised programs we can have a few classes of GCSE graduates
take the program exams and compare their scores to their own GCSEs. Correspondingly the
same for the baseline assessment and a younger classes. To measure self-assessment we can
include relevant questions in the end survey and match to the objective score difference.

THE DATA PREPARATION: a baseline survey needs to be performed, independently

from the treatment of assessment itself.
Targeted jobseekers spend varying time on allowance and are assessed though their highest
attained qualification is self-reported. For example, a baseline survey could measure problems
with self-reporting (inaccuracies with bias) but could also establish the strength of correlation
between actual highest level of qualification and treatments. A worthy addition to data gathered
would be the date of qualification; it is likely correlated with the score of claimants to the basic
assessment but may not be included in data available. This is but one of the variables that a
baseline survey including should collect.
12

EC455E Essay

Candidate ID 12265

THE NUMBER OF PARTICIPANTS REQUIRED, provided a random sample, is

largely reduced by the law of large numbers. It allows us to work with a limited sample and
measure the bounds of average outcomes with a set probability (the confidence interval, ie
95% confidence establishes limits within which the mean population outcome will be located,
around the observed sample average). Correspondingly, power calculations determine the
required sample size to detect with a set probability an effect of certain size. This is further
helped by the design requiring all assigned to second group to have training conditioned JSA,
as random assignment carries valid samples through the process.

Using software package Optimal Design Plus Empirical Evidence (Figure 3), it is possible
to show that from our population of 300 thousand, a much more limited number of 6,003
people, then assigned equally 2,001 for each group is sufficient, provided that our baseline
survey and other measurable variables (overall regional unemployment, gender, age, past
unemployment) contribute variables which account for a significant part of individual outcome

13

EC455E Essay

Candidate ID 12265

variation. This is measured by the level 2 R squared, and even a more limited (halved) such
reduction in statistical noise would allow the same power with 10,002 people, a larger number
but only 3% of the population.

RECOMMENDATIONS:
Other than the design outlined above, the following recommendations
It is paramount to monitor the implementation and not overcomplicate the RCT, for this reason
we should take full advantage of the inference power for the limited sample. This is particularly
important considering we need to provide 16 hours weekly training with highly varying class
sizes but high costs per individual with largest class sizes.
Currently, JobCentrePlus is sole policy partner but an evaluation partner is needed. In
particular, it is desirable to remove JobCentrePlus employees from the process as part of a
double blind attempt. While this is effected by the randomly assigning offices instead of
individuals, threats remain. Employees can interfere with allocation to treatment or control,
and therefore undermine the trial subjectively, especially with a number of policy safeguards
in place for at risk population. The Department of Work and pensions has experts for the
running of RCTs. However, due to the Prime Ministers recent decision not to allow early
access to senior civil service staff for review of opposition initiatives, I suggest the Labour
Parliamentary party ask the Commons Work and Pensions Select Committee to endorse the
evaluation and recruit an independent evaluation partner, such as the LSE. Finally, Parliament
involvement will define legal challenges that may be made to the RCT, and possibly overcome
them. One obvious challenge to be addressed is that JSA is an entitlement.

14