Employment
and
Metis GmbH
Donau-City-Strae 6
A-1220 Vienna
October 2012
The views expressed in this report are those of the authors and do not
necessarily reflect the official opinion of the European Commission. All errors
or omissions remain the responsibility of the authors.
Contents
Contents ......................................................................................................................... 3
page 3
7.4 A2E evaluation requirements and deliverability .................................................... 71
7.5 Some implications ................................................................................................. 73
8 Conclusions......................................................................................................... 79
8.1 Positive messages ................................................................................................ 79
8.2 Challenges for ESF A2E ....................................................................................... 79
page 4
Final A2E Synthesis Report
page 5
1 Executive Summary
page 6
Final A2E Synthesis Report
Although labour market challenges have been a pan-EU experience, the evidence is
that some Member States have enjoyed an increase in employment levels since 2007,
most have experienced a decline and a smaller number are trying to come to terms
with a substantial loss of employment. The implication of this is that the
appropriateness of ESF A2E actions will be even more variable across Member States,
and likewise with the effectiveness of A2E key areas of intervention in terms of
generating employment results.
page 7
enterprises and/or to develop intermediate labour markets, social firms and
related mechanisms.
Interventions to promote A2E fell into basically three categories: enhancing
recipient employability, supporting the creation and retention of employment and
building the capacity of public employment and training service organisations.
The prioritisation of ESF-funded A2E interventions focused principally on
disabled people, various groups of unemployed or inactive people (including
those with multiple disadvantages and complex needs), ethnic minorities,
migrants and asylum seekers, women and young people.
In terms of significant results achieved to date.
In most country reports where data were available results for employment entry
and sustainability were at or close to target. Although some experts argued the
targets were not particularly stretching, this needs to be balanced by a
consideration of the serious deterioration in the labour market from the point in
time at which the targets were initially set.
The results for employment entry and sustainability achieved varied significantly
across Member States. The difficulty with this analysis is the lack of sufficient
cohort data which take a group of recipients who have completed their
intervention and been in the labour market for sufficient time to measure
sustainable employment entry.
Employment results within Member States vary significantly across types of
intervention and priority groups.
page 8
Final A2E Synthesis Report
page 9
In relation to results based on research, the experts flagged up variations in
sampling methods used, challenges in securing sufficiently large sample sizes
and attrition in cohort studies as potential issues.
However, it is clear from many of the country reports that experts struggled to find
data that related specifically to ESF A2E types of operations focussed on the types
of final recipients prioritised under A2E. This applied to expenditure, number of final
recipients and to employment results. On employment results, there was virtually no
information on the sustainability of employment secured following an ESF A2E
operation. Additionally, for employment results many country experts were dependent
on monitoring data provided by beneficiary organisations raising issues about the
reliability of the information.
In terms of the evaluation approaches and methodologies used in preparing the
country reports:
The most common approach was to build on monitoring data and evaluations
of management and implementation processes.
Evaluations drawing on primary research were much less common.
Impact evaluations were rare and where they did exist were more likely to be
theory-based than counter-factual or other impact evaluations. The limited
number of impact evaluations reflects the difficulties involved in designing and
delivering these, including accessing data particularly micro data; identifying,
recruiting and sustaining control groups including data protection issues; and
the stage in the programming period with most Member States focussing on
monitoring implementation rather than evaluating results.
Many evaluations did not provide even the most basic information on the results
(achieving sustainable job entry) and the cost of achieving these results.
1.7 Conclusions
Positive Messages
1. The key positive message is that by December 2010 a significant ESF A2E effort
had been mounted and delivered across the 27 Member States.
2. There is a lot of evidence across the country reports that, in relation to A2E, there
have been important changes in an effort to respond to the economic crisis. It is
important not to underestimate the challenge to MAs in trying to implement OPs
designed for the very different labour market circumstances which prevailed pre-credit
crunch.
3. There is a widespread appreciation that, particularly for clients with multiple
disadvantages (or multiple and complex needs), ESF A2E interventions should reflect:
the need for interventions over a relatively long time period before sustainable
employment results can be achieved; the importance of having access to a range of
services; the need for a much more customised or individualised approach and the
value of having a key worker/personal adviser/work coach/etc who works closely
with the individual over the course of their journey towards and into employment.
4. There is a growing understanding of the value of interventions which support
individuals once they have entered employment, particularly important in the
current economic crisis where labour market fragility is the norm.
page 10
Final A2E Synthesis Report
page 11
2 Background and study objectives
1 Metis and Eurval (2011) ESF Expert Evaluation Network Synthesis Report 1
page 12
Final A2E Synthesis Report
page 13
For some groups, the principal requirement will be to provide them with the
vocational and other job related skills which meet the needs of current and
prospective employers in a rapidly changing labour market.
The most disadvantaged groups of the population seeking to enter or re-enter
the labour market typically require a spectrum of services and interventions
dealing both with personal and social issues and with vocational skill
requirements.
The implication of these issues is that ESF key areas of intervention designed for and
targeted at specific vulnerable and priority groups need to be based on a deep
understanding of the factors that restrict access to employment for the types of people
the intervention is designed to assist. One size fits all is not the effective way forward.
A2E also requires a range of different interventions depending on the needs of specific
groups, and even individuals within groups.
Those with relatively good employment characteristics such as high
qualifications may require a simple intervention such as a graduate trainee or
internship programme.
Individuals who have been unemployed for a long time may confront a series of
disadvantages and have multiple and complex needs. They may require a
significant number of different interventions over a long period of time.
Effective interventions for promoting access to employment need also to be delivered
across a range of labour market participants, in addition to individual participants who
are members of priority groups such as youth unemployed.
2 Ramb, F. (2008). Employment Gender Gap in the EU is Narrowing, Eurostat Statistics in Focus 99/2008
3 Eurostat (2012). Active Ageing in the EU, Eurostat News Release 8/2012
page 14
Final A2E Synthesis Report
page 15
European Union. A good example of this has been the employability pathway or
pipeline approach utilised by a number of Member States which has the dual merits
of:
Focussing resources on helping to prepare people for real jobs with good
sustainability in the longer term, and
Creating an employability pathway which provides support of various kinds for
more socially excluded groups of the population (e.g. health support, child care
assistance, etc) allied with the development of appropriate vocational skills for
growing sectors.
This pipeline approach can help facilitate the most effective deployment of ESF
resources alongside mainstream national government resources in the fields of health,
education and training. As yet, there are no systematic evaluations of the value added
by the employability pathway approach in practice.
A second major consequence of the global recession and the factors giving rise to it is
the retrenchment in public expenditure in a large number of Member States. As is
well known, this is particularly severe in economies such as Greece, Spain and
Portugal but there will be significant reductions over the remainder of the 2007-2013
ESF programming period. This means that the mainstream funding which is deployed
alongside ESF resources is declining in many Member States.
Given the more challenging labour market environment and the decline in public
spending in most Member States, it will be harder to achieve as good outputs and
outcomes from ESF funded interventions compared to pre-recessionary years. This
can be done, but it means that organisations in receipt of ESF funds will need to work
much more intelligently, be more innovative and create much more effective
partnership with other organisations able to support particularly more disadvantaged
individuals by deploying their mainstream funds. In effect, partnership working and the
alignment of ESF and mainstream funds need to become the norm for the remainder of
the 2007-2013 programming period, and these partnerships must be effective in
practical terms and involve more than key organisations signing statements of intent or
other forms of documentation.
Real challenges are posed for ESFfunded A2E interventions.
Reduced demand for labour makes it hard for A2E key areas of interventions to
get results.
This is particularly the case for more disadvantaged groups (often priorities for
A2E assistance).
6 Evaluation of the Reaction of the ESF to the Economic and Financial Crisis (2012). Metis and Wiiw.
page 16
Final A2E Synthesis Report
page 17
The analysis in this report underlines the challenge of assessing the effectiveness of
ESF A2E interventions in such a turbulent period for European economies. The
grouping of economies in terms of their labour market situations, their experience with
ESF and their capacity to respond to change highlights the challenge of drawing broad
comparisons across Member States within this analysis of ESF A2E.
page 18
Final A2E Synthesis Report
Figure 2 provides an insight into one of the key elements of employment change
associated with the recession, namely that where there has been job recovery it has
been stronger for part-time employment. Particularly for the some of the newer
Member States, part-time employment growth has been substantial. Part-time
employment is a key labour market characteristic influencing what A2E measures are
able to achieve. The difficulty is that part-time employment generates a lower financial
incentive to move from welfare to work. The performance of the Polish economy is
interesting here as its employment levels have grown strongly while at the same time
part-time employment has fallen.
page 19
Figure 3. % Change in Unemployment Levels (Aged 15-64), 2007 to 2011
page 20
Final A2E Synthesis Report
Are the evaluation approaches and methodologies in place to allow for robust
evaluation of the results and impacts flowing from ESF funded A2E
interventions?
In addition, the report comments upon information sources and methodological issues
in relation to the evaluation effort.
Structure of Report
The report is structured in the following way to address the key questions which require
to be considered.
Overall Progress in Delivery of ESF Access to Employment. The next
section of the report looks at ESF A2E in an aggregate fashion across the 27
Member States and considers progress to date.
Analysis of Interventions. This section discusses the types of ESF A2E
interventions and the target groups across the 27 Member States and also pulls
together information on the effectiveness of interventions.
Factors Influencing and Good Practice in Implementation. Factors inhibiting
and promoting the performance of ESF A2E interventions are considered here,
and examples of good practice are collated.
Community Added Value. The Community Added Value (CAV) flowing from
ESF A2E activity in the Member States.
page 21
Evaluation Data, Approaches and Methodologies. This section provides a
concise assessment of the data and evaluation approaches used in coming to
judgements about how much ESF A2E activity is being delivered across the
Member States, and the extent to which this activity is effective.
Conclusions. The broad conclusions emerging from the study are brought
together in the final section of the report.
page 22
Final A2E Synthesis Report
Number of programmes
Convergence Competitiveness total
Total ESF Funds per together
Funds MS in % nat/ nat/ with
Country (Meuro) EU 27 mulitreg regional mulitreg converg. regional
AT 524 0,7% 1 1 2
BE 1.073 1,4% 1 1 4 6
BG 1.185 1,6% 2 2
CY 120 0,2% 1 1
CZ 3.775 5,0% 2 2 1 3
DE 9.381 12,4% 1 6 1 11 18
DK 255 0,3% 1 1
EE 392 0,5% 1 1
ES 8.057 10,6% 3 8 2 11 22
FI 619 0,8% 2 2
FR 5.395 7,1% 4 1 5
GR 4.364 5,7% 4 3 4
HU 3.629 4,8% 2 2 2
IE 375 0,5% 1 1
IT 6.938 9,1% 2 5 1 16 24
LI 1.028 1,4% 2 2
LU 25 0,0% 1 1
LV 551 0,7% 1 1
MT 112 0,1% 1 1
NL 830 1,1% 1 1
PL 9.707 12,8% 1 1
PT 6.512 8,6% 2 1 2 1 4
RO 3.684 4,9% 2 2
SE 692 0,9% 1 1
SI 756 1,0% 1 1
page 23
Number of programmes
Convergence Competitiveness total
SK 1.500 2,0% 2 2 2
UK 4.475 5,9% 3 1 3 6
EU 27 75.953 100,0% 29 29 12 47 117
Source: Metis compilations from ESF Data base (http://ec.europa.eu/social/home.jsp)
page 24
Final A2E Synthesis Report
the UKs national employment programmes which are major A2E interventions.
Additionally, this grouping contains some of the most populous Member States.
Over 12.5 million final recipients have been engaged through ESF A2E
operations. This may be an over-estimate to some extent as in some country
reports experts noted that there was a degree of double counting due to the way
management information was collected. Nonetheless this is a substantial
number of people and the programming period still has a number of years to run.
Turning to employment results, the ideal approach would involve an assessment of
two key measures.
Job entry: the numbers finding employment (or self-employment) subsequent to
completing their ESF A2E intervention.
Job sustainability: the numbers and percentages of those finding employment
who are still in employment, not necessarily with the same employer, say 12
months after first finding a job.
The results presented in Table 2 rely entirely on job entry as only one country report
provided evidence on job sustainability. The approach to measuring job entry in the
country reports fell into two categories.
Finding a job at an unspecified point in time after completing the A2E
intervention generally derived from monitoring data.
In a job either 3 or 6 months after completing the A2E intervention, typically
based on evaluation reports.
The employment results column of Table 2 shows that:
For only 3 countries, where the analysis is based on context reports, there is no
estimate for an employment result.
A number of country reports generated percentage employment result estimates
where absolute numbers are preferred.
Almost half of the country reports were not able to specify precisely the definition
of the employment result, but on the evidence of the other country reports the
high probability is that they all measure job entry defined in one of the two ways
set out above.
Bearing in mind the above, the best estimate is that 2.4 million employment results
have been achieved.
Factors making it more likely that this is an underestimate include the fact that
we have no absolute numbers for AT, BE, BG, FI, HU, LV, and PL. Additionally,
some of the results are for shortened time periods and in the great majority of
cases the results are only until the end of 2010.
Factors leading to overestimation include measurement issues for some
countries. For example, job entry in DK is measured alongside increasing
employability and competence levels; IE includes entering education and
training; SI includes some already employed at the start of their ESF
intervention.
page 25
Table 2. Financial and Physical Progress for ESF A2E, to December 2010
SK 143.2 260,030
page 26
Final A2E Synthesis Report
page 27
nearly 50% ahead of target for employment results, but the expert argues this largely
reflects co-financing of the major public programmes for the unemployed - whose
numbers have been swollen by the recession. The expert goes on to argue, particularly
in relation to the final recipients target, that numbers well ahead of target could equate
to a diluted intervention. Although this may reach more people, it is less likely to
achieve sustainable job entries, particularly given the challenging nature of the labour
market.
page 28
Final A2E Synthesis Report
page 29
This is an intervention common to many of the Member States on the basis of
the CR evidence.
Addressing basic and core/key/employability skills. Basic skills typically
include literacy, numeracy, IT and basic language skills particularly for migrants.
Employability skills include the things that employers generally demand such as
reliability in attendance, capability for working in teams, etc. The country reports
suggest that ESF A2E contributes significantly to the former but less to the latter.
Creating new or upgraded vocational skills are a traditional feature of ESF and
remain a significant component of A2E within ESF. The economic recession
appears to have given a stimulus to this with substantial redundancies in jobs
and skills generating a significant demand for re-training.
Another common intervention is work experience/internships. These perform
a number of functions as articulated in some of the country reports.
- They help generate the employability skills and general workplace awareness
discussed above.
- They provide an experience which can feature on a CV in relation to job
search.
- They offer a direct opportunity to impress an employer and so secure a
temporary contract or permanent employment.
Graduate internships can clearly be seen as an intervention in a number of the
Member States but the work experience intervention more generally has been a
significant response to the rapidly deteriorating youth employment situation across
Europe.
Support for enhanced skills in job search, job application and interview
processes. Again this is a traditional intervention for unemployed recipients of
ESF A2E resources. It is also very common across the Member States.
Post job entry support, sometimes called aftercare, is mentioned in a smaller
number of the country reports. The rationales for this generally relate to the
following issues:
- Particularly in a context of labour market depression, jobs available are often
on a short term and temporary basis. Recipients of A2E interventions who
succeed in finding a job may also need help in keeping the job or moving
quickly to another job.
- Particularly for recipients who fall into the most disadvantaged categories and
confront Social Exclusion, they may have a higher degree of fragility in the
early stages of a new job and both the recipient and their employer can
benefit from aftercare services.
In broad terms for ESF, this type of intervention is a means of securing longer term
value from the investment already committed to get the recipient into the job in the first
place.
Supporting Creation and Retention of Employment
ESF A2E resources in a number of Member States are committed to helping
unemployed individuals into self employment and enterprise start up. This is
a recognition of the fact that self employment is a significant component of total
employment in many Member States.
page 30
Final A2E Synthesis Report
page 31
Support for self employment and enterprise startup is provided by ESF A2E
in 14 countries. In some countries this has been introduced as a result of the
recession to compensate in part for the decline in employment in existing
businesses and organisations. Some country experts have argued that as a
consequence of the recession it is extremely difficult for unemployed and other
people to make a sustainable start in business.
Employment subsidies are supported by ESF A2E in at least 11 countries, and
again are seen by some as a counter-recessionary intervention which are the
margins can stimulate in particular small businesses to expand employment but
also as a measure to make it more likely that an individual from a prioritised
group (such as the young unemployed) will obtain an employment opportunity in
an increasingly competitive labour market.
Labour market advice and guidance features in at least 10 of the Member
States with support from ESF A2E. This is increasingly important in the context
of significant labour market change and it is perhaps disappointing not to see
more activity in this sphere.
Capacity development of, in the main, national, public and training services
receives ESF A2E support in at least 9 countries. As expected these are
principally more recent members of the EU although FR and IT are also
recipients of this support.
Some types of intervention appear not to be all that prevalent in terms of receipt of
funds from ESF A2E.
Support for personal development and confidence building is mentioned in
at least 5 country reports, but this seems low given the focus of ESF A2E often
on longer term unemployed and more disadvantaged groups.
Support to find work is reported for at least 7 countries but this should be
central to any concerted approach to A2E.
Support to sustain work (i.e. post job entry support) appears in only 2 country
reports, and yet there are significant issues about sustainability of employment
secured and this becomes a greater problem in a more depressed and flexible
labour market with short-term contracting a more significant form of employment.
In exploring variations across Member States in terms of the types of ESF funded A2E
interventions, the likely broad influences on what happens in an individual country are
the existing mainstream A2E delivery (active policy) and the social security system
(passive policy). Although it is beyond the scope of this study to consider in detail how
these impact, two broad points can be made in relation to the information in Table 3.
The relative importance of capacity building of public institutions in relatively new
EU Member States has already been identified, and is confirmed in the tabular
analysis.
Beyond this, the pattern of A2E activity does not appear to vary in any obvious
way by the groupings of Member States. This is perhaps surprising as the labour
market circumstances and Member State mainstream services in relation to A2E
are likely to vary significantly. This raises interesting questions about the
planning of OPs and the detailing of delivery that flows from these. The most
obvious issue is the extent to which those involved in planning an OP take
account of complementary services, such as health and social care, when
building the content of their programme.
page 32
Draft A2E Synthesis Report
page 33
MS Assess Careers Personal Deving Vocation- Work ILMs/ Employ- Support Support Childcare Post job Support Training Inst. Others
-ment and devment key al skills exp./ social ment for self- to find and entry for up- for dev.
of A2E labour and employ- devment intern- firms subsidies employ- work, job family support skilling existing (PES,
needs market confidence ability ships ment/ search/ support (prog- employee etc.)
info., building skills for start-up interview ression) s
advice work- skills
and place
guidance
NL
PL
PT
RO
SE
SI
SK
UK
EU 8 10 6 17 23 15 6 11 14 7 17 3 3 9 9 13
27
Notes:
1. EU-12 countries shaded in blue.
2. Czech Republic, Denmark, Luxembourg, Romania, Sweden and Slovak Republic either no information on interventions provided in country report or unclear.
3. Childcare and family support 8 specifically note childcare as support; 9 implied.
4. Others include apprenticeships; education (including youth education, further education and higher education) and assistance to gain qualifications; rehabilitation
services; provision of help/assistance services for disabled individuals; adaptions to workplaces; research/PhD grants/scholarships; accreditation of existing skills and
competencies; grants for travel/equipment; support to increase geographic mobility.
page 34
Draft A2E Synthesis Report
page 35
Table 4. Target Groups
MS Unemploy- Long term Existing Women Young Young Older Graduates Ethnic Migrants/ Roma Disabled People Others
ed unemploy- employees people people people minorities refugees/ people with
ed at risk NEET or at asylum multiple
risk of seekers disadv.
NEET and
complex
needs
AT
BE
BG
CY
CZ
DE
DK
EE
ES
FI
FR
GR
HU
IE
IT
LT
LU
LV
MT
page 36
Draft A2E Synthesis Report
MS Unemploy- Long term Existing Women Young Young Older Graduates Ethnic Migrants/ Roma Disabled People Others
ed unemploy- employees people people people minorities refugees/ people with
ed at risk NEET or at asylum multiple
risk of seekers disadv.
NEET and
complex
needs
NL
PL
PT
RO
SE
SI
SK
UK
EU 27 18 14 10 17 19 4 14 5 5 11 3 12 15 14
Notes:
1. EU-12 countries shaded in blue.
2. Others include persons with low qualifications/skills; prisoners and ex-offenders; young people in/leaving care; parents returning from maternity/paternity leave or with
caring responsibilities; families of disabled persons; war veterans; persons engaged in subsistence agriculture; men; students; and over 30s.
3. Country reports for Spain and Finland contain no or limited information on target groups.
4. The Maltese country report presents data on the percentage of final recipients in each of these groups but there is no evidence they were the OP stated priorities.
page 37
4.3 Significant results achieved to date
Clearly the purpose of ESF-funded A2E is not simply to provide useful activity to final
recipients. The resources there are to produce results for individual recipients, but
also of course raise the capacity of public and other organisations working to support
these recipients. In crude terms the results can be categorised in two very broad ways.
Measurable progress towards sustainable employment in terms of increased
skill levels and qualifications, growth in self-confidence, etc.
Sustainable employment.
As indicated earlier, the evidence to date is largely about entry to employment rather
than sustainability of employment secured. Additionally, the quantitative evidence from
the country reports on progress towards employment is virtually non-existent.
Overall Results
Before discussing the employment results there are some general problems with the
data which need to be stated at the outset.
In most instances the data are derived from AIR 2010, and consequently are
principally based on monitoring data generated by beneficiary organisations.
The employment results are generally expressed as a percentage of final
recipients rather than as a percentage of a cohort of recipients who have
completed their ESF funded activity. This means that as programmes are
building up and the number of recipients is increasing percentage employment
results are artificially deflated, and cost per result inflated.
Referring back to the final set of columns in Table 2, a number of conclusions can be
reached on employment results.
In very broad terms, employment results in most are at or close to target levels.
On the downside, there is a suggestion by a number of experts that the initial
targets were notional and not particularly stretching.
On the upside, the targets were set prior to the worst economic recession for 30
years which must act to lower the chances of A2E recipients finding and
sustaining work.
Bearing in mind the earlier caveats on the measurement of employment results, Table
5 indicates that:
The spread of percentage employment results is very wide and the
corresponding variations in cost per result is similarly extensive.
The relatively low percentage achieving employment results in DE, IT, LU, etc. is
almost certainly a reflection of the failure to measure what happens to a cohort
of recipients going all the way through the intervention to completion and
beyond, but will also reflect the rate of expansion of numbers over the
programming period.
In some cases (BE, LP and PL) there is no aggregate employment result for the
Member State but a range of results for different key areas of intervention can
be reported. These demonstrate the variation within a country across different
approaches can be as great as the variation across countries in aggregate
terms.
page 38
Draft A2E Synthesis Report
2. IE Country report provided Cost Per Employment result. % Recipients Achieving Employment
Results calculated by authors using formula outlined in Note 1.
4. Apart from the 5 country reports with no employment results data, the absence of figures on
cost per employment result reflects the position for those country reports where the employment
result was presented in percentage terms.
page 39
It is disappointing not to be able to present statistical information on progress towards
employment. This is surprising as the concept of measuring distance travelled is well
established in thinking about A2E interventions.
page 40
Draft A2E Synthesis Report
page 41
% Achieving Cost Per
Employment Employment
Result Result
Finland 36.7
Employment subsidy to state, employment to ELY 28.8
(Regional MEE office)
Wage subsidy total 24.1
Apprenticeship total 59.2
Start-up money for companies 69.0
Employment benefit in labour market coaching 20.3
Subsidised employment total 36.4
Vocational labour market training 28.3
Total subsidised employment + vocational LM training 26.1
Coaching labour market training 16.5
France 32.7 1,483
Local employment and inclusion plans 34
Job coaching in Seine Maritime 37
Ireland 28.1 13,649
Labour market programmes 49 48,948
Labour market activation fund 29
(employment)
13 (self-empl.)
Specific skills training 58
Traineeship 72
Community Training Centres 41
Lithuania 40.0 2,613
ALMP Assistance to acquire professional skills 36
ALMP Employment subsidies 30
ALMP Public works 14
ALMP Vocational training 18
Social vulnerable/social risk group 30-35
Luxembourg 3.6 32,692
1.1 Increase employment rate of older workers 86 16,315
1.2 Increase womens employment rate 49 27,096
1.3 Facilitate insertion of young people in the labour 5 54,523
market
1.4 Activate persons with a large distance to the labour 45 29,425
market
page 42
Draft A2E Synthesis Report
page 43
% Achieving Cost Per
Employment Employment
Result Result
Slovenia 66.0 2,845
Job subsidies 60.0
Developing new employment opportunities 63 (including
employed and
self-employed)
Unemployed persons 55
Persons already employed 100
Overview
Although Table 6 is able to present the results in only a piecemeal way, this way of
looking at results across both key areas of intervention and priority groups has great
potential to indicate where ESF A2E is being deployed effectively or not as the case
may be. Conclusions on this would of course need to be qualified by considering also
the cost of delivery and the degree of disadvantage experienced by the final recipients
of the intervention.
page 44
Draft A2E Synthesis Report
analyse statistically with relatively large samples the results for different key areas of
intervention , while controlling for the characteristics of the final recipients.
page 45
5 Factors influencing and good practice in implementation
With such a major effort going into supporting A2E from ESF resources across
Member States, a key task for the experts analysing the experiences in their countries
was to identify the factors that:
Constrained the effective implementation of A2E key areas of intervention.
Supported the implementation of these interventions.
Although not all country reports were able to draw on evaluations with this degree of
detail, many were and below the key findings are captured. Building out from the
analysis, evidence on good practice was also collected, and this is discussed later in
this section.
page 46
Draft A2E Synthesis Report
Germany
Issues relating to administrative management and implementation.
- Fragmentation of ESF in the light of increased decentralisation of implementation
leads to large number of programmes with small or average budget for each
but same administrative requirements. With increased number of programmes
this means more capacity building required for delivery agents.
- Administrative cost is taking up a higher share of budget, reducing money for final
recipients.
- Difficult to coordinate across a large number of agencies.
Hungary
Many concerns about administrative burden ESF places on beneficiary organisations.
This is exacerbated by frequent rule changes.
Italy
Low demand for evaluation and lack of urgency in commissioning/organising these
mean that good quality feedback on intervention effectiveness is not available.
Luxembourg
Too many administrative tasks.
Complicated database and guide.
Difficulty in finding co-finance for some target groups.
Issues with stability and capacity of beneficiary organisations.
Malta
Piecemeal planning and implementation of OP.
Lack of internal controls on effectiveness of management.
Lack of internal and external evaluations.
Burdensome and bureaucratic procedures making for costly management process,
particularly damaging given small size of OP in absolute terms.
Netherlands
Administrative burden associated with ESF leads to:
- More effort on administrative issues and less on getting results.
- Due to lack of experience with ESF admin and problems with admin data, claims
lie well below actual results with the latter remaining unreported and there is a
financial burden on the beneficiary organisations.
- Communications from the Commission are viewed as inconsistent and delivered
late with too many changes in the rules of the game.
page 47
Management and Administration of ESF (cont.)
Romania
There is no real basis for evaluating impact and no serious results are expected
earlier than 2013 by which time the programme is finished. The absence of the
necessary information makes it difficult to manage the programme effectively.
SOP HRD is a large programme with 7 PAs and 21 KAIs. It has national coverage
and 11 IBs (8 at regional level and 3 at national level). Many of the staff are new to
the system, are having to work on new procedures, taking on new ways of doing
business. Despite investment in training and capacity building there has been a high
level of turnover within the MAs and the IBs.
There are serious issues around capacity of beneficiaries to access and implement
funds.
Slovak Republic
Problems around clarity of definition of indicators for results.
Lack of any serious qualitative analysis of implementation by responsible authorities.
Limited involvement of stakeholders in implementation of national programmes.
Limited cooperation by responsible agencies in international networks, leading to a
failure to bring in experience from abroad.
Slovenia
Lengthy procedures in selecting beneficiary organisations.
Long process for making claims for payment.
Problems with monitoring information systems.
Spain
Administrative and institutional arrangements have introduced severe complications
leading to late execution.
page 48
Draft A2E Synthesis Report
Portugal
Some evidence that the development of the social employment market has been
done in such a way that it becomes a social intervention rather than economic
integration into the labour market and so does not support the participation of the
vulnerable population in the labour market.
United Kingdom
Poor quality of targeting in relation to DWP and JobCentre Plus, with extensive
resource appearing to go to shorter term unemployed people more able to find work
on their own.
Denmark
Denmark starts with identifying needs for adjusting at the regional scale to changing
conditions in local labour markets. This requires close cooperation with regional
partners to understand the impact of global changes on local economies.
page 49
Finland
Broad Finnish approach is characterised by relevance and robustness of intervention
combined with flexibility which allows changes to be made in relation to economic
conditions.
- Government decided at beginning of 2009 to emphasise job retention and job
creation and ESF was redirected to projects helping this. Regional Offices of
Ministry of Employment and Economy were advised to favour calls emphasising
job retention and job creation both in relation to AP1 and AP2, the first time that
ESF measures had been so clearly aligned to government initiatives. Previously
ESF had been addressing structural challenges in Finland and the shortage of
labour principally due to the ageing workforce, high levels of retirement and a
contracting labour supply.
- Change of direction was possible because of flexibility in OP design, written in
broad terms with large funding categories.
- Change of direction was able to be implemented effectively because the
managing structure of the OP allowed for this.
- The OP design turned out to be robust, flexible and appropriate in the face of the
changing economy and labour market.
Czech Republic
Competent teams within beneficiary organisations with responsibility for managing
services.
Clarity of communication, division of responsibilities and the setting of clear rules for
employees within beneficiary organisations.
Hungary
In terms of processes, the IB is developing personalised relationships between IB
officials responsible for particular calls and project leaders.
Italy
Activities delivered in smaller territories do better perhaps due to the greater ease of
project management and closer relationships between stakeholders and agencies.
Institutional learning helps improve intervention effectiveness, including making more
use of evaluation findings which seem to be more prevalent in small territories.
Netherlands
The Municipality of Emmen has created an ESF office. The office coordinates
applications and the implementation of projects and serves as a base of reference on
ESF for all municipalities in the Dutch province of Drenthe.
The Training and Development Fund of the Fire Service has created a project
organisation that coordinates ESF programmes within this sector. The organisation
focuses on the provision of information on the quality of administration.
Poland
Prior experience of project providers and beneficiary organisations is a key factor in
facilitating effective implementation.
page 50
Draft A2E Synthesis Report
Spain
A number of the bodies administering ESF have great expertise. The OP to fight
discrimination is unique and has been identified as best practice in the EU.
- A number of the IBs are national level NGOs with significant expertise working
with vulnerable groups.
- At project level there are regional and local NGOs, trade union and employer
organisations again with great expertise and experience.
United Kingdom
The approach to contract and performance management has a bearing on effective
performance and delivery in respect of ESF provision. Important factors include:
- Openness and honesty in relationships between contract managers, prime or
lead contractors and their subcontractors or delivery partners.
- The use of performance targets used at different levels within the ESF delivery
system although, in a minority of cases, subcontractors and delivery partners to
feel that the targets given to them by prime and lead contractors were unrealistic.
- The use of result-based systems of payments to providers was perceived
beneficial in supporting performance improvement and encouraging providers to
focus on the achievement of positive results for ESF participants. However,
result-based systems were also seen by some as leading to a focus on achieving
outputs at the expense of ensuring quality in provision or targeting of individuals
perceived as being easier to help and more likely generate job results.
page 51
Engaging Hard to Reach Priority Groups
A2E interventions such as vocational training, confidence building, etc. often struggle
to achieve significant results for recipients. There is often however a prior issue. This is
how to get the more disadvantaged groups involved in the A2E activity in the first
place. General good practice in this area tends to focus on working through
organisations, typically in the non-governmental sector, who already work with the
particular sub-section of the population. This could include for example housing
organisations responsible for housing in areas where there are high concentrations of
disadvantaged groups. In relation to ethnic minority groups the engagement problem
can be severe.
Belgium
Programmes in the Brussels region engaged a high proportion of participants of non-
Belgian ethnic origin but at the same time achieved very good gross and net
employment results. This is unusual as this target group has proven previously hard
to engage and assist into employment.
Denmark
E-learning to increase labour market competencies in remote areas has been
effective.
United Kingdom
Quality outreach was important to securing additionality - the recruitment of inactive
black and ethnic minority participants required the partnership of many agencies.
Austria
Employment Companies are mainly targeted at long term unemployed giving
opportunity to get back to realistic workplace settings. At the same time, the people
are supported to help raise self confidence and self esteem. Evaluation evidence
suggests that these:
- Help raise employability but also create additional jobs for people furthest from
the labour market through secondary labour market.
- Provide a safe and confidence building atmosphere in a work situation making it
easier for furthest from the labour market to re-integrate into labour market.
- Work best for low level activities such as home and office cleaning services,
facilities management, parts, textile recycling, etc.
- Work less well where higher qualifications, work experience, physical fitness and
soft skills are required.
Hungary
The experience is that for more effective complex assistance to respond to complex
needs of some A2E target groups, it will tend to be associated with a lengthier
process before results are achieved. This has implications for how OPs are designed,
monitored and evaluated.
page 52
Draft A2E Synthesis Report
Ireland
Labour Market Activation Fund (LMAF) emphasises the need to focus on the
development of softer skills particularly for the long term unemployed or those whose
employment background had been very specific. The good practice approach is for
core skills development to be linked into an intervention pathway leading to
employment.
Italy
Tools that seem to work well have:
- High level of personalisation.
- High level of flexibility.
Ad Personam Service (Trento) and individual voucher system (Lombardy) operate
voucher systems which appear to give recipient capacity to purchase directly from a
network of accredited providers. Payment for services made to providers directly by
MA. This forces the PES and other service providers to look at clients as customers
with individual needs.
Other examples of tailor made (customised) activities include internships, work
experience, incentives and research fellowships. The key ingredients here are:
- A written agreement.
- Promotion of learning experiences in and out of the place of work.
- Development of CSR practices in offer of suitable opportunities by employers.
Poland
Particularly for most disadvantaged groups, delivery of a complex support package
with the right provider is good practice. One example is of an intervention where the
project includes psychological and group support, vocational advice, job agency,
internships and cooperation with employers. A key feature is that the project provider
has significant experience with the client group, with a customised approach to each
participant as well as time committed to working with participants families. Project
process was well planned and monitored especially in relation to recruitment, and
20% found employment, from a particularly disadvantaged group.
Romania
Customisation is achieved in some activities by the development of a systematic
ladder of progression through various measures to assist individuals back into regular
employment on an incremental basis.
United Kingdom
Personalised training programmes, involving one on one support for participants as
well as flexible and open-ended delivery were highlighted as a particular success.
page 53
Portugal
Programme Bero de Emprego supports the costs of replacing a worker on maternity
leave with an unemployed woman, diminishing some of the resistance to hiring
women. Half the substitute recruits end up being hired.
Young People
Austria
Due to the recession, greater emphasis has been put on getting young people into
employment. Apprenticeship training was introduced by the PES to make up for lack
of apprenticeships in companies, backed up by a government pledge that all young
people in Austria would be offered an apprenticeship. This pledge has been realised.
Youth employment measures help companies pre-select for apprenticeships, but
there is the danger that the apprenticeship measures themselves displace
conventional apprenticeship as companies get the taxpayer to pick up the cost.
Interventions significantly increased the number of young people with qualifications.
Young people with other issues received support through ESF co-funded measures
which helped build their self esteem and self confidence.
Denmark
There has been a focus on targeting youth education and the national target of 95 per
cent of each cohort completing youth education. This has been supported by ESF
funded activities such as Hold Fast which focused on using youth councillors as
mentors as well as other activities to avoid young people dropping out of education.
France
One of the ESF funded projects involves one to one work with job coaches which
facilitates the delivery of a customised service and a link to potential employers and
77% of recipients secured a permanent job. The key point is that job coaches work
both with the individual unemployed person and potential employer to make an
effective match.
Ireland
An example of targeting individual need and market opportunities is an assessment
and guidance programme for 1,000 unemployed people to provide individualised and
person centred education and training needs as well as matching jobseeker skills,
experience and interests with labour market opportunities.
Another example concerns the added benefits from work placements in training
programmes where there is clarity about goals, timelines and deliverables. This helps
both participants and employers understand what is the desired result and therefore
makes it more likely that the result will be achieved.
Both LMAF 2010 and review of labour market programmes emphasised the need of
education and training to give jobseekers skills to meet labour market and specific
industry needs, including opportunities emerging in new sectors prioritised in national
policy (e.g. renewables). Additionally, direct links to employment opportunities
through structured work experience and placement programmes are key success
factors.
page 54
Draft A2E Synthesis Report
Some project providers have engaged directly with employers to establish their skill
needs, to create or modify existing courses to meet these and then bring participants
onto those courses with a view to raising their employability.
Ireland
The range of partnerships and collaborative approaches involved in the
implementation of ESF A2E in Ireland were highlighted as adding significant value.
LMAF 2010 argues the value of the holistic approach to A2E. ...the mix of core and
specific industry and occupational focussed skills, the inclusion of a work placement
aspect, the provision of wider forms of support addressed to needs of individual
participants and the focus on progression rather than education and training courses
per se were identified as key characteristic of success.
United Kingdom
In the West Midlands the adoption of a consortium and network covering the whole
region was seen to have worked well allowing providers to learn from each other and
test fresh ideas and to develop a much keener understanding of the issues that each
of them are dealing with.
In Scotland, the Community Planning Partnership model, which to some extent
anticipates the Integrated Territorial Investment and Community Development
concepts advanced in the draft Regulations for 2014-20, was perceived to have
improved the efficiency of local employment services through improved coordination
and reduced duplication. Strategic commissioning was believed to represent better
value for money and to result in reduced administration compared with partnership
developed strategies and competitive bidding.
page 55
6 Community Added Value
7 A Framework to Describe the Community Added Value of the ESF (2008) Methodological Note: EC
Development, Social Affairs and Equal Opportunities DG, Evaluation and Impact Assessment Unit.
page 56
Draft A2E Synthesis Report
Germany
The Federal states have a substantial share of EU funding, a level of resource that
they would otherwise struggle to assemble. These can be deployed to meet the
needs of their regions.
European policy-orientation directs money into areas that would be served to a lesser
extent by national and regional OPs, e.g. in relation to migrants and asylum seekers.
ESF allows for more innovative and intensive support for groups such as those with
multiple disadvantages in the labour market.
EUs governance principles (including indicator based monitoring and controlling,
regular reporting and evaluation, networking and mainstreaming, and broader
stakeholder involvement) impact upon national administrations.
Ireland
ESF makes a substantial contribution to the resource available for A2E which has a
significant impact given the difficulties of the Irish Exchequer.
ESF helps to ensure that there remains a strong focus on gender, wider equality and
social inclusion in relation to A2E activities.
The Labour Market Activation Fund was also able to draw on experience of EQUAL
with innovation and new models of delivery.
ESF co-financing helps to ensure a level of planning, objectivity in target setting,
management, monitoring and reporting discipline that is unlikely to exist in its
absence.
Volume Effects
Volume effects emerge in the country reports as the most common form of CAV.
They are particularly significant for newer members of the EU that have more
limited public employment services and ALMPs.
They are also important however for older members of the EU and it is these
countries that secure the bulk of ESF A2E funding.
The importance of volume effects has clearly been heightened by the effect of
the economic crisis which has both brought forward a massive increase in
people requiring A2E assistance as well as a decline in the capacity of public
finances within Member States to resource the necessary interventions.
Some illustrations from some Member States are provided below.
page 57
CAV Volume Effects
Austria
ESF co-funding was seen by TEPs as a means of doubling their budgets for projects
for people furthest from the labour market.
Czech Republic
ESF significantly increases the total amount of resource allocated to A2E in the
range 75-100% of resources for activities delivered.
In the absence of these resources, it is estimated that 88% of recipients of PA2 and
90% of recipients of PA3 would suffer a deterioration in their situation.
Estonia
The increased volumes of A2E activities now being delivered would not be possible
without ESF support.
The sustainability of activities introduced and the capacity to introduce new activities
would not be impossible without ESF support.
Both of the above are intensified by the economic crisis, and the capacity to mitigate
the worst of the crisis would not have been possible without ESF support.
Portugal
The AIRs reveal widespread perception of considerable CAV from ESF. Levels of
activities, outputs and results would not be achievable with domestic resources alone
in relation to A2E and other policy fields.
Slovak Republic
ESF is critical for the delivery of A2E interventions, providing 54% of the funding
required to achieve the ALMP objectives of the Member State.
Slovenia
65% of the Slovene Active Employment Policy is funded by ESF and means that
existing interventions are much better resourced and assist many more final
recipients.
United Kingdom
Evaluation evidence suggests considerable added value in terms of increased
volumes of unemployed recipients.
Scope Effects
It is clear that scope effects are a very significant aspect of ESF A2E CAV, as the box
below illustrates. There are two elements here that stand out:
Support for specific groups (such as immigrants) that would not otherwise have
received as much support, or indeed any, in terms of A2E interventions.
Support for specific types of intervention, typically most costly and labour
intensive ones, which would be less likely to be available to A2E target groups.
page 58
Draft A2E Synthesis Report
Belgium
The ratio of ESF final recipients to potential recipients for 2010 amounted to 23% of
the unemployed, but 51% of disabled unemployed people.
Czech Republic
ESF projects both made it possible to work with larger number of individuals in target
groups, but also with completely new target groups.
Estonia
The at risk and other target groups associated with ESF funding support includes
some groups who would have received limited or no A2E services from the Member
State.
Lithuania
Particularly for clients with the potential to impose significant costs on public services
(e.g. ex-offenders), there is some evidence that interventions both created better
employment prospects but also helped with prevention (of crime in this instance).
Similar findings emerged on clients with addictions.
Malta
A number of the key interventions were completely new and additional to what would
otherwise have occurred, as they were not previously included in local plans. This
refers particularly to the programmes for women and for disadvantaged and
vulnerable groups.
Poland
CAV is connected with ESFs stronger focus on interventions aimed at the activation
of young unemployed persons in the labour market, and the setting up of indicators
and measurement processes. For this group policy on A2E has changed from passive
to active, and ESF has taken the lead and set standards on how to do it.
ESF funding for the central OP supports the development of interventions for groups
covered in Polish law which have specific features increasing their disadvantage e.g.
ex-offenders and members of Roma community. ESF supports tailored interventions
to meet the needs of these groups.
The Polish labour fund works with registered unemployed individuals, whereas ESF
interventions are addressed at workless people more generally including the inactive.
Slovenia
As a result of ESF a number of additional programmes have been added to the
Slovene Active Employment Policy, including on-the-job training for graduates,
subsidies for employment in the not-for-profit sector and support for specific
geographic areas.
Spain
ESF A2E support has led to a much more significant efforts in relation to the social
and labour market integration of immigrant groups.
United Kingdom
As a result of ESF support a range of key interventions are delivered, or delivered to
a much higher quality, than would have otherwise been the case. This includes
outreach activity and Intermediate Labour Markets, generally targeted at the more
disadvantaged groups of the population.
page 59
Role Effects
Role effects appear less frequently in the assessments of CAV, possibly due to the
substantial investment already made by the EC in innovation through programmes
such as EQUAL.
Austria
As the projects to receive ESF support had to bring forward new approaches to the
integration of the target groups role effects in terms of innovation were realised.
Estonia
The ability to develop innovative approaches (e.g. different training courses for
different target groups, measures to deal with disrupted studies, etc.) was stimulated
as a result of ESF support.
Finland
ESF funding is significant in triggering national innovations in enhancing employment
and skills. An example is the intervention introduced to promote job to job transitions
in redundancy situations.
A strategic evaluation found that 70% of activities would not have been launched
without ESF funding, and that it was particularly important in terms of promoting skills
and employability of jobseekers and finding new ways of delivering these results.
Process Effects
Process effects also appear less frequently as CAV in the country reports than might
have been anticipated, although it is not surprising that the three examples reported
below all refer to countries joining the EU in more recent times.
Hungary
The volumes of A2E interventions and recipients were significantly impacted by ESF,
but ESF has also contributed to experimentation and to reforming the practices of the
PES. There are considerable costs to the PES in making the required changes which
would not have been implemented without ESF support.
Malta
Accessing ESF required wide consultation with stakeholders and partners to
understand what kinds of interventions were required, and to customise these to local
circumstances.
The OP has made significant demands on institutions and systems, including
information and control systems. This applies to both public and private agencies.
Poland
The activities of the Human Capital OP contribute significantly to an increase in
common activities and approaches across public institutions. This cooperation is likely
to result in the identification and implementation of optimal solutions in relation to A2E
and Social Inclusion policies, as well as making for more effective operational working
across public institutions.
page 60
Draft A2E Synthesis Report
page 61
7 Evaluation data, approaches and methodologies
page 62
Draft A2E Synthesis Report
Output data is the most common type of data utilised with 133 evaluations
making use of this type of data. DE, ES, PL and UK in particular appear to make
extensive use of this data although this in part reflects the large number of
evaluations undertaken in these countries.
Other forms of secondary data are also widely used, with 113 evaluations
making use of results data based on monitoring, 112 making use of financial
data, and 95 making use of final recipient contact data.
Less common is primary research, with only 82 evaluations using results data
based on research.
All countries (for which we have information) make use of more than one type of
data. Many draw on all four types of data.
In terms of the accessibility, quality and reliability of data:
In most instances data were relatively accessible to experts, and hence to other
interested parties as required.
Data reliability was also not a great concern although system changes over the
life of the 2007-2013 programme meant that data currently available were often
not comparable with earlier data.
In relation to data based on research, some issues that were identified included:
The sample selected with some seeking statistical representation and others
aiming to capture a range of experiences (for example, through case studies).
The sample size with cost issues at play here. In particular, whilst overall
sample sizes may be statistically robust, they may not be sufficient to enable
analysis of specific target groups (e.g. ethnic minorities, individuals with a
disability, young people) or interventions.
Attrition in cohort surveys with diminishing participation in successive waves.
This requires reweighting of the survey results to bring them in line with the
original sample specification and to facilitate meaningful comparison between
survey waves.
However, it is clear from many of the country reports that experts struggled to find
data that related specifically to ESF A2E types of operations focussed on the types
of final recipients prioritised under A2E. This applied to expenditure, number of final
recipients and to employment results. On employment results, there was virtually no
information on the sustainability of employment secured following an ESF A2E
operation. Additionally, for employment results many country experts were dependent
on monitoring data provided by beneficiary organisations raising issues about the
reliability of the information.
Returning to the hierarchy of information that the evaluations could potentially provide:
Most weight, simply due to availability, was given to monitoring data of one kind
or another.
- The data available varied across Member States and evaluations and
therefore the capacity of evaluations to meet the most basic requirement (i.e.
to be able to identify the results and the cost of delivering these) was at
times, limited.
- In addition, most of the monitoring data available was provided by beneficiary
organisations with no independent verification. This limits the confidence
page 63
that can be placed in the results reported. Further problems result when
beneficiary and intermediate organisations maintain data on different systems
and/or interpret indicators in an inconsistent manner.
- The one key exception to this is where Member States have in place central
databases (often managed by PES or tax authorities) that contain information
on all adults of working age and evaluations have been able to draw on this.
As well as being more robust than data provided by beneficiary
organisations, this can allow some benchmarking to be undertaken.
However, this only appears to have been undertaken in a small number of
cases.
Few drew on primary research and only a sub-set of these will have used
counter-factual (control group or econometric modelling) techniques (the others
may, for example, simply have surveyed recipients about their experiences of
intervention). This means very little information is available on the additionality of
ESF-funded A2E interventions.
page 64
Draft A2E Synthesis Report
AT BE BG1 CY2 CZ1 DE DK EE ES FI1 FR GR1 HU1 IE1 IT1 LT1 LU LV2 MT3 NL PL PT RO1 SE SI SK UK Total
Final 7 3 2 - 3 13 5 2 5 1 4 4 6 2 14 3 0 - 0 0 11 0 2 0 0 0 8 95
recipient
contact
data
Financial 3 4 2 - 2 21 7 1 9 3 3 4 2 3 14 3 1 - 4 1 5 6 3 1 1 1 8 112
data
Output 5 4 3 - 2 25 3 3 9 1 3 4 7 4 16 3 0 - 4 1 13 5 3 1 1 1 10 131
data
Results 5 7 2 - 3 15 3 3 9 2 3 4 6 4 7 3 1 - 4 1 14 3 3 1 0 1 9 113
data
based on
monitoring
Results 6 5 2 - 1 18 0 2 0 1 1 0 5 1 8 1 0 - 0 0 15 0 2 0 0 0 14 82
data
based on
research
Notes:
1. Bulgaria, Czech Republic, Finland, Greece, Hungary, Ireland, Italy, Lithuania and Romania - Calculated by TERU based on individual evaluation information provided
in Annex.
2. Latvia and Cyprus information provided in Annex confusing so not included here.
3. Malta Table provided is for All AIRs reporting data. Country experts accessed AIRs for 2007-2011, so attached figure of 4 to each
page 65
Although a number of issues with data sources quality and reliability were identified in
some of the country reports, there are also some good examples to draw upon which
are captured in the box below.
Austria
Austrian monitoring data tend to be very accurate and evaluators generally have
access to a data warehouse (DWH) which gives an excellent breakdown of
participant data. This is run by the Austrian PES and Ministry for Labour, Social
Affairs and Consumer Protection. This provides anonymous person and company-
related information e.g. employment and unemployment episodes, periods out of
employment, etc. Research often is at the individual participant level. Evaluations can
draw on focus groups as well as surveys of participants.
In general, monitoring data are used to provide information on the absorption of funds
and overall progress. Research data are then used to explain the results.
Belgium
Evaluators in Belgium are able to draw on 3 different methods (beyond intervention
monitoring data) to collect information on results of ESF-funded A2E interventions.
- Crossroads Bank for Social Security which can identify whether an individual in a
given quarter of the year is on benefit, in paid employment, etc.
- PES Database which can show on a monthly basis whether an individual is still
unemployed or has left unemployment.
- Surveys of individual recipients.
Germany
In relation to general statistics, accessibility, quality and reliability are good. For
example, the PES has its own research institute and has considerable and lengthy
experience in providing, generating and analysing appropriate data. These data can
be readily disaggregated to quite small geographies.
In terms of programme monitoring data, Germany has a well established and
reasonably common approach built up from the individual projects, including
enterprises where these are the target recipients.
There is also a system which is simpler involving less documentation to be completed
by participants. This is for:
Interventions with low-level intensity.
Interventions addressing systems, rather than individuals or
enterprises/organisations.
Poland
There are two systems of data collection to help monitor the effectiveness of ESF in
Poland.
- National Information System (SIMIK) which collects data on OPs, payment
applications, non-compliance, etc. The system charts expenditure and
expenditure forecasts.
- The monitoring sub-system of ESF 2007 (PEFS) gathers data for monitoring the
achievement of goals through ESF interventions including information about final
recipients.
13 evaluations used participant surveys which are easy to conduct because PEFS
has contact details for final recipients. These surveys help measure intervention
effectiveness.
page 66
Draft A2E Synthesis Report
page 67
The focus being on approaches/methodologies based on monitoring data of one
kind or another which at best will be able to identify what the results have been
and the cost of achieving these. However, in practice, not all evaluations were
able to meet this most basic information requirement.
The approaches/methodologies used suggest limited efforts to benchmark
against national programmes, other interventions or unemployment benefit off-
flows.
Similarly, there has been limited use of counter-factual evaluation to establish
the additionality associated with ESF A2E expenditure. The limited number of
impact evaluations reflects the difficulties involved in designing and delivering
these, including accessing data particularly micro data; identifying, recruiting
and sustaining control groups including data protection issues; and the stage
in the programming period with most Member States focussing on monitoring
implementation rather than evaluating results.
page 68
Draft A2E Synthesis Report
page 69
Table 8 gives a good overview of the different approaches to evaluation. However, it is
essential to record a concern raised in a number of country reports. This relates to the
fact that there is no longer a requirement from the EC for MAs to carry out Mid-Term
Evaluations. Some country experts felt that this meant there is a risk that robust
quantitative information on the implementation of the 2007-2013 Programme may not
be available until 2013. Mid-Term Evaluations, as well as providing good quality
quantitative and qualitative data, allow an assessment to be made of any critical
changes in economic and labour market contexts, national budgetary positions, etc. In
addition, they can review the effectiveness of implementation and the structures for
managing this. All of this feedback allows MAs to consider carefully and on the basis of
robust evidence whether, to what extent and how to change their OPs. The AIR
reporting process does not generate this quality and robustness of feedback, but
perhaps the AIR process could be enhanced to provide some of the benefits that flow
from Mid-Term Evaluations.
Although there are some issues with the evaluation approaches and methods used to
try to identify the activities, outputs and results flowing from ESF A2E, there are also
good practice approaches in a number of Member States, captured in the box below.
page 70
Draft A2E Synthesis Report
Hungary
Recent evaluations have begun to take a more scientific approach and utilise more
demanding methodologies than those undertaken earlier in the OP.
- The dominant evaluation methodology is quantitative analysis based on
monitoring data although there is a slow move away from this orientation.
- Most of evaluations involve specifically focussing on A2E interventions and target
groups.
- There are some theory based and counter-factual impact evaluations, and it is
also quite common to find evaluation of results based on case studies or focus
groups, or surveys of final recipients.
However, impact evaluations are relatively rare because necessary data is missing or
hard to access, problems of identifying an appropriate control group and identifying
causality.
Latvia
Support for the implementation of regional action plans for promotion of employment
(WWS) is the flagship response of Latvian employment policy to the growth of
unemployment in recession. The principal aim of the programme was income support
for low income unemployed to bridge the gap of low or no income between jobs. The
WWS evaluation is an impact evaluation carried out on a counter-factual basis using
control groups. It also used content analysis, analysis of normative documents and
policy planning documents, a survey of households, a survey of municipalities,
interviews with representatives of municipalities, a survey of experts and analyses of
secondary statistical data.
Poland
The Polish approach to evaluating ESF A2E is multi-faceted.
- Most evaluations focus specifically on sub-set of A2E interventions and target
groups.
- To measure results, most evaluations use monitoring data, final recipient case
studies or focus groups and research or survey data with final recipients.
- Theory-based impact evaluations are common, but no counter-factual impact
evaluations are available.
- Most evaluations use quantitative, qualitative and other methods to facilitate
triangulation.
page 71
sometimes make it difficult for ESF-funded A2E interventions to be effectively
evaluated in this way.
8 EVALSED http://ec.europa.eu/regional_policy/sources/docgener/evaluation/evalsed/index_en.htm
9 Martini, A. (2009). Counterfactual Impact Evaluation: What It Can (and Cannot) Do for Cohesion Policy.
th
Paper to 6 European Conference on Evaluation of Cohesion Policy.
10 White, H. (2010). A Contribution to Current Debates in Impact Evaluation, Evaluation, Vol. 16, No. 2,
European Commission.
12 Reichardt, C. (2011). Evaluating Methods for Estimating Program Effects, American Journal of
Evaluation, Vol. 32, No. 2, pp246-272. Sage Publications on behalf of American Evaluation Association.
page 72
Draft A2E Synthesis Report
13 Crpon, B., Farracci, M., Joliret, G., and van den Berg, G. (2009). Active Labour Market Policy Effects in a
Dynamic Setting. Institute for Labour Market Evaluation (IFAU) Working Paper 2009:1.
page 73
Annex to Section 7: Analysis of evaluations based on Inventory
Overview
On the basis of updated inventories as of 19 July 2012 the broad position is that:
107 evaluation plans have been prepared.
675 evaluations have either been completed, are ongoing or are still at the
planning stage.
Of the evaluations:
- 413 are operational evaluations.
- 206 are strategic evaluations.
- 287 are exclusively process evaluations.
- 103 are exclusively impact evaluations.
The relatively limited number of impact evaluations helps explain the difficulty in finding
comprehensive data on results encountered in the country reports focussing on ESF-
funded A2E.
page 74
Draft A2E Synthesis Report
Table A6.1 shows how finalised and on-going evaluation approaches vary across
Member States.
More than half of the impact evaluations were carried out in 2 Member States
(HU and PL). This underlines the fact that impact evaluations are relatively
uncommon across most Member States.
Even in relation to process evaluations there is a high degree of concentration
with 2 countries (PL and IT) accounting for more than 50%.
There is a major job to be done to generalise both process and impact evaluations
across all Member States in sufficient quantities to guide programme implementation
and re-design where necessary.
page 75
Note: 1. Table shows count of evaluation approaches. Some evaluations listed in the Inventory do not
have an evaluation approach recorded (for example, if the evaluation is still in planning stage)
whilst others take multiple approaches and therefore have more than one approach recorded.
2. This data was extracted from the Inventories on 19 July 2012. The Inventories are updated on
an ongoing basis and therefore the numbers in this table are different from those in Table 8 which
were compiled using the data provided in the 27 CRs (which were finalised earlier in the year).
page 76
Draft A2E Synthesis Report
Table A6.2 sets out the geography of the evaluations across the Member States.
Clearly this is driven in large measure by the architecture of the OPs.
The most typical territorial level is national multi-regional accounting for 391 of
the evaluations.
There are 272 evaluations which are specifically regional.
page 77
Table A6.3 looks at the potential uses of the evaluations.
413 have potential operational value.
206 have potential strategic relevance.
page 78
Draft A2E Synthesis Report
8 Conclusions
page 79
work programmes where many of the recipients are relatively short-term unemployed.
Notwithstanding the recession it is still the case that a significant portion of the newly
unemployed tends to find work without the need of intensive programmes of support.
This raises issues about the most appropriate allocation of ESF resources between
those further from and closer to the labour market.
6. There is not much evidence coming through the country reports of a serious
appreciation at governmental level within Member States of the need to see better
service integration to support A2E across public employment services, health
services, social services, etc. but this is essential to ensure that when interventions
and services reach individuals and families who are particularly disadvantaged there is
a joined up approach.
7. The treatment of CAV in many of the evaluations underpinning the country reports
appears to be superficial. When the senior officials responsible for ESF are at the point
of beginning to bargain for the resources required for the programme beyond 2013,
someone in the Commission will be asking about the CAV arising from the 76 billion
committed! Is it too late for the Commission to do something about this in terms of
guidance to Member States? It is in the interest of Member States to be able to secure
a continuing significant flow of ESF and so they should be prepared to commit some
evaluation resource to a serious estimate of CAV provided the EC gives them a clear
definition and some working examples of what is meant by CAV. Somehow or other
more robust information on the CAV associated with ESF A2E must be collated.
8. More generally, on the evidence of the country reports, the evaluation effort around
ESF A2E is fragmented and generally lacking in robustness. The fact that Mid-
Term Evaluations are no longer required means they have generally not occurred and
so there is undue dependence on what is essentially monitoring information provided
by beneficiary organisations. By this stage of the 2007-2013 Programme the
Commission should be able to inspect much more comprehensive and robust
information on the results of ESF A2E activity across the 27 Member States. This
finding needs to be seriously considered in the design of the new ESF programmes
post-2013.
9. If good quality counter-factual evaluations can be designed, resourced and launched
this is clearly of great value in understanding the cost effectiveness and overarching
CAV of ESF A2E. However as a minimum there is a requirement for evaluations which:
Independently measure job entry and job sustainability results in a
consistent and meaningful manner, making use of national social security data
bases where possible to provide confirmation.
Benchmark job entry and job sustainability results against Member State
A2E programmes where these exist and/or statistical information on the number
of unemployed people leaving benefits to go into work based on data held by
public employment services and/or social security agencies.
These types of evaluation would help to answer the basic question is ESF A2E
getting significant number of people into jobs and helping them sustain these jobs?
They would provide a firmer base for the various types of counter-factual evaluations
essential for demonstrating the impact and CAV of ESF A2E interventions.
page 80
Draft A2E Synthesis Report
HU13 93.5 108,000 138,000 78.3 12.9% 28.5% 45.1 12.9% 28.5% 45.1
14
IE 1,051.8 213,346 59,996 59,996
IT15 1,711.8 311,400 21,250 21,250
16
LT 152.6 146,000 183,200 79.7 58,400 73,200 79.8 58,400 73,200 79.8
LU17 3.4 2,882 1,506 191.4 104 104
LV18 234.0 255,996 231,073 110.8 25-82% 30-40% 83-205% 25-82% 30-40% 83-205% 23% 24% 96%
PT22 229.4 64,854 124,088 52.3 13,367 11,600 115.2 13,367 11,600 115.2
RO23 13.7 38,288 235,000 16.3 N/A
page 81
MS Public Final Recipients Employment Results Job Entry Job Sustainability Jobs Created
Sector Cost
Achieved Target % Target Achieved Target % Target Achieved Target % Achieved Target % Achieved Target %
Committed
Target Target Target
(Meuro)
SE24 227.9 40,175 75,000 53.6 N/A
25
SI 161.3 85,912 205,900 41.7 56,700 56,700
SK26 143.2 260,030 23,319
27
UK 4,163.0 1,754,055 1,223,900 143.3 449,846 306,800 146.6 449,846 306,800 146.6
EU 23,233.3 12,581,364 2,403,339 2,403,339 6,950 63,228
Total
Note: Some employment results are reported in % terms (job entry rates) rather than absolute numbers in the evaluations that underpin the country reports.
General Note: yellow shading where no definition of employment results in country reports.
Country Notes:
1. Austria
Figures provided for PA2 (Combating unemployment). A2E activities are also undertaken under PA3b (Integration of people furthest from labour market). Data
is not available for PA3b but PA3 cost is 62.1 million, with 3,440 final recipients in 2010.
Employment results - Labour market status 3 months after end of measure.
2. Belgium
Figures estimated by Country Expert using range of assumptions as data only available at intervention level.
Cost Figures are Total Public Cost unclear whether Committed or some other measure.
Employment results - Labour market status 6 months after leaving (except BE-MS programme which is 3 months).
3. Bulgaria
Cost - Allocated spend (taken from OP HRD Operational Plan) for PA1 and PA2. Figure in brackets is Total Public Cost for PA1 (as reported in AIR 2010).
Final participants - Data for PA1 only. Data taken from AIR 2010.
Employment results Data for PA1 only. Measure is no. of people included in employment after successfully passing vocational or key competencies. Data
taken from AIR 2010.
4. Cyprus
Cost - Total Allocation of Funds (EU Contribution). Data taken from AIR 2010. Alternative figures provided in CR include Total Budget for OP EHCSC (149.7)
and Total Public Cost PA2 (49.9).
Final recipients Participants. Data taken from AIR 2010.
Employment results Job placement rate (people served by PES network). Data taken from AIR 2010.
5. Czech Republic
All figures taken from AIR 2010.
Cost Total Public Cost.
page 82
Draft A2E Synthesis Report
Final recipients Measure Number of persons who get any kind of support during the project realisation. Sum of figures for PA2, PA3 and OP Prague
Adaptability PA2.
Employment results Measure Number of jobs calculated as full time jobs created within the project (OP HRE PA2) and Number of jobs calculated as full
time jobs created for the persons vulnerable at the labour market within the project realization (OP HRE PA3).
6. Germany
Cost Calculated by TERU using following figures reported in CR. Table 3 - 2,686,837,172 allocated to A2E based on Priority Codes and Table 5 - 37.9%
cumulative commitments reported as a share of original allocation by Priority Codes presented within the OP planning documents.
Final recipient figures are based on 9 evaluations (out of 25) and employment results are based on 4 evaluations. As such these figures will be a significant
underestimate. Evaluation periods (e.g. to end June 2010, end September 2010, etc.) vary. No definition for employment results. Some % results are also
provided in main body of report.
7. Denmark
Data from AIR 2010.
Cost - Figures in country report given in Danish Krona. Converted using exchange rate (taken from FT.com) on 08/06/12 1DKK = 0.13450.
Employment results - Persons that have become employed, increased employability or increased competence levels.
8. Estonia
Cost Total Public Cost. Data from AMR 2011. Total Public Cost for 2007-13 157.2.
Final recipients Calculated by TERU using data from Table 3 Target groups and their participation in different measures of priority axis 3 Good quality and
long working life Finishing including cancelling for 2009 and 2010. Data also available for nos. starting 90,013. Data from AMR 2009 and 2010.
Employment results - Found employment and started work after participating in measures. Figures related to Measures 1.3.1 and 1.3.3 only (no data available
for 1.3.2, 1.3.4; and 1.3.5. Data from AMR 2011.
9. Spain
Figures drawn from AIR.
Cost Total Committed to 2010.
Employment results No definitions provided so uncertain what type of employment results.
10. Finland
Cost Data provided to Country Expert by Ministry of Employment and the Economy.
Final recipients - Participants in Priority II (Promoting access to employment and sustainable inclusion). Data from Strategic Evaluation (December 2011).
Employment results - Employed in open markets (26.9%) and employed with measures (9.8%) 6 months after end of measure. Figures for 3 months also
available. Relates to period 01/10/2009 to 30/09/2010. Data from MEE.
Job created Priority (Developing work organisations, labour force and entrepreneurship). Data from Strategic Evaluation (December 2011).
11. France
Cost ESF funding committed by 30/12/2010 for Mainland OP (data from AER 2011) and Total certified ESF support by December 2010 for OP La Reunion
(data from AIR 2010). Commitments for OP Priority Axis Increase participation to the labour market and decrease structural unemployment (considered by
Country Expert good proxy for A2E) are 2,516.9 (data from AIR 2010).
Final recipients Figures for OP Mainland France (AIR 2010) and OP La Reunion (Mid-term Evaluation covers period to end 2009).
Employment results OP Mainland France final recipients accessing employment (AIR 2010) plus OP La Reunion (no definition given) (AIR 2010 La
Reunion).
12. Greece
page 83
Commitments to OP HRD (data from OPs, AIRs and own work and analysis). No data available on commitments to OP Education and Lifelong Learning and
OP Administrative Reform. For information, Allocations/Payments for each OP are: OP HRD 1,039.4m/372.4m; OP E&LL 1,034.3m/184.2m; OP
Administrative Reform 8.5m/2.1m.
Final recipients - For OP HRD and OP E≪ 2007-10. Data from AIRs, own data and analysis.
Employment results No data provided in CR.
13. Hungary
Cost Spend for OP SROP Priority 1 (A2E) to end 2010. Data from AIR 2010. Note: Data on Commitments not available.
Final recipients Involved recipients for OP SROP Priority 1 to end 2010. Data from AIR 2010.
Employment results - Employment 180 days post completion of assistance. Source of data unclear from text.
14. Ireland
Cost OP HCI total cost to end 2010.
Final recipients - Data from Mid-term evaluation HCI OP. Data also available from Review of Labour Market Programmes and Evaluation of LMAF but potential
overlap so excluded.
Employment results Into employment, education or training. Data from Mid-term evaluation HCI OP. Data also available from Review of Labour Market
Programmes and Evaluation of LMAF but potential overlap so excluded.
15. Italy
Cost - Includes all Regional and 2 (out of 3) National OPs. Data from AIR 2010.
Final recipients Includes Regional OPs only. Data from AIR 2010.
Employment results - Calculated by summing employment results in the evaluation reports provided in Annex of CR. Due to data gaps, this figure is likely to be
a significant underestimate. Definition of employment unclear.
16. Lithuania
All data taken from SFMIS.
All data to end 2010.
Cost Public Sector Cost Committed.
Employment results No definition provided.
17. Luxembourg
All figures given to 2010.
No source for data given but thought to be AIR 2010.
Cost Total Public Cost.
Employment Results No definition provided.
18. Latvia
Source: http://www.esfondi.lv
Cost Total Public Cost at 31.03.12.
Final recipients To 31.12.11. Includes Measures 1.3.1.1.3, 1.3.1.1.5; 1.3.1.2; 1.3.1.4, 1.3.1.5, 1.4.1.1.1 and 1.4.1.1.2.
Employment results To 31.12.11. Includes Measures 1.3.1.1.3, 1.3.1.2, 1.4.1.1.1 and 1.4.1.1.2. See below for definitions.
Job sustainability To 31.12.11. Includes Measures 1.3.1.1.3 The share of recipients employed within 6 months after the training, 1.4.1.1.1 Share of people
supported by complex integration measures economically active after 6 months and 1.4.1.1.2 Share of people supported with measures for specific target
groups who were economically active after 6 months.
Jobs created To 31.12.11. Includes Measure 1.3.1.2 Share of recipients who have started business within 6 months after the training for self -employed and
small businesses.
page 84
Draft A2E Synthesis Report
19. Malta
Data from AIR 2010. Data to end 2010.
Cost Total Public Cost Committed.
Employment results - No. of final recipients achieving an employment result or further study 6 months after receiving assistance.
20. Netherlands
Data from various evaluations.
Figures for 2007-09 - Action A (2007, 2008, 2008-2), Action B (2007, 2008, 2009) and Action C (2007, 2008).
Cost Total Public Cost.
Employment results No definition given.
Jobs sustained Sustained employment at 6 months (Actions A and C).
Different figures given for Action C in text compared to Annex 1 - Final recipients 26,871/6,430; Employment results 9,050/11,680.
21. Poland
Data from AIR 2010. Progress to 2010.
Cost Public Cost Committed.
Employment results No definition provided.
Targets (final recipients and employment results) to end of programming period.
22. Portugal
Data from AIR 2010. Figures for 2007-2010.
Cost Total Public Cost.
Employment results No definition provided. Figures for POPH only (Rumos and Pro-emprego not available).
23. Romania
Data from AIR 2010.
Cost Total Public Cost.
Final recipients - CR expresses concerns about accuracy of participant numbers for PA6 so this may be an overestimate.
Target data calculated by TERU based on figures in CR.
Employment results No data provided in CR.
24. Sweden
Data from AIR 2010.
Cost Total Public Cost (including public co-financing).
Final recipients Unclear from text but targets thought to be to 2013.
Employment results No data provided.
25. Slovenia
Source of data unclear although most likely AIR 2010.
Cost Total Public Cost on 31.12.2011 (ESF and SLO contribution).
Final recipients Unclear from text but targets thought to be to 2013.
Employment results Nos. remained/got employment after their contractual arrangements concluded. Includes some in employment when commenced
participation.
26. Slovak Republic
Data from AIR 2010.
Cost Total Public Cost (ESF).
page 85
Employment results - Jobs created by beneficiary organisations.
27. UK
Data from AIR 2010 (except HIPP AIR 2009).
Cost Public Sector Cost Committed.
Employment results Entering employment. Note Table 1 in UK CR presents data for England based on indicator "in employment 6 months following
completion" and "entering employment" for Scotland, Wales and NI. For consistency, the English job entry figures (taken from AIR 2010) have been used in this
report in place of the employment at 6 months.
page 86
Draft A2E Synthesis Report
page 87
MS Assess- Careers Personal Deving Vocation- Work ILMs/ Employ- Support Support Childcare Post job Support Training Inst. Others
ment of and devment key al skills exp./ social ment for self- to find and entry for up- for dev.
A2E labour and employ- devment intern- firms subsidies employ- work, job family support skilling existing (PES,
needs market confidenc ability ships ment/ search/ support (prog- employee etc.)
info., e building skills for start-up interview ression) s
advice work- skills
and place
guidance
MT 19
NL
PL 20
PT 21
RO22 22b
a
SE23a 23b
SI 24
25
SK
UK
EU 8 10 6 17 23 15 6 11 14 7 17 3 3 9 9 13
27
Notes: EU-12 countries shaded in blue.
Country Notes:
1. Austria Apprenticeships.
2a. Belgium Actions to improve access to work for women.
2b. Belgium Validation of skills and competencies.
3. Bulgaria Provision of social assistant/home helper.
4. Cyprus Support for women with caring responsibilities.
5a. Czech Republic Interventions being used unclear from country report.
5b. Czech Republic Priority Axis 3 included measures to improve access to employment of women access to childcare and care of dependent persons.
page 88
Draft A2E Synthesis Report
6. Germany Programmes to reconcile family life and work (e.g. enhancing child care, local networks for families).
7. Denmark Interventions being used unclear from country report.
8. Denmark Education and training.
9. Estonia Training of disabled childminders.
10. Spain Country report provides limited details on interventions. Others include ALMP and research scholarships for graduates. In addition, personal itineraries
mentioned in section on Effectiveness of Specific Interventions.
11. Finland Apprenticeships.
12a. France Reconciling work-life balance, women in employment.
12b. France Apprenticeships; Reinforced accompaniment of jobseekers; Workers mobility and outplacement; Sectoral economic restructuring; Innovative and
preventative HR initiatives; Local Employment and Inclusion Plans.
13a. Greece Including social entrepreneurship, cooperatives and social firms.
13b. Greece Measures to improve access to employment and increase sustainable participation and progress of women in employment to reduce gender based
segregation in the labour market and to reconcile work and private life, such as facilitating access to childcare and care for dependent persons.
14. Ireland Further education; second chance education; higher education.
15a. Italy - Measures to improve access to employment and increase sustainable participation of women in employment to reduce gender segregation in the labour
market and to reconcile work and private life.
15b. Italy Financial support for training courses; Research grants; Learning pathways within compulsory education; Training vouchers; Services vouchers; Abroad
mobility; Higher education; Permanent education; Services provided by PES; Job vouchers.
16a. Lithuania Measures for better adjustment for professional and family obligations; Instructing and consulting of employers with intention of provide working
conditions that would be more favourable to family life.
16b. Lithuania Maintenance of workplaces; Social rehabilitation services.
17a. Luxembourg No section in country report on interventions. Data drawn from text on target groups which indicated what type of interventions had been most
effective. Very little information provided.
17b. Luxembourg Objective to increase womens employment rate.
18a. Latvia Help getting childcare place.
18b. Latvia Support for taking qualification tests; Improving competitiveness; Special support for people with disabilities; Grants for travel and equipment; Services of
ergotherapist/signing translator (disabled young people with hearing or vision difficulties); Allowance for assistance services for 10 hours a week (disabled young people
with physical disabilities); Negotiate with employer (people with disabilities); Set up workplace to suit special requirements (people with disabilities).
19. Malta Targeted actions supporting and lifecycle approach to work; Training and support aimed at women; Facilitating families benefiting from childcare.
page 89
20. Poland Psychological and advisory support; Support for voluntary work; Support for initiatives to increase geographic mobility; Seasonal works.
21. Portugal Job clubs.
22a. Romania No information on interventions provided in country report. Lists priority intervention codes Modernisation of labour market institutions; Active and
preventative measures to support employment; Improving equal access to employment; Modernisation of Public Employment Service; Promoting active employment
measures; Promoting social inclusion; Promoting equal opportunities on the labour market; Active employment measures for unemployed and inactive persons including
persons from the rural areas involved in subsistence agriculture, managers and employed persons; Promotion of equal opportunities on the labour market, addressing
mainly women as a disadvantaged group; Transition from school to active life; Design, introduction and implementation of reforms in education and training systems.
22b. Romania Promotion of equal opportunities on the labour market, addressing mainly women as a disadvantaged group.
23a. Sweden No information on interventions available.
23b. Sweden Initiatives contribute to EES by focusing on 4 aims including Contribute towards the development of women and men who are furthest from the labour
market so they can start work or come closer to the labour market, and broaden opportunities for women and men to work on the basis of their pre-conditions.
24. Slovenia Education/training of employed people; Preparation of vocational standards; Training of employees temporarily waiting for work; competence centres for
HRD; Company scholarship schemes; Co-financing of PhD studies; Subsidies for shorter working hours; Slovene qualification framework; Co-financing projects of social
partners for increasing working conditions; Employment of long-term unemployed people, older unemployed people, young unemployed people and hard-to-employ
people; Institutional training for national vocational framework; On-the-job training (including of candidates for graduation); Development of aids for lifelong career
orientation.
25. Slovak Republic No information on interventions available.
26. 8 specifically note childcare as support; 8 implied.
page 90
Draft A2E Synthesis Report
page 91
MS Unemploy- Long term Existing Women Young Young Older Graduates Ethnic Migrants/ Roma Disabled People Others
ed unemploy- employees people people people minorities refugees/ people with
ed at risk NEET or at asylum multiple
risk of seekers disadv.
NEET and
complex
needs
PL 35 36
37
PT
38 39 39
RO 40
SE 41 42 43
44a 44b
SI
SK45
UK 46
EU 27 18 14 10 17 19 4 14 5 5 11 3 12 15 13
Notes: EU-12 countries shaded in blue.
Country Notes:
1. Austria Low qualified.
2. Bulgaria Including school and university students.
3. Bulgaria Employees.
4. Cyprus Vulnerable groups.
5. Cyprus Over 30s.
6. Czech Republic Disadvantaged people.
7. Czech Republic - War veterans; Persons taking care of dependent person; Persons before/after release from prison; Young people in care; Parents on or returning
from maternity/ paternity leave; Job applicants; Employers; Institutions, social partners and cooperating organisations.
8. Denmark Including inactive.
9. Denmark Employed persons.
10. Estonia Including inactive.
11. Estonia Including persons who do not know the Estonian language (i.e. ethnic Russian).
12a. Estonia Employees.
12b. Estonia Families of disabled persons; persons released from prison; Other vulnerable persons; Self-employed.
13. Spain Country report notes that information on target groups in evaluations is scarce.
14. Spain Vulnerable groups.
page 92
Draft A2E Synthesis Report
page 93
Table A6: Effectiveness of ESF A2E Interventions
% Recipients Cost Per % of Cost Per
Achieving Employment Recipients Other
Employment Result Achieving Progression
Result Other
Progression
AT (Austria)
A2E Aggregate 34.9
Apprenticeships (PA2) 74
SOB/GBPs1 (PA2) Generally <20
Production schools 302
(PA2)
PA3b3 9 ( first labour 7 (other
market) vocational
15 (second training)
labour market
transitional
employment)
Young people (PA2) 67
Males 594
Females 714
Production schools
(PA2)
With school 35
leaving certificate
Without school 23
leaving certificate
Belgium (BE)5
A2E Aggregate Range 26-81
Vocational training (BE- 32
VL)
Job training and 39.8
coaching (BE-VL)
Females (BE-B) 386
Under 25s (BE-B) 37.66
Bulgaria (BG)
A2E Aggregate 35.9% 35,669
Cyprus (CY)
A2E Aggregate 16.5
PES 5.97
PES young people 8.57
Czech Republic (CZ)
A2E Aggregate
page 94
Draft A2E Synthesis Report
page 95
% Recipients Cost Per % of Cost Per
Achieving Employment Recipients Other
Employment Result Achieving Progression
Result Other
Progression
Vocational labour 36.911 6.412
market training
Total subsidised 41.111 4.512
employment +
vocational LM training
Coaching labour 30.411 2.612
market training
France (FR)
A2E Aggregate13 32.7 1,483 68.314
15
Local employment and 34
inclusion plans
16
Job coaching in Seine 37
Maritime
Greece (GR) No data
available
A2E Aggregate
Hungary (HU)
A2E Aggregate 12.917 -
Ireland (IE)
A2E Aggregate 18 28.1 13,649
Labour market 49 48,948 16 153,202
programmes 19
Labour market 29 into 4321
activation fund 20 employment
13 into self-
employment
Specific skills training22 58 13,64023
Traineeship22 72 20,64023
Community Training 41 53,27123
Centres22
Italy (IT)
A2E Aggregate 6.8 80,555
Lithuania (LT)
A2E Aggregate 40.0 2,613
ALMP Assistance to 36
acquire professional
skills24
ALMP Employment 30
subsidies24
ALMP Public works24 14
page 96
Draft A2E Synthesis Report
Latvia (LV)
A2E Aggregate Range 25-82 -
Training of unemployed 21.8
and job seekers
Phases 1 and 2 26
Training of unemployed 95
and job seekers
Phase 3 (training and
internships for
unemployed graduates
25 and under)
Malta (MT)
A2E Aggregate 12.5 22,797
27
PA3 (Promoting an 12.5 5.428
equal and inclusive
labour markets)
interventions and
projects
Training and support 029 52.630
for women
Support for vulnerable 1127 3031
groups
page 97
% Recipients Cost Per % of Cost Per
Achieving Employment Recipients Other
Employment Result Achieving Progression
Result Other
Progression
Employment aid 9432 14,395
scheme (wage subsidy
to employers to
incentivise recruitment
from
disadvantaged/disabled
groups)
Netherlands (NL)
A2E Aggregate 42.1 5,306
33,34
Action A (projects that 5
equip people with large
distance to labour
market with skills and
increase prospects for
sustainable integration)
Action B (projects that 1633 6235
improve the position of
prisoners, people on a
hospital order and
juvenile delinquents on
the labour market)
Action C (programmes 71.433
that promote the
inclusion into the labour
market of students
enrolled in professional
education and special
secondary education)
Poland (PL)
A2E Aggregate Range 18-45
Priority I, SO 2: To 20.8
broaden the impact of
ALMP
Priority VI, SO1: 45.3
Increasing coverage of (employed)
Labour Market Active 19.8 (self-
Policy employed)
Priority VI, SO2: 39.7
Increasing employment (employed)
among young people 8.36 (self-
employed)
page 98
Draft A2E Synthesis Report
page 99
% Recipients Cost Per % of Cost Per
Achieving Employment Recipients Other
Employment Result Achieving Progression
Result Other
Progression
Persons already 100
employed
Developing new 63 (including
employment employed and
opportunities self-employed)
Unemployed persons 55
Slovak Republic (SK)
A2E Aggregate38
United Kingdom (UK)
A2E Aggregate39 25.6 9,254
Note: Cells shaded in green have been calculated by authors using data from Table 2 (i.e. % Recipients
Achieving Employment Result = Employment Results Achieved / Final Recipients Received; Cost per
Employment Result = Public Sector Cost Committed / Employment Results Achieved). Given gaps in data,
care must be taken in interpreting these results.
Country Notes:
page 100
Draft A2E Synthesis Report
for taking qualification tests, training at a new workplace, improving competitiveness and special
support for people with disabilities. % of recipients employed within 6 months after the training.
27. Malta - % of final recipients achieving employment result or engaged in further education (by end
2010).
28. Malta - % of final recipients achieving a soft result such as increased personal capacity or other
progress towards employment not covered by hard results (by end 2010).
29. Malta Too early to measure employment at 6 months at end 2010.
30. Malta - % of final recipients gaining soft result, a qualification or certificate of attendance.
31. Malta - % of final recipients gaining a qualification or certification in capacity building.
32. Malta - % retained in employment after end of placement.
33. Netherlands Achieved employment results.
34. Netherlands Country expert argues employment outcomes low as most projects had not completed
at time of analysis (2007, 2008 and 2008-2).
35. Netherlands - % obtaining qualification or partial qualification (e.g. diploma).
36. Portugal Figures for POPH only (Rumos and Pro-emprego not available), 2007-2010.
37. Portugal Figures for 2010.
38. Slovak Republic Aggregate data only available for jobs created by beneficiary organisations.
39. UK Some data for nations/regions (e.g. West Wales) also available.
page 101
Table A7. Evaluation data sources
AT BE BG1 CY2 CZ1 DE DK EE ES FI1 FR GR1 HU1 IE1 IT1 LT1 LU LV2 MT3 NL PL PT RO1 SE SI SK UK Total
Final 7 3 2 - 3 13 5 2 5 1 4 4 6 2 14 3 0 - 0 0 11 0 2 0 0 0 8 95
recipient
contact
data
Financial 3 4 2 - 2 21 7 1 9 3 3 4 2 3 14 3 1 - 4 1 5 6 3 1 1 1 8 112
data
Output 5 4 3 - 2 25 3 3 9 1 3 4 7 4 16 3 0 - 4 1 13 5 3 1 1 1 10 131
data
Results 5 7 2 - 3 15 3 3 9 2 3 4 6 4 7 3 1 - 4 1 14 3 3 1 0 1 9 113
data
based on
monitoring
Results 6 5 2 - 1 18 0 2 0 1 1 0 5 1 8 1 0 - 0 0 15 0 2 0 0 0 14 82
data
based on
research
Notes:
1. Calculated by TERU for Bulgaria, Czech Republic, Finland, Greece, Hungary, Ireland, Italy, Lithuania and Romania based on individual evaluation information
provided in Annex.
2. Latvia and Cyprus information provided in Annex confusing so not included here.
3. Malta Table provided is for All AIRs reporting data. Country experts accessed AIRs for 2007-2011, so attached figure of 4 to each
page 102
Draft A2E Synthesis Report
14 i.e. evaluation that is based upon administrative resources rather than research.
15 i.e. evaluation based upon very small scale final recipient research and useful for investigating qualitative issues (may form part of a more broadly-based evaluation).
16 i.e. evaluation research through representative final recipient survey examining the gross results.
page 103
Evaluation of Evaluation of Evaluation of Evaluation of Theory Based Counter-factual Other Impact
Management Results Results Results Impact Impact Evaluation
and Using based on final based upon Evaluation Evaluation
Implementation Monitoring recipient Case sample final
Processes Data14 Study / Focus recipient
Group15 research16
target group, a (ES 4) (FR 0) (ES 5) (FR 0) (ES 4) (FR 0) (ES 1) (FR 0) (ES 0) (FR 0) (ES 0) (FR 0) (ES N/A3) (FR
subset of which (GR 5) (HU 1) (GR 6) (HU 1) (GR 0) (HU 1) (GR 0) (HU 0) (GR 6) (HU 1) (GR 0) (HU 0) 0) (GR 5) (HU
includes A2E (IE 0) (IT 0) (LT (IE 0) (IT 2) (LT (IE 0) (IT 0) (LT (IE 0) (IT 2) (LT (IE 0) (IT 0) (LT (IE 0) (IT 0) (LT 0) (IE 0) (IT 0)
final recipients. 1) (LV 0) (MT 0) 1) (LV 0) (MT 0) 1) (LV 0) (MT 0) 1) (LV 1) (MT 0) 0) (LV 0) (MT 0) 1) (LV 0) (MT 0) (LT 0) (LV N/A3)
(NL 0) (PL 1) (NL 0) (PL 3) (NL 0) (PL 2) (NL 0) (PL 3) (NL 0) (PL 3) (NL 0) (PL 0) (MT 0) (NL 0)
(PT 05) (SE (PT 0) (SE 0) (PT 0) (SE 0) (PT 0) (SE 0) (PT 0) (SE 0) (PT 0) (SE 0) (PL 0) (PT 0)
6
N/A ) (SI 0) (SK (SI 0) (SK 0) (SI 0) (SK 0) (SI 0) (SK 0) (SI 0) (SK 0) (SI 0) (SK 0) (SE 0) (SI 0)
0) (UK 4) (Cell (UK 1) (Cell (UK 1) (Cell (UK 2) (Cell (UK 1) (Cell (UK 0) (Cell (SK 0) (UK 0)
total 18) total 29) total 12) total 12) total 13) total 3) (Cell total 5)
Evaluation (AT 2) (BE 16) (AT 1) (BE 9) (AT 1) (BE 0) (AT 2) (BE 0) (AT 1) (BE 0) (AT 0) (BE 0) (AT 0) (BE 0)
which focuses (DE 0) (DK 7) (DE 0) (DK 0) (DE 0) (DK 0) (DE 0) (DK 0) (DE 0) (DK 0) (DE 0) (DK 0) (DE 0) (DK 0)
on intervention (ES 2) (FR 0) (ES 3) (FR 1) (ES 1) (FR 1) (ES 0) (FR 0) (ES 2) (FR 0) (ES 0) (FR 0) (ES N/A3) (FR
type where (GR 5) (HU 0) (GR 6) (HU 0) (GR 0) (HU 0) (GR 0) (HU 0) (GR 5) (HU 0) (GR 0) (HU 0) 0) (GR 0) (HU
subset of the (IE 0) (IT 8) (LT (IE 0) (IT 5) (LT (IE 0) (IT 1) (LT (IE 0) (IT 3) (LT (IE 0) (IT 0) (LT (IE 0) (IT 0) (LT 0) (IE 0) (IT 0)
target group are 0) (LV 0) (MT 4) 0) (LV 0) (MT 4) 0) (LV 0) (MT 0) 0) (LV 0) (MT 0) 0) (LV 0) (MT 0) 0) (LV 0) (MT 0) (LT 0) (LV N/A3)
A2E final (NL 0) (PL 2) (NL 0) (PL 2) (NL 0) (PL 2) (NL 0) (PL 2) (NL 0) (PL 3) (NL 0) (PL 0) (MT 0) (NL 0)
recipients. (PT 05) (SE (PT 0) (SE 0) (PT 0) (SE 0) (PT 0) (SE 0) (PT 0) (SE 0) (PT 0) (SE 0) (PL 0) (PT 0)
N/A6) (SI 0) (SK (SI 0) (SK 0) (SI 0) (SK 0) (SI 0) (SK 0) (SI 0) (SK 0) (SI 0) (SK 0) (SE 0) (SI 0)
0) (UK 2) (Cell (UK 2) (Cell (UK 1) (Cell (UK 2) (Cell (UK 2) (Cell (UK 0) (Cell (SK 0) (UK 0)
total 48) total 33) total 7) total 9) total 13) total 0) (Cell total 0)
Total 148 165 57 70 63 14 18
Notes:
page 104
Draft A2E Synthesis Report
Romania
2. Belgian summary table does not include row for Evaluation specifically focusing on subset of A2E interventions and target groups so figures of zero have been
allocated to each cell in this row for Belgium.
3. Spain/Latvia do not provide any information for Other impact evaluation column.
4. Malta Table provided is for All AIRs reporting data. Country experts accessed AIRs for 2007-2011, so attached figure of 4 to each category.
5. Portugal summary table only contains row Research specifically focusing on subset of A2E interventions and target groups so figures of zero have been allocated to
each cell in the other rows.
6. Sweden Data for Evaluation/analysis of management and implementation process unclear.
7. Estonia has only provided summary data for each column so not included in table. For reference, 1 evaluation of management and implementation processes; 3
evaluations of results using monitoring data; 1 evaluation of results based on final recipient case study/focus group; 2 evaluation of results based upon sample final
recipients; no theory-based, counter-factual or other impact evaluations.
page 105
Glossary
Access to Employment (A2E) is one of the key policy fields for the 2007-2013 ESF
Operational Programmes (OPs) and is described as: enhancing access to
employment and the sustainable inclusion in the labour market of job seekers and
inactive people, preventing unemployment, in particular long term and youth
unemployment, encouraging active ageing and longer working lives and increasing
participation in the labour market (Art. 3 of the ESF Regulation No 1081/2006).
Beneficiary: an operator, body or firm, whether public or private, responsible for
initiating or initiating and implementing operations. In the context of aid schemes under
Article 87 of the Treaty, beneficiaries are public or private firms carrying out an
individual project and receiving public aid (Art. 2 of the ESF Regulation No 1081/2006).
Community Added Value: The definition used is based on the methodological note
prepared by the Evaluation and Impact Assessment Unit in October 2008 which
identifies 4 key dimensions of Community Added Value (CAV). CAV measures the
difference made by ESF A2E funding and programmes compared to the efforts of
Member States alone. There are four key dimensions: volume effects, scope effects,
role effects and process effects.
Volume effects: ESF action adds to existing action, either by supporting
national action in general (mirroring) or specific areas of national policy
(boosting).
Scope effects: ESF action broadens existing action by supporting groups or
policy areas that would not otherwise receive support
Role effects: ESF action supports local/regional innovations that are taken up at
national level or national innovative actions that are then mainstreamed
Process effects: ESF action influences Member States administrations and
organisations involved in the programmes.
Final Recipient: 'Final recipients' refer to participants (i.e people) in supported
operations. These are to be clearly distinguished from beneficiaries. Beneficiary is
defined in Art. 2(4) of Council Reg. (EC) No 1083/2006 as: "an operator, body or firm,
whether public or private, responsible for initiating or initiating and implementing
operations. In the context of aid schemes under Article 87 of the Treaty, beneficiaries
are public or private firms carrying out an individual project and receiving public aid".
Beneficiary can e.g. be a NGO implementing an ESF-funded project providing services
for final recipients (participants).
Intermediate body: any public or private body or service which acts under the
responsibility of a managing or certifying authority, or which carries out duties on behalf
of such an authority vis--vis beneficiaries implementing operations (Art. 2 of the ESF
Regulation No 1081/2006).
Operation: a project or group of projects selected by the managing authority of the
operational programme concerned or under its responsibility according to criteria laid
down by the monitoring committee and implemented by one or more beneficiaries
allowing achievement of the goals of the priority axis to which it relates (Art. 2 of the
ESF Regulation No 1081/2006).
Operational Programme: document submitted by a Member State and adopted by the
Commission setting out a development strategy with a coherent set of priorities to be
carried out with the aid of a Fund, or, in the case of the Convergence objective, with
page 106
Draft A2E Synthesis Report
the aid of the Cohesion Fund and the ERDF (Art. 2 of the ESF Regulation No
1081/2006).
Priority Axis: one of the priorities of the strategy in an operational programme
comprising a group of operations which are related and have specific measurable
goals (Art. 2 of the ESF Regulation No 1081/2006).
Public expenditure: any public contribution to the financing of operations whose origin
is the budget of the State, of regional and local authorities, of the European
Communities related to the Structural Funds and the Cohesion Fund and any similar
expenditure. Any contribution to the financing of operations whose origin is the budget
of public law bodies or associations of one or more regional or local authorities or
public law bodies acting in accordance with Directive 2004/18/EC of the European
Parliament and of the Council of 31 March 2004 on the coordination of procedures for
the award of public works contracts, public supply contracts and public service
contracts shall be regarded as similar expenditure (Art. 2 of the ESF Regulation No
1081/2006).
Sub-priority axis: Level of implementation below Priority Axis in the Operational
Programmes.
page 107