CONFERENCE ON IMPACT EVALUATION: METHODS, PRACTICES, & LESSONS

Impact Evaluation of LaborBased Road Work in the Pacific

ADB Pacific Department
Aaron Batten, Christopher Edmonds, Daisuke Misuzawa, and Craig Sugden John Standingford, Lead Consultant, Transport Economist Lincy Pendeverana, Community Survey Expert

July 2012

Impact Evaluation of Labor-Based Road Work in Pacific

The labor-based methods are a construction technology using labor

substantially supported by limited equipment instead of heavy equipment handled by a little labor force.
Such methods provide an important avenue for promoting poverty

alleviation in infrastructure development because they contribute to income generation and job creation in local communitiesparticularly in rural areas where there are few wage-based employment opportunities. The LB methods have been used for small-scale infrastructure development, including road, irrigation, land development, forestry, water-supply, sanitation and small buildings.

In cooperation with partner governments in Pacific Developing

Member Countries and funding from the Government of Japan, ADBsupported projects have employed labor-based methods for road maintenance and rehabilitation of unpaved rural roads in a number of PDMCs. The ongoing evaluation is assessing impacts of road maintenance and rehabilitation projects in Papua New Guinea, Solomon Islands, and Timor-Leste.

Questions to be addressed in the evaluation

What are the changes in living standards of households in

communities located near labor-based and traditional capitalintensive road rehabilitation and maintenance projects?
Are communities near LB projects more likely to express

positive attitudes toward the road projects?

Do workers employed in LB road projects show evidence of

improved post-project employment experiences?

Do communities participating in LB projects demonstrate

greater commitment to road maintenance subsequent to the completion of the project?

What types of social and environmental externalities

associated with LB and tradition road maintenance and rehabilitation projects and practical importance in the communities and households neighboring road project areas?

Study areas and basic methods

In Papua New Guinea and Solomon Islands: Survey research based

impact evaluation with treatment and control groups tracked over time.
Key issues: defining comparator(s), rigorous sampling, logistics of

conducting surveys in a timely manner with available resources Leaving open possibility of econometric controls if treatment and control groups appear to be non-comparable

In Timor-Leste: Piloting of a lower cost, simpler evaluation method

that stakeholders can adopt for other rural development programs.

Sample selection, and definition of treatment and control groups

PNG: Select localities based on roads, every 5th house method

(citation)

Sampling groups, households proximate to: (1) LB Road

Maintenance and rehabilitation, (2) Road maintenance using capital-intensive methods, and (3) No road maintenance

SOL: GIS sample frame based on remotely sensed structures Sampling groups, households proximate to: (1) LB road maintenance, (2) Road maintenance using capital-intensive methods, and (3) No road maintenance TIM: Based on national population and housing census data

for assisted aldeias (i.e. sub-villages) supplemented with focus groups discussion and targeted household surveys in assisted communities. Control group to be identified ex-post.

Surveys collected in PNG and SOL

Household
(individual enumeration at residence)

Community
Business

(focus group)

(individual enumeration at business)

Transport service provider

transport stops or terminals)

(individual enumeration at

Road condition (direct observation and measurement

along roads in the study areas)

Papua New Guinea Study

Treatment 2: Mulitaka

Control 1: Recent rehabilitation

Treatment 1: Eagel-Kero Control 2: No rehabilitation

Solomon Islands Study

Two roads in adjacent similar areas on the outskirts of

Honiara.
One road being upgraded under ADB supported Second

Road Improvement Project, while the other is to be rehabilitated entirely by government.

Sample selected based on remotely sensed census of

structures in the areas located near the two roads, as validated in the early field work for the study.

SOL Summary of Household Interviews

Google Earth pins 11 40 46 30 44 46 31 30 26 34 61 35 22 28 116 67 48 83 798 Household interviews 4 22 23 10 10 19 14 10 10 16 19 12 0 5 39 31 15 28 287

Constellation St Martin Road

PSU 3 4 5 6 7 9 10 11 12 13 16 17 18 19 23 20 21 22

Ratio 36% 55% 50% 33% 23% 41% 45% 33% 38% 47% 31% 34% 0% 18% 34% 46% 31% 34% 36%

'The Y'

Black Post Road (Central)

Black Post Road (South) Total

Timor-Leste Study
The $3m Our Roads Our Future project (Grant 9142), funded

from the Japan Fund for Poverty Reduction, is extending earlier pilots of community-based infrastructure delivery
The budget for monitoring is small, and the method would

ideally be one that is simple enough for government agencies and other stakeholders to adopt Timorese communities are over surveyed, and the government has asked partners to maximize the use of government data and minimize one-off surveys
The control group cannot be set ex-ante, as there are many

rural infrastructure programs underway and their future location is not known

Timor-Leste Study
Baseline data has been prepared at the aldeia (ie sub-

village level) from the 2010 population and housing census, which hopefully will be updated by the 2015 census Changes in living standards to be measured by changes in asset holdings rather than more costly income/ consumption data
Relative progress of assisted and unassisted groups to

assessed by changes in ranking against a village level asset index Additional context drawn from socio-economic data of the census (eg access to basic services) and targeted information collection

Key Assumptions
The PNG, SOL, and TIM governments are

supportive to provide data and documentation. Recruitment of consulting services and procurement of data collection are conducted in a timely and effective manner. The timing of road project implementation, enable surveys to be conducted pre- and postproject completion within the suggested timeframe for the IE study.

Implementation risks
Non-response and/or logistical difficulties in

conducting survey make it difficult to collect the data required for the IE on a timely basis. maintenance and rehabilitation projects accrue to households and communities over roughly a 6-month period to enable measurement of significant differences across baseline and follow up surveys in treatment and control households and communities.

Insufficient impacts from the road

Implementation risks
How comparable are project sites? Ethical concerns - collection of data in non-

treatment villages can lead to expectations from residents

Key issues looking ahead

Duration of IE study long term nature of changes study

attempts to measure
Costs of data collection, particularly on PARD overhead

(staff time, staff travel) not covered by the IE RETA

Uncertain outcomes costs and benefits of undertaking

rigorous IE studies IE consumer appreciation of the difference between results of rigorous IE study versus more typical process/implementation evaluation Expected differences and pros/cons between the teams approach and Timor Leste approach using census data