of project
Timor-Leste
FINAL REPORT
Acknowledgements
Almost 200 people gave of their time and shared their knowledge, experience and insights to
make this evaluation possible from community members and their leaders to local government
officials, project partners and project staff to senior national government officials and
representatives of international organisations and for this we offer our sincere gratitude.
Special thanks are due to those who helped organise and facilitate the whole exercise, particularly
the field work, namely:
for their general guidance, support and facilitation in UNICEF:
Ramesh Bhusal: Chief of WASH Section
Toshiko Takahashi: Chief of Planning, M&E and Social Policy Section
Desiree Jongsma: Country Representative
Rene van Dongen: Deputy Country Representative
for setting up the field visits and accompanying the consultant:
Rodolfo Pereira: WASH Officer, UNICEF
Olga Corbafo: Consultant in Oecusse, UNICEF
Staff of the NGO partners ETADeP, HIM and AMAR
Staff of SAS in Viqueque
for assisting with translation in interviews in Dili and in the districts:
Rodolfo Pereira: WASH Officer, UNICEF
Olga Corbafo, Vicente Lopes, Indra Monemnasi and Eusebio Martins (UNICEF)
and for their reliable and professional logistical support: the administration staff and drivers of
UNICEF.
We would also like to acknowledge the quality assurance guidance and input provided by UNICEF
regional and headquarters staff, namely:
Chander Badloe: Regional WASH Advisor, UNICEF East Asia and Pacific Regional Office
Jeremie Toubkiss: WASH Evaluation Specialist, UNICEF HQ
Riccardo Polastro: Regional Evaluation Advisor, UNICEF East Asia and Pacific Regional
Office
Finally, special thanks are due to the following senior government partners for their political
support and contribution to the evaluation:
Mr. Joao Pereira Jeronimo, DG National Directorate Water & Sanitation Services, MoPW
Mr. Joao Pidade, Director National Basic Sanitation Services, MoPW.
Mr. Gustavo Da Cruz, Director National Water Supply Services, MoPW,
Mr. Martinus Nahak, Programme and SIB Officer, DNSA, SEASU, MoPW
Mrs. Tomasia De Sousa, Environmental Health Department, MoH
Mr Miguel GODINHO, Chief of Department, School Social Action, MoE
Bairro
Sub-aldeia, or neighbourhood
BeSI
BESIK
CAP
DGAS
DNSA
CLTS
DTO
EU
European Union
FGD
GMF
HH
Households
HWF
HWWS
JMP
KAP
Logframe
O&M
OD
ODF
MDG
NGO
PAKSI
PSF
SAS
SDF
Sub-District Facilitator
SIBS
Suco/Suku
TLDHS
UNEG
UNICEF
WASH
WHO
ii
Table of Contents
Acknowledgements ................................................................................................................... i
List of Acronyms, Terms and Abbreviations .......................................................................... ii
Executive Summary ................................................................................................................ vi
1.
Introduction ....................................................................................................................... 1
2.
3.
4.
4.1.1
5.
6.
4.2
4.3
Sampling ...................................................................................................................... 8
4.4
4.4.1
Methodology ....................................................................................................... 10
4.4.2
4.4.3
Field work............................................................................................................ 12
4.4.4
Findings .............................................................................................................. 12
4.5
4.6
5.2
Findings ........................................................................................................................... 15
6.1
Relevance .................................................................................................................. 15
6.1.1
6.1.2
6.1.3
6.1.4
6.2
Efficiency.................................................................................................................... 16
6.2.1
6.2.2
6.2.3
6.2.4
6.2.5
6.2.6
6.2.7
6.3
Effectiveness .............................................................................................................. 21
6.3.1
6.3.2
6.3.3
6.3.4
6.3.5
6.3.6
6.3.7
6.4
6.4.1
6.4.2
6.5
7.
8.
9.
Impact ........................................................................................................................ 28
Sustainability .............................................................................................................. 28
6.5.1
6.5.2
6.5.3
6.5.4
6.6
Equity ......................................................................................................................... 34
6.7
General ...................................................................................................................... 36
7.2
7.3
7.4
General ...................................................................................................................... 38
8.2
8.3
8.4
Recommendations .......................................................................................................... 41
Appendices
Appendix 1:
Appendix 2:
Appendix 3:
Appendix 4:
Appendix 5:
Appendix 6:
Work Programme
Appendix 7:
Documents Reviewed
Appendix 8:
Appendix 9:
List of Tables
No.
Title
Page
14
14
23
24
Functionality of water supply systems and GMFs visited during final evaluation
30
32
34
List of Figures
No.
Title
Page
5
Executive Summary
Introduction
This is the final evaluation report of the project Improving Access to Water, Sanitation and
Hygiene (WASH) in Rural School and Communities through Capacity Development. Jointly
funded by the European Union (EU) and UNICEF and implemented by UNICEF, the project was
implemented in 10 sub-districts in 5 districts1 (Aileu, Ermera, Manatuto, Oecusse and Viqueque)
in Timor-Leste, over 45 months between June 2011 and March 2015. The project originally
targeted 55 communities (aldeias2) but by the time it was completed had carried out activities in
63 aldeias in the targeted sub-districts. It was implemented together with government ministries
(public works, health, education) and national NGO partners.
The purpose of the project was to assist communities in the target areas to access clean water
and improved sanitation facilities and improve their hygiene knowledge and practices, in order to
achieve better health outcomes for children and adults and enable children to participate better
in primary education. The project was designed with three main components (expected results),
namely: i) Institutional strengthening; ii) Community and schools water supply; and iii) Household
and schools sanitation and improvement in knowledge and attitudes relating to hygiene and
sanitation, using the CLTS (community-led total sanitation) approach.
The project was implemented in response to a number of challenges faced by Timor-Leste, in
spite of progress made since the restoration of independence in 2002. When it was designed, the
under-five mortality was 64 per 1000 live births vis a vis its MDG target of 96, and the primary net
enrolment rate (grade 1-6) was 93.6%, against the MDG target of 1003. Only 61% of the rural
population were using improved water sources and only 27% improved sanitation facilities4. Poor
access to water, sanitation and hygiene was one of the main causes of diarrheal diseases in
under-five children, also contributing to the very high rates of under-five malnutrition (58.1%
stunting and 18.6% wasting5). There were also major concerns regarding the WASH situation in
schools. About 54% of primary schools did not have access to improved water supply, 35% had
no latrines in the compound and only 12% had a regular supply of soap6, all indicating the lack of
a basic enabling environment for school children to practice safe hygiene behaviour.
The overall objectives of this evaluation were to strengthen UNICEFs accountability to its donors
and key stakeholders, including beneficiaries, and to learn from the experience of this intervention
to inform future WASH policy and programming. The evaluation had to assess compliance with
contractual obligations as well as adherence to UNICEFs equity focus and the potential
contribution of the project to the sector and to UNICEFs institutional learning agenda. The
intended audience of this report thus includes UNICEF, the EU, the government and NGO project
partners, government decision makers and other organisations supporting the sector.
The methodology applied in the evaluation was based on that defined in the terms of reference,
combining qualitative and quantitative methods to ensure that as full a range of data sources as
possible was consulted. These included documents produced by the project and information
provided by the UNICEF project team, three of the five partner NGOs, the implementing
government partners at national level, in four districts and eight sub-districts, beneficiaries in 10
aldeias in seven sub-districts, and four other international organisations supporting the sector.
The documents reviewed (24 in total, including 14 produced by the project) included the recent
endline KAP (knowledge, attitudes and practices) survey, which was a major source of data. Forty
1
Now called Administrative Posts and Municipalities: see page 1 for an explanation on nomenclature.
Variously translated as community, sub-village or hamlet
3 Demographic and Health Survey, Ministry of Health, 2009-10
4 Joint Monitoring Programme, WHO/UNICEF, MDG Progress on Sanitation and Drinking Water, 2010
5 Demographic and Health Survey, Ministry of Health, 2009-10
6 National Stocktake of Facilities, and Equipment in Basic Education Schools in Timor-Leste, Ministry of
Education, 2011
2
vi
semi-structured interviews were conducted with key informants as well as 13 focus group
discussions with GMFs (water committees). Field observation included 13 aldeias and 10 schools,
while survey/interviews were conducted with 37 beneficiary households in 10 aldeias.
The sampling plan was designed to allow for capturing a diversity of situations and points of views
rather than anecdotal evidence, hence the large sample of sub-districts visited (8 out of 10),
encompassing 13 aldeias (30% of those that benefited from all project components) and 10
schools (36%). Whenever possible, the various data sources or collection methods were
combined, analysed and compared in an objective manner, acknowledging possible bias and
other methodological limitations, in order to build a comprehensive, credible and nuanced picture
of the project.
In general the approach of implementing in partnership with local NGOs who in turn partnered
with target communities organised by them, achieved positive results in this project. However, in
terms of the quality of the relationship between the NGOs and UNICEF as partners, the NGOs
perceive the relationship to be purely one of sub-contractors in which they are not genuinely
involved in joint decision making. It seems, however, that this is more likely the result of the
inevitably unequal power relations and their reluctance to express concerns more forthrightly.
Community water supply
The project was very effective in this component and amply exceeded its output targets in all
aspects. In addition, 24 point sources (protected springs and shallow wells), not originally
anticipated, were built where gravity-fed piped systems were not feasible. The 43 GMFs formed
also received additional refresher training in O&M (operation and maintenance) after completion
of their systems, and GMFs were formed per aldeia for some of the point sources. The objective
of improving water coverage was achieved, including for schools, if allowance is made for those
schools that had not been connected yet but would be within a few months.
The quality of the infrastructure built, of community participation, of documents produced and of
training provided was generally of a very high quality. An important exception was the water
system in Bahalara-Uain, Viqueque district, which had to be completed by SAS (District Water
and Sanitation Service, Ministry of Public Works) due to the poor performance of the partner NGO
which was unable to resolve the unusually difficult social and institutional problems at community
level. Unless an appropriately funded and facilitated rehabilitation intervention is undertaken on
this system, it will not be sustainable.
The effectiveness of GMFs was found to be good in most cases, with very specific problems as
the causes of dysfunctionality, namely poor leadership. Almost all the GMFs visited showed a
high level of ownership of their systems and adequate knowledge of their roles and
responsibilities. The problems noted should be able to be resolved by Sub-district Facilitators
(SDFs) but it is clear that in general they are under-funded and/or inadequately supervised in
most districts, constituting the weak link in the GMF system. This is an area that needs ongoing
attention and support.
Household sanitation and hygiene
The high level of success in achieving ODF certification, mentioned above, indicates an excellent
level of effectiveness of this element of the CLTS approach and bodes well for the future of the
strategy in Timor-Leste. There was also a significant spin-off effect in terms of an increase in the
number of permanent latrines, across all districts. At baseline 33% of households were using
these and by the end of the project the figure was 45%. Families built permanent latrines (pour
flush, in almost all cases) without any material incentive from the project, due to the messages
received and to the improved access to water.
With regard to the sustainability of latrine usage, however, it was found to be very variable.
Reversion to OD was between 42% and 53% in 30% of aldeias declared ODF (including all of
those in Manatuto), but 10% or less in 62% of them (all of which are situated in Aileu and Ermera).
This weakness will not be resolved until the health system is able to provide ongoing, frequent
follow-up at household level to sustain behaviour changes.
In Viqueque none of the target communities achieved ODF status. Several possible explanations
have been suggested, including the impact of other programmes in the district offering subsidies.
However, it was concluded that this was not a significant factor. In other districts the project had
to contend with the same challenge and managed to do so effectively. It would seem that the
weakness of the NGO was the main problem, as noted above.
viii
The evaluation confirmed the importance of the work being done to finalise a policy on incentives
to assist the most vulnerable, to be applied once communities achieve ODF status.
School WASH
Due to the good quality of design and construction and the relative newness of the facilities, their
level of functionality was high, the main problem being pending water connections or dysfunctional
water systems.
The project appears to have done everything possible to ensure maximum sustainability of the
facilities, to compensate as much as possible for the generally acknowledged shortcomings of
the Ministry of Educations facility maintenance system. The endline survey found that all but one
school visited had a PTA with specific WASH responsibilities and formal relationship with the
GMF of the water system supplying the school. The final evaluation also found that most GMFs
also take responsibility for maintenance of the water facilities at the schools.
Lessons learned
The model of institutional arrangements that was used to implement the project was based on a
partnership with government that involved sharing the responsibilities for implementation of
activities between UNICEF (and its NGO partners) and different government departments, both
at national and district levels, while UNICEF retained the responsibility for accountability to
donors. This model is appropriate and necessary to acknowledge the responsibility of government
for delivery of basic social services and to help build capacity to do so.
The model of implementing with local NGOs was also validated as one that is replicable generally
in the sector. Their comparative advantage in managing community based construction of
services that have to be managed by communities afterwards, was reflected in the high level of
ownership and effectiveness of GMFs. The NGO model is also cost effective, in relation to
commercial contractors. This is especially the case for very small jobs in remote locations which
contractors are reluctant to take on at an affordable price.
The CLTS approach worked well in this project in most districts and in all but two sub-districts,
thanks to a large extent to the intensive, repeated training of trainers and facilitators. Much has
been and continues to be learned by UNICEF and other stakeholders in the sector about applying
the methodology, the key challenge being sustainability, i.e. reversion to OD. The challenge of
sustainability will not be solved unless and until the health system is strengthened to provide
ongoing monitoring and promotion at household level. For this much more funding is needed.
Funding is also needed to scale up CLTS.
The main obstacle to moving up the sanitation ladder and enhancing sustainability is the cost of
materials and transport. The work being done currently to agree on an incentive policy to be
applied for the most vulnerable needs to be finalised and the policy applied.
Recommendations
For UNICEF and its future projects:
1.
2.
3.
While the implementation model used in the project of partnership between UNICEF and
government is sound, it could be further strengthened with a clearer definition of
governments responsibilities and tasks, especially those not funded by the project,
documented in joint work plans, with effective joint monitoring and evaluation.
In order to strengthen and enhance the sustainability of the generally effective model of
implementing in partnership with local NGOs, more responsibility should be assumed for
enhancing the organisational capacity of the NGOs.
Refresher training of NGO, UNICEF and SAS technical staff is needed on monitoring of
construction to ensure attention to detail for sustainability, with the use of checklists.
ix
For government:
4.
5.
6.
7.
8.
DGAS, Health, and Education: Accelerate the process of taking over more responsibilities as
a service provider and funder, with the donor as a supporting partner, increasing budgets,
especially for scaling up CLTS and supporting the O&M of water and sanitation in both
communities and schools. The challenge of sustainability of CLTS will not be solved unless
and until the health system is strengthened to provide ongoing monitoring and promotion at
household level.
DGAS and Health: Undertake a review of experience to date in implementing CLTS to obtain
greater understanding of factors at play in the successes and shortcomings.
DGAS: Include in the CAP (Community Action Planning) process a post-construction
mentorship period to enable SDFs to monitor GMF performance, provide further on-the-job
training and solve problems until the end of the project (with donor support in the case of
donor funded projects).
DGAS: Lobby for finalisation and approval of i) the latrine construction incentive policy for the
most vulnerable once a community has been declared ODF; and ii) the BeSI7 Committee
Guidelines and respective operating budget.
Education: Approve the School WASH Guidelines and include in the training package a
module on technical maintenance of facilities, prioritising the setting up of a functional system
that includes GMFs. DGAS: Include as a standard responsibility of GMFs the maintenance
of school water infrastructure.
1.
Introduction
This is the final evaluation report of the project Improving Access to Water, Sanitation and
Hygiene (WASH) in Rural School and Communities through Capacity Development. Jointly
funded by the European Union (EU) and UNICEF and implemented by UNICEF, the project ended
in March 2015, and in terms of the funding agreement requires an end-of-project evaluation to be
carried out after completion. Consequently the services of a consultant with the requisite
competencies and experience were procured by UNICEF. A single consultant was considered to
be sufficient, rather than a team with more members, due to the fact that an endline survey and
knowledge, attitudes and practices (KAP) study had already been carried out a few months before
the end of the project, as a key source of information for the evaluation.
This report has been written by the consultant in accordance with the terms of reference of the
evaluation (see Appendix 1). After presenting some brief background information (sections 1, 2
and 3), the report describes the methodology employed, with the respective tools (section 4), and
the activities undertaken (section 5). The detailed findings, including those of the endline survey
and KAP study, are then presented and analysed (section 6), and the key conclusions described
(section 7). Finally, the lessons learned and recommendations are presented (sections 8 and 9
respectively).
Note on terminology
The government of Timor-Leste has recently made some changes to the terms applied to the
different administrative divisions in the country. In relation to those and the related terms that are
used in this report, the following are the changes that have been made:
Previous term/name
New term/name
District
Municipality (Municipio)
Sub-district
District Administrator
Sub-district Administrator
In order to maintain consistency between the terms used in this report and those used in all other
project documents to date, to facilitate comprehension, the previous terminology will be used.
2.
Description of Project
Country context
Timor-Leste is the second youngest nation in the world. When its independence was restored in
2002, the country was in a state of ruins with shattered infrastructure and a stagnant economy,
and it has undergone numerous political, security and development upheavals since then.
However, steady progress has been made in peace and democracy with two successful
parliamentary elections in 2008 and 2012. The population of Timor-Leste is predominantly young,
with nearly half of the population under 18 years. Along with peace and democracy, the situation
of children and women has gradually improved, but a number of challenges persist.
When the project was conceived, the rate of under-five child mortality was 64 per 1,000 live births,
while more than one in ten Timorese children were acutely malnourished and almost 50 per cent
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
suffered from chronic malnutrition8. Access to an improved water source was found to be 69% in
20089, up from 52% in 2000, but with a significant disparity between urban areas (86%) and rural
areas (63%). The same JMP data showed that only 50% of Timorese had access to improved
sanitation facilities (up from 32% in 2000), with 76% of urban dwellers having access, compared
with only 40% of rural dwellers. The rate of open defecation in rural areas was 52% in 2008
(unchanged since 2000). Poor access to water, sanitation and hygiene was one of the main
causes of diseases in children, also contributing to the very high rates of malnutrition.
In the targeted rural areas in the five project districts, the major constraints faced by the children,
women and other vulnerable groups in regard to water, sanitation and hygiene can be
summarised as follows: 1) Low access to improved water sources and low capacity to manage
water facilities; 2) Low sanitation coverage and low access to information regarding sanitation
options; 3) Widespread use of unsafe hygiene practices and unsanitary conditions in schools and
communities, and; 4) Low capacity of WASH authorities, health professionals, teachers and
family health promoters.
It was found in a 2009 survey of two rural districts that 59% of schools had fully or partly
functioning water systems and 55% had fully or partly functioning sanitation systems10. The same
survey found that only 30% of community water systems were functioning one year after
completion. Other surveys also found very high failure rates within a few years of installation, with
contributing factors found to be: management failure on the part of communities, lack of spare
parts and skills and poor initial design. The national government had very little capacity to support
rural water supply and sanitation, with each of the 13 districts having only one rural water and
sanitation official assigned. Decentralisation of government services was very low, with no legal
frameworks in place to allow ministries to devolve control of resources below the national level.
Project Intervention Logic (see Logframe in Appendix 2 for details)
In order to address the issues just described, the project focused on achieving the short term
outcomes of assisting communities to access clean water and improved sanitation facilities and
on improving their hygiene knowledge and practices. In terms of the MDGs, the purpose was to
reduce the proportion of people without sustainable access to safe drinking water and basic
sanitation to work toward the achievement of Millennium Development Goal (MDG) 7.
The overall objectives of the project, in terms of the intended long term outcomes and impact,
were defined as:
Children, women and men in target areas will achieve better health outcomes
Children in target areas are better able to participate in primary education
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
3. Target aldeais and primary schools have improved sanitation and demonstrate
measureable improvement in knowledge and attitude relating to hygiene and sanitation.
The strategy for achieving Result 3 at community level was through the use of the CLTS
(community-led total sanitation) approach to eliminate open defecation (OD) without providing
any materials subsidy for latrine construction.
Based on the logframe, the specific Theory of Change guiding the evaluation can be
conceptualised as follows:
IF district authorities, particularly the Water, Sanitation and Hygiene
(BeSI) Committees are better able to lead and support WASH activities in
the five project districts, supported by national authorities, policies and
guidelines; and target aldeais (sub-villages) and primary schools have
improved water sources and are able to manage and maintain their water
systems; and target aldeais and primary schools have improved sanitation
and demonstrate measureable improvement in knowledge and attitude
relating to hygiene and sanitation;
THEN national and district WASH personnel and community groups will
be better able to sustain equitable access to, and use of, improved
drinking water and basic sanitation, and adoption of improved hygiene
practices.
The project (see Table 1, below, for a summary of key project data) was implemented in 10 subdistricts in 5 districts (two in each) over 45 months between June 2011 and March 2015. The
original project duration was planned for 36 months but received a nine months extension in June
2014.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
EU Reference No.
FED 2011/266-9801
Project Budget
Funders
Relevant MDGs
Geographic Focus
Area
Sub-District
Aileu
Liquidoe, Remexio
AMAR
Ermera
Ermera, Railaco
HIM
Manatuto
Lacio, Laclubar
NATILES
Oecusse
Nitibe, Oesilo
ETADeP
Viqueque
Lacluta, Viqueque
CPT
The location of the target districts and sub-districts is shown in the map in Figure 1, below.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
A list of the targeted aldeias in which the community water supply and sanitation activities of the
project were implemented, including changes since the start of the project, is included in Appendix
3. Appendix 4 contains a list of the schools that benefited from WASH interventions.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
3.
The overall objectives of the evaluation (see also Appendix 1), in line with standard practice, were
to strengthen UNICEFs accountability to its donors and key stakeholders, including beneficiaries,
and to learn from the experience of this intervention to inform future WASH policy and
programming. The evaluation thus needed to both assess compliance with contractual obligations
as well as adherence to UNICEFs equity focus and the potential contribution of the project to the
sector and to UNICEFs institutional learning agenda. An additional objective was to provide
strategic comment to UNICEF Timor-Leste in terms of its focus areas of support to relevant
ministries in WASH, as proposed under the current Country Programme (2015-2019).
To achieve these objectives, the evaluation was required to:
a. assess the relevance, effectiveness, efficiency, sustainability and impact of the project;
b. assess the extent to which the project contributed to broader development results at the
sub-national level, and the lessons learnt that will allow the replication and scaling up of
the interventions; and
c. draw operational recommendations and lessons learned for further improvement and
enhancement of relevant sector policies, plans and strategies through analyses of the
factors contributing to the success or failure of the project.
The detailed evaluation questions are described and discussed in the next section.
In terms of the geographical scope of the evaluation, it was necessary to take into account the
broad spread of the project across 10 sub-districts. This meant that, although ideally all sites
should be visited, the budget and thus time limitations made it necessary to select a practically
feasible sample of sub districts and communities within them to visit, that would still be
representative. The criteria for selection are described in the next section.
4.
Evaluation Methodology
The methodology applied in this evaluation was based on that defined in the terms of reference,
which is standard as per current good practice. Only minor modifications were introduced, namely:
adjustments to the distribution of time allocated in Phase 2 for desk work and interviews
in Dili, and work in the sub-districts (from 18 and 13 days respectively, to 16 and 15);
as a result, an increase in the number of sub-districts that were visited, from the specified
minimum of 3 districts and at least one sub-district in each, to 8 out of the total of 10; and
reduction in the time allocated to Phase 3 (analysis and reporting), from 20 to 16 days.
The methodology combined qualitative methods (structured and semi-structured interviews and
focus group discussions) with quantitative methods, namely gathering data from routine reports,
reports on interim external monitoring conducted by the EU, analyses and reports produced by
other interventions and, fundamentally, the data obtained by the baseline and endline surveys
and KAP studies.
The fact that an endline exercise was conducted virtually at the end of the project allowed this
final evaluation to focus more on the aspects of the project not covered in detail in that exercise,
i.e. to concentrate more on sustainability and lessons learned and less on detailed surveys to
assess KAPs at household level. This was the main motivation for increasing the sample size of
sub-districts to be visited (and thus of key role players and stakeholders to be interviewed), which
resulted in less time spent in each aldeia compared to the endline survey and KAP study.
The methodology employed is described in the sections that follow in terms of the following key
aspects:
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
4.1
Evaluation Questions
The evaluation questions included in the terms of reference were expanded by the consultant in
his technical proposal, and then some further additions made, in agreement with the UNICEF
country office, after the initial desk review of documents provided before arrival in country, and
the final list documented in the Inception Report. In particular, the recommendations for this final
evaluation made in the Endline Survey and KAP Study Report were taken into consideration, of
which the following are highlighted:
a. Some field work needs to be done to visually capture the various types of latrines households
have in the project area to better understand the Timor-Leste context in relationship to JMP
latrine definitions.
This was done.
b. Consider further exploring what other benefits have been experienced by households as a
result of increased water access beyond having increased improved drinking water sources.
Specific questions in this regard were included in the household interview checklist (see
Appendix 9H).
c. Drinking water at schools was reported to be boiled, however this was not verified through
observation. Also 70% of children were reported to bring drinking water to school with them
to drink. Suggest that both of these be verified/further assessed during the final evaluation.
As explained in section 5, it was not possible to follow up on this issue due to all but one of
the schools being closed on the day/time of the visit, due to unforeseen circumstances beyond
UNICEFs control.
d. If water quality testing has not been done, it is recommended that such testing be undertaken
during the final evaluation.
Due to the fact that there is only one laboratory able to do this, in Dili, it was decided that it
would not be feasible to include water quality testing on a large enough scale to be meaningful.
However, this an important issue and is included as a recommendation.
e. Consider collecting random household point of use water samples from water storage
containers pre/post boiling to assess water quality and the effectiveness of water treatment
methods used by the majority of households.
As explained above, this was not feasible.
The expanded evaluation questions are listed in Appendix 5.
4.1.1
M&E Indicators
Besides the evaluation questions to be answered, the project established a Monitoring and
Evaluation (M&E) plan with specific indicators to measure its impact at completion, based on the
project logical framework (logframe). The latter is attached in Appendix 2. Both the baseline and
endline studies served to inform the projects M&E plan and measurement of results and
established indicators, and the results are incorporated into this evaluation.
The studies did not fully measure the results for the indicators for Result 1 (Local authorities
are better able to lead and support WASH activities ), although sub-district facilitators (SDFs)
and water facility management groups (GMFs) were interviewed to assess their capacity
development. The results for the indicators will be assessed in this evaluation, namely:
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
All SDFs in target areas have received ToT on CAP processes and are supporting GMFs
2 national WASH training modules/guidelines developed and implemented.
Due to the fact that the results of most of the indicators had already been measured by the endline
survey, and that this evaluation by its nature as a final evaluation is mainly focused on assessing
sustainability and identifying lessons learnt, it was not guided only by the indicators in the
logframe. In the case of household sanitation in particular, other issues were explored such as
whether villages declared ODF still maintain that status, to what extent households have
subsequently moved up the sanitation ladder (i.e. upgraded to more durable facilities or ones
providing a higher level of service), and what inspired them to do so, as well the role being played
at village level by local sanitation entrepreneurs. These issues are reflected in the corresponding
evaluation questions in Appendix 5.
4.2
The details of sources of information and methods of data collection used, for each evaluation
question, are listed in Appendix 5. The actual number of each is reported in section 5. In summary,
the sources were the following:
a.
Secondary sources, i.e. documents produced by the project and others (see Appendix 7
for the list of those reviewed). These included reports of two EU-commissioned ROM
(results-oriented monitoring) reviews in 2012 and 2013.
b.
Semi-structured interviews with key informants, including UNICEF staff, NGO partners,
national and district government partners, SDFs, school coordinators and other donor
funded programmes and international NGOs active in sector. Pre-prepared checklists
were used to facilitate consistency and comparison see Appendix 9.
c.
Focus group discussions (FGDs) with GMFs (which also involved community leaders and
ordinary community members in most cases), based on pre-prepared guidelines (see
Appendix 9). No other FGDs at community level were held as this area was covered by
the baseline and endline KAP studies; as described in section 5, it was not possible to
meet with groups of school children as intended.
d.
Observation of WASH infrastructure and other aspects (including open defecation and
general cleanliness) at village level.
e.
Random sample surveys of households at all but two of the villages visited, using transect
walks, observation of latrines and other barriers against disease and brief discussions with
heads of households (except in two villages where few were at home), based on checklists
(see Appendix 9H).
All the information gathered was collated in a summarised form in a data base in Excel format
(see Appendix 10), organised as follows:
Sheet 1: Notes to form the basis for answers to evaluation questions, by question and
source of information (both as an aide to the consultant, and to assist the reader
to understand the sources of information for the observations and comments
made in this report);
Sheet 2: Observations on water supply systems and summaries of FGDs with GMFs;
Sheet 3: Observations of latrines and homesteads and summary of interviews with
householders;
Sheet 4: Observations of school WASH facilities at schools and summary of interviews
with Coordinators of two of the schools.
4.3
Sampling
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
The terms of reference specified that at least 3 districts should be visited. However, as noted
above, it was possible to organise the itinerary in such a way as to include visits to four districts
(only Manatuto could not be included, due to distance to the targeted aldeias and time constraints)
and both sub-districts in each, totalling 8 sub-districts.
The criteria used for deciding on the districts to be visited included the Endline Survey and KAP
Study Report recommendations for this final evaluation, already mentioned above, which include
the following with regard to districts and sub districts to be visited:
a. Explore what factors contributed to the differences among the targeted districts/subdistricts with respect to sanitation (especially how CLTS was implemented) and water
access. Selected districts/sub-districts should include both high performing and less
performing areas.
This was done; Viqueque was included as a less performing district in spite of the
relatively long travel distances (virtually a full day each way).
b. Consider gathering data in the communities that were dropped from the project post
baseline, such as in Viqueque and other unserved areas to provide data on non
intervention communities for comparison purposes.
In order to draw any valid conclusions for comparison it would have been necessary to
include a reasonable number of non intervention communities (say at least 3) and to
spend enough time in them to understand their dynamics. However, it would have been
difficult to gain meaningful access since the project was not active in these communities,
and there were time limitations. Another factor taken into account was the fact that the
project was not designed as a random control trial. It was thus decided not to try to
implement this recommendation.
With the above in mind, it was essential to include Viqueque district and Lacluta sub-district in
particular, as less performing and Liquido in Aileu as high performing, in terms of the
achievement of water system implementation and ODF status targets. In Viqueque both subdistricts were included in the sample.
In the other districts the initial targets were achieved or even exceeded, in terms of both water
systems and ODF status, but according to the Endline Survey, in some sub-districts, such as
Railaco in Ermera and Nitibe in Oecusse, there seems to be a fairly high incidence of reversion
to OD which needed to be analysed more closely.
In summary, the minimum list of the sub-districts to be visited, and the respective selection criteria,
were as follows:
Sub-district
1. Lacluta
2. Viqueque
3. Liquido
4. Railaco, Nitibe
As noted above, it was possible to include all the above sub-districts, including in Oecusse, as
well as the second sub-district in each district.
The main criterion for selection of aldeias to be visited in each sub-district was to include
examples of both those that had performed well and those that had performed relatively poorly.
Where there was a choice of two or more aldeias that qualified under either criterion, the issue of
practical access was considered. Aldeias that would require more than two hours travel by vehicle
from the sub-district capital were excluded. Aldeias with schools targeted by the project were also
prioritised.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
In terms of sample sizes, this means that the following was scheduled:
18 aldeias to be visited, i.e. over one third of the total number of target aldeias;
7 SDFs to be interviewed, i.e. 70% of those with whom the project partnered;
FGDs with 13 GMFs (30% of all those formed and trained); and
6 schools to be visited and inspected, representing 22% of the total of 28 supported.
In terms of the household interviews, it was planned to select them as a random cluster sample
(see Appendix 9H) with the proviso that an adult male or adult female currently residing in the
house be present. This was not possible in all aldeias, as explained in the next chapter.
4.4
The knowledge, attitudes and practices (KAP) endline study was undertaken at the end of the
project to assist in measuring the outcomes and impact of the project interventions against
baseline data collected in 2011. The data was intended to inform this final project evaluation as
well as future WASH interventions in Timor-Leste.
The endline study included a household and school survey, with 601 households with at least one
child age five and under, and 22 public school surveys11. The household surveys were carried out
in 50 of the same 55 aldeias that were surveyed in the baseline survey (five from Laline Suco in
Viqueque were excluded), as well as four which were added after the baseline survey, a total of
54. In addition focus group discussions were held with 12 GMFs and individual interviews with
four SDFs.
The results captured through the comparison of endline to baseline data are attributed to the
projects interventions in sanitation in all 54 aldeais and in the 43 aldeais where water
infrastructure projects were conducted. Changes in water supply in the 11 aldeias where UNICEF
was not working are attributed to the government supported District Development Programme
(PDD) and the NGO Triangle in Oecusse. No other organisations, or other WASH efforts, are
known by UNICEF to have been implemented in the targeted areas during the project period. As
the surveying was only done in the targeted project aldeais, the results are only representative of
the targeted communities and not necessarily to the districts and sub-districts in which they reside.
4.4.1
Methodology
The study was conducted with the use of the following methods:
Two survey instruments were developed for the baseline KAP a Household Survey and a School
Survey. These tools served as the foundation for the endline. Several questions were omitted
from the endline household survey that were not seen as relevant to measuring results and
additional questions were added to gather input on the perception of the level of open defecation
in the respondents communities and CLTS process. Questions and associated variables were
derived as much as possible from the 2009-10 Timor-Leste Demographic Health Survey (TLDHS)
and from other international recognized sources, such as the Millennium Development Goals
11
Endline was planned in all 23 schools where Baseline was done but one school was not accessible at
the time of end-line survey, so 22 schools were surveyed.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
10
(MDGs), and UNICEF documents, e.g. UNICEFs WASH in Schools Monitoring Package, to allow
for standardisation of data and to enable comparisons wherever possible.
In addition to household demographic information, the household survey included 41 possible
questions on the following subject areas:
All but one survey question was designed to be open ended to elicit respondents knowledge
without prompting, along with observational questions to assess access, conditions, and
practices. The surveys were administrated by enumerators hired through a national NGO (Belun)
to assist the consultant with conducting the endline.
The school survey was targeted to primary school principals or head teachers in the target
communities. The survey included 27 questions in addition to basic descriptive information. The
survey was administered by the enumerators through an interview and was designed to illicit as
much information as possible about WASH conditions, knowledge and practices at the school.
The survey included questions on the following areas:
The focus group discussions with GMFs and questionnaires for SDFs were implemented by the
enumerator team leaders. These questions explored the roles and tasks of the GMFs as well as
their overall functionality and perception of OD in their community.
All survey instruments were field tested and modifications to the study instruments were made
following the field tests.
4.4.2
The same sampling methodology used for the baseline was used for the endline study. Sample
size for the surveys was calculated for a margin of error of 4%, with a confidence level of 95%,
based on a 2010 census population of 22,798 from the 54 targeted aldeais, with a 50% response
distribution. This calculation resulted in a sample target 601 households to be surveyed by the
enumerators in the field. The 601 households were selected proportionately among the targeted
communities. There were some changes in the targeted aldeais from baseline to endline, however
interventions took place in the same districts/sub-districts.
The survey used a random cluster sampling method to determine what houses would be selected
for an interview. All houses had to have at least one child age five or under living in the home to
be selected. Respondents were requested to be either an adult male or adult female currently
residing in the house, while a head of household was desired for the interview, they were not
required for the survey. For the endline enumerators were asked to prioritise interviewing women
if both a male and female adult were available.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
11
4.4.3
Field work
The survey team consisted of 12 enumerators, broken into three teams, each lead by a team
leader. Training was conducted with the enumerators to prepare them for piloting/field test and
implementing the survey instruments. Field work was done over 10 days in December 2014.
4.4.4
Findings
The findings of the Endline Study and KAP Survey are reported in section 6 in the sub-sections
relevant to each theme and issue.
4.5
As noted in the terms of reference, the project was not designed on the basis of a gender
differentiated analysis of the needs of beneficiaries, nor did it define specific outcomes aimed at
greater gender equality. Thus, although this report comments on gender and human rights
aspects, this limitation was taken into account, while following the UNEG guidance on how to
integrate human rights (HR) and gender equality (GE) considerations in evaluations. In examining
the HR & GE aspects, the consultant was guided by international and regional conventions and
national policies and strategies, ensuring key informants included both genders, and from human
rights aspects, both duty bearers (government, service providers) and rights holder (beneficiaries)
were interviewed.
As mentioned above in relation to the evaluation questions, comparisons between intervention
and non-intervention communities were not made because the project design did not employ a
randomised control approach. However, this is not considered a serious limitation; such a design
is usually only employed for research projects and this project did not have research as a
significant, explicit objective.
In terms of the quantitative data collected, especially regarding issues such as numbers of outputs
and outcomes achieved, budget spent, etc, the evaluator relied mostly on data provided by
UNICEF. However, this is normal and, in general, the reliance on self-reporting is not considered
to have been excessive and thus a limitation. Other factors such as bias in respondents answers
could be managed due to the considerable prior experience of the consultant, for example, being
aware that beneficiaries tend to try and anticipate in their answers what the interviewer wants to
hear, that local leaders dont want to appear ungrateful so avoid any criticism of the project and
that implementer s tend to want to show only the most positive examples. Where only limited or
no triangulation of some data was possible, this is highlighted.
In terms of the practical aspects of carrying out the evaluation, the most significant one, which
was however foreseen, was the non-availability of some project partners. For example, the
partner NGOs responsible for Viqueque and Manatuto districts, where least success was
achieved with CLTS, were not available as they are no longer collaborating with UNICEF and did
not part on very cordial terms. Their input would have been very valuable to gain greater insight
into the challenges faced in these districts. In particular, the views of CPT could have provided
greater clarity regarding the situation encountered in Bahalara-uain in Viqueque (described in
detail in section 6). However, in recognition of this limitation, special care was taken to solicit the
opinions of more role players than normal concerning the reasons for the problems in Bahalarauain and in Viqueque District in general.
Other practical challenges that placed some limitations on the achievement of the work plan and
thus on the scope of the evaluation were:
heavy rain the night before the planned trip to Ahic in Lacluta sub-district, which prevented
the consultant from reaching the community;
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
12
delays in starting the FGD with the GMF in Bahalara-uain (due to their disorganisation),
and subsequent late arrival at the school after it was closed, thus not being able to
interview anyone there or inspect the interior of facilities;
all schools being closed in Ermera District during the week scheduled for the field visits
there, due to teacher training, hence only one interview and interior of facilities inspected
only at this school;
all schools being closed in Aileu District during the week of the visit due to a religious
festival, hence no interviews and no interiors of facilities inspected;
because of the festival activities in Remexio sub-district of Aileu, few GMF members and
no other community members were available for the FGDs.
It was thus not possible to obtain as much information as desired about the sustainability of the
school interventions in the areas of training and awareness raising. However, these aspects were
adequately covered by the endline study (conducted only a few months before the final
evaluation) which assessed retention of messages and perception of quality of training, amongst
other factors.
4.6
Ethical considerations
No difficulties were encountered in adhering to UNICEFs ethical standards, as per the UNEG
Ethical Guidelines for Evaluation. All confidential information obtained by any means was treated
in confidence. Personal, confidential and sensitive information was not discussed with, or
disclosed to, unauthorised persons, knowingly or unknowingly. The interviewees and participants
of focus groups were treated with objectivity and impartiality. In relation to the interviews with
household members and inspection of their latrines, the consultant insisted on excluding
community leaders from accompanying him, to avoid possible bias in choosing or avoiding
particular homes (such as those known to not have latrines), or unconscious intimidation to
provide the correct answers, and requested permission to visit the latrines.
5.
Activities Undertaken
The detailed overall work programme and travel itinerary for all three phases of the evaluation is
included in Appendix 6. The following is a summary of the activities undertaken.
5.1
This phase took place from the signing of the contract (18/03/2015) to arrival in Timor-Leste
(14/04/2015) and included the following:
The documents reviewed in this phase, together with others reviewed in-country and afterwards,
are listed in Appendix 6.
5.2
This phase started on 15/04/2015 and ended with the departure of the consultant from TimorLeste on 28/05/2015. The key activities undertaken are summarised below in Table 3.
Table 3: Summary of Field Work Schedule and Itinerary
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
13
Week
Location
Main Activities
15-17 April
Dili
20-24 April
Dili
27-30 April
Viqueque
4-8 May
Ermera
Dili
11-15 May
Oecusse
Dili
18-22 May
Aileu
Dili
25-28 May
Dili
Debriefings.
Departure
Activity
Description
Document
reviews
9
4
3
4
4
Interviews
10
6
22
37
2
Focus group
discussions
13
Inspections
13
10
43
The list of meetings with stakeholders and key informants interviewed can be found in Appendix
8.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
14
6.
Findings
In this section the information obtained by the final evaluation and the endline survey is presented
and analysed in relation to the evaluation questions (as listed in Appendix 5), as the basis for the
subsequent discussion of conclusions, lessons learnt and recommendations.
6.1
Relevance
6.1.1
The project and all its components remained highly relevant throughout, as confirmed by all target
groups, project partners and key sector role players in the interviews and FGDs conducted by the
consultant. At household level the priority accorded to improved access to potable water remains,
as is to be expected, extremely high. Less obvious, but no less important, is the increased
importance of sanitation and hygiene for the target communities, as expressed by them, thanks
to the projects efforts to raise awareness in this regard.
The importance of the school WASH interventions was also confirmed, the most frequent
comment (from all stakeholders) being that much more of the same is needed.
The different capacity building interventions remain highly relevant and necessary for all the
categories of beneficiaries and in this regard as well, additional needs were identified.
6.1.2
The project was fully aligned with and supportive of government policies and priorities as defined
in the Strategic Development Plan (SDP 2011-30) and specific WASH sector policies, and as
confirmed by key stakeholders interviewed. WASH in general, and water supply in particular, are
given high priority in the SDP, both as part of the health strategy and for improved nutrition as
part of rural development, being described as a vitally important element in the economic and
social development of Timor-Leste. The project design and strategies were fully in line with
government policies for the sector.
The project also contributed to developing policies such as the Basic Sanitation Policy (approved),
the National Water Supply Policy, National Water Resource Policy and Law, WASH in School
Guidelines, BeSI Committee Terms of Reference, and others. Although the latter have not been
approved yet, they are already being used in the sector. Practical support to sector development
included training of CLTS trainers, in cooperation with other donor programmes. UNICEF is
acknowledged as one of the lead organisations in the sector and a highly respected partner of
government.
6.1.3
The project was aligned with UNICEFs Country Programme strategy and objectives, WASH
being a key component of the Child Health and Nutrition component, where it is intended that
UNICEF will continue to support WASH services in rural communities, focusing on scaling-up of
CLTS in priority districts through partnership with the Ministries of Health and Public Works, NGOs
and development partners. It is also part of the Quality Education component, where the
commitment is to support the achievement of the Governments goal of ensuring that all public
schools have access to clean water and sanitation facilities by 2020, particularly in priority
districts.
The project was also aligned to EUs Country Strategy and general development objectives which
consider WASH as key enabling factors in improving nutrition, especially of children under five,
and health in general. Malnutrition (and stunting in particular) is a significant challenge in TimorLeste.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
15
6.1.4
The project was generally well designed with a clear intervention logic, based on the standard
components of a WASH intervention, namely water, sanitation and hygiene (including for
schools), and one for institutional capacity building. These Additional positive elements were the
inclusion of sanitation and hygiene in the same Result (a more appropriate arrangement when
the CLTS approach is used as in this case), and of policy advocacy activities in the institutional
capacity building Result, as well as implementation through local NGOs.
These components, together with the necessary activities to achieve them and the short, medium
and long term objectives are adequately summarised in the logical framework (LF) and describe
well the theory of change underpinning the projects design, as described in section 2. The LF
was mostly well designed, with all the elements necessary for it to use as the basis for planning
and monitoring of activities and outcomes. Some improvements to indicators were made as a
result of recommendations by the EUs ROM (results-oriented monitoring) reviews.
The assumptions were relevant and well formulated at the appropriate levels. Some of them have
not held completely and have affected the project. This is discussed under Risk management
for efficiency, effectiveness and impact, in the sections that follow.
The projects institutional design and capacity building strategy was generally appropriate, with
maximum involvement of local institutions and government, from GMFs and PTAs (parent teacher
associations) at community level, to local NGOs for implementation and local level government
to provide training and to support O&M. A dedicated capacity building component (Result 1) was
included to overcome weaknesses in all these institutions, but may not have been enough to
ensure sustainability; this is discussed further in section 6.5.
The project design placed significant emphasis on sustainability, recognising that it will depend
on government capacity to support community-based water management structures (GMFs) and
ongoing sanitation and hygiene promotion at household level. Additional training for GMFs was
included in the standard methodology.
The budget was well constructed and mostly accurate, although possibly too conservative, since
the project managed to surpass most targets (including construction of facilities) by some margin,
without exceeding the budget.
In terms of the inclusion of cross-cutting issues, there was no clear gender strategy, although
some elements were included during the course of implementation. Environmental issues were
also not specifically highlighted in the project proposal, or included as indicators in the logframe,
although in practice several aspects were included, such as protection of water sources. This
included protecting and planting trees in the upper reaches of streams, bio-fencing of intakes and
creating drain diversions for protecting water sources from contamination. All these measures are
included in the GMF training curriculum (CAP 2) and were applied in many cases.
One example of the absence of a clear policy was the lack of erosion protection guidelines for
construction; none of the reservoirs, standpipes or buried pipelines included storm water diversion
works to prevent erosion, of which a significant amount was observed. The technologies
employed were appropriate and the community water supply and school water and sanitation
facility designs used were based on governments standard guidelines, a very positive feature of
the sector in Timor-Leste.
6.2
Efficiency
The efficiency with which the project was implemented was high in most aspects; exceptions are
noted below. Planning, budgeting, monitoring and management of financial, human and other
resources were generally good and all appropriate management tools were used. All activities
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
16
were carried out, in many cases in excess of what was originally planned, due to some extent to
the nine month no-cost extension. The budget was fully spent, although there are some issues
around possible ineligible expenditure to be resolved. The quality of products was high.
6.2.1
Although the project required a nine month time extension, equivalent to an additional 25% of
time, it also exceeded most targets by about the same proportion. Early delays were largely
corrected by the end of the third year, and all targets were achieved by then, except in the case
of school WASH facilities. The general delays were due to a number of factors which are not
uncommon and have to be expected in most projects, such as occasional procurement delays
due to having to import most materials and equipment, and having to redesign some water
schemes due to factors such as being denied access to water sources by land owners. The most
significant delays were caused by the national election in 2012 and a longer than average rainy
season and flooding in 2013 which limited access to communities.
The delays in the schools component were such that by the end of the third year only 17 of the
planned target of 24 schools had been completed and the tender for the remaining schools had
not been launched yet. Amongst the causes of slow progress was the use of some private
construction companies that were unable to complete works in the stipulated time due to lack of
financial capacity to manage the required cash flow; the competitive bidding procedures were not
always effective in eliminating these financially weak companies. In some case companies lacked
qualified technicians who could understand the latrine design, resulting in errors requiring time
consuming rectifications and rebuilding. However, the advantages of supporting the still fragile
local private sector (as per government policy) seem to outweigh those of contracting larger
foreign firms.
The project team was generally effective at planning ahead to get materials on site before the
rainy seasons, and the rate of expenditure was generally in line with the time elapsed. There were
also minimal delays with processing funding tranches from the EU.
The particular delays in Viqueque and Manatuto appear to have been mainly due to the poor
performance of the two implementing NGOs, according to all sources consulted (the NGOs
themselves could not be consulted as one no longer exists and the other did not return calls). In
spite of considerable additional supervisory support provided by UNICEF and SAS (district office
of DGAS) they were not able to perform as required and their contracts were not renewed in 2014.
In Viqueque the main impact was felt in the CLTS programme, but also with the multi-aldeia water
system in Bahalara-Uain suco, which had to be completed by SAS, with less than ideal results.
The NGO was unable to resolve the unusually difficult social and institutional challenges at
community level (which remain unresolved) and it was unfortunate that it was not replaced earlier
as all efforts to assist it had not produced the desired results.
In Viqueque none of the target communities achieved ODF status. Several possible explanations
have been suggested, including cultural differences between this and other districts, the weak
social facilitation skills of the NGO and the negative impact of other programmes offering materials
subsidies to households to build latrines. It was not possible in the time available for this
evaluation to undertake a detailed enough investigation to reach definite conclusions concerning
the latter hypothesis, nor did the endline survey shed any light on the matter. However, from all
the responses obtained from multiple sources including a small sample of households themselves
in Bahalara-Uain, community leaders, the SDF, the GMF, the PSF (community-based health
promoter) who conducted some of the promotion work for the project, project staff and partners,
the consultant concludes that the impact of subsidies was not a significant factor. It would seem
that the inability of the NGO to resolve the social and institutional problems was the main reason.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
17
In other districts (especially in Oecusse) the project had to contend with similar potentially
negative impacts of other programmes offering subsidies and managed to do so effectively.
However, other programmes, such as BESIK, report this as a significant challenge.
6.2.2
The quality of project management in general was good. Regular (annual) planning and review
exercises were conducted with the NGO and government partners and the plans formalised in
annual agreements with the NGOs. They received funding tranches at quarterly intervals after
reporting on activities undertaken and accounting for the previous tranche.
The monitoring and quality control system was well designed and seems to have been
implemented effectively. An M&E plan was established, based on the logframe indicators, and
these were improved on the recommendation of the EU ROM reviews. A baseline survey was
conducted to establish baseline values for the indicators. Regular monitoring and reporting was
ensured. Quality assurance was ensured by means of technical supervision by UNICEF staff and
consultants and oversight visits from the UNICEF Regional Office.
Financial administration and accounting was done by both UNICEF Timor-Leste and the five NGO
partners, with final certification of financial statements being done by UNICEF Head Office (in
New York). The NGOs received their funding tranches on an imprest basis, i.e. having to provide
acquittals (and returning unspent funds) before receiving new tranches. The procedures were
defined in the agreements signed between each NGO and UNICEF. Some difficulties were
encountered with delays in finalising acquittals when the delay exceeded six months as UNICEFs
system cannot maintain advances as such for longer than this, requiring the process of approving
the advance to be re-initiated. In addition, the NGOs reported that they had to pre-finance their
activities for several months at the beginning of every year, and for several weeks and sometimes
months, every quarter, due to the slowness of the procedures. As with some of the other
challenges cited by the NGOs, there is no consensus as to the allocation of responsibility between
them and UNICEF, and it was not within the scope of this evaluation to conduct a detailed audit
of every delayed quarterly tranche to arrive at incontestable conclusions.
Procurement of equipment and materials was UNICEF's responsibility and the system seems to
have been relatively efficient, within the constraints of the thorough procedures of a large
organisation like UNICEF. There were also challenges with importation of some materials which
cannot be sourced locally.
Several queries were raised by the international auditing firm in its report on the expenditure
verification they conducted in 2014 on the first three years expenditure, and as a result they
considered some 122.000 (23% of the value of the sample of transactions verified) to be
ineligible. However, 41% of this amount refers to salary costs apportioned to UNICEF staff
according to the ratios defined in the budget, without time sheets to confirm that the time was
actually spent on the project. This is a standard practice in EU funded projects and it is thus
difficult to understand why it is being queried in this case. A further 45% of the expenditure
considered ineligible refers to sand and other local materials delivered without respective delivery
notes, for facilities that were verified as having being built to specification. This lack of
documentation does not seem to justify the corresponding expenditure being considered as
ineligible.
By the end of the project the budget had been fully spent. In addition, it is possible that what
UNICEF spent from its own core funding increased its total co-funding to more than the
contractual 40%. For example, UNICEF continues to provide follow-up at its own expense to the
few school sanitation contracts that have not been fully completed (e.g. two water connections
still pending) even after the end of the project. This is one of the major advantages to the EU of
co-funding with large organisations which do not rely only on project funds to ensure the smooth
implementation of projects.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
18
6.2.3
Quality of products
The quality of the infrastructure built, of community participation, of documents produced and of
training provided is judged as generally of a very high quality. In most cases, however, these
could not be assessed against norms and standards previously defined by the project, as
suggested by the evaluation questions.
Infrastructure was the exception and was built according to the standard government design
guidelines. Detailed comments of the facilities inspected are contained in the data base in
Appendix 10; these were found to be in almost all cases of solid, durable construction. Some
areas of detail could be improved, however, to reduce the burden of maintenance, as noted in
Appendix 10. Some of these are shown in the photographs in Appendix 12, namely:
Lack of overflow drainage and control on water tanks at schools, with the result that most
of those inspected were overflowing, causing muddy and unhygienic surroundings. The
inlet valves were already dysfunctional due to tampering by children because they are not
buried in protected boxes.
School toilet blocks with lightweight drain inspection box covers that are susceptible to
tampering by children and filling of the boxes with stones and/or refuse.
Lack of storm water diversion works to protect building, pipelines and standpipes from
erosion, thus increasing the risk of cracking and other damage.
Poorly built waste water drainage systems on otherwise well-built standpipe aprons, that
are already separating from the aprons.
Poorly finished standpipe aprons with plaster screed already peeling off.
The quality of community participation, documents and training was (as is normal) not specified
up-front in terms of standard norms. The quality of participation and training has to be assessed
against the results achieved. Documents are assessed against the benchmarks of others
analysed by the evaluator elsewhere. This is discussed further below.
The results achieved in eliminating open defecation are a good indication of the quality of
facilitation of the CLTS process, at least in three of the five districts, as is the mostly good
performance of GMFs as an indicator of the quality of facilitation of the CAP (Community Action
Planning) process. GMF performance can also be assessed by the mostly very efficient process
of construction of water systems with a high level of community participation.
The exceptions in Viqueque and Manatuto, due apparently to the poor performance of the NGOs,
only serve to confirm the good quality of facilitation in the other districts. In this context it is
necessary to emphasise the poor quality of the water system installed in Bahalara-Uain, the
details of which are noted in Appendix 10. Unless an appropriately funded and facilitated
rehabilitation intervention is undertaken on this system it will not be sustainable and will thus not
provide the service to the community (around 2.000 people) it was intended to, and will be an
ongoing social and political challenge for government.
The quality of documents produced by the project and by government with project support or input
is also very good, notably the WASH in Schools Guidelines, the BeSI Committee Guidelines, and
the SDFs Manual and various training/awareness materials. They are all relevant and appropriate
to local conditions and to the target audience, comprehensive, gender and human rights sensitive
and professionally laid out.
6.2.4
Cost effectiveness
In comparison with similar interventions elsewhere, the project was cost effective. The consultant
has considerable experience with rural WASH projects and has also been provided with
comparative costs by EU operations managers. The average cost per person in the target
communities that have improved their access to water and sanitation (total project expenditure
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
19
divided by total number of beneficiaries) was approximately 114 Euros. The proportion of project
implementation costs and overheads (UNICEFs total costs) to total expenditure was around 22%,
also a cost effective ratio. In the case of some organisations, including international NGOs, this
ratio can be as high as 50%, with most at around 25-30%.
Also in terms of full lifecycle costs, the use of low maintenance technologies, together with solid,
durable designs, for water systems and school WASH facilities, although more expensive in terms
of the initial investment, are more cost effective in the long run and thus more sustainable. Some
relatively minor exceptions (notably the relatively less durable water tanks at schools) are noted
in Appendix 10.
One area where it was not possible to fully adhere to the principle of building long lasting
structures was in the CLTS programme where many of the latrines built have not lasted more
than a year or two. This is the main challenge of the CLTS approach and is discussed in more
detail under sustainability.
6.2.5
In general the approach of implementing in partnership with local NGOs who in turn partnered
with target communities organised by them, was vindicated in this project. In three of the five
districts all of the advantages of the approach and none of the potential disadvantages were
confirmed. The advantages include a high level of ownership of the process and of the
responsibility for O&M, particularly of the water systems, as well as a generally thorough
understanding of how each system works, on the part of the GMFs. In addition, the combination
of culturally sensitive NGOs and proactive community leadership ensured that, in most cases, the
complex and potentially disruptive issues of ownership of and access to water sources and land,
right of way for pipelines and siting of facilities, etc were negotiated without undue delays.
The problems encountered in Viqueque, and to a lesser extent in Manatuto (where the NGO was
technically weak), have already been mentioned. In Viqueque the NGO should have been
replaced earlier. While it is understood that UNICEF wished to first exhaust all avenues to enable
the NGO to improve its performance, three years is considered too long to wait, since the losers
in this case were the community (over 1.000 families) who should be prioritised over the NGO.
The solution adopted in Bahalara-Uain of transferring the responsibility for implementation to SAS
for the last 15 months was not ideal and the results are not up to standard.
Experience with the alternative model, using private sector building contractors supervised by
government officials, has not been as positive, at least according to many of the GMFs and most
of the INGOs interviewed. In the case of school WASH facilities, for example, communities
complained of lack of communication and information about progress, lack of consultation and
delays in procurement of materials and payment of local labour. The quality of construction is not
necessarily better than that achieved by NGOs, and the costs are higher. The same level of
community participation and thus ownership, so vital for sustainability, is not achieved.
In the three districts where there were no problems with NGO performance and it was possible to
interview them, their assessment of the implementation process and the quality of project
management provided by UNICEF was mostly very positive. They acknowledged the support and
training received and the fluent communication and coordination. Some problems experienced
with funding and administration of the imprest system have been mentioned.
However, in terms of the quality of their relationship with UNICEF as partners, the assessment is
less positive. They perceive the relationship to be purely one of sub-contractors in which they are
not genuinely involved in joint decision making, although it was clear that the UNICEF team went
to considerable lengths to do just that. It would seem that this perception is more likely the result
of the inevitably unequal power relations and their reluctance to express concerns more
forthrightly. They also feel the need for support in organisational strengthening, such as provided
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
20
by a number of INGOs to their local NGO partners. This could include support for developing
organisational development strategies and even assistance in diversifying their funding sources,
as well as basic training in financial management, accounting, project management, etc.
6.2.6
Through UNICEFs very active participation in sector forums and coordination structures, the
activities of the project were effectively coordinated with those of other programmes at both
national and district levels. This was ensured right from the design phase of the project, during
which the identification of target sub-districts was made in such a way as to avoid overlap with
other donor-supported programmes. Support for development of policies and guidelines was also
effectively coordinated through participation in sector forums and regular bilateral communication
and coordination. Effective synergies were achieved with the BESIK programme in particular.
6.2.7
Risk management
Most of the risks associated with external factors preventing the project from achieving its planned
outputs, having implemented the planned activities and delivered the required products, were
anticipated in the logframe. However, in this regard the most significant external risk (not included
in the logframe) was the inability to influence government decision-making time frames. Even
though UNICEF has more influence and leverage with government than most cooperation
partners in the sector, it was not able to significantly influence these processes. One of the
outcomes is that the BeSI Committee Guidelines have still not been approved and it is clear that
without them and the accompanying formalisation of the Committees, their functioning is not likely
to improve.
Other risks, such as bad weather, procurement delays, etc appear to have been competently
managed.
6.3
Effectiveness
In general the level of effectiveness of the project was high. The targets for both water and
sanitation were exceeded by up to 25% and the (input) indicators for capacity building were
achieved. However, targets for capacity building in terms of the levels to be achieved, were not
defined; some progress was made, especially at community level, but less so within government,
as is to be expected in the context of a developing country emerging from a long period of conflict
and a stagnant economy at the time of restoration of independence.
6.3.1
The achievements of the project in terms of delivering the planned outputs and results were as
follows:
Result 1: Local authorities, particularly District WASH Committees (BeSI), are better able
to lead and support WASH activities in 5 districts, supported by national
authorities, policies and guidelines.
This component was focused on building the capacity of the project partners, particularly at district
level, through activities such as training of BeSI Committee members and SDFs, providing
technical and logistical support to the development of national WASH policies (e.g. WASH in
schools) and procedures, improving WASH information management through technical and
training support to the DNSA and Ministry of Health to improve their information systems and
conducting KAP studies.
The indicators for this Result, and respective levels of achievement were as follows:
i.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
21
UNICEF actively participated in the CAP review process initiated by DNSA with the support of
BESIK, which led to the finalisation of the improved SDF Manual. In addition, the project
developed a training module to provide refresher training to GMFs on O&M, to complement the
last module, as recommended by the second EU ROM review.
ii.
All SDFs in target areas have received ToT on Community Action Plan (CAP) processes
and are supporting GMFs
The training was carried out as planned for all the SDFs and their assessment of its usefulness
was very positive. According to the NGOs, the performance of the SDFs improved. The improved
SDF manual was especially appreciated by all those interviewed. To the limited extent that it was
possible for the consultant to evaluate the capacity of the SDFs, it seems that in general they
have the required skills, experience and motivation.
Regarding the level of support being provided by the SDFs to GMFs, achievements are very
variable from one district to another but it was not possible to obtain precise, quantified
information. SDFs report relatively regular and frequent supervisory visits in most districts
(Oecusse being an exception, due to unfilled posts), but most GMFs interviewed had not seen
their SDF for almost a year. On the other hand SDFs cite inadequate budgets from SAS to achieve
their work plans for travel to communities, whereas their SAS supervisors insist that they are
adequately funded.
However, it is clear that in general the SDFs are under-funded and/or inadequately supervised in
most districts for the GMF system to be fully sustainable. This is an area that needs ongoing
attention and support.
In conclusion, the results as defined by the indicators were achieved. However, they only measure
partially to what extent local authorities are better able to lead and support WASH activities.
The assessment of this evaluation, based on all the different sources consulted, is that while good
progress has been made in creating an enabling policy environment at national level, the project
partners at district and sub-district levels are not yet able to provide leadership effectively without
project support. The BeSI Committees are not functioning as intended yet, their terms of reference
have not been approved yet and there is inadequate funding for SAS and the Ministry of Health
to provide the leadership needed to scale up water supply and CLTS programmes, and fund
maintenance support.
Result 2: Target sub-villages and primary schools have improved water sources and are
able to manage and maintain their water systems.
The original targets for water supply included building 34 water systems through a participatory,
community-based process (the so-called CAP-Community Action Plan process), culminating in
the establishment of a community managed O&M system based on GMFs for all 34 systems. At
the same time 23 schools were to benefit from improved water supply and sanitation facilities.
The achievements are summarised in Table 5, below.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
22
Total
achieved
Above
planned
(%)
34
44
+ 26%
34
44
+ 26%
24
34
44
+ 26%
34
44
+ 26%
23
28
+ 22%
The figures show clearly that the project was very effective in this component and amply exceeded
its output targets in all aspects. In addition, 24 point sources (protected springs and shallow wells),
not originally anticipated, were built where gravity-fed piped systems were not feasible. The 44
GMFs formed also received additional refresher training in O&M after completion of their systems,
and GMFs were formed per aldeia for some of the point sources (not reflected in reports).
The indicator for this Result was: 90% of the targeted 34 GMFs and estimated 23 school PTAs
are maintaining their systems at project completion. This was intended as a measure of the
existence of functioning O&M systems rather than as a measure of their sustainability. However,
the exact meaning of functioning O&M systems was not specified. No comprehensive data is
available to assess this indicator: the project did not monitor system performance after
completion, and the data from DNSAs SIBS information system does not provide an accurate
reflection of system functioning. The endline survey included 12 systems and 22 schools, whilst
the final evaluation visited 13 systems and 10 schools.
Based on the observations of the endline survey and final evaluation, it can be confirmed that all
water systems had a GMF in place and all but a few of them were functioning reasonably to very
well, with enough funds collected for regular maintenance, uninterrupted supply and only dripping
taps (see Appendix 10 for details). All of the schools reported having some sort of maintenance
arrangement and a number of them have an arrangement with the local GMF to assist with
maintenance. 21 of the 22 schools surveyed in the endline survey had PTAs with WASH
responsibilities, although this did not include repairs. The effectiveness of these maintenance
arrangements is discussed further under sustainability.
Result 3: Target sub-villages and primary schools have improved sanitation and
demonstrate measurable improvement in knowledge and attitude relating to
improved hygiene and sanitation.
The original indicators and targets for this Result, and achievements, were:
i.
80% of target villages certified by MoH as ODF by project end (30% by mid-project)
The original number of target villages was 55. During the course of implementation, some were
excluded, mainly in Viqueque, and others added. The final number in which CLTS activities took
place was 63 (see Appendix 3 for the full list). ODF certification was achieved in 53 of these (with
a total population of over 20.000 people), which represents 84%, thus surpassing the target.
Although relatively high rates of subsequent reversion to OD were recorded in a few sub-districts
(discussed further under sustainability), this is an excellent level of effectiveness and bodes well
for the future of the CLTS approach in Timor-Leste.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
23
ii.
For reasons not explained in the report, the endline survey and KAP study did not attempt to
measure this indicator. The only hygiene related behaviour amongst children that was included
was the practice of children reported to wash their hands after using a latrine as reported by the
adult respondents in the household survey. This was found to have increased from 11% at
baseline to 46.4% at endline. This increase could be a result of handwashing campaigns at
schools but this is not known and it should be borne in mind that it is adults report of their
childrens practice.
Other aspects
Besides the indicators of the logframe, the baseline and endline surveys also measured a number
of other changes in knowledge, attitudes and practices that the project was designed to effect.
These are summarised in Table 6, below. Particularly noteworthy was the very significant increase
in knowledge amongst primary care givers of children of the causes of diarrhoea in children and
the most effective methods of prevention and treatment. On the other hand, it is clear that more
work is needed to increase the practice of hand washing after defecating.
The relatively high level of open defecation reported, in spite of all communities having achieved
ODF status, except in Viqueque, is a result of households ceasing to use the latrines built during
the CLTS process. This is discussed further under sustainability.
6.3.2
Baseline
Endline
(2011)
(2014)
58%
4,4% of adults
27%
99% of adults
96% of
children
26% of adults
34%
58%
55%
58%
17%
88%
93%
31%
13%
6%
8%
52%
70%
41%
11%
56%
The Specific Objective of the project was: National and district WASH personnel and community
groups are better able to sustain equitable access to, and use of, improved drinking water and
basic sanitation, and adoption of improved hygiene practices. However, the indicators do not
directly measure the ability to sustain equitable access although they would seem to be a better
definition of what the project was trying to achieve. They and the degree to which they were
achieved were as follows:
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
24
i.
The endline survey (with a sample size of about 13%) found that 90,3% of all households in the
43 aldeias visited where water systems were installed had access to improved water sources.
The average time needed to collect water was reduced from 52 minutes to 19 minutes. The report
does not shed any light on why some households do not have access, but it seems from an
analysis of the raw survey data that the overwhelming majority of those without access were
residents of aldeias whose water systems were not functioning at the time. In some aldeias visited
during the final evaluation, examples were reported of households who were excluded from
access due to technical impediments such as distance from the system or elevation of
homesteads above that of the source. Only one of the 13 systems visited (Bahalara-Uain in
Viqueque) was not functioning; as already noted, this system has very particular characteristics
and is not a typical example.
In the case of the schools, the final evaluation found that only seven of the 10 schools visited had
a functioning water supply; two of those without had not been connected yet and one (in BahalaraUain) was connected to a dysfunctional system. The endline survey found only one of the 22
schools visited didnt have water and 16 of the 21 with a source reported that it was very reliable.
Data for targeted schools not visited are not available.
In terms of numbers of people, the target number of beneficiaries of improved community water
supply was 16.400 whereas 18.132 gained access. The school WASH intervention benefited
5.450 against the target of 4.600.
In conclusion, the objective of improving water coverage was not achieved at the time the project
ended, but will be within a few months when those schools that had not been connected yet will
be. Perhaps the target of 100% coverage was too ambitious as there are always likely to be some
households excluded or a few systems temporarily out of order on any particular day.
ii.
This objective was not achieved. All project partners who are members of the committees in the
five districts were interviewed and confirmed that they meet only sporadically, and for specific
purposes such as to certify a community ODF, and only when project financial support is
available. In Ermera more progress has been made than in the other districts. Most respondents
reported that bilateral coordination between BeSI Committee members has improved.
One of the bottlenecks of this target is the absence of nationally approved BeSI Committee
Guidelines. UNICEF in coordination with WASH partners supported the drafting of BeSI
Guidelines. The document went through a number of national level consultation processes before
the final draft. This was submitted to the Secretary of State of Water, Sanitation and Urbanization
in Dec 2013 but it has not been approved yet. However government has demonstrated its
commitment to improve inter-ministry coordination by endorsing and implementing the Draft BeSI
Guidelines in their 2014 High Level Ministerial commitment for Sanitation and Water for All (SWA).
iii.
20% improvement in number of children that wash hands with soap or other hand washing
agent before eating and after defecating
The endline survey found that the proportion of children reported by their mothers to habitually
wash their hands before eating had increased from 82,0% at baseline to 95,8%, an increase of
17%. In the case of handwashing after defecating, there was a reported increase of 422% (from
11,0% to 46,4%). However, self-reporting and reporting from mother are unreliable indicators for
actual handwashing practices because of a number of obvious biases. Self-reporting has been
documented to overestimate actual practice by a factor of 3 to 10.
6.3.3
Contribution to MDGs
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
25
The project, as with any WASH intervention with a schools component, was expected to
contribute to the following MDGs:
MDG 2: Achieve universal primary education
MDG 4: Reduce child mortality
MDG 7: Ensure environment sustainability (Target 7C: Halve the proportion of people
without access to safe drinking water and basic sanitation)
The contribution made to the first one (MDG 2) was not measured, i.e. attendance rather than
enrolment was measured, and this is reported under Impact. The contribution to MDG 4 was not
measured. The contribution to the last one (MDG 7) was measured by the baseline and endline
surveys, with the following result:
Access to safe drinking water: Increased in aldeias where systems were built from 45,5% to
90,3% - this exceeds the MDG target for Timor-Leste in these communities.
Access to basic sanitation: The increase was from 32,9% to 44,9%, which falls short of the MDG
target. Basic sanitation was defined in the same way as improved sanitation is by the WHOUNICEF Joint Monitoring Programme (JMP), but its interpretation in the field by many of the
enumerators was incorrect. Many latrines with pit covers/slabs made with logs and earth were
incorrectly classified as not having a cover or without a slab (and hence open pit) and thus as
unimproved.
Open defecation was reduced from 58% to zero in 53 of the 63 target communities, but by the
time of the endline survey, which was up to three years later in some cases, the practice of OD
had been taken up again by 20% of households in aldeias previously declared ODF. This figure
varied from 3% to 53% from one sub-district to another (see table 8 in section 6.5.2). The resulting
figure of 73% of households using latrines is probably a more accurate reflection of sanitation
coverage achieved than the 45% registered by using the criterion of improved sanitation. This
imprecision around the definition of basic or improved, i.e. adequate sanitation was the main
shortcoming of the baseline and endline surveys.
6.3.4
The almost universal answer given by community members to the question of what the main
benefit was that they had obtained from the project, was that they now had water close to the
home. In second place the reliability of the water supply was cited by most and only in some
cases improved quality was mentioned.
In relation to increased proximity, many also linked this to the possibility of building a good quality
toilet, in other words that they had only seriously considered building a toilet after the water system
was built.
A number of households are using water from the new systems to irrigate vegetable patches in
their back yards, others to wash coffee beans.
6.3.5
Capacity building
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
26
CLTS:
The intensive and repeated training and on-the-job capacity building activities
undertaken to improve the skills of CLTS facilitators, including with the support of Kamal
Kar (the initiator and leading international expert on the approach), had the desired
impact when measured by the results. Other organisations that co-sponsored and
benefited from the training commented very favourably as well.
GMFs:
The positive impact of the post-project refresher training in O&M on the performance
of GMFs was commented on by all project partners. The new SDF Manual, produced
by DNSA with BESIK support and input from UNICEF and other agencies, is a key
enabler in this process.
SDFs:
While the SDFs consider that they have benefited substantially from the training they
received on the new SDF Manual and enhanced CAP process, and are performing
better as a result, their effectiveness seems to be constrained by factors other than
gaps in their knowledge and facilitation skills.
SAS:
Schools: The School WASH Guidelines define what teachers are expected to be able to do, but
this is not being monitored at the level of detail required for their performance to be
assessed.
6.3.6
The development of norms and standards in Timor-Leste is still at an early stage; in the rural
WASH sector they are limited to standard government design and process guidelines and
standards, which is more than exists in most developing countries, and is one of the very positive
features of the sector, as already noted. The project complied fully with all of these. As a result,
for example, the quality, functionality and durability of the facilities built is very good. Some
examples were photographed and are contained in Appendix 12. The use of the SDF Manual to
ensure a standard approach to implementing community-based water systems, and the
improvement of the manual during the course of the project, also impacted very positively on the
social and institutional development outcomes.
Of significance is the degree of involvement of government officials in all phases of the processes,
from approval of designs and guidelines to training, supervision and quality control.
6.3.7
Risk management
The risk of external factors preventing the project from achieving its Purpose or Specific
Objectives, having achieved the planned outputs, was anticipated in the logframe. The following
assumptions were made in this regard:
i.
ii.
iii.
iv.
v.
vi.
The assumptions held and no other factors prevented beneficiaries from accessing the services
provided. With regard to sustainability of access, however, inadequate SAS capacity (assumption
ii above) remains a challenge, as discussed extensively elsewhere. Reversion to OD is also an
issue which depends on government capacity.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
27
6.4
Impact
The degree of achievement of the anticipated impact was variable, in terms of the achievement
of the indicators for the Overall Objectives, with the second being achieved and the first not,
although a positive trend was found. The beneficiaries did not comment of these aspects when
mentioning the benefits received from the project, rather they focussed on improved access to
services.
6.4.1
The Overall Objectives of the project, the respective indicators and the achievements (as
measured by the Endline Survey and KAP Study) were:
a) Children, women and men in target areas achieve better health outcomes
Target:
Achieved:
Achieved:
The Endline Survey was not able to enquire into other factors which might have mitigated against
a greater reduction in the incidence of diarrhoea in spite of the improvement in access to WASH,
which included knowledge regarding the causes and most effective treatment of the condition.
None were obvious. Based on replies received by the evaluator to questions (to a small sample
of householders) regarding what messages they had received from the project, he concludes that
it is possible that the greater awareness created amongst mothers about diarrhoea in children led
them to take more notice of its frequency and thus were better able to recall and report it in the
Endline, compared to the Baseline. This needs to be verified, however.
The increased school attendance rate, especially in the case of girls, is noteworthy. However, it
is not known if any other factors, such as the considerable improvement in the economic
circumstances of the majority of households (as noted in the Endline Survey), also had an impact.
6.4.2
Risk management
No external factors prevented the project from achieving the desired health impact, having
achieved the planned outcomes of enabling access to WASH. The risks that were foreseen were
the following:
i.
ii.
iii.
The second and third are related more to sustainability of benefits and are a challenge which
the project took into account in its interactions with government partners.
6.5
Sustainability
An assessment of the sustainability of the benefits delivered by the project involves considering
a number of inter-related factors, the most important of which are:
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
28
Due to the very different challenges and enabling factors that exist for the different components
of the project community water supply, household sanitation, hygiene behaviour change and
school WASH the findings of the evaluation in relation to sustainability will be presented per
component, as follows.
6.5.1
The institutional arrangements put in place to manage the community water systems built by the
project are the standard arrangements used in Timor-Leste, as stipulated by law, namely the
Decree-Law No. 4/2004 on Water Supply for Public Consumption. This requires a communitybased management group (GMF) to be established to participate in the design of the system,
support its construction and take charge of its O&M afterwards. Local government is responsible
for facilitating and supporting the process, mainly through sub-district facilitators (SDFs) who
report to the District SAS office. Clear procedures have been established for the whole process,
called the Community Action Plan (CAP) process, which are laid out in the SDF Manual. GMFs
are required to be formally registered with SAS, although this has not happened yet anywhere in
the country.
The institutional arrangements are thus solid and permanent. Their sustainability at community
level depends however on government capacity to support them effectively. The project invested
substantially in training and on-the-job capacity building of SDFs and SAS staff in general and the
feedback from trainees was positive. However, the overall assessment of this evaluation is that
the work of the SDFs is under-funded and in most districts inadequately supervised. Most GMFs
feel that they are not adequately supported. The prospects for improvement seem to be good,
however, as considerable work is being done by DNSA, with support from the BESIK programme
in particular, to strengthen the whole system. The report of a recent (September 2013) study of
GMFs undertaken by BESIK for DNSA gives a good indication of the direction that policy and
strategy development work is taking.
The relationship between GMFs and aldeia and suco leadership is mostly very good, which is an
important enabling factor for enhancing the authority of GMFs in their communities (especially
important for tariff collection) and providing support to solve non-technical problems. In all the
FGDs with GMFs and community members, local leaders were present and no obvious tensions
were noted, on the contrary, relations seemed sound. The only two exceptions were in Bahalarauain (already mentioned) where the main problem is precisely the dysfunctional village and suco
leadership, and Asumata (see Table 7, below).
Both the endline survey and the final evaluation (particularly the latter see Table 7, below, and
Appendix 10) found a high level of effectiveness amongst GMFs, with very specific problems as
the causes of dysfunctionality, namely poor leadership in most cases. These problems should be
able to be resolved by SDFs with more resources and supervision.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
29
Table 7: Functionality of water supply systems and GMFs visited during final evaluation
Identification
No.
District
Subdistrict
Description
Aldeia/
system
Type
*
Functionality
No.
Level
fami(%)
lies
Fund
(USD)
Most
senior
woman
Ermera
Ermera
Poeana
GP
145
100
435
Treasurer
Ermera
Ermera
Huitaso
GP
57
100
209
Treasurer
Ermera
Ermera
Saramata
GP
80
100
30
Treasurer
Ermera
Railaco
Leborema
GP
67
100
35
Treasurer
Viqueque
Viqueque
BahalaraUain
GP
450
Treasurer
Oecusse
Oesilo
Webaha
GP
102
100
600
member
Oecusse
Nitibe
Baocnana
GP
115
100
123
Treasurer
Oecusse
Nitibe
Bona 1&2
GP
73
100
270
Treasurer
Oecusse
Nitibe
Bona 3
wells
175
100
Treasurer
10
Aileu
Remexio
Samalete 1
GP
27
100
60
Treasurer
11
Aileu
Remexio
Samalete 2
GP
56
100
400
Treasurer
12
Aileu
Remexio
Lerolissa
GP
102
100
Treasurer
13
Aileu
Liquidoe
Asumata
GP
73
100
Treasurer
Comments
Almost all the GMFs visited showed a high level of ownership of their systems and adequate
knowledge of their roles and responsibilities. Most of the systems are functioning well, with
adequate funds for maintenance having been collected by GMFs in 11 of the 13 visited during the
final evaluation and 9 of 12 by the endline survey.
One of the issues raised by many of the GMFs when reflecting on the support received from the
SDFs was that of on-the-job mentorship. They found the post-construction refresher training very
helpful, but in most cases had not been visited by their SDF since then. This is closely linked to
the fact that most of the SDFs virtually only travel to visit those GMFs set up by donor funded
projects if the project covers the costs, whereas this project did not envisage any projectsponsored post-construction monitoring visits. This would suggest that a post-construction O&M
mentorship phase would have been very useful to further strengthen GMFs and might have
avoided some of the problems that beset Bahalara-Uain as well as getting the GMF of Asumata
(the last one in Table 7) to function.
O&M of the systems is facilitated by appropriate technologies, sound designs and good quality
construction in almost all cases, Bahalara-Uain in Viqueque being a notable exception. The fact
that all systems are either gravity-fed piped schemes or wells using a bucket on a rope, or
protected springs, i.e. there is no mechanised pumping, ensures that the O&M costs are very
affordable. The most common tariffs are 25c or 50c per family per month and even though most
GMFs are not able to achieve a 100% collection rate, the amount collected is sufficient to cover
maintenance costs. None of the GMFs interviewed had spent more than one months tariff on
repairs, some of them after 3 years. GMFs are trained to accumulate enough funds to cover a
major repair, none having been required on any system so far.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
30
However, replacement costs are not factored into the tariff as the understanding is that this is
governments responsibility, together with O&M support limited to soft issues.
All GMFs have some women members and in almost all cases the treasurer is a woman, but no
cases of female coordinators were found.
One of the secondary issues that the final evaluation sought to understand was water user attitude
to and willingness to pay for higher levels of service, i.e. individual connections. This was
prompted by international experience which shows that providing the level of service that users
really want and are willing to pay for results in better rates of cost recovery and thus more
sustainable systems. It was found, first of all, that virtually all households consider the improved
proximity of water to be the main benefit of the project. Secondly, many households, if not the
majority, at least in some aldeias, have invested in plastic hosepipes, some of them over 100m
long, to have water virtually on tap in their yards. Thirdly, it was found that almost all households
aspire to owning a permanent pour flush toilet, for which having water close by is necessary.
It could be concluded, therefore, that the issue of individual permanent connections is high on the
agenda of most households and GMFs. However, of the 13 systems visited, only one (in Ermera)
has established rules for individual connections, which include a US$25 connection fee, and have
made three connections. Two others have established rules and procedures, but have not made
any connections yet. The others have taken a conservative approach due to their systems not
having enough water to guarantee adequate supply for everyone with individual connections, or
because they prefer the hosepipe system which allows them to maintain visual control over how
much water each household consumes.
In conclusion, it would seem that there is as yet little demand for permanent, individual yard
connections.
6.5.2
Household sanitation
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
31
Ermera
Oecusse
Manatuto
Viqueque
No. of
aldeias
Baseline (2011)
% OD*
Year ODF
certified**
Endline (2014)
% OD*
Remexio
14
23
2010-12
10
Liquidoe
73
2013-14
Ermera
13
66
2011-12
Railaco
88
2013-14
Nitibe
81
2012-13
42
Oesilo
19
2013
15
Laclubar
53
2012
53
Laclo
82
2012
52
Viqueque
64
none
53
Lacluta
87
none
88
63
58
Sub-district
Total / average
27
Notes: * The figures for no facility/bush/field/pig pen/river have been used, excluding those for pit latrine without slab/open pit (for
reasons explained in section 6.4, above).
** These are the years in which the majority of aldeias were certified; a few were certified in subsequent years in some subdistricts.
Experience elsewhere in the world has shown an association between performance in CLTS,
both in achieving ODF status and in sustaining it, and the quality of facilitation (besides other
sustainability factors). The fact that a different NGO was responsible in each district could explain
the differences, and problems in other areas of performance would bear this out in Viqueque and
Manatuto, but this does not seem to be the explanation for the differences between Ermera and
Aileu on the one hand, and Oecusse on the other. In Oecusse the consultant gained a very
favourable impression of the quality of social facilitation and involvement of GMFs in sanitation
promotion, besides their excellent performance in managing their water systems.
It is clear that there were other factors at play, besides the quality of facilitation, that need to be
studied in more depth than was possible either in the endline survey or in the final evaluation.
Anecdotal evidence gathered by the consultant includes comments by some households that they
built latrines because they were told that their aldeia would only get a water system if it achieved
ODF status. While this could be considered as an intelligent incentive strategy, it could also be
construed as a mild form of blackmail which does not involve a change in consciousness
regarding the negative impacts of OD, and thus offers little guarantee that the owner will invest in
maintenance.
Comments were also heard regarding some NGO staff actually building or helping to build latrines
for the last few households that were holding up the achievement of ODF status in a particular
aldeia. This was driven by the pressure to achieve targets for ODF aldeias each year. This does
not necessarily mean that the households were not adequately motivated, but it might help to
explain why reversion to OD takes place.
By far the most commonly expressed opinion regarding the reasons for reversion to OD was that
it was related to affordability; the poorest households could not afford to maintain their latrines.
However, what the consultant was able to observe did not corroborate this entirely; many poorly
maintained latrines belong to families with much better constructed and maintained homes and
other outbuildings.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
32
The main conclusion from these findings is that sustainability of the CLTS intervention was
variable but generally poor and that this will not be resolved until the health system develops
enough to provide ongoing, frequent follow-up at household level to sustain attitude and behaviour
changes. The medium to long term prospects appear to be good as considerable effort is being
placed on developing and implementing the Ministry of Healths new Comprehensive Service
Package for Primary Health Care which includes the kind of promotion activities required.
Movement up the sanitation ladder
One of the most effective ways to enhance the sustainability of CLTS interventions is to
encourage and/or enable and/or assist households to improve the quality or durability of their
latrines, in other words to move up the sanitation ladder. In this way maintenance requirements
are reduced and a better level of service provided, both of which encourage willingness to
continue to use and maintain the facility.
This was another of the aspects of the sanitation intervention that the final evaluation paid some
attention to, by including it in the FGDs with GMFs and household visits and interviews, and
analysing what the project had done to promote sanitation marketing.
Of the 43 homes visited, 16 had upgraded their latrines from basic dry pit latrines to pour flush
units of varying degrees of solidness and durability. Of these, however, eight had received
subsidies of about half the materials to do so (from other sources, including a government pilot
project). Thus, of the total only eight (19%) had upgraded with only their own resources. The
endline survey noted an increase from 33% at baseline to 45% in permanent latrines (virtually all
pour flush with lined pits and solid superstructures) which attests to both the promotion work of
the project and the improved access to water.
However, virtually every householder interviewed said the only barrier to upgrading is the cost,
including that of sand and transport. Many families who have very precariously constructed and
poorly maintained latrines said they would prefer to save for a permanent pour flush unit than
spend more money on maintaining their temporary one. The evaluation confirmed the
importance of the work being done to finalise a policy on incentives for the most vulnerable
households, to be applied once communities have been declared ODF.
The two small pilot sanitation marketing initiatives funded by the project were largely
unsuccessful, seemingly because inadequate research was done into what the demand was, and
into constraints such as the cost of transport to get components to each homestead. In one
community pour flush components made by one of the projects remain unsold almost three years
later, due to the cost of transport and the preference for components that have a smoother finish
(plastic or ceramic) which reduces odours.
6.5.3
The endline survey assessed the extent to which the project had been able to improve knowledge
and awareness of sanitation and hygiene related issues and influence behaviour. The most
important findings were presented in Table 6, in section 6.3.1, above. Bearing in mind that the
projects interventions had taken place up to three years before the endline survey was
conducted, in some communities, the results also reflect the extent to which the impact was
sustained. Some of the data presented in Table 6 is thus repeated in Table 9, below, to facilitate
discussion. The fourth and fifth items, being reported rather than observed behaviours, probably
also reflect knowledge more than behaviour, like the first three. All five indicate a high level of
retention of messages.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
33
unsafe water
unsafe food
contact with faeces
cook food well
wash hands with soap
safe disposal of childrens faeces
Baseline
Endline
(2011)
(2014)
55%
58%
17%
13%
6%
8%
88%
93%
31%
52%
70%
41%
11%
56%
4,4% of adults
99% of adults
96% of children
26% of adults
34%
58%
With regard to behaviour change, the last indicator reflects a somewhat disappointing result, but
behaviour change is always more difficult than raising awareness. It is possible, however, that the
enumerators in the endline survey did not consider water storage facilities and bathrooms in the
same room as pour flush latrines as constituting a specific place set aside. The final evaluation
also did not find many facilities only used for hand washing, but all those who had pour flush
latrines said they used the same water used for flushing, for washing hands.
6.5.4
School WASH
Since all of the school water and sanitation facilities are still fairly new those visited by the final
evaluation were between a few months and two years old it was not surprising to find them all
in good working order, with only minor problems of leaking or broken taps at handwashing
facilities or latrine water storage tanks. The consultant was not able to inspect interiors, however,
as already noted.
One of the water supplies installed by the project, but not visited, includes an electric pump to
raise the water from the community water supply which is at a lower elevation, and is reportedly
functioning well. The sustainability of this arrangement will need to be monitored more closely.
The project appears to have done everything possible to ensure maximum sustainability of the
facilities, to compensate as much as possible for the generally acknowledged shortcomings of
the Ministry of Educations facility maintenance system. The endline survey found that all but one
school visited had a PTA with specific WASH responsibilities and formal relationship with the
GMF of the system supplying the school. The final evaluation also found that most GMFs also
take responsibility for maintenance of the water facilities at the schools. However, the training
provided to teachers on hygiene promotion and keeping facilities clean did not include any related
to maintenance and repair of facilities.
The potential additional maintenance burden that may be caused by poor attention to details in
design and installation has been mentioned in section 6.2, above.
6.6
Equity
As noted in section 4.5 in relation to the methodology of the evaluation, care was taken to integrate
human rights and gender equality considerations in all aspects. As far as possible, within the
constraints of community convened meetings, the consultant tried to ensure that key informants
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
34
included both genders. From the human rights aspect, it was possible to ensure that both duty
bearers (government, service providers) and rights holder (beneficiaries) were interviewed.
As noted in the terms of reference of the evaluation, the project was not designed on the basis of
a gender differentiated analysis of the needs of beneficiaries, nor did it define specific outcomes
aimed at greater gender equality. For an intervention involving the provision of gender neutral
public services to whole communities this is not necessarily a serious shortcoming. During the
course of the project the CAP process guidelines, in the form of the SDF Manual, were updated
and improved by DNSA, with input from the project and other stakeholders, and these do include
a focus on gender and other equity-related issues.
In practice, the project did try and promote greater representation of women on GMFs, with some
success. As noted previously, all but one of the GMFs visited by the final evaluation had a female
treasurer and all had at least 2 and mostly 3 women members (out of a total of between 5 and 8
members), although none had a woman as coordinator/leader. An obvious benefit for women in
particular from any water supply intervention is the reduction in workload and time to fetch water;
the extensive use of plastic hosepipes from standpipes to homes is an additional advantage. The
greater improvement in school attendance of girls compared to boys (as reported above) was
noteworthy, although further analysis is required to understand all the influencing factors.
Provision was made to improve the inclusion of physically disabled persons through ensuring
wheelchair accessibility to school latrines and water system standpipes. Some GMFs reported
making special provision to site standpipes closer to the homes of disabled people. However, no
provision was made in school toilets for menstruating girls.
6.7
Based on the consultants analysis of sector documents, interviews with a wide range of
stakeholders and field observations, it can be concluded that UNICEFs WASH strategy, as
defined in the new Country Programme Document, is appropriate to the needs of the sector and
UNICEFs comparative advantages, and covers all aspects comprehensively.
In terms of priorities, it seems that the following could benefit from more attention than is being
given at present:
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
35
7.
Summary of Conclusions
7.1
General
The project was appropriately conceptualised and designed for the country context and stage of
development of the sector, was well planned and was mostly well executed. It made use of a nocost time extension to complete the school WASH component while at the same time using
unspent funds to surpass its targets for water and sanitation coverage and the number of
beneficiaries, in the other components, and to consolidate operation and maintenance
arrangements.
The MDG target for increased water coverage was achieved, being raised from 45,5% to 90,3%
in the target communities, benefiting over 18.000 people (11% more than the original target).
84% of the communities in which CLTS was implemented were declared ODF, thus providing a
sanitary environment for over 20.000 people. By the end of the project 20% of households, on
average, had reverted to OD, but in the majority of aldeias the figure was less than 10%. Assuming
the households that had not reverted (73%) were all using facilities that meet the JMP criteria for
improved sanitation, the MDG target of 55% was achieved in 70% of aldeias and in six of the
eight sub-districts.
Through the WASH interventions in 28 schools (22% more than planned), more than 5.000
children benefited from improved access to water, sanitation and hygiene information.
In terms of the projects impact on institutional strengthening and capacity building, it was not able
to clearly define up-front what it intended to achieve in this regard and used indicators that
measure inputs rather than outcomes. In practice the impact has to be evaluated in terms of
improved performance. In relation to the government partners, while good progress has been
made in creating an enabling policy environment at national level, the project partners at district
and sub-district levels are not yet able to provide effective leadership without project support.
There is inadequate funding and other resources for SAS and the Ministry of Health to provide
the leadership needed to scale up water supply and CLTS programmes, and to ensure
sustainability, particularly of CLTS.
At community level the project made a significant impact on community capacity to operate and
maintain their water systems, with good prospects for sustainability, although government
capacity to provide the necessary support is questionable.
In comparison with similar interventions elsewhere, the project was cost effective. The average
cost per beneficiary was approximately 114 Euros. The proportion of project implementation costs
and overheads (UNICEFs total costs) to total expenditure was around 22%, also a cost effective
ratio.
In general the approach of implementing in partnership with local NGOs who in turn partnered
with organised target communities, achieved positive results in this project. However, in terms of
the quality of the relationship between the NGOs and UNICEF as partners, the NGOs perceive
the relationship to be purely one of sub-contractors in which they are not genuinely involved in
joint decision making. It seems, however, that this is more likely the result of the inevitably unequal
power relations and their reluctance to express concerns more forthrightly.
7.2
The project was very effective in this component and amply exceeded its output targets in all
aspects. In addition, 24 point sources (protected springs and shallow wells), not originally
anticipated, were built where gravity-fed piped systems were not feasible. The 43 GMFs formed
also received additional refresher training in O&M after completion of their systems, and GMFs
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
36
were formed per aldeia for some of the point sources. The objective of improving water coverage
was achieved, including for schools, if allowance is made for those schools that had not been
connected yet but would be within a few months.
The quality of the infrastructure built, of community participation, of documents produced and of
training provided was generally of a very high quality. An important exception was the water
system in Bahalara-Uain, Viqueque district, which had to be completed by DNSA-SAS due to the
poor performance of the partner NGO which was unable to resolve the unusually difficult social
and institutional problems at community level. Unless an appropriately funded and facilitated
rehabilitation intervention is undertaken on this system it will not be sustainable.
The effectiveness of GMFs was found to be good in most cases, with very specific problems as
the causes of dysfunctionality, namely poor leadership. Almost all the GMFs visited showed a
high level of ownership of their systems and adequate knowledge of their roles and
responsibilities. The problems noted should be able to be resolved by SDFs but it is clear that in
general they are under-funded and/or inadequately supervised in most districts, constituting the
weak link in the GMF system. This is an area that needs ongoing attention and support.
It would seem that there is as yet little demand for permanent, individual yard connections.
7.3
The high level of success in achieving ODF certification, mentioned above, indicates an excellent
level of effectiveness of this element of the CLTS approach and bodes well for the future of the
strategy in Timor-Leste.
With regard to the sustainability of latrine usage, however, it was found to be very variable.
Reversion to OD was between 42% and 53% in just under 30% of aldeias declared ODF
(including all of those in Manatuto), but 10% or less in 62% of them (all of which are situated in
Aileu and Ermera). This weakness will not be resolved until the health system is able to provide
ongoing, frequent follow-up at household level to sustain attitude and behaviour changes.
In Viqueque none of the target communities achieved ODF status. Several possible explanations
have been suggested, including the impact of other programmes in the district offering subsidies.
However, it was concluded that this was not a significant factor. In other districts the project had
to contend with the same challenge and managed to do so effectively. It would seem that the
weakness of the NGO in social facilitation and problem solving was the main problem.
The evaluation confirmed the importance of the work being done to finalise a policy on incentives
for the most vulnerable households, to be applied once communities have been declared ODF.
7.4
School WASH
Due to the good quality of design and construction and the relative newness of the facilities, their
level of functionality was high, the main problem being pending water connections or dysfunctional
water systems.
The project appears to have done everything possible to ensure maximum sustainability of the
facilities, to compensate as much as possible for the generally acknowledged shortcomings of
the Ministry of Educations facility maintenance system. The endline survey found that all but one
school visited had a PTA with specific WASH responsibilities and formal relationship with the
GMF of the system supplying the school. The final evaluation also found that most GMFs also
take responsibility for maintenance of the water facilities at the schools.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
37
8.
Lessons learned
8.1
General
The model of institutional arrangements that was used to implement the project was based on a
partnership with government that involved sharing the responsibilities for implementation of
activities between UNICEF (and its NGO partners) and different government departments, both
at national and district levels, while UNICEF retained the responsibility for accountability to
donors. This model is appropriate and necessary to acknowledge the responsibility of government
for delivery of basic social services and to help build capacity to do so, but carries with it the risk
of UNICEF being accountable for some actions over which it does not have full control.
In general UNICEF seems to have managed this challenge well, but a number of lessons emerge
that could inform future projects, notably:
In the case of the poor performance of the NGO in Viqueque, the decision was taken not
to renew their contract and for SAS to complete the Bahalara-Uain water system, because
no alternative NGO could be found with the required social facilitation skills, local
knowledge and availability of staff, according to UNICEF. The result was a poor product
in terms of both quality of construction and social facilitation. It is likely that if UNICEF had
terminated the contract earlier, and involved other facilitators, even for a limited period, to
assist SAS, a better result could have been achieved. The lesson learned is that, although
implementing agencies have an obligation to try to assist NGOs to improve their
performance, there are limits (UNICEF tried for three years to improve the performance of
the NGO, to no avail) and the priority should be the wellbeing of the beneficiary community
(which, in this case, has not yet benefited from the project), rather than the NGO.
A number of key outputs of the project involved approval of policies and guidelines by
government at very senior level, beyond not only the control but also the influence of the
project. In future it would be advisable to define deliverables in terms of submission rather
than approval. The case of the BeSI Committee Guidelines is the one that has affected
the project most negatively, as it is also linked to formalising the committees and accessing
budgets for them to function effectively.
A related lesson is that training and other capacity building activities on their own have a limited
impact on the performance of government officials if the other factors that constrain performance
are not also addressed. In the case of SDFs and community or suco level health workers, for
example, the main constraints seem to be limited operating budgets and in some districts
inadequate supervision.
A positive lesson learned was the validation of the model of implementing with local NGOs. Their
comparative advantage in managing community based construction of services that have to be
managed by communities afterwards, was reflected in the high level of ownership and
effectiveness of GMFs. One obvious proviso is that the competency of the NGO, as with any
service provider, needs to be assured. The project did not manage this in Viqueque and Manatuto,
where project performance was poorest as a result, and took too long to terminate both contracts.
While it is acknowledged that there were limited alternatives available to UNICEF, or possibly
none, according to UNICEF, it remains a lesson for the future to improve the assessment of NGO
capacity up-front and on an ongoing basis.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
38
The NGO model is also cost effective, in relation to commercial contractors. This is especially the
case for very small jobs in remote locations which contractors are reluctant to take on at affordable
cost.
One of the requirements for getting full value from using the NGO model, however, is to assume
some responsibility for enhancing the capacity of the NGO through general training and even
support for diversifying funding sources to improve their sustainability.
8.2
Since this component involved implementing a well-established model and process, the lessons
learned were not so much related to innovation as they were to validating approaches.
The project confirmed the value of employing standard design guidelines based on appropriate
technologies and solid, durable infrastructure, to minimise the complexity of operation and the
maintenance burden and thus enhance sustainability.
A number of approaches were validated, notably:
Where GMFs are partially or totally dysfunctional they need external support to resolve the
situation, especially to solve non-technical problems, such as in Bahalara-Uain and Asumata
(Aileu). In principle the SDFs should be able to provide this support, but are clearly not doing so,
at least not in these two cases. Both situations could probably have been avoided if the project
had made budgetary provision for a post-construction mentorship period until the end of the
project, during which the NGO supported the SDF to monitor GMF performance and solve
problems.
The use of plastic hosepipes from tap stands in lieu of permanent private connections seems to
be an appropriate arrangement and should be encouraged as it reduces the workload of women
and girls in particular and facilitates greater use of water and resulting improved hygiene. The
practice appears to be well managed by GMFs, but it would be advisable to include in the training
of GMFs some guidance on regulations on the use of hosepipes, for example to prohibit
permanent connections and limiting use only to filling storage containers.
8.3
The CLTS approach worked well in this project in most districts and in all but two sub-districts,
thanks to a large extent to the intensive, repeated training of trainers and facilitators. Much has
been and continues to be learned by UNICEF and other stakeholders in the sector about applying
the methodology, the key challenge being sustainability, i.e. reversion to OD.
Amongst the lessons learned, some of them obvious, were the following:
The effectiveness of CLTS depends very heavily on the skills of the facilitators. Not
everyone can be a good facilitator as it depends on personality and sensitivity as well. In
general, NGO staff tend to be better suited than government officials or private sector
consultants.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
39
Facilitators also need to be able to adapt their messages and approaches in accordance
with local conditions. The standard shock and disgust triggering strategy may not work
in every community. Recruiting facilitators from the same area is essential.
The challenge of sustainability will not be solved unless and until the health system is
strengthened to provide ongoing monitoring and promotion at household level. For this much
more funding is needed. Funding is also needed to scale up CLTS at present very little is being
done by government outside of donor funded projects.
A key lesson learned was in relation to the design and implementation of the baseline and endline
surveys, in regard to the definition of improved sanitation. One of the categories used for
unimproved sanitation was pit latrine without slab. It is difficult to imagine what this definition is
supposed to mean, since a pit without a slab is not useable at all. One of the categories for
improved sanitation was pit latrine with slab (concrete, wood/bamboo). In practice, due to their
lack of experience, what many of the enumerators did was to define any latrine with a slab of
wood or bamboo covered by earth, but with the wood or bamboo not visible, as a pit latrine
without slab and thus as unimproved sanitation. As a result the number of households classified
as having adequate sanitation was some 28% lower than actual.
It should be recalled that the JMP definition of an acceptable pit latrine is as follows:
" a dry pit latrine whereby the pit is fully covered by a slab or platform that is
fitted either with a squatting hole or seat. The platform should be solid and can
be made of any type of material (concrete, logs with earth or mud, cement, etc.)
as long as it adequately covers the pit without exposing the pit content other
than through the squatting hole or seat."
No mention is made of pit lining or roof or squatting hole cover. All basic dry pit latrines observed
in the final evaluation met this definition, and presumably in the endline survey as well.
Another lesson learned was that the main obstacle to moving up the sanitation ladder and
enhancing sustainability is the cost of materials and transport. For many households the
prospects of ever being able to build a sustainable latrine are remote without some support in the
form of a subsidy. In particular, due to soil conditions in most areas of the country, unlined pits do
not last more than a year or two, in most cases. The work being done currently to agree on an
incentive policy to be applied for the most vulnerable once a community has been declared ODF,
needs to be finalised and the policy applied.
8.4
School WASH
The project confirmed once again the obvious advantages for sustainability of employing
appropriately simple technologies and standard designs, and investing in solid, durable
infrastructure.
The major delays experienced by the project in completing the school WASH interventions on
time would suggest that the use of commercial contractors is not necessarily the best solution in
every situation. A combination of commercial contractors where good ones were available locally,
and NGOs elsewhere, might have been a more effective strategy and would have provided a
useful comparison of the pros and cons of the different approaches.
Involving GMFs directly in the maintenance of school water infrastructure appears to be a valid
strategy to enhance its sustainability.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
40
9.
Recommendations
The recommendations that follow are based on the main conclusions and lessons learnt, as
described above. They were finalised after a process of feedback and consultation that included
debriefing meetings with UNICEF senior management, EU Managers, some of the partners and
other stakeholders, besides various iterations of the final report.
For UNICEF:
1. While the implementation model used in the project of partnership between UNICEF and
In order to strengthen and enhance the sustainability of the generally effective model of
implementing in partnership with local NGOs, the following should be considered:
More responsibility should be assumed for enhancing the organisational capacity of the
NGOs through general training (e.g. project and financial management, strategic planning)
and even support for diversifying funding sources to improve their sustainability. Use could
be made of the experience of INGOs in this approach.
Strengthen the spirit of partnership with NGOs through being more sensitive to their
relatively weaker position in the relationship and hesitance to express concerns,
especially around matters relating to non-negotiable contractual provisions.
3.
Do regular refresher training of NGO, UNICEF and SAS technical staff on monitoring of
construction to ensure attention to detail for sustainability, with the use of checklists.
4. In future baseline and endline surveys, ensure that the definition of improved sanitation is
as policies and guidelines that rely on high level government approval should be specified in
terms of submission for approval rather than actual approval.
For government:
6.
DGAS, Health, and Education: Accelerate the process of taking over more responsibilities as
a service provider and funder, with the donor as a supporting partner, increasing own
budgets, especially for scaling up CLTS and supporting the O&M of water and sanitation.
The challenge of sustainability of CLTS will not be solved unless and until the health system
is strengthened to provide ongoing monitoring and promotion at household level.
7.
DGAS and Health: Undertake a review of experience to date in implementing CLTS to obtain
greater understanding of factors at play in the successes and shortcomings: what motivates
people, how to improve community leadership, reasons for reversion to OD, potential for
involvement of local entrepreneurs, etc.
8.
DGAS and Health: Develop a mechanism, possibly through the BeSI Committees, to improve
formal coordination between Government and Implementing NGOs (both local and
international) to strengthen their technical capacity to adhere to government design and
process guidelines and policies, so that their relative strengths in community outreach and
facilitation can be optimised.
9.
DGAS: Include in the CAP process a post-construction mentorship period to enable SDFs to
monitor GMF performance, provide further on-the-job training and solve problems until the
end of the project (with donor support in the case of donor funded projects).
10. DGAS: Lobby for finalisation and approval of i) the latrine construction incentive policy for the
most vulnerable once a community has been declared ODF; and ii) the BeSI Committee
Guidelines and respective operating budget.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
41
11. DGAS: Include as a standard responsibility of GMFs the maintenance of school water
infrastructure.
12. Education: Approve the School WASH Guidelines and include in the training package a
module on technical maintenance of facilities, prioritising the setting up of a functional system
that includes GMFs.
13. Education: In future school WASH infrastructure projects:
consider a combination of commercial contractors where good ones are available locally,
and NGOs elsewhere;
change the specification water tanks at schools to HDPE tanks to ensure greater
sustainability.
14. DGAS: Encourage the use of plastic hosepipes from tap stands where permanent private
connections are not feasible, and include in the training of GMFs some guidance on
regulating this practice, e.g. to prohibit permanent connections.
15. DNSA and Health: Decentralise water quality testing to facilitate regular monitoring of sources
susceptible to contamination.
16. Health: Include in the M&E system monitoring of moving up the sanitation ladder, i.e. of the
implementation gaps in rural water and sanitation and scaling up sanitation to complement
government efforts to address the high levels of under-five malnutrition.
Report on End of Project Evaluation of joint EU-UNICEF funded WASH project. July 2015.
42
APPENDICES
Appendix 1: Terms of Reference of Evaluation
Appendix 2: Logical Framework of Project
Appendix 3: List of target aldeias and interventions in each
Appendix 4: List of schools targeted for WASH interventions
Appendix 5: Evaluation Questions and Data Collection Methodology
Appendix 6: Work Programme
Appendix 7: Documents Reviewed
Appendix 8: List of meetings and persons interviewed
Appendix 9: Data collection instruments
Appendix 10: Data base of information gathered (separate Excel file)
Appendix 11: Curriculum Vitae of Evaluator
Appendix 12: Photographs (separate file)
CONSULTANCY NOTICE
UNICEF Timor-Leste is looking for a qualified international consultant to conduct evaluation of EUUNICEF jointly funded WASH project in Timor-Leste.
1. Background
Timor-Leste is the second youngest nation in the world. With its restoration of independence in 2002, TimorLeste was in a state of ruins with tattered infrastructure and a ceased economy. The country has faced
numerous political, security and development upheavals since then. However, the nation has made steady
progress in peace and democracy with two successful parliamentary elections in 2008 and 2012. The
demography of Timor-Leste is predominantly covered by young population with nearly half of the population
below 18 years . Along with the peace and democracy, the situation of children and women has gradually
improved in Timor-Leste. According to information from the Census 2010, the under-five mortality stands
at 64 per 1000 live births vis a vis its Millennium Development Goal (MDG) target of 96; and the primary
net enrolment rate (grade 1-6) is at 93.6, against the MDG target of 100.
However there are a number of on-going challenges. The most recent WHO-UNICEF Joint Monitoring
Programme (JMP) data shows an overall increase in the coverage of drinking water by 16 percentage
points from 2000 to 2012, while the progress in sanitation has been very slow in rural areas. According to
the JMP, 61% of the rural population use improved water sources compared to only 27% accessing
improved sanitation facilities. Poor access to water, sanitation and hygiene is one of the main causes for
diarrheal diseases in under five children, also contributing to the very high rates of under-five malnutrition
(50.2% stunting and 11% wasting). There are also major concerns regarding the WASH situation in schools.
About 54% of primary schools are without access to improved water supply and 35% have no latrines in
the compound and only 12% school have regular supply of soap which clearly indicates the lacks of basic
enabling environment for the school children to practice safe hygiene behaviors even though life skill
lessons related to sanitation and personal hygiene are covered in the classes.
Under the above circumstance The European Union (EU) and UNICEF Timor Leste agreed to implement
the program Improving Access to Water, Sanitation and Hygiene (WASH) in Rural School and
Communities through Capacity Development in June 2011. This Action targeted 55 sub-villages (aldeias)
in five districts (see the Annex 1 and 2 for more details) and has been implemented in collaboration with
national NGO partners and selected contractors by UNICEF and Government and the WASH sector line
ministries namely Ministry of Public Works, Ministry of Health, and Ministry of Education. This project,
covering 45 months (including 9 months no-cost extension), received funding support of 1.5 million from
EU and 1 million from UNICEF coming from different source including National Committees, DFAT and
Thematic Fund. The Project Intervention Logic is described as below.
2. Purposes of the evaluation
This evaluation will inform the results of the project based on five criteria (relevance, effectiveness,
efficiency, sustainability and impact) recommended by OECD-DAC. This type of external evaluation at the
end of the intervention is expected to contribute to both strengthening accountability of UNICEF for its donor
and key stakeholders including beneficiaries, and to learn from this experience to inform future WASH
projects. This end of the project evaluation was planned from the beginning as part of the EU funding
requirements. More specifically, the key objectives of the evaluation are:
To assess the relevance, effectiveness, efficiency, sustainability and impact of the UNICEF-EU
jointly supported WASH Action (for detail see section 3)
To assess the extent to which the project has contributed to broader development results at the
sub-national level, and lessons learnt that will allow the replication and scaling up of the
interventions.
To draw operational recommendations and lessons learned for further improvement and
enhancement of relevant sector policies, plans, strategies through analyses of the factors
contributing to the success or failure of the project.
To provide strategic guidance to UNICEF Timor-Leste in determining its focus areas of support to
relevant ministries in WASH under the current country programme (2015-2019) in addressing
critical child health issues
Efficiency
Effectiveness
Sustainability
Impact
Did the intervention respond to the needs of the various target groups?
Was the logical framework coherent enough to achieve the results?
Did the operation support the vision of the government and was it in line with
existing policy?
Did the project contributed in national policy development and reviews?
Were resources utilized and managed in efficient manner?
How well did the operation co-ordinate with other, similar interventions (if any) for
synergy and in order to avoid overlaps?
Would it have been possible to achieve the same results at a lower cost?
Were the outputs delivered as planned and in a coherent manner?
What was the level of quality and compliance of activities to the norms and
standards previously established by the program, both for the construction and
social intermediation components?
What is the likelihood that the operation and maintenance and repair of the
installed facilities being financed at the local and sub-national level for continuity of
the services (access to clean water etc.) after the end of the project?
To what degree did the project consider the existing structure or resources to
enhance the sustainability after the end of the intervention?
What could the project have done different to improve sustainability?
What were the main impacts (positive/negative, expected/unexpected) as
perceived by the different actors and beneficiaries of the project?
The below is general proposition to develop technical proposal for the applicant. There are mainly three
phases in this evaluation exercise as below.
i) Desk review and refinement of methodology
The list of references and documentations will be shared with the selected consultant. In relation to this
project, the baseline and end-line results of a Knowledge, Attitude and Practice (KAP) study (2011 and
2014) are available. These data need to be reviewed and utilized for the evaluation together with other
existing sources (DHS, Joint Monitoring Report, Project progress reports, etc.) as appropriate by
considering the disparities based on geography, gender and other key variables based on the availability
of disaggregated data. To ensure the participatory approach and refine the methodology based on the
context of the country, the consultant will have the opportunity to finalize the methodology during the incountry visit and meeting with key stakeholders. (For more detail, please see the part 6 below).
ii) Data collection, field work (site visit/interviews)
The systematic use of qualitative (e.g, structured interviews and focus groups) and quantitative (e.g. recent
survey results, existing routine data) methods are required to collect the additional evidence during the
country visit. The meeting with key stakeholders will be accompanied by UNICEF national staff during the
field work in Dili, and by the monitoring consultant (female, NOA level) hired by UNICEF during the site visit
outside of Dili based on the needs expressed by the consultant to facilitate the communication in local
language. Due to the nature of this end of the project evaluation with time and budget limitation, it is highly
recommended to identify two to three districts among five for the project site visit and data collection, focus
group discussion in the field based on the desk review analysis. The respondents and methods/tools
(interview, questionnaires, and focus group) need to be strategically selected based on the evaluation
questions.
iii) Analysis and Reporting
The final stage of analysis and reporting will be conducted remotely (Please see section 6 below for more
detail). The results of analysis need to systematically respond to the evaluation questions, and the report
will be written in a reader-friendly manner. Before finalization of the evaluation report, UNICEF will organize
one day workshop, gathering project team, stakeholders, beneficiaries and the evaluation team, to discuss
together recommendations and action plan drawn from the evaluation. This workshop would help ensure
recommendations are appropriate and owned by the project team and stakeholders, and will be facilitated
by UNICEF WASH section, with the international consultant joining through Skype.
6. Limitation of Evaluation:
Gender and Human Rights aspects need to be considered during the evaluation to draw relevant
recommendations as a formative evaluation. However, the project was not designed / planned based on a
gender differentiated analysis of the needs of beneficiaries and does not contain proper outcomes aiming
at greater gender equality in the areas of intervention. Even if project addresses practical aspects of gender
issues by improving access to water which is an activity mainly led by women, it's not sufficient to be
considered as having a gender sensitive approach. For the OECD Gender Marker is rated "0" because they
only consider practical aspects of gender issues without a real consideration of changes/developments on
the strategic aspects of gender issues.
While baseline /end-line KAP surveys were conducted, the data will not provide seasonal variance for some
of the key indicators including diarrhea incidence as this was not a longitudinal study. However, efforts were
made to conduct the end-line and baseline survey during the same season (Nov to Dec).
7. Activities, Task, deliverables and timeframe
The assignment is for 60 working days over the period of 15 March 2015 to 15 June 2015. The consultant
will provide a detailed timetable in its technical proposal, specifying the distribution of tasks and duration to
complete each task. The proposed sequencing in the table below is an indicative proposal which could be
improved in the technical offer. The right column indicates an estimated duration for the activity.
Activity
In
country
support
Remote
support
Duration
(working
days)
13
10
Timeline
Upon completion of deliverable (a); activity 1 - 6
in section 6.
Payment
20%
30 %
50 %
9. Management
The Chief of Planning, M&E and Social Policy will supervise the consultant, with technical and logistical
support provided by the Chief of WASH of UNICEF Timor-Leste. The UNICEF East Asia and the Pacific
Regional Office (EAPRO), particularly the EAPRO Regional Adviser WASH and the EAPRO Evaluation
Officer, will provide oversight and technical support. For the quality assurance of the key evaluation
exercise, draft inception and final evaluation reports will be shared with the mentioned regional advisors
and with the WASH evaluation specialist in the UNICEF Evaluation Office, to obtain their technical input
and clearance before finalization of the inception and final Evaluation report.
10. Qualifications and experience required:
Training and experience in the field related to water, sanitation and hygiene with academic
qualifications (at least a Masters degree) in civil engineering, mechanical engineering, sanitary
engineering, social sciences or combination of it.
Credible international expert with at least 8 years of professional experience in planning,
implementation, management, monitoring and evaluation of integrated water, sanitation and hygiene
programmes in developing countries.
Proven experience with similar programme evaluations in the WASH sector in developing countries,
preferably for the fields of WASH in school, hygiene behavior change approaches including CLTS
(Community-Led Total Sanitation), and community interventions.
Proven experience for EU project evaluations is an asset.
Ability to work independently.
Good communication and report writing skills in English.
Commitments to deliver the final products in line with the set TOR within the agreed timeline.
The consultant should adhere to UNICEF Evaluation Policy; to UNEGs ethical guidelines for UN
evaluations; to UNEGs code of conduct for evaluation and to UNICEF Reporting Standards which are
enclosed in the advertisement notice.
11. Condition of Work
The consultant will be provided a working desk in the UNICEF office while in Dili. The consultant will use
his /her own laptop for the assignment.
UNICEF will facilitate the arrangement of meetings and workshops with partners as required and will
provide logistic and financial support as necessary
The consultant is expected to undertake field trips to the selected project locations in the target districts
with transportation provided by UNICEF.
Interested candidates are kindly requested to submit the following information:
Letter of interest
Technical Proposal
Proposed budget and fees
CV/P-11 form ( UN Personal History Form)
Please send the submission to dilihr@unicef.org
Closing date for receipt of proposals: COB 3 March 2015.
Only short-listed candidates will be contacted.
Intervention logic
Overall
Objectives
ER 1: Local authorities,
particularly District WASH
Committees (BESI), are better
able to lead and support WASH
activities in 5 districts, supported
by national authorities, policies
and guidelines
Specific
Objective
Expected
Results
Assumptions
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
Sub District
Aileu
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Remexio
Liquidoe
Liquidoe
Liquidoe
Suco
Suco
Tulataqeuo
Tulataqeuo
Tulataqeuo
Tulataqeuo
Acumau
Acumau
Acumau
Fahisoi
Fahisoi
Fahisoi
Faturasa
Faturasa
Faturasa
Faturasa
Acubilitoho
Acubilitoho
Aldeia
Families
Total
Population
Water
Supply
Samalete (2 bairos)
Roluly
Aikurus
Dasilelo
Lerolissa
Fatumenario
Aimerahun
Berliurai
Mautoba
Diruhati
Berelico
Fakulaun
Caitasu
Raimerhei
Hautoho
Acumata
Aitoi
178
81
95
78
102
124
200
76
100
49
63
71
39
66
44
73
25
752
451
211
360
594
897
1265
573
600
353
317
171
279
275
215
338
150
Completed
Completed
no action
no action
Completed
no action
no action
Completed
Completed
Completed
Completed
Completed
Completed
no action
Completed
Completed
Completed
Completed Type of WS
System
2011 2 - WSS
2012 2 - WSS
2012 1 WSS
Oct-13
Oct-13
Feb-15
2013
2013
2013
1
1
1
1
1
2
WSS
WSS
WSS
WSS +1PS
WSS
PS
Feb-15 1 WSS
2013 1 WSS
Dec-14 1 WSS
12 Aldeias
Sanitation
Year of
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
2010
2010
2010
2010
2013 Dec
2012
2012
2012
2013 Dec
2014 Jan
2011
2011
2011
2011
2014 Jan
2013
2014
13 W Systems
Viqueque
18 Viqueque
Bahalara Uain
Welaco
95
467
Completed
Sep-14
No Progress
19 Viqueque
Bahalara Uain
87
427
Completed
Sep-14
20
21
22
23
24
25
26
27
Viqueque
Viqueque
Viqueque
Viqueque
Viqueque
Viqueque
Lacluta
Lacluta
Ermera
Bahalara
Bahalara
Bahalara
Bahalara
Bahalara
Bahalara
Ahic
Ahic
Aidac
Aisahe
Wetalitua
Caninuc
Cailoi/Aidak
Derok Oan
Camelitur
Cai Ua
Halimean/Wecec
30
33
142
60
80
32
37
73
161
168
699
261
429
147
193
381
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Sep-14
Sep-14
Sep-14
Sep-14
Dec-13
Feb-15
Jun-13
Jun-13
No Progress
1 WSS; 2 PS
(Point Source) No Progress
No Progress
in Cailoi
No Progress
No Progress
1 WSS
No Progress
1 WSS
No Progress
1 WSS
No Progress
1 WSS
No Progress
28
29
30
31
32
33
34
25
36
37
38
39
40
41
42
43
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Ermera
Railaco
Railaco
Railaco
Manatuto
Laclubar
Laclubar
Laclubar
Laclubar
Laclubar
Laclo
Laclo
Laclo
Laclo
Oecusse
Nitibe
Nitibe
Nitibe
Nitibe
Nitibe
Nitibe
Nitibe
Oesilo
Oesilo
Oesilo
Oesilo
Humboe
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Estado
Samalete
Samalete
Samalete
Poeana
Huitaso
Hamric Metin
Tatoli
Tajaquina
Coracao De Jesus
Indau Lulic
Saramata
Moris Mesac
Culau Haburas
Lihmo
Railacan
Sinai
Aiurlala
Eraulo
Leborema
161
36
35
95
61
120
11
80
75
33
60
41
19
102
71
77
953
175
231
457
315
652
63
464
437
192
353
320
105
670
452
483
Completed
Completed
no action
Completed
Completed
Completed
Completed
Completed
Completed
Completed
no action
Completed
no action
Completed
Completed
Completed
Funar
Funar
Funar
Funar
Orlalan
Hohorai
Hohorai
Hohorai
Hohorai
Macucurian
Bamatac
Laoudo
Fahilihun
Turilalan
Anicolaun
Sabumata
Miri Huhun
Heatu Ermera
18
52
110
52
86
52
51
66
23
393
253
407
247
845
232
247
142
868
Completed
Completed
Completed
Completed
Completed
Completed
Completed
no action
no action
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
Total Completed
Total target
Additional achieved
Uain
Uain
Uain
Uain
Uain
Uain
10 Aldeias
5 W Systems
Feb-13 1 WSS
Dec-13 1 WSS
Dec-13
Dec-13
Dec-13
Dec-13
Dec-13
Dec-13
Feb-14
Nov-14 1 WSS
Nov-14 1 PS
Nov-14 1 WSS
Water supply
Water supply
Water supply
35
40
33
71
15
69
165
54
13
73
80
0
121
205
175
330
35
245
756
254
35
294
436
0
3 602
3100
502
1 178
19 269
16400
2 869
6 690
Final coverage for CLTS: 63 Aldeias, 4,368 famlies and 23,976 people
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
Completed
1
1
1
1
1
1
1
WSS
PS
PS
PS
WSS
WSS
WSS
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
2012
2012
2012
2012
2014
2012
2012
2012
2012
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
2012
2013
2013
2012
2013
2012
2014
2013
2013
2013
2014
4 W System
2012
2014
2014
2012
2012
2013
2015
2013
2013
2013
Dec-14
1 PS
1 WSS
14 PS - well
1 WSS
1 WSS
1 WSS
1 WSS
1 WSS
1 WSS
2 WSS
1 WSS
10 Aldeias
10 W Systems
53
34
19
16
44
34
10
10
aldeia
Aldeias
Aldeias
aldeia
2011
Jul-05
2011
2011
2011
2012
2012
2012
2012
2012
2012
2012
2012
2014
2013
2013
12 W Systems
2013
2013
2013
2013
Dec-14
2013
2013
7 Aldeias
Oamna/Fatubena
Oamna/ Bona (1&2)
Oamna /Bona 3
Fatunababo
Lamasi Foho
Manan/Bitila
Bocnana
Webaha (Toapenu)
Webaha (bieli)
Hoineno
Buqui /Havana
WSS
WSS
WSS
WSS
WSS
WSS
WSS
Dec-12 1 WSS
13 Aldeias
Usi-Taco
Usti-Taco
Usti-Taco
Usi-Taco
Bene-Ufe
Bene-Ufe
Bene-Ufe
Bobometo
Bobometo
Bobometo
Usitacae
1
1
1
1
1
1
1
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
ODF
WS, 24 PS
WS
WS & 24
WSS, 3 PS
53 ODF aldeias
34 Aldeia
19 Aldeia
36,99 F/
3 Aldeia
20643 P
EMIS
District
CODE
Aldeia
EB F N3
Tulataqueo
49
Aileu
Samalete
EB F Buburnaro
41
Aileu
EB Manucasa
Aileu
EB Faturasa
105
EB F Mautoba
EB Fahisoi
EB F Aiurlalan/
Samalete 2
EB F Samalete
No of Access to Access to
School im proved im proved
Children w ater
sanitation
Hand
No. of No of
WashToilet disable
ing
cabins cabins
Facility
6
2
4
Toilet
Com plete
Hygiene
prom otion
2013
conducted
2014
conducted
96
Completed
Completed
Samalete
87
Completed
Completed
Hautoho
164
Completed
Completed
Aileu
Faculau
210
Completed
Completed
30
Aileu
Mautoba
86
Completed
Completed
102
Aileu
Bereliurai
213
Complete
Completed
494
Aileu
Tatilisame
213
Completed
Completed
44
Ermera
Aiura Lalan
116
Complete
Completed
2014
conducted
Ermera
Samalete
156
Complete
Completed
2014
conducted
Mar-15 Completed
2013
conducted
2013
conducted
2014
conducted
Mar-15 Completed
10 EB 1.2 Aitura
203
Ermera
Aitura
213
Completed
Completed
2013
conducted
11 EB F Domingos
Martins Bassar
12 EBF Cocoa
30
Ermera
Saramata
96
Completed
Completed
2014
conducted
5301 Ermera
Cocoa
130
Completed
Completed
2015
conducted
13 EBC Fatuquero
243
Ermera
Fatuquero
622
Completed
Completed
2015
conducted
14 EB F Novel da Paz
96 Hituria
70
Ermera
Hatali
138
Completed
Completed
2013
conducted
1187 Ermera
544
Completed
Completed
2013
conducted
16 EB Hatu Ermera
599
113
Completed
Completed
17 EB Rubae
139
Completed
No Plan
18 EB 08 Funar
606
Manatuto Laoudo
212
Completed
Completed
19 EBF Orlalan
601
Manatuto Torilalan
307
Completed
Completed
20 EB F Fatuquenfua
111
Completed
Completed
21 EB F Lamasi
38
Completed
Completed
22 EB F Bitila
93
Completed
Completed
23 EB 1.2 Fatubena
859
Oecusse O-Amna
306
Completed
Completed
24 EB Faefnome
228
Completed
Completed
25 EBF Heubenais
15 EB 1 Nunusakari
Nunutali/
Sosoher
Manatuto Hatu-ermera
2014
conducted
Exting
conducted
2013
conducted
2015
conducted
2011
conducted
2011
conducted
2011
conducted
2013
conducted
2015
conducted
92
Completed
Completed
2015
conducted
26 EB F Bona
Oecusse Bona
114
Completed
Completed
2013
conducted
27 EB F Cailaque
Viqueque Babo
144
Completed
Completed
2014
conducted
28 EBC Bahalara-Uain
Viqueque Welaco
517
Complete
Completed
Rehab
2014
conducted
5498
Water - 28 Latrine- 27
110
30
64
28
Information to be gathered
Sources of information
- document review
- reading s
- UNICEF CP Document
- document review
- document review
- revised logframe
- document review
- document review
- semi-structured interviews
based on pre-prepared
checklists
- document review
- document review
- semi-structured interviews
based on pre-prepared
checklists
Efficiency of implementation
Information to be gathered
Sources of information
- document review
- semi-structured interviews
based on pre-prepared
checklists
- reports
- minutes of WASH forum meetings
- other programmes
- document review
- reports
- key informants: donors providing
subsidies (e.g. CVTL), local govt,
local leaders, beneficiaries, NGOs,
UNICEF
- document review
- semi-structured interviews
- household visits
- document review
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- activities planned
- activities undertaken, % completion
- monitoring activities and corrective
measures
- activity reports
- key informants: NGOs, UNICEF,
SDFs, GMFs, govt partners, school
coordinators
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- M&E Plan
- project monitoring reports
- document review
- focus group discussions
with GMFs
- semi-structured interviews
- attendance of WASH
forum meeting
Information to be gathered
Sources of information
- evidence of implementation
- semi-structured interviews
with other key informants
- quality of products
- evidence of definition of quality
standards
- inspection of infrastructure
- analysis of documents produced
- Baseline and Endline Surveys and
KAP Studies
- key informants: NGOs, UNICEF,
GMFs, govt partners
- document review
- semi-structured interviews
- observation
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- observation
- achievement of indicators of
Purpose/Specific Objective
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
Effectiveness
- document review
- semi-structured interviews
- focus group discussions
with GMFs
- observation
Sustainability
Information to be gathered
Sources of information
- analysis of documents
- semi-structured interviews
- focus group discussions
with GMFs, children (if
possible)
- own assessment based
on experience
- sample surveys of
households at each village
visited, using transect
walks, observation of
latrines and other barriers
against disease and brief
discussions with heads of
households, based on
questionnaires
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- Endline Survey
- key informants: SDFs, GMFs, local
govt partners
- observations and interviews at
household level
- analysis of documents
- interviews with key
informants
- sample surveys of
households at each village
- evidence of existence of
entrepreneurs operating independently
Information to be gathered
Sources of information
- households
- observation
- sample surveys of
households at each village
- households
- key informants: NGOs, UNICEF,
SDFs, GMFs, govt partners, school
coordinators
- sample surveys of
households at each village
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- observation
- project reports
- key informants: NGOs, UNICEF,
govt partners
- document review
- semi-structured interviews
- document review
- document review
- project proposal
- Endline Survey and KAP study
- other project reports
- key informants: NGOs, UNICEF,
govt partners
- document review
- semi-structured interviews
Impact
Equity
31.) Was the project aligned with UNICEFs
equity agenda in addressing the needs of the
target groups (i.e. to what extent the initiative
reached different groups including the most
marginalised)?
Information to be gathered
Sources of information
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- sample surveys of
households at each village
- document review
- focus group discussions
with GMFs
- semi-structured interviews
with other key informants
- sample surveys of
households at each village
- as per questions
- tipification of locally used latrine
designs
- analysis of documents
- interviews
- observation
- documentation
- key informants: national govt
partners, other donors
- analysis of documents
- interviews
Date
Location
Time
Activity
April
15
Dili, UNICEF
am
3pm
Briefing meeting
16
Dili, UNICEF
am
4pm
17
Dili, UNICEF
am
Planning, etc
3pm
18
Dili
19
Dili
20
Dili, UNICEF
21
22
Dili
Dili, UNICEF
Notes
Planning, etc
10am
2.30pm
am
3pm
9am
23
24
Dili
Dili
3pm
UNDSS briefing
5 pm
8.30am
10.30am
Interview EU
3.30pm
Interview PLAN
8am
Interview BESIK
9am
11.15am
Interview WaterAid
2.30pm
Interview CVTL
3.30pm
Interview DGSA
25
Dili
26
Dili
27
Dili
8am
Travel to Viqueque
Viqueque
pm
28
Viqueque
8am
2pm
29
Lacluta
8am
11am
Lacluta
8am
Viqueque
9am
10.30am
30
Cannot go to Oecusse
Dili
Dili
Dili
Dili
8 am
Travel to Gleno
Ermera
10.00am
2pm
Overnight in Viqueque
Overnight in Viqueque
Overnight in Lacluta
Overnight in Gleno
Date
5
6
7
Location
Time
Activity
Ermera
9am
Meet SDF
11am
pm
8am
pm
8am
Ermera
Ermera
Notes
Overnight in Gleno
Overnight in Gleno
10am
Dili
pm
Overnight in Dili
Dili
8 am
11am
5pm
Dili
10
Dili
Rest
11
Dili
8am
12
P. Macassar
8 am
Oesilo
1pm
Nitibe
8am
11am
12pm
Overnight in P. Macassar
8am
9am
10am
9am
4pm
13
14
15
P. Macassar
Dili
Overnight in P. Macassar
Overnight in P. Macassar
16
Dili
17
Dili
Rest
18
Dili
8am
Travel to Aileu
Aileu
10am
Remexio
2pm
5.30pm
Dili
7.30am
Travel to Remexio
Laulara
8am
Remexio
10am
1pm
2.30pm
3.30pm
8.30am
10am
Made presentation
pm
Dili
8am
Travel to Liquidoe
Liquidoe
9.30am
Meet SD authorities
10am
12pm
3pm
19
20
Dili
21
Dili
22
Date
Location
Time
Activity
23
Dili
Prepare debriefing
24
Dili
Rest
25
Dili
26
Dili
27
28
By 15
June
Dili
Notes
Prepare debriefing
10.30am
pm
10am
Debrief EU
3pm
Dili
Consultant departs
Home
2.
3.
4.
5.
6.
7.
8.
Final lists of communities that benefited from water supply and sanitation interventions,
and WASH in schools, as at 20 March 2015. UNICEF.
9.
10.
11.
Expenditure Verification Report for Years 1, 2 and 3. Auditors Moore Stephens LLP.
February 2015.
12.
13.
14.
15.
16.
17.
18.
19.
Comprehensive Service Package for Primary Health Care. Ministry of Health (including
Guidelines for Domiciliary Visits; and Domiciliary Visit Register: Village Health Profile),
National Directorate of Public Health. 2015.
20.
Decree-Law No. 4/2004 of 11 February 2004 on Water Supply For Public Consumption
21.
22.
Willetts, J. (2012) A service delivery approach for rural water supply in Timor-Leste:
Institutional options and strategy. BESIK, March 2012.
23.
Dwan, P. (2014) Look Back Study: An impact evaluation of Cruz Vermelha de TimorLestes water and sanitation program. Prepared by FH Designs for Australian Red
Cross, December 2014.
24.
Position
Name
Date
UNICEF
Ramesh Bhusal
UNICEF
Toshiko Takahashi
UNICEF
Country Representative
Desiree Jongsma
UNICEF
UNICEF
WASH Officer
Rodolfo Pereira
UNICEF
Hemlal Sharma
21 May
UNICEF
Takaho Fukami
22 April
UNICEF
Consultant in Oecusse
Olga Corbafo
12-14 May
European Union
Charles Dubay
23 April
10
European Union
Programme Officer
23/04, 27/05
11
European Union
Nick Heilegger
27 May
12
Ministry of Health
Chief: DSAVS
Tomasia de Sousa
16, 21 April
13
DGAS
Programme Manager
Martinus Nahak
16, 21 April
14
DGAS
Director General
24 April
15
DGAS
Director: DNSB
Joo de Piedade
24 April
16
DGAS
Director: DNSA
Gustavo da Cruz
24 April
17
AMAR
Director
Antonio Silva
20/04, 18/05
18
AMAR
Coordenador
Ronaldo Soares
20/04, 18/05
19
HIM
Director
Gilberto
20/04, 08/05
20
HIM
Coordenador Tecnico
Serafim Pereira
20/04, 08/05
21
ETADeP
Director
Victor Carvalho
20, 24 April
22
ETADeP
Assistant Coordinator
24/04,14/05
23
ETADeP
CLTS Facilitator
Augusto Varela
14 May
24
PLAN
Country Director
Terence McCaughan
23 April
25
PLAN
John McGown
23 April
26
BESIK
Heather Moran
24 April
27
WaterAid
Country Representative
Alex Grumbley
24 April
28
CVTL
24 April
29
CVTL
WASH Advisor
Stuart Bryan
24 April
30
Indep consultant
Penny Dutton
15 May
31
Viqueque District
District Administrator
Jos da Sousa
28 April
32
Viqueque District
Director: SAS
Marito da Costa
28 April
33
Viqueque District
Ilfonso Fatima
28 April
34
Viqueque District
Director: Education
Emidio Amaral
28 April
35
Viqueque District
Director: Health
Apolinario Soares
28 April
Numerous
briefing &
follow-up
meetings and
interviews
36
Bahalara-Uain
Claudio Pinto
28 April
37
Bahalara-Uain
Evaristo Saviana
30 April
38
Bahalara-Uain
39
Lacluta Sub-district
Administrator
Rui da Costa
29 April
40
Lacluta Sub-district
SDF
Rogerio Pires
29 April
41
Lacluta Sub-district
Benedito Soares
29 April
42
Ermera District
Director: SAS
Thomas da Silva
4 May
43
Ermera District
SAS: 4 SDFs
44
Ermera District
Jari Alves
4 May
45
Ermera District
Paulo Amaral
4 May
46
Poeana
Chefe Suco
Pedro Exposito
4 May
47
Poeana
Suco official
Paulino Lemos
4 May
48
Poeana
Chefe Aldeia
Toms Martins
4 May
49
Poeana
GMF Coordinator
Jacob de Carvailho
4 May
50
Poeana
GMF Treasurer
Amelia Xavier
4 May
51
Poeana
4 GMF members
52
Huitaso
Chefe Suco
Antnio Santos
5 May
53
Huitaso
Chefe Aldeia
Fernando Soares
5 May
54
Huitaso
55
Huitaso
Abilio Salsinha
5 May
56
Huitaso
GMF Treasurer
Georginha Soares
5 May
57
Huitaso
58
Saramata
Chefe Aldeia
Fernando Soares
6 May
59
Saramata
Chefe GMF
Claudio Soares
6 May
60
Saramata
Treasurer
Teresa Martins
6 May
61
Saramata
62
Fatuquero
School Coordinator
Cornelio Nascimento
7 May
63
Leborema
Chefe Aldeia
Armando da Costa
8 May
64
Leborema
Chefe GMF
Antonio Honoratus
8 May
65
Leborema
Treasurer
Maria Aveira
8 May
66
Leborema
67
Oecusse District
Domingos Maniques
12 May
68
Oecusse District
Director: SAS
Gil de Sousa
12 May
69
Oecusse District
Director: Education
Domingos Ramos
12 May
70
Oecusse District
Gastao Poto
12 May
71
Oecusse District
SAS: SDF
Jose Ili
12 May
72
Webaha
Domingos Bobo
12 May
73
Webaha
Agustinho Nono
12 May
74
Webaha
Chefe GMF
Luis Bobo
12 May
28 April
4 May
4 May
5 May
5 May
6 May
8 May
75
Webaha
12 May
76
Baocnana
Chefe GMF
Julio Calo
13 May
77
Baocnana
Secretary GMF
Jose Aloi
13 May
78
Baocnana
Treasurer GMF
13 May
79
Baocnana
8 GMF members
80
Bona
Coordinator of school
Marcelo Tamelab
13 May
81
Bona 1&2
Chefe Aldeia
Serafin Bacs
13 May
82
Bona 1&2
Chefe Bairo
Domingos Calo
13 May
83
Bona 1&2
Chefe GMF
Marcos Tefu
13 May
84
Bona 1&2
Treasurer GMF
Anjela Calo
13 May
85
Bona 1&2
8 GMF members
13 May
86
Bona 3
13 May
87
Aileu District
Secretary of Administration
Jacinto Mendosa
18 May
88
Aileu District
Director: Education
Isac Sarmento
18 May
89
Aileu District
Director: SAS
Martinho Kamoes
18 May
90
Aileu District
Amaro Belo
18 May
91
Aileu District
SAS: SDF
Cesar de Andrade
18 May
92
Remexio SD
Administrator
Carlos Araujo
18 May
93
Tulataqueo Suco
Chefe Suco
Adolfo Mau
18 May
94
Lerolisa
Tecnico of GMF
Armando da Silva
18 May
95
Lerolisa
Tecnico of GMF
Francisco Benevides
18 May
96
Samalete 1
Chefe GMF
Manuel da Costa
19 May
97
Samalete 1
Albano Dorrego
19 May
98
Samalete 2
Chefe GMF
Martino de Andrade
19 May
99
Samalete 2
Treasurer of GMF
Santina de Jesus
19 May
100
Liquidoe SD
Coordinator of PNDS
Mouzinho Santiago
22 May
101
Liquidoe SD
Mariano de Jesus
22 May
102
Liquidoe SD
Barreto, Rangel
22 May
103
Asumata
Tecnico of GMF
Celestino da Costa
22 May
104
Asumata
Rossa da Costa
22 May
105
Asumata
7 community members
13 May
22 May
Appendix 9A:
6. Challenges in WASH
O&M of water
Appendix 9B:
capacity building
partner or contractor?
general
Appendix 9C:
joint implementation?
6. Challenges in WASH
7. Strategy of government to ensure sustainability
Sanitation / CLTS
Monitoring system? (ask to see SAS monitoring reports and SDF assessments)
in general
Appendix 9D:
Was the project implemented as planned? Is every targeted aldeia ODF? Is there water
for everyone in each targeted aldeia?
What went well? What was the biggest success of the project?
Sanitation / CLTS
Monitoring system?
in general
Appendix 9E:
general
Appendix 9F:
Water supply
Sanitation
PTA?
Reliability (ask)
7. Toilets
Ratio of boy and girl pupils per toilet cubicle/stall and whether separated
Whether designed for disabled, menstruating girls, back-up when water fails
What were they able to take home and put into practice there?
Appendix 9G:
No.
Use?
Inspection of water
infrastructure
Inspection of school
infrastructure
Interviews with
school staff
District government
partners
SDFs
National government
partners
UNICEF staff
NGO partners
Analysis of documents
Own comparative
experience
Confirmed by 2
principals interviewed
Confirmed. Needed
more generic training.
Confirmed, including
Result 1.
Confirmed by all 3.
Especially school
WASH gets little other
donor support
As per govt's Strategic Development Plan
and WASH sector policies
In line with UNICEF CPD
Insufficient to assess
whether needs
adequately addressed
Technologies appropriate.
Technologies
appropriate. No erosion
protection.
Appropriate approach to
govt CB, acknowledging
that it is a slow process
Efficiency
9
10
11
12
13
Positive assessment of
project implementation
efficiency.
Complain of delays in
processing of tranches
by UNICEF and
problems with financial
system for advances.
14
15
16
Positive assessment of
quality of products
Positive assessment of
quality of products
Positive assessment of
quality of products
Positive assessment of
performance, except in
Viqueque. However not
true that there was no
progress in sanitation
there; some aldeias
almost achieved ODF
status.
Weak NGO in
Viqueque.
Positive assessment of
performance
Positive assessment of
quality of products
Detailed explanations
given of CB provided
and efforts to correct
poor performance.
Debatable whether
strategy in Viqueque
was best option.
Coordinated with
Detailed information
national and district govt provided, indicating
to avoid geographic
effective cordination.
overlaps. Active in
WASH Forum to
improve synergies and
policy/thematic
coordination.
None of 4 districts
visited raised this as a
significant issue
No negative coments on
cost effectiveness.
No negative coments on
cost effectiveness.
Positive assessment of
effectiveness
Positive assessment of
effectiveness
Compliance also
monitored by district
SAS - positive
comments.
Compliance also
monitored by DNSA and
MinEd - no negative
comments.
Positive assessment of
UNICEF's role in
coordination of sector
Effectiveness
17
18
19
20
Only 2 interviewed,
Outcomes for govt staff
unable to provide
not clearly defined,
quantitative assessment except for indicator of
of training outcomes.
BeSI Cttees meeting.
Effectiveness of training
on CAP process
measured in terms of
project and GMF
effectiveness.
Complemented
information in reports,
particularly regarding
CB of govt officials.
Sustainability
21
22
23
24
25
26
Good sustainability at
Education Section
level of policy support,
confirms weakness of
but inadequate budget
school maintenance
support for O&M. Health system.
system for PHC is
being strengthened, but
needs more budget.
Confirm weaknesses in
govt capacity and need
for much more funding
and resources to scale
up CLTS and sustain
ODF.
As above
Confirm failure of
strategy of trying to
create businesses from
scratch and not
carefully identifying
bottlenecks first.
All communities
Limited number have moved
requested financial
up ladder; nearly all
support to improve
permanent structures were
latrines and make them first version built. Main
"permanent". Need for
constraint is finances.
targeted subsidy policy.
Improve construction
finishes (see Sheet 2).
Sustainability strategy
supported by all
Support with
motorcycles, office
equipment; training in
CLTS, CAP process.
Impact
27
Support in development
of policies, strategies,
guidelines
28
29
30
Equity
31
32
Wheelchair access at
standpipes included in
design.
Children at centre of
school hygiene
promotion activities.
Lessons learnt
33
34
Model of implementing
with local NGOs is
appropriate. More
support required to
strenghten NGOs to
enhance their
sustainability.
Substantial progress
made already in
developing subsidy
policy to be applied
once aldeias declared
ODF.
Empower GMFs to
demand service from
SAS-SDFs
Strengthen efforts to
build generic capacity in
govt (e.g. SAS and
PHC system)
Strengthen efforts to
build generic capacity in
govt (e.g. SAS and
PHC system)
1) Due to road conditions, it was not possible to visit a second system as planned. However, it has reportedly been functioning well since completion 2 years ago (according to the SDF and some residents met on the road to the village before we had to turn around due to the road conditions).
2) No refresher training in O&M was reportedly done for any of the GMFs as the contract with the NGO had ended by the time this was implemented in the project.
3) The problems with the Bahalara-Uain system seem to require a higher and more intense level of intervention than has been attempted so far.
Ermera
Ermera
Humboe
Poeana
Std gravity
piped
2013
(Feb)
145
15
Fully functional
20%
$435 collected 2013,
2014
Being a coffee
growing area, nearly
all households pay
tariff annually ($6)
after harvest (JuneAugust).
Ermera
Ermera
Estado
Huitaso
Std gravity
piped
2013
(Dec)
57
Fully functional
100%
The general quality of the infrastructure observed (reservoir, 5 standpipes) is good, however the Good turnout at FGD (suco and aldeia leadership, full GMF, reps of 2 other neighbouring GMFs, several users) but only 1 woman.
$209 collected 2014 same examples of poor finishing as for Poeana were noted.
Well informed, organised and trained GMF. Sense of ownership high. 3 of 8 members are women (only one, Treasurer, present).
Pay tariff ($6)
Same applies to other 2 GMFs represented. Have considered other uses of O&M funds collected, e.g. micro credit; Saramata has
annually
already started to lend (at 10% per month). Not enough water to allow private connections - need to control consumption. No
disabled in village. Seem well informed on sanitation situation, request help to build permanent (PF) latrines. SDF has visited. PSFs
don't pay much attention to WASH.
Not all GMFs in suco manage to collect tariffs - while most collect from the majority (some from all), 2 have not managed to collect
anything (not project GMFs).
Ermera
Ermera
Estado
Saramata
Std gravity
piped
2013
(Dec)
39
Fully functional
56%
$130 collected 2014
Pay tariff ($6)
annually
The general quality of the infrastructure observed (reservoir, 5 standpipes) is good, however
most of the same examples of poor finishing as for Poeana were noted. But no erosion
observed yet and waste water drains are well built and functioning. Some standpipes have poor
screed finish on apron, so already flaking. Indicates variable skills amongst masons (and
inadequate guidance?). Too much water for reservoir overflow to cope so some taps left open.
Not very high turnout at FGD (Chefe Aldeia, 4 of 6 GMF, 3 users). Well informed, organised and trained GMF. Sense of ownership
high. 2 of 6 members are women (only one, Treasurer, present). Have started using O&M fund for micro credit. Do not allow private
connections, only hoses from taps - need to control consumption. No disabled in village. Will only assist school with maintenance if
they can pay.
Seem well informed on sanitation situation, request help to build permanent (PF) latrines (only 40% of families have PF latrines).
SDF has visited. PSFs don't pay much attention to WASH.
Ermera
Railaco
Samalete
Leborema
Std gravity
piped
2014
(Oct)
67
Fully functional
10%
$35 collected this
year, rest will be after
harvest
The general quality of the infrastructure observed (reservoir, 5 standpipes) is good, however
most of the same examples of poor finishing as for Poeana were noted. Erosion at reservoir has
already exposed pipe. Poor quality taps used, all already broken, not replaced yet.
Only 3 standpipes installed for 67 households, so heavier use than most systems (hence taps
broken so soon?). GMF insisted on few because amount of water available is limited and needs
to be carefully controlled; fewer taps make this easier. Maximum distances are still reasonable
as housing density is high.
Reasonable turnout at FGD considering it was market day (Chefe Aldeia, 5 of 6 GMF, 4 users). Well informed, organised and trained
GMF. Sense of ownership high. 3 of 6 members are women (all present: Treasurer, operator, environmental health promoter). Not
enough water to allow private connections - need to control consumption. No disabled in village.
Seem well informed on sanitation situation, request help to build permanent (PF) latrines (only 2 families have). SDF has visited.
PSF is the Treasurer of GMF, promotes sanitation.
Take responsibility for maintenance of water to school and plumbing to toilets. No APP (PTA) members on GMF.
The general quality of the infrastructure observed (reservoir, 3 standpipes) is good, however
there are a few examples of poor finishing and attention to the details that are important for
facilitating maintenance and ensuring greater sustainability, notably:
- no erosion prevention works at reservoir or at standpipes, with resulting evidence of erosion
already
- no gravel surround for standpipes or reservoir to facilitate hygienic conditions (however, the
extensive use of hosepipes from standpipes to houses means that there is not much movement
of people around the standpipes)
- in some cases wheelchair ramps at standpipes are nor positioned to be usable
- most taps were dripping
The source is an open stream which becomes dirty after rain, so a filter has been included in
the design. However, to keep costs down, a design which uses palm fibre as one of its filter
materials has been used, which requires up to weekly cleaning and/or replacement. This may
be a weakness as such frequent and labour intensive maintenance may not be sustainable. It
also requires paying some compensation (at least a meal) to the operators and community
members who do the work.
All systems fully functional with effective GMFs. Good prospects for sustainability. Suggests high quality of implementation process and supervision. High quality products and outcomes.
Oecusse Oesilo
Bobometo
Webaha
(Toapenu)
Std gravity
piped
2013
102
10
Fully functional
100%
$600 collected 2013
and 2014
Pay tariff ($3)
annually
Oecusse Nitibe
Usi-Taco
Bona 1&2
Std gravity
piped
2013
73
11
Fully functional,
some water
quality and
quantity
problems
100%
$270 collected
Pay tariff (25c)
monthly
Oecusse Nitibe
Usi-Taco
Bona 3
Hand dug
wells
2013
175
Oecusse Nitibe
Bene-Ufe
Citrana,
Bairro
Baocnana
Std gravity
piped
2014
(Nov)
115
32
Fully functional
56%
$123 collected
Pay tariff (50c)
monthly, started Jan
2015
?
The quality of the infrastructure observed (reservoir, 1 standpipe) is good. No erosion observed.
Pay tariff (50c)
No fences remain but no sign of animals around tapstands. Tapstands have 2 wastewater
monthly, started Jan drains, working well, no wastewater nearby. Hosepipes from taps to some houses. No major
2013
dripping at taps.
Wheelchair ramps at standpipes, but GMF doesn't know what they are for.
High quality of construction. Also good attention to detail. Gravel surrounds and fences at
reservoir and tapstands, waste water chanelled to soakaways where fruit trees or sugar cane
are planted. One example of pipe exposed due to erosion near reservoir. No major dripping at
taps. Some dry because water is rationed with limited opening hours.
No wheelchair ramps at standpipes.
Due to distances and lack of vehicle accessibility, the reservoir was not inspected, only 2
tapstands. High quality of construction. Also good attention to detail. Gravel surrounds and
fences at tapstands, but wastewater not chanelled away. However, the extensive use of
hosepipes from standpipes to houses means that there is not much movement of people
around the standpipes. No major dripping at taps. No wheelchair ramps at standpipes.
Due to no other source being available, one with less water than would be ideal, and high
dissolved salts content, has been used. However, the former constraint is only felt at the end of
the dry season and the latter is within safety limits while the taste of the water is acceptable.
However it does cause serious maintenance problems as deposits form in the pipes, especially
in the flat areas, such as at the school. No affordable solution appears to be available.
14 wells Fully functional No maintenance fund One well inspected. Good quality construction. GMF reports all wells always have enough
(25c collected per water (3-15 families per well). Pulley not lubricated and has started wearing. Otherwise well
family for meals when maintained.
cleaning is done)
Good turnout at FGD considering it was market day (Chefe Aldeia, Chefe Bairro, 4 of 6 GMF, 10 users). Well informed, organised
and trained GMF. Sense of ownership high. 2 of 6 members are women but not office bearers (both away at market). Not enough
water to allow private connections - need to control consumption. Have to ration through limited opening hours. No disabled. Pay
operators "incentives" ($5/month) when O&M more intensive, e.g. annual cleaning around spring, etc.
Fully involved in sanitation promotion - first case encountered of sanitation seemingly genuinely community led. Organise
community solidarity to help poorest build latrines. Request help to build permanent (PF) latrines (only 20 families have). SDF has
not visited (only 2 in district currently). Chefe Aldeia is PSF, promotes sanitation.
In spite of the visit being on market day, the whole GMF and local leadership were present (Chefe Aldeia, Chefe Bairro, 6 GMF + 3
tap stewards, 5 users). There are 3 women on the GMF, the Treasurer and two sanitation promoters. Well informed, organised and
trained GMF. Sense of ownership high. Excellent level of tariff collection. Are trained for all operations and have made necessary
repairs and preventive maintenance such as replacing palm fibre in filter at source. Operators get paid when this work and repairs are
carried out. Take responsbility for water at school and maintenance of plumbing there.
Fully involved in sanitation promotion - another case of sanitation seemingly genuinely community led. Apparently all homes have at
least temporary latrines, HWF and dish racks. Request help to build permanent (PF) latrines.
GMF is responsible for all 14 wells. 3 of the 5 members were present, plus Chefe Bairro and 7 others. Treasurer (only woman) not
present. Very satisfied with wells. Have considered installing a pump with electric motor now that electricity is being brought to the
village.
Report that all households have latrines. Need financial help to upgrade to permanent (PF) structures.
This system was not on the original project list and was implemented when additional (non-EU) In spite of the visit coinciding with a major religious ceremony in the village that day, 8 of the 14 GMF members (all the office
funding was secured. Thus the EU contribution was only used to co-fund the implementation
bearers) and local leadership were present (Chefe Bairro, 8 GMF, 5 users). Consider large GMF necessary due to size and
costs (including training), not materials. However, this was only known after the visit and the
complexity of system. Have tapstand leaders for each for cleaning and tariff collection. There are 3 women on the GMF - Treasurer,
system is thus included in this report. The sign on the reservoir includes the EU flag.
one of the operators and the sanitation promoter. Well informed, organised and trained GMF. Sense of ownership high. Are trained
It is a relatively large system supplying a semi-urban settlement that includes numerous public for all operations and have made necessary repairs and preventive maintenance such as replacing palm fibre in filter at source.
buildings and shops. The design effectively and creatively incorporated some components (e.g. Operators get paid when this work and repairs are carried out. The GMF takes responsbility for water for government and other
reservoirs) of previous systems that had ceased to function.
public buildings, such as schools (where they do maintenance of plumbing) and the clinic, as well as churches and commercial
The quality of the infrastructure observed (reservoir, tank, 2 standpipes) is good. Also good
establishments. Payments of tariffs by government and civil servants is poor. GMF is supported by SAS to reinforce their authority.
attention to detail. Gravel surrounds and fences at tapstands, but wastewater not chanelled
Sanitation promotion appears to be genuinely community led and the coordinator of the sanitation promoters (woman) also gets an
away. However, the extensive use of hosepipes from standpipes to houses means that there is incentive to encourage families to move up the sanitation ladder. The local PSF is a member of the GMF.
not much movement of people around the standpipes. No major dripping at taps. No wheelchair The SDF has not visited since the inauguration 5 months ago.
ramps at standpipes.
Very impressive results achieved, both in water and sanitation. Sanitation seems genuinely community led, due to the quality of CLTS facilitation. Due to lack of capacity in SAS, more training of the GMF was done by the NGO in this district than in others, which may have been a positive factor, although long term
sustainability (once NGO support is withdrawn) may suffer due to this lack of capacity.
10
Aileu
Remexio
Asumau
Lerolisa
Std gravity
piped
2012
102
Fully functional
11
Aileu
Remexio
Tulataqueo
2012
(Jan)
56
Fully functional
50%
$400 in fund
Pay tariff (50c)
monthly, started 2013
The quality of the infrastructure observed (reservoir, 3 standpipes) is good. No erosion observed
at tank or standpipes, but main line down from tank has been exposed in several places due to
erosion (no erosion prevention works were done). No fences remain but no sign of animals
around tapstands. Tapstands have 2 wastewater drains, working well, no wastewater nearby.
Numerous hosepipes from taps to houses. No major dripping at taps.
Wheelchair ramps at standpipes.
12
Aileu
Remexio
Tulataqueo
2012
27
Fully functional
25%
+/-$60 in fund
Pay tariff (50c)
monthly, started 2013
The quality of the infrastructure observed (reservoir, 3 standpipes) is good. No erosion observed.
No fences remain but no sign of animals around tapstands. Tapstands have 2 wastewater
drains, working well, no wastewater nearby. One of them has plaster screed on apron partially
peeled off. Hosepipes from taps to some houses. No major dripping at taps. Pipe accidentally
cut by one homeowner when laying cable last year has not been properly repaired yet.
Wheelchair ramps at standpipes, one reportedly used.
13
Aileu
Liquidoe
Acubilitoho
Asumata
2014
(July)
40
Fully functional
zero
Monthly tariff (25c)
not collected yet
Good turnout at FGD (suco and aldeia leadership, full GMF, some users). Well informed, organised and trained GMF. Sense of
ownership high. 2 of 6 members are women (Treasurer and one operator).
Have considered other uses of O&M funds collected, e.g. micro credit. Also have rules for private connections and 3 have been
made. Make provision to help disabled.
Seem well informed on sanitation situation, request help to build permanent (PF) latrines. SDF has visited. PSFs don't pay much
attention to WASH.
GMF takes responsibility for maintenance of water supply and plumbing at school toilet facilities.
Std gravity
piped
Due to major religious festival in Remexio during week of visit, it was difficult to contact GMF members. Only two (operators) were
interviewed, hence no financial data obtained. Report 7 members, 3 of them women (Treasurer, health promoter, one of 3 operators).
Few repairs needed over nearly 3 years of operation - only taps at one tapstand. All other taps still originals, not even washers
changed. So tariff collection is not seen as very important. No maintenance needed at source as it is a protected spring.
Health promoter only focuses on tapstand hygiene.
Due to major religious festival in Remexio during week of visit, it was difficult to contact GMF members. Only two (Coord +
Treasurer) were interviewed. GMF has 12 members, 4 women (treasurer, 3 tap stewards). Meets every 3 months, but not all attend
regularly, some are tiring (same since 2012), no elections held as aren't confident anyone will want to join GMF due to unpaid nature
of work. Received refresher training 2014, and toolkit. No problems with system, just taps.
GMF does maintenance at schools, which pay monthly tariff (also have blue recording book).
Health promoter is also PSF, does latrine promotion work.
No reporting system to SAS. No visit from SDF since training last year.
Due to major religious festival in Remexio during week of visit, it was difficult to contact GMF members. Only two (Coord +
Assistant) were interviewed, hence only sketchy financial data obtained. Chefe Aldeia keeps the money. GMF has 10 members, 2
women (treasurer, health promoter), same team since 2012. Received refresher training 2014, and toolkit. No problems with system,
just taps. Have to clean filter every 3 months. One person in aldeia uses wheelchair and collects water.
GMF does maintenance at schools, which pay monthly tariff (also have blue recording book).
Health promoter is also PSF, does latrine promotion work.
No reporting system to SAS. No visit from SDF since training last year.
The quality of the infrastructure observed (reservoir, 3 standpipes) is variable; 1 standpipe is
GMF has 8 members, 3 of them women (treasurer, health promoter, operator). Have been functioning for one year. Received
incorrectly sloped (hence waste water doesn't drain as intended) and has peeling screed finish. refresher training 2014, and toolkit. Not functioning adequately and have not started collecting tariff yet. No maintenance involving
Erosion observed along main line from tank and pipes exposed. Standpipe fences in disrepair
expenditure has been required yet. Some households expressed dissatisfaction with the Coordinator. The only poorly maintained
but no sign of animals around tapstands. No major dripping at taps.
tapstand is that under his responsibility. He, his deputy and the treasurer were away on private business. Another 3 members were
Wheelchair ramps at standpipes.
at an official meeting elsewhere. Only 2 members were present in the FGD, with 8 community members. No disabled in aldeia.
No reporting system to SAS. No visit from SDF since training last year (changed jobs shortly after and has not been replaced).
Generally good quality construction and maintenance, with mostly well functioning GMFs. Exception is Asumata which requires an intervention from SAS.
District
Sub-district Suco
Aldeia
Apparent
income
level *
Type of latrine*
In use?
When
built
Influenced by
project?
Trigger to build
Barrier(s)
Approximate
First or
Cost (if known) to moving
upgrade
up ladder
USD
upgrade Subsidy +/-$100,
total cost not
upgrade
upgrade known (+/-$400?)
approx $500
upgrade
without subsidy.
n/a
n/a
n/a
n/a
General
hygiene
Water
tariff
paid?
yes
yes
yes
yes
good
good
good
good
n/a
n/a
n/a
n/a
n/a
n/a
n/a
n/a
proximity, grows
vegetables
proximity
proximity (from old
system - not
connected to new)
yes
no
Project messages
recalled
HW
facility
nearby?
HWWS?
none
none
none
none
bathroom
bathroom
bathroom
bathroom
say
say
say
say
Bahalara-Uain
Viqueque Viqueque
1
Bahalara-Uain
Viqueque Viqueque
2
Bahalara-Uain
Viqueque Viqueque
3
Bahalara-Uain
Viqueque Viqueque
4
Summary of Welaco aldeia
Subsidy
no
2009
yes
PF-P
Above ave.
Welaco
Subsidy
no
2009
yes
PF-P
Above ave.
Welaco
Subsidy
no
2009
yes
PF-P
Above ave.
Welaco
Neighbours
no
2015
yes
PF-P inside
Average
Welaco
No OD observed, reported as practiced only away from homes in fields. All subsidised latrines in use. One built
Only one aldeia visited. From interviews, observations, comments from stakeholders, seems impact of subsidies is not necessarily insurmountable barrier to CLTS.
Ermera
Ermera
Humboe
Poeana
Average
PF-I
yes
2012
no
n/a
none
bathroom
say yes
good
6
7
Ermera
Ermera
Ermera
Ermera
Humboe
Humboe
Poeana
Poeana
Below ave.
Average
PF-B
only pit dug
yes
no, OD
2013
n/a
yes
yes
cost
cost
most
unwilling to talk
bathroom
no
say yes
no
good
good
$700
replace
Replace previous one that
collapsed
$50
first
project messages
n/a
first
Project messages. Little space
to build so wants to save for PF
to avoid smell near house.
Limited time available. One case of OD but fully aware of need for latrine and has started. Variable reported impact of hygiene messages.
Ermera
Ermera
Estado
Huitaso
Average
PF-B
yes
2014
yes
first
$100
cost
say yes
good
Ermera
Ermera
Estado
Huitaso
Average
PF-I
yes
2014
no
upgrade
$400
n/a
none
bathroom
say yes
good
10
Ermera
Ermera
Estado
Huitaso
Below ave.
PF-B
yes
2014
yes
upgrade
$150
cost
none
bathroom
say yes
poor
11
Ermera
Ermera
Estado
Huitaso
yes,
proximity so use more,
grow vegetables, wash pays
annually
clothes at home
yes,
proximity, quality
annual
proximity, quality
yes,
annual
yes,
annual
13
Ermera
Ermera
Estado
proximity
poor
yes
bathroom
none
cost
local materials
first
unspecified project messages
yes
2013
yes
basic dry pit, no
roof, hole cover
No OD observed or reported. Proximity of water that makes PF viable is main motivation. Clearly this is what most consider the minimum acceptable standard, refered to as "permanent"; dry pit considered "temporary".
proximity, quality
good
bathroom say yes
all (leader)
n/a
upgrade $300 + same for
proximity of water to upgrade
no
2014
yes
PF-P
Saramata Above ave.
bathroom
from first (2010), previously OD
proximity, quality
good
bathroom say yes
all (leader)
n/a
$600
upgrade
always had
no
2010
yes
PF-P
high
Saramata
14
Ermera
Ermera
Estado
Saramata
Above ave.
2015
no
15
Ermera
Ermera
Estado
Saramata
Below ave.
2014
yes
16
Ermera
Ermera
Estado
Saramata
Average
yes
basic dry pit, no
roof, no hole cover
basic dry pit, no yes(+ by
roof, no hole cover 2 other
families)
yes
Below ave. None, use no.15's
19
Ermera
Railaco
Samalete
20
Ermera
Railaco
Samalete
Leborema
Poor
21
Ermera
Railaco
Samalete
Leborema
Poor
Usi-Taco
29
Oecusse Nitibe
Usi-Taco
30
Oecusse Nitibe
Usi-Taco
no
say yes
fair
no
say yes
poor (flies,
dirty yard)
average
yes
yes
all (leader)
$300
upgrade
Replace original dry unit built
no
2014
yes
PF-P
Average
Webaha
average
yes
yes
all (leader)
n/a
$200
upgrade
with subsidy in 2006, with PF
no
2014
yes
PF-P
Above ave.
Webaha
poor
say yes
no
most
cost
local materials
first
Dignity, safety at night
yes
2012
yes
PF-B
Average
Webaha
poor
say yes
no
all (leader)
n/a
$200
upgrade
Replace original dry unit
no
2015
yes
PF-I
Below ave.
Webaha
average
say yes
no
did not participate
cost
not known
upgrade
As for 22 and 23
no
2003
no, OD
PF-B (full)
Very poor
Webaha
Almost all latrines seen (and all inspected) are PF. Only case of OD seems to be temporary until funds available to replace old latrine. Most already used latrines before project. Very good recollection of project
good
yes (tippy- not soap
latrines, HWWS
cost
local materials
first
no (prior) Right thing to do
2012
yes
PF-B
Bona 1&2 Below ave.
tap)
average
say yes
no, at
latrines, HWWS, OD
cost
local materials
first
Right thing to do - message
no
2009
yes
PF-B offset
Bona 1&2 Below ave.
kitchen
is disgusting
from Indonesian time.
poor
not soap
no
latrines, HWWS,
cost
local materials
first
Project messages
yes
2013
no, OD
PF-B (collapsed)
Poor
Bona 1&2
dish rack
good
yes (tippy- apparently
latrines, HWWS,
cost
local materials
first
Project messages (including
yes
2013
yes
basic dry pit, well
Poor
Bona 1&2
tap)
flies cause disease
"OD is illegal")
built (+roof) & maintained, hole cover
Also mostly PF and many already had before project. Good recollection of messages.
Best transmission of messages of all districts. Only OD observed seems temporary. Good GMF role in leading sanitation promotion.
Acumau
Remexio
Aileu
Summary of Lerolisa aldeia
Tulataqueo
Remexio
31 Aileu
Tulataqueo
Remexio
32 Aileu
Lerolisa
Due to schedule slippage all day long there was no time to visit homes (departure from the village was at 17:45).
?
?
?
upgrade
No-one at home
?
2014
yes
PF-P
Samalete 2 below ave.
with water close,
cost, new
local materials
first
Right thing to do, can't send
no
2006
use 31's
basic dry pit,
poor
Samalete 2
must build
house first
visitors to OD.
collapsed 2013
?
?
?
?
No-one at home
?
?
yes
basic dry pit, no
Samalete 2 average
roof, hole cover,
needs repairs
none
cost
local materials
Replace first one built 2012, no replaceno
2014
yes
Samalete 2 below ave. basic dry pit, no
ment
reason given
roof, no hole cover,
in good condition
none
cost
local materials
first
no reason given
no
2009
yes
basic dry pit, no
poor
Samalete 2
roof, hole cover,
needs repairs
?
?
n/a
n/a
No-one at home
no
n/a
n/a
none
Samalete 2 average
OD?
Due to major event in Remexio, most houses had no-one at home. Only 1 out of 6 possibly OD. Mostly basic dry pit collapsed or in need of repairs. Relatively poor aldeia.
?
?
?
?
No-one at home
?
?
yes
PF-B
Samalete 1 above ave.
33
Aileu
Remexio
Tulataqueo
34
Aileu
Remexio
Tulataqueo
35
Aileu
Remexio
Tulataqueo
36
Aileu
Remexio
Tulataqueo
average
38
Aileu
Remexio
Tulataqueo
Samalete 1
39
Aileu
Remexio
Tulataqueo
41
Aileu
Liquidoe
Acubilitoho
42
Aileu
Liquidoe
Acubilitoho
43
Aileu
Liquidoe
Acubilitoho
PF now viable,
proximity, wash coffee
proximity
no (too
poor)
some
Samalete
Oecusse Nitibe
none
cost
proximity
say yes poor (flies,
no
none (did not attend
cost
local materials
first
so as not to contaminate fields
no
2004
yes
basic dry pit
dirty yard)
meetings)
(doubtful)
poor state, flies
no queues
say yes poor (flies,
no
none
cost
local materials
first
aldeia now more populated,
yes
2012
yes
basic dry pit
dirty yard)
need to keep clean
poor state, flies
Generally much poorer aldeia. Less PF latrines. Most dry pit latrines in need of repairs. Most already used latrines before project. Poor state of cleanliness in general. Little apparent impact of hygiene/general sanitation messages.
CLTS has been effective. Little reversion to OD. But also little maintenance of "temporary" latrines. Less investment in PF than in other districts.
Railaco
28
cost
local materials
yes,
annual
Ermera
Bobometo
22 Oecusse Oesilo
Bobometo
23 Oecusse Oesilo
Bobometo
24 Oecusse Oesilo
Bobometo
25 Oecusse Oesilo
Bobometo
26 Oecusse Oesilo
Summary of Webaha aldeia
Usi-Taco
27 Oecusse Nitibe
local materials
first
yes,
annual
yes,
annual
yes,
annual
yes,
annual
proximity
poor
say yes
no
none (did not attend
cost
n/a
n/a
n/a
no
n/a
meetings)
No OD observed or reported, although 2 familes (one interviewed) use neighbour's latrine. Most already used latrines before project. Generally cleanliness not up to standard. One of two dry pits in poor condition and not repaired.
proximity
good
yes
yes
all (leader)
n/a
$500
replaced previous PF built 2006 replace
no
2012
yes
PF-P
Average
Leborema
fired local bricks
say yes poor (flies, proximity, no queues
no
OD is bad
cost
local materials
first
OD is bad
yes
2014
yes
Leborema Below ave. basic dry pit, no
dirty yard)
roof, no cover
no closer than before
fair
say yes
yes
mentioned all (from
cost
upgrade Can't recall, +/proximity of water to upgrade
no
2009
yes (+
PF-I
Leborema Below ave.
but more reliable
different project)
$150?
from first, previously OD
neighbour)
18
replace
n/a
PF-I, apparently
good condition
PF-I
no, use
37's
yes
No-one at home
n/a
No-one at home
n/a
proximity
proximity
proximity
n/a
proximity
hygiene messages.
Ease of use, still use
well in yard.
Ease of use, still use
well in yard.
proximity
proximity, cooler than
well water
yes,
annual
yes,
annual
most,
annual
paid up
paid up
paid up
n/a
paid up
paid up
paid up
partially
paid up
paid up
average
?
proximity, can bathe
children more daily
?
?
paid to
March
?
average
proximity
paid to
Feb
say yes
below
average
proximity
only $6
so far
no
average
Standpipe
nearby
Standpipe
nearby
bathroom
average
average
good
?
say yes
good
average
no
no
say yes
no
bathroom
bathroom
Due to major event in Remexio, most houses had no-one at home. No OD observed. All PF. Unclear why one (PF-I) is not being used. GMF says because too far from water, although standpipe is only 50m away.
no
proximity, quality,
good
bucket in say yes
OD is disgusting,
n/a
upgrade
yes
2014
yes
below ave.
Asumata
quantity
latrine
HWWS, childrens'
subsidy (cement,
faeces are bad
PF-I
zinc sheets,
no
proximity
good
OD is bad, HWWS, bucket in say yes
n/a
upgrade
yes
2014
yes
below ave. (some cement on
Asumata
insufficient sand)
latrine
bury refuse, security
(tho' pit
floor, bamboo
+ local materials
project messages, subsidy
of latrines for women
walls, zinc roof, collapsed)
(including bought
offset unlined pit
no
proximity (hose)
good
hose from say yes
can't remember
n/a
upgrade
yes
2014
yes
average
Asumata
sand) + own
with cement cover)
standpipe
labour
no
proximity, quantity
good
bucket in say yes
can't remember
n/a
upgrade
yes
2014
yes
poor
Asumata
latrine
Most households benefited from a materials subsidy and have built intermediate PF latrines. The subsidy was from SAS as part of a pilot project to support the most vulnerable. The project started in 2013 with an NGO promoting latrines
without subsidies, and most households built basic dry pit latrines. Then the materails arrived, intended only for the very poor ("vulnerable"), but due to the lack of clear criteria, the community demanded equal treatment for all and it was left to
the Chefe Suco to manage distribution, so in this village almost eveyone received the subsidy. However, very few actually did anyhting about improving their latrines until the EU-UNICEF project stated CLTS (May 2014) and then all those who
had received the subsidy built improved pour flush latrines.
No OD observed (2 cases of using neighbour's latrine). Interesting unintentional combination of subsidy and CLTS in Asumata. Clear link between % of permanent latrines and wealth of village.
Classifications
Refers to perceived overall standard inferred from home and outbuildings observed in all aldeias visited. Higher than most of Sub-Saharan Africa, comparable with poorer
Central American countries.
Type of latrine
Basic = Simple pit latrine made entirely of local materials, unlined pit, no cement, often without roof.
PF-B = Basic pour flush directly over partially lined pit with concrete cover, ceramic or plastic pan, floor at least partially cemented, no tank, superstructure entirely of
local materials, with roof. Usually not used as bathroom.
PF-I = Intermediate pour flush with offset, lined or unlined pit with concrete cover, ceramic or plastic pan, floor at least partially cemented, concrete tank,
superstructure mostly of local materials, zinc roof. Usually also used as bathroom.
PF-P = Permanent pour flush. As with the intermediate version, but with fully concrete floor and concrete block or brick walls. Nearly all also used as bathroom.
No. District
1
School
Name
BahalaraUain
Intervention
Rehabilitation of 2 (non-standard)
toilet blocks and connection to
new water system by NGO.
Training by NGO.
Pupils Water
No.
Observations
per availstalls
shift able?
2015
517
no 5 (1D)* School closed by time of visit so unable to inspect interior or interview anyone. No water as system
is down. Figures for no. of pupils and stalls different from reported. Ratio of pupils per stall far
exceeds standard. See below for comments that apply to all units built.
Completed
Ermera
Ermera
Humboe
Nunutali
2015
544
no
3** Ermera
Ermera
Estado
Culau
Haburas
Estado
New block, connection to new
(EMIS 139) water system by NGO.
Training by NGO.
2014
yes
Saramata
Domingos
2014
96
yes
1 SB School closed at time of visit (teacher training in District) so unable to inspect interior or interview
anyone. Tank overflowing (stop valve disfunctional and no float valve), access to toilets impossible
without stepping in mud. Waste water from HWF des not drain to spetic tank - blocked?
See below for comments that apply to all units built.
2015
340
no
2015
130
yes
2 SB School partially functioning for teacher training, Coordenador interviewed. No water (existing system
(1D) disfunctional), HIM to install new system this year. Training did not include any on maintenance of
facilities or establishing a maintenance system with responsibilities and links to GMF or SAS.
See below for comments that apply to all units built.
1 SB School closed at time of visit (teacher training in District) so unable to inspect interior or interview
anyone. List says project made connection. Inlet to tank goes through overflow pipe, overflow runs
down riser pipe (which is loose and susceptible to damage), forming pools and muddy area. Access
hole of tank on lid is for rainwater pipe, has no lid.
See below for comments that apply to all units built.
anyone. Not on UNICEF list. Sound of leaking taps inside (probably preventing tank from overflowing
as water was flowing into it, but shortening life of septic tank?). No vent pipe for septic tank.
See below for comments that apply to all units built.
Ermera
Ermera
Estado
Ermera
Railaco
Fatuquero Fatuquero
Ermera
Railaco
Railako
Leten
Ermera
Railaco
Samalete Leborema
Samalete
2014
156
yes
2 SB School closed at time of visit (teacher training in District) so unable to inspect interior or interview
(1D) anyone. List says 1 block. Not funded by EU? Dripping taps, not replaced yet by UNICEF.
See below for comments that apply to all units built.
Oecusse Nitibe
Usi-Taco
Bona
2013
114
yes
1 SB Coordinator interviewed, children too young. Described training provided to teachers and promotional
activities. Tippy-taps apparently in use still in front of school. Good relationship established with
GMF for maintenance. Children trained on HWWS before school lunch, but were observed later to
not wash hands before having their lunch, although the HWF was functional.
Taps at HWF leaking, one not functional, no funds to replace, GMF requires payment. Taps not
replaced by UNICEF. Stop valve to tank broken, no float valve.
See below for comments that apply to all units built.
Aileu
2014
87
yes
5 (1D) School closed at time of visit (religious holiday in SD) so unable to inspect interior or interview
anyone. Blocks in good condition, no taps dripping, tank inlet valve (ball type) functional, being used
to control water to tank. Severe erosion around building from rainwater.
See below for comments that apply to all units built.
2013
96
yes
2 SB School closed at time of visit (religious holiday in SD) so unable to inspect interior or interview
anyone. In good condition, taps not dripping (not originals, but not installed by UNICEF).
See below for comments that apply to all units built.
10 Aileu
Cocoa
Bona
1 SB School closed at time of visit (teacher training in District) so unable to inspect interior or interview
anyone. Report says 2 blocks, completed 2013. Ratio of pupils per stall far exceeds standard (even
with 2 old toilets included). Water not connected from tank to toilets by contractor yet (contract with
NGO had ended by the time construction was completed). Not functioning yet. Some exposed
polypipe. No signage. No vent pipe for septic tank.
See below
for comments
that (teacher
apply to training
all unitsinbuilt.
1 SB School
closed
at time of visit
District) so unable to inspect interior or interview
** N.B. School no. 3 was not funded by this project but was visited anyway as it was
beside the road and the same design and methodology was used.
* SB = Standard block = 3 stalls, one each for boys and girls, and one for disabled/wheelchair
access, plus hand washing facility. D = stall for wheelchair access.
General observations and recommendations valid for all toilet blocks inspected:
1.) Good design - functional, rugged, good quality of materials. Exceptions:
a) no facilities for menstruating girls;
b) concrete or even plastic tanks instead of galvanised ones would be longer lasting.
2.) Good quality of construction.
3.) All but 1 block had signage (M, F, disabled).
4.) Lack of attention to details that make maintenance easier and less costly:
a.) No erosion works (e.g. simple diversion channels/humps) to prevent erosion from rainwater around building. Erosion
was observed in a number of cases.
b.) Covers for some inspection boxes too light, not countersunk, susceptible to vandalism.
c.) Not all taps had been replaced with good quality ones as per UNICEF programme.
d.) Overflow from tank not piped away to avoid pools of water around tank. Many tanks observed were overflowing.
e.) Water into tank without ball valve to close off, and stop valve at waist height and vulnerable to damage, instead of being under
ground with inspection box and lockable lid.
First name
Martin
Middle name
Giles
WRITE
Not
Easily
SPEAK
Not
Easily
Not
Easily
OTHER LANGUAGES
Easily
Easily
Easily
Portuguese
X
X
X
Spanish
X
X
X
24. EDUCATIONAL. Give full details - N.B. Please give exact titles of degrees in original language.
A. UNIVERSITY OR EQUIVALENT
Please do not translate or equate to other degrees.
ATTENDED FROM/TO
DEGREES and ACADEMIC
NAME, PLACE AND COUNTRY
Mo./Year
Mo./Year
DISTINCTIONS OBTAINED
University of Stellenbosch, South Africa
1972
1973
First 2 years of B.Sc.Eng (Agric)
University of Natal, South Africa
University of Natal, South Africa
1974
1977
1976
1978
6. Sex
Male
UNDERSTAND
Not
Easily
Easily
X
X
25. LIST PROFESSIONAL SOCIETIES AND ACTIVITIES IN CIVIC, PUBLIC OR INTERNATIONAL AFFAIRS
South African Institute of Agricultural Engineers
Water Institute of Southern Africa
26. LIST ANY SIGNIFICANT PUBLICATIONS YOU HAVE WRITTEN (do not attach)
Rall, M. & de Jager, J, 1997. Greater Expectations: Technology choices for community managed water supply in the new South Africa. Water
Lines, Vol 15, No 3, Jan 1997.
Rall, M, 1997. Technical choices and operation and maintenance on the Mvula Trust groundwater supply projects. Borehole Water Journal,
Vol 35, 1997.
Rall, M. 2000. "Towards More Successful Rural Water Supply Schemes - The Mvula Trusts Experience Guides Policy Changes". Engineering
News, April 2000.
Rall, M. 2005. Mvula Trust en Sudafrica: un colaborador del gobierno independiente. Medio Ambiente y Urbanizacion, Ano 21, Nov 2005:
Removiendo Bareras: Agua y Saneamiento Para Todos (IIED-AL).
Rall, M. 2002. The Mvula Trust in South Africa: An Independent Partner to the Government. Blue Gold Series, Field Note 14, Water &
Sanitation Program (WSP-AF), World Bank.
Rall, M. 2008. Manual de Formao dos Grupos de gua e Saneamento (GAS). Direco Nacional de guas, Angola.
Rall, M. 2008. Manual de Formao de Mobilizadores Sociais. Direco Nacional de guas, Angola.
27
.
EMPLOYMENT RECORD:
A. PRESENT POST (LAST POST, IF NOT PRESENTLY IN EMPLOYMENT)
SALARIES PER ANNUM
STARTING
FINAL
TO
FROM
MONTH/YEAR
MONTH/YEAR
Feb 2007
Current
NAME OF EMPLOYER:
See below for each assignment
ADDRESS OF EMPLOYER:
See below for each assignment
N/a
Independent consultant
N/a
TYPE OF BUSINESS:
N/a
Date
Location
October
2014
Mozambique
WaterAid UK
Papa Diouf
PapaDiouf@wateraid.org
Team
Leader
Feb &
Jun
2014
Home
based
EU Mozambique
P. Calistri. piergiorgio.
calistri@eeas.europa.eu
Consultant
April
2014
Angola
Consultant
Trainer
Mar
2007 to
date
Latin
America,
Africa,
Asia
Senior
International
Expert
2013
2012
2011:
2010:
2009
2007
2006
Oct/Nov
2011,
Jul/Aug
2012
Bolivia,
Guatemala
worldwide
BPD (Building
Partnerships for
Development in Watsan)
Tracey Keatman
traceyk@bpdws.org
Consultant
Feb
2012
Mozambique
ABIODES: N. Mahalambe
nmahalambe@gmail.com
Consultant
MarchMay
2010
Angola
UNDP / SIWI
UNDP: Amaya Olivares
amaya.olivares@undp.org
Consultant
Jul-Aug
2011
Angola
Consultant
Trainer
Nigeria
Consultant
Aug
2010,
Mar
2011
Peru
Consultant
Oct
2009
Paraguay
BPD (Building
Partnerships for
Development in Watsan)
Tracey Keatman
traceyk@bpdws.org
Nov
08 to
Feb
09
Nov
2007 to
Feb 08
FINAL
Board of Trustees
NO. AND KIND OF EMPLOYEES
SUPERVISED BY YOU: 60-110
Emigration
DESCRIPTION OF YOUR DUTIES
Overall management of organisation of 60-110 staff and annual turnover of up to R 140m. Strategic leadership, stakeholder liaison,
fundraising, monitoring and evaluation, quality control, human resource management. Also project management of selected strategically
important contracts with key government and donor clients. Writing of articles for publication.
TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
FINAL
MONTH/YEAR
January 1996
Sept 2000
NAME OF EMPLOYER:
The Mvula Trust
ADDRESS OF EMPLOYER:
PO Box 32351, Braamfontein 2017, South Africa
Promotion
DESCRIPTION OF YOUR DUTIES
Management of regions, support in terms of policy guidance, capacity building, project and programme management systems, implementation
procedures, budget control, financial performance. Leadership and management of Policy Unit dedicated to co-ordinating and leading policy
development and advocacy work of organisation. Networking, partnership development, writing.
TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
FINAL
MONTH/YEAR
January 1995
Dec 1995
NAME OF EMPLOYER:
The Mvula Trust
ADDRESS OF EMPLOYER:
PO Box 32351, Braamfontein 2017, South Africa
Water Engineer
TYPE OF BUSINESS:
Johann Froneman
NO. AND KIND OF EMPLOYEES
SUPERVISED BY YOU: None
Promotion
DESCRIPTION OF YOUR DUTIES
Reporting to Projects Director, providing technical expertise in water supply to Head Office and regional offices. Included appraisal of project
feasibility studies, development of project implementation procedures and guidance to regions on appropriate technologies.
FROM
MONTH/YEAR
TO
MONTH/YEAR
July 1991
Dec 1994
NAME OF EMPLOYER:
GTZ
ADDRESS OF EMPLOYER:
Unknown - no longer operating in this area.
Dr Uli Weyl
REASON FOR LEAVING
NO. AND KIND OF EMPLOYEES
SUPERVISED BY YOU: 4
Return to South Africa
DESCRIPTION OF YOUR DUTIES
Working within provincial and district government departments, supported the planning, implementation, monitoring and evaluation of water
supply development projects in four districts. Managed the introduction of improved, hand-dug, shallow well technology, and promotion and
expansion of the programme from pilot areas into all four districts, based on community contributions, local artisans and decentralised
government support. Initiated and expanded demand responsive and community managed programme of low pressure, gravity fed, spring
supplied piped schemes.
TO
FROM
MONTH/YEAR
MONTH/YEAR
January 1990
June 1991
NAME OF EMPLOYER:
Citrinos de Manica
ADDRESS OF EMPLOYER:
No longer exists.
Chief Engineer
TYPE OF BUSINESS:
Osias Manjate
REASON FOR LEAVING
NO. AND KIND OF EMPLOYEES
SUPERVISED BY YOU: None
Career move
DESCRIPTION OF YOUR DUTIES
Responsible for upgrading of irrigation systems for 600ha of citrus plantations. Conversion of four fruit processing facilities for palletised
transport. Liaison with South African financiers and technical support.
TO
SALARIES PER ANNUM
FROM
MONTH/YEAR
STARTING
FINAL
MONTH/YEAR
January 1986
Dec 1989
NAME OF EMPLOYER:
Uniao Geral das Cooperativas Agro-pecuarias de Maputo
ADDRESS OF EMPLOYER:
No longer exists.
TO
FROM
MONTH/YEAR
MONTH/YEAR
April 1981
Dec 1985
NAME OF EMPLOYER:
Citrinos de Maputo
ADDRESS OF EMPLOYER:
No longer exists.
Chief Engineer
TYPE OF BUSINESS:
Prosperino Gallipoli
REASON FOR LEAVING
NO. AND KIND OF EMPLOYEES
SUPERVISED BY YOU: 9
Career move
DESCRIPTION OF YOUR DUTIES
Reporting to CEO, responsible for Water Supply Department, providing potable and irrigation water to small farmer co-operatives in outskirts
of city of Maputo. Managed the design, installation, training of staff and maintenance of over 100 small, appropriate technology irrigation
schemes.
SALARIES PER ANNUM
STARTING
FINAL
Chief Engineer
TYPE OF BUSINESS: