Page -1 Project Evaluation and Control

INTRODUCTION
Late July 2010, Pakistan was hit by devastating flooding that affected an area of the country stretching from the Chinese border in the north to the Arabian Sea in the south. Between 18 to 20 million people were believed to be affected by the flooding, 1.7 million homes were damaged or destroyed, and 1.985 people killed1. This flooding was believed to be the worst in Pakistans history. Plan Pakistan has been responding in the South of the country from July 26, 2010 working with local partners in emergency response interventions in the three Districts of Muzaffargarh, Layyah, and Rajanpur. In the weeks following the mass displacements, as the flood waters began to resume and life returned to normalcy, the devastation of the school infrastructure and the disruption this created for children and families was clear. Through negotiations with DFID, Plan Pakistan developed this project, funded by DFID, to get children back to school.

PROJECT DESCRIPTION
This Education in Emergencies (EiE) project titled Getting Children Back to School in three flood affected Districts of South Punjab focuses on facilitating the return of 58,500 girls and boys, to 390 targeted primary schools in river line areas of Rajanpur, Layyah and Muzafargarh. Through ensuring a safe teaching and learning environment post disaster that engages local education authorities, parents, teachers and children themselves, children will be supported in the return to normalcy, and their right to education upheld. To be implemented over a 10 month period (revised with a No Cost Extension to 13 months), the project was designed as follows: Goal/Overall objectives To provide integrated response and assistance to respond to the differentiated needs of flood affected population in Pakistan. Specific Purpose To ensure 58,500 flood affected children and their families have access to goods, services and support allowing for a return to normality. Sector: 1. Water To ensure 58,500 people (primary school attending girls and boys) have improved access to sufficient safe drinking water. Sector: 2. Sanitation To ensure 58,500 people (primary school attending girls and boys) have improved access to and use of culturally appropriate sanitation facilities. Sector: 3. PHHE (Personal Health, Hygiene Education) To ensure 58,500 (primary school attending girls and boys) trained in safer public health and hygiene practices. Sector: 4.Shelter To ensure 58,500 primary school attending children have access to adequate shelter (schools)
Page 2 Project Evaluation and Control

Sector: 5. NFI (Education related Non-food Items) To ensure that 58,500 school children (primary school aged girls and boys) have access to appropriate education NFIs. Sector: 6. Income Generation To ensure communities where 390 schools are located have increased levels of self subsistence livelihood support through income generation/income injection Sector: 7. Education To ensure 58,500 (primary school attending) children have improved access to adequate education facilities. Sector: 8. Protection To ensure vulnerable groups of flood affected beneficiaries are protected and have equal access to assistance. Sector: 9. Coordination To ensure that all implemented programmes are coordinated and accountable.

EVALUATION PURPOSE
The purpose of this evaluation is to: Assess the progress and results of the operation of the project, paying particular attention to the participation of the community in planning, identification of project beneficiaries, implementation, monitoring, and sustainability of the project. Determine the programs relevance, effectiveness, efficiency, and impact in line with the specific purpose specified in the project proposal. Comment on the present project design and effectiveness and make conclusions and recommendation at both strategy and operational levels for future programming a) for Plans emergency response and education advocacy b) for education policy and advocacy of the Provincial and National Government of Pakistan, also clearly indicating where additional investment are needed in the formal schooling system at various levels including training and monitoring. The evaluation should adopt a gendered approach in considering childrens return back to schools after the emergency and present findings accordingly It should have emphasis on DRR components of the EiE response; an area of special interest to DFID, and an area of strength for Plan Pakistan

Page -3 Project Evaluation and Control

STAKEHOLDERS
Interest or Perspecti ve Project Initiator Project Funder

Stakehold er Name Plan Pakistan DFID (Departmen t for Internationa l Developme nt) Governmen t of Pakistan USAID/ World Bank/ Asian Developme nt Bank. Children of Rajanpur, Layyah and Muzafargar h

Stakeholder Category Primary Primary

Involvemen t in the Evaluation High High

Primary Secondary

Supreme Authority Other Donors

Low Low

Tertiary

Benefiters of Project

Medium

Page 4 Project Evaluation and Control

DESCRIPTION
Need

OF

WHAT

IS

BEING EVALUATED

Plan Pakistan wishes to engage a Consultant to carry out an end of project evaluation in order to assess the impact, effectiveness and results of its DFID funded education in emergencies project interventions. Target Population

58,500 children in 3 flood-affected Districts (Rajanpur, Layyah and Muzafargarh) of Pakistan Stage of Development An end of Project Evaluation in order to assess the impact, effectiveness and results of the project. Final Project Audit and prepare Evaluation report for concerning Stakeholders

EVALUATION DESIGN
Scope of work: The evaluation should be focused on: Develop evaluation framework and methodology and refine these with the project team Develop an evaluation plan Undertake extensive document review to become familiar with the project Meeting with project staff, community and other stakeholders both in offices and in the field Prepare the evaluation draft reports and present the findings in a workshop Prepare and submit final report

Activities Consultants recruitment and finalizing process (in consultation with Plan /DFID) Desk study on provided documents and draft the tools, requirements for evaluation etc. Technical discussion with Plan staff, project management team to make some clarity about TOR (if needed) and agree on (i) the focus of the evaluation, including timeframe; (ii) methodology to be applied in the
Page -5 Project Evaluation and Control

evaluation; (iii) tools to be used in the evaluation Presenting a detailed plan of action for all activities including timeline, tasks and responsibilities parties and Finalizing the methodology, tools & framework for evaluation Preparing the tentative detailed outline of the report. Field study with support of project/Plan M&E staff/ consultant staff Multan level team review discussion Conducting a wrap up meeting with Plan Pakistan Country Directors, Program Managers, and project team Data entry and analysis of the data collected, Writing up report Sending the first draft report on evaluation and revision based on feedbacks Sending the final draft of evaluation Sending the final draft of evaluation

Deliverables:

Following are the deliverables expected from the Evaluation process: A two-page overview of the proposed methodology and timeframe, to be discussed with Plan Pakistan at an initial planning meeting Submit for review Questionnaires and Discussion Guides to be used during Individual Interviews and Focus Groups Discussion (in local language and English) A presentation of initial findings (power point presentation) and proposed draft report for input of project team according to pre-agreed timeline (Multan and Islamabad level) Islamabad level staff sharing presentation/consultation prior to leaving including broader staff such as technical staff from the CO; and management action plan to be finalised before departure (if international assignment) Presentation to the cluster on key findings and recommendations (with District level authorities invited) End of project evaluation report (intended for submission to the donor DFID) including project specific learnings including photographs as relevant (1 x hard copy, 1 x soft copy). This would include: Executive summary, methodology of evaluation, constraints faced in the evaluation, evaluation questions/points for investigation, findings, interpretation of these findings/lessons learned, recommendations, points for further clarification, Appendices to include: minutes of key informant interviews with staff and beneficiaries, report of initial findings, terms of reference for evaluation. Note that in-depth focus on DRR aspects of the EiE response is expected; including school level awareness; and the solar panel components. All raw forms of qualitative and quantitative data and completed evaluation tools both qualitative and quantitative shall be provided to Plan Pakistan upon finalization of Evaluation.
Page 6 Project Evaluation and Control

Evaluation Questions
Relevance 1. Did the objectives and activities of the project address priority needs of the target population? 2. Was the projects beneficiary targeting appropriate given the context? 3. Are there gaps in services, in terms of availability and/or quality that still need to be filled? 4. Are there other beneficiary groups in the area that were not targeted by the project, who have similar priority needs? 5. To what extent did beneficiaries participate in the design, targeting, implementation and monitoring of the project? How could their level of participation have been increased and/or strengthened? EFFECTIVENESS 1. To what extent were project objectives and expected results and expected targets met? What key factors contributed towards the achievement of these objectives and expected results, and what factors hindered it? 2. How effectively did project staff coordinate with other relevant agencies (e.g. Government or other humanitarian agencies) in the area, and nationally? 3. To what extent were lessons of implementation documented and used to modify the project? What are the promising practices that can be expanded in follow-on phases? 4. To what extent was the M&E design able to track the indicators of the project? 5. What are the sustainable institutions & best practices identified, promoted and sustained by the project inputs in the area of livelihoods, shelter and education assistance? Efficiency 1. How efficiently was the work carried out in terms of time and money?

Page -7 Project Evaluation and Control

2. Were project milestones (e.g. monthly and quarterly monitoring reports, pre-final review, etc.) met on schedule? 3. How efficiently were human resources managed? What limitations and challenges faced by the project staff? 4. To what extent did the program learn from mistakes and adapt to make processes more efficient? 5. How did the formal decision-making processes contribute to or hinder the efficiency and effectiveness of the program? 6. How efficiently was the program managed? Describe the strengths and weaknesses of the management structure as it relates to the project. 7. What mechanisms were used to control costs and how effective were these? 8. How could costs be better managed for future projects? 9. How effectively was the project able to mobilize community and government contributions to project activities? What lessons have been learnt from this process?

Impact 1. According to beneficiaries, what were the main impacts of the project, both positive and negative for all assistance? 2. How has this project impacted the lives of the vulnerable groups and what is the sustainability of the systems set up in the community for supporting them? 3. Are there any sustainable and replicable practices to be documented as case studies? What was the impact of DRR components in the project? Are communities better educated and/or prepared for future disasters?

Page 8 Project Evaluation and Control

DATA COLLECTION
Data Collection Methods In order to evaluate the project, consultant should come up with a strategy to: Develop an evaluation plan with evaluation framework and methodology and refine these with the project team Undertake documentation review to become familiar with the project and Education issues more broadly Meeting with project staff, community and other stakeholders both in offices and in the field Prepare the evaluation draft reports and present the findings in a workshop Decide on an appropriate sample, considering whether to analyse a representative or purposively selected sample of beneficiaries or schools, the choice being dependent on availability of data. Review the project documents including proposal, reports, and all training manuals for School Management Committees (SMC), Student Representative Committee (SRCs) and School Development Plan (SDP) and teacher trainings, Child Centered Community Development (CCCD), Disaster Risk Reduction( DRR) and Child protection. Develop testing and evaluation instruments to be administered to various beneficiary groups including teachers, SMC members, stakeholders, and students. Ensure unbiased data collection and qualitative responses with the help of developed instruments Analyze the collected data to quantify an evidence based program impact with respect to outputs and outcomes of the project. Make use of the data collected to date by Plan Pakistan (monthly tracking sheets/teacher questionnaires)

Expected results of the evaluation: A comprehensive reporting outlining the results of the project implemented by Plan in Pakistan, in consideration to relevance, effectiveness, efficiency, and impact Lessons and best practice from the project implementation will be identified and documented Recommendations on future programming in Education in Emergencies responses will be made for Plan and partners Solid recommendations for education policy and advocacy interventions for the provincial and national government including longer term development issues as relevant

Page -9 Project Evaluation and Control

COMMUNICATION

AND

REPORTING

Plan Pakistan is committed to communicating widely the findings of its evaluations and any lessons learned. Starting presumption is that all evaluation reports will be published, including interim reports for those evaluations that are ongoing. They will provide summaries of key findings and, where appropriate, support dissemination e.g through workshops. In addition, evaluation data will be shared with our partners. We will ensure that lessons learned are incorporated into new program and shared more broadly with relevant stakeholders. Timeline The evaluation will require approximately 25 working days, with suggested activities below. This will be finalized based on discussion with the final evaluator/s and Plan Pakistan. At least 35 schools or more would be visited over nine days of field work (approximately 3 days per district). Additionally, time will be allocated for consultation with key district (3 days), provincial (1 day) and national (1 day) level stakeholders. This would include Plan and implementing partner staff in both Multan, and Islamabad.

Page 10

Project Evaluation and Control