Evaluation of Oxfam Biosand Filter Cholera Emergency Response Program Mudzi, Zimbabwe

April 2011

Daniele Lantagne and Fungai Makoni

Executive Summary
The Zimbabwean cholera outbreak is an ongoing cholera epidemic that began in August 2008. As of January 10, 2010 there had been 98,741 reported cases and 4,293 deaths, making it the deadliest African cholera outbreak in the last 15 years. Oxfam/America (OA) and Oxfam/Great Britian (OGB) jointly responded to the epidemic along with the local partner organization, Single Parents and Widow(ers) Support Network (SPWSNet). Biosand Filters, slow-sand filters adapted for use in the home, were distributed to almost 1,000 households to address medium to long-term water needs of the households. To use the Biosand Filter, users simply pour water into the filter and collect finished water out of the outlet pipe into a bucket. Although Biosand Filters have been successfully implemented in the development context, there is little evidence of program success in the humanitarian context. Thus, although the program was established in response to a cholera outbreak, it was also viewed as a pilot program to test the introduction and use of the Biosand Filter technology in a humanitarian context. As such, the program was evaluated at numerous stages throughout the program and upon program completion. The objectives of this sustained use report, which was conducted four months after the program closed, were to provide an independent assessment on: 1) the efficacy and effectiveness of the Biosand Filters for the cholera response; and, 2) a process evaluation (for learning purposes). A mixed-methodology investigation was completed to evaluate the program, including: 1) household surveys to understand Biosand Filter knowledge, use, and sustained use; 2) water quality testing to document Biosand Filter effectiveness; and, 3) key informant interviews with program staff to characterize the response. A total of 61 beneficiary households were interviewed, with complete surveys and water quality samples collected in 29 families receiving filters with plastic housing and 29 families receiving filters with concrete housing. Overall, 56% of surveyed beneficiaries reported having Biosand Filter treated water in the household at the time of the unannounced survey visit. Results from multiple evaluations of this program (including this one) confirm the extensive worldwide data set that the use of Biosand Filters improves the microbiological quality of stored household drinking water. The percentage of the population that had drinking water contaminated drinking water before treatment with the Biosand Filter and uncontaminated after treatment was 63% in the M&E Coordinator evaluation, 43% in the follow-up evaluation, and 56% in this sustained use evaluation. It can be assumed the program reduces diarrheal disease incidence as research studies have shown that diarrheal disease incidence decreases with (even incomplete) microbiological reduction through biosand filtration. A percentage of the surveyed population that is effectively using the intervention to treat contaminated water to microbiologically safe levels can be calculated by: multiplying the percentage of the surveyed population that reported using the filter by the percentage of the reported users whose household stored drinking water was improved from contaminated to uncontaminated by using the filter. In this sustained use evaluation, 56% of surveyed households reported using the filter, and 56% of reported users had household stored drinking water that was improved from contaminated to uncontaminated by use of the filter. Thus, 31% (56% multiplied by 56%) of surveyed households were using the biosand filters effectively to reduce stored household water contamination at the time of the unannounced survey visit. This is the highest effective use percentage the consultant has seen for the use of biosand filters in the emergency context in her evaluations with UNICEF and Oxfam to date.

Page 2 of 38

The importance of adequate training on BSF use was consistently identified in this evaluation. First, training was associated with increased use of the filter, as respondents who reported receiving more than one training of any type were more likely to report treated water at the time of the unannounced survey visit than those who reported receiving one training or less. However, there were significant gaps identified in households applying their training to correct installation and maintenance of their Biosand Filter, as: The majority of beneficiaries (68%) did not have the appropriate standing water depth (4-6 cm) to correctly maintain the biolayer (schmutzdecke) in their filter. Of particular concern is the high percentage (14% each) of beneficiaries at the extreme ends with either 0 cm or !10 cm. Having the correct standing water depth was not correlated with any training or assistance from volunteer or SPWSNet indicating that training was not sufficient to ensure adequate installation. The majority of respondents reported cleaning the filter in some way (84%). However, the cleaning strategies were inconsistent, with a variety of inappropriate methods (from removing all the sand to cleaning only the sand at the top to cleaning the outside only to reinstalling the whole filter) employed irregularly by households. The vast majority of beneficiaries use the water for only drinking and cooking (82%). Given the fast flow rate in the Biosand Filters, it is recommended to encourage households to use the water for all purposes. These user training operations and maintenance difficulties, which may: 1) lead to the misperception in households that they are consuming safe water when in fact they are not; and, 2) inhibit the long-term sustained use of the filter. The data suggest that there are differences in the effectiveness and efficiency of the plastic and concrete Biosand Filters. It is not clear at this point how much the differences are due to innate differences between the filters styles and how much is due to the fact concrete filters were distributed later than plastic ones, with improved education and training. However, it does appear that concrete filters were less likely to crack. Based on the small number of families confirmed to be reached with a Biosand Filter (only 897), and the time period after the cholera in which they were reached, it can not be stated that the Biosand Filters were an appropriate intervention for the program goal of responding to the cholera. A total of 69% of surveyed respondents who received Biosand Filters also received Aquatabs as a cholera response household water treatment option from (an)other organization(s). Aquatabs are less expensive and easier to distribute, simpler to use, and more effective than Biosand Filters at reducing the cholera bacteria from household stored drinking water. However, Aquatabs are a consumable product, not a durable, and thus they only provide protection while the household uses the products until they run out. The key evaluation question for Biosand Filters is no longer on microbiological effectiveness, but instead on appropriate implementation strategies that result in users effectively using the filters to improve the quality of their stored household water in a cost-effective manner. Should OA wish to continue with Biosand Filter implementations in emergencies, it is recommended that OA: Coordinate future programs with the Government of Zimbabwe. The focus of future programs should not be on confirming microbiological effectiveness or health impact (which have been well-verified) but instead on establishing cost-effective distribution strategies that reach a large population with the training necessary to encourage consistent and correct use. Understand the potentially limited role of Biosand Filters in the acute emergency situation. Biosand Filters take time to distribute, train on, and for the schmutzdecke to develop. A chlorine-based product should

Page 3 of 38

be distributed in the emergency response time until such time as there is capacity to complete this training and development. Complete significant training with beneficiaries, that teaches how to: a) adequately install the filter, including establishing the correct standing water layer; b) maintain the filter, including where to obtain replacement parts; c) how to clean the filter appropriately and on what schedule; and, d) highlights that filtered water can be used for more than just drinking and cooking. Keep program design and scope realistic and considered. In the future, research plans should be more fully integrated into implementation.

Page 4 of 38

Table of Contents
Table of Contents !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" 1 Introduction!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!# $ Previous research !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! %
2.1 OA program!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!% 2.2 Government of Zimbabwe draft summary report !!!!!!!!!!!!!!!!!!!&$ $!' Sustained use report!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!&$

' Methodology !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &(

3.1 Household surveys!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &(

3.2 Water quality testing !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &( 3.3 Key informant interviews!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &(

( Survey Results !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! &" " Process Evaluation Results!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! $) * Discussion and Recommendations!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! $'
6.1 Independent assessment on efficacy and effectiveness of the

Biosand Filters for the cholera response!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! $' 6.2 Process evaluation!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$( 6.3 Recommendations !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$*

Annex A: Terms of Reference!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!$# Annex B: Itinerary and survey respondents !!!!!!!!!!!!!!!!!!!!!!!!!!!! '$ Annex C: Household questionnaire !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! '' Annex D: Water quality indicators!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! '#

Page 5 of 38

List of Figures and Tables

Figure 1: OA program timeline of evaluations ................................................................................................ 8 Figure 2: M&E Coordinator paired treated/untreated water E. coli samples.................................................. 10 Figure 3: Project-end evaluation paired treated/untreated water E. coli samples.......................................... 11 Figure 4: Standing water level in filters (percent of samples in ranges, by centimeter)................................. 17 Figure 5: Sustained use evaluation paired treated/untreated water E. coli samples ...................................... 18 Figure 6: An unused filter with iron staining, a filter used for storing tomatoes, a correctly installed and used filter................................................................................................................................................ 19

Table 1: Household characteristic results...................................................................................................... 16 Table 2: Training received by respondents for plastic and concrete containers ........................................... 17 Table 3: Cleaning strategies for plastic and concrete filters .......................................................................... 18 Table 4: Water quality data ........................................................................................................................... 37

Abbreviations
BSF CFU FR M&E NFI NGO OA OGB SPWSNet Biosand Filter Colony Forming Unit Female Respondent Monitoring & Evaluation Non-food Items Non-governmental Organization Oxfam/America Oxfam/Great Britain Single Parents and Widow(ers) Support Network

Acknowledgements
The authors would like to thank all those involved in the planning and execution of this study, particularly the OA/Boston, OA/Zimbabwe, SPWSNet staff, and enumerators. Without their support this report would not have been possible.

Page 6 of 38

1 Introduction
The Zimbabwean cholera outbreak is an ongoing cholera epidemic that began in August 2008. As of January 10, 2010 there had been 98,741 reported cases and 4,293 deaths, making it the deadliest African cholera outbreak in the last 15 years. Mudzi District, in north-eastern Zimbabwe on the Mozambican border, was one of the most affected districts. Although cholera is a yearly occurrence in Zimbabwe, the outbreak beginning in 2008 was worse because it started before the rainy season, affected a large number of people, had a mortality rate, and also occurred in urban environments previously spared from yearly outbreaks. Oxfam/America (OA) and Oxfam/Great Britian (OGB) jointly responded to the epidemic along with the local partner organization, Single Parents and Widow(ers) Support Network (SPWSNet). The initial response came in the form of borehole repair, distribution of non-food items, health and hygiene education, coordination, and supply of oral rehydration salts. A preliminary assessment was also undertaken on the overall situation and the groundwork was laid for the development of a diarrhea early warning system and the introduction of the Biosand Filters to address the medium to long-term needs of the households. The Biosand Filter is a slow-sand filter adapted for use in the home. The version of the Biosand Filter most widely implemented consists of layers of sand and gravel in a concrete or plastic or concrete container approximately 0.9 meters tall, and 0.3 meters square. The water level is maintained to 4-6 cm above the sand layer by setting the height of the outlet pipe. This shallow water layer allows a bioactive layer (schmutzdecke) to grow on top of the sand. Diarrheal disease causing organisms are removed through mechanical trapping, adsorption, and predation by schmutzdecke organisms. A diffuser plate with holes in it is placed on the top of the sand layer to prevent disruption of the biolayer when water is added to the system. To use the Biosand Filter, users simply pour water into the filter, and collect finished water out of the outlet pipe into a bucket. Although Biosand Filters are widely and successfully implemented in the development context, there is little evidence of program success in the humanitarian context. OA implemented a pilot program to distribute Biosand Filters to 950 families in Mudzi District in April 2009. OA purchased 450 pre-fabricated plastic filter housings from Hydraid in the United States and commissioned the manufacture of 500 locally-made concrete filter housings in Zimbabwe. Working with SPWSNet, the filters were distributed to families in stages, accompanied by filter use and maintenance training coupled with public health and hygiene education. Although the program was established in response to a cholera outbreak in north-eastern Zimbabwe, it was also viewed as a pilot program to test the introduction and use of the Biosand Filter technology in a humanitarian context. As such, the program was evaluated at numerous stages throughout the program and upon program completion. The main purpose of this post-program sustained use evaluation is to draw lessons from this experience on the viability of the technology, effective management arrangements, and minimum conditions necessary for sustainable use of the filters by communities. The Terms of Reference for this sustained use evaluation are appended in Annex A. In this report, previous research on Biosand Filters in Zimbabwe is first summarized, then results from the sustained use evaluation are presented, and lastly conclusions and recommendations are presented.

Page 7 of 38

$ Previous research
Numerous organizations, including OA, have been interested in implementing and researching Biosand Filter programs in the development and emergency situation in Zimbabwe. In this section the research completed to date on Biosand Filters in Zimbabwe is briefly summarized, including the OA program and a summary report of results from NGO programs using the Biosand Filter in Zimbabwe from the Government of Zimbabwe. The Terms of Reference for this sustained use evaluation are summarized at the end of this section and fully appended in Annex A.

2.1 OA program
The OA Biosand Filter cholera response and research program began in April 2009 and concluded in September 2010 (Figure 1). Monitoring and evaluation occurred throughout this program, as: 1) the Biosand Filter technology was new for OA, SPWSNet, and the communities and adjustments and improvements were regularly needed during program implementation; and, 2) part of the intention of the program was to learn lessons for potential expansion. This report documents the fourth major evaluation of this program (Figure 1). The first evaluation, the initial distribution evaluation, was completed by this consultant in July 2009. The second evaluation, the ongoing evaluation, was performed by an OA/Zimbabwe Monitoring and Evaluation (M&E) Coordinator in December. The third evaluation, a program-end evaluation, was conducted by OA/Zimbabwe in September 2010. The evaluation described in this report, the sustained use evaluation, was completed in January 2011. The results of these evaluations are summarized in the following sections.
2009
s 11 Mon itorin g an d Ev 12 a on-b luation oard pers on 1 10 3 2 6 Initia l Dis Visit tributio 7 n by O xfam Evaluat ion /Am erica 8 P Visit roject s ta by O 4 xfam rt /Am erica First distr 5 ibutio ns 9 4 5

2010
by O 8 xfam /Am Fina erica l Eva luatio 9 Proj ect C n and lose 10 11 6 7

2011
12 Sust aine d Us 1 e Ev alua tion

and cov prov ered bu ided cket

Educ ation

Figure 1: OA program timeline of evaluations

$!&!&

Initial distribution evaluations

At this initial evaluation, 531 filters had been distributed in 117 communities across six wards of Mudzi. This averaged to only 4.5 filters distributed per community. Although a large number of filters had been distributed to families in a short amount of time, only 3 (17%) of 17 randomly selected households had a correctly assembled filter they were using at the time of the unannounced visit. Problems were noted with maintaining the correct water level of 5 cm above the sand to ensure schmutzdecke development and with storage of filtered water in

Page 8 of 38

Tw o

visit s

open containers where recontamination is possible. Based on these results it was recommended to: 1) delay distribution of remaining filters until expansion and retraining of the Ward Volunteer network occurred; and, 2) visit each existing filter household and ensure the filter is correctly assembled, provide an informational poster, answer user questions, ensure a covered container is being used for water storage, and collect a GPS point. Additionally, it was noted that the program was attempting to be many things at once a pilot program, a research program, an implementation program, a program targeting the most needy people, a program comparing two types of filters all in an aggressive time frame in an emergency context. The recommended next steps were to: 1) determine how to ensure correct usage of the filters by providing appropriate training; and, 2) determine realistic outcome measures that can be obtained from this program. Based on the above evaluation, numerous changes were implemented, including: distribution was delayed; the Ward Volunteer network was expanded and retrained; all households were revisited to ensure correct filter assembly; more time and energy was put into training; a dedicated covered storage container was provided; distributions were clustered within communities; and, informational print materials were developed. In addition, the initial desired outcome measure of diarrheal disease reduction was dropped from the evaluation, as there was neither statistical power nor controls sufficient to evaluate diarrheal disease reduction in this program.

$!&!$

Ongoing evaluation

After the program restarted distribution, the OA/Zimbabwe Biosand Filter project M&E Coordinator conducted ongoing evaluation. Three program reports (from December, January to February, and March to May) summarizing data collected by the M&E Coordinator were made available to the consultant and are summarized in this section. The M&E Consultant reported that communities believed that the Biosand Filter treated water was safe as evidenced by only 27% exposing treated water to post-filtration treatments such as Aquatabs or boiling. A total of 29 (5%) of the 571 filters were cracked. The M&E Coordinator reported E. coli results from 16 paired household water samples one sample collected from the household before treatment and one reported treated sample collected from the household water storage container. These results were analyzed by the consultant, and are presented in Figure 2. As can be seen, the use of the Biosand Filter improves the microbiological quality of stored household water. More households had household stored water in the no risk category after treatment than had water in this no risk category before treatment. Overall, 10 of the 16 households (62.5%) had water that was contaminated before treatment (!1 CFU/100 mL of E. coli) and that was not contaminated (<1 CFU/100 mL of E. coli) after treatment.

Page 9 of 38

Figure 2: M&E Coordinator paired treated/untreated water E. coli samples Due to Oxfam administrative issues, the results from 571 questionnaires administered by the M&E Coordinator were not available to the consultant at the time of writing this report.

$!&!'

Project+end evaluation

OA in Zimbabwe conducted a program-end evaluation in September 2009. A total of 194 (23%) of the 850 confirmed beneficiaries of Biosand Filters were surveyed. The households were not randomly selected. Overall, 52% of households had access to improved sources of water, 80% covered their stored water container, and 93% reported still using Biosand Filter. Slightly over half had correct standing water levels to correctly maintain the schmutzdecke, with 29% having water levels too low (between 0 and 4 centimeters above the sand level), and 29% having levels too high (above 6 centimeters). Almost every household (99%) reported receiving training on the Biosand Filter, with 66% reporting follow-up household visits. In addition, 77% reported receiving education, including from SPWSNet (44.2%) and the Ministry of Health and Child Welfare (56%). The training from the Ministry was likely general cholera messages, and not specific Biosand Filter information. The program-end evaluation, like the ongoing evaluation, noted post-treatment after filtration with another household water treatment option, as 7.4% of respondents reported post-treatment with Aquatabs, boiling, or solar disinfection. Program challenges noted included cracking in the container, incorrect standing water levels, long maturation time for the schmutzdecke to develop, and slow filtration rates. The percent of households reporting Biosand Filter treated water and the time of the unannounced survey visit was not included in the program-end evaluation report. The water quality parameters of turbidity and E. coli were tested. Turbidity was generally low across all samples tested. Sixty triplicated household water samples were tested: 1) untreated water stored in the house; 2) water filtered through the Biosand Filter by the enumerator and collected directly from the filter without touching a storage container; and, 3) reported Biosand Filter treated water stored in the household storage container. These results were analyzed by the consultant, and are presented in Figure 3. It is clear, again, that the use of the Biosand Filter improves the microbiological quality of stored household water. Overall, 38 of the 60 households (63%) had water that was contaminated before treatment (!1 CFU/100 mL of E. coli) and that was not contaminated (<1 CFU/100 mL of E. coli) directly after filtration (no storage). Furthermore, 26 of the 60 households (43.3%) had water that was contaminated before treatment (!1

Page 10 of 38

CFU/100 mL of E. coli) and that was not contaminated (<1 CFU/100 mL of E. coli) after treatment in the stored treated water. Thus, the use of the filter improves the microbiological quality of stored household water, but there was recontamination in storage noted.

Figure 3: Project-end evaluation paired treated/untreated water E. coli samples The program-end report recommended that: Projects should provide families with technologies that allow families to use water which is close to their environments; Provisions should be made for the households to be assisted with the resources required for improved hygiene relevant to Biosand Filters; A critical mass of volunteers should be in place before the actually assembly of filters begin, at a ratio of 10 beneficiaries to 1 volunteer; Capacity building of users should target households instead of the head of household only; Biosand Filters should be given to communities as options of improving the quality of their drinking water at household level; The local community leadership should continue to give the volunteers visibility; Regular workshops and exchanges should be carried out for both users and volunteers; Printed supporting materials should be distributed; Extra materials, such as concrete, should be available; Large-scale implementation programs should consider the following issues which strongly influence outcome of the program: sources of water, voluntary spirit levels among beneficiaries, leadership support, transportation logistics for volunteers, demonstration units, and the monitoring system; and, Hygienic practices should be reemphasized.

Page 11 of 38

2.2 Government of Zimbabwe draft summary report

Given the recent interest in Biosand Filters in Zimbabwe, the Government of Zimbabwe is evaluating whether they will approve Biosand Filters formally through the Standards Association of Zimbabwe. At this point in time, the government has collated information for numerous NGO reports into one draft document. It is unclear at this point in time which Ministry will be responsible for leading the decision on Biosand Filters in Zimbabwe. In its summary document, the government report notes that a number of NGOs in Zimbabwe have been promoting the One Way Ministries manufactured Biosand Filter as a household water treatment method in different rural areas of the country on a pilot basis. They noted 2,000 filters had been installed by the end of 2010 by International Medical Corps (Bindura, Shamva and Rushinga), Single Parents and Widow(ers) Support Network (SPWSNet) (Mudzi), Institute of Water and Sanitation Development (Hatcliffe, Harare), and One way Ministries (Epworth, Harare). The objective of the Government report was to assess the efficacy of the BSF in removing pathogenic bacteria and turbidity ensuring drinking water is safe at point of consumption as presented in the different reports. The assessment also aims at providing NAC with adequate information to make an informed policy decision on the adoption, rejection or recommendations for macro field trial and further improvement and development of the Biosand Filter as a household water treatment technology. Based on the compilation of 160 water quality samples from the various implementing organizations, the Government concluded that technical efficacy of the Biosand Filter alone is not enough to ensure sustainability and for NAC to make a policy decision on the adoption or rejection of the technology and that further work is needed to improve the results and the success of the program including achieving continued and active long term adoption of the technology will require adequate data on the main interlinking elements of the program: technical (E. coli and turbidity removal rates), user perceptions, operations and maintenance, hygiene education, use of filtered water, durability, sustainability of the filters and monitoring and evaluation even beyond the program implementation period. To ensure these elements are adhered to there is need for clear Terms of Reference and support by NAC [the Government of Zimbabwe] in form of among others monitoring & evaluation.

$!' Sustained use report

The objectives of this sustained use report were to provide an independent assessment on: 1) the efficacy and effectiveness of the Biosand Filters for the cholera response; and, 2) a process evaluation. The specific questions for the efficacy and effectiveness objectives included: Was the introduction of Biosand Filters an appropriate intervention? Were the Biosand Filters effective in the cholera response? Has the quality of water improved with the use of filters for household consumption (cooking, drinking, and sanitation)? Did communities receiving the filters change their behavior in preventing disease during the program? What has been the impact on prevalence of water-borne diarrheal diseases in these communities? Is there any difference in effectiveness or efficiency between the plastic Biosand Filters compared to concrete ones? How sustainable is the use of Biosand Filters likely to be?

Page 12 of 38

The process evaluation included reviewing question of coverage; cost-effectiveness; efficiency and coordination mechanisms; communications; information management and accountability; communication and reporting; and design of program. The methodologies to complete these objectives are presented in the next section, followed by survey and process results, and conclusions and recommendations in subsequent sections. The full Terms of Reference for this sustained use evaluation are appended in Annex A.

Page 13 of 38

' Methodology
A mixed-methodology investigation was completed to fully evaluate the program, including: 1) household surveys to understand Biosand Filter knowledge, use, and sustained use; 2) water quality testing to document the effectiveness of the Biosand Filters; and, 3) key informant interviews with program staff to characterize the response. Focus group discussions were not completed due to the distance between filter beneficiaries.

3.1 Household surveys

The consultant developed a survey consisting of 44 questions on: 1) respondent and household characteristics, assets, and HWTS knowledge and products received; 2) installation, training on, use, and maintenance of the household Biosand Filter; and, 3) current treated and untreated stored household drinking water, including sample collection for later analysis. Each survey took 15-20 minutes to administer per household. Survey training and pretesting occurred on January 18, 2011, with four enumerators. A finalized survey was developed and printed in Mudzi. The survey was written in English and administered in English or Shona, depending on the family. From a line list of the 897 Biosand Filter beneficiaries, 100 households were randomly selected using the Microsoft Excel (Redmond, WA, USA) random function. All survey data was entered into Microsoft Excel and analyzed using Stata 10.1 (College Station, TX, USA). The evaluation itinerary and survey tool used in the evaluation are appended in Annex B and Annex C, respectively.

3.2 Water quality testing

Enumerators were trained by the consultant to collect a treated water sample (if available) and an untreated water sample aseptically from each surveyed household. Samples were collected in sterile WhirlPak! bags with sodium thiosulfate to inactivate any chlorine residual present, and stored in a cooler on ice for analysis. Each evening after surveys were completed, the consultant completed the microbiological testing using Millipore (Billerica, MA, USA) portable filtration stand laboratory equipment. Samples were diluted appropriately with sterile buffered water, filtered aseptically through a 45-micron filter, placed in a plastic petri-dish with a mColiBlue24 media soaked pad, and incubated in a portable incubator for 24 hours at the appropriate temperature. Red colonies were counted as total coliform, and blue colonies as E. coli. One deviation from Standard Methods is holding time before the sample was fully filtered was extended from 8 to 12 hours due to travel logistics (APHA/AWWA/WEF, 1998). Negative controls were included each day and 10% of samples were duplicated for quality control.

3.3 Key informant interviews

Key informant interviews with program staff were conducted to characterize both the programmatic response and evaluation, and the management strategies necessary to complete a program leading to sustained use. Key informant interviews were conducted formally and informally with program staff from OA/Zimbabwe, SPWSNet, and OA during the field visit and after the consultants return.

Page 14 of 38

( Survey Results
In four days of work, four enumerators located 61 of the 100 families randomly selected for survey. This is equivalent to a survey rate of 3.8 surveys per enumerator per day. The reasons 19 of the 39 families were not surveyed were: the household was not home because they were in the fields (8/19, 42%); the household was located further than a one hour walk from the road (4/19, 21%); because the car was stuck in the mud (2/19, 11%) or unable to cross the river; and, because the household moved (2/19, 11%). The remaining 20 families were unable to be located. Of the 61 families location by the enumerators, full surveys and water quality samples collected were completed in 29 families receiving plastic filters and 29 families receiving concrete filters. An additional survey (no sample collections as the surveys were completed outside the owners home) were completed with families receiving plastic filters and two samples only (no surveys as the water was collected from the beneficiaries home with a neighbor present) were completed on concrete filters. Thus, a total of 60 surveys and 61 sample collections were completed. Household characteristic results are presented in Table 1. A statistically significant difference between plastic and concrete filters is noted by a p-value of less than 0.05. Of note in the survey results are: Plastic filter beneficiaries were more likely to have received their filter in 2009, as opposed to concrete filter beneficiaries, who received their filters in 2010. The majority of survey respondents were female, and the majority attended school. A statistically significantly smaller number of plastic filter beneficiaries reported having a family member with cholera (14%) as compared to concrete filter beneficiaries (57%), indicating targeting improved over time. The majority of filter beneficiaries (69%) reported also receiving Aquatabs for cholera response. The majority of respondents reported receiving enough education for installation (95%) and maintenance (95%), had help assembling the filter (79%), and knew someone else with a Biosand Filter (93%). The majority of respondents reported ever using the filter (98%), currently using the filter (85%), and planning to keep using the filter in the future (100%). Of the nine people who report currently not using the filter the reasons why are: broken (3), and one each for it does not fit in house, it has problem, using rainwater instead, not using, have no water, and gave it away. There was a statistically significant difference in reported current use between plastic and concrete filters, with a higher percentage of concrete filter beneficiaries reporting current use. This could be because these beneficiaries received filters more recently. The majority of filters were wet (91%) on observation, which is a non-biased indicator of use. A minority of users reported sharing the filtered water (33%). The majority of respondents reported cleaning the filter in some way (84%). The majority of respondents reported storing treated water in covered containers (74%). Overall, 58% of respondents reported treated water at the time of the unannounced survey visit, with 97% of those treating (34 respondents) using the Biosand Filter. No user reported post-treatment of filtered water with another option, which differs from previous evaluations. The vast majority of beneficiaries use the water for only drinking and cooking (82%). Concrete- filter beneficiaries were more likely to use water for more than only drinking than plastic filter beneficiaries. Page 15 of 38

Table 1: Household characteristic results

Plastic filters Number of households surveyed and or sampled Number (%) received Biosand Filter in 2009 (n=58) Number (%) female respondents (n=58) Average respondent age in years (min-max) (n=58) Number (%) female respondents attend school (n=43) If FR school, average (min-max) years school (n=33) All FR average (min-max) years school (n=44) Number (%) female HOH who can read newspaper (n=57) Number (%) homes with someone with cholera in last 2 years (n=56) Average (min-max) minutes to drinking water (n=60) Number (%) report receiving Aquatabs (n=61) Number (%) report being a volunteer (n=58) Number (%) report receiving enough education for installation (n=57) Number (%) report having help assembling filter (n=58) Number (%) report knowing someone else with a Biosand Filter (n=59) Number (%) report ever using the Biosand Filter (n=59) Number (%) report currently using the Biosand Filter (n=59) Number (%) report planning to keep using the Biosand Filter (n=57) Number (%) report with currently wet Biosand Filter on observation (n=54) Number (%) report sharing Biosand Filter water (n=57) Number (%) report cleaning the filter (n=55) Number (%) report receiving enough education for maintenance (n=57) Number (%) with covered stored water (n=53) Number (%) reporting any treatment (n=59) Number (%) reporting water treated with Biosand Filter (n=34) Number (%) reporting water treated with Aquatabs (n=34) Number (%) reporting water treated with other (n=34) Number (%) use Biosand Filter water for more than only drinking (n=61) Number (%) use Biosand Filter water for more than only drinking/cooking (n=61) 4 (13%) 8 (31%) 22 (85%) 27 (100%) 21 (72%) 14 (47%) 13 (93%) 1 (7.1%) 0 (0%) 17 (56%) 30 (49%) 23 (85%) 21 (72%) 38 (18-69) 16 (76%) 8.6 (3-13) 6.5 (0-13) 19 (70%) 4 (14%) 28 (5-120) 18 (60%) 3 (11%) 27 (100%) 20 (74%) 25 (89%) 27 (96%) 21 (75%) 27 (100%) 22 (85%)

Concrete filters 31 (51%) 13 (42%) 23 (79%) 40 (15-80) 17 (77%) 7.3 (3-13) 5.4 (0-13) 15 (50%) 16 (57%) 31 (10-90) 24 (77%) 6 (19%) 27 (90%) 26 (84%) 30 (97%) 31 (100%) 29 (94%) 30 (100%) 27 (96%) 11 (36%) 24 (83%) 27 (90%) 18 (75%) 20 (69%) 20 (100%) 0 (0%) 0 (0%) 26 (84%) 7 (23%)

Total 61 36 (62%) 44 (76%) 29 (15-80) 33 (77%) 7.9 (3-13) 6.0 (0-13) 34 (60%) 20 (36%) 30 (5-120) 42 (69%) 9 (16%) 54 (95%) 46 (79%) 55 (93%) 58 (98%) 50 (85%) 57 (100%) 49 (91%) 19 (33%) 46 (84%) 54 (95%) 39 (74%) 34 (58%) 33 (97%) 1 (3%) 0 (0%) 43 (70%) 11 (18%)

p-value

<0.01 0.54

0.12 <0.01 0.14 0.39 0.19 0.36 0.25 0.29 0.05 --0.14 0.71 0.85 0.09 0.83 0.08 ------0.02 0.35

Page 16 of 38

Survey respondents reported receiving a variety of trainings (Table 2). Training was associated with increased use of the filter, as respondents who reported receiving more than one training of any type were more likely to report treated water at the time of the unannounced survey visit than those who reported receiving one training or less (p=0.05).The only statistically significant difference in training received between plastic and concrete filters was that beneficiaries of concrete filters received more pamphlets, which is attributed to the concrete filters being distributed later on in the program after written materials were developed. Twenty (57%) of the 35 respondents who reported receiving pamphlet information could produce it. Table 2: Training received by respondents for plastic and concrete containers Training received Poster/pamphlet (n=61) Household visit training (n=61) Household visit monitoring (n=61) Group training (n=61) Plastic filters 12 (40%) 17 (57%) 3 (10%) 17 (56%) Concrete filters 24 (77%) 20 (64%) 1 (3%) 20 (65%) 36 (59%) 37 (61%) 4 (6%) 37 (61%) <0.01 0.53 0.29 0.53 Total p-value

The majority of beneficiaries (68%) did not have the appropriate standing water depth (4-6 cm) to correctly maintain the schmutzdecke in their filter (Figure 4). Of particular concern is the high percentage (14% each) of beneficiaries at the extreme ends with either 0 cm or !10 cm. Having the correct standing water depth was not correlated with any training or assistance from volunteer or SPWSNet (p=0.27 and p=0.30, respectively) indicating that training was not sufficient.

Figure 4: Standing water level in filters (percent of samples in ranges, by centimeter)

Page 17 of 38

The average number of times users cleaned their filter was 1.2 times per week (range 0-7). For plastic filters this average was 1.8 times per week (range 0-7) and for concrete this average was 0.7 times per week (average 0.023.5). The cleaning strategies were inconsistent (Table 3), with 16% of respondents overall reporting never cleaning, and a variety of cleaning strategies (from removing all the sand to cleaning only the sand at the top to cleaning the outside only to reinstalling the whole filter) were employed. A particularly worrisome cleaning strategy was to wash slime out, or remove the schmutzdecke, which actually decreases the microbiological removal efficacy of the filter. Concrete filter beneficiaries were statistically significantly more like to clean just the sand at the top, which is the appropriate cleaning strategies for the Biosand Filter. This could be due to something innate in the concrete filter, or the fact that these beneficiaries received filters later, with better educational materials. In future programs, beneficiaries need sufficient training on a consistent, appropriate cleaning strategy. Table 3: Cleaning strategies for plastic and concrete filters Cleaning Strategy Remove all the sand, and clean (n=61) Clean just the sand at top (n=61) Clean outside only (n=61) Reinstall whole filter (n=61) Never clean (n=61) Other (clean parts with cloth (2), wash slime out (1)) Plastic filters 6 (20%) 7 (23%) 7 (23%) 1 (3%) 5 (17%) 2 (7%) Concrete filters 6 (19%) 18 (58%) 1 (3%) 3 (10%) 5 (16%) 1 (3%) 12 (20%) 25 (41%) 8 (13%) 4 (7%) 10 (16%) 3 (5%) 0.95 <0.01 0.02 0.32 0.96 0.53 Total p-value

Overall, 34 (59%) or 58 respondents reported no problems with their filter. The problems reported by the 41% of respondents listing problems included: slow flow/clogging (16/58, 28%); cracks (4/58, 7%); broken taps (2/58, 3%); and one respondent each (2%) for no bucket, leaking, rat fell in, rusty tap, and treatment. Three of four people reporting cracking problems had received plastic containers. An additional problem noted in household visits, but not mentioned by respondents in the survey was staining from iron in the water in the clear, white dispensing tube (Figure 6). A total of 16 paired treated/untreated water samples were analyzed in this evaluation. A total of 9 of the 16 households (56%) had water that was contaminated before treatment (!1 CFU/100 mL of E. coli) and that was not contaminated (<1 CFU/100 mL of E. coli) after treatment (Figure 5).

Figure 5: Sustained use evaluation paired treated/untreated water E. coli samples

Page 18 of 38

Figure 6: An unused filter with iron staining, a filter used for storing tomatoes, a correctly installed and used filter

Page 19 of 38

" Process Evaluation Results

OA/Boston requested the consultant to write a narrative describing her experience with the process portions of the program, beginning with her initial interactions with the program through to this evaluation. The consultant was first made aware of the program in early 2009, while she was working with a UNICEF/Oxfam contract at the London School of Hygiene and Tropical Medicine. The Terms of Reference for that work were to evaluate the effectiveness and acceptability of HWTS options in acute emergency situations. The protocol was open-ended, and the consultant was to travel to four emergencies that occurred during the time of the contract to evaluate HWTS distributions within eight weeks of emergency onset. Initially, the consultant was contacted by OA to provide input into the Zimbabwe Biosand Filter program and to assess whether this program might be appropriate for inclusion into the larger UNICEF/Oxfam program. At the time of the initial distribution evaluation by the consultant, it was clear that the process by which the program was being implemented was non-ideal. Individuals within OA/Boston had different ideas about the program goals, with some wanting an emergency program, some a health impact study, some a pilot program. OA/Zimbabwe staff received conflicting messages about the program from OA/Boston staff and were frustrated by how the program was managed. OA/Boston staff had little interaction with the implementing partner, SPWSNet, or the Government of Zimbabwe, leaving those interactions at the discretion of OA/Zimbabwe staff. Overall, this lack of communication between OA/Boston and the on-the-ground implementing team led to the program being implemented in ways that surprised OA/Boston. For example, Biosand Filters were distributed by SPWSNet to households selected by local chiefs, rather than households affected by cholera. In addition, filters were distributed in a wide geographical area to obtain the most political capital from chiefs, which is not conducive to evaluation or a research study. Filters were distributed to families with minimal group training, and families were expected to install and maintain their own filters without support. Laslty, SPWSNet did not have access to training materials and resources for correct installation of filters. In the initial distribution evaluation, recommendations were made to improve distribution and training. In addition, the consultant linked OA and SPWSNet to the Centre for Affordable Water and Sanitation (CAWST), an organization that provides technical assistance on Biosand Filters. It was mutually decided at the end of the initial distribution evaluation by the consultant and OA that this program was not appropriate for inclusion within the larger UNICEF/Oxfam program due to the lack of a program established in the acute emergency context. From July 2009 through to December 2010, the consultant was involved with the program at a minor level, sometimes answering emails or a technical question. The consultant had no substantive knowledge of how the program was progressing. In late September 2010, the consultant was contacted to conduct a sustained use evaluation of the program. Arrangements were made and the evaluation was planned for the end of January 2011. OA/Boston consistently emphasized to the consultant the importance of this evaluation being independent and unbiased. The consultant and OA/Boston worked with OA/Zimbabwe to arrange for an independent local consultant to assist with evaluation logistics in Zimbabwe. The name of the local consultant was provided to the consultant and OA/Boston the week before departure. Upon emailing the local consultant, the consultant was told by OA/Zimbabwe staff that logistics were managed and would be discussed upon arrival. Upon arrival to

Page 20 of 38

Zimbabwe, the consultant met with the local consultant, who was unfamiliar with the evaluation and surprised the government had not been contacted to approve it. Unbeknownst to OA/Zimbabwe staff, the local consultant arranged and executed meetings with Ministry of Health and Water officials. Significant concern about approving the evaluation was expressed, and noted local Mudzi government staff approval would be needed. After reconvening with OA/Zimbabwe staff, it was discovered that OA/Zimbabwe staff had, unbeknownst to OA/Boston or the consultant, arranged for SPWSNet staff, volunteers, and OA/Zimbabwe staff to travel to Mudzi and assist with the survey. No arrangements for government approval had been made. On Monday afternoon (see timeline in Annex B), OA/Zimbabwe staff, SPWSNet staff, the local consultant, and the consultant departed for Mudzi. On Tuesday, SPWSNet staff and OA/Zimbabwe staff finalized the Biosand Filter beneficiary list, and the consultant trained the enumerators and randomized the beneficiary list to select households for survey. SPWSNet and the enumerators suggested two non-standard survey methods to the consultant. The consultant firmly expressed to SPWSNet that selected households should not be called to pre-arrange a household visit and firmly expressed to the enumerators (some of whom had previously evaluated this program) that replacements of households not at home with households at home that they could locate when surveying was not to be completed. Both of these methods can bias usage results, as households will treat water if they know a visit will occur. On Wednesday, after gaining approval from the local government, surveys began in the early afternoon. In the survey results, we found that the vast majority of beneficiaries reported training, however, the quality of the training did not lead to beneficiaries having correct knowledge and ability on how to install, maintain, and clean their filter. Clearly, there was information loss in providing the correct training to the beneficiaries, but we were not able to distinguish whether that information loss occurred between OA and SPWSNet, SPWSNet and the Volunteers, or between the Volunteers and the beneficiaries, because respondents were not able to distinguish between SPWSNet staff and Volunteers trained by SPWSNet in survey responses. In retrospect, given the poor results around training knowledge, it would have been a good idea to conduct knowledge assessments of Volunteers. However, we were unaware that this would be needed during the planning of the evaluation. Pamphlets were distributed to beneficiaries who received filters later in the study, indicating the program responded to recommendations being made in the initial distribution evaluation. A pamphlet was not available to access pamphlet quality. After survey completion on Saturday, the consultant departed Zimbabwe on Sunday. As the survey data had not been entered or analyzed by the consultant at this time, the consultant did not provide a debriefing meeting with OGB or the Government of Zimbabwe. It was planned that the local consultant (who was not able to be present for Thursday surveys and for part of Friday due to a prior commitment) would present the results of this survey to the Government of Zimbabwe and interested parties. OGB did not indicate interest in disseminating the information, as they stated they were not planning to continue their Biosand Filter program. We did not provide feedback to the beneficiaries as the physical distance to each beneficiary was prohibitive. Overall, it appears there was poor communication between OA/Boston, OA/Zimbabwe, SPWSNet, the Volunteers, and the beneficiaries. Significant information seems to have been lost in this chain, and beneficiary did not have the knowledge to install, maintain, and clean their filters to achieve sustained, effective use. In the future it is recommended that all parties involved in the program align on program goals and develop training materials and training classes before filter distribution.

Page 21 of 38

The consultant feels that the monitoring and evaluation of this program was quite good with adequate metrics collected and significant data obtained. The communication difficulties between the various partners lessened the utility of the extensive monitoring and evaluation conducted, as the program did not reach its potential. However, despite these difficulties, the consultants feels that: 1) there was high sustained use of the Biosand Filters by the beneficiaries (56% of household reporting using the filter, and 56% of reporting users having improved microbiological quality); and, 2) despite the challenges with the survey, the consultant does feel that the survey was representative, as: 1) the survey was randomly implemented; 2) replacements on non-selected households were not left to the discretion of the enumerators; and, 3) households were not called the day before the survey to pre-arrange a visit. A main limitation of this evaluation, however, is the relatively lower response rate to the survey (61 or 100 targeted households located in four days of surveying) due to the dispersed nature of Biosand Filter distribution and difficulty in accessing households.

Page 22 of 38

* Discussion and Recommendations

The purpose of this evaluation was two-fold: 1) to conduct an independent assessment on the efficacy and effectiveness of the Biosand Filters for the cholera response; and, 2) to conduct a process evaluation. Based on the data collected, the specific questions raised in the Terms of Reference are answered below. Recommendations for future programs are included at the end of this section.

6.1 Independent assessment on efficacy and effectiveness of the Biosand Filters for the cholera response
Was the introduction of Biosand Filters an appropriate intervention? Based on the small number of families confirmed to be reached with a Biosand Filter (only 897), and the time period after the cholera in which they were reached, it can not be stated that the Biosand Filters were an appropriate intervention for the program goal of responding to the cholera. However, the filters were well liked and used by the beneficiaries, with 56% of survey respondents having treated water on the day of the unannounced survey visit, which indicates that they are an appropriate intervention for Zimbabwe. Were the Biosand Filters effective in the cholera response? The data suggest that no, the Biosand Filters were not effective in the cholera response. They were delivered with too little training too late to too few people to significantly reduce the risk of cholera. Additionally, 69% of surveyed respondents who received Biosand Filters also received Aquatabs as a cholera response household water treatment option from other organization(s). Aquatabs are less expensive, easier to distribute, simpler to use, and more effective at reducing the cholera bacteria from household stored drinking water than Biosand Filters. However, Aquatabs are a consumable, not a durable, product and thus they only provide protection until they run out. The evidence also suggests that the Biosand Filter program, unlike the Aquatabs distributions, had sustained uptake. Has the quality of water improved with the use of filters for household consumption (cooking, drinking, and sanitation)? The evidence consistently shows that the use of Biosand Filters improves the microbiological quality of household stored water. The percentage of the population that had contaminated water before treatment and uncontaminated after treatment ranged from 63% in the M&E Coordinator evaluation, 43% in the follow-up evaluation, and 56% in the sustained use evaluation. An important note in the survey results is the low percentage of the population that used the filtered water for anything other than cooking and drinking. Given the high flow rates possible with Biosand Filters, it is recommended to encourage users to use the water for additional purposes.

Page 23 of 38

Did communities receiving the filters change their behavior in preventing disease during the program? What has been the impact on prevalence of water-borne diarrheal diseases in these communities? The 897 confirmed filters distributed were delivered to a minimum of 121 communities (about 7.4 filters per community). Given the low percentage coverage per community, it is not anticipated that the filters changed community behavior or significantly reduced the prevalence of water-borne diarrheal disease at the community level. Is there any difference in effectiveness or efficiency between the plastic Biosand Filters compared to concrete ones? The data suggest that yes, there are differences in the effectiveness and efficiency of the plastic and concrete Biosand Filters. It is not clear at this point how much the differences are due to innate differences between the filters styles and how much is due to the fact concrete filters were distributed later than plastic ones, with improved education and training. However, it does appear that concrete filters were less likely to crack. How sustainable is the use of Biosand Filters likely to be? A total of 33 of 59 (56%) households surveyed reported having Biosand Filtered water at the time of the unannounced survey visit conducted in January 2011. This data indicates a willingness of the population to continue using their filter. Sustainability in inhibited by lack of training on operations and maintenance.

6.2 Process evaluation

Coverage: Review the adequacy of the program coverage. What was the quality of beneficiary selection and their participation in the decision making during the program? The goal of the program as reported to the consultant was to reach families affected by cholera in Mudzi District. The population of Mudzi was 130,514 in 2002, or 26,103 families. This program reached 3.4% of the households in Mudzi District. Although the selection criteria for filter distribution was supposed to be families affected by cholera, only 36% of survey respondents reported being cholera-affected. A higher percent of concrete filter beneficiaries, who received their filter later in the program, reported being choleraaffected, indicating that selection criteria may have improved after initial evaluations. Cost effectiveness: Based on available evidence is it possible to draw any qualitative conclusions as to how costeffective Oxfams intervention has been? The sub-contracted amount from OA to SPWSNet for the education component of the program was 40,000 USD. As 897 families were reached, it was thus 45 USD to reach each family with the education to complete the program. On the day of the unannounced survey visit, 56% of respondents had current Biosand Filtered water in the household. In addition, 56% of those with treated water had contaminated water before treatment and uncontaminated after. Thus, about one-third of beneficiaries (31.4% - 56% multiplied by 56%) were using the filter on the day of the unannounced visit to treat their water to WHO drinking water standards. This is equivalent to 281 households being reached effectively with the intervention (897 total reached multiplied by 31.4%), and a cost of 142 USD per household reached with effective treatment.

Page 24 of 38

It is difficult to assess the full cost-effectiveness of this program in full due to the extensive evaluations that occurred and impacted program success substantially, as well as equipment and supplies that were provided by OA to SPWSNet. Efficiency and coordination mechanisms: Review internal and external coordination. Internal coordination concerns the coordination between Oxfam America and the implementing partner as well as coordination between the HQ based specialists and country based program staff. External coordination concerns the coordination with local authorities, community volunteers. The coordination chain between the partners was as follows: 1) OA in Boston designed the program, including a research component that included a health impact study; 2) OA in Zimbabwe coordinated the work with SPWSNet to implement the program; and, 3) SPWSNet implemented the program and interacted with local authorities and community volunteers. There was no direct communication between Boston and SPWSNet. All of the communication chains were effective, except there was a missing link between Boston and the local NGO to ensure that the program was implemented in such a way to meet some of the research goals of the program. Communications, information management and accountability What communication methods and measures were used and were they effective and useful? What were the results? The use of volunteers to communicate with the beneficiaries was an effective mechanism to extend the reach of the NGO, and volunteer trainings should be prioritized in future programs. Increased training was associated with reported use of the filter, indicating that trainings were effective at encouraging sustained use of the filter. Communication to the affected people and their perceptions. The communication to the affected people (the beneficiary population) occurred not by OA, but by SPWSNet. There is a good relationship between the local NGO SPWSNet and OA in Zimbabwe, and the perception of the population for this program was positive. Communication and reporting: ongoing monitoring, communication methods between community members and the volunteers, between partners and the community members including documenting, reporting feedback from community members. The communication chains established for this program were effective, except for what is noted in other sections. Design of program: What conclusions or recommendations can be drawn about the overall design of the program? The analysis should include reflection on the planning, implementation, documentation and monitoring of activities, including partner capacity and Oxfam support. Overall, the consultant stands by the comment made in July 2011 in her first report: this program is attempting to be many things at once a pilot program, a research program, an implementation program, a program targeting the most needy people, a program comparing two types of filters all in an aggressive time

Page 25 of 38

frame. There was not the internal communication or operational capacity available within OA in Boston and Zimbabwe with SPWSNet to complete such a complicated program that would meet all of these goals. That is not to say that the program was a failure, in fact far from that. It is simply to say that some of the program goals fell by the wayside due to the complicated nature of the program. In the future, it is recommended that program design be more considered, and communication between program designers and local NGOs be improved.

6.3 Recommendations
Results from multiple evaluations in Zimbabwe confirm the extensive worldwide data set that the use of Biosand Filters improves the microbiological quality of stored household drinking water. It can be inferred that users of Biosand Filters thus have less diarrheal disease. The question is no longer on microbiological effectiveness of the filters, but on appropriate implementation strategies that result in users effectively using the filters to improve the quality of their stored household water in a cost-effective manner, and whether that is appropriate and cost-effective compared to other interventions such as Aquatabs in the emergency context. The Government of Zimbabwe has noted this, and is actively seeking data on how to encourage effective, sustained use of Biosand Filters in Zimbabwe. Should OA wish to continue with Biosand Filter implementations, it is recommended that OA: 1. Coordinate future programs with the Government of Zimbabwe. The focus of future programs should not be on confirming microbiological effectiveness or health impact (which have been well-verified) but instead on establishing cost-effective distribution strategies that reach a large population with the training necessary to encourage consistent and correct use. 2. Understand the potentially limited role of Biosand Filters in the acute emergency situation. Biosand Filters take time to distribute, train on, and for the schmutzdecke to develop. A chlorine-based product should be distributed in the emergency response time until such time as there is capacity to complete this training and development. This result is consistent with other Biosand Filter evaluations that the consultant has conducted with NGOs distributing Biosand Filters in acute emergency situations, for example after the Haiti earthquake (for UNICEF and Oxfam/UK) and during cholera in the Democratic Republic of Congo (for Oxfam/DRC). 3. Complete significant training with beneficiaries, that teaches how to: a) adequately install the filter, including establishing the correct standing water layer; b) maintain the filter, including where to obtain replacement parts; c) how to clean the filter appropriately and on what schedule; and, d) highlights that filtered water can be used for more than just drinking and cooking. 4. Keep program design and scope realistic and considered. In the future, research plans should be more fully integrated into implementation.

Page 26 of 38

Annex A: Terms of Reference

EVALUATION OF THE OXFAM AMERICA BIOSAND FILTER IMPLEMENTATION PROJECT, ZIMBABWE 1Introduction: This document sets out the Terms of Reference for the final evaluation of the Biosand Filter Implementation program in Zimbabwe. Project implementation spanned the period April 2009 September 2010 with final reports, analysis and other documentation expected to be completed by December 2010. The program was set up in response to a cholera outbreak in north-eastern Zimbabwe but was also viewed as a pilot program to test the introduction and use of the Biosand filter technology in a humanitarian context. The main purpose of this evaluation, as further elaborated below, is to draw lessons from the experience on the viability of the technology, effective management arrangements and minimum conditions necessary for sustainable use of the filters by communities. 2Background: As part of its strategy for water delivery in emergencies, Oxfam America introduced point of use Biosand Filters to 900 households in Mudzi District in Zimbabwe. This location was selected because Oxfam America was already responding to the Cholera epidemic in this part of the country. Cholera is a yearly occurrence in Zimbabwe but the 2008/09 outbreak was worse because it started before the rainy season, affected a large number of people with a high mortality rate, and also occurred in urban environments previously spared from yearly outbreaks. The reason for the high incidence of the disease in Mudzi was largely due to the poor quality of household water in the rural areas due to contaminated sources and the use of unsafe surface water sources. Oxfam America and Oxfam Great Britain jointly responded to the epidemic along with the local partner organization, Single Parents and Widowers Support Network (SPWSN). The initial response came in the form of bore hole repair, distribution of NFIs, health and hygiene education, coordination, and supply of oral rehydration salts. A preliminary assessment was also undertaken on the overall situation and the groundwork was laid for the development of a diarrhea early warning system and the introduction of the Biosand filters. As Oxfam moved forward with the programming to address the acute needs caused by the outbreak, it considered ways of halting the annual reoccurrence of cholera in Zimbabwe. The Harvard Humanitarian Initiative and Oxfam worked with SPWSN to put in place a regionally run system that would help with early detection and response. The system was adapted by SPWSN and other local stakeholders to better suit community needs, resulting in Village Cholera Committees. To address the medium to long-term needs, Oxfam America implemented a pilot program to utilize household Biosand Filters to 950 families in Mudzi District. Oxfam America purchased 450 pre-fabricated plastic filters from Hydraid and commissioned the manufacture of 500 locally made concrete filters in Zimbabwe. Working with SPWSN, the filters were distributed to families in stages, accompanied by filter use and maintenance training coupled with public health and hygiene education. Given that the technology was new for Oxfam, SPWSN as well as for the communities, adjustments and improvements were regularly needed during program implementation. Monitoring of the program
Page 27 of 38

was initially focused on filter distribution activities and logistics. Training of households in the use of filters and monitoring of water quality or hygiene practices were also only improved during the course of the program. To compensate for the lack of consistent monitoring data, the program undertook an extensive results monitoring exercise in September 2010. This exercise has made a closer examination of the proper knowledge/utilization of Biosand Filters, hygiene practices and household water quality. Data from the results monitoring exercise is still being analyzed at the time of writing this TOR. In addition, the program is planning to prepare a How-To toolkit that will document the experience of the Biosand Filter program. The Toolkit will serve the purpose of a technical, user and monitoring manual, list useful training and IEC materials prepared by this program, make recommendations on choice of monitoring indicators, present monitoring templates or formats to be used for various aspects of the program, and include guidance (based on the pilot program experience) on the minimum frequency and type of data collection needed to manage such a program. The toolkit will, in turn, benefit from the program evaluation report and will be finalized upon completion of all other program requirements. 3Evaluation of the use of Biosand Filters 3.1. Rationale, Purpose and Scope of the Evaluation Rationale: Oxfam is committed to the highest standards of learning and accountability. Accountability is our obligation to demonstrate, in a transparent way, that our actions, decisions and their results direct or indirect are in compliance with agreed norms, protocols, values and standards with respect to prudent and honest management of resources, knowledge and relationships (with people, partners, allies, donors, etc.), and allow/enable an independent assessment of our performance and results. We are accountable both internally to ourselves to draw out lessons for future improvement, and externally to our stakeholders and especially to the people we serve. According to OAs Humanitarian Monitoring, Evaluation and Learning (MEL) policy regarding programs implemented by partners, OA will evaluate select programs for accountability and learning purposes. Oxfam America is building its capacity in the water and sanitation sector in alignment with Oxfam Internationals strategy under the Single Management Structure. WASH interventions in humanitarian programs are relatively new within OA. The use of Biosand Filters for provision of safe drinking water to cholera-affected communities was piloted in Zimbabwe with the intention of learning from the experience for possible replication in other contexts. This evaluation will allow Oxfam to capture lessons and learn from its new experience as well as to demonstrate the efficacy of the Biosand Filter use during emergencies. 3.2. Purpose: The purpose of this evaluation is two-fold: 1) to provide an independent assessment on:Efficacy and effectiveness of the Biosand Filters for the cholera response: Was the introduction of Biosand Filters an appropriate intervention? Were the Biosand Filters effective in the cholera response? Has the quality of water improved with the use of filters for household consumption (cooking, drinking, and sanitation)? Did communities receiving the filters change their behavior in preventing disease during the program? What has been the impact on prevalence of water-borne diarrheal diseases in these communities? Is there any difference in effectiveness or efficiency between the plastic Biosand Filters compared to concrete ones? How sustainable is the use of Biosand Filters likely to be?

Page 28 of 38

2) Process evaluation (for learning purposes) Coverage: Review the adequacy of the program coverage. What was the quality of beneficiary selection and their participation in the decision making during the program? Cost effectiveness: Based on available evidence is it possible to draw any qualitative conclusions as to how cost-effective Oxfams intervention has been? Efficiency and coordination mechanisms: Review internal and external coordination. Internal coordination concerns the coordination between Oxfam America and the implementing partner as well as coordination between the HQ based specialists and country based program staff. External coordination concerns the coordination with local authorities, community volunteers. Communications, information management and accountability What communication methods and measures were used and were they effective and useful? What were the results? o Communication to the affected people and their perceptions o Communication and reporting: ongoing monitoring, communication methods between community members and the volunteers, between partners and the community members including documenting, reporting feedback from community members Design of program: What conclusions or recommendations can be drawn about the overall design of the program? The analysis should include reflection on the planning, implementation, documentation and monitoring of activities, including partner capacity and Oxfam support. 4- Intended use and evaluation audience This evaluation is expected to serve an important learning function for the design and management of similar programs in other contexts. It is also expected to give Oxfam partner, SPWSN, valuable feedback on their role, management, staffing as well as M&E functions. This will serve as beneficial input for improving their organizational capacity to undertake similar or different humanitarian response programs in the future. External audience: In addition to sharing evaluation results with OAs implementing partner, findings will also be circulated amongst OGB and other NGOs, local authorities and with communities using the filters. A modified version of the evaluation report may be shared with donors as relevant. Oxfam America will develop a separate advocacy or lessons sharing plan with relevant stakeholders. Internal audience: The evaluation results will be shared across OA along with the how-to tool kit being developed by the program where key lessons learned and best practices will be compiled to complement tools already developed from other literature and experiences with the use of Biosand Filters. 5- Evaluation Principles and Methods The evaluation will be guided by the following principles: Participation: Evaluation must incorporate the views and perceptions of the affected people and program participants, duly contextualized in the proper perspective. Participants need to be allowed to shape the very design/intent of the evaluation. Rigorous: Evaluation must be methodologically rigorous and stand the scrutiny of evaluators and practitioners detailing the evidence base, and wherever possible, triangulating the information. Credible: the evaluation must be conducted impartially Ethical: Must respect the dignity and privacy of the people and informants. It must also adhere to internationally recognized standards and norms for humanitarian response evaluation.

Page 29 of 38

6- Methodology: Documentation review. Observation of key events and processes, Photographs and maps. Random water quality testing. Interviews with stakeholders including staff. Staff includes all relevant OA staff based in Boston and in Zimbabwe. Non-staff stakeholders that will be important to interview include local authorities, community leaders, leaders and members of womens groups or users of Biosand Filters. Focus group discussions separately with women, children, men, etc. Gathering/documenting people perceptions using key informants, personal open-ended interviews and discussion. 7- Management and Supervision of the evaluation The evaluation will take place in January 2011. The process will be facilitated in Zimbabwe by the OA Humanitarian Program Coordinator with oversight from the Deputy Director of Humanitarian Programs and the Humanitarian MEL Manager in HRD, Boston. 8- Time Frame and budget The evaluation mission will be conducted in January 2011 and will involve: - documentation review 1 day - interviews with Oxfam staff/management 1 day - field investigation and interviews 5 days - debriefing Oxfam/partner staff in Harare & Boston 1 day - analysis and report writing 2 days - travel 3 days The first draft of the report should be submitted to Oxfam by February 5 2011, whereas the final report is expected to be completed within 5 days of receipt of any consolidated comments from Oxfam. Budget: Details related to the consultant fees and expenses will be stipulated in consultant contracts. 9- Deliverables Evaluation Report The final report should be no more than 25 pages, clearly written in English, using font size of 11 points, single spaced. An executive summary of no more than three pages should summarize the major insights, conclusions, and recommendations of the study. As noted before, since the evaluation will serve a capacity building or learning purpose, the report should draw on lessons from the program and be forward looking in its recommendations. The report should clearly identify the purpose of the evaluation, what was evaluated, how the evaluation was conducted, the data considered, the conclusions drawn and recommendations made and lessons identified. The report should explain how each conclusion derives from the findings, and what their limitations are. Recommendations should be linked to conclusions.

Page 30 of 38

10- Evaluation Team The evaluation team will be composed of an international consultant (Team Leader) and one national consultant. The Team Leader will have extensive expertise in the evaluation of humanitarian programs especially in emergency WASH assistance. The national consultant will also have an evaluation and WASH background and will be responsible for contextualizing the evaluation methodology and approach to local social and cultural practices. The national consultant will play a central role in organizing and conducting individual and group discussions with official and community representatives. The Team Leader will have final responsibility for the design of the evaluation methodology and report writing.

Page 31 of 38

Annex B: Itinerary and survey respondents

The itinerary of the evaluation was as follows: Monday, January 17th: Arrive into Harare Meet with Ransam Mariga, OA/Zimbabwe Meet with Fungai Makoni, Local Consultant Meet with Fungai Makoni, MOW Staff, MOH Staff Meet with Oxfam/Great Britian/Zimbabwe Depart for Mudzi Tuesday, January 18th: Wednesday, January 19th: Train enumerators on survey Meet with government officials to obtain permission First day of survey sampling Water Quality testing Thursday, January 20th: Second day of survey sampling Water Quality testing Friday, January 21st: Third day of survey sampling Water Quality testing Saturday, January 22nd: Fourth day of survey sampling Water Quality testing Return to Harare Sunday, January 23rd: Depart Zimbabwe

This investigation was not formally reviewed by an Internal Review Board for adherence to ethical standards, as: 1) this was not required by Oxfam; and, 2) this type of investigation is generally considered exempt from review because it is program evaluation and not research. However, the author conducts all of her investigations in accordance with generally accepted ethical research standards and Internal Review Board procedures. As such, the informed consent paragraph read to each potential survey participant before the survey began (see Annex C), reads: No one except the researcher will know that it was you who provided these answers. In addition, the researcher, in accordance with standard practice, stripped all identifiers from household survey data on data entry, and deleted the key linking household survey number to household name after analysis was complete. Thus, the names of individual survey respondents cannot be provided to OA. However, the line list of all filter beneficiaries, collated by OA/Zimbabwe and SPWSNet staff, is provided with this report.

Page 32 of 38

Annex C: Household questionnaire

Page 33 of 38

Page 34 of 38

Page 35 of 38

Page 36 of 38

Annex D: Water quality indicators

The water quality indicators tested in this evaluation included the bacteria E. coli (by the consultant, presented in Figure 5) and the indicator organism Total Coliform (by the consultant). For completeness, E. coli and Total Coliform results are presented in tabular form in this annex. Table 4: Water quality data
Untreated Stored Household Water Total Coliform CFU/100 mL >400 20 96 >400 >400 256 >400 8 >400 >400 88 >400 149 >400 136 >400 >400 >400 150 >400 74 116 >400 244 >400 >400 334 >400 >400 0 >400 >400 >400 220 154 Untreated Stored Household Water E. coli CFU/100 mL 192 0 16 40 130 16 18 2 84 0 8 8 2 0 6 282 200 100 10 0 64 16 >400 108 180 >400 2 70 42 220 4 0 16 >400 170 38 4 74 2 Treated Stored Household Water Total Coliform CFU/100 mL 0 47 20 14 30 60 0 0 >400 0 >400 74 >400 >400 >400 30 >400 >400 >400 Teated Stored Household Water E. coli CFU/100 mL 0 44 0 0 0 0 0 0 10 0 0 0 >400 78 4 0 350 48 0

Page 37 of 38