Final Evaluation Report Project 6676 Using the Organoleptic Analysis of Fine Chocolate to Improve Market Access of Small-Scale

Cocoa Growers in Ecuador Project

BACKGROUND
The Ecuadorian Non-Governmental Organization, Conservacion y Desarrollo (CyD), along with its partners, The National Autonomous Institute of Agriculture Research (INIAP) and 11 Growers Associations (GAs), undertook a project that originated from a desire of many cacao growers to be able to taste and analyze their own chocolate, not just rely on client reports, in order to improve the quality and better market differentiated product. To that end, sophisticated equipment that can produce relatively small batches (30 kilos) of chocolate was purchased for INIAPs Experimental Station in Pichilingue, Ecuador, so that cacao growers associations would always have access to chocolate making facilities. An aim of the project was also to showcase the large diversity of cacao flavors that exist in Ecuador, due to genetic as well as environmental factors, something that is often referred to as terroir. Thus, in addition to the making of 11,000 chocolate bars, the project also worked closely with the associations to produce maps of their territory, reconstruct oral histories, and create a marketing identity with their own slogan and logo that reflects their identity. The result is a set of 11 bars with unique and delicious tastes and aromas, and farmers who are aware and empowered by their local identity. The bars are used as samples for fine chocolate makers and consumers around the world to know the diversity of Ecuadorian cacao and work directly with the growers associations that interest them. This has the advantage of created differentiated markets that depend on a particular geography and community, to provide a specific organoleptic 1 profile and thus buffers these associations from the price and volume fluctuations associated with commodity markets. Moreover, the project intended to improve the GAs awareness between cocoa quality and chocolate taste through training sessions on production and post-production issues, as well as traceability2.

1 2

Organoleptic refers to the qualities of a substance such as taste, color, odor and feel. Traceability refers to the completeness of the information about every step in the process chain.

Project Objectives 1. To equip The National Autonomous Institute of Agriculture Research (INIAP) with small batch chocolate making machinery so they can provide growers associations with a low cost service of processing chocolate samples that associations can use as samples 2. To train and empower farmers from 11 growers associations to produce the high quality, differentiated cocoa. 3. To educate cocoa buyers on the diversity of cocoa flavors within Ecuador and encourage more direct buying relationships.

Project Theory of Change

Farmer training sessions* Install Chocolate lab at INIAP

Farmer knowledge on connection between post harvest and quality

Farmer knowledge on selling and communicati ons and market ing increased

Farmer knowledge on sensorial properties of chocolate for their region increases

Video, blog, media coverage

Local knowledge on geography, history and story collected and synthesized in marketing materials (map, boxes)

Chocolate bars produced

INIAP has capacity to make chocolate

Analyze sensorial properties of cacao from diff. regions

Information dissemination

Increase Quality

Increased Price

Increase Demand

Chocolate makers aware of sensorial properties of Ecuadorian cacao

Long term sustainability of project objectives

Key: Activities * Outputs Outcomes

*Farmer training sessions refer to 3 training sessions that were held at the main collection center of each growers associations (GA), they usually lasted 4 hours and included lunch. The trainers included 2-4 members of the CyD-INIAP training team. The topics of the 3 sessions were the following (please see video and photos on all events at: http://cacaodesabor.blogspot.com): Training session #1 (July 2009): Introduction to project, Organoleptic analysis, quality, postharvest, Fermentation testing, traceability, Evaluation, baseline interviews Training session #2 (May 2010): Feedback from baseline, Developing the idea of terroir (participative mapping, collecting the history of the association, designing logo and motto), tasting bad and good quality cocoa paste, taping video. Training session #3 (May 2011): Show video, Chocolate tasting, Training on communication and marketing for fairs, preparation for fair in Pichilingue. Cocoa Fair Pichilingue (July 2011): Tasting of all chocolates, tour of chocolate making lab, awarding of prizes, distribution of bars and maps, endline survey. Chocolate fair in Quito (May 2011) and Guayaquil (Nov. 2011): providing tastings to consumers and buyers. Seven associations participated in the fair in May, 3 in the fair in November. Chocolate making at INIAP (August 2011): Two representatives from each association were invited to INIAP for 3 days to participate in making their own associations chocolate bars. 10 associations participated Evaluation questions (based on outputs and outcomes in Theory of Change3): 1.) Has farmer knowledge on the sensorial properties of chocolate for their region changed? 2.) Has farmer knowledge changed in respects to the connection between post-harvest practices and quality? 3.) Are chocolate makers more aware of the sensorial diversity of Ecuadorian cocoa? 4.) Has cocoa quality increased? 5.) Has price increased? 6.) Has demand increased? Specific tasks of the final evaluation To analyse monitoring and evaluation data collected during the project (baseline and endline survey, workshop evaluations, fermentation tests, follow up interviews with leaders of the associations, cocoa buyers and other key stakeholders)
4

See http://www.theoryofchange.org/ for more information on this methodology.

To assess and reflect on change and effectiveness of project interventions as well as sustainability. To provide learning and insight for further interventions

METHODOLOGY
1. Baseline survey Our design was to have each Association pick a representative (in terms of gender, age and geography) group of 20 participants who we would track through the three training sessions. Unfortunately, these 20 participants did not all participate in the first training session so we ended up picking a convenience sample based on the participants who showed up if there were more than 20. During the baseline phase there were only 10 growers associations. **All baseline data was returned to the associations both during training session #2 and in printed form** 2. Training session evaluations At the end of each training session we asked the group to dived into sub-groups with 5-6 people in each group. There they would reflect on 3 questions as a small group for 20 minutes and then write their responses on large pieces of paper. At the end of the activity the group would discuss their observations. The responses that were written on the paper were kept as distinct data points. For training session #1 there were 38 subgroups, 18 for #2 and 48 for #3. Questions for training session #1: 1. What do you expect from this new project? 2. What and how can you as a farmer support this project so it achieves its goals? 3. Name 3 or more things that you learned in the workshop today? Questions for training sessions #2 and #3: 1. Name 3 or more things that you remember/ learned from the previous workshop 2. Name 3 things that you learned in the workshop today 3. How can you apply what you learned in the workshop today?

3. Endline surveys The original 20 who were interviewed in each association did not, for the most part, participate in the other 2 trainings and attend the fair. Thus, during the fair the at INIAP in July 2011, we decided to take advantage of the presence of approximately 30 farmers per association to
5

randomly select participants to take the endline survey. We did this by giving the first 10 farmers in line for each association a sticker to be interviewed, thus we expected to interview 110 farmers. In the end 58 farmers were interviewed from the 11 associations: FEDECADE (7) UCOCS (8) SAN CARLOS (2) MISS ECUADOR (5) ELOY ALFARO (10) COCPE (1) BUENA SUERTE (7) APROCA (10) APOV (3) FORTALEZA DEL VALLE (2) COAGRICACE (3) Of the 58 interviewed, 33 had participated in Training 1, 29 had participated in training 2 and 28 had participated in training 3. The surveys were conducted by 3 interviewers, all who had been trained by the head evaluator on how to conduct the survey. The interviews lasted approximately 15 minutes. 4. Follow up email with those who received samples: We compiled a list of important chocolate makers worldwide who would be interested in receiving the sample. The list was compiled by talking to our current network of cocoa and chocolate experts, and suplementing that list with members of the World Cocoa Foundation. From this exercise, 50 key international actors and buyers were determined and they were sent samples and the map between August and September 2011. Follow up emails were sent to all of the recipients in October 2011 asking if they received the samples, what general impressions they had and if they were interested in working with a particular association, 21 responded to the email. 53 samples were also sent out to key actors in Ecuador, mostly to government and nongovernmental agencies, but also to exporters and chocolate makers. This list was compiled through the CyD database that we have amassed over 10 years working in the cocoa sector of Ecuador. We received feedback from two of the national chocolate makers (Pacari and Confiteca). 30 samples where sent through ProEcuador, the Ecuadorian Governments export promotion organization, to international chocolate fairs in Brazil and England, we are still waiting the final report on impact.
6

5. Follow up email/ phone/ in person interview with representative of each association: Each association was asked the following two questions in an email, telephone call or in person conversation in November 2011: 1. What are the final reflections of the association on their participation in the World Bank project (3 workshops, the fairs in Quito and Pichilingue, production of bars etc.)? Has it had any effect on the association in terms of knowledge/practices/ sales? Are you going to do any special follow-up with the results from the project? 2. What did you do/ are going to do with the chocolate bars? 3. Did you strengthen or develop commercial linkages as a result of the project? Did you obtain new clients? 6 of the 11 associations responded to the questions.

FINDINGS
This section is presented thematically, where findings from the methods listed above are presented in their respective thematic area. There are 6 thematic areas of findings: Participation Cocoa quality improvement Organoleptic analysis Organizational capacity and communication Increase in price and income through better market linkages Scaling up and sustainability

Participation
Below a table with total number of members in each association and those that participated in each training session and the fair in Pichilingue: Association Forteleza del Valle COCPE Buena Suerte FEDECAD E UCOCS San Carlos Miss Ecuador CORIAGRI CACE APOV Eloy Alfaro APROCA # of active Training members #1 668 27 490 131 470 250 508 36 450 200 1000 308 37 30 17 19 Training canceled Training canceled Didnt participat e 30 72 56 Training #2 19 33 50 25 10 14 23 13 31 44 8 Training #3 7 22 25 15 10 30 Training canceled 17 22 25 17 7 9 13 Recurrent participants 6 7 9 3 6 2 INIAP % of fair total* 39 3.4 40 15 28 5 8 29 45 48 30 6 27 3.8 7 3 43 4 16 5 9

*average number of participants training sessions #1, #2, #3 and INIAP fair, represented as a percentage of the total number of members of the association.

100
% de miembros por asociacion

90 80 70 60 50 40 30 20 10 0 Asociaciones de Cacao

APOV COCPE ELOY ALFARO APROCA UCOCS FEDECADE BUENA SUERTE FORTALEZA DEL VALLE SAN CARLOS MISS ECUADOR

100
% de miembros por asociacion

90 80 70 60 50 40 30 20 10 0 Asociaciones de Cacao

UCOCS FEDECADE MISS ECUADOR BUENA SUERTE SAN CARLOS ELOY ALFARO COCPE FORTALEZA DEL VALLE APROCA APOV

100
% de miembros por asociacion

90 80 70 60 50 40 30 20 10 0 Asociaciones de Cacao

FEDECADE UCOCS APOV MISS ECUADOR COCPE APROCA SAN CARLOS BUENA SUERTE FORTALEZA DEL VALLE ELOY ALFARO

In the endline survery 96% of farmers can correctly differentiate CCN51 from cocoa nacional. 4 In the endline survey when participants were asked to rank the 3 most important things they learned from the project, 19% mentioned that they felt that the most important learning from the project was how to be better farmers; 7% reported the most important thing was that it allowed them to improve quality and post-harvest and 4% said the most important thing was it helped them to maintain Nacional varieties. In the workshop evaluation for training #3, 63% of the participants stated that they will apply what they learned in the workshop about improving quality. Specifically they mentioned: Delivering better quality cocoa to the associations collection center That they realize better cocoa equals better chocolate That they will improve fermentation That they will use tastings to improve quality Conserve biodiversity (15%) At the end of training session #3 when subgroups were asked what were the 3 most important things they learned during the previous workshops, 58% mentioned physical analysis to determine fermentation levels, and 48% mentioned traceability.

The representative of Buena Suerte wrote: [participating in the project] was a very good experience because the farmers could participate and be trained in what is traceability and cut tests and the sensorial taste of cocoa, all of this allowed them to have greater knowledge and elevate the self-esteem of the producers

In Ecuador there are two main types of cocoa, Nacional which refers to thousands of local landraces that are prized for the floral aroma, and CCN51, which is an improved clonal variety that does not have floral qualities but does have higher yields. This project was focused only on working with Nacional varieties, because our hypothesis is that these are the ones that will bring long term economic gains to Ecuadorian farmers by having a differentiated premium product.

11

100
% de miembros por asociacion

90 80 70 60 50 40 30 20 10 0 Asociaciones de Cacao

MISS ECUADOR ELOY ALFARO COCPE UCOCS SAN CARLOS BUENA SUERTE APROCA APOV FORTALEZA DEL VALLE FEDECADE

The representative of COCPE reports that while most of the participants appreciated the workshops, the new president didnt and was skeptical of the project, but he only attended one workshop where they tasted bitter cocoa, which he didnt like

100
% de miembros por asociacion

90 80 70 60 50 40 30 20 10 0 Asociaciones de Cacao

SANCARLOS MISS ECUADOR ELOY ALFARO FORTALEZA DEL VALLE UCOCS COCPE APOV FEDECADE BUENA SUERTE APROCA

100
% de miembros por asociacion

90 80 70 60 50 40 30 20 10 0 Asociaciones de Cacao

COCPE ELOY ALFARO SAN CARLOS APOV MISS ECUADOR FEDECADE FORTALEZA DEL VALLE APROCA UCOCS BUENA SUERTE

The representative of APROCA reports that the workshops were important in motivating farmers, especially tasting their own chocolate

Organizational capacity/ communication

91% of farmers in the endline survey reported that the project had strengthened their organization (9% said it was the same), 11% specifically mentioned that they had improved the association. In the small group evaluation of training #3, 100% of the participants stated that they will apply what they learned in the workshops to improve their communication ability and organization. Specifically they mentioned: They have improved their capacities as an association They will share with partners/ clients They will distribute the technical information sheets that were provided at training sessions.
The leader of CORIAGRICACE reports: The assistance at the fairs in Quito and INIAP represented important spaces where for the first time we could be known as CORIAGRICACE at the national level

54% mentioned the creation of the maps that allow them to see the zone of influence of the project as well as where the members of the association are located as an important learning from past workshops. 6% mentioned knowing the location of their association in comparison to the rest of the associations as important. 10% mentioned the importance of learning to work as a team.

Increase in price and income through better market linkages

In the endline 100% of the farmers believe that the results of this project will give them more power to negotiate with this clients due to: More knowledge about the product and the market (33%) Possibilities of passing over an intermediary (21%) Better quality/ post-harvest (21%) The associations chocolate bar (9%)
A leader from APROCA reported in his follow-up interview that the bars were a great marketing tool. He had recently returned from a business trip to Italy and Spain where he distributed the bars to potential cocoa buyers and chocolate makers. He felt proud distributing such a beautiful package and the clients loved it.

14

A representative from Aproca showing the bars to a buyer in Italy

Ecom Trading Bouga CacaO

Hasnt decided Hasnt decided

Lake Champlain Chocolates Theo chocolate Amano chocolate Friis-Holm

Hasnt decided Hasnt decided Interested in who is Fair Trade certified Hasnt decided

made some initial inquiries about samples. In September said he would like to be in touch with some associations, but still hasnt indicated which. Said in September that he will do a formal tasting of the bars and if he likes them he will ask for samples of 100% chocolate. He also tried to put an association he works with in touch with INIAP to make bars for them. There has been some communication but nothing has materialized. In October said they would do a formal tasting and were interested in doing small batch Ecuadorian chocolate In September was going to do a formal tasting and get back to us. Wants to visit potential farms Wants to do formal tasting in October

The project also participated in 2 Ecuadorian Chocolate fairs, organized a fair in Ecuador and Participated in 2 international fairs. Fair Project participants (from GAs) 6 GAs Non-project attendees Comments

Salon de Chocolate, Quito, May 2011

It is estimated that at least 500 people stopped by our stand including international buyers, domestic chocolate makers and consumers Over 500 other participants including other farmer associations, buyers and donors

The project paid for the GAs participation in the fair (travel and lodging expenses)

Feria de aromas y Sabores de Chocolate Ecuatoriano, organized and financed by the project, July 2011 at INIAP Pichilingue Brazil Chocolate Fair,

500 participants from the 11 GAs

The project paid for the GAs participation in the fair (travel and lodging expenses). Wide media coverage

None, sent

Thousands of consumers Still waiting for report

16

August 2011

samples and maps through ProEcuador None, sent samples and maps through ProEcuador 3 associations

and industry leaders

from ProEcuador, they represented us without cost.

London Chocolate Week, September, 2011

Thousands of consumers Still waiting for report and industry leaders from ProEcuador, they represented us without cost. Estimated that 250 We were not able to consumers passed by the finance the participation stand, including small of GAs. domestic chocolate makers

Feria de Aromas de Chocolate y Caf, Guayaquil, November 2011

Four associations have reported attending other regional fairs and selling the chocolate bars for $1 each (profit margin roughly $0.60/ bar).

Scaling up and sustainability

The association Miss Ecuador reports that they have interested the municipal government of Sucumbios in the project, and they are interested in financing the purchase of their own chocolate making lab for $100,000. They first want to make more bars with INIAP and solidify potential markets. The association COCPE used the fair at INIAP as an opportunity to interest their local government by inviting them along. As a result, in August, the municipality sponsored a chocolate fair in Quininde that was attended by almost 500 people. COCPE was able to sell some of their bars, but people find the price too high. Use of bars Association San Carlos Fedecade COCPE APROCA UCOCS Has been distributing them at cocoa fairs in Tena, Santo Domingo and Sucumbios They all went to the farmers Collection center, samples, events, some sales Ordered and paid for an additional 1000 bars from INIAP. They have sold 100 bars so far, mostly at fairs. Also have used them as samples for the industry. Using them to promote the association to governmental institutions who
17

CORIAGRICACE Buena Suerte Miss Ecuador

are active in investing in the area Has approached INIAP to make more bars to sell for Christmas Distributed some among farmers so they could taste the chocolate, others were used at fairs for samples and they still have some. They are looking at the possibility of making more for the local market. Has been selling and distributing them at fairs, bought 1000 more from INIAP

The project was able to put together a scientific oral presentation on the organoleptic diversity of Ecuadorian chocolate and present it at the VIII International Symposium of Genetic Resources for Latin America and the Carribean to be held in Quito in November 2011. It is hoped that this event, as well as the public availability of main project materials at http://cacaodesabor.blogspot.com will provide public goods that can be used by other organization to adapt and sustain the interventions developed The head of the cocoa quality lab during the project. There have been almost 1500 page visits to the at INIAP, Juan Carlos Jimenez blog in the past year. reports that he was recently in the Interviews with the principal researchers involved with the project from INIAP reveal that they feel significant change in capacity, knowledge, skills and attitude due to the project. Changes include:
field and was recognized by a farmer who saw him on TV and asked if he could help him make chocolate. The technician feels there is more trust and interest on the part of farmers to work with INIAP. He know offers farmers to process 1 kilo of chocolate fee out of friendship/

They know how the facilities and knowledge of how to produce fine chocolate. They have been trained in innovative methods for working with farmers including learning by doing. They feel more camaraderie and mutual respect with farmers and associations who now approach them with their concerns and needs.

CONCLUSIONS/ Lessons Learned/ Recommendations

Participation The participation for many of the associations was around 5% of associates, or between 10-30 people per training. We estimated that there would be around 100 farmers participating in at least one of the training events during the project cycle. We expected such high numbers because the trainings were particularly innovative, hands-on, allowed for a high degree of participation and included the distribution of tangible and desired products, like laminated technical sheets, maps and chocolates. However, our expectation for 100 participants per association was high
18

compared to previous experience. This experience has made us question the general strength of the producers associations and how much buy-in there is from their members. We would like to investigate the status and role of growers associations in a future project. It is hard to assess how much producers valued the workshops. While the evidence shows that there were important changes in knowledge due to the workshops, and there are many testimonies that they were worthwhile, attendance did not always reflect that. We also received a few testimonials that the workshops were not that helpful. The production of the chocolate bars, was, however, unequivocally appreciated. It is hard to come to a conclusion about the workshops, without them we would never have had the local knowledge and buy-in to create appropriate marketing materials and the organoleptic map. However, in the future, it might be worthwhile to work with other community partners to get higher attendance from farmers than we get solely working through the association. We also worked with a large number of associations, which was very difficult to coordinate logistically. It might have been more reasonable in retrospect to work with only 5 associations more profoundly to test out the innovation. There was also considerable time between trainings, often 12 months. It is unclear what impact this had on participation, but clearly, there was not a lot of momentum from training to training, however the levels of recall of previous training learning were high. Often when trainings are too close together participation also suffers. Evaluation question #1: Has farmer knowledge on the sensorial properties of chocolate for their region changed? There was a dramatic change in farmers knowledge of fine chocolate in general and the chocolate from their association specifically due to the project. Before the project, farmers largely did not have the vocabulary to undertake sensorial analysis of chocolate, and by the end, 70% did. Likewise, roughly 15% knew the taste profile of the association before the intervention and 50% did after. We hypothesize that the end number would have been much higher if we had had the organoleptic maps ready by the 3rd training session so we could have specifically focused on what the experts believe to be their profile in order to guide the tasting more. Evaluation question #2: Has farmer knowledge changed in respects to the connection between post-harvest practices and quality? The baseline reveals that farmers already knew that fermentation was important for producing high quality cocoa. The barrier they probably face is that high quality cocoa is not sufficiently valued in the market.
19

The other important aspect of fermentation is that even when farmers and associations knew that fermentation was important, they may not have known how to produce good results and monitor levels of fermentation. As a result of the project, knowledge of how to perform a cut test to monitor levels of fermentation went from roughly 40% to 90%.

Evaluation question #3: Are chocolate makers more aware of the sensorial diversity of Ecuadorian cocoa? None of the targeted chocolate makers came out and stated that they were surprised by the diversity of flavors that come out of Ecuador. However, a few key ones have still not done the full tasting. Also, many buyers were more focused on either the less than optimal quality of some of the chocolate in terms of tempering and refining or else by the beautiful packaging. While none commented explicitly on the diversity of flavors, many commented on flavors that were unusual or new to them. Moreover, during the public fairs in Ecuador that the project participated in, there were many comments by consumers and other stakeholders on the diversity of flavors. Evaluation question #4: Has cocoa quality increased? The results of the cut test would suggest that there was not progress between the 1 st and 2nd training session, but it is possible that there was a big jump in quality between before the project started and after the 1st training session. We do not have a baseline of quality before the intervention started. Associations report more awareness and use of good post-harvest practices. Evaluation question #5: Has price increased? The findings show that the answer to this question is not yet this is likely to be a longer term change, as these commercial relationships take a long time to establish, and often include introducing a new origin in the market by businesses that are often small and do not have a lot of capital for experimentation. However, the feedback from buyers is encouraging and suggests that change will occur. More importantly, the evaluation findings reveal that the associations and farmers feel more able to negotiate and attract new business, the fairs in particular, provided them with an important contact with final consumers and buyers. One price issue that this project hoped to address was that farmers do not receive a sufficient premium to make up for yield and time losses for well fermented Nacional cocoa verse the CCN51 variety. During this project there was no evidence of progress being made to widen the price differential between these two quality levels, but this might have been exacerbated by

20

political problems in the Ivory Coast, a major cocoa exporter, that caused global cocoa prices to rise across the board in 2010. Evaluation question #6: Has demand increased? We can answer this question in a similar fashion to evaluation question #5 in terms of the export market we envisioned at the beginning of the project. However, an unexpected local market emerged during the project, where over half of the associations have had enough success selling bars to order more with their own funds from INIAP. The project itself has sold almost 100 boxes for $20 each, a $15 profit margin per box, with almost no effort. There is an interesting demand for these nicely packaged and marketed bars in the local market that should be pursued further. Empowerment Another impact of the project that was not entirely envisioned at the beginning of the project, was the increase in feelings of farmer empowerment. This emerged during the evaluation as farmers and leaders consistently talked about the pride they felt in their chocolate bars. We think this arose from many of the unplanned activities of the project such as participative mapping, focus groups to formulate the narrative and logo for each association, the use of video that featured the farmers prominently and the participation of farmers in national cocoa fairs. Other activities that contributed to the sense of empowerment were conceived as part of the project design from the beginning, such as the emphasis on using well designed materials for all the project products (technical sheets, map, banners, chocolate boxes) to show the farmers that we value them as recipients and co-creators of these products, as well as the heavy emphasis on press coverage that is motivating for the farmers.

Sustainability Since this project represents an innovation, there are no plans to continue it in its current configuration, rather we hope that specific techniques and interventions will continue into the future. The include: The Ministry of agriculture is interested in working with 50 more farmer associations to produce bars and make organoleptic maps. Participating and non-participating farmers associations are paying INIAP to make more bars to sell in the local market. INIAP has invested in paying the salary of a full time lab technician to make chocolates, they are also investing in expanding the chocolate lab further
21

TCHO chocolate company and Equal Exchange are interested in working with us in 2013 to strengthen some of the farmers associations that worked with the project. Both INIAP and the associations have increased capacity and knowledge from this project to continue innovating on how to increase small-scale farmers revenues.

22