Independent evaluation of Paris Pact Phase III

A Partnership to counter the trafficking and consumption of opiates originating in Afghanistan

Reconvened Policy Consultative Group Meeting Vienna, 3 September 2012

Mr Johannes Chudoba (Evaluation Team Leader) Mr Marcio Carvalho (Evaluation Specialist)

Overview
Introduction Methodology Background / Context Evaluation Findings Conclusions Recommendations

Introduction
Independent two-month evaluation Two evaluators Emphasis on improving Phase IV
Focus on effectiveness & engagement with stakeholders

Scope:
Paris Pact Initiative (PPI) UNODC GLO K31

Methodology
Comprehensive qualitative and quantitative approach:
50+ documents & ADAM reviewed, 123 responses on websurvey, 64 interviews and 23 questionnaires & 1 field visit. 44 partner countries, 14 partner organisations and UNODC.

Triangulation Participatory process Criteria:

Relevance Effectiveness Sustainability and Impact Partnerships and Cooperation Efficiency of UNODC Project GLO K31

Methodology
Limitations and Constraints
PPI is not a development project Time:
For the PPI/Project to show results in a 24-month framework For conducting the evaluation

Data environment Impact

Background / Context
Paris Conference Paris Pact Phase I: Partnership / consultative framework Moscow Conference Paris Pact Phase II: Strengthened analytical capacity in priority locations for evidence based policies Paris Pact Phase III: Refining the tools: more focused evidence for action Vienna Conference

Global Afghan Opium Trade - A Threat Assessment, 2011

Global potential opium production, 1997-2011 (tons)

WDR 2012

Prevalence of the use of opioids (heroin, opium and non-medical use of synthetic opioids) in 2010

WDR 2012

UNODC Project GLO K31

1. Consultative Mechanism 2. National Strategic Analysts (NSAs) 3. Automated Donor Assistance Mechanism (ADAM)

Budget: USD 2,972,290

Duration: 24 months from June 2010

Evaluation Findings

Relevance
The extent to which the objectives are consistent with beneficiaries requirements, country needs, global priorities and partners and donors policies. Web survey: PPI remains the primary forum on opiates originating in Afghanistan. Questionnaires: PPI is vital for continuously raising awareness and building political will among stakeholders UNODC GLO K31 Project:
the Paris Pact could not exist without the projects support

Relevance

Effectiveness (PPI)
To serve a diverse international partnership as a platform for information exchange; To provide the venue for consultations among experts and policy makers on issues related to opiates originating from Afghanistan;

88% 88%

10% 9%

To provide evidence for strategic planning, policies, and action;

To promote good practices on counter narcotics enforcement and opiates demand reduction; To identify weaknesses and recommend priority actions at the geographic and thematic level;

81% 79%
86% 73%

15% 17%
11% 21%

To coordinate counter narcotics related technical assistance.

Effectiveness

What kind of information provided on ADAM is relevant to your needs?

NSAs products*

28% 26% 22% 28% 22% 21% 29%

0% 10% 20% 30%

36% 33% 33% 31% 39% 43% 26%

40% 50%

7% 6% 17% 15% 13% 11% 13% 17%

60% 70%

8% 8% 5%

16% 10% 14% 13% 15% 11% 14%

90% 100%

Drugs and crime news

TA projects

9% 9%

7% 7%

Recommenda ons

Special messages/updates

8% 5% 10% 2% 10% 3%
80%

Mee ng informa on

Info on PPI process

Highly relevant

Relevant

Somewhat relevant

Limited relevance

Not relevant

N/A

Effectiveness
UNODC Project GLO K31
71% of respondents to the web survey indicated that the project serves the partnership effectively and very effectively. 81% of respondents that have participated in PPI meetings rated UNODCs role in the meetings as good or excellent.

Sustainability and Impact

The continuation of benefits after major development assistance has been completed, and probability of continued long-term benefits. Positive and negative, primary and secondary long-term effect produced, directly or indirectly, intended or unintended.

Among the most sustainable efforts are ones which are difficult to quantify: improved communication, contacts and trust.
Similarly, increased awareness, a stronger willingness to find a common language, greater consensus, buy-in into a multi-lateral process are all sustainable (though not irreversible).

Sustainability and Impact

Partnerships and Cooperation

Perception of PPI support in terms of collaboration between partners

Exchanging lessons learned and promoting best practice

Facilitating cooperation on joint operations Enabling coordination among policy makers to facilitate the Enabling communication among experts Creating synergies among initiatives on counter-narcotics

82% 67% 82% 91%

11% 19% 13% 6%

84%
73%

10%
17%

Avoiding the duplication of efforts

Efficiency of UNODC Project GLO K31

Challenging to measure efficiency: results defined broadly, monetary value of elements partners value most is difficult to assess. Respondents were unable to respond directly on questions of efficiency: Inferred that UNODC must support the project well. Project budget allocated and spent as planned, with the exception of the P3 position in the PPCU (not filled in Phase III). Respondents mentioned good support for meetings, and observed that the project has delivered more than could be expected.

Conclusions

A.PPI is relevant and should continue. Most relevant value-added: PPIs contribution to convergence around balanced, comprehensive approach, and political commitment generated. B. PCG: Providing consistent, clear, solution-focused guidance, building on expertise of EWGs; has a particular responsibility to lead partners efforts.

C. EWGs: more focused & better geared to providing actionable recommendations following 2009 evaluation; room for improvement remains: follow-up.
D.NSAs connect policymakers, experts & projects, are main contributors to ADAM. Further efforts to connect them better with partners & projects under way. E. ADAM: One of the most advanced tools of its kind; usage has increased in Phase III, but is still below critical mass. Considerable room for further improvement remains in terms of usage.

F. Connection between capitals/headquarters and the field, between policymakers and experts, and between donors and priority countries, deserves to be clarified and strengthened, as one of the potential greatest benefits of the PPI.
G.PPI has been relying on a relatively narrow funding base compared to the number of partners; the implementation period of just 24 months has complicated project logistics.

Recommendations

1. Build on the Vienna Declaration: Clearer focus; renewed commitment; balanced, comprehensive approach; more consistent follow-up. 2. Project document for Phase IV: Clear, actionable objectives & responsibilities for stakeholders; strengthened PPCU to facilitate follow up. 3. Reconfirm roles of PCG, EWGs and UNODC; consistent chairs for PCG and EWGs a year in advance; focal points in capitals to follow up between meetings; 4. PPCU and NSAs to facilitate follow up with chairs and partners, including beyond Vienna.

5. Nominate experts to rosters / consistently send them to EWGs; strengthen follow-up mechanisms via PPCU; 6. PPCU: maintain rosters and facilitate follow-up to EWG recommendations. 7. Define and endorse NSAs role in Phase IV as liaison officers (continuing research & capacity-building); additional countries?; 8. UNODC: better define role of NSAs & integrate them further with other projects. 9. Engage more directly in using ADAM for interaction between meetings, including to facilitate preparation and follow-up; 10. PPCU to conduct thorough assessment of best use of ADAM as an information platform.

11. Consistently keep experts in priority countries engaged in the PP process; and ensure follow-up and sustainability; 12. UNODC: strengthen PPCUs connection with other projects. 13. Enable PPCUs work through more predictable, equitable, long-term funding; 14. UNODC to consult with partners on long-term funding for a Phase IV of four years.

Main Issues
Clarity / commitment
Consistency / follow-up

Capacity / resources

Independent evaluation of Paris Pact Phase III

Mr Johannes Chudoba (Evaluation Team Leader) johannes.chudoba@ucentralasia.org Mr Marcio Carvalho (Evaluation Specialist) macarvalho2008@gmail.com