Here, I’m sharing some points from the American Evaluation
Association (AEA) Conference that was held from 16 to 20 October 2013,
including the professional development workshop on the last day. The event was
held at Washington DC, USA.
First of all, I’d like to thank the Mercy Corps and AEA for
accepting my paper for presentation and providing financial support to attend
the event. Without those and other logistic supports, the participation was not
possible.
Remaining, I’ll try to briefly share my learning on the
event and from some of the presentations that I attended. The AEA conference
2013 was the huge event that I’d ever seen or attended. More than 37 hundred
participants presented papers that were categorized under close to nine hundred
sessions. The link takes to the detailed list of sessions and the abstracts of
the papers therein: http://www.americanevaluation.org/search13/allschedule.asp.
The email of the presenters is also provided to facilitate the communication in
case a paper is of interest to the reader. Besides, another link compiles the
list of participants that also lead to the presentation of the participant: http://www.americanevaluation.org/search13/choosepresenter.asp.
I presented a paper in the
session entitled ‘Around the World: Cost Experiences and Insights’. The paper
is entitled as Estimating Social Return on Investment of Social Protection
Programs:
Piloting an Approach in Irrigation Canal Improvement Project in Far-western Hills of Nepal. The paper concludes that irrigation scheme is a strong investment to social benefit of target communities. However, the high benefit cost ratio (BCR) poses whether irrigation scheme alone is attributed to increased production. The cost of other interventions (such as training, demonstration, seed distribution) needs to be considered for their attribution and contribution to the increased production. Participants raised their interests to know on what basis discount rate of 10% and time horizon of five years were assumed to estimate BCR. It was clarified that those were assumed based on the standard used by Asian Development Bank (1009) and other studies undertaken in disaster risk reduction in Nepal (White and Rorick 2010 and Willenbockel 2011). There were some suggestion to undertake the sensitivity analysis by altering the discount rate and time of benefit to see at what levels the cost of intervention outweighs the net present value. The abstract of my paper that I submitted at the time of call is available at the link: http://www.americanevaluation.org/search13/session.asp?sessionid=2037&presenterid=0.
Piloting an Approach in Irrigation Canal Improvement Project in Far-western Hills of Nepal. The paper concludes that irrigation scheme is a strong investment to social benefit of target communities. However, the high benefit cost ratio (BCR) poses whether irrigation scheme alone is attributed to increased production. The cost of other interventions (such as training, demonstration, seed distribution) needs to be considered for their attribution and contribution to the increased production. Participants raised their interests to know on what basis discount rate of 10% and time horizon of five years were assumed to estimate BCR. It was clarified that those were assumed based on the standard used by Asian Development Bank (1009) and other studies undertaken in disaster risk reduction in Nepal (White and Rorick 2010 and Willenbockel 2011). There were some suggestion to undertake the sensitivity analysis by altering the discount rate and time of benefit to see at what levels the cost of intervention outweighs the net present value. The abstract of my paper that I submitted at the time of call is available at the link: http://www.americanevaluation.org/search13/session.asp?sessionid=2037&presenterid=0.
Jon Kurtz, Mercy Corps Director
for Research and Learning, Lara Hilton, an Evaluation Expert (PhD scholar at
Claremont Graduate University) and I presented jointly a paper entitled Connecting
program and social science theory in a Nepalese microfinance program
evaluation. The paper is based on the baseline evaluation of PROMISE
project. The purpose of the evaluation is to examine who accesses and
benefits most from VSLAs, thus answering the question of whether they
constitute an effective vehicle for social protection of very vulnerable
groups. Besides, it will contribute to answering the broader question of: Are
the lack of formalization and capacity of existing savings and loan groups in
rural Nepal major constraints to their abilities to contribute to improving
their members’ livelihoods and abilities to cope with risks and shocks? During
presentation, it was emphasized that Mercy Corps relies increasingly on program theory-based
evaluation to increase knowledge about effectiveness of its intervention
strategies. The theoretical approach to move beyond the current practice of
performance monitoring evaluation was discussed. Participants were curious to
know whether there was any difference between eastern and western districts.
Abstract of the paper submitted to the organization is posted with a different
title: What They Don't Tell You: Connecting Program and Social Science
Theory, in a Nepalese Microfinance Program Evaluation at: http://www.americanevaluation.org/search13/session.asp?sessionid=2145&presenterid=1795.
Jon Kurtz, Mercy Corps
Director for Research and Learning presented a paper entitled Mercy Corps'
Contingency Approach to Measuring Resilience. The paper highlighted the methods
applied to generate evidence on what contributes to strengthening resilience in
the arid lands of Africa. The main methods are: analysis of post-shock
cross-sectional data to predict resilience; quasi-experimental impact
evaluation designs to determine attributable program effects on resilience
outcomes; and ex-post program evaluations to understand how target populations
recover and manage subsequent shocks.
Scott Chaplowe, International
Federation of Red Cross and Red Crescent Societies (IFRC) presented a paper on
Counting People Reached – Reality versus Fiction. Harry Carr
from Catholic Relief Services presented the paper on Tracking
Beneficiary and Service Delivery - Lessons From Catholic Relief Services. Michael
Wallace, Mercy Corps Senior Monitoring, Evaluation and Learning Advisor had
facilitated the discussion. In
the pursuit of demonstrating accountability, measuring both output and outcome
level results have been the pressing need. However, measuring direct versus
indirect people-reached, as well as avoiding double-counting are contested. The
papers discuss the challenges in the development and roll-out of the Monitoring
System. IFRC has the system of counting only once the direct recipients of
services within regardless of how may services they receive. They only count
indirect recipients when a credible approximation can be made. The paper
suggests to count Households to determine the number of people reached, based
on the credible evidence that household has received service and based on
reliable statistic of the average household size. The purpose of collecting
people reached data is an important organizational decision.
Jane
Buckley, Cornell University and Thomas Archibald Cornell Office for Research on
Evaluation facilitated the workshop on Evaluation Thinking (ET). They define ET
as a cognitive process motivated by inquisitiveness and a belief in the value
of evidence. The process includes: 1) identifying assumptions, 2) posing
thoughtful questions, 3) pursuing deeper understanding through reflection and
perspective taking, and, 4) making informed decisions in preparation for
action. In the development context, the interventions are undertaken with the assumption
that they produce short-run to long-run results. For e.g. irrigation canal is
constructed with the assumption that it will increase access to irrigation
facility to cultivate crops. With irrigation facilities, the farmers are
assumed to increase the production of the crops that they have been
cultivating, or the farmers are assumed to select those varieties of the same
crop that are potential to bear higher production in irrigated condition or the
farmers are assumed to change their cropping pattern to introduce new crops
that are highly producing and fetching good market price. With increased
production, the farmers are expected to increase the value of production from
their crops. Besides, the farmers are assumed to meet their own food
requirements and sale surplus. These sorts of assumptions will help come up
with evaluation questions: does irrigation intervention increase land
under irrigation? Have farmers increased their production with the crops they
have been producing for years? Have farmers changed the variety of the same
crop? Have farmers introduced new crop? Have farmers increased the monetary
value of the crops?. The question will help establish claims: irrigation
intervention increases area under irrigation, area under irrigation increases
choice of crops that are suitable for irrigated conditions. There, come the
parts of what approach, methodology and tools will be applied to measures the
variables to establish these claims: discussion, survey, or what else. Thus,
evaluative thinking is based on identifying the assumption and believing on the
value of evidence to establish the claims.
