My few learning from American Evaluation Association (AEA) conference 2013

Here, I’m sharing some points from the American Evaluation Association (AEA) Conference that was held from 16 to 20 October 2013, including the professional development workshop on the last day. The event was held at Washington DC, USA.

First of all, I’d like to thank the Mercy Corps and AEA for accepting my paper for presentation and providing financial support to attend the event. Without those and other logistic supports, the participation was not possible.

Remaining, I’ll try to briefly share my learning on the event and from some of the presentations that I attended. The AEA conference 2013 was the huge event that I’d ever seen or attended. More than 37 hundred participants presented papers that were categorized under close to nine hundred sessions. The link takes to the detailed list of sessions and the abstracts of the papers therein: http://www.americanevaluation.org/search13/allschedule.asp. The email of the presenters is also provided to facilitate the communication in case a paper is of interest to the reader. Besides, another link compiles the list of participants that also lead to the presentation of the participant: http://www.americanevaluation.org/search13/choosepresenter.asp.

I presented a paper in the session entitled ‘Around the World: Cost Experiences and Insights’. The paper is entitled as Estimating Social Return on Investment of Social Protection Programs:
Piloting an Approach in Irrigation Canal Improvement Project in Far-western Hills of Nepal. The paper concludes that irrigation scheme is a strong investment to social benefit of target communities. However, the high benefit cost ratio (BCR) poses whether irrigation scheme alone is attributed to increased production. The cost of other interventions (such as training, demonstration, seed distribution) needs to be considered for their attribution and contribution to the increased production. Participants raised their interests to know on what basis discount rate of 10% and time horizon of five years were assumed to estimate BCR. It was clarified that those were assumed based on the standard used by Asian Development Bank (1009) and other studies undertaken in disaster risk reduction in Nepal (White and Rorick 2010 and Willenbockel 2011). There were some suggestion to undertake the sensitivity analysis  by altering the discount rate and time of benefit to see at what levels the cost of intervention outweighs the net present value. The abstract of my paper that I submitted at the time of call is available at the link: http://www.americanevaluation.org/search13/session.asp?sessionid=2037&presenterid=0.

Jon Kurtz, Mercy Corps Director for Research and Learning, Lara Hilton, an Evaluation Expert (PhD scholar at Claremont Graduate University) and I presented jointly a paper entitled Connecting program and social science theory in a Nepalese microfinance program evaluation. The paper is based on the baseline evaluation of PROMISE project. The purpose of the evaluation is to examine who accesses and benefits most from VSLAs, thus answering the question of whether they constitute an effective vehicle for social protection of very vulnerable groups. Besides, it will contribute to answering the broader question of: Are the lack of formalization and capacity of existing savings and loan groups in rural Nepal major constraints to their abilities to contribute to improving their members’ livelihoods and abilities to cope with risks and shocks? During presentation, it was emphasized that Mercy Corps relies increasingly on program theory-based evaluation to increase knowledge about effectiveness of its intervention strategies. The theoretical approach to move beyond the current practice of performance monitoring evaluation was discussed. Participants were curious to know whether there was any difference between eastern and western districts. Abstract of the paper submitted to the organization is posted with a different title: What They Don't Tell You: Connecting Program and Social Science Theory, in a Nepalese Microfinance Program Evaluation at:  http://www.americanevaluation.org/search13/session.asp?sessionid=2145&presenterid=1795.

Jon Kurtz, Mercy Corps Director for Research and Learning presented a paper entitled Mercy Corps' Contingency Approach to Measuring Resilience. The paper highlighted the methods applied to generate evidence on what contributes to strengthening resilience in the arid lands of Africa. The main methods are: analysis of post-shock cross-sectional data to predict resilience; quasi-experimental impact evaluation designs to determine attributable program effects on resilience outcomes; and ex-post program evaluations to understand how target populations recover and manage subsequent shocks.

Scott Chaplowe, International Federation of Red Cross and Red Crescent Societies (IFRC) presented a paper on Counting People Reached – Reality versus Fiction. Harry Carr from Catholic Relief Services presented the paper on Tracking Beneficiary and Service Delivery - Lessons From Catholic Relief Services. Michael Wallace, Mercy Corps Senior Monitoring, Evaluation and Learning Advisor had facilitated the discussion. In the pursuit of demonstrating accountability, measuring both output and outcome level results have been the pressing need. However, measuring direct versus indirect people-reached, as well as avoiding double-counting are contested. The papers discuss the challenges in the development and roll-out of the Monitoring System. IFRC has the system of counting only once the direct recipients of services within regardless of how may services they receive. They only count indirect recipients when a credible approximation can be made. The paper suggests to count Households to determine the number of people reached, based on the credible evidence that household has received service and based on reliable statistic of the average household size. The purpose of collecting people reached data is an important organizational decision.

Jane Buckley, Cornell University and Thomas Archibald Cornell Office for Research on Evaluation facilitated the workshop on Evaluation Thinking (ET). They define ET as a cognitive process motivated by inquisitiveness and a belief in the value of evidence. The process includes: 1) identifying assumptions, 2) posing thoughtful questions, 3) pursuing deeper understanding through reflection and perspective taking, and, 4) making informed decisions in preparation for action. In the development context, the interventions are undertaken with the assumption that they produce short-run to long-run results. For e.g. irrigation canal is constructed with the assumption that it will increase access to irrigation facility to cultivate crops. With irrigation facilities, the farmers are assumed to increase the production of the crops that they have been cultivating, or the farmers are assumed to select those varieties of the same crop that are potential to bear higher production in irrigated condition or the farmers are assumed to change their cropping pattern to introduce new crops that are highly producing and fetching good market price. With increased production, the farmers are expected to increase the value of production from their crops. Besides, the farmers are assumed to meet their own food requirements and sale surplus. These sorts of assumptions will help come up with evaluation questions: does irrigation intervention increase land under irrigation? Have farmers increased their production with the crops they have been producing for years? Have farmers changed the variety of the same crop? Have farmers introduced new crop? Have farmers increased the monetary value of the crops?. The question will help establish claims: irrigation intervention increases area under irrigation, area under irrigation increases choice of crops that are suitable for irrigated conditions. There, come the parts of what approach, methodology and tools will be applied to measures the variables to establish these claims: discussion, survey, or what else. Thus, evaluative thinking is based on identifying the assumption and believing on the value of evidence to establish the claims.

Overall, AEA conference 2013 was a giant forum. My learnings are few drops of water from the ocean.