Monitoring, evaluation and learning
CSA plan’s monitoring, evaluation, and learning (ME&L) component develops strategies and tools to track progress of implementation, evaluate impact, as well as facilitate iterative learning to improve CSA planning and implementation. CSA Plan’s ME&L delivers processes and products to support achieving and documenting program goals and adaptively managing implementation. The primary audience of the ME&L component of CSA Plan is program and project designers and managers.
CSA aggregates outcomes from individual efforts towards the common set of goals of a more productive, resilient and lower emission food system. While the scale and scope of activities certainly differ, they may all contribute to CSA objectives. The fundamental premises underlying CSA Plan’s ME&L is that CSA is a gradient and not an endpoint and that CSA is context (place and time) specific. The triple win of productivity, resilience and mitigation may not be achievable or prioritized in all places. This means that the standards for performance and learning need to be specific to a societal aim and specific to a time and place. This precondition of clear objectives will typically be informed by step 1 in CSA Plan (‘situation analysis’) and set in Step 2 (‘targeting and prioritizing’).
Outcomes are measured using indicators and metrics where at least two key principles should be considered:
- There should be a minimal set of high-level indicators tracked across programs to allow easy measurement and facilitate comparability. Additional higher resolution and indicators that are more specific may be important for specific projects. It is important to note that there are many groups globally (e.g. World Bank and CCAFS, see cases) working to set this high-level set of indicators and initial release should be forthcoming in April 2016.
- Indicators should be ‘SMART’: simple, measurable, accurate, reliable and time bound, which allow tracking of performance and change in condition.
CSA plan ME&L approach uses a simple three-step process to help determine which type of metrics, indicators, and monitoring approaches are suitable for a given context. The decision tree in Fig. 1 helps to determine the categories of relevant CSA indicators. Oftentimes, implementation of CSA across the entire program lifecycle will require all three categories (or use-cases) of indicators that follow:
Figure 1. Simple decision tree to determine which use- case to select from. Typically, you will need to select and use indicators from each group throughout the project lifecycle. Each use case is described below. Based on an extensive database of indicators, CCAFS in the context of collaboration with USAID Feed the Future has developed a new tool to plan CSA projects and indicators (see the CSA Programming and Indicator Tool).
CSA Plan’s indicators use-cases for the project lifecycle:
- Readiness indicators
- Process indicators
- Progress/impact indicators
Readiness indicators: The first use case is to determine readiness for a CSA intervention. Readiness is important because CSA requires many supporting structures for success and sometimes the intervention necessitates preliminary capacity building to precondition the location or target population. Readiness indicators need to be tied to the type of intervention designed (See case study 3). For example, if a project desires to scale-up agroforestry in the Western Kenya, it will be important to know items such as the status of the extension system, network of nurseries, land tenure, etc. The information needed may be significantly different from that for an intervention designing an early warning system or insurance scheme.
Process indicators: CSA Plan advocates inclusive, transparent, and responsive processes to enable adaptive management, learning, and tracking of project implementation. Process-oriented indicators allow programs to be evaluated for meeting CSA and planned objectives. Certain ME&L indicators provide insights into implementation processes such as the diversity and gender of team composition, number and quality of interactions with communities, timeliness of reporting, etc. While often neglected in many development programs, CSA Plan proposes that the importance of these indicators cannot be understated as they provide purchase for just, equitable and transparent processes.
Progress/impact indicators: A key distinguishing feature of M&E for CSA is the need to understand the impact on all three objectives. One can use either input or outcome indicators to measure progress or impact. Input indicators typically rely on assumptions about the relationships between activities in an outcome. For example, adoption of conservation agriculture increases the resilience to intra-seasonal droughts. Or use of nitrogen fertilizer affects nitrous oxide emissions. While tractable, in many cases the relationships between inputs and outcomes are not well documented or are site specific. Thus, the magnitude of change is difficult or impossible to ascertain. Alternatively, one can measure direct indicators of outcomes of interest (see case studies 1 and 2). Numerous individual indicators can describe each pillar. For example, productivity may be illustrated by changes in yield, calories, or income (see Measuring productivity below). Or mitigation may be characterized by carbon sequestration or greenhouse gas emission per unit product (see Measuring mitigation below). Resilience indicators are more challenging, with both proxy and process indicators being commonly use (see Measuring adaptation and resilience below). The diversity of indicators with productivity and mitigation and the varying opinions and options of indicators for resilience further reinforce the need to select indicators will end users in a participatory process so that they reflect their goals (see toolboxes for further references).
Changes to productivity can be measured in different ways. The most common approach is to measure yields. Yield measurement techniques vary between crops, and range from weighing harvested grain from the entire field to weighing representative samples from a plot area after plants have reached physiological maturity (Lauer 2002:1). There are also many mathematical approaches for computing basic yield.
Productivity measurements can also be expanded to include food security indicators such as calorie availability, poverty indicators, and availability of dietary diversity.
Improving the nutritional relevance of food security measurement means using indicators that capture both macro- and micronutrient consumption, that can be measured at the individual level, and that give some sense of acute food insecurity (such as seasonal shortfalls or consumption shocks).
Examples of indicators used to measure productivity include:
- Yield (e.g. product per unit of land, water, energy, nutrients, labour)
- Income (e.g. gross margin, net present value)
- Labour (e.g. person hours, labour allocations by gender)
Additional indicators to measure food security include:
- Per capita food consumption in terms of e.g. calories, protein, dietary diversity
- Food deficits, such as number of hungry months
- Food prices
- Percentage of income spent on food
- Children’s nutritional status ( e.g. upper arm measurements to indicate wasting or stunting)
For mitigation, the UNFCCC has well-established protocols for measurement. These are classified as:
- Tier 1 (reference approach): The simplest approach, which uses IPCC default emission factors and requires the most basic and least disaggregated activity data.
- Tier 2 and Tier 3 (sectorial approaches): Different methods can be used to estimate emissions or removals from most source and sink categories. The selection of a particular method will depend on the desired degree of estimation detail, the availability of activity data and emission factors, and the financial and human resources available to complete the inventory (UNFCCC, 2009). The difference between Tiers 2 and 3 is mainly an increase in the degree of detail required by the method.
Work that is more recent seeks to devise cost-effective measurement approaches for smallholder farmer landscapes. These include theStandard Assessment of Mitigation Potential and Livelihoods in Smallholder Systems (SAMPLES) program, which provides guidance for measuring emissions from smallholder systems. GHG calculators such as Cool Farm Tool, Ex-Act Tool, and SHAMBA Tool etc. also allow users to measure mitigation outcomes.
Examples of indicators used to measure mitigation include:
- Emissions of methane, nitrous oxide and carbon dioxide from all agricultural sources including energy, soils
- Removals and accumulation of carbon in biomass and soils
- Changes in land use, particularly conversion of high C land uses such as forests and peatland
- Fuel wood consumption
- Biofuel use
Measuring adaptation, adaptive capacity and resilience is complicated. They tend to be measured in terms of attributes of the system (rather than as outcomes for farms and people). Context-specificity is important – for example, a more diverse system may be more adaptive in many cases, but not always.
Examples of indicators used to measure adaptation and resilience include:
- Access to capitals (financial, human, social/political, physical, natural)
- Access to services (particularly climate information services)
- Level of skills, knowledge and access to extension on climate change
- Diversity in livelihoods and income sources
- Market access (for food, agricultural inputs and agricultural product markets)
- Gender equity (e.g. labour, income differences)
- Biodiversity (e.g. Shannon, N %)
- Pests/pathogens (e.g. % loss, damage rates)
- Erosion/Soil loss (e.g. kg/ha)
- Soil quality (e.g. changes in C, N, soil water balance, etc.)
- Income levels
- Access to credit
- Land rights/tenure
- Access to insurance
- Proportion of income from climate-prone sources
- Enabling policy and regulation environment
- Incentive systems
- Subsidies (directed away from maladaptive practices towards resilience practices)
- Safety net schemes
- Early warning systems and disaster recovery strategies
Gujit I, Woodhill J. 2002. Managing for impact in rural development. Rome, Italy: IFAD.
This Guide has been written to help project managers and M&E staff improve the quality of M&E in IFAD-supported projects. The Guide focuses on how M&E can support project management and engage project stakeholders in understanding project progress, learning from achievements and problems, and agreeing on how to improve both strategy and operations.
The main functions of M&E are: ensuring improvement-oriented critical reflection, learning to maximise the impact of rural development projects, and showing this impact to be accountable. The Guide is meant to improve M&E in IFAD-supported projects, as a study found that most projects have a fairly low standard of M&E. The Guide provides comprehensive advice on how to set up and implement an M&E system, plus background ideas that underpin the suggestions.
World Bank. 2004. Monitoring and evaluation (M&E): Some tools, methods & approaches. Washington, DC: World Bank.
Government officials, development managers and civil society are increasingly aware of the value of monitoring and evaluation (M&E) of development activities. M&E provides a better means of learning from past experience, improving service delivery, planning and allocating resources, and demonstrating results as part of accountability to key stakeholders. Yet there is often confusion about what M&E entails. This booklet therefore presents a sample of M&E tools, methods and approaches, including several data collection methods, analytical frameworks, and types of evaluation and review. For each of these, a summary is provided of the following: their purpose and use; advantages and disadvantages; costs, skills, and time required; and key references.
UNFCCC. 2010. Synthesis report on efforts undertaken to monitor and evaluate the implementation of adaptation projects, policies and programmes and the costs and effectiveness of completed projects, policies and programmes, and views on lessons learned, good practices, gaps and needs. Subsidiary Body for Scientific and Technological Advice 32nd session. Bonn, 31 May to 9 June 2010. Bonn, Germany: UNFCCC.
This document synthesizes information contained in submissions from Parties and organizations and in other relevant sources on efforts undertaken to monitor and evaluate the implementation of adaptation measures, including projects, policies and programmes. This document synthesizes efforts in this area and also reports on the development and use of adaptation indicators. A summary of lessons learned, good practices, gaps and needs is provided, and the document concludes by raising issues for further consideration.
Lamhauge N, Lanzi ER, Agrawala S. 2012. Monitoring and evaluation for adaptation: Lessons from development co-operation agencies. OECD Environment Working Paper No. 38. Paris, France: OECD Publishing.
In the context of scaled up funding for climate change adaptation, it is more important than ever to ensure the effectiveness, equity and efficiency of adaptation interventions. Robust monitoring and evaluation (M&E) is an essential part of this, both to ensure that the prospective benefits of interventions are being realised and to help improve the design of future interventions. This paper is the first empirical assessment of M&E frameworks used by development co-operation agencies for projects and programmes with adaptation-specific or adaptation-related components. It has analysed 106 project documents across six bilateral development agencies. Based on this, it identifies the characteristics of M&E for adaptation and shares lessons learned on the choice and use of indicators for adaptation.
Colomb V, Bernoux M, Bockel L, Chotte J-L, Martin S, Martin-Phipps C, Mousset J, Tinlot M, Touchemoulin O. 2012. Review of GHG calculators in agriculture and forestry sectors: A guideline for appropriate choice and use of landscape based tools. Rome, Italy: Food and Agriculture Organization of the United Nations.
Climate change and its consequences are now recognized amongst the major environmental challenges for this century. Land based activities, mainly agriculture and forestry, can be both sources and sinks of greenhouse gases (GHG). In most countries, they represent significant share of total GHG emissions, around 30 % at global level. In order to reach global or national reduction target, as well as meeting food security challenges, agriculture and forestry sectors need to evolve. In parallel to IPCC work and progress on methodological issues, many GHG tools have been developed recently to assess agriculture and forestry practices. Denef et al. (2012) classify these tools as : calculators, protocols, guidelines and models. This review focus on calculators, i.e, automated web-, excel-, or other software-based calculation tools, developed for quantifying GHG emissions or emission reductions from agricultural and forest activities. This review considers calculators working at landscape/farm scale, including several productions: crop, livestock and forest. Eighteen major calculators were identified, amongst them EX-ACT, ClimAgri, Cool Farm Tool, Holos, USAID FCC and ALU.
Approaches and tools
In this section, we will describe select approaches, tools and cases that guide and exemplify an entire process or to facilitate specific elements of the process.
CCAFS has created a database of over 378 CSA-related indicators gathered from several international development agencies (FAO, DFID, GIZ, IFAD-ASAP, World Bank, USAID) to develop a public access CSA Programming and Indicator Tool to contribute to address both, the need of good instruments for CSA programming and better metrics for tracking outcomes and impacts. The Tool proposes a shared framework for agricultural programs to: i) examine to what extent current or planned intervention(s) address each CSA pillar, ii) compare the scope and CSA intentionality among different project designs to make future programming more climate-smart, and iii) support the identification and selection of an appropriate set of indicators to measure and track CSA related outcomes.
“The capacity to manage plan, implement, and monitor climate finance and activities related to climate change is a condition known as climate readiness” (Wollenberg et al. 2015). 1 Climate readiness in the agricultural sector is highly variable, and indicators for climate readiness can provide guidance to implementers and investors to target their efforts. Drawing upon some experience from REDD+ readiness, Wollenberg et al. (2015) 1 has provided examples of climate readiness indicators across five work areas. These include (examples only, see Wollenberg et al. 1 for full list):
Effective governance and stakeholder engagement:
- Lead ministry or inter-ministerial body designated to manage and coordinate climate-ready activities with clear decision-making processes and transparency.
- Institutional roles are clear in agencies and local jurisdictions.
- Platforms exist for stakeholder engagement and consultation, including the private sector and ensuring the inclusion of affected and vulnerable groups, such as smallholders, indigenous groups and women.
Knowledge base and information services:
- Classification exists of current agricultural production systems related to adaptation needs and mitigation opportunities.
- Identification of options and priorities for CSA, including reducing agricultural greenhouse gas (GHG) emissions is consistent with agricultural development objectives.
- Climate information services are available and accessible to farmers and other agricultural decision makers.
- Social and environmental impacts of CSA anticipated before programs are scaled up, particularly for vulnerable groups in agriculture assessed.
Climate-smart agriculture strategy and implementation framework:
- Agreed-upon vision and goals exist for the agricultural sector that balance food security, adaptation and mitigation and help meet United Nations Sustainable Development Goals.
- Identification of priority interventions, geographic areas and potential reductions in vulnerability and emissions exists.
- Cost-benefit analysis of program options exists to inform business viability and finance needs.
- CSA mainstreamed into predominant agricultural programs.
National and sub-national capabilities to develop sustainable CSA infrastructure and investment strategies and practices:
- Rural credit is available for CSA practices.
- Capacity exists to support access to seed banks and make adapted seeds available to farmers.
- Private sector and rural farmers’ organizations, including women and youth groups, support innovation, learning and implementation.
National information system for monitoring and accounting in agriculture:
- Criteria and measureable indicators for resilience, climate change mitigation and productivity or food security identified.
- Monitoring systems for climate threats and vulnerability assessments exist.
- National system in place to measure, monitor, report and verify GHG emissions and multiple-benefit indicators relevant for agriculture, in coordination with other monitoring activities.
The Adaptation for Smallholder Agriculture Programme (ASAP) uses 10 specific and measurable indicators to evaluate programme progress. These indicators are (IFAD 2012a, p. 16): 2
- # of poor smallholder household members whose climate resilience has been increased because of ASAP disaggregated by sex.
- % of new investments in ENRM in IFAD 9th Replenishment compared to IFAD 8th Replenishment.
- Leverage ratio of ASAP grants versus non ASAP financing
- % increase in number of non-invasive on farm plant species per smallholder farm supported
- # of tonnes of GHG emissions (CO2e) avoided and/or sequestered
- # increase in hectares of land managed under climate resilient practices
- % change in water use efficiency by men and women
- # of community groups involved in ENRM and/or DRR formed or strengthened
- $ value of new or existing rural infrastructure made climate-resilient
- # of international and country dialogues where IFAD or IFAD-supported partners make an active contribution
These indicators have been identified based on ASAP goals, purpose, and outcomes and helps measure progress towards 2020 targets. While selecting projects for funding, ASAP evaluates their potential contribution towards these indicators. For selected projects, as part of project monitoring process, the progress to these indicators is measured and thereafter aggregated globally.
The CSA working group of the World Business Council is developing the Corporate CSA measurement protocol for Sustainable Development (WBCSD), with technical support from CCAFS. The protocol envisages the identification and assembly of the most appropriate tools and methodologies to measure progress under each CSA pillar, under an over-arching framework. Currently in the conceptual stage, the protocol considers various approaches to measurement under each pillar at the site/business-unit level. These are outlined below:
Pillar 1: Productivity
In addition to measuring yield metrics, this pillar focuses on collection of additional metrics such as earnings and incomes among smallholder suppliers and wider rural populations to understand the contributions to food security and livelihood goals.
Pillar 2: Climate change resilience, incomes & livelihoods
Under this pillar, companies are considering two approaches to measurement. One involves establishing a theory of change that identifies a plausible chain of causality of how CSA interventions build resilience. The other approach involves calculating a “climate smartness factor” for these interventions, and calculating the overall resilience by combining the climate smartness factor together with adoption of these interventions.
Pillar 3: Climate Change Mitigation
The protocol envisages measuring changes in greenhouse gas emissions per area (CO2e per ha) or per product (CO2e per kg) for different activities at the site or business unit levels. Tools including GHG calculators such as the Cool Farm Tool, Small-Holder Agriculture Monitoring and Baseline Assessment (SHAMBA) Tool, EX-ACT Tool, and CCAFS-MOT will be considered for measuring these changes.
Background and purpose
The World Bank is currently developing a set of indicators on climate-smart agriculture with the tentative aim to provide policymakers and development practitioners with a framework for implementing the necessary policy, technical, and monitoring and evaluation (M&E) framework to make CSA fully operational. The framework will help decision-makers prioritize strategies and interventions that most effectively lead to more climate friendly outcomes.
Use and users
The World Bank CSA indicators are currently conceived as three sets of indexes: one on policy, one on technology and practices, and one on results. The three indexes are based on a theory of change, which describes the “impact pathways” through which project outcomes are arrived at.
The CSA Policy Index (CSA Policy) comprises three themes. Readiness refers to the capacity of countries to plan and deliver adaptation, mitigation, disaster risk and land rights policies and programs in ways that are catalytic and fully integrated with national agricultural development priorities. Services and Infrastructure, reflects the ability to leverage agricultural investments through the provision of services and enabling environment such as extension, research and development, roads, social safety nets, GHG inventory and risk management systems, and support for starting agribusinesses. Coordination and Implementation, assesses collaboration between agricultural institutions and agencies in related fields such as environment, and the extent to which adaption and mitigation are mainstreamed in agricultural programs.
The CSA Technology and Practices Index (CSA Technology) comprises three themes: Productivity (P), Resilience (R), and Mitigation (M). Ex-ante application of the index reveals how project interventions can lead to productivity gains and environmental benefits. It is particularly useful in identifying the most appropriate technologies for a CSA project during its planning and design stages.
The CSA Results Index is intended to help project leaders measure an agricultural project’s performance towards achieving the CSA triple-wins individually and jointly. It can be applied to monitor progress during a CSA project’s implementation and to evaluate impacts once the project has closed. It affords project planners substantial flexibility in customizing a CSA intervention according to the specific context in which it will operate.
- Presentation at the CSA Conference in Montpellier, France, April 2015, http://csa2015.cirad.fr/var/csa2015/storage/fckeditor/file/L2.1e.CSA%20Indicators%20EmenanjoL.pdf
For more information, please contact Ademola Braimoh, World Bank, firstname.lastname@example.org
Background and purpose
A fundamental requirement for operationalizing CSA is understanding (1) which climate risks can be addressed by CSA;(2) how changing from “conventional” to “CSA” management practices changes farm-level outcomes in terms of productivity, resilience and climate change mitigation; and (3) how both these risks and outcomes are expected to change under climate change. Unfortunately little empirical evidence has been put forth to systematically evaluate CSA. For these reasons the World Agroforestry Centre (ICRAF), CGIAR Research Program on Climate Change, Agriculture, and Food Security (CCAFS), Food and Agriculture Organization of the United Nations (FAO), and the International Center for Tropical Agriculture (CIAT) are attempting to unpack the CSA concept in a systematic through a quantitative meta-analysis of the published scientific literature to date on CSA practices. The objective of meta-analysis is to combine the available information to (1) map where there is evidence and where there is not and (2) synthesize the information to answer key questions for decision makers, in this case program developers/implementers, policy makers, and investors.
Use and users
The review is based on data found in English-language peer-reviewed journals with searches using terms relevant to CSA practices and outcomes. Promising practices identified as potentially CSA were organized into five general themes: agronomy, agroforestry, livestock and aquaculture, postharvest management, and energy systems. Under these themes, 73 practices were selected. For each outcome of CSA, there are many dimensions and potential indicators that can be measured. For example, increased food security may result from changes in availability of food (e.g. increased yield), accessibility of food (e.g. increased income), utilization of food (e.g. increased food safety) or stability (less variable harvests) (FAO 2002). Similarly, mitigation benefits can come from emission reductions, enhanced removal of GHGs, or future emissions avoided through adoption of CSA technologies (Smith et al 2008). A comprehensive list of indicators were selected to represent the range of economic, environmental, and social impacts possible and desired from CSA practices (see Rosenstock et al. forthcoming). Data are currently organized in Microsoft Excel workbooks, however, the resulting data will be compiled into a searchable Web-based database and analytical engine.
- Poster at the 3rd Global Science Conference on CSA, Montpellier, France http://www.slideshare.net/cgiarclimate/panacea-orpropogandaposterreduced
For more information, please contact Todd Rosenstock, CCAFS, email@example.com
Wollenberg E, Zurek M, De Pinto A. 2015. Climate readiness indicators for agriculture. CCAFS Info Note. Copenhagen, Denmark: CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS).https://cgspace.cgiar.org/bitstream/handle/10568/68685/CCAFS%20info%20note%20readiness%20indicators%20Oct%202015%20final%20A4%20%281%29.pdf?sequence=11&isAllowed=y Countries vary in their institutional technical and financial abilities to prepare for climate change in agriculture and to balance food security, adaptation, and mitigation goals.Indicators for climate readiness provide guidance to countries and enable monitoring progress. Readiness assessments can enable donors, investors and national decision-makers to identify where investments are needed or likely to be successful. Examples of climate readiness indicators are provided for five work areas: 1. governance and stakeholder engagement, 2. knowledge and information services, 3. climate-smart agricultural strategy and implementation frameworks, 4. national and subnational capabilities and 5. national information and accounting systems.
IFAD. 2012a. Adaptation for Smallholder Agriculture Programme (“ASAP”) Programme Description. Rome, Italy: International Fund for Agricultural Development.http://www.ifad.org/climate/asap/note.pdf
ASAP is a new direct entry point in IFAD to channel earmarked climate and environmental finance to smallholder farmers. ASAP funds will co-finance projects using clear selection criteria and applying a results framework which contains 10 specific and measurable indicators of achievement. An important element of ASAP will be a knowledge management programme that will develop and share climate adaptation lessons and tools across IFAD‘s programmes and with key external partners. Based on a thorough monitoring and evaluation system, this is expected to demonstrate the value of investing climate finance in smallholders to the Green Climate Fund and other climate initiatives. Investment areas are determined by the needs identified by partner communities.