Although significant progress has been achieved in the fields of climate change adaptation (CCA) and disaster risk reduction (DRR) in recent years, experience in policy monitoring and evaluation (M&E) is still rather limited and hence fundamental conceptual and methodological challenges remain. Furthermore, despite their common ground in managing risks and reducing negative impacts of climate change and disasters, CCA and DRR have evolved separately to a large extent, and thus opportunities for knowledge exchange have been limited. However, as the number of countries developing M&E schemes to track the implementation of relevant policies and assess their impacts is growing, interest in sharing experiences and learning from each other, as well as in aligning relevant processes when such opportunities exist, becomes greater.
In response to this interest our session at this year’s European Climate Change Adaptation Conference (5-9 June 2017, Glasgow) focused on fostering dialogue and learning on M&E of CCA and DRR policies. Five speakers shared their experience in designing and implementing M&E systems, presenting examples from Europe and beyond. The discussion that followed revolved around the challenges that evaluators often encounter when attempting to develop meaningful and effective M&E procedures, and the lessons learned so far.
Highlights from the discussion are presented below, and presentation slides and information on the session speakers are provided at the end of this post. For further information please contact Eleni Karali.
Discussion highlights
Fundamental challenges
Long-term perspective: Measuring the effect of a policy is a time-consuming task and demands a long-term perspective. For example, countries need to develop a system now in order to carry out a meaningful evaluation of today’s decisions in 10 or 15 years’ time. This long-term perspective is not always well-received by governments, and thus strong leadership and political will are needed to support relevant decisions.
Uncertainty: Uncertainty is one of the most fundamental challenges when developing or implementing a scheme for M&E of CCA and DRR policies, due to the need to take into account many aspects of a socio-ecological system that go beyond the way that climate system works. For example, some measures may seem ‘ideal’ from a climate science perspective, yet evaluators need to consider a range of pertinent factors and aspects when selecting the ones to be implemented, such as how society might look in the future. Heavy stakeholder engagement, forecast evaluations of policies and they way societies have evolved in the past can be useful tools for informing relevant decisions.
Baseline: Limited or inappropriate data is a fundamental constraint that evaluators are often confronted with and expected to overcome. This is critical especially when trying to establish the baseline conditions for assessing the effect of a policy or measure.
Attribution / accountability: Evaluators are in need of frameworks and methods that will allow them to measure learning and track its evolution in organisations. Outcome-based indicators are also important to assess whether (or not) societies become more resilient after the implementation of adaptation policies and measures.
Developing synergies: Additional efforts are needed to establish links and synergies between Climate Change Adaptation, Sustainable Development (SDGs) and Disaster Risk Reduction (Sendai Framework). A first step could involve the establishment of a core set of indicators that are relevant for all three fields.
Lessons learned
- Having acknowledged that there is no perfect, shining example of an M&E system, participants underlined the importance of people who design M&E schemes reflecting on and sharing their experiences of what worked and what didn’t.
- Taking time to define the purpose and objectives of a national M&E system at an early stage is an important step that can save time and resources, and help in making better decisions at a later stage.
- Learning is often one of the fundamental purposes of M&E systems. While it can be as complex as adaptation itself, evaluators should try to distinguish between different types: technical learning; social learning; systemic learning. A focus on systemic learning, which aims to improve policy-making and implementation, seems to be lacking in many of the current M&E systems.
- There seems to be a gap between people carrying out M&E activities and their ability to translate the results of these activities into meaningful messages for policy-makers. This gap must be bridged if we aim to use M&E results to inform policies and other relevant decisions.
- No single indicator is perfect. A single indicator might tell a very different story compared to the one reflected by when the same indicator is seen in combination with others. This is why indicators should be seen as a set rather than individually.
- Sharing information across countries can be useful. Yet borrowing a set of indicators only can be helpful only if evaluators have a clear idea of what they aim to achieve and what they try to assess.
- To obtain the knowledge needed to revise indicators and/or the overall methodological approach (i.e. what improvements could or should be implemented and where), evaluators need to go through a full M&E cycle at least once.
Speaker biographies
Eleni Karali is a post-doctoral researcher at the Foundation Euro-Mediterranean Centre on Climate Change (Fondazione CMCC, Italy). Her recent research has been focused on monitoring and evaluation of policies, the use of scientific knowledge to inform policy-making, and the links between CCA and DRR.
Markus Leitner is a specialist in Climate Change Adaptation, Environmental Impact Assessment and Strategic Environmental assessment at the Environment Agency Austria. His expertise ranges from adaptation research coordination and funding to adaptation planning at national, regional and European level.
Willem Ligtvoet is Programme Manager on Water Climate and Adaptation at the PBL Netherlands Environmental Assessment Agency. His research encompasses both policy evaluation and assessments, focusing mainly on the interactions between water, climate change and spatial development on national level (the Netherlands), European and global level.
Patrick Pringle is Director of UKCIP and leads their work on monitoring and evaluation for climate change adaptation. He authored UKCIP’s AdaptME Toolkit and was also lead author on two recent European Environment Agency (EEA) reports on national-level monitoring, reporting and evaluation of climate change adaptation in Europe.
Victoria Sword-Daniels’ background and interests lie in the areas of disaster recovery, disaster risk reduction and urban resilience, as well as monitoring, evaluation and learning in climate and disaster related programmes, and knowledge exchange. In 2016 she joined the Climate Change theme at Itad, where she works on the DFID BRACED programme as part of the monitoring and results reporting team, as well as providing M&E support to other climate projects.