Abstract
For the evaluation to have an impact, it has to be communicated clearly to those who can influence plans for the future, as well as others who could benefit from what has been learned (e.g. program managers, coffee extensionists and other coffee farmers). It is important at this stage to think back to Task A) the purpose of your evaluation and B) your evaluation questions. These should have been developed in conjunction with key stakeholders – what did they say they wanted the M&E process to achieve? The answer to this question will help in deciphering which information different groups will want to receive.
- Develop a plan for sharing the findings with different audiences and in various formats
- Documentation of lessons learned and development of a case study for the c&c tool box
- Audience-appropriate communication approaches implemented
- Identified opportunities for including lessons learned in future plans
Theory
In addition to providing recommendations for the future, it is important that the lessons learned influence the development of plans for what to do next. A simple table that organizes information about dates for planning decisions may be useful to align data collection and the analysis with decision-making needs, especially where decisions have long-term implications (e.g. choices about planting new varieties of coffee or choosing where to set up new plantations).
Guiding questions for using the findings:
- Who would benefit from hearing about what happened?
- What opportunities are there for bringing learning in to inform and improve future plans and who has an influence in this?
- How will the lessons from the evaluation be articulated and shared?
- How will progress on recommendations be monitored and assessed?
- How might you develop learning further?
- What further questions should we be asking?
Think carefully about the audiences that you want to benefit from the findings and what messages and format would be most appropriate. For example, some coffee sector experts may be interested in technical details while others may want a simple summary of findings (policymakers often appreciate a succinct executive summary of just a few pages). In contrast, reaching farmers with key messages might involve reporting back to meetings, holding informal events on a market day or using local radio. Think about dissemination as soon as you can so you can include it in your budgeting. Record the key outputs from this task in the M&E plan template, see Step 5, Task A.
Sharing experiences and case studies through the c&c toolbox
One easy and effective way to share your experience of implementing adaptation activities is to fill in the case study template in Step 5, Task A and provide it for upload into the c&c toolbox. Adaptation options that show the intended results and promise to be a solution to increase adaptive capacity serve as an example for others in the coffee sector.
The c&c toolbox is continuously updated and amended. You are encouraged to become part of the c&c network by enriching the toolbox with your experience. This can be done by creating your own case study of implementing a certain adaptation option and sharing it with the c&c network. Further information about the case studies can be obtained through direct contact with the c&c team.
List of websites and information hubs
The following are useful sources of further information on good M&E practices:
UKCIP’s AdaptMe Monitoring and Evaluation for adaptation tool: https://climate-adapt.eea.europa.eu/en/metadata/guidances/adaptme-toolkit-adaptation-monitoring-evaluation
Learning to ADAPT: monitoring and evaluation approaches in climate change adaptation and disaster risk reduction – challenges, gaps and ways forward https://www.ids.ac.uk/download.php?file=files/dmfile/SilvaVillanueva_2012_Learning-to-ADAPTDP92.pdf
CARE’s Community-Based Adaptation Toolkit https://www.un-redd.org/document-library/care-community-based-adaptation-toolkit
CARE’s Manual “Participatory Monitoring, Evaluation, Reflection and Learning for Community-based Adaptation” (PMERL) https://careclimatechange.org/wp-content/uploads/2019/06/2014_PMERL.pdf
World Resources Institute report “Making Adaptation Count” providing concepts and options for Monitoring and Evaluation of Climate Change Adaptation pdf.wri.org/making_adaptation_count.pdf
UKCIP Guidance Note: “Twelve reasons why climate change adaptation M&E is challenging” https://www.betterevaluation.org/sites/default/files/2014_01_SEA_Change_UKCIP_GN1_12_Reasons_why_CCA_MandE_is_challenging_0.pdf
Practical Guidance
Objectives
- To ensure that the findings from the evaluation are shared effectively with others who would benefit from them.
- To ensure that necessary changes are made to future plans.
Expected outputs
- A case study written in the c&c template and uploaded to the c&c toolbox.
- An assessment of future decisions that need to take into account the impacts of climate change, and the timing of such decisions.
- As required, a plan for taking identified actions forward.
- Completion of part E of the evaluation plan.
Required time
Approximately half a day per case study. Filling in the templates can be done in 30 minutes to an hour depending on what planning discussions have already taken place and the group’s level of understanding.
Guiding questions
- Has there been an increase in awareness of climate change adaptation in farming households and the wider stakeholder group involved in the implementation process? Who led this?
- What kind of stakeholder engagement was most successful in building capacity to adapt and why (e.g. training, interviews, field visits, etc.)?
- How can the c&c approach, or specific aspects of it, be improved (e.g. the Climate Witness Workshop, FFS, etc.)?
- What differences were there between the adoption rate by female and male farmers for particular practices? What differences existed between how easy and/or hard it was to encourage female and male farmers to adopt the suggested adaptation measures?
- How would you prioritize the different adaptation options that were implemented, keeping in mind their effectiveness towards climate change adaptation, but also the feasibility to implement them with both female and male farmers, as well as their families?
- How costly was the implementation? Were there costs (e.g. start-up costs or maintenance costs) that only became evident later?
Procedure
- Document lessons learned by developing a case study for the c&c toolbox.
- Summarize what you learned in the implementation process of the c&c approach (covering all activities in Steps 1 to 4) using the template provided in the c&c toolbox. This summary can then be uploaded to the c&c toolbox in order to support others in the development of their own adaptation activities. The case study template (see example on page 166) provides a useful format to help you reflect on the lessons learned in relation to the tools used and activities implemented, and also on other important issues such as stakeholder participation and the role of gender.
- Use the implementation and activity planning templates to ensure your recommendations inform future plans (more about implementation and action planning below).
Implementation and activity planning templates
Implementation template
Once you have a number of recommendations, this template can help you get a sense of priority and see how easy a particular activity is to implement (e.g. in relation to cost, timing, who needs to be involved, etc.).
Activity planning template
This template can be used if you have a list of recommendations from the evaluation that you would like to include in future plans.
Participatory tools for evaluation
Spectrum lines
Objectives
- To share and collect basic information along a spectrum from one extreme to another.
- To share and collect ideas on a single dimension of an issue, e.g. the degree, extent or amount of something.
When to use spectrum lines
Spectrum lines are very versatile. They can be used to collect basic information about who participated in training or c&c activities and how participants felt about the support they were given. They provide a sense of how confident farmers feel about building climate resilience. If you want to get extra data on a particular question, e.g. if there are differences between men and women, you could ask men and women to use different colored marks.
How to use spectrum lines
There are two main types of questions you can ask on spectrum lines:
- A question about basic information with objective values marked on the spectrum, e.g.: “How much coffee does your farm produce each year?”
- A question that asks for impressions and attitudes, e.g.: “How likely are you to recommend the training you received to other coffee farmers?” or “Support to produce an adaptation plan”
These spectrum lines allow you to probe further into why a participant has placed a mark on a particular spot, either through direct questioning or asking participants to mark it on the line, like in Figure 51.
You can start to consider what would have to change to make them move it to the right or left. Also watch for trends over time, as well as hopes and ambitions, by asking participants to mark where they were in relation to this question a year before and where they might like to be a year (or some other time period) in the future.
Objectives
- To draw out what supports and what hinders a particular line of action.
- To evaluate progress by undertaking an H diagram at different points of time.
- To share a range of views on level of progress, and what supports or hinders this.
- To identify what next steps would be most effective in making progress towards a desired goal.
When to use an H diagram
The H diagram can be adapted to fit a number of applications, e.g. as a tool to assist locals in evaluating the performance of partnerships, programs, agencies and initiatives; as a tool to identify local indicators for M&E, as a way to identify next steps and priorities for action and in the evaluation of workshops and training.
How to use an H diagram
The H diagram is shaped like a wide letter ‘H’ that can be used in numerous settings to rate something along a scale of zero to ten (e.g. confidence in responding to climate change and assessment of the level of farmer participation in a decision-making process). It provides an easy-to-understand, visual representation of participant responses that is assessed to determine what is going well, what is blocking progress and how the situation can be improved. ´
Objective
To understand the factors that influence a particular situation, either by driving movement toward a particular goal (motivating forces) or blocking such movement (constraining forces or barriers). These forces can be very dynamic, varying both over time and with the experience and awareness of those tasked with identifying them. They can include aspects such as motivations, values, needs, personalities, goals, anxieties and ideals, as well as more structural aspects of organizational decision-making.
When to use force fields
Force fields can be used in all stages of project design and planning. It can also be used to analyze why an adaptation process has evolved as it has and plan how the project might change in the future. Force fields help you to understand what positive factors have contributed to progress and also what obstacles have been overcome.
How to use force fields
- First, decide which issue you will use the force field to explore. For example, “assessing what supported and what constrained the implementation of an adaptation activity plan”.
- Next, encourage participants to list all the forces that supported the implementation of the adaptation activity plan and write these onto pieces of paper with arrows on one side of the paper.
- List all the forces that got in the way of implementing the activity plan and add them to the other side of the paper. Each driving or constraining force can then be ranked by placing it closer to or further from the center of the paper, according to the strength of the force.
- Encourage participants to explore how you might increase the driving factors and decrease the constraining factors. You can identify what next steps you might take to put this into practice. For example, in the illustration below, one of the things that slows down progress is listed as “the information about how a climate is changing seems uncertain”. As it is unlikely that you can increase the uncertainty of the information, you could discuss how you might make a decision despite this uncertainty, e.g. by exploring ‘no-regret’ decisions, which are still useful in a range of different future climates.
Objectives
- River of life exercises can be used to reflect on the progress of an adaptation process to date, to identify what worked well and what difficulties were encountered.
- To share diverse perspectives on the same project and identify common ground.
When to use rivers of life
This tool that can be used to help people get to know one another, reflect on their relationships, explore hopes and fears about a new venture, reflect on what was surprising or difficult in a project that has now ended.
How to use rivers of life
- Using the metaphor of a river, ask participants to draw a picture that reflects their experience of participating in the adaptation process. They can either draw their own picture or one as a group.
- Have them reflect on their experiences and identify positive aspects that were achieved, key choices (represented by tributaries), and challenges or setbacks encountered (rough waters, rocks or monsters). Other images (boats, fish, bridges, rocks, marsh/shallow ground, etc.) can be used to represent different aspects of the journey.
Objectives
- To represent how participants see all aspects of a situation, including the issues, people, problems, processes, relationships, conflicts and drivers.
- To share what participants see as important, why they may be feeling stuck and where they might start looking for ways to improve the situation.
When to use rich pictures
- Rich pictures can be used in the evaluation stage to share perspectives on the purpose of evaluating an adaptation process.
- Depending on the question asked, rich pictures can also be used to compare different experiences of the same process and the reasons for these differing experiences, or how different participants value different aspects.
Participants often worry about the standard of their drawing skills, but often end up producing pictures that demonstrate their thoughts and concerns far more concisely than straightforward answers in an interview. Below are some examples of rich pictures illustrating how a group of participants perceived an adaptation process they were part of.
How to use rich pictures
- Begin with large sheets of paper and lots of colored pens and ask participants to simply draw what they see happening in a specific situation, or in response to a particular request. They should use pictures to represent the situation in a way that can be communicated to other people. This should take between 10 and 15 minutes.
- Give them instructions, such as the following:
- “Draw a picture that explains your experience of participating in this process.”
- “Draw a picture that explains how you are currently feeling about responding to climate change.”
- Have them explain their pictures to another participant or to the larger group (if they are willing). Sharing different rich pictures of how people see the adaptation process allows you to identify connections, differences, opportunities and contradictions that might have been missed in straightforward discussions. It allows participants to question their assumptions about what is happening, which may result in rethinking how they understand the situation and thus influence what is to be evaluated. A facilitator can serve to identify where common themes and different ideas emerge.
Objective
Conversation maps present different understandings and insights of a situation that emerge from a conversation.
When to use conversation mapping
This tool can be used to capture different observations, feelings and general reactions to a ‘trigger’ issue. The trigger issue is normally phrased as a question, e.g. “What was your experience of participating in the implementation of climate change adaptation options?” Or, “What did you most value from your participation in the implementation of climate change adaptation options?”
How to use conversation mapping
- Write the trigger issue in the center of a large piece of paper with a circle drawn around it.
- Ask one participant to respond to this trigger to start a conversation which is then recorded on the paper and linked to the central trigger.
- Have the others respond to this or the central trigger and also record their contributions so that the map branches out. There should be no attempt to shape this process as this is a tool for understanding the whole range of perspectives with no attempt to collect or analyze them at this stage.
- If the map becomes too large for the paper, tape on new paper. It is normal for the map to become very messy and full of information.
- When people have had enough (or time runs out), it is time to reflect on the map (and the process of making it). Ask participants what the map shows and about their perceptions or understandings of the trigger. Draw out key themes and anything that has changed in people’s perspectives as a result of creating the map.
Ranking exercises
Many different types of participatory ranking exercises are useful for an evaluation – particularly in the data gathering and analysis stages – to determine priorities for action or how different options perform against an agreed set of criteria, among other information. Latter ranking was described in “Designing a plan to gather evidence” of Step 5.
Venn diagrams
Venn diagrams are a way to represent the relationships between stakeholders and the power differences between them. They can be used in an evaluation to compare how the relationships between key stakeholders were thought to be at the start of the implementation of adaptation options with how they were at the end. They also demonstrate how relationships change and why. Venn diagrams are described in the Section 2, Step 2.
Additional visual and narrative approaches
There are a number of additional participatory tools that can be used to gather more qualitative evidence, which is often necessary to answer other types of evaluation questions, but requires more time and resources. The approaches are not discussed in depth here, but are simply mentioned in order to show the range of what is available for those who would like to conduct an evaluation process that investigates the experiences of participants in more depth. For more information about each of the approaches, please follow the links provided.
Participatory theatre
Participatory theatre is a multi-stakeholder dialogue that uses theatre as an informal, creative but serious way of exploring different perspectives and responses along the coffee value chain. It allows different actors to exchange information, formulate and prioritize issues and identify collective solutions through short story writing and performance.
It does not involve a formal play or production. The term ‘theatre’ simply refers to participants as being both ‘actors’, who narrate, write and act short stories, and as being an ‘active audience’ that watches short plays. The idea is that, following each short story, a dialogue is held between the actors and the audience, in contrast to traditional theatre in which the actors present and the audience listens. The role of the audience is to help each group of participants to refine the problem and the solutions presented.
This approach has been used extensively across the world (it is sometimes called ‘theatre for development’) and has been especially successful with the coffee sector in Uganda.
A participatory video is a tool that can be used by participants to tell the story of what has changed as a result of implementing adaptation to climate change in their area. The finished film can be used to present their ideas directly to funders and decision-makers outside of their local area. Thus, their learning can be shared more widely to influence decision-making processes beyond their local context.
Participatory photo stories
Photo stories can be used in a similar way to explain how livelihoods, vulnerabilities and opportunities have changed throughout the adaptation process, and what has influenced these from the point of view of a specific participant, e.g. the coffee farmer.
This is quite a simple tool. You simply lend each participant or group of participants a camera and ask them to take pictures of things they feel represent something about their experience in the adaptation process. This can include what has worked well and what has not, and any surprising things that were perhaps unexpected.
You can also be more specific with instructions. E.g. ask participants to take 10 photos of things they think went well, and 10 photos of things that represent whatever needs further improvement. When cameras are returned and photos printed, you can use these to compare and contrast, and to stimulate discussion. Look for overlaps of where people photographed the same things.
Outcome mapping was developed by the International Development Research Centre in Canada (IDRC) as a way of planning, monitoring and evaluating with a focus on what contributes to outcomes made by interventions, rather than attempting to directly measure the attribution of change to a particular organizational intervention.
Outcome mapping has a lot to offer in the evaluation of adaptation options, as it accepts that adaptation processes are complex and dynamic; it recognizes that there may be unexpected and unintended consequences of a given option. It also acknowledges that outcomes arise as the result of a number of factors rather than a single action. Outcome mapping provides a way to draw together different contributions to an outcome, which is essential in order to learn more about what supports successful adaptation.
This approach is based on listening to what people (beneficiaries, participants, stakeholders, etc.) think have been the most significant changes resulting from an adaptation process. It does not require any professional skills to facilitate and is easy to communicate across cultures, as people generally find it easy to tell stories about events they think were important.
There is also no need to explain what an indicator is and it is a good way to pick up unanticipated changes and changes that may challenge assumptions of what is happening. This approach encourages people to engage in the analysis as well as data collection stages of an adaptation process, as they have to explain why they believe one change is more important than another. It can be used to monitor and evaluate bottom-up initiatives that do not have predefined outcomes against which to evaluate.
A learning history is described as a ‘jointly told tale’ between outsider researchers and insider actors around a tangible outcome. It aims at bringing together analysis and story in a way that has value both for those originally involved in the work and others seeking to learn from it.
A learning history account attempts to get into the ‘messy’ human aspects of what happened during implementation. It aims to present a number of perspectives on a situation rather than synthesizing several accounts into one story. It is presented as a multi-voiced and multi-leveled account that charts what happened, and provides quotes from those involved and reflections from researchers during the process and analysis of .