Abstract
In the analysis, the evidence is reviewed to assess progress, next steps are identified and lessons are shared with others. This stage is an opportunity to bring key people together to share perspectives on what has worked well, what supports and constrains it and whether capacity for resilience has been developed. This can be done through a learning workshop.
- Learning workshop for everyone involved in the project to assess the information emerging from the data collection (this involves group discussions and ranking exercises)
- More ambitious narrative approaches to make sense of the emerging evidence (e.g. participatory theatre, videos and photos)
- Deeper understanding of evidence gathered to help answer evaluation questions
- Identification of gaps in data availability and challenges to assumptions
- ‘Stories of change’, which explain how the project has built capacity and increased resilience to climate change
Theory
You may find it useful to return to both your evaluation questions and your project pathway for the analysis. Your evaluation questions are a good starting point and may provide a useful structure on which to base your analysis, while your project pathway may help you understand what you have learned about the assumptions you originally made and the outcomes you expected.
It is important to give those who participated in gathering evidence an opportunity to see what resulted from the process and offer their feedback. Sharing the task of analysis with the key people involved, including farmers, is more time-consuming than if only the evaluation team is involved, but it has many benefits, e.g.:
- It provides an opportunity for checking how trusted the data collected is, thus increasing its quality and depth.
- Participants are able to see where their ideas and experience resonate with other groups and where there is disagreement.
- There is an opportunity for ‘co-learning’ between participants through combining different perspectives, seeing underlying patterns and drawing out questions for further reflection.
- It increases the confidence participants have in their knowledge and ability to contribute to wider decision-making processes, and their capacity to question assumptions about how to build resilience – which is useful for building resilience in the future.
- It improves an understanding of the wider coffee production system and where changes need to be made in order to ensure resilience in the long term.
- Any next steps devised are likely to be more relevant and useful if they have been developed in conjunction with those involved in the implementation.
The evidence can be analyzed to answer the key evaluation questions. For example, for the question “What are the most effective activities for reducing rust?” case study write-ups and evidence from farmer interviews could be brought together into a table, e.g.:
This table could be presented to farmers again at a learning workshop or in interviews to get their feedback on whether this seems correct, what is missing and what new questions the findings pose.
However, if your evaluation question was about whether capacity for resilience has been developed by farmers, identifying characteristics of resilience could require a qualitative assessment of how well these characteristics were demonstrated in the work to date. This should highlight areas where things are going well and areas where improvements can be made. Record the key outputs from this task in the M&E plan template, see Step 5, Task A.
Table 19: Example evaluation of adaptation option
Affordability | Acceptability to farmers | Timing | Effectiveness | |
Planting rustresistant variety A | *** | * | ** | * |
Planting rustresistant variety B | ** | * | ** | - |
Planting shade trees | * | *** | * | ** |
Planting only above 1,000 ft | n/r | n/r | * | *** |
*** = very good, ** = good, * = fair, - = poor, n/r = not relevant | ||||
Practical Guidance
Objectives
- To verify that the evidence gathered can be trusted.
- To make sense of different types of evidence so that it can be used to answer the evaluation questions.
Expected outputs
- An assessment of the evidence in relation to how it answers the evaluation questions.
- Key messages from the evaluation process that can be used in broader dissemination.
- Completion of part D of your evaluation plan.
Required time
Variable depending on the approach, how many people are engaged and the depth of the information needed.
Guidance
Verification
It is important to verify that the evidence gathered is accurate and trustworthy. Checking information takes time, but it increases the quality of data considerably. Checking back with participants helps you gain a deeper shared understanding of the issues and allows you to uncover additional insight. It is important to be clear about what is being verified. For example, you may want to:
- verify who has taken part and whose views have and have not been expressed
- verify where there is agreement and where there are different perspectives
- check that the views expressed represent what people genuinely believe and that the learning that emerges from an exercise has been properly understood and represented
With thanks to
Procedure
- Consider organizing a learning workshop for key stakeholders in order to both provide evidence and assist in the analysis (this would address both parts C and D of Step 5). At each stage of analysis, judgments need to be made about which pieces of evidence are most useful for answering the evaluation questions. You should start the analysis stage with a set of untested ideas and end up with a smaller number of key messages and recommendations. Facilitators should be in place to ensure the processes of recording, presenting and prioritizing evidence takes place and that the workshop stays on track.
- Even if a full learning workshop is not feasible, there are a number of exercises that can be used with stakeholders in interviews and focus groups to help collect and prioritize lessons from the evidence.
- Fill in the M&E plan template to help organize future actions.
Technical methods
- Many of the participatory tools mentioned previously for data gathering can also be used in the analysis stage, in methods such as conversation mapping, rivers of life, ranking, H diagram, force fields and, if time allows, narrative approaches such as participatory photos or learning histories. These are described at the end of this section.
- A learning workshop is also helpful for identifying exercises that enable participants to move from a broad understanding (e.g. brainstorming) to a focus on key issues. This will allow them to draw out key lessons and transferrable messages and to identify the short-, medium- and longer-term steps for building resilience to climate change.
Exercises that might help in carrying out specific aspects of the analysis include:
I. Determining whether activities were carried out in an efficient, affordable, appropriate and timely way
For each of the activities implemented in your adaptation process, assess how well they met the criteria of being efficient, affordable, appropriate and timely. This may require prior discussion about what the words mean in relation to the aims of the adaptation process, i.e. the people involved in implementing the project should identify a locally defined range for each word in relation to their specific coffee production.
Use a table such as the one below to record these definitions (Table 43).
Table 43: Assessment of implemented adaptation activities
Criteria | Excellent | Satisfactory | Poor | Unsatisfactory |
Efficiency | Activity achieved the intended goal very effectively | Activity achieved the intended goal reasonably effectively | Activity was poor at achieving the intended goal | Activity did not achieve the intended goal |
Affordability | Activity was considered to be excellent value for money | Activity was considered reasonable value for money | Activity was considered to be expensive | Activity was so expensive that farmers would not consider using it |
Appropriateness | Activity was very suitable for implementation by the people required to do so | There were few concerns about the suitability of the activity for those implementing it | Concerns were raised about the suitability of the activity by those required to implement it | The activity is unsuitable for use by those required to implement it, e.g. because it was too strenuous, culturally unacceptable, etc. |
Timeliness | The activity fit easily into the existing schedules of the implementers | There was an acceptable fit into the existing schedules of the implementers | There were concerns about how the activity fit with existing schedules | The timing implications of the of the activity made it impossible to fit into existing schedules |
II. Evaluating the effectiveness of stakeholder engagement
Resilience to climate change requires a connection between different people involved in coffee production and their wider networks, e.g. between the community, government or businesses. It also requires flexible processes and opportunities to learn from experience and to make sure that learning is used to inform future plans. At the end of an adaptation process, it is important to assess how well farmers and other stakeholders were able to participate in the design, implementation and analysis.
The following table provides some suggestions for collecting evidence that can answer questions about the level of stakeholder inclusion in design and decision-making, or stakeholder investment in . Feel free to adapt or add to this list to suit your needs. These questions can be answered through focus groups, one-onone interviews or in a workshop setting. There will inevitably be an element of subjectivity in assessing the quality and level of participation. However, through these conversations, it should start to become clear what meaningful participation means for farmers and whether they (as well as other stakeholders) felt that opportunities to participate were appropriate and sufficient and, if not, how the approach might be improved next time.
Table 44: Example for collecting evidence of stakeholder engagement
Questions about stakeholder participation | Example indicators |
Who or what determined the rules of implementation? |
|
What is the balance of power between different people and organizations? |
|
At what stages of the process are coffee farmers and local communities involved? |
|
How much influence/control do coffee farmers and local communities have on decisions? |
|
What investment is made in developing and sustaining coffee farmer (and local community) participation? |
|
What investment is made in developing and sustaining coffee farmer (and local community) participation? |
|
Does the participation strategy allow for a variety of ways into the process for farmers and local community representatives? |
|
How accessible are local meetings? |
|
Are local project meetings run in an effective and inclusive way? |
|
III. Summarizing key messages about which activities are most effective
It is helpful to use a ranking exercise that assesses how different participants viewed each activity in the adaptation process. There are many ways to conduct ranking exercises. A simple approach is to give participants four stones (or a different small object, such as a piece of candy) and ask them to place them on a table marked out on a large piece of paper. For example, if you were comparing three activities, Table 45 could be used.
If a participant thought the activity was extremely efficient, reasonably affordable, and suitable for the farmers, but that the timing was terrible, their stones might look like Table 45 below.
If you asked another five participants to place their stones, you might get something that looks more like Table 46.
From this exercise, facilitators can start to pick up patterns and discuss them with the group. For example, the facilitator could ask, “There seems to be some agreement about the poor timing of this activity. Is this how you see it and if so, could someone tell me more about why this is?” Or, “There seems to be a spread of views about the efficiency of the activity. Could someone who considered it to be ‘very efficient’ tell me why they placed their stone there? Could someone who placed their stone on ‘very inefficient’ explain why they put their stone there?” It is normally during these discussions that the most interesting lessons can be learned.
Table 45: Example of assessing activities by using the ranking exercise (1 person answered)
Criteria | Excellent (++) | Satisfactory (+) | Poor (-) | Unsatisfactory (--) |
Efficiency | * | |||
Affordability | * | |||
Appropriateness | * | |||
Timeliness | * |
Table 46: Example of assessing activities by using the ranking exercise (5 persons answered)
Criteria | Excellent (++) | Satisfactory (+) | Poor (-) | Unsatisfactory (--) |
Efficiency | ** | * | * | ** |
Affordability | ** | *** | * | |
Appropriateness | **** | ** | ||
Timeliness | * | ***** |
IV. Other participatory tools and exercises
There are many participatory tools and exercises that can be helpful in one-on-one discussions with farmers, focus groups or as part of a learning workshop in order to gather data and examine it in more depth. These tools and exercises can help participants map out and understand complex relationships, interactions and influences. They are listed in Table 47 and are described in more detail at the end of this section.
In addition to evaluating the adaptation process at the farm level, it might also be interesting to consider how the broader aspects of the coffee production system can also be addressed. Extension workers, trainers or others using the c&c approach can use Table 48 as a checklist to reflect to reflect on how resilience to climate change can be built for coffee production on a larger scale. Good practice at the farm level needs to be supported by good practice in other parts of the coffee production system if it is to adapt to climate change to the best possible extent.
The checklist will allow you to identify strengths and weaknesses and to consider where resources should be invested to build long-term capacity for resilience in your local coffee production system.
For example, the ability to understand new information about climate change and to assess new risks is an important skill to incorporate into decision-making processes when adapting to climate change. An extension worker should consider, for example, how well farmers are able to access and understand new data on climate change. They should consider how they can support the farmers in this and how much of a priority developing this capacity should be. They may decide that this is not a priority in relation to other more pressing activities, such as introducing new irrigation systems.
Table 47: Examples of participatory tools and exercises for evaluation
Specific participatory tools (for use in interviews, focus groups and workshops) | Other visual and narrative exercises (more time and resources needed) |
|
|
|
|
|
|
|
|
|
|
|
|