Identifying why, what and who for your M&E process

Abstract

This task is about exploring and defining what are you hoping to achieve from the M&E process, what you are trying to monitor and evaluate and whom you need to involve. Addressing these three issues will allow you to identify evaluation questions and develop a plan to collect information and analyze it in a useful way.

Method

Focus group discussions or one-onone discussions moderated by local extensionists
Participatory exploration methods to gather and share differing perspectives such as conversation mapping or rich pictures
Participatory ranking exercises to prioritize which areas to focus on in M&E processes
Cross-checking with previous stakeholder engagement work (Steps 2 and 3)

Expected Results

A shared understanding of what stakeholders would like the M&E plan to achieve
Clear boundaries for the scope of the evaluation, e.g. is it evaluating a single option, a set of options, a program? Which impacts are being considered (e.g. drought, increase in disease, etc.)?
Clarification of which stakeholders are to be involved and how they will contribute to the M&E process

Theory

Table 16: Identifying the purpose of the M&E process

What is your motivation for undertaking an evaluation at this time?	Further considerations: What aspects of the project do you want to know more about? Who are the audiences for the evaluation and what are their needs?
Do you need to demonstrate to others that you have done what you said you would do?	What evidence is needed to demonstrate this?
Do you need to demonstrate how successful the adaptation activities were?	How might different stakeholders define success? What evidence is needed to determine success?
Do you want to share what is working well and what supports this?	With whom do you want to share this? What formats would be most useful for them (e.g. technical information or stories of people’s experiences in adaptation)? What level of detail is required? How can you check this?
Do you want to improve decision-making processes?	What particular decisions are you interested in? What information do you need to gather about the context of these decisions?
Do you want findings to act as a guide to future work?	Which aspects might be of most relevance to future activities? Would it be beneficial to synchronize your evaluation with future planning cycles?
Do you want to motivate other people to act?	Whom do you want to motivate? What sort of evidence would be most useful to them? What would be the best way to communicate key messages to this group?

I. Why: Identifying the purpose of your M&E process

It is important to be clear on why you want to undertake M&E activities, as this will shape how they are designed, who becomes involved and the evidence you choose to gather.

Reasons to undertake M&E:

To demonstrate that you have done what you said you would do
To track progress
To share what is working well and what supports this
To improve the process of decision-making
To guide future work
To motivate other people to act

In addition to assessing the technical performance of adaptation measures (e.g. is a rainwater harvesting system gathering water effectively?), M&E also examines the outcome of these measures and how well they worked in different situations.

For a rainwater harvesting system, this might include comparing the efficiency of this system to other available options, measuring cost-effectiveness, determining who benefits from the system (and who does not) and establishing whether the system has any negative consequences (e.g. encouraging the use of excess water).

When identifying a purpose, it is also important to consider when you intend to undertake the evaluation aspect of your M&E approach. For example, if you are conducting a mid-term evaluation, an important purpose might be to understand how aspects of the project might be improved or enhanced in the short term.

Ensure that you discuss the purpose of your M&E process with key stakeholders. If you explain why the evaluation is taking place, and initiate a dialogue on the benefits of the M&E outputs (including the benefits for them), stakeholders are likely to engage with the process in a more positive way.

II. What: Identifying what you are monitoring and evaluating

It is also important to consider what exactly you will be monitoring and evaluating. There are a huge number of possible adaptation options and approaches in response to a range and combination of possible impacts (heat, rainfall, disease, wind speed, etc.). You may also be monitoring and evaluating a single adaptation option or a whole project or program. However, given the objectives of c&c it is reasonable to assume that you will need to monitor and evaluate:

How your adaptation options are helping build the resilience of smallholder coffee farmers to climate change
How your adaptation options are improving the adaptive capacity of smallholder farmers

These broad themes will require further refinement. Setting clear boundaries for what you are going to evaluate will help you select the most appropriate methodology. By carefully considering what you are monitoring and evaluating, and linking this to the objectives outlined in your project pathway, you should have a strong foundation on which to develop a) appropriate indicators to monitor and b) a strong set of evaluation questions.

Guiding questions for defining what you are monitoring and evaluating:

Which direct or indirect climate impacts are you responding to (e.g. reducing the adverse impacts of drought, disease outbreak, landslides, etc.)?
How will you build resilience (e.g. through improvements to coffee plants, diversifying household income, improving access to markets, etc.)?
Will the work focus on a particular beneficiary group?
Are you looking to build the capacity of farmers directly or by training extensionists?

Learning from experience

Example of individual adaptation option: Use of deeper polybags

For interventions following the c&c approach, M&E and learning is best achieved when integrated into the different steps alongside the approach. In the c&c pilot project in Brazil, drought was identified as a key problem in the triangulation process in April 2012. A detailed cost-benefit projection from later in 2012 identified seedlings in large polybags as a promising adaptation option: seedlings remain in the nurseries for an additional six months and are delivered in larger polybags. The price per seedling was higher, but mortality was predicted to be lower and initial yields superior.

A nursery was found to produce these seedlings and farmers were invited to use them – and encouraged to plant them directly alongside normal seedlings from small polybags. This allowed for close monitoring of how the plants performed. After one year, the seedlings were assessed and compared to the control group on the same plot. Mortality had dropped by 20% and all other indicators, from plant height to stem diameter, showed a strong positive effect of using large saplings. Farmers from other groups were invited to visit the plot and to observe and discuss the effects in the field. This convinced farmers to try out this alternative method as well.

The formal assessment of plant parameters and the discussions with farmers formed the foundation for a case study. The tool was rated by extensionists according to its effectiveness (e.g. if it achieved its purpose), acceptability (e.g. if farmers were willing to accept the new technique), affordability (e.g. if farmers afford to use this tool) and timing (e.g. when benefits occurred). It was only on the criterion of timing that the tool received an unfavorable rating, as farmers incurred costs while planting and the benefits only occurred when the trees started yielding.

When the trees on test plots start producing, a more formal cost benefit assessment of the tool can be done, again using the set-up of large seedlings and small seedlings on the same plot for comparison reasons.

For more detailed information, see the tool Deeper polybags.

III. Who: Planning whom to involve in the M&E process

Deciding who will take part in the M&E process and the roles they will play requires a balance between including everyone with useful knowledge and experience, and managing what is practical in terms of the available time and resources.

Guiding questions for planning whom to involve in M&E:

Who is responsible for what is being monitored and evaluated?
Who is expected to benefit from, or be affected by, what is being evaluated (directly and indirectly)?
Who is able to influence what is being monitored and evaluated?
Who is able to affect whether the outputs of the evaluation are implemented?

As a minimum, key people such as farmers and extensionists should contribute by providing information and experience. However, more participatory approaches to monitoring and evaluation can be especially useful. These require more active involvement of farmers and other stakeholders in the development and design of the M&E process. They would also play a key role in discussions on how the success of the adaptation process is defined, the evidence that is needed and data collection and analysis. The role of the person or team responsible for implementing M&E processes can thus shift from full control of the whole process to a facilitation role for others.

Guidance

Participatory monitoring, evaluation, reflection and learning for community-based adaptation (PMERL)

A ‘good’ facilitator serves mainly as a catalyst or stimulator, rather than a leader, drawing out and bringing together inputs from different types of stakeholders. This requires key skills of negotiation and in some cases conflict resolution. Facilitators should ask the right questions at the right time listen well, build trust, encourage the sharing of ideas, and at the same time keep the group focused.

Being clear about how you decide who will participate will help people understand what is expected of them. Bringing coffee farmers and other local stakeholders together to evaluate and learn from the work requires methods that appreciate the value of different perspectives and different types of evidence (e.g. opinions, experiences, factual data and cultural values). Some participants may need additional support (e.g. time, guidance, access to data) to enable them to contribute effectively.

A facilitator can be used to build the M&E team’s capacity to engage people, support fair participation and encourage locals to take an active role in managing the M&E process. If there are significant power differences between different participant groups, it may be worth meeting each group on a one-on-one basis before bringing them together. This will allow them to consider their individual perspective before sharing it with others.

Also refer back to the stakeholder mapping created in Step 2. Many of the stakeholders identified and engaged during the assessment and adaptation planning stages may also need to be engaged in the M&E process. Reflect on how these stakeholders have been involved to date and consider how they might contribute to, or benefit from, the M&E process.

Table 17: Principles of participatory evaluation from PMERL

Participation	Design your M&E to include those most affected by the work being undertaken.
Negotiation	Encourage open discussion about what will be monitored and evaluated. It should not just be based on the views of the most influential people.
Learning	Everyone involved in the evaluation process should be open to learning and should be supported to do so through access to information, sharing of experiences and facilitation of reflection to think more deeply about the practical implications of these findings.
Flexibility	Allow plans to change over time to incorporate new learning and understanding.

for deciding whom to involve and in what ways:

Who has a useful perspective or evidence to offer that is either affected by the adaptation process or influential in it (e.g. project managers and field staff, local partners, local NGOs, local government, communities?)
Whose absence will mean important information is missed? What would hinder their presence and how can this be avoided?
How can you support the participation of vulnerable coffee farmers in an evaluation process that might feel quite unfamiliar to them?
Whose capacity for monitoring should be strengthened to ensure the sustainability of the process?
Who should be involved in making sense of what is collected?
Do those involved (scientists, farmers, advisors, funders, etc.) value different types of information equally? If not, how can this be managed?
Will participants change over time? How will this be managed?
Are there any ethical implications that should be considered when engaging people in M&E?
How much do ‘upstream’ factors (e.g. institutions, markets, governance) affect what can be achieved at the farm level? What are the implications for this and who needs to be involved?

Practical guidance

The following section will provide suggestions for exercises that will help you to develop and implement an M&E plan, which was introduced in Step 5. You are not required to use every exercise suggested here or to follow any particular sequence, but rather to choose what is appropriate for you – exercises work better for some groups than others.

Objectives

To identify and agree on the purpose of the M&E process.
To identify the scope of your implementation.
To identify who needs to be involved in M&E and in what ways.

The overall purpose of the c&c initiative is to build the resilience of the local coffee sector to climate change. However, it is important to contextualize this broader purpose for the local level and to think through the attributes that a resilient local coffee production system might have (see introduction in Section 1). Use the following table as a template to record the outcomes of each of these steps. Once you have completed each task and filled in each section of the template, you will have a full M&E plan.

Expected outputs

A clear purpose and learning objective for the evaluation that has been accepted by those implementing and/or affected by the adaptation process.
If time and resources allow, a detailed stakeholder engagement plan that explains who should be engaged when and for what purpose.
Completion of part A(I-III) of your evaluation plan (purpose, scope and whom to involve).

Required time

As required to understand the purpose and given the resources available.

Procedure

I. Define a purpose

Refer back to the original project objectives, as the purpose of the evaluation is likely to be closely linked to these.
Discuss the guiding questions below with key people involved (extension workers, funders, farmers, etc.) either in focus groups or oneon- one to find out what they would like to gain from the evaluation. The following list of common reasons for evaluations may be helpful in discussions. Decide which apply to your evaluation specifically:
- to evaluate effectiveness
- to assess efficiency
- to understand equity
- to provide accountability
- to assess outcomes
- to improve learning
- to improve future activities or interventions
- to compare outputs with other similar activities or interventions
Identify where there are agreements or potential conflicts between those involved and how they relate to the purpose of the evaluation.

II. Define the scope of the project

Referring back to the project objectives and project pathway, identify the key areas of focus for your evaluation, e.g.:
- Will it focus on implementation of a single adaptation option or a series of options?
- Which hazards are of interest, e.g. increased drought, increased heat, storminess or a combination of hazards?
- Which groups are of interest, e.g. all local coffee farmers and their families, coffee growing communities or only farmers?

III. Determine whom to involve and how

Refer to any previous stakeholder analyses made during the adaptation process and identify which groups, organizations and individuals have been involved to date and how exactly they have been involved.
If no previous stakeholder analyses have taken place or if they are incomplete, participatory exercises can help you map out who has been involved and how they might contribute to M&E.
It is important to be clear about:
- who is responsible for the evaluation
- who is expected to benefit, or be affected by, the evaluation
- who is able to influence the evaluation
- who is able to affect whether the outputs of the evaluation are implemented
If you have the time and resources, use the following template to create a stakeholder engagement plan for the evaluation.

Table 38: Stakeholder action plan (template)

Stakeholder action plan
1	Define a purpose What do you want to achieve in the evaluation process? You can then decide who needs to be involved in order to achieve this objective (refer to part A of the M&E plan).
2	Identify stakeholders Who needs to be involved? Refer to previous work on stakeholders and any outputs from past exercises, e.g. brainstorms, Venn diagrams, influence and importance matrixes. Establish: who is responsible for the evaluation who is expected to benefit, or be affected by, the evaluation who is able to influence the evaluation who is able to affect whether the outputs of the evaluation are implemented Also ask: How much do larger factors (e.g. institutions, markets or governments) affect what can be achieved at the farm level? What are the implications of this on stakeholder involvement?
3	Establish roles Who takes on which responsibilities? Roles in M&E include change throughout the process e.g. : defining the purpose setting the scope of the evaluation questions designing how to gather evidence gathering evidence analyzing evidence making recommendations reporting findings reviewing the report and recommendations communicating the report
4	Understand stakeholders What do you already know about these stakeholders? What can already be figured out from previous discussions about the stakeholders, including their perceptions, knowledge, interests, decisions, power differences, and patterns of interaction among themselves and with others?
5	Select stakeholders Which individuals can act as good representatives for their organizations or communities in the M&E process? Once you have identified relevant categories of stakeholders, you now need to get the names of appropriate people.
6	Look at logistics When is the best time to approach these stakeholders? Is it best to involve them at the beginning, at a later stage or throughout? In which season, on which day of the week or at what time of day will you approach them? Where is the best place to meet stakeholders? Will you meet them separately or in shared meetings? In a community building, a public space, at home or at their place of work?
7	Anticipate problems What problems can already be identified? Do those involved (scientists, farmers, advisors, funders, etc.) value different types of information? If not, how can this be managed? How can you support the participation of poor coffee farmers in an evaluation process that might feel quite unfamiliar to them? Will participants change over time? How will this be managed?

Guiding questions for identifying why, what and who

What do you see as the purpose of your evaluation?
What would you like to learn? Who else should be learning, what should they be learning and how might this best take place?
How might you manage conflicting purposes? What trade-offs might you have to make and can these be justified?
Who are the audiences for the evaluation and what are their needs?
What do you already know about the stakeholders from the previous steps?
Which individuals can act as good representatives for their organizations or communities?
What logistical issues in engaging stakeholders can be anticipated?

Technical methods

Focus group discussions or one-on-one discussions moderated by local extension officers.
Participatory exploration techniques to gather and share different perspectives on the situation, such as conversation mapping or rich pictures.
Participatory ranking exercises to prioritize the focus areas of the evaluation.
It is likely that a stakeholder analysis will have already been done at an earlier stage in the process, meaning you may need to simply revisit this work. If no previous stakeholder analysis has occurred, suggestions for methods to help you undertake a stakeholder analysis are given in the Section 2, Step 2.