Identify your evaluation questions

Abstract

Having defined the purpose of your M&E process, it is now time to consider the evaluation questions that will help fulfil this purpose. A good starting point for creating questions is to consider the logic behind your adaptation measures and the assumptions that you made during the planning stage. If you have developed a project pathway (see Step 3), you will have already mapped out this logic and its inherent assumptions. Evaluation questions should test and challenge the logic outlined in your project pathway and help you understand what has or has not worked well and why.

Method

Group discussions with key stakeholders involved in the project design
Discussions informed by the project pathway (see Step 3), M&E purpose and the outputs of the previous discussions and workshops

Expected Results

A prioritized set of questions to create a focus for the M&E process

Theory

Table 18 below provides a general example of the types of questions you may need to consider. These need to be tailored to your particular context and adaptation options. The number of questions you select will depend on the complexity of your adaptation process, the resources you have available and the purpose of your M&E. It is best to identify three or four critical evaluation questions, which might also have further sub-questions.

Some of the most valuable lessons can be learned by looking beyond what you expected to happen and exploring what was not anticipated. This could be positive (e.g. the peer-to-peer farmer training directly led to the sharing of labor at key times during the year) or negative (e.g. the technical interventions worked well, but uptake was poor because we didn’t involve community leaders at the start).

Be sure to include some open questions to take these factors into account. For example, rather than simply asking, “Did farmers find the training useful?” ask “What did the farmers find useful during the training?” It is important to capture these lessons and also to ask, “What had most value for the people involved?” – it may not be what you expect. This will help you to improve plans for the next phase.

Table 18: Example evaluation questions for adaptation process

Aspect of the adaptation process	Example evaluation questions
The progress of planned activities (“Did we do things right?”)	Did what you achieved match what you expected at the start? Were the planned activities undertaken in an efficient, affordable, appropriate and timely way? Were your inputs sufficient in enabling you to carry out the planned activities?
The roles, responsibilities and level of engagement of farmers and other stakeholders involved in implementation	Did activities target the right people and scales to build resilience? How were key groups engaged in the activities? Who took which roles at different stages? What was their experience like?
The appropriateness of the logic in the operational plan, including assumptions (“Did we do the right things?”)	Did the activities result in the anticipated outputs? Which assumptions were challenged and in what ways? What new understanding has emerged about how change happens and what constrains and supports it? Have priorities changed during the course of the work due to external changes?
If (and how) unexpected or unintended outcomes have arisen	What was surprising or unanticipated and what challenged your understanding of how change happens?

Practical Guidance

Objectives

To identify the evaluation questions, which will determine whether you are ‘doing things right’ and ‘doing the right thing’.
To identify questions that will help to capture information that fulfills the purpose of the evaluation, as well as the following:
- the progress of planned activities and outcomes
- the roles, responsibilities and level of engagement of farmers and other stakeholders involved in implementation
- the appropriateness of the logic you used in preparing the operational plan, including assumptions about how activities help to achieve the overall purpose
- if and how unexpected or unintended outcomes have arisen and the consequences of these

Expected outputs

A list of questions to be used in the evaluation process.
Completion of part B of the evaluation plan.

Required time

Variable, depending on the amount of engagement involved.

Guiding questions

The following table provides an example of evaluation questions for each of the four key aspects of evaluating an adaptation process. Feel free to change these questions to suit your own evaluation or think of additional questions for each aspect.

Procedure

Create questions to ask during the evaluation that will track progress made towards achieving your original objectives for implementation.
In addition, use the assumptions identified in your project pathway (in Step 3) to formulate evaluation questions that look at whether these assumptions were reasonable and still hold true.
If you did not develop a project pathway, use the table in the following section as a guide for formulating evaluation questions.
Revisit the evaluation purpose(s) and check that the questions you have identified adequately address them.

Table 39: Determining your evaluation questions – some examples

Four key aspects of an adaptation process evaluation	Examples of corresponding evaluation questions
The progress of planned activities ‘Did we do things right?’	Did what you achieved match what you expected to achieve? Were the planned activities carried out in an efficient, affordable, appropriate and timely way? Were your inputs sufficient enough to enable you to carry out the planned activities?
The roles, responsibilities and level of engagement of farmers and other stakeholders involved in implementation	Did activities target the right people and scale to build resilience? How were key groups engaged in the activities? Who took on which roles during different stages? What was their experience like participating in the activities? What was of most value to them in taking part in the work?
The appropriateness of the logic in the operational plan, including assumptions about how activities link to achieving the overall purpose ‘Did we do the right things?’	Did the activities result in the anticipated outcomes? What assumptions were challenged and in what ways? What new understanding has emerged about how change happens and what constrains or supports it? Have priorities changed during the course of work due to external changes?
If and how unexpected or unintended outcomes have arisen	What was surprising or unanticipated, or what challenged your understanding of how change happens?

Technical methods

Group discussions with key people involved in the project design (see section A), informed by the project objectives, assumptions from developing a theory of change and the outputs of the previous discussions and workshops.