Required citation:

van den Berg, H., Phillips, S. & Morales-Abubakar, A.L.C. 2023. Monitoring, evaluation and learning in farmer field school programmes – A framework and toolkit. Rome. https://doi.org/10.4060/cc5160en

The designations employed and the presentation of material in this information product do not imply the expression of any opinion whatsoever on the part of the Food and Agriculture Organization of the United Nations (FAO) concerning the legal or development status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific companies or products of manufacturers, whether or not these have been patented, does not imply that these have been endorsed or recommended by FAO in preference to others of a similar nature that are not mentioned.

The views expressed in this information product are those of the author(s) and do not necessarily reflect the views or policies of FAO.

ISBN 978-92-5-139178-5

© FAO, 2023

Some rights reserved. This work is made available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo/legalcode).

Under the terms of this licence, this work may be copied, redistributed and adapted for non-commercial purposes, provided that the work is appropriately cited. In any use of this work, there should be no suggestion that FAO endorses any specific organization, products or services. The use of the FAO logo is not permitted. If the work is adapted, then it must be licensed under the same or equivalent Creative Commons licence. If a translation of this work is created, it must include the following disclaimer along with the required citation: “This translation was not created by the Food and Agriculture Organization of the United Nations (FAO). FAO is not responsible for the content or accuracy of this translation. The original [Language] edition shall be the authoritative edition.”

Disputes arising under the licence that cannot be settled amicably will be resolved by mediation and arbitration as described in Article 8 of the licence except as otherwise provided herein. The applicable mediation rules will be the mediation rules of the World Intellectual Property Organization http://www.wipo.int/amc/en/mediation/rules and any arbitration will be conducted in accordance with the Arbitration Rules of the United Nations Commission on International Trade Law (UNCITRAL).

Third-party materials. Users wishing to reuse material from this work that is attributed to a third party, such as tables, figures or images, are responsible for determining whether permission is needed for that reuse and for obtaining permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user.

Sales, rights and licensing. FAO information products are available on the FAO website (www.fao.org/publications) and can be purchased through publications-sales@fao.org. Requests for commercial use should be submitted via: www.fao.org/contact-us/licence-request. Queries regarding rights and licensing should be submitted to: copyright@fao.org.

Contents

Acknowledgements

Acronyms and abbreviations

Before you get started…

Who is this document for?

What will you find in the document?

Introduction

Developments in farmer field schools

What is monitoring, evaluation and learning (MEL)?

The need for a MEL framework and toolkit

What is impact assessment?

The need for impact assessment guidelines in FFS programmes

PART I.

Framework for setting up a monitoring evaluation and learning system for farmer field schools

1 The farmer field school MEL framework

1.1 Using the results chain to understand linkages between farmer field school, their impact and MEL

1.2 Reinforcing four capital domains to build sustainable rural livelihoods

1.3 Combining the results chain and capital domains to develop a framework for MEL and impact assessment

2 Using the MEL framework: defining impact targets and milestone indicators

2.1 Setting the impact targets

2.2 Identifying milestone indicators towards impact targets

2.2.1 Human domain

2.2.2 Social domain

2.2.3 Natural domain

2.2.4 Financial/physical domain

2.3 Summarizing the framework

3 Learning and adaptation to ensure FFS quality

3.1 Linking the quality of FFS activities to the results of FFS programmes

3.2 Enhancing and modifying FFS activities at programme level

3.2.1 Setting up mechanisms and tools to enhance existing activities

3.2.2 Organizing regular reviews of FFS implementation

3.2.3 Introducing structural modifications of activities and methods

3.2.4 Revising guidelines and refresher training

3.2.5 Pilot testing and assessing structural modifications

3.3 Setting up MEL mechanisms at farmer level

4 Tools for monitoring and evaluation

4.1 Monitoring tools

4.2 Evaluation tools

5 Management of MEL

5.1 Organization and oversight

5.1.1 Roles of stakeholders in the management of MEL

5.1.2 Financial and human requirements of a MEL system

5.2 Sampling, reporting and data management

5.2.1 Selecting the sample

5.2.2 Reporting

5.2.3 Data management

5.2.4 Analysis of data

5.2.5 Review and feedback

5.3 Institutionalizing MEL

5.4 Overview of a typical MEL system for FFS programmes

6 Setting up impact assessment for FFS programmes

6.1 Discovering impact assessment

6.2 Capturing early results and unintended outcomes

6.3 Selecting tools for impact assessment design and analysis

6.4 Planning an impact assessment

6.4.1 What are the questions to be answered?

6.4.2 Is impact assessment the right approach?

6.4.3 What are the target and eligible groups of the study?

6.4.4 What is the time span for answering the questions?

6.4.5 What data type and sample size will be needed?

6.4.6 Which design option should be used?

6.4.7 Should the impact assessment be external or not?

6.4.8 What can be done to ensure that impact assessment is going as planned?

6.5 Managing an impact assessment

6.5.1 Selecting the type of impact assessment

6.5.2 Identifying internal evaluation teams or external institutions to carry out impact assessment

6.5.3 Organizing workshops to plan, review, report on results

6.5.4 Disseminating results of the impact assessment

PART II.

Toolkit for MEL and impact assessment for FFS programmes

How to use this toolkit

1 Tools for monitoring

1.1 AESA records

1.2 Cost–benefit records (at farm level)

1.3 FFS start-up reports

1.4 Field day reports (with photos)

1.5 Participant lists

1.6 Participatory evaluation of FFS session

1.7 Participatory performance tracker

1.8 Ballot box: pre-/post-test results

1.9 Reports of facilitator meetings

1.10 Reports of monitoring visits

1.11 Results of study plots/comparative experiments

1.12 Quality matrix

2 Tools for evaluation

2.1 Focus group discussion

2.2 In-depth interviews/key informant interviews

2.3 Spider diagramming

2.4 Most significant change

2.5 Participant observation

2.6 Picturing change

2.7 Questionnaire survey

2.8 Survey of farm logbooks

2.9 Community visioning/vision mapping (combined with action planning for a new project or activities within a project and evaluation of community action plans)

2.10 Kasese tool

3 Design tools for impact assessment

3.1 Baseline survey

3.2 Before–after comparison

3.3 With–without comparison

3.4 Difference-in-differences

3.5 Capturing diffusion effects

3.6 Stepwise implementation scheme

3.7 Accounting for FFS ‘dropouts’

4 Analytic tools for impact assessment

4.1 Difference-in-differences estimates

4.2 Matching

4.3 Combining analytical techniques

4.4 Triangulation of data sources

Glossary

References

Annex 1

MEL reorientation in existing programmes

Annex 2

Application of the sustainable livelihoods framework to assess the potential benefits of livestock FFS on the livelihoods of FFS members

Annex 3

Sample terms of reference for external institution for impact assessment study

Figures

PART I.

Framework for setting up a monitoring evaluation and learning system for farmer field schools

Figure 1. Link between project design, MEL and impact assessment

Figure 2. FFS learning cycles

Figure 3. Experiential learning cycle, applied to planning, monitoring and evaluation as a continuous cycle at programme level

Figure 4. The results chain

Figure 5. Milestones towards an impact target in the results chain

Figure 6. Linking results and interventions

Figure 7. Four domains of results of the farmer field school

Figure 8. The FFS Mandala: Framework on the results of the FFS, showing the results chain, with examples of impact targets in each of the four domains

Figure 9. The MEL framework

Figure 10. Focus on the results of the results chain

Figure 11. Focus on the interventions of the results chain

Figure 12. A. Diagram of activities followed by results; B. Diagram showing how within the FFS activities, methods and their results are closely interrelated

Figure 13. Diagram showing regular review leading to enhancement and modification of activities for implementation

Figure 14. The cycle of learning and adaptation

Figure 15. Schematic illustration of the MEL cycle at programme level and farmer level

Figure 16. Budget categories required for FFS programmes

Figure 17. The monitoring flow chart

PART II.

Toolkit for MEL and impact assessment for FFS programmes

Figure 1. Economic benefits of field study plots

Figure 2. Hypothetical example of spider diagramming for six indicators in the social domain, showing the situation now (red line) and the situation before the FFS (blue line)

Figure 3. Left: Household map; right: Summary of observations on pesticide storage and disposal practices

Figure 4. Examples of ‘Picturing change’ by farmers from Cambodia

Figure 5. Schematic diagram showing some options for design of an impact assessment study

Figure 6. Design option that captures diffusion effects by selecting two control groups: a nearby and far-flung control

Figure 7. Implementation phases according to the stepwise crossover scheme

Annexes

Figure A1. Proposed steps in the reorientation of an existing MEL framework

Figure A2. Schematic example of a comparison between the MEL framework presented in this document and the existing system

Tables

PART I.

Framework for setting up a monitoring evaluation and learning system for farmer field schools

Table 1. Description and application of capital domains in farmer field schools

Table 2. Impacts, outcomes and outputs in the human domain at individual level, with examples of targets, measures and tools for data collection

Table 3. Impacts, outcomes and outputs in the social domain, with examples of targets, measures and tools for data collection

Table 4. Impacts, outcomes and outputs in the natural domain, with examples of targets, measures and tools for data collection

Table 5. Impacts, outcomes and outputs in the financial domain, with examples of targets, measures and tools for data collection

Table 6. MEL framework with domains and results chain. Examples are given for indicator types, measures and tools as selected by a hypothetical programme

Table 7. Examples of ‘early results‘ in the four domains that could become visible during implementation of the FFS or follow-up activities. Monitoring of these ‘early results’ during the FFS could inform the need to strengthen or modify the activities and methods

Table 8. Examples of types of quality indicators of the FFS for monitoring purposes that can be included in a quality matrix or checklist

Table 9. Enriching the MEL framework with indicator and measure of social gain defined by farmers

Table 10. Main monitoring tools for collecting data of FFS activities (see Part II)

Table 11. Example of an assessment of possible monitoring tools for collecting data of FFS activities, with potential users of the data presented for each tool

Table 12. Possible evaluation tools for collecting data of FFS activities, with a short description and purpose (see Part II)

Table 13. Enriching the MEL framework with evaluation tools for farmers’ use

Table 14. Stakeholders and their roles in the management of MEL

Table 15. MEL framework with domains and results chain. Examples are given for methods, roles and resources for a hypothetical programme

Table 16. WHO needs to know WHAT for what purpose (WHY)?

Table 17. Design tools and analytic tools for impact assessment (see Part II)

Table 18. Advantages and disadvantages of internal, external and combined impact assessments

Table 19. Contents of the impact assessment planning workshop

PART II.

Toolkit for MEL and impact assessment for FFS programmes

Table 1. Overview of tools for monitoring, evaluation and impact assessment, indicating the degree of efficiency, ease of implementation and cost

Table 2. Main monitoring tools for collecting data of FFS activities

Table 3. Attendance list of participants and facilitators

Table 4. Summary of attendance: Number of farmers per session

Table 5. Matrix for ballot box test questions and sample questions

Table 6. Results of ballot box test (pre- and post-ballot box test)

Table 7. Analysis of ballot box test results

Table 8. Cotton IPM farmer field school monitoring report

Table 9. Data collection on components of yield and productivity

Table 10. Summary of data on averages of components of yield and productivity from treatments

Table 11. Recording economic parameters

Table 12. FFS activity quality matrix for IPM programmes

Table 13. Sample matrix for monitoring FFS quality

Table 14. Possible evaluation tools for collecting data of FFS activities, with a short description and purpose

Table 15. Design tools for impact assessment

Table 16. Analytic tools for impact assessment

Table 17. Calculation of difference-in-differences estimates, with a numeric example on crop yields

Table 18. Examples of household characteristics that are unaffected, possibly affected and likely to be affected by the FFS activities in the medium term (<5 years)

Boxes

PART I.

Framework for setting up a monitoring evaluation and learning system for farmer field schools

Box 1. The farmer field school

Box 2. ‘Non-negotiables’ for farmer field schools and FFS programmes

Box 3. Key steps in building a MEL system and impact assessment for FFS programmes

Box 4. Diverse impact dimensions of FFS programmes worldwide

Box 5. How the FFS influences all domains of people’s livelihoods: The unexpected beauty of FFS – a case study from Indonesia

Box 6. Unplanned impacts – voices from the field

Box 7. SMART indicators

Box 8. Empowering women through FFS

Box 9. I tell my husband what to do in the field, Nam Dinh province, Viet Nam

Box 10. FAO’s farmer field school changed Fatima’s farming practices towards sustainability

Box 11. Farmers’ journey from FFS to a cooperative, Kampong Cham province, Cambodia

Box 12. Common challenges in ensuring the quality of FFS during implementation and in documenting impact of FFS programmes

Box 13. Regular review workshop

Box 14. From the field

Box 15. Collecting quantitative versus qualitative data

Box 16. Unplanned impacts – Voices from the field

Box 17. Essential elements of FFS impact assessment

Box 18. Challenges in impact assessment

PART II.

Toolkit for MEL and impact assessment for FFS programmes

Box 1. Field day report (sample information)

Acknowledgements

This guidance note was prepared by Henk van den Berg,[1] Suzanne Phillips, and Alma Linda C. Morales-Abubakar. Significant contributions to the development of the publication were made by Anne-Sophie Poisot, Marjon Fredrix, James Okoth and the Malawi team. The document was reviewed by Deborah Duveskog, Emilia Bretan, Giacomo DeBesi, Jan Willem Ketelaar, Maria Hernandez Lagana, Renata Mirulla, Stefano Mondoví, Teatske Bakker and Tiko Hema. Marut Jatiket and Thai Education Foundation hosted the workshop where some of the ideas in this guidance note were first developed and Zoe Despretz provided coordination support to the development of this work on impact assessment. Editing was by Clare Pedrick. Graphic design was by studio Pietro Bartoleschi.

Special thanks go to the facilitators, farmers, farmer field school partners and practitioners, as well as to FAO colleagues who were involved in all the debates, discussions and sharing of ideas and insights that have provided the main source of information and experiences contained in this framework and toolkit on Monitoring, Evaluation and Learning and Impact Assessment in Farmer Field School Programmes.

[1] As part of a letter of agreement between FAO and Wageningen University in Kingdom of the Netherlands.

Acronyms and abbreviations

AESA

agroecosystem analysis

CAESA

cotton agroecosystem analysis

CAP

community adaptation plan

DD

difference-in-differences

ETL

economic threshold level

FAO

Food and Agriculture Organization of the United Nations

FFS

farmer field school

FP

farmers’ practice

FGD

focus group discussion

GAP

group adaptation plan

GIS

geographic information system

HESA

human ecosystem analysis

IA

impact assessment

ICT

information and communications technology

IGA

income generating activity

IPCC

International Panel on Climate Change

IPM

integrated pest management

IPM FFS

integrated pest management farmer field school

M&E

monitoring and evaluation

MEL

monitoring, evaluation and learning

NE

natural enemies

NGO

non-governmental organization

OECD

Organisation for Economic Cooperation and Development

PESA

pastoralist ecosystem analysis

PPT

Participatory Performance Tracker

PRA

participatory rapid appraisal

PRR

pesticide risk reduction

PTD

participatory technology development

RCT

randomized controlled trial

ToF

Training of Facilitators’ course

TOR

terms of reference

The elements of this illustration summarize the chapters contained in Part I and the tools for MEL and impact assessment in Part II. If you are using a digital copy of the document, you can reach each chapter by clicking on its title.

Before you get started…

The basic project cycle can be summarized in three phases: (i) preparatory, (ii) implementation and (iii) post-implementation. Detailed steps in each phase vary depending on the requirements of resource partners and main stakeholders. There are management tools that field projects and programmes use to facilitate planning and monitor change processes to achieve positive and sustainable results, such as the logical framework and the results chain. The results chain, or causality framework, is based on a process of change that shows that a programme’s activities can lead to certain outputs, outcomes and impacts. It is assumed that certain actions/activities, specific skills, outputs and outcomes are required to reach a desired impact target. Built on a logical framework (or logframe), a results chain is always based on assumptions and should be constantly adjusted as additional information becomes available that may bring new insights into the process of change.

Project design and the planning of monitoring and evaluation are closely linked (see Figure 1). A logical strategy, detailed plan, clear measurable objectives, well-designed activities, and well-defined indicators provide the structure for monitoring and evaluation. Monitoring and evaluation data can be used to improve programme implementation and to understand the results of an impact assessment.

Figure 1. Link between project design, MEL and impact assessment

Source: Authors' own elaboration

More than 12 million smallholder family farmers have now been empowered by the quality of the farmer field school (FFS) learning process and helped to continue growing using skills and knowledge gained from farmer field schools. Increasing interest and demand are emerging from countries around the world to build good quality FFS programmes and ensure their correct implementation and sustainability. To this end, the Farmer field school guidance document: Planning for quality programmes (FAO, 2016) was prepared to facilitate the development of high-quality FFS programmes that are relevant to the specific needs of countries and that are flexible to be adapted to local conditions. The FFS Guidance Document provides details about the preparatory phase (the design stage) of quality FFS programmes.

This document, Monitoring, evaluation and learning in farmer field school programmes: A framework and toolkit, focuses on the implementation phase and provides guidance and tools that will help farmer field schools to bring about the change expected by the programme and the communities it supports, by setting up a robust monitoring, evaluation and learning (MEL) system. It also introduces impact assessment as an integral management function linked to monitoring, evaluation and learning in FFS programmes.

In this guide, the term farmer field school encompasses all FFS variants such as farmer life school, climate field school, agropastoral field school, livestock field school, pastoralist field school, aquaculture field school, farmer business or market school, and the other adaptations of farmer field schools that share the same principles, approaches, methodology and vision.[2] The term ‘farmer’ is used as a short form for ‘all types of agricultural primary producers’ including all those involved in farming activities, such as crops, livestock, aquaculture, pastoralists, agroforestry and forest dwellers.

Who is this document for?

This document is aimed at project and programme designers, managers and staff involved in the planning, monitoring, evaluation and learning processes of project/programmes with a farmer field school component. It also targets those who will be involved in the initial design, planning, follow-up of monitoring, evaluation and learning and impact assessment to improve programme implementation. In addition, it will be of interest to anyone engaged in developing new FFS projects, programmes and policies based on lessons learned from the impact assessment.

What will you find in the document?

This document outlines background information and tools to help ensure that implementation of FFS programmes will lead to the anticipated outputs, outcomes and impacts, by setting up a robust MEL system and impact assessment methodology/framework for the improvement of ongoing FFS projects and programmes and the design of new ones.

PART I

The first part of the document, the MEL framework, outlines MEL considerations from the design stage to the implementation and final evaluation of a programme, highlighting the role that communities play in assessing their own progress towards the changes they wish to see. The framework provides ideas on management of a MEL system, including the coordination, oversight, reporting, data management, annual review and institutionalization of MEL.

An overview is given of impact assessment tools and approaches, as well as issues and challenges to take into account in order to select the appropriate impact assessment approach for project needs and identify proper institutions and experts to carry out the impact assessment.

PART II

The second part of the document provides a toolkit – including tools for monitoring, evaluation and impact assessment. It shares thoughts about what kind of information is needed by whom and for whom, and how to collect this information – ranging from information needed for decision-makers and donors to that required by local communities. It outlines the mechanisms and tools needed to collect relevant information during FFS implementation, and feedback loops to correct or improve the programme activities. It also provides basic information on the tools that programme leaders must be familiar with in order to conduct dialogue with experts and institutions regarding the requirements for conducting an impact assessment.

ANNEXES

The annexes contain more detailed information on how to reorient an existing MEL framework, as well as monitoring tools such as the FFS quality matrix and a sample ‘terms of reference’. These tools can be used by external institutions for guidance of programme leaders who will commission an impact assessment.

[2] The terms ‘project’ and ‘programmes’ will be used interchangeably albeit it is understood that projects refer to those initiatives with strictly defined outputs, i.e. what you gain upon completing the project, within a specific timeframe and budget while programmes could include multiple projects with outcomes that often are not easily attributable and depend on the collective benefits of the different projects. The processes for monitoring, evaluation and learning (MEL) will be the same in projects and programmes.

Introduction

Developments in farmer field schools

The farmer field school was developed in the late 1980s in Asia as an educational investment and capacity-building approach for farmers and farmer groups. The FFS nurtures understanding and skills among farmers to prepare them for a process of continued learning and action in their environment (see Figure 2. FFS learning cycles). Since the 1980s, FFS approaches have proved capable of empowering farmers in improving their agricultural and livelihood situation, using ecological learning, systems analysis, experimentation and group-building (Van den Berg, 2004; Tripp et al., 2005; Braun et al., 2006; Waddington et al., 2014; Van den Berg et al., 2021). Over the past three decades, the FFS has evolved to suit the needs of different groups of communities, and methods and curricula have been adapted to diverse contexts, topics and disciplines. For example, the FFS was initially intended to be facilitated by dedicated staff, but it soon emerged that graduate farmers made excellent facilitators in their localities (if supported by dedicated staff), and contemporary data indicate that the majority of farmer field schools are now run by farmer facilitators.

Box 1. The farmer field school

A farmer field school (FFS) is a group learning process. In a typical FFS, 25–30 agricultural producers meet regularly for an entire season/production cycle and carry out experiential learning activities to gain an ecological perspective of managing ecosystems and skills in informed decision-making based on location-specific conditions.

An FFS is generally facilitated by skilled extension workers or producers. Employing non-formal education methods, the field (or the orchard; or forest; or a group of animals or fish pond) is used as the primary resource for discovery-based learning. The process is facilitative and respects the experience that producers bring with them.

Producers work in small groups to ensure that each one’s ideas are shared. Activities are designed to respond to the immediate needs of producers and are geared towards encouraging creativity and independence. Farmer field schools prepare producers to work together to address agricultural (livestock, aquaculture, etc.) and broader community concerns as FFS alumni groups, farmer organizations or cooperatives.

©FAO

©FIELD Indonesia

Source: Adapted from Morales-Abubakar, A.L. et al., 2013. Empowering farmers to reduce pesticide risks. Bangkok, FAO. www.researchgate.net/publication/259080275_Empowering_Farmers_to_Reduce_Pesticide_Risks

Over the years, the FFS has seen major developments in terms of numbers of farmers trained and expansion from Asia to other regions of the world. A recent global survey estimated that 0.4–1 million farmers graduate from farmer field schools every year (Van den Berg, H., Ketelaar, J.W., Dicke, M. & Fredrix, M. 2020.). This suggests that the FFS education approach has been extensively adopted, although it has yet to be mainstreamed by most national agricultural extension systems. Agricultural extension services have faced limitations in terms of inadequate budget allocation, few available field officers, aging workforce, high staff turnover, and inadequate funds available for farmer-based activities. To address these challenges, the supplementary role of FFS programmes and farmer facilitators is crucial to farmer education, as is the use of information and communications technology (ICT) in discovery-based training.

©William Settle

There are other activities that have built upon or derived from the FFS approaches, but which do not always use the FFS name. Examples include Farmer Life Schools, Farmer Market Schools, Danish Stable Schools, Science Field Shops, and Field Laboratories. More recently, ICT-based approaches have been woven into discovery-based farmer training programmes. These developments, which can be expected to continue in the near future, show the lasting legacy of the FFS methodologies and approaches of field-based discovery-learning.

Conversely, in some countries, initiatives are being implemented that use the FFS name, but are in fact not ‘genuine’ farmer field schools, because key principles in programme design, local ownership and learning are compromised (CIP-UPWARD, 2003). For example, the FFS has been used for the purpose of technology adoption, which does not create ownership of knowledge that empowers farmers for continued learning. This is NOT aligned with the true intention of FFS. In response to these concerns, FAO produced an FFS Guidance Document for the planning and adaptation of quality FFS programmes (FAO, 2016), which provides support for, among other things, the design, capacity-building and key non-negotiable elements of the farmer field school (see Box 2).

Box 2. ‘Non-negotiables’ for farmer field schools and FFS programmes

- ➔ Farmers’ needs define and drive FFS and FFS programmes – not technology

- ➔ Farmers’ local knowledge co-produces and co-creates new knowledge and science and new public services (i.e. pluralistic extension) alongside science-based knowledge and formal extension systems

- ➔ The learning process and knowledge generation are central to FFS and FFS programmes:

1. Farmer field schools are based on fields (or animals or trees) through which to learn and experiment; structured hands-on, experiential learning is primarily used;

2. adult learning cycles emphasize observation, monitoring, critical analysis, sharing and debate, conclusion/decision (i.e. functional skills, and not only technical skills;

3. implementation enhances knowledge and decision-making skills that combine local and science-based knowledge;

4. learning is a continuous process – regular meetings are held at critical crop/enterprise development stages to correspond with the decision-making of farmers;

5. the practical and critical development of skills and competencies is the focus (i.e. not only knowledge);

6. diversity in age, gender and experience enriches an FFS when all are involved in production.

- ➔ Building trust and strengthening groups to develop:

1. critical analysis skills;

2. feedback and evaluation skills;

3. planning skills;

4. basics of group work and collaboration (group dynamics exercises).

- ➔ Facilitation of the learning process: competent master trainers and facilitators (technical, methodological)

- ➔ Development of organizational skills

- ➔ Situation/location-specific activities, i.e. locally appropriate learning curriculum

Source: Adapted from FAO, 2016. Farmer field school guidance document: Planning for quality programmes. Rome. www.fao.org/documents/card/en/c/d7d4db1f-826f-4d81-b097-44292ff7eeca/

What is monitoring, evaluation and learning (MEL)?

Monitoring is the routine tracking of information about the FFS activity, seeking to answer the general question of whether implementation is going according to plan, whether it is of high quality and considered useful by farmers (“what are we doing?”).

Evaluation is the activity that seeks to answer whether or not the implementation leads to the expected effects (“what results are we having?”). Monitoring of implementation and evaluation of the effects are combined in monitoring and evaluation (M&E). Monitoring and evaluation is a form of experiential learning (for resource partners and main stakeholders) at project, programme or policy level. To underscore the learning value of monitoring and evaluation, the term monitoring, evaluation and learning (MEL) has been adopted for the FFS. The learning process transforms M&E into a robust MEL component of the FFS intervention.

At FFS level, each part of the process of planning, monitoring and evaluating farmer field school activities feed into the others. M&E provides feedback based on which activities are modified (as a result of the learning process) to increase benefits to the community. The MEL process is similar to the experiential learning cycle – a central feature of the FFS – where participants learn, evaluate and make plans for another round of the learning cycle (see Figure 2).

Figure 2. FFS learning cycles

Source: Adapted from FAO. 2016. Farmer field school guidance document: Planning for quality programmes. Rome. www.fao.org/documents/card/en/c/d7d4db1f-826f-4d81-b097-44292ff7eeca

The same learning cycle applies at the programme level, where assessments lead to conclusions, which provide feedback for planning and actions to make improvements in the activities (see Figure 3).

Figure 3. Experiential learning cycle, applied to planning, monitoring and evaluation as a continuous cycle at programme level

Source: Adapted from FAO. 2016. Farmer field school guidance document: Planning for quality programmes. Rome. www.fao.org/documents/card/en/c/d7d4db1f-826f-4d81-b097-44292ff7eeca

Conducting MEL in FFS programmes will allow farmer participants and the project/programme team to analyse and reflect critically on their experiences, and plan for modifications to ongoing or follow-up farmer field schools, a second phase of an existing project, or to use these experiences in the formulation of new programmes/projects with similar activities and strategies.

Source: Cartoon by Terry Smutylo – https://www.ox.ac.uk/research/using-research-engage/policy-engagement/guidance-and-resources/how-do-i-monitor-evaluate-and-learn-about-policy-engagement/monitoring-evaluation-and-learning-about-policy-engagement-approaches-questions-and.

The primary purpose of MEL is to improve the quality of the FFS programme, in order to achieve the desired results (the outcomes and impacts); we can call this “adaptive management”. A secondary objective of MEL is to be accountable to farmers as the main beneficiaries of the FFS; involving farmers and local leaders in tracking their performance, making activities more appropriate, and building local capacity and ownership. A tertiary purpose of MEL is to account for the use of resources to the funding body and institutions involved. Additionally, information on FFS implementation obtained from MEL can be used to prepare an impact assessment (see Chapter 6 Setting up impact assessment for FFS programmes) and contribute to an analysis of the effects observed after the intervention.

The importance of the learning element of MEL highlights the need to ensure that adequate resources (staff time, travel) are provided for monitoring and evaluation, since the results will be utilized to improve activities and programme management and achieve the desired impacts of current FFS programming, as well as to enhance evidence to inform the design of FFS programmes in the future.

MEL implies that a system is in place that allows adaptations to be taken on board during the course of implementation. The MEL system should be set up by a project or programme to ensure that the intended effects are achieved as planned, through timely, efficient and high-quality implementation of activities. The time span of a learning cycle of MEL can vary from one to several seasons/production cycles, depending on the type of methods used.

The need for a MEL framework and toolkit

The diversity of applications and uses of FFS programmes has increased, which presents challenges to practitioners in developing appropriate methods for MEL. These methods should identify what to monitor and evaluate, selecting key indicators, which monitoring and evaluation tools to use and how to assess the performance of the FFS, before incorporating the lessons learned to improve the design and implementation of ongoing and future activities. Furthermore, results from the global FFS survey carried out in 2019 (Van den Berg, 2021) suggest that FFS programmes are commonly overwhelmed by the volume and complexity of the data collected from the field. This raises the question of whether data are efficiently collected and used for learning purposes to improve ongoing field implementation and programme management. Hence, many FFS practitioners have expressed a need for a framework and guidance on monitoring, evaluation and learning.

©Agrorural_MINAGRI

What is impact assessment?

Impact assessment is a methodological approach used to establish whether or not the observed impacts can objectively be attributed, or accredited, to the intervention. Impact assessment can be used as a tool to improve programme implementation. MEL and impact assessment are linked in that MEL data can help to explain the impact assessment results. Training quality and learning achievements are monitored and the findings are assessed and analysed to identify problems that are then used to modify current activities or improve the design of subsequent programmes.

Over the past two decades, many impact studies of farmer field school (FFS) programmes have been produced. Unfortunately, most of these studies presented outcomes and impacts as stand-alone results, with little or no reflection on the linkages across the results chain. A lack of impact may have been due to various factors along the results chain. However, without the studies explicitly exploring causality, it is difficult to relate the outcomes and impacts to the way that the FFS intervention was designed or implemented.

The need for impact assessment guidelines in FFS programmes

Impact studies have been carried out in many places, sometimes by the projects themselves, others by external investigators from universities or other institutions. Some studies have been published in peer-reviewed journals, while many have been documented in the grey literature.

In 2004, a review was conducted to synthesize existing impact studies on integrated pest management farmer field schools (IPM FFS) (Van den Berg, 2004). It showed that diverse methodologies, processes and tools were employed in these studies. One challenge that was highlighted was how to capture the diversity of impact areas (social, economic, environmental, policy) at various levels (trained individuals, households, communities, national institutions), while limiting costs and time for the studies. Furthermore, it proved difficult to provide an accurate picture of local realities, and to ensure that identified changes were indeed attributable to FFS interventions. The review revealed a demand for commonly agreed frameworks on basic data needed for FFS programmes, and for tools to capture innovations that can be refined based on local contexts. In 2014, another study reviewed the use of FFS for improving farming practices and farmer outcomes (Waddington et al., 2014). However, there continues to be a need for guidelines on impact assessment in FFS programmes.

Box 3. Key steps in building a MEL system and impact assessment for FFS programmes

The chapters in Part I of this document will guide the reader through the main steps of setting up and running a MEL system for FFS programmes, as summarized below.

©Cheythyrith Chou

PART I

Framework for setting up a monitoring evaluation and learning system for farmer field schools

1
The farmer field school MEL framework

This section will provide the reader with key information on the proposed farmer field school MEL framework for FFS programmes, as summarized below:

1. 1.1 Using the results chain to understand linkages between farmer field schools, their impact and MEL

2. 1.2 Reinforcing four capital domains to build sustainable rural livelihoods

3. 1.3 Combining the results chain and capital domains to develop a framework for MEL and impact assessment

1.1
Using the results chain to understand linkages between farmer field school, their impact and MEL

As previously mentioned, the results chain is based on a process of change that shows that a programme’s activities can lead to certain outputs, outcomes and impacts. To reach an impact target, certain actions/activities, specific skills, outputs and outcomes are required – in that sequence – otherwise, the desired impact will not be achieved.

The results chain is also sometimes called the ‘causal chain’. This means that programme activities have the intention to cause a process of change that leads to the desired results (Douthwaite et al., 2003).

Source: Cartoon by Julie Smith – https://unesdoc.unesco.org/ark:/48223/pf0000186231

The results are what can be influenced through our activities (for example, the food security levels of the FFS participants) (see Figure 4). Interventions are the actions that we can control, including the set of inputs and activities needed to address the problems identified (such as organizing training of farmers in sustainable land management practices through the FFS). Monitoring, evaluation and learning helps in understanding the relationship between the inputs/activities (the interventions) and the results of those activities.

Figure 4. The results chain

Source: Authors' own elaboration

When designing, revising or evaluating an intervention, many activities can be undertaken over many months: desk review of documentation, data collection, stakeholder analysis, problem analysis and prioritization exercises with stakeholders and communities, technical or economic studies, key informant interviews, GIS mapping, surveys, community focus groups, workshops and more. The result chains is a useful way to help summarize the vision for the intervention in a concise and clear way; and to test how robust the design is.

The results chain starts with problem identification (for example, seeds that meet local climatic and socio-economic requirements are not available) and setting the impact targets (such as improved food security in the long term for farmers and their families). When a specific impact is targeted, it is useful to think ‘in reverse’ about the outcomes, outputs and actions/activities that serve as ‘milestones’ leading to the impact target (see Figure 5).

Figure 5. Milestones towards an impact target in the results chain

Source: Authors' own elaboration

After thinking about the milestones leading to the impact target, work backwards, to make explicit the series of activities that need to happen, the key stakeholders who should be involved, and the intermediate results (outputs and outcomes) that need to be achieved if the FFS programme is to reach its impact targets.

Then review the interventions using the following questions:

➔ Problem:

- How was the problem and its contributing factors identified?

- Were pertinent problems and their contributing factors addressed by the programme?

- Has new information emerged that suggests other, more relevant problems that the programme should tackle?

➔ Design:

- Does the design adequately address the problem and lead to the desired change process, without potential negative trade-offs?

- Does the design of activities give flexibility to allow for adaptations in the programme design to respond to emerging needs?

➔ Inputs:

- Based on the design, what inputs (financial and human resources) are needed to carry out the activities?

- Are inputs mobilized to carry out planned activities?

➔ Activities (These include the FFS, Training of Facilitators (ToF) courses, follow-up activities for FFS alumni groups, coordination meetings, advocacy, and any other supportive activity): Does the way the activities are implemented have direct implications on the results they will attain?

NB: We need to ensure that the farmer field school processes are implemented following the key principles of FFS. As such, process-based monitoring is set up to ensure good quality of FFS implementation.

After reviewing the interventions, the expected outputs, outcomes and impacts should be clearly defined. The following questions may be used to identify the results:

➔ Outputs:

- What immediate results will the activities bring?

- What farmer capacities, abilities, knowledge and skills will be directly improved by the FFS?

- Will the changes be achieved during the duration of the FFS (for example, acquired knowledge and skills)?

➔ Outcomes: (These can be short-term (see Outcomes 1; step 3 in Figure 5) and medium-term (see Outcomes 2; step 2 in Figure 5).

- What can the farmers do differently or better as a result of the capacities, abilities, knowledge and skills gained directly from the FFS? (for example, Outcome 1: changed attitudes and practices and Outcome 2: collective action, improved production and savings).

- Are the changes within the control of the FFS, or can they only be influenced by the FFS?

- Are the changes expected to happen during or after participation in the FFS?

➔ Impacts:

- What would be the long-term consequence resulting from the outcomes of the FFS?

- What improvements will the FFS bring to the lives and capital domains of participants (such as increased food security and well-being, increased biodiversity)?

- How long will it take to see the impacts of the FFS?

Careful observations of the outputs and outcomes will help to understand if and how the activities lead to the results. If the desired results do not materialize, critical reflection should be undertaken in order to adapt the activities towards achieving the desired results (reorientation process). A lack of impact may have been due to various factors along the results chain (see Figure 6).

Figure 6. Linking results and interventions

Source: Authors' own elaboration

For instance, lack of impacts could be related back to poor outcomes, activities or design of the FFS. In the context of the example provided in Figure 4. The results chain, a lack of improvement in food security could possibly be linked to poor achievements in the selection of locally adapted seed or to poor development of farmers’ skills for seed production. It is also possible that the identified problem of lack of locally adapted seed was not the main limiting factor of agricultural production.

1.2
Reinforcing four capital domains to build sustainable rural livelihoods

The FFS is intended to empower farmers and trigger a process of continued learning in multiple dimensions. Empowered farmers have the ability and means to take control over their lives (see Box 4. Diverse impact dimensions of FFS programmes worldwide). The sustainable livelihoods framework takes a holistic approach to rural development and is appropriate for monitoring and evaluating farmer field schools. It views farmers, communities or villages as part of their existing environment over which they have limited control, but shows that by combining available livelihood resources, they can embark on strategies to improve their livelihood situation (Scoones, 1998).

The sustainable livelihoods framework comprises four types of livelihood resources (see Figure 7) – so-called capital domains, namely the human, social, natural and financial/physical domains (see Table 1).[3]

Figure 7. Four domains of results of the farmer field school

Source: Authors' own elaboration

Table 1. Description and application of capital domains in farmer field schools

Domain

Description

Application in FFS

HUMAN CAPITAL

Refers to all attributes of the individual human being, including knowledge, skills, creativity and drive to perform and innovate, and the health and strength to work. These attributes are necessary for each human to function and attain well-being within the person’s environment.

The FFS uses educational concepts such as the experiential learning cycle (Kolb, 1984), the learner-centred approach for adult education (Rogers, 1969), and the framework for the technical, practical and emancipatory domains of learning (Habermas, 1971; Pontius et al., 2002). These concepts, if applied properly, are expected to enhance human capital.

SOCIAL CAPITAL

Refers to relationships, norms, cooperative action and networks. It gives the members of a community additional meaning, power and opportunities, compared with the situation where individuals act on their own.

Since the FFS is a group approach, it can be expected to increase social capital.

NATURAL CAPITAL

Refers to the natural resources (such as water, soil, biodiversity) and the environmental services (such as natural regulation of water, climate or plant pests) available to rural communities. Farmers and pastoralists strongly depend on their natural capital for maintaining or improving their livelihood situation.

The FFS plays an important role in preserving and strengthening the natural capital, mainly by improving FFS members’ ecological literacy and understanding of the agroecosystem and mobilizing farmers to protect their ecosystems.

FINANCIAL/PHYSICAL CAPITAL

Refers to the cash, savings and assets that are needed to maintain or improve people’s livelihood situation.

The FFS can contribute to financial capital by teaching financial skills, increasing family income (for example through improved market access), or by offering opportunities for savings and loans.

Source: Authors' own elaboration

Box 4. Diverse impact dimensions of FFS programmes worldwide

- ➔ As part of the Regional IPM Programme in Asia, FFS farmers reduced costs of farming by 70 percent on average, Participants in Viet Nam reduced pesticide spraying frequency by 80 percent and FFS participants in Bangladesh reduced volume of pesticide used by 76 percent.

- ➔ Farmer Water Schools in India were set up as platforms to strengthen the farming community on demand and supply dynamics of water and empower them to negotiate for appropriate water releases at minor level. Impact included reduced risks associated with groundwater distress; reduced groundwater pumping while improving economic returns per unit water; and better management of common property resource and crop diversification (from 4 to 19 crops cultivated in a given area) with nutritional implications.

- ➔ In Viet Nam, as an outcome of FFS activities, in 2015 the Ministry of Agriculture and Rural Development issued Directive No: 2027/QD-BNN-BVTV supporting the Programme on Strengthening the Application of IPM in Crop Production (2015-2020).

- ➔ In Pakistan, as a result of IPM FFS, significant improvements were observed in gender equity and social inclusion, community trust and collaboration, and in collective action (beyond agriculture).

- ➔ In Lebanon, the support provided to participants of poultry FFS has helped generate income for many vulnerable households. In total, 40 000 laying hens and 525 tonnes of feed were distributed. The poultry FFSs allowed beneficiaries to learn by implementing good poultry husbandry and egg-production practices, including construction of bio-safe and biosecure poultry coops. It is estimated that the eggs produced by each set of 50 hens should increase gradually to at least 40 a day. Moreover, all the poultry FFS households are consuming eggs produced by their own hens, thus increasing their protein intake.

©FAO Lebanon

- ➔ In Kenya, the value of crop productivity/acre for FFS members increased by 80 percent, and by 200 percent for female-headed households.

- ➔ In Mozambique, integrated practices introduced via FFS are resulting in more resilient and productive farmers, with diversified livelihoods and improved food and nutrition security. Visible gains are observed in social capital and knowledge exchanges.

- ➔ In Malawi, several FFS programmes are ongoing and collaborating at country level. Emerging results of FFS activities include: forest regeneration in a number of conservation belts, improvement of dietary diversity among participants, improved yields, improved commodity deals registered with private dealers and accumulation of assets in group savings, which are used as capitalization for small on- and off-farm investments.

©Rachel Nandelenga

Sources: FAO. 2018. Farmer field schools for small-scale livestock producers – A guide for decision makers on improving livelihoods. FAO Animal Production and Health Guidelines No. 20. Rome; Yerraconda, S. 2018. Unpublished; FAO. 2019. Farmers taking the lead: Thirty years of farmer field school; Van den Berg, H., Ketelaar, J.W., Dicke, M. & Fredrix, M. 2020. Is the farmer field school still relevant? Case studies from Malawi and Indonesia. NJAS-Wageningen Journal of Life Sciences, 92:100329.

The publication Farmer field schools for small-scale livestock producers – A guide for decision makers on improving livelihoods demonstrates the application of the sustainable livelihoods framework to assess the potential benefits of livestock FFS on the livelihoods of FFS members (see Annex 2). Examples of potential benefits from the livestock FFS using the sustainable livelihoods framework are incorporated in the measures (i.e. indicators) of the proposed MEL framework (see section 1.3 Combining the results chain and capital domains to develop a framework for MEL and impact assessment). See further examples in Boxes 4 and 5.

Box 5. How the FFS influences all domains of people’s livelihoods: The unexpected beauty of FFS – a case study from Indonesia

In Indonesia, the first FFS programme was set up in 1988 as a response to pest outbreaks in rice. Over time, it developed into a huge national integrated pest management (IPM) programme involving millions of Indonesian farmers. Today, IPM FFS are still promoted by the national Plant Protection Directorate. The Indonesian IPM Farmer Association has 1.2 million members and continues to promote community-based IPM and farmers’ rights.

The Field Foundation, a non-governmental organization (NGO) that emerged from the IPM FFS programme, uses the FFS approach to address broad farmer issues such as community forest management, water distribution, biogas production and wastewater management. In Kebon Agung Village, East Java, two FFS groups were set up: one focusing on managing the rice ecosystem and another on management of the water and sanitation ecosystem. Both FFS groups steadily moved into working beyond the initial FFS theme:

- ➔ The first FFS evolved to cover more than just agricultural technology, to develop and use all natural resources available to the community.

- ➔ The second FFS evolved to be more than a water and sanitation (watsan) facility into a multipurpose facility for the community.

This led to very diverse outcomes:

- ➔ Ecological food was produced for members and the community: ecological rice, vegetable and fish.

- ➔ The watsan facility became a meeting place, community laboratory and community learning centre for the group, as well as for other farmers and organizations.

- ➔ Income was obtained for the watsan facility and for financing group development.

The FFS farmer groups became more active, since various community interests could be accommodated through increasingly diverse activities. Those activities cover various aspects of livelihoods: food, water and sanitation, income, intercommunity training, community organization, community networking, and multistakeholder collaboration.

©FIELD Indonesia

Source: Van den Berg, H., Ketelaar, J.W., Dicke, M. & Fredrix, M. 2020. Is the Farmer field school still relevant? Case studies from Malawi and Indonesia. NJAS-Wageningen Journal of Life Sciences, 92:100329. https://agris.fao.org/agris-search/search.do?recordID=NL2020005031

All four capital domains are equally important to rural communities for pursuing their strategies to improve their livelihoods sustainably: if any one of the capitals is missing or weak, the potential to improve their livelihoods in a sustainable way will be compromised.

1.3
Combining the results chain and capital domains to develop a framework for MEL and impact assessment

The proposed framework on the results of the FFS, as presented in Figure 8, combines the results chain and the four capital domains. The FFS is presented in the centre of the FFS ‘Mandala’ diagram, with the understanding that the FFS (in the central circle) includes several steps, from problem analysis, design and inputs to activities, and covers supportive activities (such as training of facilitators and follow-up).

Figure 8. The FFS Mandala: Framework on the results of the FFS, showing the results chain, with examples of impact targets in each of the four domains

Source: Authors' own elaboration

In the monitoring and evaluation of the FFS, a balanced evaluation should be adopted across the four domains, because farming communities need each of the four assets for their livelihood strategy. As shown in Figure 8, the FFS leads to outputs in each of the four domains, namely: analytic skills, social skills, ecosystem analysis and financial skills. These outputs lead to immediate and intermediate outcomes, which in turn may produce impacts.

The framework provides examples of impacts in each domain; any of these (or other impacts) could be targeted by a programme as the impact targets. Any of these impacts may emerge through a unique route from outputs and outcomes. The framework does not prescribe the outputs and outcomes that lead to a certain impact; rather, the path for change can happen across domains. Hence, the route to reach the impact target needs to be examined on a case-by-case basis by a programme.

MEL should not become too complex. Collecting strong data on a few indicators is preferable to collecting weak data on many indicators. Therefore, when designing the FFS programme and the corresponding MEL framework, it is suggested to select not more than one impact target in each of the four domains (human, social, natural and financial/physical). Countries should adjust the framework to suit their particular needs, for example by separating the financial and physical capitals when appropriate.

“As much as necessary, as little as possible”…

Brent Anderson

It is acceptable to place more emphasis on one or two domains, depending on the focus of the project or programme. But the inclusion of all four domains stimulates a balanced evaluation and vision of change. Data from human, social, natural and financial domains can complement each other. This could provide more convincing results to guide decisions at programme level.

Figure 9. The MEL framework

Source: Authors' own elaboration

[3] More recent modifications of the sustainable livelihoods approach added a fifth capital domain, the physical capital (DfID, 1999). In the case of the FFS, and to keep our framework as simple as possible, the physical capital is combined with the financial capital (Pontius et al., 2002).

2
Using the MEL framework: defining impact targets and milestone indicators

This chapter describes how the basic MEL framework, with its results chain and four capital domains (see Chapter 1), can be developed by countries or programmes into a concrete structure or system for MEL. This structure should define the expected process of change and key milestone indicators, as the basis for monitoring and evaluation, as summarized below:

1. 2.1 Setting the impact targets

2. 2.2 Identifying milestone indicators towards impact targets

3. 2.3 Summarizing the framework

The focus of this chapter will be on the results (see Figure 10), while Chapter 3 concentrates on the interventions and their quality. Common challenges in assuring the quality of FFS during implementation and in documenting impact of FFS programmes are mentioned in Box 12 at the end of this chapter.

Figure 10. Focus on the results of the results chain

Source: Authors' own elaboration

New and existing FFS programmes are encouraged to prepare their worked-out MEL framework (see template in section 2.3 Summarizing the framework), with impact targets in each domain, results chain, and milestone indicators, as discussed in sections 2.1 Setting the impact targets and 2.2 Identifying milestone indicators towards impact targets. Existing FFS programmes will need to follow additional steps for a reorientation process, presented in Annex 1 MEL reorientation in existing programmes.

New programmes: When designing or developing a new FFS programme, it is important to spend ample time working out a framework for MEL and IA: which type of impacts are expected; which causal stage will lead to each impact; how activities can be designed in such a way that the expected outcomes or impacts are reached; who should be involved in implementing activities, collecting information and reporting; and the timelines. Care must be taken that adequate funds are allocated in the budget for MEL and IA, because this will be a core programme management activity (see section 5.1.2 Financial and human requirements of a MEL system). The MEL and IA framework should then be incorporated into the logical framework, or similar tool for programme management, so that it becomes an integral part of the new programme.

Existing FFS programmes: Existing FFS programmes and projects mostly have protocols in place for monitoring and evaluation. These protocols have generally been developed in accordance with programme logical frameworks, ideally based on a results chain. The logical framework and results chain should be the basis for MEL and IA, although it will be useful to reorient the existing system of monitoring and evaluation towards a MEL system with a stronger learning component and a better thought-out framework, as well as more adequate design and planning, feedback loops and review processes. This reorientation may require changes in protocols for data collection, together with the use of new tools, and strengthening of the review process. Details of the reorientation process are presented in Annex 1 MEL reorientation in existing programmes.

Alice: Would you tell me, please, which way I ought to go from here?

The Cheshire Cat: That depends a good deal on where you want to get to.

Alice: I don't much care where.

The Cheshire cat: Then it doesn't much matter which way you go.

If you don't know your impact targets, it does not matter what you monitor

Cheshire Cat in Lewis Carrol "Alice in Wonderland"

2.1
Setting the impact targets

All programmes aim to have a certain impact that will address the problems identified and implement their activities, with the aim of achieving that impact. Therefore, targets should be aimed at impact level, such as those presented in Figure 8. An impact target is what we ultimately want to reach.

Even though some impacts may be beyond the direct reach of the programme and the given time frame, the programme would still aim to contribute towards achieving these impacts. Hence, outputs and outcomes become milestones towards reaching the impact targets. This is helpful because outputs and outcomes may be more easily accomplished and measurable by a programme than impacts.

Farmers may be interested in different outcomes and impacts compared with programme managers. Farmers should be able to track whether or not they have achieved what they wanted from the intervention. It is therefore important to include participatory tools to identify the outcomes and impacts most valued by farmers (such as ‘spider diagrams’, ‘picturing change’, ‘most significant change’ tools; see Part II, section 2 Tools for evaluation).

Box 6 illustrates how impact types can differ from the original theme tackled by FFS groups. Depending on the availability of time and resources – and whether or not it makes sense – the MEL system could even include more than one impact per livelihood domain.

Box 6. Unplanned impacts – voices from the field

In the Democratic Republic of the Congo, households traditionally sold their produce individually: after 2 years, 68 percent of farmers negotiated as the FFS group, 56 percent operated in a farmer business association and 30 percent sold through agricultural collection centers. The formation of farmer business associations allowed the farmers better market access, although the initial focus of FFS activities was on increased yield and application of agricultural good practices.

In Senegal, a farmer involved in IPM FFS programmes commented: “I see that the increased income, especially in case of women (who mostly take charge of school-related expenses for kids), had a positive impact on education. More resources are available, for example for school supplies.” Another farmer said: “Apart from the practical changes in my agriculture-related activities, increased opportunities and increased income, I definitely started to care more about others and to share systematically what I learned. I’m more sensitive.”

©Marjon Fredrix

Source: FAO. 2019. Farmers taking the lead: Thirty years of farmer field schools

2.2
Identifying milestone indicators towards impact targets

An indicator is a variable that can be measured and which provides clear information about a subject of interest. For example, if we are interested in the health of a rice ecosystem, we could decide to use the number of spiders – one of the best biological control options – by square metre as an indicator. There are many more components that make up a healthy rice ecosystem because it is very diverse and difficult to grasp. However, if we decide that spiders are representative of an ecosystem’s health, given that spiders are easily observed, the spider density could be an indicator of a healthy rice ecosystem. Hence, an indicator is something that we can measure quite easily, and which is a good reflection of the subject of interest. The same concept applies to indicators in the four domains.

Several factors complicate the identification of indicators: overlapping domains; vague distinction between outputs, outcomes and impacts; and finding measurable indicators.

Overlapping domains. Even though the division into four livelihood domains is helpful, some indicators are a mixture of two or more domains. For example, food and nutrition security at household level has a natural component, a human, and a financial component. Food security (related to food production) may be inclined towards the natural domain), while nutrition security (how nutritious food maintains the human body) inclines towards the human domain. Nonetheless, even if these overlap, specific indicators can be created to capture the desired effect.

Source: Cartoon by Virpi Oinonen – Virpi/Businessillustrator.com

Outputs-outcomes-impact[4] The distinction between outputs, outcomes and impacts is not always very clear; what is called an outcome in one study may be called an impact in another study. Hence, these terms should be clearly defined within the context of each programme or project. In general, what is more important than the categorization of these terms is to recognize that there are usually several steps or phases in which the direct results of the FFS (such as outputs) lead to indirect results (such as outcomes, impacts).

Measurable indicators Often, it is not easy to provide standard measures of the indicators in quantitative ways. Many indicators demand serious thought about how they could be measured or described in a meaningful and reliable manner, and they often require a combination of tools using quantitative and qualitative methods to provide an overview, as well as deeper understanding and insight (see Table 12. Possible evaluation tools for collecting data of FFS activities, with a short description and purpose).

Some indicators may require systematic study in order to reliably attribute the observed result to the FFS; this methodological approach is referred to as impact assessment (see Chapter 6 Setting up impact assessment for FFS programmes). Impact assessment is best conducted using quantitative data, but the results are most valuable when combined with qualitative data (such as those collected through in-depth interviews and other tools), to provide verification and in-depth information, and to explore unintended outcomes or impacts of the intervention.

Box 7. SMART indicators

Ideally, when defining indicators to track and measure progress and impact in programmes, including FFS, it is good practice to select qualitative and quantitative SMART indicators:

Specific to what we want to measure and assess

Measurable in a clear and cost-effective way so everybody can do it

Attainable/achievable so the targets are realistic

Relevant to the implementation plan and the intended impact

Time-bound, where a time frame is defined for the indicator’s achievement and measurement

Source: Adapted from https://eca.state.gov/files/bureau/a_good_start_with_smart.pdf. Cited 24 March 2023.

The following sections aim to assist programmes and countries in developing their own set of SMART indicators in the human, social, natural and financial domains. Examples of indicator types of impact, short- and medium-term outcomes (outcomes 1 and outcomes 2, respectively) and outputs are presented in tables pertaining to each domain. Methods for measuring each indicator type, and tools that could be used, are provided in the tables. More details about the tools are presented in Part II of this document.

2.2.1 Human domain

In the human domain, it is important to look for indicators that describe how the human capital has changed at the individual level, whether people have strengthened their capability, functioning and well-being within their environment.

If we select ‘empowerment of farmers’ as the main impact target (see Table 2), we could monitor the outputs and outcomes that are likely to contribute to that impact. These outputs and outcomes could, for example, include the quality of agroecosystem analysis,[5] self-confidence when presenting AESA results to the FFS group, the number of problems solved, self-perceived change in mindset, and people’s level of control over resources and funds. Since this is just an example, programmes should construct their own results chain.

Table 2. Impacts, outcomes and outputs in the human domain at individual level, with examples of targets, measures and tools for data collection

Results chain

Examples of targets

Examples of measures

Examples of tools

Impacts

Empowerment of farmers

Signs of continued learning; capacity to create opportunities, take action

In-depth interviews; focus group discussion

Quality of life

Wellness; sense of belonging; time to spend positively; access to health services, education level

In-depth interviews; focus group discussion; observation

Outcomes-2

Confidence, motivation

Self-perceived confidence level; number of exchanges with other farmers; sense of purpose

Questionnaire survey; in-depth interviews

Change of attitude, perception, mindset

Self-perceived change in attitude, mindset

Case studies; in-depth interviews

Improved nutrition

Number of meals, food groups; household dietary diversity score

Questionnaire survey ; focus group discussion

Outcomes-1

Decision-making capacity

Tracing how decisions were made, trade-offs, scenarios considered

Case studies; in-depth interviews; number of decisions made; scenarios

Critical thinking

Signs of interest in the causes of problems and ways to solve them

In-depth interviews

Questioning of existing norms/habits

New norms accepted, critical about some old cultural habits

Case studies; in-depth interviews

Innovation, experimentation

Number, sequence and results of farmer studies

Questionnaire survey; focus group discussion; monitoring reports; observation

Problem-solving

Number or examples of solved problems

Case studies; in-depth interviews

Outputs

Strengthened skills of analysis and adaptation

Level of participation in agroecosystem analysis (AESA); quality of analysis in AESA

Improved livestock management practices, including hygiene and sanitation issues

Appraisal of AESA; participant observation

Increased knowledge, awareness

Knowledge scores; number of master trainers and farmers trained per topic

Knowledge surveys (e.g. FFS ballot box test); % farmers graduated; pre-post test results; follow-up action after graduation

Source: Authors' own elaboration

Box 8. Empowering women through FFS

In an FFS project in Senegal, gender disparities were perceived to hinder the participation of women farmers in activities and the successful achievement of the programme’s livelihood objectives. This was addressed through brainstorming at various levels on potential actions that could facilitate women’s participation – ranging from the selection of facilitators, to the scheduling of FFS sessions that avoided market days or times that hampered women’s attendance; the programme trained its staff, master trainers and facilitators to become more gender-sensitive; role plays on gender roles were created by communities; and a participatory video on gender dynamics was created with villagers, and viewed and discussed in village meetings. “The process resulted in significantly improved participation and voice of women in village activities and FFS; women were elected to the board of farmers’ organizations." The programme staff reported an overall positive process for all involved. Similar processes have since been put in place by many FFS programmes around the world.

©Lucie Chocholata

Source: FAO (forthcoming). Champs-écoles des producteurs, égalité de genre, inclusion sociale et autonomisation des communautés au Sénégal. Rome.

2.2.2 Social domain

In the social domain, it is important to look for indicators related to the interactions between people, or mutual support, norms and rules, collective action and networks, which are expected to enhance social capital in the community.

If we select ‘good cooperation at household and community level’ as the impact target (see Table 3), the programme could monitor the outputs and outcomes that are likely to contribute towards that impact. These include farmers’ presentations and discussions in the FFS, group cohesion, the role and contribution of women in decisions made, the number and types of collective actions, and the existing structure for organization and planning.

Table 3. Impacts, outcomes and outputs in the social domain, with examples of targets, measures and tools for data collection

Results chain

Examples of targets

Examples of measures

Examples of tools

Impact

Good cooperation at household and group level

Signs of organizational structure; joint planning; common funds

In-depth interviews; focus group discussion

Empowerment of the farmer group

Signs of continued learning by the group; control over use of resources/money; creation of new opportunities; contributions made

In-depth interviews; focus group discussion

Leadership and community development

Number of leaders, women leaders; vision promoted; motivating community members; bylaws developed

Case studies, focus group discussion; key informant interviews

Outcomes-2

Group decision-making

Number of decisions taken by group; number of members contributing to a decision

Questionnaire survey; focus group discussion

Collective action

% of groups with joint action implemented; numbers and types of actions; participants per action

Questionnaire survey; focus group discussion

Level of access to agroclimate information services

% of groups receiving regular information on technologies, climate forecasting

Questionnaire survey; focus group discussion

Increased role of women in decision-making

Number and type of decisions taken by or with woman at the level of household, group, community

Questionnaire survey; in-depth interviews

Reduced conflict in the household, community

Role-sharing in household tasks; number of conflicts; harmony in relationships; mutual respect

Questionnaire survey; in-depth interviews; case studies

Responsibility for use of natural resources

Number and type of actions to protect/enhance natural resources

Questionnaire survey; focus group discussion

Outcomes-1

Trust, group cohesion

Group dynamics (friendship, respect, rivalry, cooperation), % active group members; gender inclusion; social inclusion

Questionnaire survey; in-depth interviews; case studies; focus group discussion

Speaking out, reaching out

Number and type of contacts with authorities, neighbouring farmers

Questionnaire survey; in-depth interviews

Outputs

Skills of presenting, communicating, team-working, negotiation

Equitable participation in FFS presentations/discussions

Participant lists; participant observation

Source: Authors' own elaboration

Box 9. I tell my husband what to do in the field, Nam Dinh province, Viet Nam

Bui Thi Cuc, now 51, became a member of the Women’s Union at the age of 21. Her husband, Bui Dang Nga, is the Chair of their hamlet in Yen My commune, Y Yen District, Nam Dinh province.

She says, “I attended the farmer field school on IPM/pesticide risk reduction in Summer 2014. There were 40 of us, all women. Men were not available to join the FFS because they either work in factories or furniture shops. Before I attended the FFS, I would transplant many seedlings per hill and use a lot of nitrogen. Through field studies in the FFS, I tried myself and experienced that the number of seedlings per hill and the amount of nitrogenous fertilizer could be reduced, but I still got higher yields.” The practices included use of fewer seeds for transplanting and wider spacing between plants. The farmers saw that the plants grew bigger and better compared with their conventional practices.

©Dung Tien Ngo

“The rice plants were stronger because they received more light. They were more resistant and there was also less insect and disease incidence and better yields. I told my husband about the results of the FFS. He was very happy about it. He works with me in the field. Based on what I learned in the FFS, I tell my husband what to do in the field.”

Bui Thi Cuc’s husband had this to say: “I am happy she participated in the FFS. She has been able to apply what she learned in our field. She can share her experiences with our neighbours and they too can increase their income. And she continues to be a good wife and mother.”

Source: FAO. 2016. Towards a non-toxic Southeast Asia. Stories from the field: Women working towards a non-toxic environment. Penang. www.fao.org/agroecology/database/detail/en/c/1187176/

2.2.3 Natural domain

The natural domain refers to the specific skills, practices and impacts related to the use of natural resources and environmental services that are available to farmer communities, and how the use of those resources is made more productive, secure and sustainable (as well as being socially acceptable and financially viable).

If we select ‘sustainable production system’ as the impact target (see Table 4), the programme could monitor the outputs and outcomes that are likely to contribute to that impact. Examples are farmers’ skills of agroecosystem analysis and management, and farmers’ experimentation with agricultural practices that are adapted to local conditions, leading to locally optimized food production and crop diversification.

Table 4. Impacts, outcomes and outputs in the natural domain, with examples of targets, measures and tools for data collection

Results chain

Examples of targets

Examples of measures

Examples of tools

Impacts

Sustainable production system

Stable yields; reduced reliance on chemical fertilizers and pesticides

Longitudinal or retrospective surveys

Reduced food insecurity

Access to diverse food sources; size and frequency of meals; feeling of anxiety over food quantity/quality

Questionnaire survey; in-depth interviews

Sustainable ecosystem services

Constant or enhanced availability of water, soil fertility (with compost and manure fertilization), pollination, natural control agents

Longitudinal study with systematic design

Outcomes-2

Improved food production

Yield per ha; crop cycles per year; post-harvest loss

Questionnaire survey

Improved biodiversity, crop diversification, cropping pattern, natural resource management

Area under inter/multicropping; rotations; density of pollinators, natural control agents, fodder conservation and reforestation

Questionnaire survey; systematic entomological surveys

Increased conservation of natural resources

Forested area protected by bylaws

Focus group discussion; key informant interviews

Outcomes-1

Increased adaptation of ecosystem management practices

% of FFS alumni using improved and locally-appropriate practices

Questionnaire survey

Improved efficiency in the use of farm inputs

Amount of seed, fertilizer and pesticide inputs per ha; % of FFS alumni reducing overuse of inputs

Questionnaire survey; sales data

Outputs

AESA skills, technical skills

Active participation in FFS; quality of AESA; quality of decisions

Participant lists; appraisal of AESA; participant observation

Knowledge of practices, climate influences

Level of knowledge about what was taught

Knowledge surveys; pre-post test results

Source: Authors' own elaboration

Box 10. FAO’s farmer field school changed Fatima’s farming practices towards sustainability

With simple and adoptable husbandry techniques promoted by FAO’s farmer field schools (FFS), Fatima Al Youssef became a skilled livestock keeper who is capable of dealing with various livestock keeping challenges. Now, together with her husband, Fatima is planning to go beyond her farm practices and get into the market. Her newly acquired knowledge on producing healthy animal feed and fodder storage techniques has helped her cow produce more milk of better quality. The know-how sessions received enabled her to become a competitive milk producer, who can provide extra independent income for her family. Fatima – as a livestock keeper and a mother of two young children - has been enrolled along with fifteen other female livestock keepers to attend the FAO FFS in Abu Jrien, Rural Aleppo. Fodder supply is not always available and often of poor quality; Fatima has learnt to make fodder mixtures on her own, setting the proportions and calculating the quantities her cow needs to be healthy. She said that the fodder mixtures produced are better than the ones available in the market. “My cow is now producing better milk, I can tell with the better taste and I can make better cheese for my family”. The FFS has also served to upgrade feeding practices, which reduced fodder loss, depletion of reserves and the consequent cost of production. “In my barn, it was difficult for my cow to get its feed, because I used to place the fodder at an extremely lower level, which also caused fodder loss. At FFS I have learnt useful livestock keeping tips like raising the fodder tank, which allowed my cow to eat more comfortably,” Fatima added. The integration of topics presented by the farmer field school has also strengthened Fatima, and her fellow female livestock keepers, with techniques for the identification of infections, which often cause contaminated milk that is not fit for human consumption. Those simple techniques made easier early identification of any infection with a fast full recovery of the animal. Group discussions at FFS enriched Fatima’s connection with her surrounding environment. She was happy to share ideas and practices with her neighbouring female livestock keepers. Exchanging ideas and experiences among participants at the school was another advantage of attending FFS. “I enjoy discussions with my neighbours during the class, it opens my eyes to other useful ideas. I remember that one day during an FFS session, my colleague Husama proposed milking the cow using different quarters each time during the milking phase. It was a piece of great advice which we all used in our barns,” said Fatima. There has been a horizontal spread of skills learnt at the FFS. Fatima and other farmers are happy to apply different farming techniques and so upgrade their skills. The FFS were able to help livestock keepers enhance and sustain their feed and food productions and become self-reliant.

©FAO/Andrea Theodor

Source: FAO. 2022. FAO’s Farmer Field School changed Fatima’s farming practices towards sustainability. www.fao.org/syria/news/detail-events/en/c/1468963/

2.2.4 Financial/physical domain

In the financial/physical domain, it is important to know how capital (cash, savings, assets) is generated and available to farmers to secure their incomes and consumption patterns, cope with emergencies, and improve their living standards.

If we select ‘financial security’ as the impact target (see Table 5), the programme could monitor the outputs and outcomes that are likely to contribute to that impact. Examples are the skills of cost–benefit analysis and marketing, increased profits through better accounting, capacity to save and improve their ability to access to loans for investments, returns on their investments, and overall improved income.

Table 5. Impacts, outcomes and outputs in the financial domain, with examples of targets, measures and tools for data collection

Results chain

Examples of targets

Examples of measures

Examples of tools

Impacts

Financial security

Household spending on food, education, health; surplus spending; stability of income

Questionnaire survey; in-depth interview

Improved living conditions

% houses with paved floor; mobility; presence of kitchen, toilet, electricity, etc.,

% of time dedicated to work (incl. domestic work) vs leisure

Questionnaire survey; in-depth interview

Sustainable, inclusive and responsive markets

Signs of market adapting to the needs of farmers (incl. women), and to new (value-added) products

Key informant interviews; case studies; market research

Outcomes-2

Access to loans; savings

Presence of village savings and loans schemes; amount of loans, savings; presence of local banks

Questionnaire survey; focus group discussion; data from local banks

Accumulation of physical assets

Amount of land, number of livestock owned;

% households with phone, radio, bike, stone floor, metal roof, etc.

Questionnaire survey ; focus group discussion

Premium quality of produce

Value addition; selling price; number of certified farms

Questionnaire survey ; focus group discussion

Outcomes-1

Reduced input cost, increased profits

Input costs; production outputs; profits per ha

Survey of farm logbooks; cost-benefit analysis

Diversification of income sources

Number of income sources per household; presence of non-farm income in households

Questionnaire survey; focus group discussion

Market access, linkage established

Contact with cooperatives; signs of promotion of produce

Focus group discussion

Outputs

Skills of tracking costs, analysing cost–benefit

Active participation in FFS; number of farmers keeping records, logbooks

Participant lists; survey of farm logbooks

Marketing skills

Knowledge of break-even price; negotiation skills

Questionnaire survey; focus group discussion

Market research carried out

Records of market research; best commodity and timing identified

Questionnaire survey; focus group discussion

Source: Authors' own elaboration

Box 11. Farmers’ journey from FFS to a cooperative, Kampong Cham province, Cambodia

“I never dreamed that I would become an officer of a cooperative or that I would produce chili for export,” says Kheang Sipho. “My parents were farmers and knew how hard it was to be a farmer. When I was small, they encouraged me to study hard so that I could I become a professional, a government officer. But history changed all that. After the Pol Pot regime, we were very poor and I could not go to school.” However, with a diploma from a farmer field school, Sipho did indeed become a professional, as the Treasurer of the Kampong Cham Organic Farming Cooperative.

Over the period 2009–2011, the Swedish-supported National IPM Programme, in partnership with the International Fund for Agricultural Development-supported FAO project on Enhancing Agricultural Competitiveness of Rural Households in the Greater Mekong Sub-region, worked with chili farmers in four villages of Kampong Cham province. For the farmers, this involved participating in farmer field schools on organic chili implemented by the Provincial Department of Agriculture and producing chili for export products sold by the Thai Organic Food Company.

©Kaan Ritkhachorn

After the initial FFS, follow-up activities were facilitated in areas where additional training was needed, such as technical aspects of production, as well as group organization and management. Encouraged by their experience of playing a key role in the value chain, and seeing the advantage of working together for stronger collective bargaining, the four farmers’ groups decided to form the Kampong Cham Organic Farming Cooperative.

At the outset, in 2012, the cooperative had just USD 410 in its bank account, primarily made up of membership registration fees (USD 2.50 per person) and initial monthly deposits to the savings account (USD 5 per member). By the end of 2015, the figure had grown to USD 4 800. In 2014 alone, the cooperative sold about 25 tonnes of fresh chili and 800 kg of dried chili, generating an income of USD 23 625.

“We had no idea that our group could become an enterprise… that we could export organic chili. Now we do not only talk about growing organic chili,” said Sipho. “We can grow other organic crops. We talk about how to improve our production; how to make farming profitable….a business.”

Source: FAO. 2018. Farmer’s journey from FFS to a cooperative. www.fao.org/cambodia/programmes-and-projects/success-stories/farmers-journey-from-ffs-to-a-cooperative/ar/

2.3
Summarizing the framework

The indicator types selected by a programme, together with the measures and tools should be listed in the MEL framework. An example of a partially complete MEL framework is given in Table 6. If data collection for several indicators can be combined, this improves the efficiency of MEL. Certain measures or tools could be used for more than one indicator type. Having an overview of the indicators, measures and tools helps programmes with their planning for the human and financial resources needed to collect and manage the data, aspects that will be discussed in Chapter 5.

Table 6. MEL framework with domains and results chain. Examples are given for indicator types, measures and tools as selected by a hypothetical programme

Source: Authors' own elaboration

For this purpose, the rest of the columns 5–8 (shaded in the matrix below) will be completed as part of Chapter 5.

As will be explained in Chapter 4, several indicators of the quality of the FFS process itself will be defined for data collection. Once identified, these should also be listed and compared with the indicators and data needed to assess impact targets, trying to combine the indicators, measures and tools to improve efficiency of data collection and planning. Responsibilities/roles and timelines should be defined to improve the efficiency of the MEL system.

If you want to set up your project’s MEL system, go to Chapter 3 Learning and adaptation to ensure FFS quality. If you are developing an IA go to Chapter 6 Setting up impact assessment for FFS programmes.

Box 12. Common challenges in ensuring the quality of FFS during implementation and in documenting impact of FFS programmes

- ➔ Project staff and master trainers, and even more so facilitators and farmers, often lack an understanding of MEL concepts and tools, and it takes time and resources to build such capacities.

- ➔ Efforts to focus limited resources on field activities often result in improper budgeting for MEL (for workshops to fine-tune the MEL system; training of master trainers and staff; salary of a MEL technical assistant at programme level; coaching of facilitators by master trainers; field supervision missions).

- ➔ When it comes to MEL, the participatory approach inherent to FFS is often eroded in favour of more top-down approaches, with little community and stakeholder participation; programme management pressures and targets tend to take precedence over quality and locally-driven processes (including participation in adapting the learning curricula and indicators to local needs). Participatory MEL is rooted in a culture of accountability to rural communities and local users, which is rarely the norm for public sector activities in most contexts.

- ➔ Once a MEL system is put in place and MEL tools are defined, programmes may face challenges in following through with rigorous implementation and continuous use of the tools by facilitators, zonal focal points, master trainers and programme staff. A MEL system requires sustained management efforts, time and discipline, and initial good intentions may erode with time.

- ➔ Not all programmes have robust built-in feedback loops for communities and grassroots staff to participate in, and influence, the regular programme reviews.

- ➔ The quality of the FFS determines the impacts. Quality considerations need to be integrated into all phases of the project cycle. The MEL system should ensure monitoring for quality during FFS implementation, to identify weaknesses and bottlenecks and address them in a timely manner. Not all programmes have adequate monitoring systems and tools in place.

©Diana Rosero

- ➔ The FFS activities typically last over one or more cropping season or animal cycle and can generate a great deal of data. This includes information on the FFS operations (location, topic, number of participants by gender, number of sessions), as well as technical data (AESA in comparative experiments, yields, and economic analysis), qualitative feedback and planning by the group towards the end of the cycle. Many programmes use field diaries for collection of these data. However, not all projects collect this information, or they may fail to make use of it. That is unfortunate as such information on FFS implementation is relatively easy to collect (compared for instance with impact assessment data), and it provides good feedback on whether the project is on the right track.

- ➔ The FFS generate impacts in the natural, human, social and financial domains, reflecting the basic principles and concepts that underpin farmer field schools. A recent synthesis of FFS impacts shows that most of the efforts in measuring impacts focus on the natural capital domain (such as increased yield, reduced input use), while other domains and dimensions that are difficult to measure are often neglected (Van den Berg, H., Phillips, S., Dicke, M. & Fredrix, M. 2020.), thereby missing essential results of FFS activities (such as on empowerment and unintended results or community initiatives arising from empowerment).

- ➔ Programmes/projects do not always explicitly define the results chain, and the impacts they would like to see, or have not clearly related the data they are collecting from FFS groups to these impacts. This can lead to collecting data that are not relevant to inform decision-making and tracking progress, or collecting too much or too little data.

- ➔ Most programmes lack the time, knowledge or human capacities to analyse the data collected.

- ➔ Changes that occur after the FFS has been completed (evaluation) are not always recorded or documented, or there may be no resources for evaluation once the programme is over. Also, the indicators and the way they are measured may vary (e.g. from one programme to another), making comparisons difficult.

Source: Fredrix, M. & Phillips, S. 2019. Unpublished.

[4] Outputs are defined as the immediate results of the activities, which in the case of FFS are the knowledge and skills gained. Outcomes are short-term (outcomes 1) or medium-term (outcomes 2) consequences of the outputs in terms of changes in behaviour and actions that result from the use of the outputs. Impacts are the long-term consequences of the outcomes (Van den Berg et al., 2020b).

[5] Agroecosystem analysis or AESA refers to the analysis of the systematic field observations of the plant(s) or animal(s), under study, which is conducted weekly by farmers in the FFS learning site, and the interactions of biotic and abiotic factors in the environment, to make informed and timely management decisions.

3
Learning and adaptation to ensure FFS quality

Programme activities are supposed to trigger a process of change that leads to the impact targets. The way the FFS programme is designed and implemented determines whether the desired change will or will not take place.

An FFS programme includes various activities such as: the FFS (which incorporates various methods), the training of facilitators, follow-up activities for FFS graduates, and other supportive activities. In the broad sense, the activities also include problem identification and the design of activities at the beginning of a programme and during programme formulation (see Figure 11).

Figure 11. Focus on the interventions of the results chain

Source: Authors' own elaboration

Guided by the results chain, the MEL process of learning and adaptation is best started by looking for the linkage between interventions and results (see 3.1). Strengthening the interventions (inputs and activities) can involve the enhancement or substantial modifications of activities intended to reach a particular impact target.

This section will show the reader how FFS programmes can use monitoring and evaluation to strengthen the process of learning and adaptation, both at the programme level and at farmer level, as summarized below:

1. 3.1 How does the quality of FFS activities link to the results of FFS programmes?

2. 3.2 Enhancing and modifying FFS activities at programme level

3. 3.3 Setting up MEL mechanisms at farmer level

3.1
Linking the quality of FFS activities to the results of FFS programmes

A common notion is that impacts occur only after the FFS activities have been completed and should be measured in the period following those activities. This is only partially correct. The early signs of the expected outputs and outcomes can already become visible within the period of implementation of the FFS (and during the follow-up activities). During implementation, the causal link between the methods and the early results (outputs and outcomes) can most easily be established (see Figure 12).

Figure 12. A. Diagram of activities followed by results; B. Diagram showing how within the FFS activities, methods and their results are closely interrelated

Source: Authors' own elaboration

A good example of an early result is the analytical and communication skills that participants develop during the FFS sessions. These skills are the direct result of field observations and group discussions practised as part of FFS key learning activities. If these skills are not adequately developing among participants, this is a sign that the field observations or group process should be changed. Hence, the activities-result linkages may directly inform a programme about improvements needed – so that activities can be modified in subsequent FFS seasons/production cycles. If this issue is not addressed, the impact targets will be difficult, or worse, may not be attained.

The key guiding principle of the FFS enables the methodology to be adapted to address a wide range of location-specific issues, allowing practitioners to mould the FFS according to local needs, demands challenges (see the FFS Guidance Document). This implies the constant improvement of FFS activities, so that it remains relevant and brings the intended results in all aspects of farmers’ lives. In FFS activities, the four livelihood domains (human, social, natural and financial) can be closely intertwined. For example, the core activity of agroecosystem analysis stimulates analytic and critical thinking (human domain), supports group cohesion (social domain), promotes ecological learning (natural domain), and helps farmers in tracking costs and benefits (financial domain). Modifications in FFS activities can potentially influence the outcomes in multiple domains.

Table 7 presents examples (taken from Tables 2–5) of some types of ‘early results’ (outputs and outcomes) during the FFS, or during its follow-up activities. For instance, in the FFS, farmers improve their communication skills; they start questioning old norms or habits; women may increase their role in the group; women’s contributions could become increasingly frequent; or farmers could establish a savings and loans scheme. Some of these ‘early results’ contribute directly to the impact targets. Programmes should identify in their results chain those early results that contribute to achieving the impact targets in each domain. Those early results should be given particular attention in the monitoring of FFS activities, for example, by using FFS quality checklists or matrices for FFS monitoring and supervision activities.

Table 7. Examples of ‘early results‘ in the four domains that could become visible during implementation of the FFS or follow-up activities. Monitoring of these ‘early results’ during the FFS could inform the need to strengthen or modify the activities and methods

Domain

Outputs

Outcomes 1

Outcomes 2

HUMAN

Improved analytic skills Knowledge, awareness

Decision-making capacity Critical thinking, questioning Experimentation

Problem-solving

Confidence, motivation Change of attitude, perception

SOCIAL

Skills in presenting

Skills in communicating and negotiating

Trust, group cohesion Speaking out, reaching out

Group decision-making Collective action

Increased role of women (in discussions, decisions)

Reduced conflict

NATURAL

AESA skills, technical skills Knowledge of practices

Use and adaptation of ecosystem management practices

Improved food production Crop diversification

FINANCIAL

Skills in tracking costs Marketing skills Market research done

Reduced input cost Added sources of income

Access to loans Savings

‘Early results’ could be outputs and outcomes.

Outputs = the immediate results of activities, which in the case of FFS are the knowledge and skills gained.

Outcomes = the short-term (outcomes-1) or medium-term (outcomes-2) consequences of the outputs in terms of changes in behaviour and actions that result from the use of the outputs.

Source: Authors' own elaboration

When the linkage between the activities and the results becomes clear, plans can be made on how to enhance or modify the activities (see Figure 13). These plans should include the suggestions of farmers, which are often highly relevant. For example, it will be clear that exercises in record-keeping, calculation of inputs and outputs, and knowledge about the break-even price are essential to help farmers become ‘financially’ literate. Hence, where financial literacy is vital to achieve financial security, such exercises should be incorporated in the FFS curriculum.

Figure 13. Diagram showing regular review leading to enhancement and modification of activities for implementation

Source: Authors' own elaboration

In some cases, the specific linkage between the activities and results is less straightforward, but needs to be explored. For example, certain changes in the group work or in plenary discussions at the FFS could increase participation and confidence among women, but some changes might work better than others. This could be discovered through pilot testing.

3.2
Enhancing and modifying FFS activities at programme level

A central concept of MEL is to use the collected data to reflect on what was found and to use that knowledge to improve the FFS activities, with the intention of attaining the impact targets. This process of learning and adaptation is most critical during the programme’s pilot stage, but learning and adaptation should continue during a programme’s full duration.

The data on FFS implementation obtained through monitoring and evaluation should be collected and reviewed regularly, to reflect on the progress made towards achieving the desired change (see Figure 14). Such regular reviews produce recommendations on how to enhance or modify the activities to improve their quality, which, after conducting refresher training for facilitators, will be implemented or pilot tested. As such, in this chapter, the focus will be on improving the quality of the interventions (problem identification, design and activities) that are part of the results chain (see Figure 4. The results chain).

Figure 14. The cycle of learning and adaptation

Source: Authors' own elaboration

3.2.1 Setting up mechanisms and tools to enhance existing activities

When an impact target has been set in each domain, and milestone indicators towards reaching the impact targets have been identified (see section 2.2 Identifying milestone indicators towards impact targets), it is appropriate to examine how the existing activities could be enhanced or adjusted.

For example, if ‘farmer empowerment’ is the impact target in the human domain, it will be crucial to ensure that elements such as critical thinking, experimentation and a change of mindset are adequately addressed or fostered, in the agroecosystem analysis, or through group discussion or planning exercises. In part, these elements can be addressed through quality assurance, for example, by paying extra attention to the quality of data analysis and decision-making resulting from field observations of field studies and AESA carried out by farmers in FFS sessions.

The quality of FFS implementation is to a large extent determined by the ability, competencies and motivation of the facilitators. The training and mentoring of FFS facilitators, and the involvement of farmer facilitators, deserves frequent evaluation to identify areas for improvements (for example, through refresher training).

Programmes should develop their own quality matrix for the purpose of monitoring their operations, so that the activities can be enhanced and steered towards achieving the expected outputs, outcomes and impact targets. The quality matrix also assists in safeguarding the quality of the FFS in the general sense, for example, by ensuring that the educational principles of the FFS are not compromised or neglected. Examples of quality indicators of the FFS activities that can be included in a matrix are shown in Table 8. Non-negotiable elements of FFS (as presented in the Introduction) can also be a starting point to build FFS quality matrices adapted to the programme (see Part II, Tool 1.12 Quality matrix and Table 15. MEL framework with domains and results chain. Examples are given for methods, roles and resources for a hypothetical programme).

Table 8. Examples of types of quality indicators of the FFS for monitoring purposes that can be included in a quality matrix or checklist

Category

Quality indicator type

Considerations

Preparation

FFS location

Criteria for selecting FFS locations revisited

FFS participants

Optimal group size and gender ratio Criteria for selecting participants revisited

Local ownership

Farmers involved in identifying problems, setting objectives, adapting curriculum, planning FFS study plots/comparative experiments

Process

Agroecosystem analysis

Communal study plots/sites with simple/relevant comparative treatments Comprehensive observations and records at fixed intervals

Observations form the basis for critical decision-making

Comparison made between ‘farmer practice’ and ‘improved practice’ plots Decision considers all observations

Comparison made with AESA from previous week ‘What if’ scenarios discussed

Special topics

Relevant and timely topics, flexible to include emerging issues Facilitation instead of lecturing

Technical depth and clarity of topic

Participation

Consistent attendance

Active participation in field, group work and discussions Equal gender participation

Participatory evaluation conducted

Field day

Participants (farmers, local leaders) Active sharing and discussion

Management

Facilitator

Skills in facilitation and leadership Management of FFS sessions according to plan Certified training and refresher training received

Record-keeping

Relevant records, documentation and feedback for local use Reporting to supervisor/programme management

Support

Supervisory support received, regular coordination meetings

Source: Authors' own elaboration

3.2.2 Organizing regular reviews of FFS implementation

Most programmes already have a mechanism in place for interim review or evaluation. Such mechanisms allow the programme to be adjusted or corrected, based on an evaluation of the ongoing FFS. Usually, however, the interim review is not a regular activity, but typically takes place only once – at mid-term and focuses more on the process of implementation, rather than on the activities that should lead to the impact. Interim reviews are not sufficiently frequent to ensure that learning and adaptation take root.

In a MEL system, a regular review at annual or semi-annual intervals is a key element in the cycle of learning and adaptation. Regular review is ideally carried out in the form of a workshop, with the participation of farmers, facilitators, master trainers and resource persons. In the workshop, data from monitoring and evaluation are reviewed, and methods to improve the activities discussed. It is advisable to appoint a technical team to oversee and plan the regular review process.

Box 13. Regular review workshop

- ➔ Purpose: To review monitoring and evaluation data in order to assess the status of the activities based on the logframe and plan how to make improvements as needed

- ➔ When: At annual or semi-annual intervals

- ➔ Who: MEL team and representatives of farmers, facilitators, master trainers, resource persons

In preparation for the regular review, monitoring and evaluation data from the previous period of implementation (e.g. one or two seasons/production cycles) are compiled and the results summarized. This is done by the MEL team, or a third party.

The summarized results with key findings regarding quality indicators and milestone indicators contained in the logframe are reviewed by the workshop participants. The results are the basis for discussions on how to improve programme implementation, which may involve enhancement or modifications of activities or adjusting timelines and responsibilities (see section 3.2.3). In the case of pilot testing of modified activities in the preceding season/production cycle, the data are used to decide whether the modifications have been successful.

Source: Cartoon by Julie Smith – https://www.espa.ac.uk/files/espa/Intro-To-Monitoring-And-Evaluation.pdf.

A workplan with timeline should be prepared for the introduction of the enhanced or modified FFS programme. This could involve a technical workshop on modification of activities, followed by the updating of guidelines, refresher training for facilitators, and planning for actions for pilot testing or implementation of the modified activities.

3.2.3 Introducing structural modifications of activities and methods

Besides improving existing activities, it may be deemed necessary to make more structural modifications in the FFS to achieve the desired impacts. This could be initiated through a workshop to adapt the FFS curriculum or to design new components of activities.

©FAO/Andrea Theodor

Some examples of structural modifications at programme level, or at FFS level, are given here. It is recommended that programmes or countries develop their own enhancements and modifications according to the local situation.

- ➔ If ‘conflict management in communities’ or ‘good cooperation at household level’ is targeted as the impact in the social domain, a programme could ensure that communication skills, trust and mutual respect are paid special attention in the FFS (PELUM-Uganda, 2016). For example, more time could be devoted to group work and presentations during the FFS, or group dynamics exercises could be introduced that build trust or transparency. A programme could also make structural modifications by introducing methods that facilitate dialogue around gender and social norms among men and boys; or by enabling FFS participants to share lessons with their non-participating spouses at home; or by selecting both husband and wife as FFS participants, thereby aiming at cooperation and harmony within the household.

- ➔ As another example, if ‘financial security’ is the impact target in the financial domain, it will be crucial to ensure that farmers learn about record-keeping, cost-benefit analysis, marketing and market research, and that women increase their influence in household decision-making. This may require that specific learning topics are added to the FFS curriculum or as a follow-up activity. Many FFS programmes in all regions of the world have evolved to include elements of farmer market schools, farmer business schools or village savings and loans associations.

- ➔ If a ‘sustainable production system’ is the impact target in the natural domain, the farmers or programme could decide to expand the field study plots/comparative experiments of the FFS, to increase options for experimentation by farmers, and to promote crop/animal diversification and soil nutrient management.

In West Africa, the FFS on cotton evolved after 2004 to take into account the whole cotton production system, including its rotations, associations and livestock integration (cotton-cereal-leguminous crops-livestock FFS). In Andhra Pradesh, India, single crop farmer field schools evolved after 2018 into year-round FFS based on intensive polycropping, integration with livestock and trees and year-round successions and livelihood models based on the agroecological approach of the Community-Managed Natural Farming Programme. Programmes can also introduce structural change by adding follow-up activities in support of experimentation in the years after the FFS has been conducted.

More substantial modifications may be necessary if a programme revisits its technical entry point or problem identification, which served as the basis for the design of the programme. If the programme finds that the problem identification was too restricted at the outset, or proved to be less relevant in certain targeted regions, the set of activities may require a redesign to accommodate local needs and demands. Also, criteria for targeting FFS locations and selecting FFS participants may require modification in accordance with lessons learned.

This process of learning and adaptation is most critical during the programme’s pilot stage, but it may also be necessary to introduce structural changes later in a project or programme.

3.2.4 Revising guidelines and refresher training

Guidelines for implementation of activities should be appropriately updated to accommodate any new enhancements or modifications made. The programme’s master trainers and FFS facilitators will then require refresher training in the adjusted methods before the new methods can be implemented or tested.

Every ToF course should include specific sessions on monitoring and evaluation of farmer field schools. The training sessions should be accompanied by clear and updated MEL templates and guidelines to be reviewed by the participants. Moreover, in line with the continuous improvement process that underpins the implementation of FFS, master trainers and facilitators should be retrained periodically (for example, at the end of each learning cycle). The content of retraining (called ‘refresher courses’) should be based on a review of the MEL results and a participatory assessment of additional training needs, as well as on the performance of facilitators and master trainers as assessed by programme teams, or on new themes, technologies or tools that the programme wants to introduce.

The assessment may cover:

- ➔ an analysis of the shortcomings identified in the content of the initial training of master trainers and facilitators;

- ➔ an analysis of the shortcomings identified in the process of conducting FFS and in the support given to facilitators while conducting the FFS;

- ➔ an analysis of training needs accounting for any new challenges faced by FFS facilitators and members.

Different methods are used to identify the need for further training (refresher courses) of master trainers and facilitators.

When needs for refresher courses are identified, the programme should always ensure:

- ➔ their relevance to the problems to be solved; and

- ➔ the feasibility of the training in relation to costs, available skills, motivation and commitment of the possible participants.

3.2.5 Pilot testing and assessing structural modifications

Structural modifications of the activities may require pilot testing in selected locations, followed by further learning and adaptation, before being adopted by the entire programme. The results of pilot testing provide the programme with important learning opportunities, especially if well documented and shared. Hence, intensified monitoring should be in place to evaluate if, and how, the modified activities contribute to the desired results. These findings will serve as important feedback for the next learning workshop.

3.3
Setting up MEL mechanisms at farmer level

So far, most of the discussion on the planning for MEL has referred to MEL at programme level, seeking to enhance or modify a programme’s activities to achieve the desired impact targets. To encourage congruence between ideas and approaches throughout the programme, farmer and community representatives should be part of the team designing MEL from the outset. A separate but equally important platform for MEL is at farmer level or FFS group level, conducted by the farmers and local leaders as the direct beneficiaries of the activities (Figure 15).

Figure 15. Schematic illustration of the MEL cycle at programme level and farmer level

Source: Authors' own elaboration

Participatory MEL helps farmers to evaluate the process, facilitation and contents of the FFS sessions; monitor and track their own performance and progress (as individuals, as a group); evaluate the functioning of their groups; and evaluate whether the FFS has assisted farmers in improving their livelihood situation. Several participatory methods for monitoring and evaluation by farmers are available, as will be discussed in Chapter 4 Tools for monitoring and evaluation. For example, spider diagramming is a simple tool for farmer participatory evaluation of the effects of an FFS.

The results of participatory monitoring and evaluation will help in deciding how to enhance or modify the activities. This could be carried out in a workshop at farmer level.

©FAO/Francisco Martinez

The purpose of participatory MEL is to generate ownership among participating farmers of the activities, so that adaptations in FFS methods can be made when needed, and so that innovation and learning will continue after the completion of programme activities. Participatory MEL also aims to provide FFS facilitators and programme management with feedback on the acceptability, feasibility and sustainability of activities. For example, the MEL Framework (see Table 6. MEL framework with domains and results chain. Examples are given for indicator types, measures and tools as selected by a hypothetical programme, section 2.3) can be enriched with indicators and measures for social gains defined by farmers (an example is shown in the callout in Table 9). These are based on farmers’ awareness of the conditions in which they live and their capacity to act to improve their families’ and community’s development (for example, FFS alumni selected as members of village committees due to their improved analytical and planning skills; see Table 9).

Box 14. From the field…

In the FFS in Togo, during the debriefing workshop at the end of a cropping season, farmers were asked what they liked, what they found irrelevant and what improvements they would suggest. In cases where the problems identified were about FFS facilitation, the discussions had to be moderated by other facilitators, to allow farmers to speak freely. For some cases and issues, it would also be useful to talk separately with men, women and youth.

Source: Bakker, T. 2021. Unpublished.

Table 9. Enriching the MEL framework with indicator and measure of social gain defined by farmers

Source: Authors' own elaboration

Part II of this toolkit includes several tools to involve FFS producers and their communities in monitoring and evaluating the activities. See in particular section 1 Tools for monitoring and 2 Tools for evaluation.

4
Tools for monitoring and evaluation

The sections in this chapter provide an overview of several tools to collect quantitative and qualitative data for monitoring and evaluation of an FFS. More detailed information on when and how to use each tool, and what they are, is presented in the toolkit – Part II of this document.

Box 15. Collecting quantitative versus qualitative data

Monitoring and evaluation may involve the collection of quantitative or qualitative data. Some examples of quantitative data are the number of FFS participants (and percentage of women participants), the duration of FFS sessions, yields of field plots/comparative experiments, and the scoring results of perceived benefits. Some examples of qualitative data are the quality of participation (such as a description of how women contributed to the discussions), the process and quality of decisions made (such as which considerations were made, which scenarios included), the quality of facilitation (such as how participants were stimulated to think and discover), and the empowerment of women farmers.

Quantitative data are generally easier to collect, process and analyse than qualitative data. For this reason, monitoring, evaluation and impact assessment studies often use quantitative data. However, many aspects of the implementation and results of the FFS cannot be captured in numbers or percentages, but need to be described in narrative terms. For instance, the quality of an exercise on agroecosystem analysis or the learning process in the FFS cannot simply be measured by numbers, but demand a narrative description of the process. The use of scores (for example, 1–5 as in spider diagramming) can in some cases reduce a complex process or outcome to a simple measure. For example, rather than farmers describing how they have experienced becoming empowered, they could simply be asked to give a score (1–5) to indicate their perceived level of empowerment. The score is much easier to measure, but the measure provides limited insight into the subject matter (in this case, empowerment).

A preferred approach is a ‘mixed methods’ evaluation, using a combination of quantitative and qualitative data sources (such as focus group discussion and spider diagramming), so as to measure a given process while providing a deeper understanding of it. Such a combination also provides triangulation (or cross verification) of results from more than one source.

Source: Bamberger, M. 2012. Introduction to mixed methods in impact evaluation. Impact Evaluation Notes, 3:1–38. www.interaction.org/wp-content/uploads/2019/03/Mixed-Methods-in-Impact-Evaluation-English.pdf

4.1
Monitoring tools

Monitoring of FFS activities is intended to answer questions related to the quality and scale of implementation, specifically in relation to the impact targets. Monitoring data contributes to evaluation. Monitoring progress of implementation is also important for course correction, adaptive management and to account for the use of resources to donors and government stakeholders.

Participatory monitoring and evaluation allows farmers to monitor their progress, and to make an appraisal of how the FFS is being implemented and facilitated, and how it is benefiting them.

Potential monitoring tools include those identified in Table 10. In addition, narrative reports (including photographs, videos) can provide important information about FFS quality, progress and emerging problems.

Table 10. Main monitoring tools for collecting data of FFS activities (see Part II)

Tool

When

Description

As indicator of

AESA records

Weekly

Weekly farmer-drawn graphs and records

Quality of AESA process, decisions

Cost–benefit records

Throughout the season/production cycle, cost of inputs is recorded

Outputs calculated at end of the season/production cycle

Quantity and prices of inputs and outputs from experimental plots

Financial comparison between plots

FFS start-up reports

At start of the season/production cycle

Description of farmer participation in identification of problems and objectives, curriculum development

Local programme ownership; local adaptation of curriculum

Field day reports, with photos

End of the season/production cycle

Description of field day at closure of FFS

FFS quality, dissemination, linkages

Participant lists

Every FFS session

FFS attendance records (with gender, social category – if appropriate)

Average and consistency of attendance

Participatory evaluation

Every FFS session and end of the season/production cycle

Participants’ appraisal of FFS process

Quality of facilitation, process, relevance

Participatory performance tracker

Schedule to be determined by the group

Self-reporting on key practices or behaviours at individual and group level

Progress in use of key practices or behaviours

Pre/post-ballot box test results

At start and end of the season/production cycle

Knowledge test before vs after the FFS

Progress in knowledge

Reports of facilitator meetings

During each facilitator’s meeting

Description of progress by facilitators

Progress, problems, emerging issues, lessons learned

Reports of monitoring visits

During each monitoring visit

Description of progress by supervisors

Progress, problems, emerging issues, lessons learned

Results of study plots/treatments*

End of the season/production cycle or on completion of the study

Yield and other parameters of each experimental plot/treatment

Yield comparison between plots

Quality matrix

During each monitoring visit

Norms and standards for quality FFS

Progress, problems, emerging issues, lessons learned

* When available from many FFS locations, these results give an impression of the potential to improve farmer practices and can highlight locations where further attention is needed.

Source: Authors' own elaboration

Start-up reports are of particular importance, as they are crucial for monitoring and evaluating whether or not the participant farmers are given adequate ownership of the FFS process, by identifying local problems, setting the objectives and suggesting content and adaptations in the FFS curriculum prior to implementation of the FFS.

Recent developments have seen the use of ICT tools in field-level monitoring, particularly applications for mobile/smartphones or tablets to monitor FFS implementation. Similar to the more common approaches used in MEL systems (such as use of quality matrices), feedback from ICT applications (such as WhatsApp) is used as inputs in improving FFS implementation to ensure quality FFS.

The availability of a long list of monitoring tools does not mean that all tools should be selected by a programme. Nor does it mean that all collected data should be reported to the central level and stored in a database. It is important to consider that certain data are very useful at the local level, but if data are collected from all locations, they may overwhelm project management, and end up sitting in a database or in filing cabinets without being used, thereby wasting resources used for collecting data. Not all data are needed or desired at programme or national level. Lean and purposeful monitoring systems are preferable.

The data collected through participant lists, AESA records, participatory performance tracking, and participatory evaluation of each FFS are useful learning tools for participant farmers and their facilitators, to monitor progress made by individuals and by the group (see Table 11). A sample of such data, from a representative selection of FFS units, is useful at programme management level to monitor the quality of FFS activities. Donors and government stakeholders will be interested in aspects of participation, results of study plots/comparative experiments, and cost–benefit records, but will require a simple summary of those data, supplemented by some in-depth case studies. The same is true of the international community. Sharing summarized data through dedicated websites, newsletters or research articles will be of benefit to FFS practitioners and experts in other countries and regions.

Table 11. Example of an assessment of possible monitoring tools for collecting data of FFS activities, with potential users of the data presented for each tool

* Public domain

Notes: The table indicates whether all data, a sample of data, or a summary of data is needed for each user and tool. Programmes must not select all tools at all levels, but select those needed for reaching their monitoring objectives.

Source: Authors' own elaboration

Reports of the FFS, monitoring visits or facilitator meetings provide options for obtaining open-ended feedback from the field, including unexpected problems or emerging issues, as well as positive or negative consequences. These reports will help facilitator teams to address problems or shortcomings. Key issues should be flagged and taken up at programme management level.

The intensity of data collection depends on the stage of a programme. In the pilot stage, it may be desirable to collect monitoring data from all FFS locations, but once the programme expands, the monitoring apparatus should be adapted to collect only data from a representative sample of locations.

4.2
Evaluation tools

Evaluation tools differ from tools that are used for routine monitoring. The evaluation is concerned with the results of the FFS activities, but monitoring data of the implementation process can enrich the evaluation by providing information about the quality of the FFS programme that is being evaluated. Locations with higher quality or relevance of the activities are expected to have better results.

Apart from a survey of farm logbooks (individual or FFS record books or diaries), most evaluation tools are general tools for data collection that can be used to gather various qualitative and quantitative data. Several evaluation tools are presented in Table 12.

Table 12. Possible evaluation tools for collecting data of FFS activities, with a short description and purpose (see Part II)

Tool

Description

Use for

Focus group discussion

Discussion among farmers on specific topic of interest

Insight into diversity of opinions and experiences among farmers with similar background

In-depth interview/Key informant interview

Structured or open-ended interview to capture rich information from a stakeholder

Deeper understanding about changes in behaviour, causes, reasons, attitudes Insight into pressing issues

Spider diagramming

Assigning scores to selected parameters by farmers

Simple quantification of farmer perceptions or other indicators

Most-significant-change

In-depth stories about significant changes caused by the activities

Identifying most significant changes experienced by farmers

Participant observation

Qualitative observation of participant behaviour; this should be triangulated with other data and/or documented with photos or videos

First-hand evidence of a change in behaviour, practices, interactions

Picturing change

Farmers’ own description of perceived impact, using photographs

Understanding the diversity of impacts Can be used in addition to other data sources by highlighting unexpected impacts

Questionnaire survey

Brief set of questions for medium/large sample of farmers

Quantitative or qualitative results of the FFS (baseline and end-line surveys) Can be used to ask a large sample of farmers about quantitative information regarding production or qualitative data such as perceptions or behaviour

Survey of farm logbooks

Farmer records of farm inputs, outputs

Expenses, income, profit

Source: Authors' own elaboration

There are more participatory tools available for evaluation purposes. CARE’s Participatory Performance Tracker is a self-monitoring tool that can be used to measure progress on any practice or behaviour, in response to a programme’s specific focus; this tool, which uses tailor-made tracking sheets on practices or behaviours, is used at individual and group level (CARE, 2015). Spider diagramming is used for any variable (such as level of confidence; financial benefits) and asks farmers to give a score (from 1 to 5) of how they perceive the variable (5 is very good), comparing this with the scoring before the FFS (Mancini et al., 2007). Spider diagramming is a quick and simple tool for participatory evaluation, but it does not provide in-depth insight into each variable. Community visioning/vision mapping, when combined with other tools, may be useful for evaluation to understand what the community wants to achieve.

Likewise, the MEL Framework (see Table 6, section 2.3) can be enriched with tools that farmers can use to measure the indicator for certain gains they have defined (see callout under tool in Table 13).

Table 13. Enriching the MEL framework with evaluation tools for farmers’ use

Source: Authors' own elaboration

5
Management of MEL

A MEL system will require actions by several stakeholders at various levels: farmers, facilitators, local partners, project managers, etc. These stakeholders will have different roles in the collection, reporting and management of the data. An effective MEL system could potentially be linked to, or adopted by, national-level frameworks for monitoring and evaluation, increasing prospects for sustained implementation of quality FFS programmes (Schwandt, 2018).

This section will provide the reader with key information on how to manage MEL, as summarized below:

1. 5.1 Organization and oversight

2. 5.2 Reporting and data management

3. 5.3 Institutionalizing MEL

5.1
Organization and oversight

5.1.1 Roles of stakeholders in the management of MEL

Defining and setting up a well-structured and operational mechanism for an MEL system is a critical starting point. For instance, at national level the system should be built on a structure based on focal points at municipal, regional, departmental and national level (it will differ from country to country). Focal points have specific terms of reference and must follow a well-defined reporting system, with a flow that rises from the local to national level. Within this system, the FFS supervision will be ensured by field visits through focal points who directly assist the facilitators and farmers/producers in the field, and by periodic meetings between the focal points and the facilitators. Table 14 provides ideas on the roles of the different stakeholders in the management of MEL in government-led FFS programmes. The actors and roles may be different in non-government-led FFS projects, but the matrix below can be adapted.

Table 14. Stakeholders and their roles in the management of MEL

Stakeholder

Roles and responsibilities

Steering committee

Provides technical oversight of the design and implementation of MEL, assign roles in MEL, and provide leverage in mobilizing the necessary resources for MEL.

Where needed, establishes technical working groups on pertinent topics, e.g. develop guidelines, adapt activities, or initiate an evaluation study on specific indicator types or impacts.

Ensures that the MEL activities satisfy the needs of the farmers, funders and government stakeholders.

Project coordinator and MEL expert or unit

Takes the lead in coordinating monitoring, evaluation and impact assessment activities.

Organizes the reorientation process (see Annex 1 MEL reorientation in existing programmes), oversees and plans the regular review process and the modification of activities.

Ensures that the outcomes of the reorientation process and regular review are shared with the funders and with government stakeholders.

Master trainers and zonal FFS focal points

Plays a fundamental role in both the design and implementation of an FFS MEL system.

Supports the programme coordinator as well as the MEL expert in designing the MEL tools and templates and the MEL system (i.e. defining who collects what information, when, why, how and to send to whom).

Master trainers

Provides a sounding board for the coordinator or the MEL expert on the feasibility and appropriateness of indicators and MEL processes and time frames proposed.

Plays an essential role in training facilitators in use of the tools, making sure they are clear and simple to use, and in getting feedback from facilitators to fine-tune the templates or processes being introduced.

Plays a key role in information collection – supervising facilitators to ensure that MEL templates are filled in and submitted according to schedule (i.e. if the master trainer is also a focal point in the area).

Relays to the programme management the concerns and issues encountered by facilitators; suggests regular improvements; and alerts on sub-performance and emerging issues from the field.

Other actors

At field level, farmers and facilitators

Farmer representatives validate the results chain and choose tools (e.g. picturing change) to track whether the intervention is successful from their point of view.

Engage in MEL within farmer groups.

Are involved in monitoring performance through attendance rates, participation levels, agroecosystem analysis, knowledge test sheets, self-reflection, and participatory evaluation.

Farmer representatives participate in MEL review workshops.

At district or central level, master trainers and MEL teams

Conduct monitoring visits of the FFS and other activities, to mentor and supervise the facilitators and improve quality of methods, specifically steering towards the impact targets.

Monitor and assure quality of activities (including timely implementation), to evaluate the outputs and outcomes of the activities.

Collaborating research teams and resource persons

Conduct evaluation studies or impact assessment studies on specific outputs, outcomes and impacts in each domain, while providing advice on how activities could be modified to achieve the desired impacts.

Universities and consulting firms

Can offer training in a range of assessment methodologies and tools for impact assessment studies.

Civil society

Plays a major role in implementation of a monitoring and evaluation strategy by promoting informed debate on administrative policies, based on the findings of monitoring and evaluation.

Source: Authors' own elaboration

In countries where one or more programmes or projects use the FFS approach, it may be desirable to establish one harmonized framework and system for MEL. Harmonization of MEL across programmes and projects will increase the efficiency of MEL activities and consistency of the activities and data collection, provided that adequate attention is paid to local differences (see section 5.3 Institutionalizing MEL).

5.1.2 Financial and human requirements of a MEL system

A functional MEL system requires the necessary financial and human resources, which should be a standard programme cost with its own budget line (see Figure 16). As a rule of thumb, programmes or projects should assign approximately 5–10 percent of their budgets to monitoring and evaluation. However, some small FFS projects may not have an existing M&E component in their budget or workplan.

Figure 16. Budget categories required for FFS programmes

Source: FAO. 2016. Farmer field school guidance document: Planning for quality programmes. Rome. www.fao.org/documents/card/en/c/d7d4db1f-826f-4d81-b097-44292ff7eeca/

Because human and financial resources for MEL may be limited, it is critical that redundancies in data collection and data processing are regularly identified and removed from the MEL system. Redundancies are data collected that are not processed or used. Freeing up resources by making the MEL system more target-oriented and leaner may benefit the process of learning and adaptation, for example, by introducing frequent reviews and modifications.

In addition, programmes and projects should actively engage with research agencies to explore the opportunities for hosting undergraduate or graduate students to conduct their field research – a generally underutilized but low-cost resource option to assist programmes in their evaluation activities. For example, a student could be hosted to conduct questionnaire surveys and data analysis exploring specific human or social impact targets.

Source: Cartoon by Julie Smith – https://www.ifrc.org/sites/default/files/2021-09/IFRC-ME-Guide-8-2011.pdf

In the reorientation towards MEL, it may be justified to allocate extra resources to the MEL system, especially when the added value of MEL in improving activities and reaching impact targets is recognized. Such extra resources could be used to support regular review, learning, adaptation and pilot testing activities of the FFS programme.

5.2
Sampling, reporting and data management

5.2.1 Selecting the sample

Pilot projects or small-scale programmes may be able to monitor or evaluate all the FFS units, or all the activities covered. However, larger programmes may not have the resources to cover all FFS units, or all activities, with monitoring and evaluation. Therefore, these programmes should select a share (sample) of FFS units or activities, to represent the entire population covered by the programme.

The unit of sampling should be carefully identified. The sampling unit can, for example, be the FFS unit (such as FFS units to be monitored during a season/production cycle), or individual persons (such as persons included in a questionnaire survey; or agricultural production per farmer). In general, monitoring most commonly uses the FFS as a sampling unit, whereas evaluation commonly uses the farmer as a sampling unit; however, there are many exceptions.

The sample is selected as a subset of the total number of FFS units, or the total number of targeted farmers. A sample consists of a number of sampling units (such as FFS units, or farmers). Not all FFS units, or farmers trained, can be included in the sample because most programmes only have limited financial resources and time available for monitoring and evaluation.

In order to be representative of the programme, the size of the sample needs to be large enough to capture the diversity of the FFS and/or farmers participating in the programme. For qualitative analysis, a good maximum sample size is usually 10 percent of the total; for instance, if we have 100 farmer field schools as part of a project, interviewing 10 would be considered sufficient; however, the larger the sample, the more representative the results.

Once the size of the sample is defined, the FFS or farmers should ideally be randomly selected using a set of criteria. Failing to select randomly can result in a sample that is biased. For example, a sample of FFS units might be biased towards FFS units that are easily accessible by road, thus neglecting poorly accessible areas. Or a sample of farmers taken for conducting a questionnaire could easily be biased towards those farmers who are more vocal or dominant than others, unless the sample is taken randomly from the list of eligible farmers. A biased sample will not give an accurate representation of the targeted units or population.

Random selection of sample units is typically done by taking a comprehensive list of all numbered FFS units or lists of all individual farmers covered by the programme. From the list, a fixed number of individual units – the sample size – is selected through a lottery, or through a similar method of randomization (for example, using the MS Excel formula: =RANDBETWEEN(1,y)). If this method of randomization is too cumbersome, a cruder but quite acceptable method is to select units at regular distances from the lists, for example, to select every 5th or 10th item on the list, until the sample size is met.

5.2.2 Reporting

Data collected should mostly be used at the level at which they are obtained, for the direct benefit of those involved. For example, monitoring data on the quality of agroecosystem analysis, or the participatory evaluation of the FFS process by farmers, should mainly be discussed within that group.

Where the FFS has two facilitators, the division of roles as lead facilitator and reporting facilitator will enable peer assessment, thereby leading to improved implementation.

Reporting of data implies the transfer of those data from the level where they were collected to the next level upwards, to the programme management team. Data transfer is conducted by sending written records or reports, or by transmission of data in real time through tablets or mobile apps. The purpose of reporting is to inform those responsible for decisions on implementation and use of funds (master trainers, district teams, programme management, steering committee, donors) about the performance and results of the activities.

Reporting of monitoring and evaluation data is an essential part of the MEL learning cycle. Reported data are the input for the regular review process. These data allow the programme to track the progress and quality of its activities and examine whether the outputs and outcomes are leading towards the impact targets.

Some programmes use phone apps to send data directly to their supervisors or data managers, as a way of accelerating the data reporting. However, this tool has its limitations when it comes to qualitative data or sharing of experiences.

A protocol for data collection should include details of the data types, reporting format, data collection frequency and sample size. Also, the appropriate method of data processing should be determined at each level. For example, the results of participatory evaluation could be discussed locally between facilitators and FFS participants, while the main points of these discussions are reported to the master trainers or programme management.

A clear mechanism with well-defined actors and ToRs is important to ensure a functioning MEL system. Establishing an efficient MEL system is not an easy task as it involves several people at different levels, each with different tasks and using several tools. The responsibilities, the reporting flow and formats, the timing and tools to be used have to be very clear to all actors. A single link in the chain that does not work is enough for the whole chain to collapse.

5.2.3 Data management

Data management is the processing, storing and quality assurance of reported data. Data management can take place at different levels. At the level of the FFS, data are mostly kept and processed in paper format. However, the use of tablets or mobile applications can enable transmission of the collected data in real time to the computer server maintained at central level as shown below in the photograph from the ongoing project “Transforming Indus Basin with Climate Resilient Agriculture and Water Management” in Pakistan. The illustration shows how the use of mobile-based data collection and management facilitates effective monitoring and evaluation of FFS.

Not all data should be processed at programme level. When certain data are no longer considered relevant at central level, these data should not be included at that level. For example, in a pilot project it may be useful to keep track of detailed outcomes of FFS sessions, but as more experience is gained after some years, these detailed data are no longer required, so do not need to be processed and stored.

At programme level, relevant data should be processed and entered into a database to allow production summary statistics and to flag locations with field problems. Both quantitative data (numbers) and qualitative data (such as text, photos) should be processed, as long as they are relevant to the programme. A well-organized database ensures learning from, and systematization of, the experiences and results of each FFS (such as best practices and experiments) for the design of future farmer field schools. A good database should also include information on human resources (master trainers and facilitators), referenced with zones of FFS interventions. Depending on the size of the programme, the MEL unit should include a full-time or part-time database manager.

To assure data quality, the MEL unit makes spot-checks (selected samples) of the incoming forms, supplemented by field observations, to look for anomalies. Issues of poor data quality should be addressed through appropriate corrective actions.

5.2.4 Analysis of data

The analysis of MEL data (including the results of impact assessment studies) involves the production of summary statistics for information sharing, as well as the exploration of more complex data. Data analysis is accompanied by the interpretation of results. Interpretation of complex results, including qualitative data, will benefit from the assistance of resource persons with a technical or social science background.

The main results should be packaged into concise portions (summarized text, tables, graphs) that could be tailor-made for feedback to the community, facilitators or master trainers. The results may need different packaging (for example with more details on methods and analysis) for targeting of donors, national partners or researchers.

©FAO/K. Jam

5.2.5 Review and feedback

The main purpose of MEL is to improve FFS activities. Therefore, the results obtained through monitoring and evaluation should be used as feedback to master trainers and FFS facilitators, in order to make gradual improvements towards achieving the impact targets.

An annual or semi-annual review workshop is a critical element in the MEL learning cycle (see section 4.1 Monitoring tools). The regular review should be used to check if all collected data are still relevant, or if some types of data may no longer be needed at field or programme level – and should therefore be removed from the data collection protocol. Also, the burden of data collection and data management should be critically assessed to determine whether the MEL system should be made leaner, or if resources should be added for MEL.

As part of the annual review process, the need for new evaluation studies should be determined and necessary resources mobilized, or partnerships established to conduct those studies.

The main results of the regular review should be communicated to all those who are actively involved in MEL (including to the community level), for example, through a newsletter. These results include progress in implementation and the achievements of FFS results. The feedback serves two purposes. First, it fosters programme ownership among farmers, facilitators, master trainers and government stakeholders. Second, the feedback is essential for making constructive improvements in activities.

The main results of the annual review should also be disseminated to donors, policy-makers and decision-makers. Dissemination could be effected through an annual meeting or seminar, or through preparation of an annual report. The purpose of dissemination is to account for the use of programme resources, and to advocate for further funding or policy changes.

Figure 17 shows the monitoring flowchart that was presented in the FFS Guidance Document (which illustrates the elements of organization, oversight (see section 5.1), reporting and data management (see section 5.2) of FFS programmes.

Figure 17. The monitoring flow chart

Source: Adapted from FAO. 2016. Farmer field school guidance document: Planning for quality programmes. Rome. www.fao.org/documents/card/en/c/d7d4db1f-826f-4d81-b097-44292ff7eeca/

5.3
Institutionalizing MEL

As mentioned earlier, in countries where one or more programmes or projects use the FFS approach, it may be desirable to establish one harmonized framework and system for MEL. Harmonization of MEL across programmes and projects may increase the efficiency of MEL activities and the consistency of the activities and data collection, provided that adequate attention is paid to differences between programmes. For example, two programmes may have different impact targets. It may also be useful to institutionalize the annual (or semi-annual) review meetings at national level. One advantage of harmonizing the MEL system among programmes is that it facilitates making comparisons or sharing lessons between programmes, for example in shared workshop sessions, while it also enables the exchange of experts between programmes, to provide feedback and advice.

Moreover, an effective MEL system could potentially be linked to, or adopted by, national-level frameworks for monitoring and evaluation, thereby increasing prospects for sustained implementation of quality FFS programmes (Schwandt, 2018). Key FFS actors, including farmers, farmer organizations, NGOs, extension services and government institutions

Framework for setting up a monitoring evaluation and learning system for farmer field schools at different administrative levels, need to be involved in the institutionalization process from the outset. That way, they can gain full awareness of the importance of the MEL process, as well as a common understanding of FFS objectives and methodology. Government institutions, in particular, should understand how MEL can support the quality of FFS programmes and use evidence and results to inform future activities. Stakeholders’ involvement is key to awakening their interest and ensuring their availability to support the MEL process. The MEL system could also contribute to the global reporting system on the Sustainable Development Goals, if linkages with relevant institutions are established.

Source: Cartoon by Julie Smith – https://www.rcrc-resilience-southeastasia.org/event/consultancy-to-strengthen-m-and-e-of-rri-workshop-february-2017/

5.4
Overview of a typical MEL system for FFS programmes

Table 15 gives an overview of a MEL system for an FFS programme, including the kind of information that is relevant, how it should be collected, when, at what level, and by whom. This is the type of table that should result from outlining the MEL management system and following the steps highlighted in the first part of the document.

Table 15. MEL framework with domains and results chain. Examples are given for methods, roles and resources for a hypothetical programme

Source: Authors' own elaboration

The table is based on the example presented in 2 Using the MEL framework: defining impact targets and milestone indicators. Each programme will need to build its own MEL system. The table, by serving as a model, provides a starting point for each programme to plan its MEL activities. In addition, the table provides a basis for navigating the various tools that are presented in the toolkit.

6
Setting up impact assessment for FFS programmes

The terms ‘impact assessment’ and ‘impact evaluation’ have been used interchangeably, which can be confusing. For the purpose of this document, ‘impact assessment’ signifies a methodological approach used to establish whether the observed impacts can reliably be attributed, or accredited, to the FFS activities. ‘Impact evaluation’ is used as the general term for any observations or study on impact (see example under Part II, section 3.4 Difference-in-differences design). Consequently, impact assessment is a component of impact evaluation (see Figure 1. Link between project design, MEL and impact assessment).

The results of an impact assessment can support a project in its planning, management and reporting functions. It not only demonstrates the results achieved, but also helps to produce reference points for monitoring and decision-making. In the end, impact assessment, must be able to show:

- ➔ whether or not the targeted changes have been achieved

- ➔ whether or not the observed changes were the result of project activities

This chapter will provide the reader with the background information needed to carry out simple internal impact studies and/or engage in discussions with impact evaluation experts, as well as to support the initial design and planning and provide follow-up to more complex impact assessment studies. The sections that follow are:

1. 6.1 Discovering impact assessment

2. 6.2 Capturing early results and unintended outcomes

3. 6.3 Selecting tools for impact assessment design and analysis

4. 6.4 Setting up and managing impact assessment

MEL and impact assessment are linked in that MEL data can serve to explain impact assessment results. In this context, training quality and learning achievements are monitored and the findings are assessed and analysed to identify problems that are then used to modify current activities or improve the design of subsequent programmes.

Impact may take a very long time to materialize and will depend on a combination of different factors (such as actors/stakeholders involved, enabling environment, people’s attitudes) beyond the FFS activities. Hence, impact assessment must already be considered during project formulation and rollout.

6.1
Discovering impact assessment

The main purpose of impact assessment is to produce evidence that is convincing to ‘outsiders’ who have not directly been involved in the FFS, including donors, decision-makers, policy-makers, and the academic and international community. Outside stakeholders may not be easily persuaded by the routine data collected, narratives shared, or by their personal field visits, which they may label as ‘biased’, ‘subjective’ or ‘open to interpretation’. They may demand more objective and convincing evidence of specific outcomes and impacts of interest, so that important decisions regarding policy, programmes or funding can be justified.

As well as the outsiders, the ‘insiders’ may also require impact assessment for their own purposes. Insiders, such as the farmers, facilitators, MEL teams and the project management team, who have been directly exposed to the observations and data collection at field level, may have personally seen the results of the FFS in aspects of people’s lives and in the agricultural sphere. However, this may not be adequate for them to evaluate how effective the activity was. Hence the need for impact assessment (see Table 16).

Table 16. WHO needs to know WHAT for what purpose (WHY)?

Who needs to know?

What?

Why (for what purpose)?

Governments

Whether goals are achieved

Scale of implementation

Decision-making on whether the government should buy in and support future programmes

Designing future programmes

Donors

Whether goals are achieved

Scale of implementation

Determining whether investments were worth the outlay

Universities and international communities

Whether goals are achieved

Scale of implementation

Designing future programmes

Supporting future programmes

FFS participants and communities

Monitor their progress

Appraise how the FFS was implemented and facilitated

Deciding whether participation is of benefit to them

Facilitators and programme management

Monitor their progress

Quality of implementation

Improving implementation of programmes, i.e. adaptive management

Accounting for use of resources to government stakeholders and donors

Designing future programmes

Source: Authors' own elaboration

An impact assessment should not be a stand-alone study, but the results from the impact assessment should ideally be evaluated vis-à-vis other data routinely collected through the MEL system (see Figure 4. The results chain). Monitoring data on elements such as programme design, level of local ownership and methods of facilitation are important for interpreting and explaining the results of an impact assessment. Also, evaluation data on external processes that influenced the results (such as the support received from local authorities) can be instrumental in understanding the assessment’s results. Multiple data sources are a strength to the assessment.

Impact assessment can be planned from the start of a programme, to establish a baseline before the activities, but it can also begin later during the programme cycle, by establishing baselines for new FFS groups, or by skipping the baseline altogether (see sections 4.2 Evaluation tools and 6.3 Selecting tools for impact assessment design and analysis).

Impact assessment can be planned and conducted internally, by the programmes themselves, or externally, by independent study teams. Each option has its own advantages and disadvantages (see section 6.4 Planning an impact assessment).

6.2
Capturing early results and unintended outcomes

It is important to have a clear results chain, define the impact targets upfront and design activities that will lead to the desired results. However, we must accept that surprises from unexpected outputs and outcomes will always be part of the process, which can lead to some of the most desirable impacts (such as farmers’ empowerment). As such, programmes should integrate indicators from observations and discussions with stakeholders and MEL tools that are well suited to capture unintended outcomes of activities. This includes the use of impact assessment in combination with more open ‘qualitative’ tools for MEL (see Table 16. WHO needs to know WHAT for what purpose (WHY)?).

Source: Cartoon by Julie Smith – https://unesdoc.unesco.org/ark:/48223/pf0000186231

Box 16. Unplanned impacts – Voices from the field

As part of FFS activities in Kenya, FFS farmers significantly increased their production (maize yield from 555 kg/ha in 1997 to 3 335 kg/ha today) and reduced chemical inputs, resulting in higher gross margins from agriculture. However, an unexpected yet critical impact was the emancipation of women. Mixed FFS groups promoted equal participation, breaking social customs related to gender roles. This increased the role of women in agriculture and changed perceptions of the role of women in the community.

Source: FAO. 2019. Farmers taking the lead [video]. www.fao.org/3/ca5131en/ca5131en.pdf

As discussed in section 3.1 Linking the quality of FFS activities to the results of FFS programmes, early signs of outcomes or impacts are visible during the FFS sessions and can be measured during routine FFS monitoring visits or surveys. For example, farmers often improve their analysis and decision-making skills after some FFS sessions. In Kenya (see Box 15) women’s participation in mixed FFS groups demonstrated early signs of emancipation. Such changes are promising signs of outcomes and impacts which, if monitored during the FFS season/production cycle, can provide the programme with a measure of quality of the activities and help in the preparation of impact assessment (for example, identification of indicators, selection of appropriate impact assessment methods).

Box 17. Essential elements of FFS impact assessment

- ➔ Impact assessment (as any evaluation) should ideally be planned from the beginning, before the interventions start, so that robust baseline data are collected.

- ➔ The combination of impact assessment and MEL can be used to cross-check results and to better understand those results.

- ➔ The evaluation of post-FFS results needs to be included in budgets, usually prepared at the conception stage of programmes.

- ➔ Researchers should ensure that sampling is representative of the target population and that data are collected and interpreted in an objective and scientifically sound manner.

Source: Authors' own elaboration

6.3
Selecting tools for impact assessment design and analysis

A considerable number of studies have assessed the impact of farmer field schools. These studies varied in their focus, approach, methodology and statistical robustness. This diversity produced a range of outcomes from different perspectives. However, it is difficult to compare the results between studies because these were designed to be either statistically rigorous (but with a restricted scope, focusing on a few variables) or comprehensive (looking at a broad range of impact indicators, but without solid statistics), but never both. In order to evaluate the benefits from more than one perspective, and to enhance the comprehensiveness of the overall evaluation, methods or results of different data sources should be combined.

There are several design tools available for impact assessment under programmatic conditions (see Table 17. Design tools and analytic tools for impact assessment). These tools are described in more detail in Part II Toolkit for MEL and impact assessment for FFS programmes, section 3. They address the challenges that are commonly encountered in impact assessment.

Table 17. Design tools and analytic tools for impact assessment (see Part II)

Type

Tool

Description

Design tools

Baseline survey

Establishing situation prior to activities

Before–after comparison

Comparing results from before vs after the activity

With–without comparison

Comparing results from the activity vs the control

Difference-in-differences (DD)

Combining before–after and with–without comparison

Capturing diffusion effects

Using two control groups, one nearby and one far-flung

Stepwise implementation scheme

Scaling-up locally before scaling-out, to retain control

Accounting for FFS ‘dropouts’

Removing ‘dropouts’ from the baseline sample

Analytic tools

Difference-in-differences estimates

Calculating DD estimates for testing of differences

Matching

Ensuring similar characteristics of activity vs control

Combining analytic methods

Strengthening results by using more than one method

Triangulation of data sources

Verifying data by comparison from different sources

Source: Authors' own elaboration

In addition, several analytic tools are available for use in impact assessment, some of which are presented in Table 17 (and described in Part II (‘the Toolkit’). The purpose of the design tools and analytic tools for impact assessment is to produce results of impacts that can reliably be attributed to the activities.

Due to the different interest groups of FFS impact assessment (see Table 16. WHO needs to know WHAT for what purpose (WHY)?), the use of just one method cannot adequately respond to the needs of several divergent groups, for example by exclusively applying the difference-in-differences estimates approach. For this reason, it is important to combine impact assessment with MEL. Different methodological approaches are necessary to provide timely and relevant feedback to all interest groups (such as farmers, facilitators, project team, government and donors).

A good example of a best practice for impact assessment is the work conducted by the FAO Regional IPM/Pesticide Risk Reduction Programme in Asia (Morales-Abubakar et al., 2013). The programme demonstrated how to set up a statistically rigorous study using the ‘double delta’ methodology as a tool to reduce the risk of biased and ambiguous results in key parameters. The study included a scientifically sound and qualified assessment of the impact of FFS training, which produced valid, meaningful and comparable results. An accompanying case study measured the population levels of insects as indicators of ecosystem health, as a measure of the impact of training on pesticide risk reduction. The in-depth case study provided another perspective from a broader scope of information (agroecosystem data), albeit using a less statistically rigorous method.

©Lira Chea

Box 18. Challenges in impact assessment

An FFS impact assessment seeks to establish if the observed changes at farmer and community level can be reliably attributed to the FFS activity. However, impact assessments face several challenges, such as:

- ➔ Contemporary changes: During the implementation period of the FFS programme, other factors will also influence farmers’ livelihood situation, such as other programmes on rural development or health operating within the same area. There could be changes taking place in the market environment, changes in agricultural policy, or other societal or environmental changes (such as human migration to cities, the emergence of new pests, dry spells, soil degradation). All these factors will influence the farmers’ situation, making the attribution of changes directly to the FFS a challenge.

- ➔ Anticipated planning: To assess the impact of the programme objectively, attention should be paid to ensuring that so-called ‘spillover effects’ do not affect those outside programme beneficiaries and communities, so that the effects of the programme can be ‘isolated’ for comparison with the situation in communities where the programme is not implemented (the ‘control’).

- ➔ Selection bias: FFS programmes commonly select their locations and participants by using certain criteria. These selection procedures have implications for identification of the ‘control group’ that will help to compare progress, which may not be subjected to the same selection criteria. Consequently, the activities group (project beneficiaries) may not be comparable with the control group (those who are not beneficiaries) at the outset. This leads to selection bias.

- ➔ Keeping the control group: A common challenge in programmes that seek to roll out activities gradually over an entire area is that the control groups chosen at the beginning may be covered by the programme activities in subsequent years. As a result, the control is ‘lost’ after some time, until all farmer groups have been covered by the programme. This is a challenge, particularly for studying medium- or long-term impact because, after a number of years, most of the control groups that were established at the beginning will have been covered by the activity.

- ➔ FFS ‘dropouts’: Yet another challenge in the assessment of FFS impact is that some participants may have dropped out from the FFS during implementation. If the assessment compares the situation before the FFS with the situation afterwards, there may be fewer farmers in the sample after the FFS, when some have dropped out.

Source: Authors' own elaboration

6.4
Planning an impact assessment

An impact assessment study can be very costly and the planning phase is therefore critical. Reportedly, several previous FFS projects have spent major resources on an impact assessment that did not yield valuable or meaningful results. The key to a successful impact assessment (IA) is planning and consultation. Several considerations should be kept in mind to enable an impact assessment that generates meaningful and reliable results. It is crucial to ensure that these steps in the planning stage are conducted in consultation with technical resource persons who have a good track record in study design and data analysis.

6.4.1 What are the questions to be answered?

The purpose of any study is to answer ‘burning’ questions – questions that need to be answered. If there are no burning questions, one could ask why the study is needed in the first place. Consequently, a study should take its research question(s) as its starting point, which should be formulated in a clear and succinct manner, such that the scope of the study becomes clear. The research question could, for instance, refer to an impact target in one or more domains (such as the degree of diversification of agricultural systems). The study could also refer to a certain outcome (such as collective action, savings, crop production). It is also important to know from whom the questions originate: Are the questions posed by the government, by the programme or its beneficiaries, or by the funder? These issues should be clarified at the outset of the study.

6.4.2 Is impact assessment the right approach?

In general, impact assessment is the correct approach if the objective is to convincingly attribute changes or impacts to the programme or project. This normally involves a comparison between the intervention group and the control group, or between before and after the intervention. Such systematic assessments will be much easier for quantitative data than for qualitative data. In this regard, impacts in the natural and financial domains may be simpler to measure with quantitative data than impacts in the human and social domains, which often involve qualitative data. Therefore, careful consideration of the right approach, or combination of approaches, is critical when planning the study. MEL generates qualitative data on the evaluation of outcomes and impacts, which could be used to supplement an impact assessment because the latter concentrate on quantitative data only. The combination of approaches may therefore provide the best of both worlds.

6.4.3 What are the target and eligible groups of the study?

The research question should be answered for the selected target group, which will be eligible groups or eligible individuals, according to certain criteria. If the question refers to the entire project, all farmer groups covered from the outset will be eligible for inclusion in the target group. However, the study could also specify its target group, for example, only those FFS groups that started in a certain year, only those that are located in a certain province, or only those groups or individuals who are particularly vulnerable. Hence, the study must define its target group.

6.4.4 What is the time span for answering the questions?

Should the change or impact be recorded from the beginning to the end of the project, or from the beginning to the end of an FFS season/production cycle? To clarify this important point, there should be a baseline, at the beginning of the project, or at the beginning of the FFS season/production cycle. These baselines will enable the before-after comparison of the impact assessment. It is also possible to compare the intervention group with a control group, in the absence of a baseline. In that case, there is no time span for answering the questions because the comparison could be theoretically made at any point in time. The before–after comparison and the with–without comparison each has its own shortcomings, as discussed in Part II. Design options to overcome these shortcomings are also discussed in Part II.

6.4.5 What data type and sample size will be needed?

What type of data will be required to answer the research questions? Will quantitative data suffice, or should these data be supplemented by qualitative data? It is important to think about the type of data at the design stage of the study because this will have implications for data analysis, including the technical resources needed.

The sample unit can be an FFS unit, as a group, or the individual farmer or community member. The sample size is the number of sample units that will be needed. The sample size should be chosen so that it is large enough to allow the detection of a significant difference between the comparison groups (before–after, or with–without). It is difficult to define the correct sample size because this depends on the anticipated difference in the results between the comparison groups (the ‘effect size’). Consultation with a technical resource person with expertise in study design and data analysis will be important in deciding on the sample size.

6.4.6 Which design option should be used?

Several design options for an IA are discussed in Part II. The main options for IA are the before–after comparison and the with–without comparison (intervention vs control group). The third option is the combination of the before–after and the with–without comparison, called the difference-in-differences. This third option is superior to the other two options because is accounts for sources of bias that could influence the results; it therefore produces more accurate results. It is recommended that programmes select the difference-indifferences design for their impact study.

Randomized controlled trials (RCT) are often used for research purposes, but are not usable for evaluation of an FFS. That is because in RCTs, the decision as to whether or not a group receives the intervention is determined by randomization (by chance). This is clearly not the case in FFS programmes, where the FFS is assigned to a farmer group based on certain criteria – not randomly.

6.4.7 Should the impact assessment be external or not?

The decision to conduct the IA internally within the project, or to contract an external entity to conduct the study is an important one. After the initial planning has been completed regarding the questions to be answered, the target group, time span, data types and sample size, it will become clear as to whether the task at hand can feasibly be conducted using the technical expertise within a project and its collaborators, or if the study will be technically too demanding for the project to conduct on its own. External entities, such as research groups in universities, or specialized consultants, could be selected to undertake the task of the IA, if they have a proven track record in the design and analysis of field-based or social science studies. The advantage of the external arrangement is the presumed objectivity of the external researchers. A risk of this arrangement is that the external group may not fully understand the principles or objectives of the FFS project, or may deviate from the questions or objectives of the study. A continuous engagement of the project team with the external research team through joint workshops or meetings is therefore, essential (see section 6.5).

6.4.8 What can be done to ensure that impact assessment is going as planned?

Regardless of whether the IA is conducted internally or externally, it is crucial that the project team maintain supervision of the study to ensure that the original questions are answered, that observations remain within the scope and target group of study, and that the quality of data collection is not compromised. Workshops and meetings with the research team will help to ensure that the study is going as planned (see section 6.5).

6.5
Managing an impact assessment

6.5.1 Selecting the type of impact assessment

A recent global survey identified that approximately half of the FFS impact assessment studies produced between 2005 and 2018 were internal studies – conducted by the programme or by farmers – while the other half involved external studies conducted mostly by researchers (Van den Berg et al., 2021). Table 18 summarizes the advantages and disadvantages of internal, external and combined impact assessment study types.

Table 18. Advantages and disadvantages of internal, external and combined impact assessments

Study type

Advantages

Disadvantages

Internal studies
(including those carried out by farmers, facilitators and/or project staff)

Those involved are likely to understand the methods and purposes of the FFS approach, which may benefit the scope and relevance of data collected.

Ability to link impact data to MEL data collected to data on how the activities were implemented, which assists in the interpretation of results.

Enhances the technical capacity and sense of programme ownership among those involved.

Prone to reporting bias, which occurs through selective revealing or suppression of information, because those involved may have a vested interest in a positive result of their programme

External studies
(including those contracted out to universities or consultants)

Independent, without vested interest in the outcomes of the programme, and are thus expected to be more objective.

Teams may not have full understanding of the FFS and may miss some key elements and impacts

Tend to be costlier

Combined studies
(engage internal staff as well as external researchers)

Internal staff could ensure that methods are based on a good understanding of what the FFS is trying to achieve, while external researchers ensure that sampling is representative of the target population and that data are collected and interpreted in an objective and scientifically sound manner.

Teaming up with researchers makes it more likely that the study results will be adequately documented and disseminated.

Could be the best option if it combines the advantages of internal and external studies.

Source: Authors' own elaboration

Participatory impact assessment does not mean that farmers are used to collect data for researchers. Participatory impact assessment of FFS programmes implies that farmers are actively involved in different stages of an evaluation study, including the planning and implementation of the study and the evaluation of its results. It is important to select those tools for participatory evaluation that will help farmers to collect the data they need to answer their questions.

6.5.2 Identifying internal evaluation teams or external institutions to carry out impact assessment

Internal studies may be carried out with the involvement of farmer leaders, FFS facilitators, master trainers, government field staff, the project management team, and/or the MEL unit. The focal points who are FFS facilitators or master trainers in the areas would normally lead the impact assessment exercise in their locations.

External institutions to be contracted by the FFS project are selected based on a set of criteria defined by the project. External institutions are usually selected by the MEL unit and the project team, based on:

- ➔ sufficient expertise and experience in agricultural projects and statistical techniques to be applied to the study;

- ➔ sufficient knowledge about the FFS concept, approaches and methodologies;

- ➔ ability to provide suitable references to assure the project that it has the required expertise and practical experience; and

- ➔ willingness to subscribe to the concept and facilitate the process of impact assessment as an integral management function of the project, rather than just another study to be conducted.

Annex 3 provides an example of a Terms of Reference (TOR) that can be used in a contract with an external institution commissioned to conduct the impact assessment study. The TOR covers the description of activities/services to be undertaken, outputs and duration of the study, as well as the monitoring and reporting responsibilities. The TOR may be drafted by the MEL unit or project team and revised by the selected external institution, based on the results of the planning workshop before the agreement is finalized by the project.

©FAO/Philbert Yandemye

6.5.3 Organizing workshops to plan, review, report on results

Regardless of whether an internal, external or combined impact assessment study is chosen, a series of workshops will have to be organized to plan the study, and review and report on its results.

A planning workshop should be organized with the participation of farmer representatives, government field staff, the project team and MEL unit and several resource persons, to advise on technical and logistical aspects of the impact assessment study. The role of resource persons will be crucial, especially if the project does not have sufficient technical expertise. If the project has decided that the study will be carried out by an external institution (such as a university), the external team should participate in the workshop.

Using the framework on the results of FFS (see Figure 8. The FFS Mandala: Framework on the results of the FFS, showing the results chain, with examples of impact targets in each of the four domains) as a reference, the workshop should include the content outlined in Table 19.

Table 19. Contents of the impact assessment planning workshop

Expected output

Sample questions to ask during the workshop

Setting the impact targets

What are the impacts of prime interest to donors, governments, farmers?

What is the most important research question we want to answer?

What are the targets we want to achieve?

What early signs (i.e. outputs and outcomes) may be observed during the FFS sessions?

Identification of indicators

What do we want to observe in order to verify whether, or to what extent, it is true that progress has been made?

Definition of methods for data collection

Will it be an internal, external or combined study?

Who will conduct the study, and what will be their roles?

Will it be a before–after comparison, with–without comparison, or a combination?

How will the study account for ‘dropouts’?

How can ‘contamination’ be avoided between comparison groups (e.g. with vs without the FFS)?

What should the sample size be and how should sampling be done?

Which data would provide the most reliable answer to the research question?

Definition of special studies (e.g. case studies) to support the impact assessment

Which additional data would provide better understanding of the impact results (e.g. qualitative data from interviews; M&E data collected during FFS implementation)

Preparation of a workplan for the study and setting the timelines for a baseline survey and impact assessment surveys in the short and long term

What is the timeline for expected impact (e.g. how many years after the FFS)?

Considering all the above aspects, how can we make best use of the available funds for the study?

Source: Authors' own elaboration

The timing of the planning workshop for an impact assessment could be at the beginning of the project (to establish the baseline before project activities start), or at the start of a new FFS season/production cycle in a certain area (to establish a baseline right before the FFS activities begin). See also section 6.1 Discovering impact assessment on establishing a baseline later in the project or skipping the baseline altogether.

When a participatory impact assessment study is carried out, a second planning workshop will be needed for the purpose of training farmer leaders and facilitators to collect impact data. This is to ensure that those who collect the data follow the same style of interviewing, to ensure data quality. A second workshop will be necessary:

- ➔ to train farmer leaders and facilitators in analysis of the impact data collected

- ➔ to prepare the outline for the report

In a participatory impact assessment, the role of the project team, MEL unit and resource persons will be crucial in both workshops, to advise on technical and impact assessment aspects. Similar exercises may be carried out by the external institutions, for example, as part of a contract with the FFS programme or project.

On completion of the impact assessment report, a workshop should be organized to present the findings. The project team should take the lead in organizing the workshop and inviting representatives from government, donors, international communities, FFS participants and communities and facilitators.

During implementation of the impact assessment study, data are collected by means of standardized interviews, participatory monitoring and self-evaluation methods, or experiments that can be carried out in farmers’ fields. The collected data are analysed using the selected methods (such as the ‘double delta’[6] approach). Periodic meetings may be carried out by the organizing team (such as the MEL Unit or project team or the external institution) to review and verify the collected data for accuracy and consistency.

6.5.4 Disseminating results of the impact assessment

Once the impact assessment report has been accepted by the project management and submitted to the relevant offices (such as donors), it is important to share the results to give FFS practitioners, policy-makers and researchers, locally and in other countries, access to the findings. Dissemination could be effected through meetings, online posting of documents on dedicated websites, or in the form of technical documents or journal articles.

[6] The ‘difference-in-differences’ or ‘double delta’ approach combines a before–after comparison and a with–without comparison.

PART II

Toolkit for MEL and impact assessment for FFS programmes

How to use this toolkit

The Toolkit for MEL and Impact Assessment of FFS projects and programmes will assist field implementers, the MEL unit and the project team in deciding on the selection of tools for monitoring, evaluation and impact assessment to meet their needs and requirements. They can use the toolkit for their regular tasks, such as deciding on the type of data to collect during monitoring of farmer field school (FFS) projects and how to evaluate FFS quality so that improvements can continuously be made to support the learning element within projects. The toolkit also provides basic information on impact assessment tools to guide project and programme leaders in the selection of experts who may be conducting impact assessment of FFS projects. Each tool is briefly described in Part I, while details are provided here in Part II.

STEP 1.

Become familiar with the tools listed below and presented in Part I.

STEP 2.

Review the matrix on efficiency, ease of implementation and cost of each tool for monitoring, evaluation and impact assessment (Table 1).

STEP 3.

Study the information provided on those tools that meet the requirements of, and will be suitable to your FFS programme. Part II provides guidance for each tool on

(i) what it is;

(ii) why use it; and

(ii) how to use it.

Examples, with photographs or illustrations, are provided where available.

Table 1. Overview of tools for monitoring, evaluation and impact assessment, indicating the degree of efficiency, ease of implementation and cost

Legend:

L – low M – medium H – high

Notes: Efficiency – the results that can be obtained by using the tool with the given resources and time frame.

Ease of implementation – the level of difficulty involved in using a tool from start to finish, including development of the content, identifying qualified staff, material preparation, training of data collectors/enumerators, report preparation, etc.

Cost – cost of a using a tool, which may include professional fees, salaries, costs of training, transport, needs assessment, content development.

Legend:

L – low M – medium H – high

Notes: Efficiency – the results that can be obtained by using the tool with the given resources and time frame.

Ease of implementation – the level of difficulty involved in using a tool from start to finish, including development of the content, identifying qualified staff, material preparation, training of data collectors/enumerators, report preparation, etc.

Cost – cost of a using a tool, which may include professional fees, salaries, costs of training, transport, needs assessment, content development.

Source: Authors' own elaboration

Poultry FFS participant presenting the final results of their study

©FAO/Lebanon

1

Tools for monitoring

Several tools for monitoring are proposed, as presented in Part I (see section 4 Tools for monitoring and evaluation). The timing for implementation, together with a description and the purpose of each monitoring tool, is summarized in Table 2.

Table 2. Main monitoring tools for collecting data of FFS activities

* When available from many FFS locations, these results give an impression of the potential to improve farmer practices and highlight locations where further attention is needed

Source: Authors’ own elaboration

1.1
AESA records

what is it? Agroecosystem analysis (AESA) is a qualitative instrument that can be used to guide farmers in developing skills and knowledge about ecosystems to develop critical thinking and make better decisions. In the FFS, farmers work in groups to observe and analyse the field situation regularly (such as crops, insects, water, weather), on the basis of which informed decisions are made. The AESA process contributes to developing skills of observation, critical thinking, analysis and informed decision-making and strengthens farmer confidence. AESA activities can also give the MEL team an indication about the group development and organizational processes. The AESA process can be adapted to other types of field schools, e.g. pastoralist ecosystem analysis (PESA) in pastoralist field schools, human ecosystem analysis (HESA) in farmer life schools.

why use it? The weekly farmer-drawn representations and records of agroecosystem analysis (or system analysis, where the FFS tackles a system other than agriculture) are an important documented outcome of the FFS learning activities that can be used for monitoring purposes. Records of observations are compared at different times in the season/production cycle to assess the progress of farmers’ groups or learning gaps to identify areas for improvement in the FFS programme. AESA is a qualitative instrument that can complement formal evaluation techniques, including assessment of the development process of farmers’ groups.

how to use it? Decision-making in an FFS requires an analysis of interactions between components of the ecosystem. Farmers take data from field observations in small groups (to ensure that everyone is involved), then all the groups reach a consensus to make an informed management decision. The weekly farmer-drawn sheets and records of the agroecosystem situation provide insight into the comprehensive nature of drawings and supporting observational data, and the quality of analysis, critical thinking and decision-making performed by the FFS participants.

AESA data collection

©FAO/Philbert Yandemye

AESA presentation

©William Settle

In 2021, Pakistan developed a variation of the tool to carry out agroecosystem analysis in cotton-wheat, rice-wheat, sugarcane intercropped systems. The tool uses a pre-designed AESA board and pre-printed colored circles to replace numeric and textual data that remains a challenge for farmers – especially women – who cannot easily write and read data for use in analysis and decision making. The colored stickers reflect the situation of the various components of the agroecosystem as observed by farmers but actual specimen is also collected and attached to or drawn on the board to support field observations. The tool ensures that all participants regardless of FFS members’ age, education level, gender equality, wealth position and social activism are able to participate in the activity.

Livestock FFS data collection

©FAO Solomon

It is even more effective to monitor the actual AESA activities for quality indicators, such as the presence of communal plots with two or more treatments; records taken at fixed (usually weekly) intervals; observations forming the basis of decisions made; comparisons made between ‘farmer practice’ and ‘improved’ plots; comparison made with the AESA of the previous week; ‘what if’ scenarios being discussed; and whether all participants are actively engaged in discussions and working together.

A representative sample of AESA drawings, supplemented by observations of the AESA process, will allow facilitators, their supervisors and the MEL team to monitor the quality of learning and skills development, and to flag issues or elements that need addressing.

> Further reading: Jam, M.K. 2021. Understanding the farmer field school agro-ecosystem analysis board. Pakistan. FAO. https://doi.org/10.4060/cb6742en

AESA observations

©Manuela Allara

1.2
Cost–benefit records (at farm level)

what is it? Cost–benefit analysis, sometimes also called benefit–cost analysis, is a quantitative tool that uses a systematic approach to estimate the strengths and weaknesses of new knowledge, alternative practices and skills to determine the options that provide the best approach to achieving benefits, while reducing costs. In an FFS, farmers should learn to keep systematic records of inputs and outputs of field study plots/comparative experiments, to track their costs and benefits.

why use it? Throughout the season/production cycle, the quantities and costs of inputs in the field study plots/comparative experiments are recorded and outputs are recorded at the end of the season/production cycle. Cost–benefit records can be used as a monitoring tool during the season/production cycle to obtain an indication of the development of financial skills. When calculated at the end of the season/production cycle, it can show the benefits derived from knowledge, practices and skills introduced by the training programme.

how to use it? Records of costs and benefits are used to calculate profits in each field study plot/treatment obtained from the application of new knowledge, practices and skills. The records can also be used to calculate break-even prices, to help farmers negotiate sale of their produce. The presence and quality of these records is indicative of the development of financial skills. In addition, the financial outcomes of the field study plots/comparative experiments can be used in meta-analysis across all FFS units, to provide insight into prospects for raising the profitability of farming.

example 1

Figure 1. Economic benefits of field study plots

Source: Center for Initiatives on Community Empowerment and Rural Development (ICERD). 2021. Stories from the field. Hanoi. https://icerd.vn/stories-from-the-field

Participant in FFS on rice-fish production

©Dung Tien Ngo

example 2

FFS logbook, India at https://www.fao.org/3/cc2712en/cc2712en.pdf

1.3
FFS start-up reports

what is it? FFS start-up reports are a qualitative tool to compile information about local production practices, field problems, farmers’ specific needs, and requests to be considered in designing the training programme. Using various methods to generate information, the main purpose of start-up reports is to give participant farmers a sense of ownership over the FFS process and to ensure that the programme is relevant to the farmers. Used as a monitoring tool, the reports can be reviewed vis-à-vis what actually happens in FFS sessions, to assess the quality of the programme.

why use it? At the beginning of each FFS unit, farmers are consulted about their production practices, field problems and their specific needs and requests. Information obtained from the consultation with farmers and local leaders – processed and summarized as start-up reports – is subsequently used by facilitators and their supervisors to ensure that the curriculum and methods of the FFS are adequately adapted to the local situation and needs.

how to use it? Over several meetings before the FFS sessions start, the local facilitator and members of the community – such as farmers, local leaders and resource persons – carry out several activities to collect information listed as examples below. The exact sequence, intensity and type of method should be determined in a flexible manner, based on the experience and capacity of facilitators, the field situation and results of the activities. The activities – intended to collect information on farming practices, field problems and their specific needs and requests – may include:

1. transect walk and or participatory mapping and field observations to become acquainted with the physical and functional structure of the village;

2. informal interviews with farmers (men and women), traders, consumers and community leaders to collect information on perceptions of community members with regards to production and/or post-production issues;

3. focus group discussion using various participatory rapid appraisal (PRA) methods to discuss problems encountered by the community related to production/post-production issues; and

4. community feedback meeting to summarize information from the previous meetings and plan the FFS curriculum with farmers.

Results from whatever methods were used are analysed and summarized in FFS start-up reports. Used in combination with participatory evaluation tools, FFS start-up reports are useful to review whether FFS methods were sufficiently adapted to local needs. The availability and content of start-up reports are important quality indicators, ensuring local relevance and fostering local ownership of the FFS programme.

Participatory resource mapping, Bangladesh

©Hein Bijlmakers

> Further reading
Cambodia National IPM Programme. 2009. Participatory Rapid Appraisal of Mungbean Production and Protection Practices in Cambodia. Unpublished. www.fao.org/3/ca8287en/ca8287en.pdf

FAO India. 2019. FFS Logbook. Unpublished. https://docs.google.com/document/d/1hCWMX-4XLXOPn4LDPL1clMarTM0qdft6/edit

Outcomes of participatory mapping by FFS group members in Peru

©Agrorural_MINAGRI

example

Participatory Rapid Appraisal of Mungbean Production and Protection Practices in Cambodia (Table of content of report)

Background

Survey objectives

Methods

Focus Group Discussions

Individual interviews

Informal interviews

Field observation and transect walk

Inventory of pesticide shops

Major findings

Mungbean cultivation, yields and profits

Fertilizer use

Pest and disease problems and solutions

Pesticide use

Gender division of labor in mungbean production

Pesticides and applicator’s health

Inventory of pesticide shops

Conclusions

Recommendations

Source: Cambodia National IPM Programme. 2009. Participatory Rapid Appraisal of Mungbean Production and Protection Practices in Cambodia. Unpublished. www.fao.org/3/ca8287en/ca8287en.pdf

1.4
Field day reports (with photos)

what is it? The field day is the culmination activity of a farmer field school. The purpose of the field day is to disseminate the outcomes of the FFS and promote the sharing of experiences among farmer field schools, non-FFS farmers and other stakeholders (such as extension staff). Field days can help to generate demand, as well as local government support for farmer field schools within the village. FFS-trained farmers organize exhibits (such as cost–benefit analysis of field studies, compilation of agroecosystem analysis, insect zoo experiments, photographs of FFS activities) to explain the FFS activities and results (such as field studies) to various stakeholders. The exhibits will vary depending on each locality and what the FFS farmers did during the season/production cycle. Field day reports submitted by facilitators involve documentation of the activities that took place during the event. The report should include the proceedings and highlights of discussions that took place, summaries of speeches, farmers’ post-FFS or follow-up plans and photographs of the activities. The list of attendees is also included in the report.

why use it? The FFS is normally completed by holding a field day for neighbouring farmers, local authorities and other stakeholders. Field day reports (with photographs) are prepared by facilitators as part of the FFS documentation. The documentation, used as a qualitative tool, can be used when the FFS has been completed to obtain an indication of the quality of the training programme and the facilitators, as well as the benefits that a farmer field school brings to farmers. It can be used to improve the design of future FFS programmes and to lobby for support and buy-in from local governments.

how to use it? Field day reports provide indicators of quality of the FFS and facilitators, drawn from the experiences and results that the farmers share and the confidence that they show as they deliver presentations (captured in the photographs). Field day reports also provide an indicator of the extent of dissemination of information about the training programme within the village. Moreover, established linkages between the FFS group and other stakeholders attending the field day can become evident from their participation and feedback in the activity. Photographs of the field day activities can be analysed vis-à-vis indicators in other monitoring tools (such as the quality matrix), to provide evidence of the quality of the FFS.

Presenting experiment at field day in Kenya

©FAO/Ami Vitale

Bangladeshi farmers presenting folk media

©Alma Linda Morales-Abubakar

Bangladeshi farmers performing

©Alma Linda Morales-Abubakar

example

Box 1. Field day report (sample information)

- ➜Name of reporting officer

- ➜Date, location

- ➜Description of activities

1. Exhibits of insect zoos, pot studies, various species of aquatic organisms, livestock organic feed formulation

2. Presentation of field study results

3. Speeches from guests

4. Speeches from FFS farmers

5. Folk media presentations

6. Photo exhibits

- ➜Agenda (attach)

- ➜Number of participants (attach list)

- ➜List of special guests

- ➜Comments on media coverage (if applicable)

- ➜Reactions from neighbouring/non-FFS farmers and guests

Source: FAO-EU Cotton IPM Programme, 2002. Unpublished.

FFS field day in Puno, Peru

©Claudio Guzmín

1.5
Participant lists

what is it? Participant lists usually contain the name, gender and age of FFS participants. Additional information may be collected depending on the need of the FFS programme, such as participants’ addresses. The list is used by facilitators to keep a record of the actual attendance of each participant in routine FFS sessions or when home visits need to be made for other FFS concerns.

why use it? A participant list is a routine monitoring tool used in farmer field schools. It is compiled during each FFS session to collect information on regular attendance. At the end of the FFS programme, a summary of attendance (for example, per person, percentage of women) of all the sessions can be prepared for follow-up or adjustments to the programme.

how to use it? The monitoring tool can be used to track the actual attendance of each participant and, if needed, to follow up on the person/s to check for problems and encourage more regular attendance. Summarized attendance records are used for monitoring the attendance rate (average number of farmers per FFS session), and percentage of women, to flag FFS units with low attendance or a low proportion of women participation, for feedback and action.

Besides the attendance rate, the consistency of attendance (number of sessions attended per participant) is a useful indicator of quality, because occasional attendance, or attendance by a representative, is expected to weaken the educational outcome of the FFS. Hence, participant lists should also be used to indicate whether a participant attended in person, or whether he/she was represented by someone else. Data on consistency of attendance could be linked to the pre-post test results per individual at the level of FFS and programme to provide insight into the importance of consistent attendance.

FFS participants learning about animal feeding and how to use fodder choppers as part of field school on animal fattening and milk processing in Alzzuhra district, Yemen.

©FAO Yemen

example

Table 3. Attendance list of participants and facilitators

Source: FAO Asia Regional IPM/Pesticide Risk Reduction Programme. 2018. Unpublished.

Table 4. Summary of attendance: Number of farmers per session

Source: FAO Asia Regional IPM/Pesticide Risk Reduction Programme. 2018. Unpublished.

1.6
Participatory evaluation of FFS session

what is it? Participatory evaluation is a crucial activity to give farmers a voice in evaluating how the FFS was implemented, whether it met expectations, whether the curriculum was considered relevant, and whether FFS methods were adequately adapted to local needs and demands.

why use it? Participatory evaluation, carried out at every session or at the end of the programme, allows farmers to give an appraisal of the FFS – including the quality of facilitation, the process and relevance.

how to use it? Participatory evaluation can be carried out through open discussion in the FFS, using an itemized response technique, or through a short questionnaire conducted at the end of the FFS. The results of the participatory evaluation should be taken up by the facilitators and supervisors to make improvements in how the FFS programme is implemented. A sample of the results of participatory evaluation should inform the MEL team and programme management about the farmers’ perceptions of the FFS, to suggest the need for improvement. Some programmes incorporate participatory evaluation in every FFS session, to give farmer participants the opportunity to make an appraisal of the facilitator and the curriculum. Such a feedback mechanism could serve as an incentive for facilitators to perform optimally in FFS sessions.

example

Itemized response technique

Purpose:

- ➜Evaluate the FFS session

- ➜Identify and reinforce good things that happened

- ➜Draw suggestions and recommendations for the improvement of future sessions

Materials: Blackboard and chalk or newsprint and marking pens

Procedure: Divide the blackboard or newsprint into two columns. On the left column, write the heading: What went well. On the right column, write the heading: Needs improvement.

Ask farmers to think about the day’s activities. Then ask them what they think went well during the day – the good things that happened during the session during that day. Write down all the responses. There are no right or wrong answers in this activity. When all the responses have been gathered, ask the farmers to focus on the things that need improvement, or to say what can be done to improve the next session. Again, write down all the responses.

> Further reading: Association of German Development NGOs (VENRO)/NGO-IDEAs, 2011. NGO-IDEAs Impact Toolbox: Participatory monitoring of outcome and impact. Bonn. (www.ngo-ideas.net/mediaCache/impact_toolbox/NGO_Ideas_Toolbox_v05.pdf)

1.7
Participatory performance tracker

what is it? The Participatory Performance Tracker (PPT) is a self-monitoring tool developed and used by the NGO CARE to measure the adoption of practices at individual and group level (CARE, 2015). The tool can be tailored to measure progress on the application of any practice or behaviour, in response to the specific focus of FFS programmes. The PPT consists of two parts: the individual-level sheet and the group-level sheet. The individual-level sheet documents the performance of individual group members in relation to the individual performance areas laid out by programme staff and group leaders. The group-level sheet tracks the performance of the group, including group leadership, gender equity governance and data management.

why use it? The PPT is used both as a management tool and an outcome monitoring tool. Adapted for FFS programmes, it can be used to evaluate group dynamics and performance. Farmers determine the schedule of meetings for the exercise. Regular meetings are organized by facilitators to conduct dialogue, focusing on progress in the use of key behaviours and practices at individual and group levels.

Source: Adapted from CARE. 2015. CARE’s Participatory Performance Tracking Tool. A step by step guide for use.

how to use it? When adapted for use in FFS programmes, the individual tracking sheet could be used for self-reporting of more ‘generic’ behaviours or practices, such as making regular field observations, keeping records of inputs and outputs, and active use of scenarios in decision-making. This is done by scoring (from a list of five key practices) the key practices that he or she has applied.

The group-level sheet tracks the performance of the group, including group leadership, gender equity, governance and data management. The combination of the individual self-reporting and the reporting by groups provides insight into the progress at each level. The data are for use by farmers themselves, but can also be collected and used by programme management for monitoring purposes. This data can be aggregated at the district, regional, national or global level to analyse how groups progress over time. Data may also be disaggregated by the year that groups began participating in the programme, to determine success at different time points. Cohorts can be compared in order to learn what is working in the field and where there may be gaps in programme implementation.

> Further reading: CARE. 2015. CARE’s Participatory Performance Tracking Tool. A step by step guide for use. www.care.org/news-and-stories/resources/ppt-step-by-step-guide/

©J.W. Ketelaar

1.8
Ballot box: pre-/post-test results

what is it? The ballot box pre-test is a field-based diagnostic tool to identify learning needs to be addressed when finalizing the design of the FFS training curriculum, as well as to reinforce those needs in implementation of the FFS sessions. The post-test gives an indication of the learning that has taken place at the end of the course.

why use it? The ballot box pre-test is carried out at the start of the FFS programme, and the post-test is conducted at the end of the training course. Aside from using the test as a diagnostic tool and to gauge improvement in knowledge, the ballot box pre-/post-test results can also be used by facilitators and their supervisors as a method of evaluating the quality of the FFS, while paying particular attention to low test scores and their possible reasons. The information can be used to improve the curriculum and design of future FFS programmes.

how to use it? A matrix for ballot box questions is prepared as a guide for formulating the ballot box test, to ensure that the facilitator evaluates all the possible curriculum content areas that should be assessed. The test questions – related to the core objectives of the course – are formulated in such a way that these: are field-based and site-specific; are focused on functions, not names; and test field skills and field knowledge. In most cases, the total number of questions asked is 20. This may, however, be changed, depending on the need and the field situation.

Ballot box test

©Agrorural_MINAGRI

The multiple-choice test question is written on a thick paper board, mounted on a stick and placed like a sign in the field next to a real condition or object that is the focus of the questioning. The sign is connected by a piece of string to the plant part that is damaged, or the object being asked about. Each farmer is assigned a number and given several small pieces of paper (ballots), with the same number specific to him or her. On each test question, envelopes or small containers marked A to D are made available, where the farmers can drop in their ‘ballots’. If the correct answer is A, then the person puts one piece of his/her number in the envelope/container. About 20 questions should be prepared for the test. After everyone has taken the test, the facilitator should walk with the group to each question, determine the correct answer, and discuss why.

Ballot box test

©Agrorural_MINAGRI

When used as pre- and post-tests, the tests must ‘talk to each other’, that is, both must test the same level of skill and/or knowledge. Care must be taken that the focus of the exercise is on content and not on the scores of participants, ensuring that the farmers are aware that the exercise is intended to improve the FFS curriculum and activities. Pre-/post-test results produce simple scores that can be collected at programme level. However, these test scores are mostly a reflection of knowledge level (or sometimes skills level), and do not capture the full educational outcomes of the FFS.

The Matrix for ballot box questions is intended as a guide for formulating the ballot box test. Using the matrix will ensure that the facilitator evaluates all the possible areas that should be assessed. Questions should be distributed among the various areas and content. Care must be taken that the questions are formulated in such a way that participants are not merely identifying specimens, but are analysing functions. In most cases, the total number of questions asked is 20. This may, however, be changed, depending on the need and the field situation.

example

Table 5. Matrix for ballot box test questions and sample questions

Source: FAO Programme for Community IPM in Asia. 1998. Unpublished.

Table 6. Results of ballot box test (pre- and post-ballot box test)

Source: FAO-EU Cotton IPM Programme. 2002. Unpublished.

Summary of pre-/post-ballot box test (maximum score: 100)

Pre-FFS scores:

lowest:

mean:

highest:

Post-FFS scores:

lowest:

mean:

highest:

Source: FAO-EU Cotton IPM Programme. 2002. Unpublished.

Table 7. Analysis of ballot box test results

Source: FAO-EU Cotton IPM Programme. 2002. Unpublished.

1.9
Reports of facilitator meetings

what is it? Reports of facilitator meetings provide an overview of progress in implementation, problems and emerging issues in FFS programmes, as well as possible mitigation measures identified. The reports are a major monitoring tool for the MEL team.

why use it? During the FFS season/production cycle, it is important that regular coordination meetings are held for facilitators together with the supervisors, to discuss implementation problems and to share lessons learned/experiences on how to improve field school quality. The documentation of these facilitator meetings can be used immediately to take action to address the problems and issues that have been reported.

how to use it? Local problems flagged in the reports of facilitator meetings could be cross-checked with other monitoring tools – such as FFS start-up reports and AESA records – as a basis for follow-up visits to the sampled FFS. The meeting reports can also be the basis for identifying topics on which to provide coaching to facilitators and/or other support needed to improve programme implementation (such as coordination with local leaders). In addition, the reports can be used to identify topics for possible inclusion in refresher courses for facilitators during the season/production cycle, or before the next FFS courses are organized.

example

Report on facilitator meetings

Prior to the meeting, the project team consolidates findings from monitoring visits to determine the agenda/activities to include in the schedule. If needed, resource persons are invited to provide inputs during the meeting.

Reporting officers: ________________________

Location and date/s: _____________________

Programme entity/project: ________________

Date of report: ____________________________

1. Description of detailed activities (based on agenda/programme) and major findings, highlights or issues raised

- ➜E.g. Progress report on FFS and post-FFS activities by trainers

- ➜E.g. Problems encountered in FFS/post-FFS implementation and how these were resolved

2. Inputs from resource persons (i.e. process-related or technical topics identified based on monitoring visits such as facilitation skills, biological control agents, immunization of livestock, organic feed for fish)

3. Discussions on workplans for the next period

- ➜E.g. Farmer Field Day/s

- ➜E.g. Farmers’ congresses

4. Administrative matters relative to FFS implementation

5. Action/follow-up needed from project team

Attachments:

- ➜Detailed schedule of activities

- ➜List of participants

Source: FAO Asia Regional IPM/Pesticide Risk Reduction Programme. 2018. Unpublished.

1.10
Reports of monitoring visits

what is it? Monitoring visits of FFS activities by supervisors (such as master trainers) or members of the MEL team have the dual function of obtaining monitoring data and providing mentoring support. Reports of monitoring visits are invaluable sources of data on FFS implementation (for example, highlighting problems such as low attendance of farmers) and inform about the type of support and corrective actions taken – including providing moral support to facilitators or sharing of technical information about project-related topics, based on the request of field staff and/or field problems.

why use it? Reports of monitoring visits (prepared by master trainers or members of the MEL unit) should be submitted no later than one week after the visits were carried out, to enable supervisors and the project team to provide timely action to address problems and issues reported. The reports can also be reviewed after corrective actions have been taken, to check for changes and improvements in implementation. At the end of the FFS programme, reports of monitoring visits may be analysed for any indications of early impacts (outcomes) that will be useful for impact assessment of the programme.

how to use it? Supervisors (such as master trainers) or members of the MEL team are responsible for writing up summary reports or highlights of visits to the FFS. Reports should be submitted one week after the monitoring visit, so that action required to address problems and issues can be taken in a timely manner. It is advised that problems requiring immediate action be directly reported to the project team. However, problems that may be discussed and solved at field level should be addressed immediately with concerned FFS facilitators. In addition, the experience should be noted, for the sharing of lessons learned with other facilitators during regular facilitator meetings (see Table 8) or during refresher training courses. Likewise, the monitoring visit reports may also contain information on early impacts (outcomes) that will be important inputs for the impact assessment of the programme.

The recent integration of information, communication technology (ICT) to strengthen the knowledge and skills of facilitators necessary for improving FFS programmes has provided the opportunity to use open data source[7] tools to improve FFS monitoring and evaluation. FAO Pakistan used the Open Data Kit (ODK)[8] to monitor the facilitation of FFS sessions in remote locations among widely spread out FFS. The facilitators use mobile phones to collect information during field visits and upload these into the system. For every session, facilitators take up to 3 pictures including a group session, field session and Agro Ecosystem Analysis (AESA) presentation which is sent to the system in real time as evidence of their presence in the field (see Example 3). Supervisors, managers and master facilitators remotely monitor the facilitator’s work in the field through activity reporting formats.

example 1

Table 8. Cotton IPM farmer field school monitoring report

Source: FAO-EU Cotton IPM Programme. 2002. Unpublished.

example 2

FFS Session Checklist, Farm Forestry Field School l

Source: Adapted from FAO, JICA and KFS. 2011. Farmer field school Implementation Guide: Farm Forestry and Livelihood Development. Rome. knowledgecentre.resilientfoodsystems.co/assets/resources/pdf/i2561e.pdf

example 3

Use of open data source software and tools for improved monitoring and evaluation

Source: Adapted from FAO Pakistan. 2019. Unpublished.

> Further reading: FAO (forthcoming). E-empowerment tips for facilitators. Information and communication technologies for farmer field schools. Pakistan.

1.11
Results of study plots/comparative experiments

what is it? The typical FFS has communal field study plots/comparative experiments with the ‘farmer practice’ and ‘improved’ treatments, and possibly one or more additional treatment plots (in the case of livestock FFS, the unit of analysis is not the field plot, but an animal, pastureland or forage field). The purpose of these experimental plots/treatments is to develop skills of observation, analysis and decision-making on improved practices and, in addition, to make a comparison with the farmer practice. After harvest, yield samples are taken from each plot, and a comparison is made. Depending on the nature of the FFS (e.g. Farm Forestry and Livelihood Development)[9], corresponding templates may be developed to summarize the results of activities (see Example 2).

why use it? At the end of the training course, and assuming that the study plots/comparative experiments were well designed and managed, the results of study plots/experiments can be used by facilitators and their supervisors as an indicator of performance of the individual FFS. When combined in a database, the results of many studies can be used at the completion of an FFS programme to analyse differences in results between different locations, for the purpose of improving future programmes.

how to use it? The yields of field study plots/comparative experiments are first and foremost used by participant farmers to evaluate the results of their production cycle or season-long decision-making vis-à-vis the conventional farmer practice. The results can also be used by facilitators and their supervisors as an indicator of performance of the FFS.

The field study plots/comparative experiments are non-replicated studies, with limited scientific value. However, when the results of many studies are combined in a database, this can provide a programme with useful insights into the prospects for improving farming practices and regional differences in results. Such ‘meta-analysis’ of field study plots/experiments can inform the programme about possible enhancements or regional adaptations in the activity methods.

example 1

Table 9. Data collection on components of yield and productivity

Notes: - In each plot, 5 sampling points will be identified and data collected from 1 hill/sampling point

- - The weather and climate for the whole season/production cycle will be noted

- - H: Hill

- - Ave.: Average

Table 10. Summary of data on averages of components of yield and productivity from treatments

Treatment

Treatment 1 (Average) Farmers’ (conventional) practice

Treatment 2 (Average) Improved practice

Parameter

N^0 of panicles/hill

N^0 of panicles /m^2

N^0 of grains/panicle

N^0 of filled grains/panicle

Weight per 1 000 grains

Yield (tonnes/ha) at 14% moisture

Table 11. Recording economic parameters

Source: FAO Programme for Community IPM in Asia. 2002. Unpublished.

example 2

Result of Cost Benefit Analysis, Farm Forestry Field School

Calculation of total expenditure

Plot 1

Plot 2

Plot 3

Seeds

Ksh.

Ksh.

Ksh.

Manure

Ksh.

Ksh.

Ksh.

Fertiliser

Ksh.

Ksh.

Ksh.

Other farm input

Ksh.

Ksh.

Ksh.

Total

Ksh.

Ksh.

Ksh.

Cost Benefit Analysis

Plot 1

Plot 2

Plot 3

Yield

Kg

Kg

Kg

Price* per kg

Ksh.

Ksh.

Ksh.

Total Value

Ksh.

Ksh.

Ksh.

Total Expenditure

Ksh.

Ksh.

Ksh.

Profit

Ksh.

Ksh.

Ksh.

Quantification

Plot 1

Plot 2

Plot 3

Quantified yield per acre

Kg

Kg

Kg

Quantified product value per acre (1)

Ksh.

Ksh.

Ksh.

Quantified cost/expenditure per acre (2)

Ksh.

Ksh.

Ksh.

Quantified profit per acre (1) - (2)

Ksh.

Ksh.

Ksh.

Result of Discussion/Conclusions/Recommendations:

Source: Adapted from FAO, JICA and KFS. 2011. Farmer field school Implementation Guide: Farm Forestry and Livelihood Development. Rome. knowledgecentre.resilientfoodsystems.co/assets/resources/pdf/i2561e.pdf

example 3

FFS group final report template, Revitalizing agricultural clusters and Ulimi wa Mdandanda through farmer field schools in Malawi at https://www.fao.org/3/cc2721en/cc2721en.pdf

1.12
Quality matrix

what is it? The quality matrix is a table that describes the activities, critical points for quality of each activity, the objectives and aim of each activity, and the indicators for quality (see sample matrix in Table 12). Monitoring teams of the MEL unit can adapt the quality matrix for use during their field visits to note their observations on the FFS (some including the preparatory and post-FFS stages). In some FFS programmes, scores have been assigned to the indicators, such that they become quantifiable quality indicators. This makes it easier for monitoring teams or the MEL unit to summarize their monitoring visit reports.

why use it? There are standards for implementation of farmer field schools. These ‘non-negotiables’ establish norms for the quality of the FFS. While there is plenty of room for variation in the content (such as sectors and topics), depending on the situation, location-specific conditions and problems, the resulting process of any FFS must be learner-centred and participatory, and rely on an experiential learning approach. The quality matrix describes actions and behaviours that an observer should be able to see when the FFS is implemented. Indicators based on standards of FFS (quality process) are used to note observations for the purpose of helping facilitators to improve the FFS and of helping farmers to have optimal conditions for learning in the FFS. Quality matrices have been designed for use at the preparatory, implementation and post-FFS stages, depending on the FFS programme.

how to use it? The quality matrix can be adapted for use as a checklist to note observations during field visits and as a tool for reflection for the monitoring officer. Reflections on the answers should be entered into the monitoring officer’s monitoring form (see example in Table 13. Sample matrix for monitoring FFS quality). As part of the monitoring visit, the monitoring officer should ask to discuss with farmers whether the FFS has met their expectations (as defined by the farmer groups at the beginning of the season/production cycle), and if they have had the chance to express their thoughts and ideas on how to strengthen the project. The information that emerges from this discussion should also be written in the monitoring officer’s diary for discussions with the MEL unit and the facilitators during regular meetings.

example

Table 12. FFS activity quality matrix for IPM programmes

Activity

Critical points for quality

Notes

Indicators of quality

What is this?

Dialogue to focus attention on functions in ecosystem, not names

Question process

Must be open ended; lead to answer; help learners discover answer; force critical analysis.

No direct answers given, leading questions asked

Function-related questions asked

Result

Learner discovers answer to his or her own question without being given the answer.

Participants able to state or define functional relationships in the agroecosystem

Agroecosystem analysis

Primary activity for developing good IPM habits:

weekly observations

analysis

decision-making

Observation and drawing of agroecosystem

Participants need to be clear about process of observation and its purpose or objective.

Participants in the field observing, taking notes, collecting specimens.

Purpose of drawing is to summarize observations and be the focus of analysis.

Before activity, participants told goal of activity; and process to be followed in the activity

Participants all in the field

Process of observation includes whole plant

Observations written down

Specimens collected

Drawings summarize observations

Farmers as experts

Analysis

Problems posed, questions asked, and scenarios presented to participants. The purpose is to discuss field conditions in depth and solve ‘what if’ scenarios. The objective is to improve decision-making skills based on ecosystem analysis. The second objective is to improve observation and analytical skills. Trainer must help to achieve objectives by asking the group to analyse and discuss the agroecosystem process, issues and decisions.

Leader poses problems and asks question appropriate to analysis

Groups discuss field conditions and agroecosystem relationships

‘What if’ scenarios discussed

Previous week’s agroecosystems drawings used for comparisons

Field management decisions made and critically examined before being accepted

Other factors in addition to economic thresholds are considered (farmers to develop thresholds based on experience)

Participants active and working together in small groups

Leader, using leading questions, helps participants to analyse activity and their learning

Results

Participants improve observation, analytical and problem-solving skills.

Participants’ knowledge of the agroecosystem is increased.

Participants can state/define differences between pests and natural enemies and their roles

Decisions based on levels of insect populations and analysis of their relationships in the field ecology (including stage of growth of plants)

Special topics

focus on particular aspects of IPM (biology, ecology, economics, rats, etc.)

Statement of goal

Participants must know purpose of activity and what they will learn.

Before activity begins participants told goal and process of activity

Process

Participants clear about what they must do and why. All materials at hand. Process unimpeded by logistics.

All participants active and involved in the activity

No small group dominated by one person to the point that others are totally excluded

Results

Participants achieve the objective. Activity is analysed by participants, with leaders asking open questions so that all know what happened and why. The purpose of special topics is to provide participants with opportunity to learn about issues important to IPM.

Participants present results of their work during the activity, summarizing what has happened and why

Participants can state what they have learned because of activity

Leader asks open-ended questions to help participants examine steps in process of activity and apply learning to ‘real life’

Group dynamics

focused on enhancing teamwork and problem-solving skills

Process

Participants informed about objectives and process before activity begins. Materials for activity, if needed, are on hand before activity begins. Time allowed for activity is sufficient to achieve objective. Logistical issues do not disturb process.

Before activity begins participants told goal and process of activity

All participants involved/active, no single individual dominating the activity

Synthesis

Leader takes time to review objective of activity. Leads discussion concerning what happened during the activity. Draws out important points experienced during the activity. Helps participants learn from their experience in the activity.

Leader

- a) reviews goal and process of activity;

- b) helps participants identify key learning points based on activity;

- c) asks questions that help participants learn from the experience

Results

Participants achieve greater understanding of teamwork or problem-solving issues.

Participants can state what they have learned and its application to real life

Ballot box

evaluation process used as a diagnostic pre- and post-test to determine IPM field skills

Preparation

Questions are field-based and site-specific. Focused on functions, not names. Tests

field skills and field knowledge. When used as a pre- and post-test, the tests must ‘talk to each other’ – both test same level of skill and/or knowledge.

Testing of field-based knowledge and skills

Latin names are not used

Results

Tests results used as a tool that: evaluates learning, learning needs, and reinforces learning.

Learner uses ballot box to reinforce learning; focus is on content, not on scores

Source: Pontius, J.C., Dilts, D.R. & Bartlett A. 2002. Ten years of IPM training in Asia: From farmer field school to community IPM. Bangkok, FAO. www.fao.org/docrep/005/ac834e/ac834e00.htm

For each matrix, the left-hand column indicates items for observation. The right-hand column should be filled in by the observer. Yes and No answers are discouraged. More specific comments will be appreciated. It may be best to ask either the farmers or the trainers for information on items that cannot readily be observed. Guide questions have been indicated to help the monitoring team. For every FFS or follow-up activity observed, at least three participants and one trainer should be asked for comments.

Table 13. Sample matrix for monitoring FFS quality

ITEMS FOR OBSERVATION

1. Attendance: Total number; regular attendance (at least 80% of full attendance in every session); # of men/women

2. Study fields:

Types of studies

Crop stand/plant development

Pests

Diseases

Natural enemies

Other field conditions, e.g., Are there field problems like golden snails that the field school is not addressing. Why?

Is the field representative of the whole paddy in terms of soil fertility, etc.? Is it accessible?

COMMENTS

3. Local government support

3.1. Kind/type of field school according to funding source: local government, non-governmental organization, FAO

3.2. Village officials can explain what the programme is; show interest in and support for the programme

(Highlights of dialogue with village officials)

3.3. Farmers can explain what the programme is; show interest in and support for the programme

(Highlights of consultation with farmers)

4. What is this?

4.1. No direct answers given, leading questions asked

As observed:

4.2. Function-related questions asked

As observed:

4.3. Participants able to state or define functional relationships in the agroecosystem

As observed:

5. Agroecosystem

5.1. Before activity, participants told goal and process to be followed

As observed:

5.2. Participants all in the field

As observed:

5.3. Process of observation includes the plant and the ecosystem

As observed:

5.4. Observations written down

As observed:

5.5. Specimens collected

As observed:

5.6. Drawings summarize observations

As observed:

5.7. Sufficient time allowed for all groups to ask questions; discussions not cut short to give way to special topics

As observed:

6. Farmers become experts

6.1. Facilitator poses problems and asks questions appropriate to analysis of drawing, uses Field Guide questions

As observed:

6.2. Groups discuss field conditions and agroecosystem relationship

As observed:

6.3. ‘What if’ scenarios discussed

As observed:

6.4. Previous weeks’ agroecosystem drawings used for comparisons

As observed:

6.5. Field management decisions made and critically examined before being accepted

As observed:

6.6. Other factors in addition to economic threshold levels (ETL) are considered; farmers developing feeling/intuitive threshold

As observed:

6.7. Participants active and working together in small groups

As observed:

6.8. Leader, using leading questions, helps participants to analyse activity and their learning

As observed:

6.9. Participants can state/define differences between pests and natural enemies and their roles in the agroecosystem

As observed:

6.10. Decisions based on levels of insect populations and analysis of their relationships in the ricefield ecology (including stage of growth of rice plants)

As observed:

6.11. All possible options for crop protection considered before arriving at decision; participants implement decision as agreed

As observed:

7. Special topics

7.1. Before activity begins, participants told goal and process

As observed:

7.2. All participants active and involved in the activity

As observed:

7.3. No small group dominated by one person to the point that others are totally excluded

As observed:

7.4. Participants present results of their work during the activity, summarizing what has happened and why

As observed:

7.5. Participants can state what they have learned because of activity

As observed:

7.6. Leader asks open-ended questions to help participants examine steps in process of activity and apply learning to ‘real life’

As observed:

7.7. Special topics respond to problems at hand

As observed:

8. Insect zoos

As observed:

9. Group dynamics focused on enhancing teamwork and problem-solving skills

9.1. Before activity begins, participants told goal and process

As observed:

9.2. All participants involved/active

As observed:

9.3. Facilitator a) reviews goal and process of activity; b) helps participants to identify key learning points based on activity; c) asks questions that help participants to learn from the experience

As observed:

9.4. Participants state/summarize what they have learned

As observed:

10. Other group processes

10.1. Trainers/farmer trainers work in groups and not as individuals (cooperative/group process)

10.2. Farmers come on time and stay until the session is finished

10.3. Participation and involvement of farmers (Are farmers talking more or are trainers/farmer trainers talking more?)

10.4. How well prepared are trainers/farmer trainers for the field school session (Are all materials ready? Can trainers provide additional information to what farmers have shared or clarify misconceptions?)

As observed:

As observed:

As observed:

As observed:

10.5. Trainers/farmer trainers meet regularly to discuss training activities and issues that may arise

(How often does your group, i.e. trainers meet? What do you discuss during meetings?)

10.6. Problem-solving is done during meetings and not in front of participants

(What do you do when there are issues your team cannot agree on?)

11. Ballot box evaluation

11.1. Testing of field-based knowledge and skills

(Ask for three sample ballot box questions.)

11.2. Latin names not used

11.3. Leader uses ballot box to reinforce learning; focus is on content, not scores

(What do you do with results of the ballot box test?)

12. Farmers’ ideas about field school

12.1. Feedback farmers offer about field school

12.2. Farmers’ plans about continuing IPM activities

(What can you say about the field school?)

(What are your plans for IPM?)

13. Programme management/support for training activities

13.1. Necessary supplies and materials in right amount are available and on time

(Name activities for the last 2 sessions and list materials used for these)

13.2. Training funds of right amount available and on time

(What problem do you have about funds and why?)

13.3. National/provincial/area programmes prepared, implemented and managed by field staff, with the national team providing technical support

(Who decides what IPM activities you will implement? How are you involved in planning for these activities?)

13.4. Permanent coordinators and monitoring teams regularly visit training centres and project areas for problem-solving sessions

(How often do permanent coordinators visit you? What do you discuss?)

13.5. Training teams have access to national office for problems that require immediate attention from that level

(When you have a problem which requires immediate action, how soon do you get help?)

13.6. Trainers/farmer trainers’ team works effectively and efficiently (e.g. good teamwork)

As observed:

13.7. Trainers/farmer trainers maintain good relations with farmers and local leaders

As observed:

13.8. Monitoring/evaluation results are used as inputs to improve training quality

(Is monitoring/evaluation done on the field school programme? How is this done? What were the results? How did you use the results?)

Source: Viet Nam National IPM Programme. 1998. Unpublished.

[7] Open data is the kind of data that can easily be accessed, modified, reused and shared online.

[8] Open data kit (ODK) is a free set of open-source application (e.g. Google) that allows the creation of questionnaires that can be filled out on mobile phones or tablets and allows real time mapping of responses. The data can be aggregated on a server and retrieved for analysis. In addition to collecting information, the application allows the collection of photos, videos

[9] The farmer field school methodology was customized for use in Farm Forestry Field Schools (FFFS). The FFFS sought to intensify social forestry and improve livelihoods of communities in Kenya.

2

Tools for evaluation

Several tools are available for the evaluation of FFS programmes. Their description and potential uses are outlined in Table 14.

Table 14. Possible evaluation tools for collecting data of FFS activities, with a short description and purpose

Source: Authors' own elaboration

2.1
Focus group discussion

what is it? A focus group discussion (FGD) is a technique to collect data through group interaction. It is a useful evaluation tool to increase understanding of different perceptions, opinions and experiences among 5–10 selected farmers with a similar background (Valters et al., 2016).

why use it? An FGD is used to capture different opinions and ideas that exist in the group, for example, with respect to the management of natural resources or about cultural norms. It is particularly useful where interaction is desired. Focus group discussions are used when there is a need to address issues emerging from a questionnaire survey that demand further insight, by capturing various viewpoints and describing underlying reasons. Conversely, the tool can also be used as a first step in developing an appropriate questionnaire (Valters et al., 2016).

how to use it? A focus group discussion requires bringing people from similar backgrounds or experiences together to discuss a particular topic of interest. The common method for selecting participants is to choose members of the community who could give the best information. A facilitator asks “why”, “what” and “how” questions or topics for discussion by the group and keeps the focus of the discussion (which can sometimes be difficult when many ideas are raised) on track. After the discussions, the comments of the group are analysed by identifying the ideas that repeatedly occur. The FGD can be useful in combination with tools that are easier to implement, but are less in-depth, such as spider diagramming, semi-structured interviews or surveys.

example

This example section contains a set of questionnaires used to explore the effects of Junior Farmer Field and Life Schools[10] (JFFLS) using focus group discussions. Discussions were held with different types of stakeholders, followed by semi-structured interviews with selected key informants following the FGDs.

Guide for focus group discussions with JFFLS participants

This is a participatory tool to help evaluate the EFFECTIVENESS and IMMEDIATE IMPACT of the JFFLS programme and participants’ satisfaction. A focus group is an interview with small groups of relatively homogenous people with similar backgrounds and experience. Participants are asked to reflect on the questions asked by the interviewers, provide their own comments, listen to what the rest of the group have to say and react to their observations. The information obtained is not the opinion of a single individual, but rather a consensus of opinions.

The following is a set of core discussion points or issues to explore during the focus groups with youth participants to evaluate their satisfaction with the programme and to solicit problems encountered, as well as suggestions for improvement. The guide serves as a checklist during the focus group and ensures that basically the same information is obtained from different groups. Each programme should decide what information is necessary and add additional questions to the interview guide, as necessary. The same guide can be used for groups with different characteristics, such as boys, girls, younger children, those about to finish school, etc.

- ➜Do you think that what you have learned in JFFLS is useful? How and why?

- ➜What did you like most about the JFFLS?

- ➜What did you like least about the JFFLS?

- ➜What are the most significant changes that happened in your life because you participated in the JFFLS?

- ➜If you are asked to talk to your friends and fellow students about JFFLS, what would you say?

- ➜In what ways do you now contribute differently at home because you attended a JFFLS?

- ➜What skills and attitudes have you learned or changed because you attended a JFFLS?

- ➜How much of what you learned in the JFFLS did you already know? Where did you learn those things before?

- ➜Did attending the JFFLS cause any difficulties in your life such as family relationships, social activities, school performance? If so, what kind of problems?

- ➜What improvements or new topics would you like to see in the JFFLS for the next cycle?

Guide for focus groups discussions with caretakers of JFFLS participants

The following is a set of core discussion points that can be used in focus group settings with caretakers of the youth participants to evaluate their satisfaction with the programme and to solicit ideas about problems encountered as well as suggestions for improvement. The groups will be composed of caretakers of JFFLS participants and may vary by characteristics, such as caretakers of boys, girls, younger children, those about to finish school, etc.

- ➜What skills and ideas have the youths learned or changed because they attended a JFFLS?

- ➜Do you think that what the youths have learned in the JFFLS is useful? How and why?

- ➜What are the main positive changes you have noticed in the youths attending the JFFLS?

- ➜In what ways do the youths now contribute differently at home because they attended a JFFLS?

- ➜If you are asked to talk with your friends and acquaintances about JFFLS, what would you say?

- ➜Were there any negative aspects of the JFFLS on family life, community relations, school performance, other matters? If so, what were they?

- ➜What improvements or new topics would you like to see in the JFFLS for the next cycle?

Sample note-taker’s form for FGD

Group composition________________________________________________________________________________

Note-taker’s name _____________________________________________________________________________________________

Number of participants _______________________________________________________________________________________

Venue ___________________________________________________________________________________________________________

Date _______________________________________________________________________________________________________________

Starting time _____________________________________ Ending time __________________________________

Seating chart (indicate the participants and their number/identifier)

Discussion points

Responses

Observations

FGD debriefing form

1) What are the main themes that emerged in this focus group?

2) Did any information contradict what you learned in previous focus groups?

3) What did participants say that was unclear or contradictory to your ideas?

4) What did you observe that would not be evident from reading a transcript of the discussion (e.g. group dynamics, individual behaviours)?

5) What problems did you encounter (logistical, behaviors of individuals, questions that were confusing, etc.)?

6) What issues will you follow up (e.g. what other questions can be asked to clarify, or get additional information on)?

7) Does the note-taker have any suggestions for the moderator, and vice versa?

Semi-structured interview guide for FGD

Key informant interviews with district officials, local government officers and other local stakeholders

This is a participatory tool to help evaluate the RELEVANCE, EFFECTIVENESS and IMMEDIATE IMPACT of the JFFLS programme. Key informants are persons who provide information that can assist in understanding the context of a programme or project, or in clarifying particular issues or problems. They are not chosen randomly, but intentionally, to give a voice to the different groups that have a stake in the JFFLS.

This guide contains a set of core questions or issues to explore during the interview. It serves as a checklist during the interview and ensures that basically the same information is obtained from a number of people. Each programme should decide what information is necessary from key informants and add additional questions to the interview guide, as needed. The interview should not last more than 30–45 minutes.

- ➜What do you know about the JFFLS in your community (district)?

- ➜In your opinion, how well is the JFFLS addressing the vulnerability of children?

- ➜In your opinion, what has changed in the community and for the participants as a result of the JFFLS?

- ➜To what extent do you think the JFFLS activities are meeting participants’ and stakeholders’ needs?

- ➜To what extent are JFFLS programmes aligned with the strategic objectives of local development partners?

- ➜Do you think that the JFFLS has implemented its planned activities in an effective and timely manner?

- ➜What problems and difficulties have you noticed that affect how well the JFFLS functions?

- ➜What are your suggestions for improvements in the JFFLS programme for future cycles?

Semi-structured interview debriefing form for FGD

1) What are the main themes that emerged from these interviews?

2) What did respondents say that was unclear or contradictory?

3) What did you observe that would not be evident from reading a transcript of the interviews (e.g. gestures, body language)?

4) What problems did you encounter (e.g. logistical, questions that were confusing)?

5) What issues will you follow up (e.g. what other questions or additional information can be asked to clarify issues)?

Source: FAO. 2010. Monitoring and evaluation toolkit for Junior Farmer Field and Life Schools. Rome. www.fao.org/3/i1489e/i1489e00.pdf

2.2
In-depth interviews/key informant interviews

what is it? In-depth interviews/key informant interviews used as an evaluation tool may be structured or open-ended, formal or informal, conducted face-to-face or by telephone to capture rich information from a stakeholder. In contrast to focus group discussions, this tool allows the interviewer and the interviewee to discuss other points or change the direction of the discussions, if needed.

why use it? In-depth interviews/key informant interviews are used when there is a need to obtain a better understanding of people’s attitudes and behaviours, and changes taking place in people’s lives or communities. It is also used when there is a need to gain insight into pressing issues. In-depth interviews are used when the aim is to obtain information about a particular topic or issue that has already come up (such as experience in empowerment, change in respondent’s role in the community). In-depth interviews can be conducted with anyone.

Key informants are experts who can provide information about key issues or problems (including sensitive information) affecting the community-at-large. Key informant interviews are used to explore a topic before obtaining more details about that topic, or to provide insight into particular problems at community level (such as the younger generation leaving agriculture; poor access to markets) and help in finding solutions.

how to use it? The interviewer asks questions and records the answers of a small number of carefully selected individuals. Respondents can explain their experiences in their own language and setting. Care is taken that the interviewer does not influence the respondent’s response by asking leading questions. In-depth interviews are a used to supplement some less in-depth methods (such as spider diagramming, questionnaire survey), and are particularly useful for describing outcomes and impacts in the human and social domains.

example 1

Informal interview guide

This activity aims to collect information in an informal way about the perception of community members regarding mung bean production and marketing issues. The following topics that could be discussed for each respondent category are suggested:

Farmers (2 men and 2 women):

Prior to the interview, the farmers will have taken part in group exercises (such as focus group discussions, community mapping) on the following topics:

- ➜cropping pattern;

- ➜role of women and men in crop production, marketing and utilization;

- ➜reasons for choosing the crop and a specific variety;

- ➜utilization patterns of crops cultivated;

- ➜problems faced with regard to (1) production; (2) marketing; and (3) utilization of the crops cultivated;

- ➜solutions to the above problems;

- ➜perceived opportunities for improving the farming enterprise.

In addition to the group exercises, possible questions for the informal interview could include:

- ➜What crops do you grow?

- ➜What is the size of the field/plots that you use for growing mung bean? Other crops?

- ➜What pest management strategy do you practise (for mung bean)?

- ➜Why did you decide to use chemical pesticides (for mung bean)?

- ➜How did you choose chemical pesticides (for mung bean)?

- ➜What is the size of your sprayer/tank (for mung bean)?

- ➜How many crops of mung bean a year do you grow (calculate the total litres of pesticides used in mung bean for one year)?

- ➜How much do you spend on chemical pesticides/crop (herbicides, insecticides, etc.)?

Traders (2 traders selling agricultural inputs and 2 traders selling agricultural products)

Suggested topics include:

- ➜local marketing networks;

- ➜marketing procedures;

- ➜constraints in (1) marketing and (2) utilization of crops commonly grown in the area;

- ➜perceived opportunities for improving the farming enterprise.

Possible questions:

- ➜What agricultural inputs/products are sold?

- ➜Where does the shop owner get the agricultural inputs/products from?

- ➜What other consumer goods are sold in the pesticide shop?

- ➜On an annual basis, what quantity of pesticides enters the shop? What quantity of other agricultural inputs/products enters the shop?

- ➜What is the best-selling agricultural input? Product?

- ➜What is the best-selling pesticide (indicate trade name, common name; what is it used for)?

- ➜If pesticides are sold, how are they stored in the shop?

- ➜How does the seller handle pesticides?

- ➜Are promotions or advertisements used to sell pesticides?

- ➜Does the shop owner give any advice/suggestions to farmers when they come to buy chemical pesticides?

- Is the shop used as a residence?

- What happens if a farmer does not have enough money to buy agricultural inputs or products?

- What are the constraints in (1) marketing and (2) utilization of crops commonly grown in the area?

- How can the farming enterprise be improved? (How can farmers do better farming? How can farmers better sell their produce?)

Community leaders (1 person)

Suggested topics include:

- ➜role in agricultural development in the village;

- ➜constraints in agricultural enterprises;

- ➜perceived opportunities for improving the farming enterprise.

Possible questions:

- ➜How does the local government support agricultural development in the village?

- ➜What are some constraints in (1) marketing and (2) utilization of crops commonly grown in the area?

- ➜How can the farming enterprise be improved? What can the local government do to improve the farming enterprise?

Source: Cambodia National IPM Programme. 2018. Unpublished.

example 2

Researcher (Name) _________________________________________________________________________________________   Date ____/ ____/ ____

District/Ministry ___________________________________________________________________________________________________

Name of Key Informant _____________________________________________________________

KNOWLEDGE ON CLIMATE CHANGE

What changes have you observed in your district that are attributed to climate change?

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

Climate change effects

How have the observed changes affected the following fields of agriculture in the district?

Crops

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

Livestock

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

What strategies are being used by the communities to cope with the effects of climate change in the district?

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

ADAPTATION PRACTICES

Adaptation refers to adjustment in natural or human systems in response to actual or expected climatic stimuli or their effects, which moderates harm or exploits beneficial opportunities.

What practices can be used by the communities in your subcounty/district/region to adapt to climate change?

1. __________________________________________________ 5. __________________________________________________

2. __________________________________________________ 6. __________________________________________________

3. __________________________________________________ 7. __________________________________________________

4. __________________________________________________ 8. __________________________________________________

Which of the mentioned adaptation options is being practiced by the communities in your district?

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

Which climate change adaptation activities is your institution engaged in? (Use list below as reference to discuss exhaustively and take notes, about the different activities happening the district in as many sectors as possible)

Adaptation practices and concepts

Crop

Conservation agriculture

Integrated pest management

Crop intensification

Livestock

Improved pasture management

Improved grazing management

Improved manure management

Forestry

Agroforestry

Sustainable forest management

Afforestation, reforestation and forest restoration

Land management

Sustainable land management

Improved crop and grassland management

Restoration of degraded lands and organic soils

Water management

Irrigation modernization

Rainwater harvesting

Cross-sector

Efficient energy use

Reduced post-harvest losses and waste recycling

Disaster risk management

Adoption of suitable crop, plant and animal varieties and strengthening of seed system

EXPOSURE AND SENSITIVITY

What are the most important climate-related hazards/threats the distric and/or ecological zone faces?

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

What measures have been put in place to reduce the potential effects of the hazards/threats on the agricultural sector

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

If the threat occurred in your zone (or when it occurs), what would happen (or what happens) to the person or asset that are exposed to it, that are fully in contact with it?

_________________________________________________________________________________________________________

_________________________________________________________________________________________________________

Options for climate change adaptation in the agricultural sectors

Altering exposure

- Assess impacts and map hazard zones

- Conduct proper land and wateruse planning

- Protect watersheds and establish flood retention zones

- Resettle humans and restructure agriculture

- Change cropping patterns

Reducing sensitivity

- Develop or adopt suitable crop, plant and animal varieties

- lmprove irrigation and drainage systems

- Enhance soil nutrition and onfarm water management

- Diversify cropping and agricultural activities

- Adopt disaster-prevention construction standards

lncreasing adaptive capacity

- Develop adaptive strategies and action plans

- Diversify sources of household income

- lmprove water and other infrastructure systems

- Establish disaster and crop insurance schemes

- Promote technical transfer and capacity building

Source: Adapted from Mwesigwa, B. 2013. Unpublished.

> Further reading:

Bakker, T., Dugue, P., Roesch, K. & Phillips, S. 2022. Methodological recommendations to better evaluate the effects of farmer field schools mobilized to support agroecological transistions. Rome, FAO. www.fao.org/publications/card/en/c/CB9925EN

Bakker, T., Dugue, P. & de Tourdonnet, S. 2021. Assessing the effects of farmer field schools on farmers’ trajectories of change in practices. Agronomy for Sustainable Development, 41(2):18. www.researchgate.net/publication/349799063_Assessing_the_effects_of_Farmer_Field_Schools_on_farmers’_trajectories_of_change_in_practices

Friis-Hansen, E., Duveskog, D. & Taylor, E.W. 2012. Less noise in the household: the impact of farmer field schools on Gender Relations. Journal of Research in Peace, Gender and Development, 2(2):044–055. www.researchgate.net/publication/267782272_Less_noise_in_the_household_the_impact_of_Farmer_Field_Schools_on_Gender_Relations

2.3
Spider diagramming

what is it? Spider diagramming is a simple tool for visualizing findings or perceptions on several indicators. It is typically composed of six axes, with concentric rings indicating the scale of those axes (1–5). Indicators are assigned to each of the axes, scores are given, and the dots are connected to resemble a spider’s web [see Figure 2. Hypothetical example of spider diagramming for six indicators in the social domain, showing the situation now (red line) and the situation before the FFS (blue line)].

why use it? Spider diagramming can generate valuable information about outputs, outcomes or impacts that may be difficult to measure with other tools. The participants are asked to give a score (1–5) to certain indicators or questions; the scores are given for the current situation and retrospectively for the situation before the FFS, or at the beginning of the project (see Figure 2). The results from spider diagramming, in the form of simple scores, provide an indication of outputs, outcomes or impacts. Nevertheless, the scores given may be quite subjective, or difficult to interpret for outsiders. Therefore, spider diagramming should be seen as an initial tool, to highlight indicators or impact areas that may need further study by using more in-depth tools.

Figure 2. Hypothetical example of spider diagramming for six indicators in the social domain, showing the situation now (red line) and the situation before the FFS (blue line)

Source: Authors' own elaboration

how to use it? Spider diagramming should be used in a participatory setting, by farmers participating in a workshop to identify the indicators to be evaluated, to define the scores given to each indicator (1–5), and to conduct the actual evaluation (Mancini et al., 2007; FAO, 2015). Selected indicators could be anything, from yields to perceived level of empowerment or confidence.

> Further reading:

Mancini, F., van Bruggen, A.H.C. & Jiggins, .L.S. 2007. Evaluating cotton integrated pest management (IPM) farmer field school outcomes using the sustainable livelihoods approach in India. Experimental Agriculture, 43:97–112. https://doi.org/10.1017/S001447970600425X

FAO. 2015. Handbook for monitoring and evaluation of child labour in agriculture: Measuring the impacts of agricultural and food security programmes on child labour in family-based agriculture. Rome. www.fao.org/3/i4630e/i4630e.pdf?utm_source=visiting+cards&utm_medium=qrcode&utm_campaign=occ-book-cards

©Claudio Guzmín

2.4
Most significant change

what is it? ‘Most significant change’ is a participatory storytelling technique used for monitoring and evaluation of outcomes in an open way (Dart et al., 2003). It is especially helpful to unearth unexpected outcomes of interventions that have been most valued by its participants, which cannot be tracked with indicators.

This tool actively involves project stakeholders in deciding the type of change to be recorded and in analysing the data. It can be considered a ‘story approach’ because it tends to answer the following questions:

- ➜Who did what?

- ➜When?

- ➜Why?

- ➜Why is it important?

For example, in FFS programmes it can generate stories by farmers about the changes they experience due to FFS activities.

why use it? ‘Most significant change’ is a qualitative instrument that can be used to assess the perceptions of project effectiveness. It has some similarities with the ‘picturing change’ method, but it is more in-depth. Most significant change can complement formal evaluation techniques, especially where the impact of an intervention can be described more effectively by qualitative rather than quantitative indicators.

how to use it? The process is organized into seven steps and involves collecting significant change stories emanating from the project/programme activities and then including designated stakeholders or staff in selecting them. The designated staff are first involved in searching for the project impact. Once changes have been captured, the people sit down together, document and read the stories aloud and have regular and often in-depth discussions about the most significant changes they have experienced.

STEP 1.

Introduce Most significant change approach to project stakeholders and foster interest and commitment in participation. Jointly define the broad domains of change areas (such as improved food security, resilient and sustainable production system).

STEP 2.

Collect stories describing a specific change from persons most directly involved in programme activities or from beneficiaries of programmes. They should be encouraged to report on why they consider that particular change to be the most significant.

STEP 3.

Analyse stories and filter them up through the levels of authority typically found in an organization or programme. Each involved group selects the most relevant stories and highlights the criteria used to select them, and feeds back to all interested stakeholders.

STEP 4.

After this process has been conducted for some time, produce a document that includes all selected stories. Ask the external stakeholders (such as donors) to select the stories that best represent the kind of outcomes that interest them the most and document the reasons for their choices. Give feedback about this information to programme managers.

STEP 5.

Validate the selected stories by visiting the sites where the events described took place.

STEP 6.

When a change is described, it is possible to include quantitative information as well as qualitative information. It is also possible to quantify the extent to which the most significant change identified in one place has taken place in other locations within a specific period.

STEP 7.

Look at who participated in the process, how this influenced the contents, and how often different types of changes were reported.

example

Sample questions to be asked:

- ➜What was the reason for the FFS at this site?

- ➜How were participants selected?

- ➜In your opinion, what was the main reason for the FFS?

- ➜What was the situation before the FFS and what is it meant to achieve?

- ➜What changed after the FFS? (this question will highlight the perception of change)

- ➜How effective was the FFS?

- ➜What was done well and what could have been done better? (this question will highlight the intervention approach, competence and attitude)

- ➜How have the FFS activities helped individuals to learn and change?

- ➜How have FFS activities helped the farmers (where there is a farmers’ group, or if talking to a government agency) to change? Has it made a difference to the decision-making process, provided incentives and opened up new opportunities for the group?

- ➜Have those who participated been able to share their knowledge?

- ➜Is there any institutionalization or scaling-up of the activity? (this question will highlight sustainability issues)

- ➜What needs to change in the future to make the FFS more effective?

> Further reading:

FAO. 2015. FAO approaches to capacity development in programming: Processes and tools. FAO capacity development. Rome. www.academia.edu/7728062/FAO_APPROACHES_TO_CAPACITY_DEVELOPMENT_IN_PROGRAMMING_PROCESSES_AND_TOOLS

Association of German Development NGOs (VENRO)/NGO-IDEAs, 2011. Tiny Tools. Bonn. www.ngo-ideas.net/mediaCache/Tinytools_Overview/Tinytools-Overview.pdf

2.5
Participant observation

what is it? Most evaluation tools rely on information passed on by respondents, for example to describe a change in farming practices, or a change in someone’s role within the farmers’ group. This information can be accurate or inaccurate. Participant observation is a qualitative method that employs the process of observation and/or engaging in the activities of a group of people with a shared identity (such as FFS participants). The process allows the observer to gain knowledge and understanding about the interactions, the site and the events that occur in the site related to the subject of the study (such as reduction in the use of pesticides).

why use it? Participant observation can be used to complement questionnaire data, to make independent observations of possible changes, behaviours or roles. It is particularly valuable when language difficulties are anticipated in the data collection process. It is useful when there is a need to provide descriptive information on the context and changes observed.

how to use it? Participant observation is conducted by direct observation (and documentation) of field activities, communications and group meetings. Information on who was involved, what happened, when, and where and how it happened, is recorded in a register or diary. The purpose is to describe behaviours and roles of a few carefully selected persons, for example to describe how processes of decision-making or negotiation take place between persons. The quality and usefulness of the data depend largely on the writing and observational skills of the observer and findings may be open to interpretation. These risks need to be addressed.

©Agrorural_MINAGRI

example

Health risks related to household storage and disposal of pesticides[11]

- ➜The observation is carried out before the start of an FFS programme and again at the end of the season. Participants work in pairs.

- ➜Distribute the printed illustration of the household map to each participant. Each pair agrees on the timing of the visit to each other’s house to carry out the observation (with the agreement of the local leaders, non-FFS participating farmers may be included in the observation and data compared with FFS farmers at the end of the season).

- ➜In the illustration, indicate where pesticides were observed in the household – storage and disposal.

- ➜In the FFS session, participants analyse whether pesticide storage and disposal are: child-, food-, water-, livestock-safe – for the house/s that they visited. Each farmer can then enter the data in a summary chart, either using different coloured symbols or using (+) for yes or (o) for no. Percentages can be calculated.

Figure 3. Left: Household map; right: Summary of observations on pesticide storage and disposal practices

Source: Adapted from Murphy, H. 1997. The health effects of pesticide use: Methods to conduct community studies with school age children. Jakarta, FAO. www.fao.org/3/ca8270en/ca8270en.pdf

2.6
Picturing change

what is it? Picturing change (also called ‘Picturing impact’) is a participatory tool for evaluating impact. The tool uses photographs and explanatory text from FFS graduates to describe what they experienced as impact of the FFS (Pontius, 2003).

why use it? Picturing change is an evaluation tool used to help farmers increase their awareness, confidence and determination (Pontius, 2003). The pictorial reports can be used to complement other evaluation tools, as they present a diversity of perceived impacts, several of which could be easily missed in centrally planned evaluation studies.

how to use it? Groups of farmers from several villages are selected to participate in a workshop to learn about the objectives and methods of evaluation. Each group is given a camera (or uses the camera on a smartphone), which is passed among farmers in the group, to take photographs of what they experienced as impact of the FFS. In a follow-up workshop, the farmers produce narratives explaining the impacts illustrated in each of their selected photographs.

Figure 4. Examples of ‘Picturing change’ by farmers from Cambodia

Tonnes and tonnes of cow dung are brought daily to Viet Nam from Cambodia through the Prey Veng-Tay Ninh border. “We request the members of our IPM Farmers’ Club not only to save money but to also save cow dung for compost. The local government must take action and issue sanctions to prevent those collecting cow dung and selling them for their own benefit. We need to act as a community.” Mrs Chorn Nhor was voicing out the thinking of other members of her IPM Farmers’ Club. Their own agricultural production could benefit from the incorporation of cow dung to improve soil structure and nutrient content. Her awareness about soil health and how this results in better yields resulted from learning in the farmer field school (FFS).

©Alma Linda Morales-Abubakar

In Snor village, Angkor Reach commune, Prash Sdach district, the members of the IPM Farmers’ Club proudly informed us about their toilets. Snor village was the first one in the province to have a toilet in 2007 under a Department of Rural Development-initiated programme. But the model was still unhygienic and escaping gas caused problems with air quality. Loans and earnings from dividends from their IPM Farmers’ Club savings funds made it possible for them to avail of and construct improved toilets with slabs, lid covers, chamber boxes and leach pits.

©Alma Linda Morales-Abubakar

The Snor village IPM Farmers’ Club was formed in 2009. By 2014, almost almost all houses in the village had their own toilet offering convenience and dignity and more importantly safety and privacy especially for women and girls. “We have uplifted the quality of life by improving health and sanitation in the community. It would not have been possible before we joined the IPM Farmers’ Club. Before, if we had some money, we kept it at home where it was very easy to access. We would end up using it and had no savings.”

©Alma Linda Morales-Abubakar

Source: FAO. 2018. Beyond farmer field schools. www.fao.org/cambodia/programmes-and-projects/success-stories/beyond-ffs/ar/

2.7
Questionnaire survey

what is it? A questionnaire survey is a general evaluation tool with applicability for a wide range of topics. The questionnaire contains a series of structured or semi-structured (open) questions for use in relatively short interviews. A questionnaire survey can be used to ask a large sample of farmers about quantitative information regarding production, or to solicit qualitative data such as on perceptions, attitudes or behaviour.

why use it? A questionnaire survey is used to reach a large sample at the same time. It is used to impose uniformity by asking all respondents the same questions. The tool is useful to facilitate the comparison and compilation and recording of basic data.

how to use it? Questionnaire surveys conducted at different points in time (such as baseline, post-FFS) should ensure that the methods are consistent. Inclusion of a control group should adequately address the problem of selection bias in the sample methods. If a baseline is missing, the survey could include questions asking respondents to recall the situation before the activities. In general, recall data are reliable for clear events that people remember (such as change of crop variety), but are not so reliable for collecting less eventful or subjective information (such as perceived financial situation, perceived level of confidence).

The clarity of questions must be ensured because the quality of answers depends largely on the understanding of the questions. Furthermore, it is sometimes difficult to persuade respondents to complete and return the questionnaires, so steps must be taken to ensure that these concerns are addressed.

example

Baseline survey for FFS on IPM in fruit fly

Date:

Name of interviewer:

Village:

Commune:

District:

Province:

I. Background information of farmer

1.6. Family Status:  Married   Single    Widow/Widower

1.7. What crops do you grow? What is the total land area?

 Vegetable: ....…..…........... ha  Fruit trees: ....…..……....... ha    Rice: …...…….............. ha

1.8. What major vegetable crops do you grow? 1: .......................... 2: ..........................3: ..........................

1.9. What major fruit crops do you grow? 1: ................................. 2: ............................3: ..........................

1.10. How many years have you been growing vegetables and fruit trees?

1. vegetables: ..............year 2. fruit trees: .............. year.

II. Fruit crop (heavily infested by fruit fly)

2.1. Fruit crop (heavily infested by fruit fly)

2.1.1 Cultivation method of fruit crop

2.1.1.1 Type of crop: ......................number of planting/year ........ planting time.......... land area m²

2.1.1.2. Seed:....................Variety: ........................ What is the reason for selecting this variety? .....................................................................................................................................................................................

2.1.1.3. Fertilizer application

No.

Name of fertilizer

Quantity (kg/land area)

Method of use

1

2

3

4

5

2.1.1.4. Crop management

2.1.1.5. Yield: .................... kg/land area (previous season)

2.2. Fruit crop (average infestation by fruit fly)

2.2.1 Cultivation method of fruit crop

2.2.1.1 Type of crop: ......................number of planting/year ........ planting time.......... land area m²

2.2.1.2. Seed:..........................Variety: ..............................Why did you select this variety? ................................................................................................................................................................................................

2.2.1.3. Fertilizer application

No.

Name of fertilizer

Quantity (kg/land area)

Method of use

1

2

3

4

5

2.2.1.4. Crop management

2.1.1.5. Yield: .................... kg/land area (previous season)

III. Damage and yield losses

3.1. Damage and yield losses by fruit fly on fruit crop

Crop

Area cultivated (ha)

Season of crop

Peak fruit fly infestation month

Severity in 2020–2021

% area infested

% losses reported

Pomelo

Orange

Guava

Mango

Sapodilla

Custard apple

Jujube

Milk apple

Jackfruit

IV. Opinion of farmer (on how to control fruit fly and improve cultivation technique)

What subjects/topics do you want to learn so as to be able to control fruit fly and improve your cultivation technique?

..............................................................................................................................................................................................................

..............................................................................................................................................................................................................

..............................................................................................................................................................................................................

Source: FAO Asia Regional IPM/Pesticide Risk Reduction Programme. 2018. Unpublished.

2.8
Survey of farm logbooks

what is it? A logbook survey is a tool to evaluate the use of logbooks (versus non-use) by FFS graduates, in comparison with the control group. Moreover, the information contained in the logbooks can provide relevant details on farming practices that could be used, for example, to evaluate changes after the FFS has been completed in the use of inputs, costs, crop choice, yields, marketing and profits.

why use it? Farm logbooks are used to identify issues for future consideration and provide evidence for action, change and impact to support respondents’ perspective. Data from farm logbooks can be reviewed against master data files, administrative data, training materials and correspondence.

how to use it? To build financial skills, the FFS teaches farmers to keep records and logbooks of their agricultural inputs, practices, outputs, sale and profits. Farm logbooks contribute to better informed decision-making on farming and marketing. However, not all FFS farmers keep farm logbooks. At the end of the season/production cycle, the use of inputs, costs, crop choice, yields, marketing and profits are compared with their own results in the same season/production cycle of the previous year to evaluate changes resulting from participating in the FFS. These can also be compared with FFS farmers who do not use farm logbooks and control farmers.

©Stephen Sherwood

example

Template for record keeping, FFS for family poultry producers

Income and operating costs per production cycle

Sales income

Input/material costs

Labour costs

Income and operating costs per activity

To be completed for each product or activity of the poultry FFS.

Production/activity _____________________________________________________________________________________________

Unit of production _______________________________________________________________________________________________

Number of units of production _______________________________________________________________________________

Duration of each cycle (in months) __________________________________________________________________________

Number of cycles per year __________________________________________________________

Total income and costs

Source: Adapted from FAO & IFAD. 2022. Farmer field schools for family poultry producers: A practical manual for facilitators. Rome, FAO. www.fao.org/3/cc0254en/cc0254en.pdf

2.9
Community visioning/vision mapping (combined with action planning for a new project or activities within a project and evaluation of community action plans)

what is it? Community visioning/vision mapping is a participatory tool that is used as the first step in the planning process. The planning process engages members of the community and various stakeholders (including farmer field school participants) in problem identification and the formulation of action plans/interventions to address the problem. In FFS programmes, community visioning and/vision mapping is combined with other evaluation tools to determine the progress towards achieving targets or goals (i.e. the vision) of the FFS group or the community.

why use it? The process of community visioning/vision mapping gives stakeholders the opportunity to decide on what and how they will address or solve issues/problems. The process can be used to track results and evaluate changes compared to what was initially imagined. The involvement promotes ownership of the process, makes them accountable and more committed to achieving results. Community members feel that their contributions (e.g. ideas) are valued thereby strengthening social cohesion and motivation to work towards making change.

how to use it? In the process of community visioning, participants discuss and answer basic questions, such as:

- ➜where are we now (including challenges, opportunities and resources available)

- ➜where are we going (the vision based on existing challenges, opportunities and resources available)

- ➜where do we want to be based on common aspirations and shared values (with defined parameters or measurable indicators)

- ➜what actions do we take or how do we get to where we want to go

- ➜how do we know if progress has been made (including monitoring, evaluation, reporting and designing adjustments needed)

When this tool is used In FFS programmes, usually participants graphically illustrate the current situation (in their community/landscape or watershed) and their vision and then carry out action plans within a given timeframe. The maps/drawings are then revisited and discussions are held to assess progress towards the goal, e.g. what happened, what did not happen and why. If needed, the FFS group/community may decide to re-draw a new vision. The evaluation is best done at the end of the first FFS cycle or at the end of a programme. The maps/drawings constitute a strong tool for evaluating progress against a vision and as a basis to integrate the activities and aspirations within the FFS group into the broader landscape level.

> Further reading: Green, G., Halnes, A. & Halebsky, S. 2000. Building our future: a guide to community visioning. Cooperative Extension Publications. Wisconsin. https://cdn.shopify.com/s/files/1/0145/8808/4272/files/G3708.pdf

example 1

Map of the Kayokwe Watershed, Muyebe and Murama hills, Burundi in 2019

©Stefano Mondovi

Map of the vision for the Kayokwe Watershed, Muyebe and Murama hills, Burundi in 2024

©Stefano Mondovi

example 2

Results of visioning exercises for FFS groups set up under the KULIMA project in Malawi

©FAO/Anne-Sophie Poisot

©FAO/Anne-Sophie Poisot

©FAO/Anne-Sophie Poisot

2.10
Kasese[12] tool

what is it? The Kasese tool is a simple tool to introduce group and individual goal setting. Similar to the community visioning/vision mapping tool described previously, this tool can be used by participants at the beginning to identify priorities and at the end of an FFS programme to review the extent to which they have progressed towards their targets/goals. The tool measures changes that may or may not have resulted from the group’s efforts or from contributions of others who support the group. Different tools (e.g. theory of change or influence matrix) are used to analyze the contributions.

why use it? Communities are the best source of information about the impact of development projects. They have deep knowledge of how they made use of projects, what changed and the reasons for the changes. The Kasese tool allows community members to reflect on changes in a systematic way as well as encourages them to be open and self-critical. The process results in relevant and valid information and empowers communities to be more in control of their realities.

how to use it? Led by a facilitator, participants share their objectives for change and discuss the extent to which they have achieved the change and what desired change has not been achieved. The assessment is done using a numerical scale (0-10) and the response of every participant in indicated on the board. Participants are asked why they gave the corresponding scores and the comments are recorded.

Source: Adapted from graph by P. Strauch

The discussion is followed by asking the reasons for the achievement or non-achievement of their targeted change, the factors that hindered them from achieving change and how these could be overcome. This is followed by the preparation of an action plan that details what immediate steps the group will take next as well as their longer-term plans.

It is important to record the stories shared by members of the group and requires one or two people to do the documentation. The documentation (i.e. the original records) should be kept by the group to enhance ownership. The tool could be repeated any time to compared the group’s progress in moving towards achieving their goals.

> Further reading: Association of German Development NGOs (VENRO)/NGO-IDEAs, 2011. Tiny Tools. Bonn. www.ngo-ideas.net/analyzing_trends/index.html

3

Design tools for impact assessment

There are several available design tools for impact assessment under programmatic conditions (see Table 15). These tools address the challenges that are commonly encountered in impact assessment.

Table 15. Design tools for impact assessment

Source: Authors' own elaboration

3.1
Baseline survey

what is it? A baseline survey is a study conducted at the start of the project to collect information on the status of the subject prior to the intervention. It aims to measure key indicators (such as crop yields) at the time prior to the activities against which the endline and post-FFS surveys are compared, in order to assess impact of the activities.[13]

why use it? A baseline can be established at different levels: at the start of a programme or project (for measuring impact during the project period), or at the beginning of each FFS season/production cycle (for measuring impact of each set of FFS units). The advantage of the latter is that the baseline can concentrate on those farmers selected as participants of the FFS, who can be followed up individually after the FFS has been completed.

how to use it? A baseline could establish main agricultural parameters, such as yields, cropping pattern, inputs and outputs prior to the activities. Baselines could also include other indicators of outcomes and impacts, for example knowledge, attitudes and practices of farmers, their collaboration with other farmers, their access to services and loans, and their standard of living. Only parameters that were included in the baseline will enable a comparison to be made after the activities.

Figure 5. Schematic diagram showing some options for design of an impact assessment study

Source: Authors' own elaboration

example

Cotton IPM Impact Assessment

Checklist for Data Collection

Farm-household survey questionnaire

Questionnaire Number _________________________________________________________________________________________

Name of interviewer _____________________________________________________________________________________________

Date of interview _________________________________________________________________________________________________

Duration of interview: ___________________ min

Head of household and address ____________________________________________________

Information collected

Source: Adapted from FAO-EU IPM Programme for Cotton in Asia. 2004. Unpublished.

3.2
Before–after comparison

what is it? Before–after comparison relates to a situation that exists before an intervention is introduced and the situation that exists afterwards (see Figure 5. Schematic diagram showing some options for design of an impact assessment study). The comparison is indicated by differences between data obtained before and after the intervention (such as an FFS) is implemented. This evaluates whether a change has occurred during the period over which the FFS was conducted.

why use it? Before–after comparison can be a suitable tool for studying aspects that are not easily influenced by other contemporary factors. For example, studying farmer knowledge, or farmer skills, before and after the FFS can provide useful results. However, when the period from ‘before’ and ‘after’ spans several years, it is likely that other factors (apart from the FFS) also influenced farmer knowledge or skills. For this reason, two issues should be considered when planning a before–after comparison: how likely the results are to be influenced by other contemporary factors, and the optimal time span that should be used.

how to use it? Situations or behaviours of project beneficiaries are evaluated before the intervention (such as an FFS), and then again afterwards. The differences between before and after are often attributed to the FFS. However, the challenge with this comparison is that there may have been other changes that occurred during the same period. For example, if a study compared the crop yields before and after the FFS, it may find an increase or a decline in yield. Such changes are dependent not only on what farmers learned in the FFS, but also on other factors, such as the weather conditions. Moreover, other factors such as programmes, policy changes, market effects or influences may also have affected the results. A framework for using the results chain to understand linkages between farmer field schools and their impact (see Part I, section 1.1) may be referred to in order to support attribution of the observed changes to the intervention.

©Khalid Jam

example 1

The article, Effects of training on acquisition of pest management knowledge and skills by small vegetable farmers, documents the effects of training in farmer field school and conventional training (i.e. classroom lectures) on the acquisition of knowledge and skills by vegetable farmers. The study was carried out in Yunnan province, China under the FAO-supported Vegetable IPM Programme in Asia during the period 2003 to 2007 (Yang et al., 2008). FFS farmers demonstrated significant gains of both simple (e.g. life cycle of a vegetable pest) and complex IPM knowledge (e.g. agroecosystem analysis, food webs and food chains) while only simple knowledge improved among farmers trained using conventional methods.

> The full article can be accessed on: https://agris.fao.org/agris-search/search.do?recordID=US201301561984

example 2

The article, Reducing Pesticide Exposure and Associated Neurotoxic Burden in an Ecuadorian Small Farm Population, documents how community-based interventions – including training in farmer field schools (FFS) in Integrated pest management (IPM) – reduces pesticide exposures and associated negative effect on the nervous system of small farm households in Ecuador. The study used baseline questionnaire surveys to collect information on pesticide-related knowledge, practices and exposure. Effects on the nervous system were assessed using a WHO Neurobehavioral Core Test Battery, a pencil and paper test on various domains (e.g. psychomotor, perceptual coding and motor speed, attention and short-term memory and mood). Pesticide applications by FFS and non-FFS farmers were compared.

A year later, the survey and neurobehavioral[14] tests were repeated. FFS graduates reduced their pesticide applications; households had increased their pesticide-related knowledge and better handling practices and their neurobehavioral tests had improved compared with non-FFS participants.

> The full article can be accessed on: www.tandfonline.com/doi/abs/10.1179/oeh.2007.13.3.281

> Further reading:

Yang, P., Liu, W., Shan, X., Li, P., Zhou, J., Lu, J., & Li, Y. (2008). Effects of training on acquisition of pest management knowledge and skills by small vegetable farmers. Crop Protection, 27(12), 1504–1510. https://agris.fao.org/agris-search/search.do?recordID=US201301561984

Cole, D. C., Sherwood, S., Paredes, M., Sanin, L. H., Crissman, C., Espinosa, P., & Munoz, F. 2007. Reducing pesticide exposure and associated neurotoxic burden in an Ecuadorian small farm population. International Journal of Occupational and Environmental Health, 13(3), 281–289. https://doi.org/10.1179/oeh.2007.13.3.281

3.3
With–without comparison

what is it? The with–without comparison is another design for an impact assessment. It is also called cross-sectional comparison (see Figure 5. Schematic diagram showing some options for design of an impact assessment study). No measurements are taken ‘before’ the activities; only ‘after’ the activities.

why use it? In using the with–without comparison method, the comparison is made between FFS graduates and farmers who did not participate in FFS (the control group) directly after the FFS is completed (immediate results), or several years after the FFS (long-term results).

how to use it? Comparison is made between FFS graduates (the activities group) and farmers who have not been engaged in the FFS (the control group). This can be a very practical design for impact assessment because there is no need for a baseline, or information collected before the activities. All that is needed is to select the two comparison groups. However, selecting two groups that are comparable can be a challenge. For example, programmes select FFS groups according to certain criteria, or may unintentionally select those that are within easy access by road. When the control group is selected at the time of study, it may be difficult to select according to the same criteria as the FFS groups were selected, thereby leading to selection bias. It is therefore important to match the control with the activities group. Methods for matching are discussed in section 4.2.

example

The publication, ”Reducing pesticide risks to farming communities: cotton farmer field schools in Mali”, documents the impact of integrated pest management training on the adoption of alternatives to the use of hazardous insecticides among cotton-growing households. Data was collected over the period 2003-2010 on pesticide use and yields of farmers who were trained in FFS and those who did not receive training (i.e. control). The analysis showed a reduction of about 92.5 percent among trained farmers compared with the control albeit no evidence was established for changes in yield resulting from changes in pest management practices.

> The full report can be accessed on: royalsocietypublishing.org/doi/10.1098/rstb.2012.0277

3.4
Difference-in-differences

what is it? The ‘difference-in-differences’ tool is increasingly being used in development programmes, including FFS programmes. This tool combines a before–after comparison and a with–without comparison, as illustrated in Figure 5. The method is also referred to as the ‘double-delta’ method.

why use it? The activities group (FFS farmers) and control groups (non-participating farmers) are selected before the start of activities. Both groups are sampled before and after the activities (see Figure 5. Schematic diagram showing some options for design of an impact assessment study).

how to use it? Contemporary changes are measured both in the activities and control groups before the start of the activities and after the activities (see Figure 5). The difference between the results can be considered to be the impact that is attributed to the activities (such as FFS activities). At the beginning, the activities and control groups may have different characteristics due to selection bias. However, by sampling both groups again after the activities, the difference between their results can be assessed, in order to isolate the impact that is ascribed to the FFS activities.

example

The publication, Empowering farmers to reduce pesticide risks, documents impact assessment work done on FAO-supported community education for pesticide risk reduction efforts implemented through farmer field schools under the National IPM Programmes of Cambodia and Viet Nam during the 2007–2012 period (Morales-Abubakar et al., 2013). The report compensated for non-project-related changes by comparing the results before and after FFS training with those from non-IPM/pesticide risk reduction control villages in a difference-in-differences design. Only farmers for whom both baseline and impact data were available in both IPM and non-IPM-PRR control villages were included in the analysis.

> The full report can be accessed on: www.researchgate.net/publication/259080275_Empowering_Farmers_to_Reduce_Pesticide_Risks

3.5
Capturing diffusion effects

what is it? A diffusion effect is when the control group becomes influenced by being indirectly exposed to the intervention activities. Diffusion is a desirable result in development programmes because it increases the scale of influence of the activities. However, a diffusion effect can complicate an impact assessment study, when the control group is no longer a ‘pure’ control, but becomes influenced by the intervention.

why use it? Capturing diffusion effects is a method for impact assessment of the FFS. The method is used to address the challenge of diffusion (the influence of the FFS on neighbouring farmers) by comparing data before and after the intervention among FFS farmers, neighbouring farmers and outside control farmers (see Figure 6). The method provides useful information for future replication of FFS interventions.

Figure 6. Design option that captures diffusion effects by selecting two control groups: a nearby and far-flung control

Notes: The comparison of results between the activities and the outside control is used for assessment of impact. The comparison of results between the nearby and the outside control is used to measure the diffusion effect.

Source: Authors' own elaboration

how to use it? To address the challenge of ‘diffusion’ or ‘spillover’ effects, the best design in impact assessment of the FFS is to include not one, but two types of control group: one control group taken from within the village where the FFS has been conducted (neighbouring farmers), and one control group taken from outside the village where the FFS has been conducted (the outside control farmers). The neighbouring farmers within the same village as the FFS may be partially influenced by the FFS, even without participating in the activities. However, the outside control farmers from villages without an FFS are less likely to have been exposed to such influence. The comparison of results between the FFS farmers and the outside control is used for assessment of impact. The comparison of results between the neighbouring farmers and the outside control is used to measure the diffusion effect. For example, the neighbouring farmers may have learned about a certain agronomic practice from the FFS alumni, whereas the outside control farmers may not.

example

The impact assessment report on FAO-supported community education for pesticide risk reduction in Asia, Empowering farmers to reduce pesticide risks, documented spillover effects of IPM/pesticide risk reduction training in Cambodia to fellow farmers who did not participate in the FFS (exposed farmers), but who may have learned indirectly from those who attended the field schools (Morales-Abubakar et al., 2013).

> Further reading: Khanal, A., Regmi, P.P., Bahadur, G.Kc., Bahadur, D.Kc. & Dahal, K.C. 2021. Determination on the level of adoption of IPM technology in western Nepal. International Journal of Environment, Agriculture and Biotechnology, 5(6). https://dx.doi.org/10.22161/ijeab.56.18

©FAO/Bahag

3.6
Stepwise implementation scheme

what is it? The stepwise implementation scheme is a design option that can help to extend the ‘useful period’ of the control group in a programmatic context. This design option uses a stepwise crossover scheme (also called the stepped-wedge scheme), recognizing that programmes cannot achieve universal coverage of the FFS activities overnight. Due to limitations in capacity, most programmes roll out the FFS gradually over a number of years.

why use it? When a programme gradually rolls out an activity over an entire area, a control group selected at the beginning may be covered by the activity in the following years. Hence, the control is ‘lost’ after some time. It can be difficult to address this challenge of ‘losing’ the control group. This challenge is particularly encountered by programmes aiming for high coverage of the farmer population with the FFS activities, or where farmer trainers set up new groups within their area. Small projects that only target a minor fraction of the farmers in a given area do not face this challenge. Even though a control group is crucial for measuring long-term impact, it may be unacceptable (or even unethical) to deliberately withhold a proven activity from farmers to maintain a control. This is when the stepwise implementation scheme can be used.

how to use it? Assume that all farmer groups in an area will eventually receive the FFS activities, but that it will take 10 years to achieve full coverage. The programme has different options for rolling out the FFS. It could allocate its FFS units evenly across all targeted villages, or it could scale up the FFS within the village before scaling out to other villages. In the first option, the outside control is quickly lost because all villages will soon have one or more FFS units. In the second option, the outside control group can be maintained for a longer period, until the programme starts scaling out to those outside control villages (see Figure 7).

Figure 7. Implementation phases according to the stepwise crossover scheme

Notes: The FFS activities are scaled up within villages (A-B) before being scaled out to other villages (C). Empty hexagons denote farmer groups without – and filled hexagons denote farmer groups with the FFS activities.

Source: Authors' own elaboration

A programme could also choose to roll out in a hybrid fashion, by building some level of critical mass within some villages before moving to the next village, thus benefiting from diffusion effects (Witt et al., 2008).

3.7
Accounting for FFS ‘dropouts’

what is it? A ‘dropout’ is a participant who stopped attending FFS sessions, or who attended fewer than the minimally required number of sessions (70 percent of sessions). It is important for any FFS programme to track the number of FFS participants who drop out before graduation. If the dropout rate towards the end of the FFS exceeds 10 percent, a programme should reconsider how participants are selected, and how they are informed about what to expect from the FFS.

why use it? In the FFS, there will always be dropouts. Accounting for dropouts should be covered if the impact study uses a before–after comparison or a difference-in-differences design (see section 3.4 Difference-in-differences design).

how to use it? If the assessment compares a sample of farmers before the FFS and another sample after FFS graduation, without consideration of dropouts, the results may be biased because the dropouts are not represented in the post-FFS sample. The assessment will be more accurate, but also more tedious, if it can account for dropouts. This is the case if the assessment follows a cohort of the same individuals before and after the FFS (called panel data). The data on dropouts can then easily be removed from the pre-FFS sample.

©FAO/Fulvio Cenci

[10] The Junior Farmer Field and Life School (JFFLS), applying the basic principles of farmer field schools, has been designed to enable young women and men, boys and girls to enhance their agricultural and entrepreneurship skills;

[11] Adapted from Murphy, H. Farmer-to-farmer studies

[12] Kasese is a town in western Uganda where the tool was originally designed and used by a group of persons with disabilities (PWD) to raise awareness about disability in the district.

[13] A baseline differs from a benchmark survey, which is the activity conducted at the beginning of a project to provide essential information about local problems, practices, socio-economic conditions and local demands that is used in the design of a project and its activities.

[14] Neurobehavioral relates to the relationship between the action of the nervous system and behavior disorders. In this example, the behavior disorders refer to those resulting from pesticide exposure, e.g. inability to follow a pattern of small circles, inability to perform skillful movements with hands and arms, inability to write symbols in blank spaces as fast as possible, recall digits in series forwards or backwards immediately after hearing them, confusion, etc.

4

Analytic tools for impact assessment

Various analytic methods are available to help programmes with the non-random assignment of their FFS units. The analytic methods seek to improve the quality of the impact assessment studies (see Table 16). Because these methods involve specialized modelling and statistics, it is prudent for programmes to team up with capable research groups to assist in the analysis. The purpose of the analytic tools for impact assessment is to produce results of impacts that can reliably be attributed to the activities.

Table 16. Analytic tools for impact assessment

Source: Authors' own elaboration

> Further reading: Van den Berg, H., Phillips, S., Dicke, M. & Fredrix, M. 2020. Impacts of farmer field schools in the human, social, natural and financial domain: a qualitative review. Food Security 2020, 12:1443-1459. https://doi.org/10.1007/s12571-020-01046-7

4.1
Difference-in-differences estimates

what is it? The difference-in-differences (DD) estimates compare the change in outcome in the activities group before and after the FFS intervention to the change in outcomes in a control group over the same period. The method enables an estimate to be made of the difference-in-differences, in what is also referred to as the ‘double-delta’ method.

why use it? The difference-in-differences can be estimated where baseline data are available, and a cross-sectional comparison is made between the activities group (the FFS) and the control group.

how to use it? Table 17 illustrates how the DD estimate is calculated. The FFS group increased its yield from 2.3 to 3.5 t/ha; an increase of 1.2 t/ha. The control group increased its yield from 2.1 to 2.5 t/ha; an increase of only 0.4 t/ha. This shows that even control farmers were able to increase their yields, possibly due to better rainfall. Hence, to obtain the yield increase that is attributable to the FFS we have to deduct the 0.4 from the 1.2 t/ha. This produces the DD estimate. The DD could be estimated for any quantitative parameter.

Table 17. Calculation of difference-in-differences estimates, with a numeric example on crop yields

Source: Authors' own elaboration

If separate DD estimates are produced for a range of farmers, or for several areas, these DD estimates will make it possible to test whether or not the difference in findings between activities and control is compelling (testing for statistical significance). Difference-in-differences analysis is quite a robust method if the comparison groups are relatively similar.

example

The impact of farmer field school training in integrated pest management on IPM knowledge, adaptation/adoption of IPM options, and appropriate application of pesticides and awareness of related health hazards was examined among vegetable producers in Cotonou, Benin. The difference-in-differences model was used to compare the knowledge and practices before and after the project of 54 vegetable producers using semi-structured interviews. The results of the study showed increased knowledge about IPM that was to some extent adapted into the farmers’ production systems. The report indicated that there was no significant difference for the type of synthetic pesticides used by the participants of the study.

> The full article can be accessed on: https://www.cambridge.org/core/journals/international-journal-of-tropical-insect-science/article/abs/farmer-field-schoolipm-impacts-on-urban-and-periurban-vegetable-producers-in-cotonou-benin/4A136C018449FF648004EFF4A06185FC

4.2
Matching

what is it? Matching is a technique designed to purposely select the samples of the activities group (the FFS) and the control group, so that both groups have similar characteristics; these characteristics are assumed to have been in place at baseline. Matching can be facilitated by making use of the programme’s selection criteria for farmers or groups to be enrolled in the FFS, and applying the same criteria to selection of the control group. When the two groups are similar at baseline, it is more likely that the difference that can be observed between the two groups after the intervention can be attributed to the FFS.

why use it? When no baseline data are available, there are other ways of evaluating results of the activities, using single difference (without baseline) instead of double difference estimates.[15] Single-difference analysis relies on the comparison between the activities (the FFS) and control groups after the FFS has been completed, to test whether the FFS farmers are performing better than the control group of farmers, as an indication of FFS impact. However, without a baseline, it is unknown whether or not the FFS farmers were already performing better prior to the FFS, for example if selection criteria favoured highly motivated farmers. Consequently, the main challenge of a single-difference analysis is to establish that the activities and control groups are similar. Matching is used to ensure that this issue is addressed.

how to use it? Matching with just one or a few characteristics (such as age category, education level) will be relatively simple, but may miss out on some important variables that could influence the results. For example, if the activities and control groups are matched according to age and education level, but the farmer’s participation in previous projects was not considered, then it is possible that the activities group had higher participation in previous projects as compared with the control group (or vice versa), which may affect the results.

©FAO/Harben Quinto

Characteristics used for matching are assumed to have been in place at baseline and not to have been affected by the activity. For this reason, socio-economic status may not be the right characteristic for matching, because the FFS graduates of the activities group may have increased their profits as a result of the FFS. Hence, a clear distinction should be made between baseline characteristics that are unaffected by the FFS, and characteristics that are likely to be affected by the activities – which could be considered as indicators of impact (see Table 18). Land ownership is an example of a characteristic that may be unaffected in the short term (say, within 1–3 years after the activities), but which may be affected in the medium term due to the accumulated savings of FFS graduates, enabling them to purchase land.

Table 18. Examples of household characteristics that are unaffected, possibly affected and likely to be affected by the FFS activities in the medium term (<5 years)

Source: Authors' own elaboration

Matching of individual samples between the comparison groups according to several variables, or characteristics, can be very tedious. Research groups and experts use statistical techniques (such as propensity score matching) to assist in matching, but these techniques are beyond the scope of this guidance document.

example

In the longitudinal study on the “Impact of farmer field schools on agricultural productivity and poverty in East Africa” the authors used quasi-experimental methods (e.g. matching, difference-of-differences, cross sectional design, before–after comparison) to show evidence on economic and production impact of a farmer field school project. The project demonstrated how FFS training could bring about a positive impact on production and income of small holder farmers – especially women with limited literacy.

> The full article can be accessed on: www.sciencedirect.com/science/article/pii/S0305750X11001495

4.3
Combining analytical techniques

what is it? Combining analytical techniques refers to the integration of at least two methods to enable a more complete and synergistic utilization and analysis of data than would be achieved by using quantitative and qualitative approaches separately.

why use it? Combining more than one analytical method aims to strengthen the results of a study. It offsets weaknesses from using only one method and confers the benefits of using a combination of methods. A combination of analytical techniques is used when there is a need to obtain a deeper understanding of why change is taking place, or is not taking place as planned, or to help understand contradictions between qualitative findings and quantitative results.

how to use it? For example, matching combined with difference-in-differences analysis can produce better comparison groups from the start, resulting in more convincing double-difference estimates. Similarly, any of these analytical methods can be combined with regression analysis, to provide more insight into the role of explanatory variables. Regression analysis is a technique that is used to find relationships between different variables by analysing trends in the data sets; for example, to find out whether poorer farmers benefit less from the FFS than farmers who are better-off; or whether farmers with higher knowledge scores use fewer pesticides than those with lower knowledge scores.

©FAO/Lekha Edirisinghe

example

The Chinese Ministry of Agriculture (MoA) instituted farmer field schools to address the problem of inefficient and excessive use of fertilizers among farmers. As to support plans to scale up farmer field schools, an assessment of the impact of FFS on fertilizer use was carried out in Anhui and Hebei provinces over the period 2011 to 2013. Matched pair random assignment of villages was used to select treatment and control groups and additional farmers were randomly selected for the exposed group to study diffusion effects. Comparison of means, ordinary least squares and instrumental variables regression were used to analyze impacts of the intervention.

> The full report can be accessed on: https://documents.pub/document/nicholas-burger-assessing-the-impact-of-farmer-field-mary-assessing-the-impact.html

4.4
Triangulation of data sources

what is it? Triangulation is the use of multiple methods or data sources for the cross-verification of or to analyse results of the same study. It validates the findings and results obtained from using different tools. Triangulation enables an assessment to be made of factors that influence the results of the study.

why use it? Impact assessment should not be a stand-alone study, but the results should be compared with the data collected using other qualitative and quantitative tools through the MEL system. The results of a quantitative impact assessment can be strengthened by additional qualitative data on the self-reported attribution of impact. Triangulation of data sources is used to test the validity of information, as well as to address the need to develop a comprehensive understanding of the study results or of a phenomenon.

how to use it? The combination of data sources is essential for interpreting and understanding the impact assessment results. For example, if the impact assessment did not find an effect of the FFS on farming practices, it would be crucial to examine the monitoring data, in order to determine whether the activities had been well-designed, the curriculum optimally adapted, and the activities implemented according to the programme’s quality standards.

To take another example, assume that the impact assessment demonstrates that FFS graduates have increased their income. In addition, qualitative data obtained from questionnaires or in-depth interviews could provide separate indications of income. Combining these disparate data sources may enhance the results (if the data obtained from the different sources are consistent), or may flag inconsistencies between the data sources for follow-up study.

©FAO/Philbert Yandemye

example 1

Focus group discussions for exploratory purposes and a formal survey using structured questionnaires were used to assess the “Impact of farmer field schools on human and social capital”. A sample of 70 FFS graduates and 70 non-FFS Ghanaian farmers were involved in the study that explored the impact of training in FFS on technical knowledge, experimentation, knowledge diffusion, group formation and social skills. The study highlights the contribution that FFS can make in strengthening human and social capital consequently serving as a springboard for other development activities in cocoa growing communities.

> The full report can be accessed on: https://www.researchgate.net/publication/233470610_The_Impact_of_Farmer_Field_Schools_on_Human_and_Social_Capital_A_Case_Study_from_Ghana

example 2

Using data from structured questionnaires, in-depth interviews, FGDs and key informant interviews, the study “The empowerment route to well-being: An analysis of farmer field schools in East Africa” explored outcomes of FFS in three countries (i.e. Kenya, Uganda and Tanzania) in Africa. Empirical relationships between participation in FFS and increased well-being, between participation in FFS and empowerment, and between empowerment and enhanced well-being were assessed. Analysis of data from two thousand questionnaires indicate the link between empowerment and well-being and the need for agricultural development programmes to focus on the process of empowering farmers rather than provide technical solutions.

> The full report can be accessed on: https://www.sciencedirect.com/science/article/pii/S0305750X1100132X

©FAO/Philbert Yandemye

[15] In the absence of a baseline, the assumption made is that local variables have remained unchanged following introduction of the FFS.

Glossary

Evaluation is the periodic assessment of data or experience from an ongoing or completed project/programme design, implementation and results.[16] Frequent evaluation provides useful information and lessons learned recorded by monitoring that can be used for decision-making on adjustments to the design or implementation of a current or future project/programme. Evaluation is aimed at examining the relevance and achievement of the goals and objectives, efficiency and effectiveness of the intervention (such as intended or unintended, direct or indirect resulting changes), impact and sustainability of the project/programme.

Experiential learning: a process where learners are engaged and learn by doing and reflecting on the experience.

Impact: the long-term, cumulative positive and negative, primary and secondary effect of a development intervention. This may be a direct or indirect, intended or unintended change resulting from the intervention.

Impact assessment is a methodological approach used to establish whether the observed impacts can reliably be attributed, or accredited, to the intervention. Impact assessment (IA) shows whether or not targets have been achieved and if the observed changes are resulting from the intervention or not.

Impact evaluation analyses what worked, what did not work and why. Impact evaluation covers a wider range of issues than impact assessment, such as the appropriateness of the intervention design, the cost and efficiency of the intervention, its unintended effects and how to use the experience from the intervention to improve the design of future interventions.

Indicators may be quantitative or qualitative parameters or factors that provide a simple and reliable means to test assumptions about how change is expected to happen, measure achievement, reflect changes connected to an intervention, or assess the performance of a development stakeholder.

Interventions are structured actions that include identifying the problem, designing and implementing activities to improve the quality of the project and move towards achieving its objectives.

Logical framework or logframe: a management tool used, most often at project level, to improve the design and monitor the implementation of development interventions. It identifies core elements (such as inputs, outputs, outcomes, impact) and their causal relationships, as well as indicators and assumptions or risks that could influence success or failure of the interventions.

Milestones: signs of project progress at certain reference points, such as output or outcome levels.

Outcomes: likely or achieved short-term and medium-term effects of the outputs of an intervention.

Outputs: changes resulting from the intervention that are relevant to the achievement of outcomes.

Results chain: the sequence of inputs, activities, outputs, outcomes, impact and feedback needed to achieve the desired objectives of a development intervention.

Sustainable livelihoods framework: an effort to conceptualize livelihoods in a holistic way. It captures the many complexities of livelihoods, as well as the constraints and opportunities they are subjected to, such as factors that are outside the control of individuals (for example, local norms and institutions), together with factors over which people have control (for example, household or individual assets).

[16] The term ‘results’ will be used in the document to refer to the set of outputs, outcomes and impacts of programme implementation. The term ‘effects’ is also used in other contexts and is understood as its synonym. This choice is in line with FAO’s results-based framework.

References

Bamberger, M. 2012. Introduction to mixed methods in impact evaluation. Impact Evaluation Notes, 3:1–38. www.interaction.org/wp-content/uploads/2019/03/Mixed-Methods-in-Impact-Evaluation-English.pdf

Braun, A., Jiggins, J., Roling, N., Van den Berg, H. & Snijders, P. 2006. A global survey and review of farmer field school experiences. Wageningen, the Netherlands. www.researchgate.net/publication/228343459_A_Global_Survey_and_Review_of_Farmer_Field_School_Experiences

CARE. 2015. CARE’s Participatory Performance Tracking Tool. A step by step guide for use. [online]. [Cited 13 April 2022). www.care.org/news-and-stories/resources/ppt-step-by-step-guide/

CIP-UPWARD (International Potato Center - Users’ Perspectives With Agricultural Research and Development). 2003. Farmer field schools: Emerging issues and challenges. Los Baños, the Philippines. https://books.google.it/books?id=8kx_xspVL2MC&printsec=frontcover&hl=fr&source=gbs_ge_summary_r&cad=0#v=onepage&q&f=false

Dart, J. & Davies, R. 2003. A dialogical, story-based evaluation tool: The most significant change technique. American Journal of Evaluation, 24:137–155. 10.1177/109821400302400202

DfID (Department for International Cooperation). 1999. Sustainable livelihoods guidance sheets. London. www.livelihoodscentre.org/-/sustainable-livelihoods-guidance-sheets

Douthwaite, B., Kuby, T., van de Fliert, E. & Schulz, S. 2003. Impact pathway evaluation: An approach for achieving and attributing impact in complex systems. Agricultural Systems, 78:243–265. https://doi.org/10.1016/S0308-521X(03)00128-8

FAO (forthcoming). Champs-écoles des producteurs, égalité de genre, inclusion sociale et autonomisation des communautés au Sénégal. Rome.

FAO. 2015. FAO approaches to capacity development in programming: Processes and tools. FAO capacity development. Rome. www.academia.edu/7728062/FAO_APPROACHES_TO_CAPACITY_DEVELOPMENT_IN_PROGRAMMING_PROCESSES_AND_TOOLS

FAO. 2015. Handbook for monitoring and evaluation of child labour in agriculture: Measuring the impacts of agricultural and food security programmes on child labour in family-based agriculture. Rome. www.fao.org/3/i4630e/i4630e.pdf?utm_source=visiting+cards&utm_medium=qrcode&utm_campaign=occ-book-cards

FAO. 2016. Farmer field school guidance document: Planning for quality programmes. Rome. www.fao.org/documents/card/en/c/d7d4db1f-826f-4d81-b097-44292ff7eeca/

FAO. 2019. Farmers taking the lead: Thirty years of farmer field schools. Rome. www.fao.org/3/ca5131en/CA5131EN.pdf

Gertler, P.J., Martinez, S., Premand, P., Rawlings, L.B. & Vermeersch, C.M.J. 2016. Impact evaluation in practice. Second edition. Washington, DC, World Bank. http://hdl.handle.net/10986/25030

Habermas, J. 1971. Knowledge and human interests. Boston, USA, Beacon Press. www.wiley.com/en-us/Knowledge+and+Human+Interests-p-9780745694177

Kolb, D.A. 1984. Experiential learning: Experience as the source of learning and development. Englewood Cliffs, USA, Prentice-Hall. www.researchgate.net/publication/235701029_Experiential_Learning_Experience_As_The_Source_Of_Learning_And_Development

Mancini, F., van Bruggen, A.H.C. & Jiggins, J.L.S. 2007. Evaluating cotton integrated pest management (IPM) farmer field school outcomes using the sustainable livelihoods approach in India. Experimental Agriculture, 43:97–112. https://doi.org/10.1017/S001447970600425X

Morales-Abubakar, A.L., Arnst, R., Chung, D.K., Ketelaar, J.W., Pananurak, P., Praneetvatakul, S., Seng, S., van der Wulp, H., Walter-Echols, B. & Walter-Echols, G. 2013. Empowering farmers to reduce pesticide risks. Bangkok, FAO. www.researchgate.net/publication/259080275_Empowering_Farmers_to_Reduce_Pesticide_Risks

Murphy, H. 1997. The health effects of pesticide use: Methods to conduct community studies with school age children. Jakarta, FAO. www.fao.org/3/ca8270en/ca8270en.pdf

PELUM-Uganda. 2016. Adapting the Gender Action Learning System (GALS) in development programmes. Models of integrating GALS in functional adult literacy, agricultural extension, advocacy, village savings and lending associations. Kampala, Participatory Ecological Land Use Management (PELUM) Uganda; OXFAM.

Pontius, J.C. 2003. Picturing impact: Participatory evaluation of community IPM in three West Java villages. Paper presented at the “International learning workshop on Farmers Field Schools: Emerging issues and challenges”, 21-25 October 2002, Yogyakarta, Indonesia.

Pontius, J.C., Dilts, D.R. & Bartlett, A., eds. 2002. Ten years of IPM training in Asia: From farmer field school to community IPM. Bangkok, FAO. www.fao.org/docrep/005/ac834e/ac834e00.htm

Rogers, C. 1969. Freedom to learn. Columbus, USA, Merrill. www.worldcat.org/title/freedom-to-learn/oclc/912482312

Scoones, I. 1998. Sustainable rural livelihoods: A framework for analysis. IDS Working Paper 72 (unpublished).

Schwandt, T., D’Errico, S., Ofir, Z., Lucks, D. & El-Saddick, K. 2018. Embedding evaluation in national plans and policies to foster transformative development. IIED Briefing. London, International Institute for Environment and Development. https://pubs.iied.org/sites/default/files/pdfs/migrate/17464IIED.pdf

Tripp, R., Wijeratne, M. & Piyadasa, V.H. 2005. What should we expect from farmer field schools? A Sri Lanka case study. World Development, 33:1705–1720. www.sciencedirect.com/science/article/abs/pii/S0305750X05001300

Valters, C., Cummings, C. & Nixon, H. 2016. Putting learning at the centre: Adaptive development programming in practice. London, Overseas Development Institute. www.alnap.org/help-library/putting-learning-at-the-centre-adaptive-development-programming-in-practice

Van den Berg, H. 2004. IPM farmer field schools: A synthesis of 25 impact evaluations. Rome, FAO. www.fao.org/3/ad487e/ad487e00.htm

Van den Berg, H., Ketelaar, J.W., Dicke, M. & Fredrix, M. 2020. Is the farmer field school still relevant? Case studies from Malawi and Indonesia. NJAS-Wageningen Journal of Life Sciences, 92:100329. https://agris.fao.org/agris-search/search.do?recordID=NL2020005031

Van den Berg, H., Phillips, S., Dicke, M. & Fredrix, M. 2020. Impacts of farmer field schools in the human, social, natural and financial domain: A qualitative review. Food Security 2020, 12:1443–1459. https://doi.org/10.1007/s12571-020-01046-7

Van den Berg, H., Phillips, S., Poisot, A.S., Dicke, M. & Fredrix, M. 2021. Leading issues in implementation of farmer field schools: A global survey. Rome. Journal of Agricultural Education and Extension, 27:341–353. https://doi.org/10.1080/1389224X.2020.1858891

Van Rijn, F., Burger, K. & Den Belder, E. 2012. Impact assessment in the sustainable livelihood framework. Development in Practice, 22:1019–1035. London, Routledge. http://dx.doi.org/10.1080/09614524.2012.696586

Waddington, H., Snilstveit, B., Hombrados, J.G., Vojtkova, M., Anderson, J. & White H. 2014. Farmer field schools for improving farming practices and farmer outcomes in low-and middle-income countries: A systematic review. Campbell Systematic Reviews, 10. https://onlinelibrary.wiley.com/doi/10.4073/CSR.2014.6

Witt, R., Pemsl, D.E. & Waibel, H. 2008. The Farmer Field School in Senegal: Does training intensity affect diffusion of information? Journal of International Agricultural and Extension Education, 15:47–60. 10.5191/jiaee.2008.15204

Annex 1
MEL reorientation in existing programmes

Existing FFS programmes mostly have established protocols and methods for monitoring and evaluation in place. Adopting an improved MEL system would therefore involve a process of learning and reorientation. Hence, existing programmes are encouraged to reflect critically on the established system for monitoring and evaluation, in order to transform to a MEL system with a stronger learning component and a better thought-out framework.

Several steps are proposed for the reorientation (see Figure A1):

1. The process of reorientation is best started by preparing a proposed MEL framework.

2. A situation analysis is conducted of the existing system of monitoring and evaluation.

3. A side-by-side comparison between the two systems is made.

4. The MEL framework is consolidated within the programme’s management system.

Figure A1. Proposed steps in the reorientation of an existing MEL framework

Source: Authors' own elaboration

The reorientation activities are preferably carried out through workshops or focus groups, with broad participation of farmers, facilitators, master trainers and resource persons (including from the funding agency). Farmers and facilitators will play a critical role by advising on indicators for outputs and outcomes.

Preparing the MEL framework

New or existing FFS programmes are encouraged to prepare their worked-out MEL framework, with impact targets in each domain, results chain, and milestone indicators. Each targeted impact is selected through consensus in a workshop setting. Tools for data collection should be outlined for measurement of each indicator.

The indicators and tools needed for MEL should be listed to provide an overview for easy reference. These indicators and tools should include those for monitoring the quality of activities, as well as those for evaluating the results.

Situation analysis

An analysis of the monitoring and evaluation system is essential to identify the gaps and possibly the redundancies in the existing system of monitoring and evaluation, and to examine the discrepancies between the existing system vis-à-vis the MEL framework.

Programmes typically use a logical framework (also known as logframe) as the basis for their protocols for monitoring and evaluation. Mostly, these logframes have been prepared according to the requirements of the funding agency, government authorities or other agencies. In the event of there being more than one FFS programme, separate analyses should be conducted for each programme.

The situation analysis could be conducted by the M&E officer or M&E team, or be contracted to a third party. The primary analysis involves some main questions:

1. Which tools for monitoring and evaluation are being used?

2. Which types of data are being collected using each tool?

3. What are the key indicators of outputs, outcomes and impacts?

Then, a recap is conducted to produce a list of tools used and data types collected. This list of tools and data types will be used in the comparison with the MEL framework.

The secondary analysis involves several additional questions:

1. How are data reported, and to which level?

2. Is impact assessment conducted or planned?

3. How are the collected data being used, and for what purpose?

4. How frequently is an internal review or evaluation conducted?

The secondary analysis thus provides further insight into the reporting and utilization of data, for comparison with the MEL framework.

Comparing MEL framework with existing system

After preparation of the MEL framework and the situation analysis, the two systems will be compared with each other in order to determine where they overlap, and where they differ. This comparison, and its consolidation, are ideally carried out in a workshop setting to establish consensus.

The list of indicators needed for MEL is compared vis-à-vis the list of indicators in the existing system. The comparison will reveal which of the indicators needed for MEL are already covered by the existing system, and which indicators should be added for MEL (see Figure A2).

Figure A2. Schematic example of a comparison between the MEL framework presented in this document and the existing system

Notes: Numbers signify indicators. Arrows point to indicators that are in the MEL framework, but are lacking from the existing system. Question marks show indicators in the existing system that are not in the MEL framework; this could suggest redundancy.

Source: Authors' own elaboration

The comparison also reveals whether any data being collected by the existing system do not have a meaningful purpose in the MEL framework. Such data may have a specific purpose for the donor or for another stakeholder, but alternatively, the data may be redundant.

Similarly, the lists of data collection tools are compared between the MEL framework and the existing system. This comparison reveals whether the tools necessary for MEL are in place, or if certain tools should be added. Tools unnecessary for MEL could be removed from the existing system if they serve no other purpose.

Checking for redundancies is important, because a recent global survey reported that some FFS programmes are being overwhelmed by the data they collect on a routine basis, suggesting the need to collect only the most relevant data that can be put to good use (Van den Berg, 2020).

Next, the results of the secondary analysis of the existing system are examined. This will show whether important data on the milestone indicators are being reported to the central programme level. Also, it will reveal whether data needed only at local levels are being unnecessarily reported to central level, thus burdening programme management. For example, detailed data from FFS implementation may only be needed at local level. As a result, changes may be needed in the reporting structure of certain data types.

Finally, utilization of the data in the existing system should be inspected. If the data end up in reports for the donor or government authority, without being used for the internal benefit of the programme, some opportunities for learning and adaptation are being missed. In this respect, the mechanism and frequency of internal review or evaluation is indicative of whether or not the data are regularly used for reflection, learning and adaptation.

Consolidating MEL within the programme

The comparison between the MEL framework and the existing system will result in recommendations. These should include action steps on how to incorporate or merge the MEL framework within the existing programme to improve the quality of activities and to reach the desired impacts. When the MEL framework has been formally approved by the funding agency, programme management can start planning for incorporating the changes into the programme’s structure for monitoring and evaluation. This involves adjustment or updating of protocols and guidelines, and refresher training for those involved in data collection. Functional and efficient MEL is characterized by being target-oriented, avoiding redundancy of data collection or data reporting, and having the flexibility to adapt to changes over time.

©FAO/Deborah Duveskog

At this stage it is crucial to assess the feasibility of the MEL framework, with its data types to be collected and reported. In the event that the expectations from the MEL system are over-ambitious, the data collection and/or reporting should be reduced, so as to be manageable with the available resources. In particular, redundant reporting of details from each FFS to central level should be avoided, so that programme management can concentrate on the milestone indicators. Guidelines and protocols need to be regularly updated to accommodate the modifications made after each annual MEL learning cycle.

In capacity-building on MEL, master trainers and FFS facilitators should learn about the benefits of MEL and use of the MEL guidelines and protocols. Capacity-building on MEL should start with training workshops for master trainers in MEL, and subsequently move onto including a dedicated MEL module in the training or refresher training of FFS facilitators. Separate guidelines and protocols on MEL may be needed for regular facilitators (dedicated staff of government or NGOs) and farmer facilitators (farmers who became facilitators in their own location).

Annex 2
Application of the sustainable livelihoods framework to assess the potential benefits of livestock FFS on the livelihoods of FFS members

Livelihood capital*

Potential benefits

Human
(Skills, knowledge, health and ability to work, etc.)

Improved critical and logical decision-making

Improved organizational and critical skills

Improved management and entrepreneurial skills (farming as a business)

Better understanding of the local ecosystem

Increased awareness of the community’s common challenges

Improved livestock management practices, including hygiene and sanitation issues

Improved nutrition through diversification of diets and changes in food consumption patterns

Increased awareness of sustainable management of natural resources and the relation to productive and income-generating activities

Climate change and drought preparedness, adaptation and mitigation

Social
(Family relationships, social resources, informal networks, membership of formalized groups, trust, etc.)

Increased cohesion among FFS group members and strengthened ability to work together, solve problems and take decisions together

Creation of informal networks for economies of scale, cooperation between farmers and financial agreement between members

Increased trust between members and with service providers (e.g. extension services)

Women’s empowerment and gender equality, e.g. increased women’s influence and participation, involvement in leadership, access and control over household resources, changes in gender roles and division of labour

Improved access to basic services (health, education, veterinary, etc.)

Resolution of problems such as conflicts and cross-border trade

Increased status of FFS groups and their members (especially women) within the community

Strengthened linkages with research, development partners, and other progressive producers

Natural
(Natural resources such as land, wildlife, soil, water, forests and fisheries)

Improved access to pastures and fodder trees

Restored soil fertility with compost and manure utilization, and conservation with erosion mitigation techniques and zero tillage

Improved water conservation

Improved cross-border and regional grazing resource management

Improved natural resource management including pasture establishment, fodder conservation and reforestation

Physical
(Basic infrastructure (roads, water & sanitation, etc.), agriculture infrastructure, schools, producer goods including tools, livestock and equipment)

Improved livestock infrastructure (milk parlours, sheds, poultry coops, etc.)

Improved access to adapted genetic resources

Adoption of new technologies and tools

Increased number of livestock due to reduced mortality and improved fertility

Increased output of animal products

Financial
(Financial resources including savings, credit, and income from employment, trade and remittances)

Increased income and development of new income-generating activities

Improved access to financial services and to input and output markets

Development of saving and credit schemes

Increased financial responsibility and entrepreneurship

*As defined in the Sustainable Livelihood Framework of the UK’s Department for International Development, DFID (for more information see: http://www.eldis.org/vfile/upload/1/document/0901/section2.pdf).

Sources: FAO. 2018. Farmer field schools for small-scale livestock producers – A guide for decision makers on improving livelihoods. FAO Animal Production and Health Guidelines No. 20. Rome. www.fao.org/3/I8655EN/i8655en.pdf

Annex 3
Sample terms of reference for external institution for impact assessment study

A sample is provided of a terms of reference that was taken from a contractual agreement between Hanoi Agricultural University and FAO in 2009.

Description of activities/services

In collaboration with Kasetsart University, and in consultation with the FAO IPM Programme, CTA and Programme Development Officer, local FAO experts and counterparts, Hanoi Agricultural University will:

- ➔ provide inputs for the development of locally suitable methodologies and instruments for baseline/impact assessment of the Pesticide Risk Reduction Programme, including surveys on pesticide use and their impact on human health and the environment;

- ➔ design the baseline/impact assessment of the Pesticide Risk Reduction Programme (study targets, number and selection of sites, respondents, time frame, etc.);

- ➔ submit an inception report which includes the workplan, objectively quantifiable indicators, research sites, sampling design, methodologies, etc. for the baseline study; this should be done in close consultation with the National IPM Programme;

- ➔ report informally on the status of the study to the FAO Regional IPM Programme at the end of each month;

- ➔ coordinate with IPM trainers to identify sites and farmers to be involved in the study;

- ➔ develop and pre-test the farm-household survey questionnaire and other data collection methods;

- ➔ train enumerators and supervise data collection from 200 farmers in 2 IPM communes and 2 control communes for the year 2008 baseline values;

- ➔ immediately review and verify data for accuracy and consistency;

- ➔ supervise data entry and processing;

- ➔ prepare and submit a draft report;

- ➔ prepare and submit the final report; and

- ➔ present results at a regional workshop (costs not included).

Definition of outputs

A final report will be produced under this contract, covering the following items:

- ➔ Introduction

- ➔ Research design and methodology

- ➔ Socio-economic status and agricultural production in the study communes

- ➔ Socio-economic status of farmers, pesticide dealers in the study commune

- ➔ Knowledge, awareness and attitude of farmers, commune leaders, dealers on pesticide risks and concern for food safety

- ➔ Decision-making and pest, pesticide management practices and behaviours of commune leaders, farmers and pesticide dealers

- ➔ Community-based actions and institutionalization towards pest and pesticide management

- ➔ Status of health risks caused by pesticides

- ➔ Status of environmental risk caused by pesticides (environmental indicators observed)

Duration and timing

Starting on 17 March 2008 and finishing on 30 September 2008.

As mentioned in the reporting section of the contract, the recipient organization will submit the following reports:

An inception report in English with workplan will be submitted to the FAO Regional IPM Programme Development Officer within two weeks after signing the agreement.

A draft final report in English, together with the database files, will be submitted to the FAO Regional IPM Programme Development Officer on or before 30 September 2008.

A final report in English will be submitted to the FAO Regional IPM Programme Development Officer no later than one month after acceptance of the draft report. The report should be made available in both hard copies (2) and in diskette. The final report should be accompanied by a final unaudited statement of accounts.