Required citation:

FAO. 2024. Surveillance Evaluation Tool (SET) – A guide for SET evaluators. FAO Animal Production and Health Manual, No. 29. Rome, FAO. https://doi.org/10.4060/cc8992en

The designations employed and the presentation of material in this information product do not imply the expression of any opinion whatsoever on the part of the Food and Agriculture Organization of the United Nations (FAO) concerning the legal or development status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries. The mention of specific companies or products of manufacturers, whether or not these have been patented, does not imply that these have been endorsed or recommended by FAO in preference to others of a similar nature that are not mentioned.

ISBN 978-92-5-139361-1

© FAO, 2024

Some rights reserved. This work is made available under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 IGO licence (CC BY-NC-SA 3.0 IGO; https://creativecommons.org/licenses/by-nc-sa/3.0/igo/legalcode).

Under the terms of this licence, this work may be copied, redistributed and adapted for non-commercial purposes, provided that the work is appropriately cited. In any use of this work, there should be no suggestion that FAO endorses any specific organization, products or services. The use of the FAO logo is not permitted. If the work is adapted, then it must be licensed under the same or equivalent Creative Commons licence. If a translation of this work is created, it must include the following disclaimer along with the required citation: “This translation was not created by the Food and Agriculture Organization of the United Nations (FAO). FAO is not responsible for the content or accuracy of this translation. The original English edition shall be the authoritative edition.”

Disputes arising under the licence that cannot be settled amicably will be resolved by mediation and arbitration as described in Article 8 of the licence except as otherwise provided herein. The applicable mediation rules will be the mediation rules of the World Intellectual Property Organization http://www.wipo.int/amc/en/mediation/ rules and any arbitration will be conducted in accordance with the Arbitration Rules of the United Nations Commission on International Trade Law (UNCITRAL).

Third-party materials. Users wishing to reuse material from this work that is attributed to a third party, such as tables, figures or images, are responsible for determining whether permission is needed for that reuse and for obtaining permission from the copyright holder. The risk of claims resulting from infringement of any third-party-owned component in the work rests solely with the user.

Sales, rights and licensing. FAO information products are available on the FAO website (www.fao.org/publications) and can be purchased through publications-sales@fao.org. Requests for commercial use should be submitted via: www.fao.org/contact-us/licence-request. Queries regarding rights and licensing should be submitted to: copyright@fao.org.

Photo cover: © FAO/Anis Mili

Contents

Abbreviations

1. Background

1.1. Context

1.2. Development of SET

1.3. Outputs of SET

1.4. Links with other assessment tools available

1.5. SET implementation in Africa and Asia

1.6. SET Biothreat Detection Module

1.7. Objective of this manual

2. Implementation of SET

2.1. General principles

2.2. External and self-assessments

2.3. Cost of missions

2.4. Implementation of SET missions

3. Preparatory phase

3.1. Invitation from Ministry

3.2. Identifying evaluation team members and their responsibilities

3.3. Sharing and reviewing the SET package

3.4. Request for information and reviewing documents of the system

3.5. Defining mission dates, stakeholders to interview and programme

3.6. Preparation checklist

4. In-country mission

4.1. Team briefing

4.2. Launching meeting

4.3. Interview process

4.4. Field portion of the mission

4.5. Scoring

4.6. Reviewing results

4.7. SWOT analysis

4.8. Developing the action plan

4.9. Closing meeting

4.10. Team debriefing

5. Post-mission

5.1. Feedback to the project leads and updating SET

5.2. Drafting the final report

5.3. Implementing the action plan

5.4. Following up on progress

6. Resources

Annexes

I. Tables listing the SET 2.0 core, direct support and indirect support indicators

II. Performance attributes definition and resources

III. Calculation of performance attribute scores

IV. Preparing a mission using SET-BT Biothreat Detection Module

V. SET mission checklist

Figures, Tables and Boxes

Figures

1. Assessment tools available to countries - ranging from general to specific

2. Overview of the implementation of SET missions

3. Using the SET Excel file to enter scores, justifications and notes for each indicator

4. Graphical outputs generated by SET. A: Core competencies; B: Performance attributes.

5. SWOT analysis definitions

6. Framework used to develop SET recommendations

7. Steps in drafting and publishing SET reports

A.IV.1. Biothreat detection module categories (fictive country)

Tables

1. Areas and categories evaluated by SET

2. Characteristics of external and internal evaluations

3. Cost matrix for SET missions

4. Documents in the SET packet

5. Guideline of major stakeholders to meet during SET missions

6. Suggested template for SET mission programme (external evaluation)

7. Guidance to develop SMART recommendations

A.I.1. Case detection (Core)

A.I.2. Case confirmation (Core)

A.I.3. Reporting (Core)

A.I.4. Data analysis and interpretation (Core)

A.I.5. Feedback (Core)

A.I.6. Surveillance design (Core)

A.I.7. Standards and guidelines (direct support)

A.I.8. Training (direct support)

A.I.9. Supervision (direct support)

A.I.10. Monitoring and evaluation (direct support)

A.I.11. Coordination (indirect support)

A.I.12. Communication (indirect support)

A.IV.1. Documents in the SET-BT subpacket

A.IV.2. Template SET-BT mission programme (external evaluation)

Abbreviations

CVO

chief veterinary officer

DSA

daily subsistence allowance

FAO

Food and Agriculture Organization of the United Nations

GAC

Global Affairs Canada

GHSA

Global Security Agenda

INTERPOL

International Criminal Police Organization

JEE

Joint External Evaluation

LMT

Laboratory Mapping Tool

NFPs

national focal points

PVS

Performance of Veterinary Services

RFI

request for information

SET

Surveillance Evaluation Tool

SET-BT

SET Biothreat Detection Module

SMART

specific, measurable, attainable, relevant, time-bound

SWOT

strengths, weakness, opportunities, threats

USAID

Unites States Agency for International Development

WHO

World Health Organization

WOAH

World Organisation for Animal Health

1. Background

1.1. CONTEXT

In 2016, some African countries in Phase 1 of the Global Health Security Agenda (GHSA) (United States Agency for International Development, 2018) requested a methodology to comprehensively assess animal disease surveillance systems and identify specific activities to improve veterinary services’ capacities to conduct surveillance. In response, the Food and Agriculture Organization of the United Nations (FAO) developed the Surveillance Evaluation Tool (SET) with funding from the United States Agency for International Development (USAID).

1.2. DEVELOPMENT OF SET

The first step in developing SET was to conduct a comprehensive review of available resources to assess whether existing tools or methodologies already existed to fit the countries’ needs. During this process, the OASIS tool (“Outil d’analyse des systèmes de surveillance”),[1] developed by the French Agency for Food, Environment and Occupational Health and Safety, was selected as the basis for SET.

Additional assessment criteria from FAO’s Epidemiology Mapping Tool were then incorporated into the following categories: cross-sectoral collaborations, epidemiology workforce capacities, outbreak investigation and risk assessment. Finally, the tool’s structure, scoring system (1 to 4) and graphical outputs were harmonized with FAO’s Laboratory Mapping Tool (LMT).

Two piloting sessions were conducted in the United Republic of Tanzania (12–21 June 2017) and Liberia (4–13 September 2017) to test SET in real-time situations in the East and West African contexts. Following these missions, the toolkit and evaluation methodology were updated to reflect feedback and lessons learned during each of those piloting missions. Lastly, the final version of SET was made available in English and French for implementation in the rest of GHSA Phase 1 countries in Africa in September 2017.

Following its widespread implementation in 22 countries up to December 2021, the tool was updated to improve its effectiveness. To this end, SET was subjected to an internal review by SET project leads at FAO headquarters, taking into account feedback received during the various country assessments. Edits, deletions and additions were proposed to the indicators, and scoring criteria were shared with various content experts and experienced SET users who attended two virtual meetings summarizing the key updates and were invited to participate in the tool’s review and update during 2021 and 2022. Due to the changes introduced in the updated version (herein after referred to as “SET version 2.0”), a straight comparison between results of SET 1.0 and SET 2.0 evaluations will not be possible, i.e. for countries that have used SET 1.0 for baseline evaluation and are re-evaluating their animal health surveillance system capacity using SET 2.0. In this case, enhancement can be demonstrated by showcasing progress in implementation of SET 1.0 recommendations against respective SET categories and areas, which remained largely the same.

1.3. OUTPUTS OF SET

The tool automatically generates two types of graphical outputs in the form of spider graphs that allow evaluators to understand the strengths and weaknesses of the surveillance system evaluated.

• Core results for the operation of the surveillance system: these are calculated through the scoring of 96 indicators grouped into 19 categories and 7 major areas (Table 1) describing various aspects of a surveillance system. The indicators are ranked in descending order in the back-end formulas in terms of their importance under three groups: core, direct support and indirect support indicators (Annex I). These groups were inspired by the core and support surveillance functions defined in the World Health Organization (WHO) communicable disease surveillance and response systems (World Health Organization, 2006). The rank reciprocal weighting method (Roszkowska, 2013) was used to convert the ordinal ranking of the three groups into weights as follows: 0.55 (core indicators), 0.27 (direct support indicators) and 0.18 (indirect support indicators). The core group includes the indicators covering case detection, case confirmation, reporting, data analysis and interpretation, and feedback; the direct support group includes the indicators covering standards and guidelines, training, supervision, and monitoring and evaluation; and the indirect support group includes the indicators covering coordination and communication aspects.

TABLE 1 Areas and categories evaluated by SET

Area

Category

Nº of indicators

Institutional organization

Central institutional organization

7

Field institutional organization

8

Intersectoral collaborations

5

Laboratory

Operational aspects

2

Technical aspects

10

Analytical aspects

3

Surveillance activities

Objectives and context of surveillance

3

Surveillance data collection

15

Surveillance methods (other than active surveillance)

10

Active surveillance

4

Risk assessment

2

Epidemiologic surveillance workforce

Workforce management

5

Training

4

Data management

Information system

2

Data processing and exploiting

4

Communications

Internal communication

4

External communication

4

Evaluation

Monitoring and evaluation

2

External evaluation

2

Total indicators

96

Source: SET toolkit version 2.0.

• Performance attributes of the surveillance system: these results are calculated using weighted coefficients assigned to the scores obtained for each indicator. They include representativeness, rapidity, flexibility, reliability, stability, acceptability, simplicity and utility. Annex II provides definitions, references and the relationship between these attributes as well as the indicators scored. Annexes II and III provide more information on these attributes.

1.4. LINKS WITH OTHER ASSESSMENT TOOLS AVAILABLE

When considering SET (either initial version, updated version 2.0 or further updates), it is important to place it in the context of other evaluation tools available to countries. Several international organizations have developed their own assessment methodologies, such as the Performance of Veterinary Services (PVS) from the World Organisation for Animal Health (WOAH) and the Joint External Evaluation (JEE) by WHO. While both of these tools provide important insights into disease surveillance, the purpose of each tool is to conduct a broader assessment of their target systems, and a limited number of indicators are dedicated to animal disease surveillance (Figure 1).

FIGURE 1
Assessment tools available to countries - ranging from general to specific

Source: Authors’ own elaboration.

With its 96 indicators, SET can be used concurrently with the PVS and JEE tools to corroborate their findings, leading to a deeper understanding of the strengths and weaknesses related to animal disease surveillance systems. In this context, decision-makers may wish to use SET individually or can turn to the toolkit following previous assessments to get deeper and more comprehensive results for further investigating issues related to their animal disease surveillance network at the central, intermediary and grassroots levels. This would result in targeted action plans conducive to tangible progress.

1.5. SET IMPLEMENTATION IN AFRICA AND ASIA

Between 2017 and 2024, 39 countries implemented SET in Africa, Asia and Latin America with support from various donors including USAID, the European Union, the United States Defense Threat Reduction Agency, Global Affairs Canada (GAC) and the FAO Regular Programme funding.

1.6. SET BIOTHREAT DETECTION MODULE

Throughout history, several countries and non-state actors[2] have used biological agents against animals. When these agrocrimes are implemented to coerce sociopolitical objectives, they are classified as agroterrrorism, a subset of bioterrorism specifically targeting livestock and crop production. Well-functioning surveillance systems are essential for effective preparedness and response against bioterrorism events. These systems can rapidly detect and contain diseases, identify perpetrators through case tracing and monitor the population to prevent accidental and deliberate disease outbreaks.

SET’s comprehensive assessment of all aspects related to animal disease surveillance generated interest in a project on agroterrorism and agrocrime funded by GAC. The project’s goal is to build resilience against agroterrrorism and agrocrime attacks on livestock that could have devastating social, economic and public health consequences. Launched in 2019, the project is being implemented by a consortium including FAO, WOAH and the International Criminal Police Organization (INTERPOL). Its initial phase assessed baseline capacities in the target regions of the project using SET, LMT, WOAH’s PVS and WHO’s JEE. Within this framework, FAO developed biothreat modules under SET and LMT. In combination, these tools informed the subsequent phases of the project to develop national preparedness and response capacities to agroterrrorism and agrocrime.

The SET Biothreat Detection Module (SET-BT) includes specific indicators related to the surveillance of potential terrorist and criminal animal health events, including but not limited to collaboration between law enforcement and veterinary services, mechanisms to suspect terrorist and criminal animal health events, and mechanisms for joint epidemiologic and criminal investigations, among others. FAO, WOAH and INTERPOL animal health and law enforcement experts developed the module based on literature review and expert consultations with 14 biothreat reduction experts with different technical and geographic backgrounds. The module was piloted in early 2021 and is now available for wider use during SET missions. Annex VI provides more information on the SET-BT module, how to use it and additional considerations when planning a mission that will use the SET-BT module.

1.7. OBJECTIVE OF THIS MANUAL

The objective of this manual is to provide SET evaluators with clear guidelines on how to implement the SET version 2.0 using a comprehensive and standardized methodology. This is essential to ensure that the quality of assessments in countries are comparable to one another.

[1] See Hendrikx et al. 2011.

[2] Non-state actors are individuals or entities not acting under the lawful authority of any state in conducting the described activities. Source: United Nations Security Council resolution 1540 (2004).

2. Implementation of SET

2.1. GENERAL PRINCIPLES

Several key principles guide the implementation of any SET mission:

• Participatory – Involvement of national focal points (NFPs) during all aspects of the mission is essential to ensure engagement by national veterinary services and to help identify realistic recommendations to improve a country’s animal disease surveillance system. Interviews with multidisciplinary stakeholders of surveillance are a key element of the evaluation. These are conducted in a participatory manner, using an informal approach that fosters dialogue with stakeholders and can be adapted to different contexts. Although no structured questionnaire exists to conduct interviews, a guide with sample questions is available to support this process.

• Objectivity – Outputs of the assessment should reflect the realities of the surveillance system in the country. The data-gathering phase of the SET mission and findings from interviews with multidisciplinary stakeholders at all levels of the system should therefore guide scoring and the development of recommendations. The evaluators’ can lead to specific biases based on their own experiences with surveillance and it is important to recognize and address these biases to ensure that outcomes of the evaluation and recommendations support the needs of the systems assessed. Remaining neutral and objective will also ensure that evaluations are repeatable and comparable regardless of the composition of the evaluation team.

• Evaluation for action – The ultimate purpose of SET evaluation is to produce relevant recommendations to guide the development of countries’ capacities for animal disease surveillance. As such, a large part of the SET approach involves following up the evaluation portion of the mission with the development of the action plan. Assessments that do not culminate in the development of a relevant action plan and final evaluation report should be considered incomplete.

2.2. EXTERNAL AND SELF-ASSESSMENTS

SET missions can be conducted as an external evaluation or an internal assessment. Each methodology comes with its own strengths and weaknesses, and it is up to the country to decide on the approach that best suits them (Table 2).

TABLE 2
Characteristics of external and internal evaluations

External evaluations

Internal evaluation

• Led by FAO staff from headquarters, regional or external country offices who have participated in multiple SET missions.

• Less flexibility on the mission’s programme or date, to accommodate external evaluators – missions have to be done following the SET programme template.

• External mission leader guides every aspect of the mission, including the preparatory phase and post-mission items.

• External assessor leads the development of the post-mission report, with support from members of the evaluation team from the country assessed.

• Led by in-country staff from FAO or the project financing the evaluation.

• In-country mission leaders must have received training in the use of the tool, and contact the SET focal point at FAO headquarters prior to planning to use the tool to ensure all updated materials and methodologies are used.

• More flexibility on the mission’s dates and programme – for example, more time can be taken to conduct field interviews, score the system or develop recommendations.

• Country evaluators manage all aspects of the evaluation, including the preparation of the mission, identification of stakeholders and development of the final report.

Source: Authors’ own elaboration.

2.3. COST OF MISSIONS

SET was originally developed with funding from USAID under FAO’s contribution to the GHSA. As such, implementation of the tool did not fall under regular FAO activities during the initial years of SET implementation and external funding was needed for FAO to support countries in using SET.

To facilitate budgeting SET’s implementation, the following matrix breaks down the cost of specific items needed for SET missions (Table 3). It is important, however, to adapt this table to the local context – some items may be added or removed depending on the local needs.

TABLE 3
Cost matrix for SET missions

Source: Authors’ own elaboration.

In general, total costs for external SET evaluations can range between USD 20 000 and USD 35 000, including the time needed for preparation and the in-country mission, as well as days required to develop the report. These costs can vary greatly depending on factors listed in Table 3, and on whether these were external or self-assessments, how many evaluators were involved and how many locations were visited in the country.

2.4. IMPLEMENTATION OF SET MISSIONS

The organization of SET evaluation can be divided into three phases: 1) Preparation, 2) In-country mission and 3) Post-mission follow-up items (Figure 2). Each phase is described in further detail in the manual’s next section.

FIGURE 2
Overview of the implementation of SET missions

Source: Authors’ own elaboration.

The preparatory phase for SET missions should start at the latest one month prior to the start of the in country mission and the first interviews – although an even earlier start can facilitate the process further.

3. Preparatory phase

Proper planning is the key to a successful SET evaluation, leading to recommendations that effectively guide future capacity development activities. As such, it is important to devote proper time and resources to this process.

During this time (at least one month prior to the mission), the following steps are important to help the organizers ensure a smooth assessment process.

3.1. INVITATION FROM MINISTRY

SET evaluations must be conducted upon the request of the ministry or delegated authority overseeing the animal disease surveillance system, usually the Ministry of Agriculture. As such, an invitation letter from the relevant authority is needed prior to the start of the mission.

There is no standard format for this invitation document, and in past missions a formal letter or email may be sufficient, depending on the local practices.

A draft concept note is available to provide decision-makers with background information on the process and the mission in the country (Table 4). More information is available on the SET web page (FAO, 2024).

TABLE 4 Documents in the SET packet

File name

Document use

0 – “How to use the SET documents”

Guide to using the documents of the SET packet.

1 – “SET concept note for the country”

Document sent to the veterinary services of the country four weeks before the beginning of the mission, so that they can understand SET missions.

2 – “Request for information” (Excel file)

This Excel-based file will facilitate the collection of information and documentation relevant to the surveillance system prior to the in-country mission. It should be filled out and returned by the country’s veterinary services at the latest one week prior to the mission. Note that there are multiple tabs to fill out.

3 – “Stakeholder checklist and interview sign-in sheet”

Guide to record all stakeholders interviewed during the mission. It can be adapted to the local situation. Pages 2 and 3 can be used as interview sign-in sheet.

4 – “Sample questions”

Examples of questions to ask during interviews with stakeholders. It can be used in the field for reference.

5 – “Interview feedback”

Questionnaire used to gather the comments and expectations of the persons interviewed about the mission (implementation, outputs, recommendations, etc.).

6 – “SET scoring grid” (Excel file)

Excel file used to 1) compile all the information gathered during the mission;

2) hold the scoring session, and 3) automatically calculate the outputs of the mission.

7 – “Glossary of terms used throughout SET evaluation”

Used to clarify definition of terms used in evaluation and support common understanding during the mission.

8 – “SET scoring guide”

Scoring guide with all SET indicators as a stand-alone document to use during the scoring session.

9 – “SWOT analysis worksheet”

Document to be used by team members to conduct SWOT analysis part of the evaluation. This can be done by the team at the end of each interview day and/or as a group after scoring.

10 – “SMART worksheet for SET recommendations”

Worksheet used to develop specific, measurable, attainable, relevant, time-bound (SMART) recommendations and organize them into the final format used for the SET report.

11 – “Evaluation team feedback”

Questionnaire used to gather the comments and opinions of the evaluation team members about the SET procedure at the end of the mission.

Source: Authors’ own elaboration.

3.2. IDENTIFYING EVALUATION TEAM MEMBERS AND THEIR RESPONSIBILITIES

The second step is to identify and finalize the list of members of the evaluation team(s). Evaluators should be familiar with the principles of animal disease surveillance and/or be veterinary epidemiologists. A team lead is designated to ensure the success of the mission – this person usually has the most experience with the toolkit.

The evaluation team should comprise a minimum of three members, including:

1. FAO animal health specialist(s), including

a. External assessor(s) (one or two evaluators if external mission)

b. Country FAO epidemiologist(s)/livestock officer(s) or regional FAO animal health and production officer, if available

2. NFP(s) from the veterinary services (ideally working in the central veterinary services with surveillance-related mandates)

Depending on the number of evaluators available, it is possible to divide the team into smaller groups in order to maximize the number of areas visited and the number of stake-holders interviewed.

All members of the team should actively participate in all aspects of the mission including preparation, implementation and follow-up. Specific responsibilities include:

• Country or regional FAO staff:

– Liaising with national government to provide information on the toolkit and evaluation methodology

• NFPs from veterinary services:

– Communicating in the language spoken by external evaluators and supporting translating interviews and technical documents for the evaluation team

– Flagging any requirements for meeting with decision-makers or stakeholders (e.g. customs, tea breaks, food, etc.)

– Providing a first proposal on stakeholders for interviews and areas to visit

– Liaising with stakeholders at the national and subnational level to coordinate interviews and logistics

– Following up and requesting the relevant documents that have not been shared during the interviews

– Requesting clearance of finalized SET report to the country’s chief veterinary officer (CVO) or equivalent position

•. In addition, it is essential that all team members:

– Review and provide feedback on mission programme, stakeholders to meet and areas to visit

– Collect and review all necessary technical documents before the mission (see section 3.4)

– Lead interviews with stakeholders

– Take notes during interviews

– Review relevant documents of the national surveillance system during the interviews and keep a list of which documents have been mentioned during discussion

– Summarize and share their findings at the end of each day

– Objectively score all indicators, providing justification from interviews

– Review findings from scoring and conduct a strengths, weaknesses, opportunities, threat (SWOT) analysis

– Develop recommendations

– Present to key decision-makers on SET methodology and findings

– Contribute to the development and finalization of the SET report

Additional team members are considered full members of the evaluation team and are welcome to participate in SET missions to receive training in using the tool.

Depending on the country context, it may be needed to include translators in the evaluation team to facilitate interaction with stakeholders, and review of documents provided to the team.

Once finalized, all evaluation team members should be available for teleconferences prior to the mission to provide their input. It is recommended to have weekly teleconferences with everyone at the latest one month prior to the start of the mission.

3.3. SHARING AND REVIEWING THE SET PACKAGE

A package with all relevant SET documentation (Table 4) is available from FAO in English, French and Spanish. This package can be accessed freely by those interested in conducting a SET evaluation through contacting the SET focal points at the Animal Health Service in FAO headquarters. These documents should be distributed to all members of the evaluation team as soon as they are identified so that they can familiarize themselves with the tool. The SET packet should be reviewed with all evaluators during a teleconference prior to the start of the mission to ensure all team members are familiar with each document.

3.4. REQUEST FOR INFORMATION AND REVIEWING DOCUMENTS OF THE SYSTEM

Documents relevant to animal disease surveillance in the country should be shared with the evaluation team for review prior to the start of the mission (at least two weeks prior to the mission). This will give the evaluators a general understanding of the country’s capacities and will greatly support the subsequent interviews during the mission. With support from NFPs or dedicated translators, documents should be provided in a language spoken by the evaluators, or translations should be made available. For larger documents, at least the table of contents can be translated, and evaluators, along with NFPs, can jointly identify the most relevant parts to review.

To guide this process, a request for information (RFI) is available in the SET packet. This Excel spreadsheet contains questions for the veterinary services to fill out and should be returned at the latest one week prior to the mission. It is important to note that the second tab of the RFI should be completed by all laboratories participating in surveillance.

In general, documents to review include (but are not limited to):

• Laws, policies and regulations regulating animal disease surveillance

• A list of priority animal diseases

• Surveillance plans for priority animal diseases

• Disease data, analyses and reports

• Protocols and standard operating procedures for all surveillance activities

• Organogram of the surveillance system

• Terms of reference for staff in the surveillance system

• Memorandums of understanding, letters of agreements for intersectoral cooperation

3.5. DEFINING MISSION DATES, STAKEHOLDERS TO INTERVIEW AND PROGRAMME

SET missions last about 10–12 days between arrival and departure of the external evaluators. Assessments conducted as self-evaluations may have more flexibility on the duration of the mission, but evaluators should still follow the methodology described in this document and be trained before leading their own mission.

The mission should not be planned over important holidays or official events in the country evaluated, or during significant events that may render stakeholders unavailable (e.g. ongoing disease outbreak, significant weather phenomena such as flooding, etc.).

The duration of the mission should balance the logistics of travelling to the field with the need to gather input from a representative sample of stakeholders.

Before finalizing the mission’s programme, it is therefore important to identify the major stakeholders of the surveillance system in the country. Table 5 details the guidelines for stakeholders to meet and the average interview times. It is important to note that these guidelines are not prescriptive, and the evaluation team should decide together on who to interview and how long to meet with them. Team members from the country’s veterinary services and FAO office know the local context best and they are expected to actively participate in this process.

TABLE 5 Guideline of major stakeholders to meet during SET missions

Central level

Estimated duration of interview

CVO

0.5–1 hour

Veterinary services’ epidemiology unit

2–3 hours

Central veterinary laboratory

1.5–2 hours

Regulatory, policy and legislation unit

1–1.5 hours

Ministry of Health – staff involved in surveillance of zoonotic diseases

1–1.5 hours

Ministry of Environment/Wildlife – staff involved in wildlife surveillance

1 hour

One Health platform/coalition

1 hour

Private laboratory (if relevant)

0.75–1 hour

2–3 representatives of livestock owners associations (can be done together)

1 hour

Subnational level

Estimated time of interview

4–5 district veterinary offices – staff involved in animal disease surveillance data collection and analysis

1–2 hours

2–3 abattoirs

1 hour

5–6 community animal health workers (or equivalent)

1 hour

3–4 subnational/district laboratories

1.5–2 hours

2–3 livestock traders/markets

0.75–1 hour

3–4 border post officers/veterinarians – who examine imported/exported animals

1 hour

At any level

Estimated time of interview

Up to 3 NGOs working with livestock, if relevant

0.75–1 hour

2–3 private veterinarians (can be done together)

0.75–1 hour

Animal research institutions, if relevant

1–1.5 hours

Veterinary faculties (e.g. departments of zoonoses and/or infectious diseases), if relevant

1.5–2 hours

Any other important stakeholder of animal disease surveillance in the country, at the discretion of the national veterinary services

Variable

Source: Authors’ own elaboration.

If the evaluation team contains sufficient people, it can be divided into several groups to conduct parallel interviews and visit more areas of the country. A member who is experienced in conducting SET evaluations should lead each subgroup. In past missions, between 40 and 200 individuals were interviewed and up to three teams of evaluators were used to gather data.

Once a complete list of stakeholders to interview is available to the evaluation team, a final mission programme can be drafted. There are some considerations to keep in mind as the team finalizes this programme:

1. It is important for the evaluators to get a balanced perspective of the surveillance capacities in the country. Therefore, stakeholders that represent both advanced and weak capacities should be selected for interviews, and an effort should be made to meet with people who are not always in contact with the veterinary services. As well, it may be useful to visit areas of high/low animal densities to compare capacities. This is critical to avoiding biased assessment results and recommendations that are ill-adapted to the realities of surveillance in the country.

2. The team should spend enough time at the central level (at minimum two days) to ensure all the relevant stakeholders are met and gain a broad insight of animal disease surveillance in the country prior to travelling to the field.

3. Travel times should be accounted for to reach each stakeholder for interview, including in the capital city.

4. Accommodation during the field portion of the mission must be identified ahead of time and should have access to internet.

5. Any security issues should be considered during the planning of the mission and the time for a United Nations security briefing should be accounted for in the final agenda, when necessary.

Table 6 shows a template SET programme for external evaluations, which can be adapted to the local context.

TABLE 6
Suggested template for SET mission programme (external evaluation)

Source: Authors’ own elaboration.

With the support from country FAO staff, the local NFPs have the responsibility to liaise with stakeholders and schedule meetings with them to conduct interviews. This should be done well in advance to confirm that the stakeholders will be available during the mission. Subsequent reminders should be given closer to the actual interview time.

3.6. PREPARATION CHECKLIST

To support the preparation of the mission, a checklist is available to the evaluation team in Annex V.

4. In-country mission

The in-country missions are the basis for objective evaluations, leading to outcomes that are useful to countries. It is therefore essential to use this time efficiently to collect, review and analyse the data needed to score the system properly.

Whether external or led by in-country teams, all SET evaluation missions must complete the following components.

4.1. TEAM BRIEFING

This informal meeting should be done as early as possible when all the members of the evaluation team are together. This is the opportunity to meet the rest of the team face to face, review the mission’s programme and stakeholders to meet, review team members’ roles/responsibilities, and address any last-minute items.

It is important to conduct the briefing prior to the launch of the mission.

4.2. LAUNCHING MEETING

The launching meeting represents the official launch of the mission with decision-makers, partners and technical persons. Participants to this meeting may vary depending on the country context and NFPs should guide the team during the preparatory phase regarding who should attend. In general, attendees represent high-level decision-makers and partners of the surveillance system, as well as representatives from the most important stakeholders. These may include:

• The CVO and/or high-level representatives from the ministry responsible for agriculture/livestock

• Veterinary services epidemiology unit

• Representatives from the veterinary laboratory network

• Representatives from the ministries responsible for public health and the environment/wildlife

• The private sector, including representatives from private veterinary practitioners and livestock associations

• Local NGOs involved in animal diseases

• International organizations and financial partners involved in animal disease surveillance

• Any other relevant decision-maker as identified by the local team members

The programme of the launching meeting may be adapted to the country’s preferences. A representative from the veterinary services (usually the head of the epidemiology unit) should prepare a presentation describing the national animal disease surveillance system and reporting system, including priority diseases and relevant legal mandates, perceived strengths and gaps and any other relevant information they feel the evaluators should know.

The SET team lead also presents the development of the tool and its methodology as well as the objective of the mission and areas to be visited. A template launching presentation is available to the evaluation team in the SET package provided during the preparatory phase.

Last, the launching meeting is an opportunity to review the proposed mission agenda with the attendees and make any last-minute changes as needed.

The launching meeting may last between 1 and 3 hours depending on the discussion with the group. Interviews should be initiated immediately after the mission launch. It is important to keep on time. If the FAO office of the country under evaluation is interested in posting communication about the SET mission (such as social media posts and newsletters), then the evaluation team would contribute to the formulation and revision of the communication material before posting it through the various communication channels.

4.3. INTERVIEW PROCESS

Proper interviews with all relevant stakeholders are essential to gather the evidence for scoring. It is important that all members of the evaluation team familiarize themselves with the indicators of SET to better lead the interview process and get all the information needed from the interviewees.

There are different methodologies to conduct interviews, each with their own advantages and gaps. A mix of both approaches can be used:

• Individual interviews – conducted face to face, where the evaluators meet the interviewee at their place of work. This is the opportunity to focus the conversation and get detailed information on the stakeholder’s role in surveillance. Relevant documents guiding their activities (e.g. protocols, standard operating procedures, communication documents) can also be reviewed at the same time. This process, however, can minimize the number of interviewees met as it adds significant travel time.

• Plenary meetings – bringing together many stakeholders to conduct a facilitated discussion on their role in surveillance. The plenary meetings should bring people from the same backgrounds to better focus the discussion (e.g. livestock owners and veterinary practitioners). Depending on the number of attendees, this may reduce the amount of time each interviewee can provide information, as attention should be shared among everyone present. It is also difficult to review relevant documents as they are discussed, since interviewees may not be in their place of work. In contrast, it does provide the team with access to a larger number of stakeholders to meet.

Prior to the start of interviews, participants should be made aware of the interviews’ purpose. This can take the form of a verbal briefing or a short presentation, but it should not run over the time allocated for the interview itself. The purpose is to ensure the stake-holders understand the reason for the interview, what the information they provide will be used for, and the expected outcomes of the assessment mission.

The interviews should be open-ended discussions where the interviewers guide the conversation to the topic of interest. Conversations may diverge on topics that are not always relevant to surveillance (e.g. vaccination programmes or livestock production activities) and the interviewers should work together to ensure that the discussion remains on topic.

There is no standardized questionnaire for each specific stakeholder interviewed, but a list of questions is available based on SET categories to guide the process. The team lead(s) may also use responses to the RFI to follow up on specific questions. In the document, a column exists for interviewer notes during the mission. It is very important that all interviewers are familiar with the areas evaluated by SET so that they can properly guide the interview process.

Some other aspects to consider during interviews:

• Interviewing similar stakeholders provides the opportunity to cross-reference the information given, as well as cover topics that were not discussed before due to lack of time for example.

• As all areas of a country are unlikely to be visited, it is important to ask interviewees whether conditions in the rest of the country resemble their own context. This ensures a more representative perspective of the country’s strengths and weaknesses.

• At the end of the day, evaluators should review their findings, identify information missing and prepare for the next day.

• As much as possible, information provided by stakeholders should be supported by tangible documentation. The evaluators should request to review these documents during the interviews and keep a list of which documents have been mentioned during discussion. The team should follow up and request the relevant documents that have not been shared during the interviews.

• Be mindful of conflicts of interest when interviewing a supervisor together with the staff they supervise. This may cause unease and bias in the responses of the supervisees. It may be prudent in some cases to interview these groups separately.

• Be mindful about the general political and cultural context in the country, particularly when it comes to openness about information sharing and transparency. In some countries, the acknowledgement of certain disease presence or even outbreaks is not obvious, and even unacceptable because it might be seen as a personal or collective failure.

Note-taking during the interview process is critical to keep track of all the information gathered. The quality scoring session depends on using the interviews as a reference for scoring each indicator. The evaluation team can decide on their strategy for taking notes–they may choose a specific note-taker for each interview, or each assessor may take and share their own notes. Notes may be taken on a notebook or laptop computer, but it is recommended that all notes eventually be digitized to make locating specific information easier.

Last, a sign-in sheet is available in the SET packet (Document 3 – “Stakeholder checklist and interview sign-in sheet” in Table 4). Each person interviewed should fill this form to keep a list of who provided feedback for the mission. This list should be transcribed into Excel for analytical purposes. A feedback form for interviewees is also available for them to complete after the meeting (Document 5 – “Interview feedback” in Table 4).

4.4. FIELD PORTION OF THE MISSION

To understand the realities of animal disease surveillance in a country and to reach stakeholders that may not be able to attend interviews at the central level (e.g. livestock owners or field veterinary personnel responsible for data collection), travelling to the field is essential.

Depending on the conditions, travel can be challenging due to road quality, security issues or geographic factors such as remoteness. It is important to identify these issues during the preparatory phase and to plan accordingly.

Accommodation must be identified ahead of time. If possible, internet access should be obtained to facilitate the evaluators’ work, especially if there are multiple teams conducting the evaluation throughout the country.

In some occasions, evaluators may decide to get a local SIM card for regular internet access and coordination with others. It is also recommended to create a messaging group with all the members of the evaluation team to facilitate communications.

It is important to end the days in the field at an appropriate hour to allow time in the evenings for the team to rest in the hotel, eat, review their notes and prepare for the next day. As a general guideline, evaluators should be back at their hotel around 18:00, although this can depend on local security rules. It is important to get and respect the country-specific instructions of the United Nations Department for Safety and Security regarding the allowed travelling time.

Last, it is recommended to plan for a day of rest prior to starting the scoring process. This is usually done on Day 7 of the mission (Table 6).

4.5. SCORING

While the field portion of the mission may be physically tiring, the scoring of the system using SET can also be a demanding activity, so it is important that the whole team be well rested and prepared prior to the start of scoring.

If done properly, scoring usually takes up to two days to thoroughly discuss each of the indicators objectively, decide on a score and enter an appropriate justification for each indicator.

Below are some indications to properly score the surveillance system using SET:

1. Who conducts scoring:

• All members of the evaluation team must participate in the scoring process, including the team members from the country’s veterinary services and FAO office.

• It is not recommended that stakeholders or other individuals external to the evaluation team attend the scoring session, as a large group participating may lengthen the process. On occasion, a content expert in a specific field (e.g. a laboratory expert) may be brought in to elucidate a specific question when the team is missing information.

2. Where scoring is done:

• A room with a projector or screen is needed so the whole team can follow the scoring process together. The room should be large enough to accommodate all members of the team comfortably.

• If possible, it is recommended to avoid conducting scoring in the offices of the veterinary services or FAO. This will reduce the chances of team members leaving the scoring session to attend to other business.

3. Before starting – establishing ground rules:

• Scoring can be a lengthy process, so the team should agree on ground rules to facilitate the activity.

• It is important to keep distractions such as phones and emails away during the scoring session – these will prevent team members from being engaged in the process and eventually lengthen the session. The team may decide to keep cell phones in a designated place in the room and only check them during breaks.

4. The scoring process:

• A printed copy of the SET scoring guide (Document 8 in Table 4) should be made available to all team members so that they can follow the process on their own.

• Each indicator is projected onto the screen and discussed as a group. After discussion, NFPs from the veterinary services propose a score.

• Each indicator should receive a full score based on the situation that fits the country best. In other words, there are no decimal scores. If a country finds itself in between two scores, the lower score should be given.

• Scores are incremental, meaning that all criteria from lower scores should be met before receiving a higher score. For example, even if it appears that a country meets the requirements for a score of 4, this score cannot be assigned unless the country also meets the requirements for scores 1 through 3.

• All scores must be justified in the appropriate column to ensure a transparent process.

• Enter the appropriate information in each designated column in the SET Excel file (Figure 3):

– Column K – Enter the score for each indicator. Only 1, 2, 3 or 4 should be entered here; do not enter 0 or text.

– Column L – Enter “N/A” here if the indicator is non-applicable for the specific surveillance system.

– Column M – Enter a detailed justification for the score that can allow reviewers to understand the reasoning behind the score given.

– Column N – Indicate difficulties in scoring this indicator, or recommendations to improve for the developers of SET.

– Columns G and I (hidden) – Enter the scores for subsequent evaluation missions to compare progress between assessments on the same graph.

5. Evidence-based scoring and avoiding bias.

• Scoring biases may be introduced based on team members’ affiliations or previous experiences in the country. It is important to recognize and avoid these biases during the scoring process. Scores that are over-or underestimated may lead to recommendations that are ill-adapted to the real situation in the country.

• Scores should be based on evidence and represent the realities seen during the mission.

• The team should use information from interviews and documents reviewed to support the decision. The justification behind a score must be entered in the corresponding column for each indicator.

• If the indicator requires the existence of formalized documents, they must be made available to the evaluation team for review. If the documents in question are mentioned to exist but are unavailable, then the criterion is not met.

FIGURE 3
Using the SET Excel file to enter scores, justifications and notes for each indicator

Source: SET toolkit version 2.0.

6. Efficiency, time management and teamwork

• The importance of scoring cannot be understated, as it provides the team with the final outputs of the surveillance system. Therefore, efficiency and teamwork are essential to properly conduct this activity.

• Although all team members should provide input for the scoring process, specific roles may be given to various team members to facilitate this process, for example:

– Someone skilled in facilitation should ensure that the discussion remains on topic and that everyone is engaged in the activity

– A note-taker should enter the score in the SET Excel spreadsheet, as well as the justifications from the discussion

• The team may decide to allocate a specific amount of time per indicator for the discussion and scoring, based on the total time available for the activity. This can help keep track of time and assess if the scoring session is on time, faster or slower than expected.

• Breaks should be used wisely during scoring to ensure that people remain engaged in the activity. It is also important not to spend too much time at lunch breaks to avoid being delayed (e.g. have lunch on site or nearby).

4.6. REVIEWING RESULTS

At the end of scoring, graphical outputs characterizing the core competencies of a surveillance system are produced automatically in Tab 4 of the SET Excel file (Figure 4). The graph shows the strengths and weaknesses of the system in each of the 19 categories and 7 areas of SET (Table 1) as a percentage compared with a “perfect” system (scores of 4 received for all indicators). A similar approach is used in the SET-BT module, which creates its own graphical outputs in a separate spreadsheet.

FIGURE 4
Graphical outputs generated by SET

Source: SET toolkit version 2.0.

Performance attributes are also calculated for the surveillance system (Figure 4), and a key to the indictors that are considered for these outputs is available in Tab 4 of the SET Excel file (colored cells in the range F7:U102). A weighing system is used for the calculation of the attributes. This system is described in Tab 4 of the SET Excel file, Tab 6 of the SET Excel file showing that indicators that carry more weight in the columns titled “Coeff”. Annexes I and II provide more detail on the definition of the performance attribute and the weighing system used for their calculations.

4.7. SWOT ANALYSIS

Once the outputs are calculated, they can be used to conduct a thorough SWOT analysis to better understand the results of the evaluation (Figure 5). The objective of this activity is to review the spider graph depicting the core competencies and explain the strengths and weaknesses observed, comparing them with the experience from the previous week of interviews. This is also the opportunity to validate the outputs and identify if there are any results that are surprising or unexpected for the reviewers – this may indicate the need to reassess some scores.

FIGURE 5
SWOT analysis definitions

Source: Authors’ own elaboration.

A worksheet is available to the evaluation team to conduct this activity (Document 9 in Table 4) and can be projected on a screen for everyone to follow together. The team then lists strengths and weaknesses under each of the seven areas listed in Table 1, based on the graphical results from SET and the experience from the interviews. Although not necessary, opportunities and threats may also be subdivided according to SET areas.

An initial working version of the SWOT table can be done as a group activity, and word-smithing may be conducted afterwards to directly incorporate the SWOT analysis into the closing presentation and final report. A similar approach is used to conduct SWOT analysis specific for the SET-BT module.

4.8. DEVELOPING THE ACTION PLAN

The findings from the SWOT analysis should directly lead to the development of an action plan with recommendations that are adapted to the country.

An initial step in the development of the action plan is to group together weaknesses that relate to one another. The idea is not only to address each weakness by an activity that can be incorporated into a larger overarching recommendation, but also not to ignore the strengths that should be maintained or improved. Ideally, it is better to regroup several recommended actions into primary recommendations as much as possible so as not to overwhelm veterinary services with recommendations. In past SET missions, an average of ten recommendations were identified. This is not prescriptive, however, and it is more important that the number of recommendations fits the needs of the assessment rather than sticks to a specific number.

To ensure that recommendations are useful to the country and not too general, the SMART approach is used during the development of the action plan (Table 7).

TABLE 7 Guidance to develop SMART recommendations

Specific

• What is the objective of the recommendation?

• What are the steps/milestones to implement this recommendation?

Measurable

• What are the results of this recommendation?

• What will tell us that this recommendation has been fully implemented?

Attainable

• What resources are needed to implement this recommendation? (E.g. financial but also technical and logistical resources).

• Which partners should be involved in implementing this recommendation? What will they provide?

• What are some challenges that will make this recommendation difficult to reach? How can these challenges be addressed?

Relevant

• Which weaknesses will this recommendation address? (Refer to SWOT analysis and SET spider graph).

• How will this recommendation contribute to improving SET indicators’ scores?

Time-bound

• When should the recommendation be completed?

• What about each milestone?

• What factors will prevent this due date from happening? (Refer to Threats section of SWOT analysis if needed and adapt due dates).

Source: Authors’ own elaboration.

A worksheet is available in the SET packet (Document 10 “SMART worksheet for SET recommendations”) to help the evaluation team develop SMART recommendations that can be incorporated into the closing presentations and SET report. The first page of the worksheet guides the team in the formulation of the SMART aspects of the recommendation, while the second page serves as the template of the final recommendation that can be directly copied/ pasted into the closing presentation or final SET report. The evaluation team can project the worksheet and fill it out together for each recommendation. Each final recommendation identifies a strategy for application, responsible parties, deliverables and a timeline for delivery.

The deliverables identified during the development of each recommendation will be used in the future to track the progress of the country along the action plan. Therefore, they should be designed in a way that makes tracking progress clear and efficient.

Once all recommendations have been identified and drafted, they must be numbered and ordered into three priority levels based on their feasibility and agreed timeline for completion. Each evaluation mission may define these due dates differently (i.e. SET evaluations in different countries may not have the same timelines for the three levels of priorities).

The finalized and agreed recommendations constitute the last action plan for the evaluation. Once again, the role of the NFPs from the country government is critical to identify the impact, feasibility, roles/responsibilities, strategies and due dates for each recommendation. The action plan is then presented to key decision-makers during the closing meeting of the mission for feedback and validation before finalizing it to incorporate into the evaluation report. A summary of the framework used to develop SET recommendations is available in Figure 6.

FIGURE 6
Framework used to develop SET recommendations

Source: Authors’ own elaboration.

4.9. CLOSING MEETING

Once the outputs of the evaluation have been generated, SWOT analysis conducted and action plan developed, the evaluation team drafts a presentation to report findings of the mission to the key decision-makers of animal disease surveillance in the country during a closing meeting.

Attendees of this meeting should reflect those present at the launching of the mission and can include:

• The CVO and/or high-level representatives from the Ministry responsible for agriculture/livestock

• Veterinary services’ epidemiology unit

• Representatives from the veterinary laboratory network

• Representatives from the ministries responsible for public health and the environment/wildlife

• The private sector, including representatives from private veterinary practitioners and livestock associations

• Local NGOs involved in animal diseases

• International organizations and financial partners involved in surveillance

• Any other relevant decision-maker identified by the local team members

A template for this presentation is available in the SET package provided to the evaluation team during the mission’s preparatory phase. The first part of the presentation summarizes the programme of the evaluation mission, areas visited and stakeholders met. Most of the presentation should focus on the findings from the evaluation, however, reviewing the SWOT and validating the recommendations.

At this point, it is likely that the attendees of the meeting have significant feedback on the missions’ findings or recommendations developed. This constructive criticism of the evaluation’s output is welcomed, and the attendees should be reminded that the results shown represent the first draft of the mission’s findings. Their input is needed to validate and finalize these results so that they can be incorporated into the final SET report. However, the team should be prepared to face criticism against some observed weaknesses that is not necessarily constructive or even relevant resulting from what might be perceived as bad employee performance.

4.10. TEAM DEBRIEFING

The last activity of the SET mission is the evaluators’ team debriefing. This can be done in an informal way after the closing meeting is over and should only involve those individuals who are members of the evaluation team – including the NFPs.

The goal of this meeting is to discuss what went well during the mission and what can be improved in the SET toolkit and methodology. A feedback form is available in the SET packet to capture comments from the rest of the team.

5. Post-mission

Although the in-country mission is over, significant work remains to finalize the findings of the mission and develop the SET report.

5.1. FEEDBACK TO THE PROJECT LEADS AND UPDATING SET

Comments from the team debriefing, including the feedback forms, should be shared with the developers of SET at FAO to improve the SET process and documents. The evaluation team may decide to conduct a teleconference with the developers of SET to further discuss their experiences in using the tool.

5.2. DRAFTING THE FINAL REPORT

Drafting the final report is an integral part of any SET evaluation, and as such, the process cannot be considered complete until the report is finalized.

All SET reports should follow FAO’s publication guidelines, available in multiple languages (FAO, 2017).

Specific templates for SET reports are available in the SET packet to ensure reports are standardized and to guide the evaluation team in the development of the documents. A specific template for preparing reports when using the SET-BT module is available in the SET-BT subpacket.

In general, the layout of SET reports includes the following sections:

1. Background on SET – This section remains the same for all SET reports and covers the inception and development of the tool and the assessment methodology. A paragraph related to the funding that led to the evaluation mission can be adapted to the specific context of the mission.

2. Livestock situation in the country – This provides the reader with some basic context on the local livestock situation in the country, as well as important animal disease events. A census of different animal species can be included here, as well as graphs and charts if needed. It is important to keep this section short (no more than two pages of text) to avoid diverting the focus of the manuscript from the evaluation.

3. Mission summary – A short summary of the mission should be entered, including the composition of the evaluation team, programme of the in-country mission, areas visited and the number/summary of stakeholders met. A full list of stakeholders met should not be listed here to keep the section short and may be added as an appendix later.

4. Evaluation results – This section is divided into two parts. The first part consists of a narrative that describes the findings of the mission based on the interviews. This can be very text heavy, and its headings should reflect SET’s categories (Table 1). The purpose of this is to provide the reader a background of what was seen during the mission so that they can understand the strengths and weaknesses in the graphical outputs generated by the tool. The second part of this section provides the reader with the spider graphs generated after scoring.

5. SWOT analysis and action plan – The SWOT analysis summarizes some of the strengths and weaknesses discussed previously and can be listed as bullet points for each SET area. They will lead into the action plan, which is the finalized collection of recommendations as developed above.

6. References – Any reference used in the report should be listed here, following the FAOStyle guidelines. Several references relating to the development of SET are common to all reports and should not be removed.

7. Appendices – Any additional information can be listed here, at the discretion of the evaluation team. This could include a summary of the findings which can be used as a stand-alone document for decision-makers, more details on the stakeholders interviewed or whatever else the team feels should be listed here.

A general framework to develop SET reports is described in Figure 7 and some considerations should be remembered during this process:

• Authorship and responsibilities of the evaluation team – All members of the evaluation team are considered authors of SET report. As such they should all be actively involved in the development of the manuscript. A lead writer can be designated to ensure that the writing style remains consistent throughout the report. Other team members can contribute to specific sections, e.g. evaluators more familiar with the country can draft the section on livestock background. Acknowledgements can be included at the beginning of the document for parties who contributed significantly to the development of SET, in-country mission and/or validation of the findings.

• Clearance for publication – Once the team finalizes the report, they should send it to the CVO for validation and authorization to be published online. If the CVO has some specific requests or edits, they should be accommodated so long as the report still reflects the reality of the evaluation. The CVO should provide written (letter/email) confirmation that the report is validated and can be posted/shared online. In some instances, a country may choose to keep the final report confidential and for their own internal use. This can be done but it is not recommended, because if partners cannot access the report, this will significantly reduce the chances of the action plan being implemented. In case the SET-BT module is used, additional written confirmation to publish the report will be needed from relevant or leading law enforcement authorities. The details of this process are listed in point 4 of annex IV. Drafting a SET-BT report in Annex IV on preparing missions using the SET Biothreat Detection Module.

• Timeline to finalize – The due date to deliver the report should be communicated to the decision-makers during the launching and closing presentations. It is understandable that shifting priorities may affect the completion of the manuscript but SET evaluations cannot be considered finished until a final and approved report has been produced. In previous missions, a first draft of the report is usually produced within 60 days of the mission’s end, constituting about 35 days of work. Clearance to validate and publish the report thereafter may depend on the country’s internal protocols and may take longer.

FIGURE 7
Steps in drafting and publishing SET reports

Source: Authors’ own elaboration.

Once the report is finalized and approved for sharing, it can be submitted through FAO’s Publications Workflow System to be posted online. At this point, minor formatting changes may occur based on requirements from the Office of Corporate Communications. Last, all published reports should be posted on the SET project web page (FAO, 2024).

5.3. IMPLEMENTING THE ACTION PLAN

A finalized SET report is not considered to be a to-do list for any single organization such as the veterinary services or FAO. Rather, it is a guide for all partners involved in building surveillance capacities.

The veterinary services should integrate findings from SET into their strategic plans as much as possible, and FAO will also support the implementation of the action plan where possible. However, the report should be distributed to different partners, including local and international organizations, to guide the implementation of activities in a cohesive manner.

The publication of the SET report online is therefore essential to leverage partnerships and increase the chance that a SET action plan is implemented.

5.4. FOLLOWING UP ON PROGRESS

The idea behind SET evaluations is to conduct “evaluations for action”. As such, the idea of keeping track of the implementation of the action plans and providing support as needed is essential.

Regular meetings with stakeholders of animal disease surveillance in the country should be conducted to review the SET action plan, identify challenges and successes, and amend the action plan if necessary. The goal is to ensure that relevant activities are occurring on a regular basis in between SET evaluations.

Last, it is recommended that a follow-up SET evaluation occur every three to five years.

6. Resources

Centers for Disease Control and Prevention. 2001. Updated guidelines for evaluating public health surveillance systems – Recommendations from the guidelines working group. Morbidity and Mortality Weekly Report, 50(RR12): 1–35. https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5013a1.htm

Centers for Disease Control and Prevention. 2004. Framework for evaluating public health surveillance systems for early detection of outbreaks. Morbidity and Mortality Weekly Report, 53(RR05): 1–11. https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5305a1.htm

Health Canada. 2004. Framework and Tools for Evaluating Health Surveillance Systems. Ottawa. http://publications.gc.ca/collections/Collection/H39-4-46-2004E.pdf

World Health Organization. 1997. Protocol for the Evaluation of Epidemiological Surveillance Systems. Geneva. http://apps.who.int/iris/handle/10665/63639

REFERENCES

FAO. 2017. Publishing at FAO: Strategy and guidance. Office for Corporate Communication – Publications Branch (OCCP). https://www.fao.org/3/i7429e/i7429e.pdf

FAO. 2024. Animal health. In: Surveillance Evaluation Tool (SET). Rome, FAO. [Cited 11 October 2024]. https://www.fao.org/animal-health/resources/tools/set

Hendrikx et al. 2011. OASIS: an assessment tool of epidemiological surveillance systems in animal health and food safety. Epidemiology and Infection, 139(10): 1486–1496.

Roszkowska, E. 2013. Rank ordering criteria weighting methods: A comparative overview. Optimum. Studia Ekonomiczne, 5(65): 14–33. https://pdfs.semanticscholar.org/f983/e8c4eb7d7c30694dd72c5849dd6fee8a5c79.pdf

United States Agency for International Development. 2018. Global Health Security Agenda (GHSA). https://2017-2020.usaid.gov/sites/default/files/documents/1860/Global_Health_Security_Agenda_-_May_18.pdf

World Health Organization. 2006. Communicable disease surveillance and response systems: guide to monitoring and evaluating. Lyon. https://iris.who.int/handle/10665/69331

Annexes

Annex I
Tables listing the SET 2.0 core, direct support and indirect support indicators

TABLE A.I.1
Case detection (Core)

26

Percent of the priority diseases that can be tested within the laboratory network

31

Availability of diagnostic investigation personnel to support field investigation agents

51

Legal authority to conduct surveillance

54

Surveillance at borders and points of entry

55

Mechanisms for event-based surveillance of potential animal health threats

56

Existence of passive surveillance whose results are representative

57

Enhanced surveillance during emergencies/outbreaks

58

Existence of community-based or participatory surveillance approach in the system

59

Existence of operational and formal syndromic surveillance approach

61

Surveillance of priority diseases in susceptible wild animals

62

Vector surveillance and control

63

Implementation of epidemiological animal health investigations

67

Completeness of active (planned) surveillance

71

Number of staff available to conduct core, routine and emergency surveillance activities

TABLE A.I.2
Case confirmation (Core)

22

Adequacy of human, material and financial resources for diagnostic needs within the national animal health laboratory network conforming to the needs of the system

23

Application of quality control for the tests undertaken

27

Relevance of collected samples for diagnostics of priority diseases

28

Standardization of collected samples

29

Quality of samples collected

30

Percent of accredited animal health laboratories

32

Relevance of diagnostic techniques

TABLE A.I.3
Reporting (Core)

14

Representativeness of the field agents’ coverage of lowest administrative units covered by surveillance

15

Adequacy of infrastructure, material and financial resources at the field level only

40

Timeliness of reporting by field agents

42

Animal health reporting tools

44

Relevance of data collection tools (excluding laboratory tools)

46

Relevance and use of the standard criteria for classifying whether a unit of interest (animal, herd or flock, etc.) is suspected, probable or confirmed to have the priority diseases (relevance and use of the case definitions)

47

Simplicity of the case or threat definition

48

Quality of the completion of investigation forms

50

Simplicity of the notification procedure from the field to the central level

TABLE A.I.4
Data analysis and interpretation (Core)

1

Existence of an operational/functional central surveillance unit

7

Adequacy of the necessary infrastructure, material and financial resources for central surveillance unit only

33

Technical level of data management at the laboratory

73

Minimum qualifications of staff conducting epidemiological analyses

74

Adequate skill level in epidemiology of members of the central unit

79

Adequacy of the data management system for the needs of the system (relational database, etc.)

81

Designated staff available and trained in data entry, management and analysis

83

Complete descriptive processing of data

84

Analysis of data fits the needs of the system

TABLE A.I.5
Feedback (Core)

34

Analysis deadlines at the laboratory between sample analysis and reporting of results to the central unit (formalization, standardization, verification and transfer of results to the central epidemiological unit)

35

Quality of laboratory reports delivering results

87

Reporting of individual test results to field actors and data collectors

88

Systematic distribution/feedback of analyses of the surveillance to field actors (outside of a newsletter)

TABLE A.I.6
Surveillance design (Core)

36

Relevance of surveillance objectives

37

Level of detail, accuracy and formalization of objectives

38

Consistency of the priority diseases under surveillance with the sanitary situation (existing/exotic diseases or threats)

39

Existence of a formalized surveillance plan for each disease or threat under surveillance

45

Existence of standard criteria for classifying whether a unit of interest (animal, herd or flock, etc.) is suspected, probable or confirmed to have the priority diseases (case definition of priority diseases)

64

Relevance and suitability of active (planned) surveillance protocols

65

Representativeness of the populations under active (planned) surveillance (selection bias)

66

Precision of sample under active (planned) surveillance (sample size)

68

Implementation of animal health risk assessment

69

Usefulness of risk assessment activities for informing surveillance priorities

TABLE A.I.7
Standards and guidelines (direct support)

4

Organization and operations of the surveillance system laid down in regulations, a formal agreement or a convention established between the partners

24

Level of the standardization of work between laboratories of the national animal health laboratory network

25

Proportion of tests for priority diseases submitted to interlaboratory proficiency testing (samples sent for confirmation testing are not included)

41

Defined intervals between the detection of a case or threat and the delivery of results

43

Standardization of data collected from the field

49

Defined intervals between the detection of a case or threat and reporting

70

Terms of reference (TORs)

72

Human resources planning

80

Data input interval in accordance with the objectives and use of system results

82

Data verification and validation procedures formalized and operational

TABLE A.I.8
Training (direct support)

75

Initial training implemented for all field agents when joining the system

76

Objectives and contents of initial training of system field actors adequate for operational surveillance needs

77

Regular refresher training specific to the surveillance system

78

Adequacy of material and financial resources for training

TABLE A.I.9
Supervision (direct support)

6

Supervision of intermediary units by the central level

8

Existence of formal intermediary units covering the entire territory/country

9

Active role of intermediary units in the function of the system (data validation, management, feedback)

10

Implementation of supervision by the intermediary level

12

Adequacy of infrastructure, material and financial resources for intermediary units only (field will be addressed later on)

TABLE A.I.10
Monitoring and evaluation (direct support)

3

Existence of a surveillance system technical committee

93

Monitoring and performance indicators developed and validated by the directors of the surveillance system

94

Performance indicators regularly measured, interpreted and disseminated

95

External evaluations carried out

96

Implementation of corrective measures

TABLE A.I.11
Coordination (indirect support)

2

Existence of a surveillance system steering committee that is representative of the partners

5

Frequency of meetings of the central coordinating body

11

Harmonization of intermediary units’ activities

13

Existence of coordination meetings at the intermediary level

16

Coordination with private sector

17

collaborations with academia and research institutions, and other relevant public agencies (i.e. food safety authority)

18

Coordination with Public Health Sector

19

Coordination with Environmental Health Sector

20

Formal coordination mechanism is established on priority zoonotic diseases between Ministry of Health and Ministry of Agriculture (or equivalent), Ministry of Environment, WHO, FAO, WOAH, etc.

21

Effective integration of laboratories in the surveillance system

TABLE A.I.12
Communication (indirect support)

52

Acceptability of the consequences of a suspicion or case for the source (i.e. farmer or livestock owner)

53

Acceptability of the consequences of a suspicion or case for the collector of data (i.e. field agent)

60

Existence of awareness building and/or incentive programmes for data sources (farmers) in a passive (event-based) surveillance system

85

Regular release of surveillance reports to international organizations such as WOAH

86

Regular release of surveillance reports to decision-makers and surveillance system actors at the central level

89

Presence of a communications system organized horizontally and vertically between field actors (mail, web, telephone…)

90

Regular dissemination of a relevant public information newsletter

91

Solid communication policy with decision-makers and other external partners (exclude mandatory reporting to AU-IBAR or WOAH – this is evaluated elsewhere)

92

Adequacy of material and financial resources for communication

Annex II
Performance attributes definition and resources

Attribute

Definition

Representativeness

Ability of the system to accurately and timely describe the epidemiological situation of the diseases/threats covered by the system

Rapidity/timeliness

The interval between the occurrence of an adverse health event and (i) the report of the event to the appropriate agency, (ii) the identification by that agency of trends or outbreaks, and/or (iii) the implementation of control measures

Flexibility

The ability of the surveillance system to be easily adapted to new notifiable diseases, new reporting needs in response to changes in the nature or importance of the health event, the population monitored, or the resources available

Data quality (reliability)

Reflection of the completeness and validity of the data recorded in the surveillance system

Stability

The surveillance system’s ability to be operational when it is needed

Acceptability

Assessed by the willingness of persons conducting surveillance and those providing data to generate accurate, consistent and timely data

Simplicity

Refers to both its structure and ease of operation. Surveillance systems should be as simple as possible while still meeting their objectives

Utility/usefulness

The usefulness of a surveillance system is assessed by whether it leads to prevention or control or a better understanding of health events

Sources: CDC, 2001; CDC, 2004; Health Canada, 2004; WHO, 1997

Annex III
Calculation of performance attribute scores

Indicators taken into consideration for the calculation of the attributes

Weight

Representativeness

8.Existence of formal intermediary units covering the entire territory/country

3

9.Active role of intermediary units in the function of the system (data validation, management, feedback)

3

12.Adequacy of infrastructure, material and financial resources for intermediary units only (field will be addressed later on)

2

14.Representativeness of the field agents’ coverage of lowest administrative units covered by surveillance

3

15.Adequacy of infrastructure, material and financial resources at the field level only

3

29.Quality of samples collected

2

32.Relevance of diagnostic techniques

2

36.Relevance of surveillance objectives

2

39.Existence of a formalized surveillance plan for each disease or threat under surveillance

2

54.Surveillance at borders and points of entry

2

55.Mechanisms for event-based surveillance of potential animal health threats

3

56.Existence of passive surveillance whose results are representative

2

57.Enhanced surveillance during emergencies/outbreaks

2

58.Existence of community-based or participatory surveillance approach in the system

2

59.Existence of operational and formal syndromic surveillance approach

2

60.Existence of awareness building and/or incentive programmes for data sources in a passive surveillance system

2

61.Surveillance of priority diseases in susceptible wild animals

2

63.Implementation of epidemiological animal health investigations

3

64.Relevance and suitability of active (planned) surveillance protocols

2

65.Representativeness of the targeted populations under active (planned) surveillance (selection bias)

2

66.Precision of sample under active (planned) surveillance (sample size)

2

67.Completeness of active (planned) surveillance

2

Rapidity

5.Frequency of meetings of the central coordinating body

1

12.Adequacy of infrastructure, material and financial resources for intermediary units only (field will be addressed later on)

3

13.Existence of coordination meetings at the intermediary level

2

14.Representativeness of the field agents’ coverage of lowest administrative units covered by surveillance

3

15.Adequacy of infrastructure, material and financial resources at the field level only

3

21.Effective integration of laboratories in the surveillance system

3

22.Adequacy of human, material and financial resources for diagnostic needs within the national animal health laboratory network conforming to the needs of the system

3

26.Percent of the priority diseases that can be tested within the laboratory network

3

31.Availability of a diagnostic investigation personnel to support field investigation agents

2

34.Analysis deadlines at the laboratory between sample analysis and reporting of results to the central unit (formalization, standardization, verification, transfer of results to the central epidemiological unit)

3

40.Timeliness of reporting by field agents (reporting to intermediary and/or central levels)

3

41.Defined intervals between the detection of a case or threat and the delivery of results

1

42.Animal health reporting tools

2

44.Relevance of data collection tools (excluding laboratory tools)

1

49.Defined intervals between the detection of a case or threat and reporting

1

50.Simplicity of the notification procedure from the field to the central level

2

51.Legal authority to conduct surveillance

3

52.Acceptability of the consequences of a suspicion or case for the source (i.e. farmer or livestock owner)

2

53.Acceptability of the consequences of a suspicion or case for the collector of data (i.e. field agent)

2

72.Human resources planning

1

80.Data input interval in accordance with the objectives and use of system results

2

81.Designated staff available and trained in data entry, management and analysis

2

82.Data verification and validation procedures formalized and operational

2

83.Complete descriptive processing of data

3

84.Analysis of data fits the needs of the system

2

87.Reporting of individual test results to field actors and data collectors

1

88.Systematic distribution/feedback of analyses of the surveillance to field actors (outside of a newsletter)

1

Flexibility

1.Existence of an operational/functional central surveillance unit

3

2.Existence of a surveillance system steering committee that is representative of the partners

3

3.Existence of a surveillance system technical committee

3

13.Existence of coordination meetings at the intermediary level

2

14.Representativeness of the field agents’ coverage of lowest administrative units covered by surveillance

2

15.Adequacy of infrastructure, material and financial resources at the field level only

3

16.Coordination with private sector

2

17.collaborations with academia and research institutions, and other relevant public agencies (i.e. food safety authority)

2

18.Coordination with public health sector

2

19.Coordination with environmental health sector

2

20.Formal coordination mechanism is established on priority zoonotic diseases, between Ministry of Health and Ministry of Agriculture (or equivalent), and Ministry of Environment, WHO, FAO, WOAH, etc.

3

22.Adequacy of human, material and financial resources for diagnostic needs within the national animal health laboratory network conforming to the needs of the system

3

30.Percent of accredited animal health laboratories

1

31.Availability of a diagnostic investigation personnel to support field investigation agents

2

32.Relevance of diagnostic techniques

1

33.Technical level of data management at the laboratory

1

42.Animal health reporting tools

3

44.Relevance of data collection tools (excluding laboratory tools)

2

45.Existence of standard criteria for classifying whether a unit of interest (animal, herd or flock etc.) is suspected, probable or confirmed to have the priority diseases (case definition of priority diseases)

3

46.Relevance and use of the standard criteria for classifying whether a unit of interest (animal, herd or flock etc.) is suspected, probable or confirmed to have the priority diseases (relevance and use of the case definitions)

3

47.Simplicity of the case or threat definition

1

50.Simplicity of the notification procedure from the field to the central level

3

51.Legal authority to conduct surveillance

3

52.Acceptability of the consequences of a suspicion or case for the source (i.e. farmer or livestock owner)

2

53.Acceptability of the consequences of a suspicion or case for the collector of data (i.e. field agent)

2

55.Mechanisms for event-based surveillance of potential animal health threats

2

56.Existence of passive surveillance whose results are representative

3

57.Enhanced surveillance during emergencies/outbreaks

3

58.Existence of community-based or participatory surveillance approach in the system

2

59.Existence of operational and formal syndromic surveillance approach

2

60.Existence of awareness building and/or incentive programs for data sources in a passive surveillance system

3

63.Implementation of epidemiological animal health investigations

3

71.Number of staff available to conduct core, routine and emergency surveillance activities

3

72.Human resources planning

1

74.Adequate skill level in epidemiology of members of the central unit

2

77.Regular refresher training specific to the surveillance system

2

78.Adequacy of material and financial resources for training

3

93.Monitoring and performance indicators developed and validated by the directors of the surveillance system

2

94.Performance indicators regularly measured, interpreted and disseminated

2

96.Implementation of corrective measures

3

Reliability

6.Supervision of intermediary units by the central level

2

8.Existence of formal intermediary units covering the entire territory/country

2

9.Active role of intermediary units in the function of the system (data validation, management, feedback)

2

10.Implementation of supervision by the intermediary level

2

11.Harmonization of intermediary units’ activities

2

12.Adequacy of infrastructure, material and financial resources for intermediary units only (field will be addressed later on)

2

15.Adequacy of infrastructure, material and financial resources at the field level only

2

23.Application of quality control for the tests undertaken

3

24.Level of the standardization of work between laboratories of the national animal health laboratory network

3

25.Proportion of tests for priority diseases submitted to interlaboratory proficiency testing (samples sent for confirmation testing are not included)

3

27.Relevance of collected samples for diagnostics of priority diseases

3

28.Standardization of collected samples

3

29.Quality of samples collected

2

30.Percent of accredited animal health laboratories

2

32.Relevance of diagnostic techniques

3

33.Technical level of data management at the laboratory

3

35.Quality of laboratory reports delivering results

3

36.Relevance of surveillance objectives

3

37.Level of detail, accuracy and formalization of objectives

3

38.Consistency of the priority diseases under surveillance with the sanitary situation (existing/exotic diseases or threats)

3

39.Existence of a formalized surveillance plan for each disease or threat under surveillance

3

43.Standardization of data collected from the field

3

65.Representativeness of the targeted populations under active (planned) surveillance (selection bias)

3

66.Precision of sample under active (planned) surveillance (sample size)

3

67.Completeness of active (planned) surveillance

3

68.Implementation of animal health risk assessment

3

69.Usefulness of risk assessment activities for informing surveillance priorities

3

73.Minimum qualifications of staff conducting epidemiological analyses

2

74.Adequate skill level in epidemiology of members of the central unit

3

79.Adequacy of the data management system for the needs of the system (relational database, etc.)

3

80.Data input interval in accordance with the objectives and use of system results

1

81.Designated staff available and trained in data entry, management and analysis

2

82.Data verification and validation procedures formalized and operational

3

83.Complete descriptive processing of data

3

84.Analysis of data fits the needs of the system

3

93.Monitoring and performance indicators developed and validated by the directors of the surveillance system

2

94.Performance indicators regularly measured, interpreted and disseminated

2

96.Implementation of corrective measures

2

Stability

1.Existence of an operational/functional central surveillance unit

3

2.Existence of a surveillance system steering committee that is representative of the partners

3

3.Existence of a surveillance system technical committee

3

4.Organization and operations of the surveillance system laid down in regulations, a formal agreement or a convention established between the partners

3

5.Frequency of meetings of the central coordinating body

2

6.Supervision of intermediary units by the central level

3

7.Adequacy of the necessary infrastructure, material and financial resources for central surveillance unit only

1

8.Existence of formal intermediary units covering the entire territory/country

2

9.Active role of intermediary units in the function of the system (data validation, management, feedback)

2

10.Implementation of supervision by the intermediary level

1

11.Harmonization of intermediary units’ activities

1

12.Adequacy of infrastructure, material and financial resources for intermediary units only (field will be addressed later on)

3

13.Existence of coordination meetings at the intermediary level

2

14.Representativeness of the field agents’ coverage of lowest administrative units covered by surveillance

3

15.Adequacy of infrastructure, material and financial resources at the field level only

3

21.Effective integration of laboratories in the surveillance system

2

22.Adequacy of human, material, and financial resources for diagnostic needs within the national animal health laboratory network conforming to the needs of the system

3

26.Percent of the priority diseases that can be tested within the laboratory network

2

31.Availability of a diagnostic investigation personnel to support field investigation agents

3

42.Animal health reporting tools

3

50.Simplicity of the notification procedure from the field to the central level

2

51.Legal authority to conduct surveillance

3

52.Acceptability of the consequences of a suspicion or case for the source (i.e. farmer or livestock owner)

2

53.Acceptability of the consequences of a suspicion or case for the collector of data (i.e. field agent)

2

71.Number of staff available to conduct core, routine and emergency surveillance activities

3

72.Human resources planning

3

73.Minimum qualifications of staff conducting epidemiological analyses

1

74.Adequate skill level in epidemiology of members of the central unit

1

75.Initial training implemented for all field agents when joining the system

2

76.Objectives and contents of initial training of system field actors adequate for operational surveillance needs

2

77.Regular refresher training specific to the surveillance system

2

78.Adequacy of material and financial resources for training

3

79.Adequacy of the data management system for the needs of the system (relational database, etc.)

3

81.Designated staff available and trained in data entry, management and analysis

3

93.Monitoring and performance indicators developed and validated by the directors of the surveillance system

1

94.Performance indicators regularly measured, interpreted and disseminated

1

96.Implementation of corrective measures

2

Acceptability

2.Existence of a surveillance system steering committee that is representative of the partners

2

3.Existence of a surveillance system technical committee

2

4.Organization and operations of the surveillance system laid down in regulations, a formal agreement or a convention established between the partners

2

5.Frequency of meetings of the central coordinating body

2

11.Harmonization of intermediary units’ activities

2

16.Coordination with private sector

3

17.Collaborations with academia and research institutions, and other relevant public agencies (i.e., food safety authority)

3

18.Coordination with public health sector

3

19.Coordination with environmental health sector

3

20.Formal coordination mechanism is established on priority zoonotic diseases, between Ministry of Health and Ministry of Agriculture (or equivalent), and Ministry of Environment, WHO, FAO, WOAH, etc.

3

21.Effective integration of laboratories in the surveillance system

3

51.Legal authority to conduct surveillance

2

52.Acceptability of the consequences of a suspicion or case for the source (i.e. farmer or livestock owner)

3

53.Acceptability of the consequences of a suspicion or case for the collector of data (i.e. field agent)

3

60.Existence of awareness building and/or incentive programmes for data sources in a passive surveillance system

2

75.Initial training implemented for all field agents when joining the system

2

87.Reporting of individual test results to field actors and data collectors

2

Simplicity

42.Animal health reporting tools

3

43.Standardization of data collected from the field

3

44.Relevance of data collection tools (excluding laboratory tools)

3

47.Simplicity of the case or threat definition

2

Utility

26.Percent of the priority diseases that can be tested within the laboratory network

2

27.Relevance of collected samples for diagnostics of priority diseases

2

29.Quality of samples collected

2

31.Availability of a diagnostic investigation personnel to support field investigation agents

2

32.Relevance of diagnostic techniques

3

33.Technical level of data management at the laboratory

3

34.Analysis deadlines at the laboratory between sample analysis and reporting of results to the central unit (formalization, standardization, verification, transfer of results to the central epidemiological unit)

2

35.Quality of laboratory reports delivering results

3

36.Relevance of surveillance objectives

3

37.Level of detail, accuracy and formalization of objectives

3

38.Consistency of the priority diseases under surveillance with the sanitary situation (existing/exotic diseases or threats)

3

39.Existence of a formalized surveillance plan for each disease or threat under surveillance

3

44.Relevance of data collection tools (excluding laboratory tools)

3

48.Quality of the completion of investigation forms

3

64.Relevance and suitability of active (planned) surveillance protocols

3

68.Implementation of animal health risk assessment

3

69.Usefulness of risk assessment activities for informing surveillance priorities

3

79.Adequacy of the data management system for the needs of the system (relational database, etc.)

3

83.Complete descriptive processing of data

3

84.Analysis of data fits the needs of the system

3

86.Regular release of surveillance reports to decision-makers and surveillance system actors at the central level

2

88.Systematic distribution/feedback of analyses of the surveillance to field actors (outside of a newsletter)

2

89.Presence of a communications system organized horizontally and vertically between field actors (mail, web, telephone, etc.)

3

91.Solid communication policy with decision-makers and other external partners (exclude mandatory reporting to AU-IBAR or WOAH – this is evaluated elsewhere)

3

93.Monitoring and performance indicators developed and validated by the directors of the surveillance system

2

96.Implementation of corrective measures

3

Annex IV
Preparing a mission using SET-BT Biothreat Detection Module

1. GENERAL

1.1. Context

Attacks using animal pathogens can have devastating socioeconomic, public health and national security consequences. The livestock sector has some inherent vulnerabilities which put it at risk for deliberate or accidental spread of disease. Growing concerns of countries about the risks of agroterrorism and agrocrime has led to efforts to prepare against potential attacks. In 2019 the project “Building resilience against agro-terrorism and agro-crime affecting animal health” (World Organisation for Animal Health, 2020) was launched, with financial support from GAC, and implemented by a consortium between WOAH, FAO and the International Criminal Police Organization (INTERPOL). Given the importance of strong surveillance systems in the preparedness and response to agroterrorism and agrocrime, the WOAH-FAO-INTERPOL consortium decided to use SET to obtain a detailed understanding of the beneficiary countries’ animal disease surveillance systems. However, the surveillance of potential deliberate animal disease outbreaks, particularly the investigation of these events, requires additional coordination and activities beyond that of routine surveillance, such as forensic investigation, forensic sampling and testing, proper chain-of-custody processing, among others (World Organisation for Animal Health, 2015). Therefore, a specific Biothreat (BT) Detection Module was developed to be used within SET to assess the capacity of countries to detect unusual animal health events that are indicative of agroterrorism or agrocrime.

1.2. Development of the SET-BT

The SET-BT module was developed by animal health and law enforcement experts from FAO, WOAH and INTERPOL with a background in epidemiological surveillance, animal health emergencies, veterinary diagnostics and biothreat reduction. An initial draft was developed between March and May 2020 based on an extensive literature review of more than 50 documents including international and national strategies and guidelines, workshop reports, peer-reviewed articles, legislation, international conventions and more. The draft was then reviewed by 14 biothreat reduction experts with different technical and geographic backgrounds between July and September 2020. Finally, the module was piloted in November 2021 in Tunisia and May 2022 in Jordan.

1.3. Outputs of SET-BT

The Biothreat Detection Module consists of a list 32 of indicators divided into 7 categories related to the surveillance of potential deliberate animal disease outbreaks, including their investigation. Like SET, the indicators are scored from 1 to 4 based on the country’s capacity. Once all indicators are scored, the module automatically generates a spider graph of results by category.

FIGURE A.IV.1
Biothreat detection module categories (fictive country)

Source: SET biothreat module.

2. PREPARATION

2.1. Timeline and general considerations

Regular SET missions are planned at least 1 month in advance. However, SET-BT missions require additional approvals and clearances from relevant ministries governing national security forces, such as Ministry of Interior, Ministry of Defense and Ministry of Finances (for customs authorities). These approvals can be lengthy. As such, it is advisable to begin planning SET-BT missions at least two months in advance.

When planning SET-BT mission, it may be advisable to liaise with INTERPOL National Central Bureau or Headquarters and WOAH agroterrorism/agrocrime focal points (where present), especially in initial discussions, to facilitate and avoid any bottlenecks during the mission planning. Below is the list of tasks INTERPOL may support in a SET-BT mission depending on availability and country needs:

• Providing guidance on relevant stakeholders from security forces to involve in the SET-BT mission

• Liaising with NFPs of security forces and national security forces to facilitate organization

• Providing guidance and support in drafting the invitation and inviting stakeholders from security forces

• Ensuring relevant information/documents related to bio-/agroterrorism and bio-/agrocrime are collected during interviews, particularly from security forces

• Providing a secure platform to share sensitive information and documents (only provided by INTERPOL)

• Guiding the team on matters related to security forces, bio-/agroterrorism and bio/agrocrime

• Participating as a member of the evaluation team

Further details on the above support are provided in this annex.

2.2. Invitation of security forces

In order to involve the relevant security forces, an invitation should be sent to the ministries overseeing them. FAO and NFPs should discuss and agree on the best approach for these invitations and which security forces to involve in the mission as stakeholders to interview or evaluation team members. Nevertheless, due to SET’s focus on animal health, the lead ministry is that in charge of overseeing veterinary services (Ministry of agriculture, livestock or similar). The FAO national or regional focal point(s) should manage this with the NFP(s) from veterinary services and from security forces.

2.3. Evaluation team

In addition to the members described in section 3.2. Identifying evaluation team members and their responsibilities of the SET manual, NFPs from relevant security forces should be included in SET-BT missions. As with other team members, NFPs from relevant security forces should actively participate in all aspects of the mission including preparation, implementation and follow-up. Specific responsibilities include:

• Ensuring relevant information/documents on bio-/agroterrorism and bio-/agrocrime are collected during interviews, particularly from security forces

• Guiding the team on matters related to security forces, bio-/agroterrorism and bio-/agrocrime

• Ability to communicate in the language spoken by external evaluators

• Liaising with national or regional FAO focal point on best process to involve security forces in mission (e.g. invitation process)

• Flagging any requirements for meeting with decision-makers or stakeholders from security forces (e.g. diplomatic protocols, security protocols, etc.)

• Providing a first proposal on stakeholders for interviews and areas to visit that are relevant for agroterrorism and agrocrime detection

• Liaising with security forces stakeholders at the national and subnational level to coordinate interviews and logistics

• Following-up and requesting the relevant documents that have not been shared during the interviews

• Requesting clearance of finalized SET-BT report to the directors of relevant security forces

It may be possible to include an INTERPOL expert in the evaluation team; however, this will depend on the availability of a limited experts’ pool. It is also possible to include a national or regional WOAH agroterrorism/agrocrime expert/focal point where available. Even if not available for the full mission, experts from these organizations may provide valuable support in the planning and follow-up of the mission.

2.4. SET-BT subpacket

A separate subpacket with all relevant SET-BT documents is available within the SET packet. Similar to the SET packet, the SET-BT documents should be distributed to all members of the evaluation team as soon as they are identified so that they can familiarize themselves with the tool.

TABLE A.IV.1
Documents in the SET-BT subpacket

File name

Document use

0 - “How to use the SET-BT documents”

Guide to use the documents of the SET-BT packet.

1 – “SET and SET-BT concept note”

Document sent to the veterinary services and security forces of the country two months before the beginning of the mission, so that they can familiarize themselves with SET and SET-BT missions. This should be used in lieu of the SET concept note mentioned in Table 4 of the SET manual.

2 - “SET-BT Request for information” (Excel file)

This is an additional Excel sheet included in the SET RFI. The sheet facilitates the collection of information and documentation relevant to national agroterrorism/agrocrime detection system prior to the in-country mission. It should be filled out and returned by the country’s veterinary services and security forces at the latest 10 days prior to the mission.

3 - “SET-BT Additional stakeholder checklist and interview sign-in sheet”

Guide to record all stakeholders interviewed during the mission.

It can be adapted to the local situation. Pages 2 and 3 can be used as interview sign-in sheet. This should be used in addition to the SET stakeholder checklist.

4 - “SET-BT Sample questions”

Examples of questions to ask during interviews with stakeholders. It can be used in the field for reference. This should be used in addition to the SET sample questions.

5 - “SET-BT scoring grid” (Excel file)

Excel file used to: 1) compile all the information gathered during the mission; 2) hold the scoring session, and 3) automatically calculate the outputs of the mission.

6 - “Glossary of terms used throughout SET-BT evaluation”

Used to clarify definition of terms used in evaluation and support common understanding during the mission. This should be used in addition to the SET glossary.

7 - “SET-BT scoring guide”

Scoring guide with all SET-BT indicator as a stand-alone document to use during the scoring session.

8 - “SET-BT SWOT analysis worksheet”

Document to be used by team members to conduct SWOT analysis of SET-BT results.

9 - “SET-BT Report Template”

Document to be used for preparing the national report of the SET-BT assessment.

2.5. Request for information and reviewing documents of the system

Documents relevant to the detection of agroterrorism and agrocrimes affecting animal health in the country should be shared with the evaluation team for review prior to the start of the mission. This will provide the evaluators with a general understanding of the country’s capacities and will greatly support the subsequent interviews during the mission.

In general, documents to review include (but are not limited to):

• Organogram of relevant security forces – this is extremely important to allow evaluators to understand the system and follow interviews. If this is not possible to share for security reasons, ask security force NFPs to present the organigrams during one of the mission preparatory meetings.

• Laws, policies and regulations regulating agrocrime and agroterrorism

• List of pathogens and toxins of concern for agroterrorism and agrocrime

• Strategies and plans for the detection, reporting and investigation of agroterrorism and agrocrime

• Protocols and standard operating procedures for all activities related to the detection, reporting and investigation of agroterrorism and agrocrime

• Memorandums of understanding, letters of agreements for intersectoral cooperation on agroterrorism and agrocrime

• Any other document relevant to the detection of agroterrorism and agrocrimes in the country.

Some documents may be sensitive to share. INTERPOL and security forces NFPs should guide the team on the best way to handle any sensitive documents related to agroterrorism and agrocrime detection. An option may be to have documents shared and saved through a secure platform set up by INTERPOL. A document detailing the procedures for handling such documents may also be shared by INTERPOL.

2.6. Defining mission dates, stakeholders to interview and programme

SET-BT missions last at least 13 days between arrival and departure of external evaluators. Guidelines on which stakeholders to meet, in addition to those listed in Table 5 of the SET manual and the SET stakeholder checklist (available in the SET packet), can be found in the SET-BT Additional stakeholder checklist (available in the SET-BT subpacket). It is important to note that these guidelines are not prescriptive, and the assessment team should decide together on who to interview and how long to meet with them. Team members from the country’s veterinary services, security forces and national/regional FAO office know the local context best and are expected to actively participate in this process. A few points should be considered when selecting stakeholders to interview during the SET-BT mission, in addition to those listed in section 3.5. Defining mission dates, stakeholder to interview and programme in the SET manual. These are:

1. It is important for the evaluators to get a balanced perspective of the country’s capacities for detection of criminal or terrorist animal health events. Therefore, stakeholders to be selected for interviews should represent both advanced and weak capacities. In addition, it may be useful to speak with stakeholders of areas that are considered at high risk for criminal or terrorist animal health events. These areas may be dangerous to visit, therefore consider making arrangements to bring stakeholders from these areas to the central level or a safer nearby area for interviews.

2. Interviews with directorates of security forces within a same Ministry or broad category can be combined to save time (e.g. military forces combined; forensics, investigative and judiciary police forces combined); however, it is not advisable to combine all security forces in the same interview. Interviews with security forces should be divided at least at ministry level.

A template SET-BT programme for external evaluations is shown below (Table A.IV.2), which can be adapted to the local context. It is the responsibility of the NFPs to liaise and schedule interviews with stakeholders. Interviews with stakeholders in security forces is the responsibility of the NFPs from security forces. Keep in mind that meetings with directors of security forces may require some protocol, respect of hierarchy and security measures. These factors should be taken into account and planned accordingly, if needed. It is also the responsibility of the NFPs from security forces to guide the rest of the team on the appropriate protocols and security measures to meet with relevant security services.

Note that the launching meeting can take place online during the week before the mission. If the launching meeting is done online before the mission, the central-level interviews can occur during 2.5 days instead of 3 days.

Moreover, the approximate durations of tasks during the scoring session of a SET-BT mission (Week 2 of mission) are as follow:

• Scoring of the 32 SET-BT indicators – approximately 0.5 day

• Scoring of the 96 SET indicators – approximately 1.5-2 days

• SWOT analysis and development of recommendations for SET –BT – 0.5-1 day

• SWOT analysis and development of recommendations for SET – 1-1.5 day

Finally, note that INTERPOL experts are not allowed to participate in missions for more than 1 week. Therefore, should an INTERPOL expert be part of the evaluation team, modifications will need to be made in the mission agenda to accommodate this. This may be done by

• Having the INTERPOL expert(s) participate in person in the first week of the mission

• Having the INTERPOL expert(s) participate online in the SET-BT scoring session, and

• Organizing SET-BT scoring sessions in a way that INTERPOL expert(s) are not required to connect for more than half-a-day (morning or afternoon) each day

TABLE A.IV.2
Template SET-BT mission programme (external evaluation)

* Scoring sessions include the following sequential activities: Scoring of indicators, SWOT analysis and development of recommendations

3. IN-COUNTRY MISSION

The in-country mission follows the same structure as a regular SET mission except for a few additional points. These points are listed below.

a. Launching meeting

In addition to the decision-makers listed in section 4.2. Launching meeting of the SET manual, relevant decision-makers from security forces and other sectors involved in the detection of agroterrorism and agrocrime events should be invited to the launching meeting. Participants to this meeting may vary depending on the country context and NFPs of security forces should guide the team during the preparatory phase regarding who should attend. These may include:

• Ministry of Defense

• Ministry of interior

• Customs

• Other relevant security department/ministry[1]

• INTERPOL, WOAH and other international organizations and financial partners involved in biothreat resilience building

When conducting a mission using the SET-BT module, use the SET-BT launching meeting template presentation instead of the SET template. The template is available in the SET-BT subpacket.

b. Scoring

Scoring of SET-BT indicators follows the same process as conventional SET. After scoring all indicators, the SET-BT module in Excel will automatically generate a spider graph showing strengths and weaknesses in each of the 7 categories of the SET-BT module as a percentage compared to a “perfect” system (scores of 4 received for all indicators).

Templates are available for the SET-BT SWOT analysis. To develop recommendations, the team may use the SET SMART recommendation development template available in the SET packet.

c. Closing meeting

As with conventional SET missions, missions using the SET-BT module end with a presentation of the key findings and preliminary recommendations of the SET-BT module to improve national capacities in detecting potential criminal or terrorist animal health events. The participants should reflect those present at the launching meeting.

When conducting a mission using the SET-BT module, use the SET-BT closing meeting template presentation instead of the SET template. The template is available in the SET-BT subpacket.

4. POST-MISSION

a. Drafting the final report

When conducting a mission using the SET-BT module, use the SET-BT report template instead of the SET report template. The template is available in the SET-BT subpacket. In general, the layout of SET-BT reports includes the following sections:

1. Background on SET and SET-BT

2. Livestock situation in the country

3. Mission summary

4. Evaluation results of SET

5. SET SWOT analysis and action plan

6. Evaluation results of SET-BT – follows same structure as evaluation results of SET but instead describes the current situation in the country for the detection of terrorist or criminal animal health events

7. SET-BT SWOT analysis and action plan – follows the same structure as the SET SWOT analysis and action plan but relates to results from the SET-BT module

8. References

9. Appendices

Validation and publication: After the SET-BT report is finalized, it should be sent to the CVO and directors of relevant security forces for their validation and authorization to publish online. It is the responsibility of the NFPs of veterinary services to share the report with the CVO for validation while it is the responsibility of the NFPs of security forces to share the SET-BT report to the directors of the relevant security forces for validation. The CVO and the relevant directors of security forces should provide written (letter/email) confirmation that the report is validated and can be posted/shared online. In some instances, a country may choose to keep the final report confidential and for their own internal use. This can be done, although it is not recommended as this will significantly reduce the chances that the action plan is implemented if partners cannot access the report. It is also possible, if deemed appropriate, to keep the sections concerning results of the SET-BT module as confidential while publishing the sections on conventional SET results.

b. Implementing the action plan and following up on progress

As with the conventional SET action plan, relevant authorities may integrate the SET-BT action plan into their strategies for improving detection of potential criminal and terrorist animal health events. The action plan may also inform capacity building or be leveraged by projects such as the GAC-supported WOAH-FAO-INTERPOL agroterrorism and agrocrime project.

In addition, regular meetings with stakeholders of animal disease surveillance and security forces in the country should be conducted to review the SET-BT action plan, identify challenges and successes, and amend the action plan if necessary. The goal is to ensure that relevant activities are occurring on a regular basis in between SET-BT evaluations.

Lastly, similar to SET, it is recommended that follow-up SET-BT assessments occur every three to five years.

RESOURCES

Centers for Disease Control and Prevention & Federal Bureau of Investigation. 2016. Joint Criminal and Epidemiological Investigations Handbook (International Edition). Washington, DC, FBI and Atlanta, CDC. https://www.cdc.gov/phlp/docs/crimepihandbook2016.pdf

United States Food and Drug Administration & United States Department of Agriculture. 2008. Criminal investigation Handbook for Agroterrorism. Silver Spring, FDA and Washington, DC, USDA. https://www.fsis.usda.gov/news-events/publications/criminal-investigation-handbook-agroterrorism

Gioia, G.V., Lamielle, G., Aguanno, R., ElMasry, I., Mouillé, B., De Battisti, C., Angot, A. et al. 2021. Informing resilience building: FAO’s Surveillance Evaluation Tool (SET) Biothreat Detection Module will help assess national capacities to detect agro-terrorism and agrocrime. One Health Outlook, 3(14).

World Organisation for Animal Health. 2018. Guidelines for Investigation of Suspicious Biological Events (Guidelines for National Veterinary Services). Paris, WOAH. https://www.woah. org/fileadmin/Home/eng/Our_scientific_expertise/docs/pdf/Guidelines_Investigation_Suspicious_Biological_Events.pdf

REFERENCES

Centers for Disease Control and Prevention. 2001. Updated guidelines for evaluating public health surveillance systems – Recommendations from the guidelines working group. Morbidity and Mortality Weekly Report, 50(RR12): 1–35. https://www.cdc.gov/mmwr/ preview/mmwrhtml/rr5013a1.htm

Centers for Disease Control and Prevention. 2004. Framework for evaluating public health surveillance systems for early detection of outbreaks. Morbidity and Mortality Weekly Report, 53(RR05): 1–11. https://www.cdc.gov/mmwr/preview/mmwrhtml/rr5305a1.htm

Health Canada. 2004. Framework and Tools for Evaluating Health Surveillance Systems. Ottawa. http://publications.gc.ca/collections/Collection/H39-4-46-2004E.pdf

World Health Organization. 1997. Protocol for the Evaluation of Epidemiological Surveillance Systems. Geneva. http://apps.who.int/iris/handle/10665/63639

World Organisation for Animal Health. 2015. Reducing biological threats thanks to resilient animal health systems. Bulletin No. 2015-3. Paris. https://www.woah.org/fileadmin/Home/eng/Publications_%26_Documentation/docs/pdf/bulletin/Bull_2015-3-ENG_new.pdf

World Organisation for Animal Health. 2020. Our action: International cooperation against agro-terrorism. In: World Organisation for Animal Health. https://www.report2019oie.fr/en/international-cooperation-against-agro-terrorism/

1 Other relevant security department/ministry will vary between countries and may include royal guards, gendarmerie, national guards, military police, among others.

Annex V
SET mission checklist

SURVEILLANCE EVALUATION TOOL (SET) MISSION WORKPLAN:
[INSERT COUNTRY]

Project goals:

1. Evaluate [Insert country]’s animal disease surveillance system using the standardized SET toolkit

2. Jointly identify and prioritize recommendation for improvement in [Insert country]

SET mission timeline:

FAO ANIMAL PRODUCTION AND HEALTH MANUALS

1. Small-scale poultry production, 2004 (En, Fr)

2. Good practices for the meat industry, 2004 (En, Fr, Es, Ar)

3. Preparing for highly pathogenic avian influenza, 2007 (En, Ar, Ese, Fre, Mke)

4. Revised version, 2009 (En)

5. Wild bird highly pathogenic avian influenza surveillance – Sample collection from healthy, sick and dead birds, 2006 (En, Fr, Ru, Ar, Ba, Mn, Ese, Zhe, Th)

6. Wild birds and avian influenza – An introduction to applied field research and disease sampling techniques, 2007 (En, Fr, Ru, Ar, Id, Ba)

7. Compensation programs for the sanitary emergence of HPAI-H5N1 in Latin American and the Caribbean, 2008 (Ene, Ese)

8. The AVE systems of geographic information for the assistance in the epidemiological surveillance of the avian influenza, based on risk, 2009 (Ene, Ese)

9. Preparation of African swine fever contingency plans, 2009 (En, Fr, Ru, Hy, Ka, Ese)

10. Good practices for the feed industry – implementing the Codex Alimentarius Code of Practice on good animal feeding, 2009 (En, Zh, Fr, Es, Ar)

11. Epidemiología Participativa − Métodos para la recolección de acciones y datos orientados a la inteligencia epidemiológica, 2011 (Ese)

12. Good Emergency Management Practice: The essentials – A guide to preparing for animal health emergencies, 2011 (En, Fr, Es, Ar, Ru, Zh, Mn**)

13. Investigating the role of bats in emerging zoonosese – Balancing ecology, conservation and public health interests, 2011 (En)

14. Rearing young ruminants on milk replacers and starter feeds, 2011 (En)

15. Quality assurance for animal feed analysis laboratories, 2011 (En, Fre, Rue)

16. Conducting national feed assessments, 2012 (En, Fr)

17. Quality assurance for microbiology in feed analysis laboratories, 2013 (En, Zh**)

18. Risk-based disease surveillance – A manual for veterinarians on the design and analysis of surveillance for demonstration of freedom from disease, 2014 (En)

19. Livestock-related interventions during emergencies – The how-to-do-it manual, 2016 (En, Zh**)

20. African Swine Fever: Detection and diagnosis – A manual for veterinarians, 2017 (En, Zh, Ru, Lt, Sr, Sq, Mk, Es)

21. Lumpy skin disease – A field manual for veterinarians, 2017 (En, Ru, Sq, Sr, Tr, Mk, Uk, Ro, Zh)

22. Rift Valley Fever Surveillance, 2018 (En, Fr, Ar)

23. African swine fever in wild boar ecology and biosecurity, 2019 (En, Ru**, Fr**, Es, Zh**, Ko, Lt)

24. Prudent and efficient use of antimicrobials in pigs and poultry, 2019 (En, Ru, Fr**, Es**, Zh**)

25. Good practices for the feed sector - Implementing the Codex Alimentarius Code of Practice on Good Animal Feeding, 2020 (En, Vi)

26. Good emergency management practice: The essentials – A guide to preparing for animal health emergencies. Third edition, 2021 (En, Es, Ru, Fr, Ar, Zh)

27. Conducting After Action Reviews for animal health emergencies, 2022 (En)

28. Manual for the management of operations during an animal health emergency, 2022 (En, Fr)

29. African swine fever in wild boar – Ecology and biosecurity. Second edition, 2022 (En)

Availability: October 2024

Ar

-

Arabic

Ba

-

Bashkir

En

-

English

Es

-

Spanish

Fr

-

French

Hy

-

Armenian

Id

-

Indonesian

Ka

-

Georgian

Ko

-

Korean

Lt

-

Lithuanian

Mk

-

Macedonian

Mn

-

Mongolian

Pt

-

Portuguese

Ro

-

Romanian

Ru

-

Russian

Sq

-

Albanian

Sr

-

Serbian

Th

-

Thai

Tr

-

Turkish

Uk

-

Ukrainian

Vi

-

Vietnamese

Zh

-

Chinese

Multil

-

Multilingual

* Out of print

** In preparation

e E-publication

The FAO Animal Production and Health Manuals are available through authorized FAO Sales Agents or directly from Sales and Marketing Group, FAO, Viale delle Terme di Caracalla, 00153 Rome, Italy.

FAO ANIMAL HEALTH MANUALS

1. Manual on the diagnosis of rinderpest, 1996 (En)

2. Manual on bovine spongifom encephalophaty, 1998 (En)

3. Epidemiology, diagnosis and control of helminth parasites of swine, 1998 (En)

4. Epidemiology, diagnosis and control of poultry parasites, 1998 (En)

5. Recognizing peste des petits ruminant – a field manual, 1999 (En, Fr)

6. Manual on the preparation of national animal disease emergency preparedness plans, 1999 (En, Zh)

7. Manual on the preparation of rinderpest contingency plans, 1999 (En)

8. Manual on livestock disease surveillance and information systems, 1999 (En, Zh)

9. Recognizing African swine fever – a field manual, 2000 (En, Fr)

10. Manual on participatory epidemiology – method for the collection of action-oriented epidemiological intelligence, 2000 (En)

11. Manual on the preparation of African swine fever contigency plans, 2001 (En)

12. Manual on procedures for disease eradication by stamping out, 2001 (En)

13. Recognizing contagious bovine pleuropneumonia, 2001 (En, Fr)

14. Preparation of contagious bovine pleuropneumonia contingency plans, 2002 (En, Fr)

15. Preparation of Rift Valley Fever contingency plans, 2002 (En, Fr)

16. Preparation of foot-and-mouth disease contingency plans, 2002 (En)

17. Recognizing Rift Valley Fever, 2003 (En)