2.1 Different Types of Interlaboratory Comparisons

There are two types of interlaboratory comparison studies: proficiency tests, which aim at monitoring the proficiency of laboratories and test performance studies (TPS; or collaborative method validation studies, collaborative trials, ring tests), which aim at evaluating the performance of (a) specific test(s) and whether it (they) is (are) fit for purpose (ISO 17025 2005, EPPO 2014 PM 7/122(1)). The main differences between TPS and proficiency tests are shown in Fig. 2.1. In this booklet only TPS organisation is covered.

Fig. 2.1
figure 1

Main differences between test performance studies and proficiency tests in plant health

Full size image

A TPS is usually organised to monitor the performance of a newly developed test to detect and/or identify ‘emerging’ pests or strains of known pests or to compare the performance of different tests. The results of TPS also provide information on how (a) test(s) perform(s) in different laboratories; i.e., on different equipment, with different reagents, different personnel. This allows a better estimation of the accuracy and reproducibility of tests. The organisation of a TPS is a complex process that requires significant efforts from the organisers in terms of time, expertise and finances. A TPS may be organised following a request from different stakeholders involved in plant health, such as National Plant Protection Organisations (NPPOs), companies producing commercial tests, and diagnostic laboratories, as part of their activities.

TPS organisation includes different steps that are connected and are mutually dependent (Fig. 2.2). The required steps in TPS organisation are sometimes sequential, while others can occur simultaneously. The first two steps are presented in Sects. 2.2 and 2.3 of this chapter whereas the other steps are presented in Chap. 3.

Fig. 2.2
figure 2

Steps involved in test performance study organisation

Full size image

2.2 Selection of Pests

For the validation of test(s) for a specific pest a choice should be made by the laboratory to organise a TPS or to perform an intralaboratory validation. The selection of the pests for which a TPS should be organised is important considering resources needed for validation processes (e.g., the personnel, financial requirements, biological material, etc.). The aim of selection is to direct resources to critical points and/or where improvement of tests has the highest beneficial impact. Ideally, test selection should involve different stakeholders and reflect current and potential future needs in plant health. Pests for which there is a need for an appropriate test are selected taking into account their current importance or their potential to have a significant impact in the near future. The expression of a need to conduct a TPS may come from different stakeholders. The VALITEST Project focused on pathogens and partners selected 11 pests (including viruses, bacteria, nematodes and fungi) that were of interest to stakeholders in the region. The first six target pests for the first round of TPS were selected during the preparatory phase of the Project, by matching a list of candidate pests with the experience of the VALITEST partner laboratories. This ensured that laboratories with expertise in specific pests organised the relevant TPS. The pests that were selected for this first round are listed in Table 2.1.

Table 2.1 Pests and associated test performance study organisers selected for the first round of Test performance studies in the framework of the VALITEST Project
Full size table

For the second round of the VALITEST TPS, the targets were selected based on stakeholder and market needs and identified through two online surveys by two online surveys launched by EPPO at the beginning of the Project (Agstner and Jones 2020). For the second round of the VALITEST TPS, the targets were selected based on stakeholder and market needs (Trontin et al. 2021). At the beginning of the Project, two online surveys were launched for the list of pests categorised by the EU, one for laboratories and one for NPPOs. Respondents to both surveys were asked to indicate their current testing priorities. The laboratories were instructed to rank the pests according to the number of tests performed. NPPOs were instructed to rank the pests based on current plant health priorities. In the second survey representatives of NPPOs were instructed to rank their top 10 pests based on current plant health priorities. The surveys were distributed through the EPPO networks, and the results were combined for the final selection of the pests. The six pests selected for the second round of TPS are listed in Table 2.2.

Table 2.2 Pests and associated test performance study organisers selected for the second round of test performance studies in the framework of the VALITEST Project
Full size table

2.3 TPS Organiser

The capacity (in terms of both staff expertise and availability) of the TPS organiser is essential to ensure the reliability of the results of such study. The requirements for the organisation of interlaboratory comparisons are described in EPPO PM 7/122(1) 2014 and ISO 17043 (2010). TPS organisers should have appropriate technical and scientific competence in relation to the organism and the test(s) to be performed, they should remain impartial, and should maintain confidentiality throughout the process. In practice this means that an external reader of a TPS report should not be able to identify which results were obtained by which participant (unless the participant waives confidentiality). When a laboratory commits to organise a TPS, this should be carefully planned and communicated with the interested stakeholders. Laboratories organising TPS should carefully prepare the plans and determine a feasible time frame, they need to take into account unforeseen problems. Some empirical examples of how certain unforeseen events can affect the time frame are given in Chap. 3. A laboratory organising a TPS should define a person responsible for the organisation of this TPS, and should ensure the availability of personnel, equipment and facilities needed for the organisation. The organiser should have a quality assurance system in place (ISO 17025 (2005) or equivalent). A TPS organiser needs to keep records of all the activities and to provide traceability of the measurement results.

2.3.1 Selection of TPS Organisers

Test performance studies can be organised individually or, in some cases, in batches (i.e., more than one TPS at the same time, organised by the same or different organisers or consortium) when necessary. When several laboratories are part of a network a choice may need to be made of which one should organise the TPS. For the first round of TPS organised in the framework of VALITEST, TPS organisers were selected based on laboratories’ expertise with a selected list of pests (Table 2.1). For the second round of TPS (Table 2.2), TPS organisers were selected based on the results of an online poll, where each potential organiser expressed their interest to organise one or more specific TPS on a preselected list of prioritised pests. Organisers judged their expertise regarding the list of pests. When more than one laboratory expressed an interest in organising a TPS for one pest, the TPS organiser was selected based on predefined criteria that were transparent and approved by all the parties involved. These criteria are described below.

2.3.2 Minimum Requirements for TPS Organisers

A list of minimum criteria that should be met by any TPS organiser was defined and is detailed below. A TPS organiser must also fulfil all requirements that are prescribed for TPS participants (see Sect. 3.3.2 “Definition of weighted criteria for the selection of TPS participants”). In addition, it is an advantage if a TPS organiser has experience in organising similar interlaboratory comparisons.

2.3.2.1 Involvement in Diagnostic Activities with Different Methods

A TPS organiser needs to have significant experience in the performance of diagnostic activities (e.g., performing routine analyses) using a range of different methods and matrices (e.g., type of plant material: seed, leaves, etc.) for the pest in question or in the case of an emerging pest for a pest of the same genus or family.

2.3.2.2 Experience with the Development and Validation of Diagnostic Tests

The personnel in the laboratory organising a TPS needs to demonstrate continued experience in the validation of diagnostic tests in their field, and to have adequate knowledge, competence and experience. Experience in the development of diagnostic tests can be considered as an additional advantage. Records of staff qualifications and training should be available, for example through a quality management system.

2.3.2.3 Experience with the Shipment of Material

Materials for TPS often require special ship** conditions. These can include special requirements for ship** on dry ice or for ship** certain quarantine pests, for which participants might need to provide specific documentation, such as a Letter of Authority. Consequently, it is important that a TPS organiser has experience in ship** similar materials abroad. Completing the official paperwork for ship** specimens according to Plant Health Regulations can take time, which should be considered by the TPS organiser when planning the TPS.

2.3.2.4 Collaboration with National and International Bodies in the Field of Plant Pest Diagnostics

It is preferable that TPS organisers are part of diagnostic networks at national, regional (e.g., EPPO) or international (e.g., IPPC) level or participate in Euphresco. Participation in such networks facilitates collaborations within the field, ensures an efficient flow of information, and provides an opportunity to communicate and disseminate TPS results to the target audience, thereby increasing the impact of the Project.

2.3.2.5 Technical Requirements

A TPS organiser should use agreed terminology and processes, have an established quality assurance system, be able to prepare reference material, and ensure that the validation processes are properly documented. In the EPPO region, terminology is defined in PM 7/76 (EPPO 2018a) and processes for validation defined in PM 7/98 (EPPO 2021a). The use of harmonised terminology can prevent misunderstandings. It is also an advantage if a TPS organiser has already participated in at least one interlaboratory comparison study, preferably in a TPS.