Figures
Abstract
Background
To support the development of early warning and surveillance systems of emerging zoonoses, we present a general method to prioritize pathogens using a quantitative, stochastic multi-criteria model, parameterized for the Netherlands.
Methodology/Principal Findings
A risk score was based on seven criteria, reflecting assessments of the epidemiology and impact of these pathogens on society. Criteria were weighed, based on the preferences of a panel of judges with a background in infectious disease control.
Conclusions/Significance
Pathogens with the highest risk for the Netherlands included pathogens in the livestock reservoir with a high actual human disease burden (e.g. Campylobacter spp., Toxoplasma gondii, Coxiella burnetii) or a low current but higher historic burden (e.g. Mycobacterium bovis), rare zoonotic pathogens in domestic animals with severe disease manifestations in humans (e.g. BSE prion, Capnocytophaga canimorsus) as well as arthropod-borne and wildlife associated pathogens which may pose a severe risk in future (e.g. Japanese encephalitis virus and West-Nile virus). These agents are key targets for development of early warning and surveillance.
Citation: Havelaar AH, van Rosse F, Bucura C, Toetenel MA, Haagsma JA, Kurowicka D, et al. (2010) Prioritizing Emerging Zoonoses in The Netherlands. PLoS ONE 5(11): e13965. https://doi.org/10.1371/journal.pone.0013965
Editor: Adam J. Ratner, Columbia University, United States of America
Received: July 8, 2010; Accepted: September 27, 2010; Published: November 15, 2010
Copyright: © 2010 Havelaar et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The EmZoo project is funded by the Dutch Ministry of Agriculture, Nature and Food Quality (www.minlnv.nl). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript, other than guidance and feedback by the EmZoo Supervisory Committee on the relevance of criteria for policy decisions.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Human health is threatened by a wide variety of pathogens transmitted from animals to humans. In the Netherlands, a systematic approach for early warning and surveillance of emerging zoonoses and a blueprint for an efficient network of collaborators from the medical and veterinary professions to prevent and control emerging zoonoses are being developed by a consortium of national institutes for human and animal health (the EmZoo consortium). To support this task, a prioritized list of emerging zoonotic pathogens of relevance for the Netherlands was needed. The HAIRS Group in the UK [1] has developed qualitative decision trees to assess the zoonotic potential of emerging diseases [2] and to classify the risk to public health, based on probability and impact of infection [3].
Priority setting is a multi-dimensional problem, in which technical information is often intertwined with value judgments. Traditionally, a priority setting procedure entails asking a limited number of experts to reach consensus. An example of this approach in the domain of emerging zoonoses has been published in France [4]. This method is relatively straightforward, but not very transparent and the repeatability is low. Currently, semi-quantitative methods are frequently used in which criteria are divided into a limited number of classes (e.g. low, medium and high). Criteria may also be scored on arbitrary scales (e.g. 0, 1, …, 5), while scores for all criteria are aggregated to produce an overall score. An example of this approach was published in Belgium [5], and a similar approach was taken for animal diseases by McKenzie et al. [6] in New Zealand. Here, the transparency and the repeatability are improved, but the classes are chosen rather arbitrarily. Linear relations between the different classes of a criterion or between criteria are often assumed but are not supported by data. For the current project, the aim was to develop a quantitative method to rank emerging zoonoses using clearly interpretable criteria, expressed on natural numerical scales. Furthermore, weights were incorporated for these criteria, elicited by a systematic procedure from a panel of judges, independent from the authors or scientific experts in the project. The method was designed to simultaneously be the basis of a web-based knowledge management system.
The quantitative method is based on the well-established multi-criteria analysis (MCA) method. This method has been used in many decision making contexts including animal health [7]. MCA offer methods and techniques to structure complex decision-making. After completing the different phases, information can be introduced or modified without the necessity to completely redo the analyses. This is especially valuable in the priority setting of emerging zoonoses, where information changes constantly. In our approach to MCA, we combined objective information on the epidemiology and societal impact of zoonotic pathogens with subjective information on the relative weights of different criteria. The objective information was based on scientific evidence, while for the subjective information the values of individuals involved in the control of infectious diseases were sought.
Methods
Selection of pathogens
Zoonoses are defined as diseases that can be transmitted between vertebrate animals and man under natural conditions. An emerging zoonoses is a zoonosis that is newly recognized or newly evolved, or that has occurred previously but shows an increase in incidence or expansion in geographic, host, or vector range [8]. Of 1415 known species of human pathogens, there are 868 zoonotic pathogens [9], but only a limited number of them is considered relevant as emerging zoonoses for the Netherlands.
Information from recent published studies on emerging zoonoses in the Netherlands [10] and from other European countries [4], [11], [12], [13], [14], [15] was taken into account. Furthermore, relevant information was gathered from signals of emerging zoonoses from internet sources of public health and veterinary organizations including the WHO, OIE, HPA and CDC and ProMED-mail. In addition, expert members of the Emzoo consortium were invited to suggest additional pathogens. This process resulted in a long-list, including all pathogens (174) mentioned as emerging zoonoses in one of the sources mentioned above. Only pathogens with a proven zoonotic potential [2] were included in our final list. To condense the resulting long-list to a more manageable short-list, five additional decision rules were applied. A zoonotic pathogen was excluded from the list if:
- non-human primate species form its only known reservoir. These reservoir species are not likely to occur as free ranging species in Europe and the pathogens have little public health significance other than very specific occupational risks, e.g. Simian foamy virus;
- its specific only known reservoir species is absent in Europe, e.g. Sin nombre virus;
- its vector (in case of a vector-borne zoonotic pathogen) family (not vector species) is absent in Europe, e.g. Trypanosoma spp.;
- the zoonotic aspects involved a single species jump, after which the pathogens further evolved and became effectively and essentially transmissible from human to human e.g. new influenza H1N1 or HIV.
This analysis finally resulted in a short-list of 86 emerging zoonotic pathogens of relevance for the Netherlands (see database in Annex S2), which are evaluated by the risk-ranking method.
Listing and structuring of criteria
We quantified the risk to public health of emerging zoonoses by applying seven criteria that covered the complete pathway from introduction to societal impact (Figure 1). All criteria were scored on a natural scale, and were divided into 4-5 levels; often covering several orders of magnitude in terms of effects (see Table 1 and Annex S1). For subsequent analysis, each class was represented by a point estimate, representing a central value in the range.
Evaluating pathogens on the selected criteria
Where possible, levels were assigned to pathogens based on published literature. Values were to reflect the current situation in the Netherlands, given the existing level of prevention and health care including vaccination and infrastructure (water supply, sewerage, food safety controls) et cetera. We, therefore, mainly used data from industrialized countries. For many pathogens currently available data were insufficient, and in those cases we tried to evaluate criteria using simple decision rules. In the absence of both sufficient data and decision rules, expert opinion was employed and related uncertainty was expressed by assigning a pathogen to more than one level. All assignments were made from the societal perspective, i.e. the impact on all affected parties and sectors of economy was considered.
Determining the weight of each criterion
Weights were based on panel sessions with different groups of participants, representing different professional groups involved in infectious disease control:
- Risk managers from the Dutch Ministries of Agriculture and Public Health (n = 7);
- Infectious disease specialists from medical microbiological laboratories and from regional public health services (n = 11)
- Students in the medical and veterinary faculties of Utrecht University (n = 11).
Each panel session started with an explanation of the objectives and approaches of the project. Panel members were invited to comment on the approach and ask questions about any aspect. Discussion was specifically stimulated on the criteria and their scores, as ranking these was the core task of the panel members.
For the ranking exercise, five groups of seven scenarios were generated. Each scenario (designated by a two letter code, e.g. QJ) represented a hypothetical zoonotic agent, by randomly choosing a level for each criterion, subject to certain constraints: scenarios were chosen as not to ‘majorize’ each other (i.e. no scenario should have a higher risk level on all criteria than any other in the same set), and implausible scenarios (i.e. with low animal prevalence yet very high costs) were omitted. Each scenario was presented to the panel members on a small card (Figure 2). Panel members were asked to place the scenario that they considered to represent the lowest risk to the left of their table and the highest risk scenario to the right. They were then asked to arrange the remaining five scenarios in between these two extremes, in order of increasing risk. To alleviate potential effects of training and fatigue, the five groups of seven scenarios (denoted by G1, …., G5) were offered to one half of the panel members in the order G1, G3, G5, G4, G2 and to the other half in the order G3, G2, G4, G1, G5. Data were entered in a Microsoft Excel spreadsheet independently by two analysts, and any discordance was resolved by referring to the original data sheets.
The numbers 1–7 represent the criteria C1–C7 (for details see Table 1).
Panel rankings were checked for consistency in two ways. Firstly, scenario group G2 included two scenarios that also occurred in G1, G3 contained two scenarios from G2 and so on. Consistency was evaluated by calculating the number of pairs that were ranked differently (with a maximum of 4). Secondly, all panel members received G2 again by (e-)mail two weeks after the session and were asked to re-rank the scenarios. Results were considered inconsistent if the rank of a scenario shifted two or more positions, and the number of inconsistencies (with a maximum of 30) were counted.
Data-analysis was carried out by probabilistic inversion, as described by Kurowicka et al. [16]. Further technical details on probabilistic analysis as a method to model stakeholder preferences can be found in Nesloo and Cooke [18]. Detailed results and software code used for this particular project can be obtained from one of the authors (d.kurowicka@ewi.tudelft.nl). Probabilistic inversion consisted of the following steps:
- Evaluation of randomness.
- Transformation of values (Table 1).
- Optimization of constraints.
- Main analysis (probabilistic inversion)
A simpler method to prioritize infectious diseases for surveillance was proposed by Krause et al. [17]. To compare with our approach to elicit preference-based weights, panel members were also asked to directly assign a rank order to the seven criteria and mean ranks were calculated.
Aggregation of data
A linear model was applied, which combined the mean weights from the panel session with transformed values for all 86 zoonotic agents. The model calculates the score Si of a pathogen as:where Xij is the (transformed) value assigned to pathogen i on criterion j and Bj is the weight of criterion j.
These results were then normalized to a value between 0 and 1 by calculating the scores for the pathogen with the highest and lowest theoretical risk (i.e. for which the values on all criteria were at the highest or the lowest level).
Uncertainty in the transformed scores was included as discrete distributions with equal weights, and quantified by Monte Carlo simulation in @RISK Professional Version 5.0 (Palisade Corporation, Ithaca, NY USA), an add-in to Microsoft Excel.
Sensitivity analysis
To assess the impact of different model assumptions on the outcomes, several alternative scenarios were evaluated. These included:
- Equal weights. Instead of using the preference-based weights from the panel sessions, each criterion was assigned an equal weight.
- Semi-quantitative method. Instead of assigning a transformed value to each level as shown in Table 1, values of 1 … 5 were assigned to all criteria. Scores were calculated using equal weights.
- Deterministic model. An interactive website (Emerging Zoonoses Information and Priority system (EZIPs; http://ezips.rivm.nl) was developed that allows the user to change scores for any pathogen on each criterion to evaluate the possible impact of uncertain or modified information. It is also possible to exclude one or more criteria from the ranking, to compute scores with equal weights or to introduce a new pathogen and compare it with pathogens already in the database. For technical reasons, a stochastic model could not be implemented in the website and, therefore, uncertain values were replaced by single estimates. Single estimates were chosen so that the score was as close to the mean score from the stochastic model as possible. However, as there are only few levels per criterion, deviations could not be avoided. In addition to the results of the MCA, the website also contains descriptive information on all pathogens in 5 categories: Taxonomy, Human and Animal Disease, Reservoirs, Transmission, and Geographical distribution.
Cluster analysis
Policy makers may to better grasp a categorization of diseases when expressed in qualitative terms (low, middle and high importance), than when expressed as a continuous number. We therefore implemented a cluster analysis. Based on an adapted version of the methodology used in Cardoen et al. [5], groups of different importance were identified by Classification and Regression Tree analysis (CART Version 6.0, Salford Systems, San Diego, California, USA [19]). As the normalized score is a continuous variable, we aim to obtain subgroups with minimal within group variance (grouping zoonoses with similar importance). Starting with all the pathogens the method will in first instance obtain a binary split into two groups (nodes) that are most homogeneous with respect to the normalized score. The two subgroups will then be further split so that the “purest” subgroups are obtained. The process is then continued until the nodes can not be further “purified” using a technique called cross-validation [20]. In contrast with [5], we did not use the mean total scores per disease (i.e. one value per disease) as input, but the output of the Monte Carlo simulations. This accounts for the existing uncertainty in the normalized scores. The categorical variable comprising the names of the pathogens was used as a discrimination variable. In this way, Monte Carlo samples of the same pathogen were kept together in the different clusters of pathogens.
Results
Listing and structuring of criteria
Details of criteria are given in Table 1, a full description can be found in Annex S1, including decision rules for assigning levels in absence of data.
Evaluating pathogens on the selected criteria
A full table of scores of criteria of each of the 86 pathogens is presented in Annex S2.
Determining the weight of each criterion
An example of a group of randomly generated scenarios that were ranked in panel sessions is presented in Table 2. The consistency between ranking in the panel session and the repetition after two weeks was good: 11 panel members did rank the scenarios in the same order in both sessions, and 10 provided only one answer that was not consistent with the previous ranking. 6% of scores resulting from ranking the same group after two weeks were considered inconsistent, and no panel member scored more than 20% inconsistencies. It was concluded that scores were sufficiently consistent to warrant further analysis. The results for group 1 (G1) are given in Table 3 as an example. Scenarios GF and WL represent the highest risk by the panel's opinion, while NW and QJ are considered to represent the lowest risk. Scenario VG is ranked as of medium risk, and there is considerable disagreement between the panel members on the risk of scenarios JR and ZC.
Including all signals in the model in which four or more panel members ranked the scenario at a particular position in the analysis (as indicated in Table 3 for G1) resulted in 51 constraints to be taken into account from the combined dataset of G1, G2 and G5. The scores of two out of five groups were not significantly different from random ordering and these groups were excluded from further analysis. The linear model was sufficient to reproduce the panel members' preferences.
Table 4 shows, for each criterion, the weights obtained and their standard deviation. Based on rankings by panel members, probabilistic inversion identified the human case-fatality ratio and animal-human transmission the most important criteria, whereas they considered transmission between animals, human morbidity and economic damage in animals least important. The coefficient of variation (standard deviation / mean) varied between 14 and 28%, reflecting deviating opinions between panel members about the relative importance of criteria.
Table 4 compares the weights derived by probabilistic inversion with the simple ranking method as proposed by Krause et al. The participants consider C5, C7 and C4 as the more important criteria when they rank them directly but the probabilistic inversion excludes C5 as important criterion. There is no significant correlation between both methods (p = 0.29, linear regression).
Aggregation of data
Figure 3 shows the results of combining in the linear model the levels per pathogen with the mean weights as described above. The confidence intervals reflect the valuations of a random stakeholder, given uncertainty on criteria levels of the zoonoses. The model appears to have good discriminative power. Within the possible range for normalized scores of 1 to 0, there is a rather continuous decrease in normalized scores from 0.68 for the pathogen with the highest risk (Influenza A virus (avian) H5N1) to 0.15 for the pathogens with the lowest risk (Dhori virus). The error bars around the normalized scores reflect uncertainty about the epidemiological characteristics of the pathogens, which is particularly large for many exotic viruses. Note however that the uncertainty tends to be greater for pathogens with lower normalized scores. Inspection of Annex S2 shows that the greatest uncertainty was associated with criteria relating to transmission in the animal reservoir (C2) and from animals to humans (C4). There was little uncertainty in the transmission between humans (C5).
Three groups of statistically different importance were identified by Classification and Regression Tree analysis and are represented by dashed lines. Mean (standard deviation) of the full dataset: 0.423 (0.124). Mean (standard deviation) of the three clusters: 0.577 (0.047); 0.476 (0.044); 0.317 (0.083).
Sensitivity analysis
Figure 4 shows relatively good correlation between scores obtained with the baseline model using preference-based weights and an alternative model in which each criterion is given equal weight. Yet, even relatively small differences in scores may significantly affect the ranking of pathogens.
A comparison between the quantitative method proposed in this paper and the semi-quantitative method currently used by many authors (both models with equal weights) showed that despite a general tendency for ranks to increase in parallel, the discriminative power of the quantitative method was much larger. The semi-quantitative method can only assign a discrete number of scores, whereas the quantitative method uses the full scale in a continuous manner. Rankings according to both methods may also be quite different (Figure 5). Most pathogens were ranked from five places lower to 15 places higher, but extremes from 16 places lower to 25 places higher did occur.
Cluster analysis
Three statistically different groups of importance were identified by CART and are indicated by (dashed) lines in Figure 3. The optimal number of subgroups was 29, but for the sake of practical use of the results, we report the three main clusters only. The clusters comprise 18, 28 and 40 pathogens, respectively. Splitting the tree further in e.g. five clusters subdivided the cluster with the lowest normalized scores and hence is not very informative for risk management purposes.
Among the first cluster including 18 pathogens with the highest normalized scores, there are one prion, 7 viruses, 9 bacteria and one protozoan parasite. 8 are already present in the Netherlands while 10 are not. Helminths are not represented in this group. The obtained grouping is not very sensitive to the number of Monte Carlo simulations. Indeed, the grouping obtained with 400 and 200 Monte Carlo simulations only differed by one pathogen shifting from one group to another. No difference was noted between 400 and 600 Monte Carlo simulations, indicating that 600 simulations were more than sufficient for a robust grouping.
Discussion
We describe a quantitative, stochastic method to rank the risk of emerging zoonotic pathogens for the Netherlands. The approach differs from several previously published methods. We decided to restrict the number of criteria. With higher numbers, it becomes increasingly complex to develop validated databases in which pathogens are assigned to multiple possible values. Furthermore, choosing between different scenarios as in our panel studies becomes less meaningful as respondents will only use a limited number of criteria to base their judgment on. By choosing criteria at a high level of integration, we do, however account for many criteria that are used in similar exercises, either explicitly by incorporating them in decision rules or implicitly in the transmission criteria.
In contrast to most current approaches, we scored our criteria using associated numerical scales, rather than non-informative ad-hoc scales. This forces explicit consideration of the available scientific evidence and we suggest that our quantitative approach is less arbitrary in assigning values to possible levels that a criterion can take, and is therefore more realistic than a semi-quantitative approach. Our comparison with currently used semi-quantitative methods (Figure 5) shows that there are considerable differences between the quantitative and semi-quantitative approach. We also introduce preference-based weights in the calculation of the pathogen scores. The weights are reflecting the preferences of a panel of decision makers, in our case professionals involved in infectious disease control. Our comparative analysis shows that using weights does affect ranking, but to a lesser extent than introducing numerical scales. We also found that our elaborate method of establishing weights through choice experiments provided weights that were very different from those obtained with a simple ranking exercise.
Assigning levels to the 86 pathogens on the short-list was found to be a difficult process that required several iterations involving literature studies and evaluation by pathogen-specific experts. Nevertheless, considerable uncertainty remains, part of which was expressed in uncertainty ranges around the normalized scores. By identifying the factors that contribute most to the uncertainty in quantified risk for pathogens with high normalized scores, these results can be used to prioritize additional data collection and analysis. The current method can easily be updated to incorporate new data in a transparent way. Furthermore, the web tool allows all users of the system to explore the impact of different value assignments in an interactive mode.
The pathogens with the highest score according to the baseline model would be proposed as priorities for risk management activities. Subdivision into smaller groups with different implications for risk management is suggested. This is illustrated by considering the 18 pathogens in the cluster with the highest normalized scores. A major subdivision is between pathogens already established in the Netherlands and pathogens that are not. Surveillance and risk management strategies are likely to be different for these categories. As a next step in the EmZoo project, all pathogens were evaluated for the availability of hum and and veterinary diagnostic methods, and surveillance systems. Results showed that many gaps in diagnostics and surveillance exist, also for the zoonoses in the first cluster. It was suggested that many of these gaps can be complemented by developing generic surveillance systems, which, in an efficient way, monitor for more than one pathogen at a time. Thus, the development of mosquito monitoring, tick monitoring, rodent monitoring, and syndromic surveillance in humans and horses was recommended.
Instead of using expert panels, the same method could also be used to identify issues that are important for the general public (citizens) as their weighing of criteria could be different. These results might offer opportunities to improve risk communication to the general public. Moreover, the method could also be used in another context (e.g. in developing countries) in order to prioritize pathogens that should be addressed in developmental aid programmes.
The model for priority setting presented here is based on criteria reflecting the epidemiology and societal impact of zoonotic diseases. Risk perception by the general public is not included in this model, but may pose additional challenges to policy makers. Further work to include risk perception as a second dimension in the priority model is recommended.
In summary, the EmZoo project has resulted in:
- the development of a cross-disciplinary network to deal with zoonoses threats;
- the development of systematic, explicit and quantitative estimates of risk; and
- a web-based knowledge management system.
Supporting Information
Annex S1.
Criteria: definitions, ranges, point estimates, and decision rules.
https://doi.org/10.1371/journal.pone.0013965.s001
(0.07 MB DOC)
Acknowledgments
The EmZoo project is a collaboration between:
- Animal Health Services, Deventer;
- Central Veterinary Institute, Lelystad;
- Faculty of Veterinary Medicine, Utrecht University;
- National Institute for Public Health and the Environment, Bilthoven.
The authors would like to thank scientific experts (Boyd Berends, Fred Borgsteede, Titia Kortbeek, Rob Moormann, Daan Notermans, Wim van der Poel, Chantal Reusken, Barbara Schimmer and Fred van Zijderveld) for assistance with scoring of pathogens. They also thank panel members for their participation in the panel sessions, and Jeroen Selier, Marjolein de Leeuw, Rudy Otzen, Wilbert Cortz and Merel Postma for assistance in development of the EZIPs website. Two referees made helpful suggestions to improve the manuscript.
Author Contributions
Conceived and designed the experiments: AHH JAPH RMC. Performed the experiments: FvR. Analyzed the data: FvR CB MAT DK NS. Wrote the paper: AHH FvR DK JAPH MAHB. Designed and performed panel sessions: FvR JAH. Collected and interpreted literature data: MAT MFML JWBvdG MAHB.
References
- 1. Walsh AL, Morgan D (2005) Identifying hazards, assessing the risks. Vet Rec 157: 684–687.
- 2. Palmer S, Brown D, Morgan D (2005) Early qualitative risk assessment of the emerging zoonotic potential of animal diseases. BMJ 331: 1256–1260.
- 3. Morgan D, Kirkbride H, Hewitt K, Said B, Walsh AL (2009) Assessing the risk from emerging infections. Epidemiol Infect 137: 1521–1530.
- 4.
Valenciano M (2001) Définition des priorités dans le domaine des zoonoses non-alimentaires. Paris, France: Institut de Veille Sanitaire.
- 5. Cardoen S, Van Huffel X, Berkvens D, Quoilin S, Ducoffre G, et al. (2009) Evidence-Based semiquantitative methodology for prioritization of foodborne zoonoses. Foodborne Pathog Dis 6: 1083–1096.
- 6. McKenzie J, Simpson H, Langstaff I (2007) Development of methodology to prioritise wildlife pathogens for surveillance. Prev Vet Med 81: 194–210.
- 7.
Mourits M, Oude Lansink AGJM (2007) Multi-criteria decision making to evaluate quarantaine disease control strategies. In: Oude Lansink AGJM, editor. New approaches to the economics of plant health. Heidelberg, Germany: Springer. pp. 131–144.
- 8.
Anonymous (2004) Report of the WHO/FAO/OIE joint consultation on emerging zoonotic diseases, 3–5 May, Geneva, Switzerland. WHO/FAO/OIE. 72 p. WHO/CDS/CPE/ZFK/2004.9 WHO/CDS/CPE/ZFK/2004.9.
- 9. Taylor LH, Latham SM, Woolhouse ME (2001) Risk factors for human disease emergence. Philos Trans R Soc Lond B Biol Sci 356: 983–989.
- 10.
Van der Giessen J, Isken LD, Tiemersma EW (2004) 99 p. Zoonoses in Europe: a risk to public health Bilthoven, the Netherlands: National Institute for Public Health and the Environment. 330200002 330200002.
- 11. Vorou RM, Papavassiliou VG, Tsiodras S (2007) Emerging zoonoses and vector-borne infections affecting humans in Europe. Epidemiol Infect 135: 1231–1247.
- 12. King LJ, Marano N, Hughes JM (2004) New partnerships between animal health services and public health agencies. Rev Sci Tech 23: 717–725.
- 13. Brown C (2004) Emerging zoonoses and pathogens of public health significance - an overview. Rev Sci Tech 23: 435–442.
- 14.
Tait J, Meagher L, Lyall C, Suk J (2006) Foresight. Infectious Diseases: preparing for the future. T2: Risk Analysis. London: Office of Science and Innovation.
- 15. Fauci AS (2005) Emerging and reemerging infectious diseases: the perpetual challenge. Acad Med 80: 1079–1085.
- 16.
Kurowicka D, Bucura C, Cooke R, Havelaar A (2010) Probabilistic inversion in priority setting of emerging zoonoses. Risk Anal: e-pub ahead of print.
- 17. Krause G (2008) Prioritisation of infectious diseases in public health - call for comments. Euro Surveill 13:
- 18.
Nesloo REJ, Cooke R (Accepted for publication) Modelling and validating stakeholder preferences with probabilistic inversion. Applied Stochastic Models in Business and Industry.
- 19.
Steinberg D, Colla PL (1995) CART: tree-structured non-parametric data analysis. San Diego, CA: Salford Systems.
- 20. Speybroeck N, Berkvens D, Mfoukou-Ntsakala A, Aerts M, Hens N, et al. (2004) Classification trees versus multinomial models in the analysis of urban farming systems in Central Africa. Agricultural Systems 80: 133–149.