Figures
Abstract
Background
The Foodborne Disease Burden Epidemiology Reference Group (FERG) was established in 2007 by the World Health Organization (WHO) to estimate the global burden of foodborne diseases (FBDs). This estimation is complicated because most of the hazards causing FBD are not transmitted solely by food; most have several potential exposure routes consisting of transmission from animals, by humans, and via environmental routes including water. This paper describes an expert elicitation study conducted by the FERG Source Attribution Task Force to estimate the relative contribution of food to the global burden of diseases commonly transmitted through the consumption of food.
Methods and Findings
We applied structured expert judgment using Cooke’s Classical Model to obtain estimates for 14 subregions for the relative contributions of different transmission pathways for eleven diarrheal diseases, seven other infectious diseases and one chemical (lead). Experts were identified through international networks followed by social network sampling. Final selection of experts was based on their experience including international working experience. Enrolled experts were scored on their ability to judge uncertainty accurately and informatively using a series of subject-matter specific ‘seed’ questions whose answers are unknown to the experts at the time they are interviewed. Trained facilitators elicited the 5th, and 50th and 95th percentile responses to seed questions through telephone interviews. Cooke’s Classical Model uses responses to the seed questions to weigh and aggregate expert responses. After this interview, the experts were asked to provide 5th, 50th, and 95th percentile estimates for the ‘target’ questions regarding disease transmission routes. A total of 72 experts were enrolled in the study. Ten panels were global, meaning that the experts should provide estimates for all 14 subregions, whereas the nine panels were subregional, with experts providing estimates for one or more subregions, depending on their experience in the region. The size of the 19 hazard-specific panels ranged from 6 to 15 persons with several experts serving on more than one panel. Pathogens with animal reservoirs (e.g. non-typhoidal Salmonella spp. and Toxoplasma gondii) were in general assessed by the experts to have a higher proportion of illnesses attributable to food than pathogens with mainly a human reservoir, where human-to-human transmission (e.g. Shigella spp. and Norovirus) or waterborne transmission (e.g. Salmonella Typhi and Vibrio cholerae) were judged to dominate. For many pathogens, the foodborne route was assessed relatively more important in developed subregions than in developing subregions. The main exposure routes for lead varied across subregions, with the foodborne route being assessed most important only in two subregions of the European region.
Conclusions
For the first time, we present worldwide estimates of the proportion of specific diseases attributable to food and other major transmission routes. These findings are essential for global burden of FBD estimates. While gaps exist, we believe the estimates presented here are the best current source of guidance to support decision makers when allocating resources for control and intervention, and for future research initiatives.
Citation: Hald T, Aspinall W, Devleesschauwer B, Cooke R, Corrigan T, Havelaar AH, et al. (2016) World Health Organization Estimates of the Relative Contributions of Food to the Burden of Disease Due to Selected Foodborne Hazards: A Structured Expert Elicitation. PLoS ONE 11(1): e0145839. https://doi.org/10.1371/journal.pone.0145839
Editor: Srinand Sreevatsan, University of Minnesota, UNITED STATES
Received: June 2, 2015; Accepted: December 6, 2015; Published: January 19, 2016
Copyright: © 2016 World Health Organization. This is an open access article distributed under the Creative Commons Attribution IGO License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. http://creativecommons.org/licenses/by/3.0/igo/, This article should not be reproduced for use in association with the promotion of commercial products, services or any legal entity.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: This study was commissioned and paid for by the World Health Organization. Copyright in the original work on which this article is based belongs to WHO. The authors have been given permission to publish this article. Aspinall & Associate and Gibb Epidemiology Consulting LLC provided support in the form of salaries for authors [WA, HJG], but did not have any additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The specific roles of these authors are articulated in the ‘author contributions’ section. The work by WPA was further supported by the United Kingdom Natural Environment Research Council [Consortium on Risk in the Environment: Diagnostics, Integration, Benchmarking, Learning and Elicitation (CREDIBLE); grant number NE/J017450/1]. This Council provided support in form of salary, but did not have additional role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: Co-author WA is the owner of Aspinall & Associates. Co-author HJG is the owner of Gibb Epidemiology Consulting. There are no patents, products in development or marketed products to declare. This does not alter the authors' adherence to all the PLOS ONE policies on sharing data and materials as detailed online in the guide for authors.
Abbreviations: FERG, Foodborne Disease Burden Epidemiology Reference Group; FBDs, Foodborne diseases; SATF FERG, Source Attribution Task Force; DOI, Declaration of Interests; SEJ, Structured Expert Judgment
Introduction
Foodborne diseases (FBD) are an important cause of morbidity and mortality worldwide [1, 2, 3]. The human health burden due to contaminated food is currently unknown. In order to fill this knowledge gap, the Foodborne Disease Burden Epidemiology Reference Group (FERG) was established in 2007 by the World Health Organization (WHO) to estimate the global burden of FBD.
Estimating the burden of FBD is complicated because most of the hazards causing foodborne disease are not transmitted solely by food. The relative impact of each route differs depending on the epidemiology of the disease causing microorganism (bacteria, virus or parasite) or chemical hazards. Other factors such as the geographical region, season, and food consumption patterns also influence the role of different exposures routes [4, 5]. The estimation of the burden of FBD, therefore, requires a delineation of the major transmission routes including contaminated food, water, soil, air, or contact with infected animals or humans. Previous efforts to quantify the contribution of specific sources (including types of foods) and transmission routes have been gathered under the term ‘source attribution’ or ‘human illness attribution’ [6, 7]. The applicability of available methods for source attribution of FBD at the global level was recently assessed by Pires [4].
Knowledge of the most important sources of exposure can foster better, more targeted control measures, and support risk managers in decisions on allocating resources to achieve the greatest public-health benefits. Source attribution is an important tool for identifying and prioritizing effective interventions to prevent and control FBD [8]. The need for reliable source attribution estimates has prompted a growing body of research focusing on attribution, particularly for infectious agents [4, 7, 9, 10]. However, comprehensive attribution studies based on surveillance data and/or food monitoring and exposure data are still limited in scope, and to date have been performed for a few hazards only or in a limited number of countries [11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]. In addition, existing studies have focused mainly on identifying specific food sources or animal reservoirs, whereas other potential transmission routes are often not quantified due to lack of data or neglected due to the complexity of attribution models. Many studies, often designed as randomized controlled intervention trials, have been conducted to assess the importance of water, particularly for the transmission of diarrheal diseases (reviewed by [24, 25]). However, other transmission routes, such as soil, air and direct contact with infected humans or animals, are generally not considered in those studies. Thus, for most countries, and at the global level, relevant studies and data for quantifying attribution of potential FBD to the major transmission routes do not exist.
In such situations, structured elicitation of scientific judgment may be used [4, 26]. When data are not available, or undertaking primary research is not feasible, a structured elicitation offers a transparent and mathematically rigorous way of evaluating and enumerating uncertainty distributions from the judgments of many individual researchers, for quantifying risk models. Scientific judgment elicitations have been used in several risk domains including climate change [27, 28, 29, 30], critical infrastructure [31], genetically modified organisms [32], and volcanic risks [33, 34, 35, 36]. Within food safety, the approach has been applied to provide national estimates for the proportion of illnesses attributable to food for specific infectious diseases [37, 38, 39, 40, 41, 42, 43, 44], or to inform modeling of foodborne disease risk assessment models by estimating specific model parameters and their uncertainty [45, 46].
In this paper, we describe an expert elicitation study conducted by the FERG Source Attribution Task Force (SATF) to estimate the relative contribution of food and other major transmission pathways to the global burden of diseases commonly transmitted by food. The resulting estimates were used as input to the disease burden framework developed by the FERG Computational Task Force [47] to estimate the global burden of FBD [48].
Materials and Methods
Overall, the study was designed to provide estimates of the percent of illness acquired through different major routes of exposure. Major exposure routes considered were: food, environmental (water, soil, air), human-to-human transmission, direct animal contact and a variety of potential lead exposure sources. Exposure route attribution estimates were developed for 19 individual hazards for each of the fourteen subregions (Table 1, Fig 1). Three hazard-based task forces within the FERG, the Enteric Diseases Task Force, the Parasitic Diseases Task Force, and the Chemicals and Toxins Task Force identified, from their prioritized lists of hazards, those to be included in the expert elicitation [49, 50, 51]. Certain hazards were considered 100% foodborne, i.e., Listeria monocytogenes, Mycobacterium bovis, all foodborne trematodes, Taenia solium, Trichinella spp., cyanide in cassava and peanut allergens. For aflatoxin, inorganic arsenic, cadmium, dioxin, and methyl mercury, the Chemicals and Toxins Task Force determined that adequate data on foodborne exposure existed to use a risk assessment approach for estimating the foodborne disease burden, thus negating the need for attribution. The remaining hazards were included in the structured expert elicitation (Table 1).
Several experts had experience in more than one subregion. The subregions are defined on the basis of child and adult mortality as described by Ezzati et al. [63]. Stratum A: very low child and adult mortality, Stratum B: low child mortality and very low adult mortality, Stratum C: low child mortality and high adult mortality, Stratum D: high child and adult mortality, and Stratum E: high child mortality and very high adult mortality. AFR = African Region; AMR = Region of the Americas; EMR = Eastern Mediterranean Region; EUR = European Region; SEAR = South-East Asia Region; WPR = Western Pacific Region. A list of countries per subregions can be found in Havelaar et al. (48). The use of the term ‘subregion’ here and throughout the text does not identify an official grouping of WHO Member States, and the ‘subregions’ are not related to the six official WHO regions.
Identification of experts
An iterative peer nomination process based on a professional network sampling technique, sometimes called ‘snowball sampling’ was used to identify a pool of potential expert participants for this study. The first points of contact were identified through FERG members and other networks (e.g. Global Foodborne Infections Network—GFN, Global Environment Monitoring System—GEMS, International Network of Food Safety Authorities—INFOSAN, Joint FAO/WHO Expert meeting on Microbial Risk Assessment—JEMRA, Joint FAO/WHO Expert Committee on Food Additives—JECFA, European Food Safety Authority’s (EFSA) scientific panels, and WHO regional food safety advisors). These persons were asked to use their professional contacts and recognized expertise in relevant areas to nominate additional experts. Since the purpose of this process was to identify an adequately large pool of appropriate experts, rather than to identify the entire expert network, the process of referral continued until an adequate size pool was identified to fill panels of typically 8 to 12 experts per panel.
Selection of experts
In collaboration with the hazard-based task forces of the FERG, the SATF defined a set of criteria for inclusion of experts. These criteria considered the experts’ background (education, and current and past positions), years of experience within the field and geographic coverage of expertise within the panel. The WHO invited nominated experts to participate, and the experts were asked to complete a declaration of interests (DOI) and an expert sheet providing information on their research/working area, highest education, current position, geographical experience, and years of experience. The experts were asked also to indicate which panel(s) they believed themselves suited for. The experts were not offered any compensation for their participation. The chairs of the three hazard-based task forces and the SATF reviewed the experts’ information and CVs and a final selection was made. FERG task force chairs and members of the SATF were not eligible for the study. DOI’s were evaluated by WHO.
Given the broad nature of the attribution task, care was taken to include a suitably wide range of scientific backgrounds and professional experience, and to ensure adequate geographical representation. Frequently, expert elicitations use publication record as the measure of recognized expertise [52]. However, in this study, restricting expert selection to choices based solely on publication records would have eliminated important groups of experts, in particular public-health field workers and food-safety professionals in developing countries.
Expert panels
The panels for Brucella spp., Hepatitis A virus, parasitic diseases including intestinal protozoa and lead were structured as global panels, meaning that all experts in those panels were asked to provide estimates for all fourteen subregions (Table 1, Fig 1). The panels for the eight bacterial and viral pathogens causing diarrhea and Salmonella Typhi were structured as sub-regional panels, i.e., separate panels were created for each subregion. In general, experts on all panels were free to decide the subregions and/or hazards for which they provided their judgments.
Analytical method
The study used Cooke’s Classical Model for expert elicitation [52, 53, 54]. This approach uses calibration or seed questions to develop performance weights used in aggregating experts’ judgments. The paradigmatic seed question is one for which the true value is not known at the time the experts answer the question, but will be known or is expected to become known post hoc. Thus the experts are not expected to know these values, but should be able to capture a majority of them reliably by defining suitable credible intervals.
Analysis of the experts’ performance on the seed variables has two main purposes: 1) to evaluate the expert’s statistical accuracy when assigning values to probability outcomes against the seed values (i.e., how reliably the expert’s credible interval responses capture the true values of the seed variables, statistically), and 2) to evaluate the expert’s informativeness when providing uncertainty distributions over the seed variables (i.e. how concentrated (narrow) are the distributions provided). Experts are thus scored with regard jointly to statistical accuracy (calibration score) and informativeness (information score). The statistical accuracy is measured as the p-value at which one would falsely reject the hypothesis that the expert’s probability assessments were statistically accurate), and informativeness is measured as Shannon relative information with respect to a user supplied background measure. Informativeness scores are not absolute, but relative to a set of experts assessing the same variables. The calculated calibration and information scores are used to aggregate experts’ judgments on target variables. The same measures can be applied to any combination of the experts' assessments to implement criteria for aggregating the assessments.
Cooke’s Classical Model provides a rigorous, quantitative means for estimating model parameters and their uncertainties and is the only elicitation procedure that has objective empirical control on expert scoring. Moreover, it allows formal optimization of aggregated uncertainty distributions in terms of statistical accuracy and informativeness [53]. The expert judgment processing software EXCALIBUR (http://www.lighttwist.net/wp/excalibur) also allows direct comparison of the results that would be obtained from unweighted aggregation of expert judgments with those produced by weighted linear pooling (or other combination schemes). For a more detailed description of the Classical Model see (S1 File) that also provide a list of references for different applications.
Seed questions
It is not always possible to develop seed questions that are in the paradigmatic form of asking about a future event or measurement that has not been made, but could be made, in principle. The essential feature of a viable seed question is that the expert is not expected to know the exact value but, if they are a subject-matter expert, should be able to define a narrow uncertainty range that captures the value. Therefore, an alternative is to ask about selected data or values in the topic domain, about which the expert will not have perfect knowledge, nor access to realization values at the time they are answering the seed questions–but, for which the values are known to the analyst. Such retrospective questions are frequently used in expert elicitations applying the Cooke Classical Model (see e.g. [30, 38]).
In the present case, the seed questions we formulated were a mixture of retrospective and prospective seed variables. It is possible that expert uncertainty judgments vary by subject matter domain. In this study, the possibility of such biases relevant to foodborne illness source attribution was of concern. Therefore, the seed questions were designed to focus on questions that are substantively related to foodborne illness source attribution. Further, to account for the wide range of scientific backgrounds and experiences, seed questions covered a range of substantive topics relevant to source attribution. Five main categories of seed questions were identified for the panels on biological hazards (diarrheal pathogens and parasites): 1) dietary patterns and food supply; 2) under 5 years mortality rate; 3) access to improved water and sanitation; 4) disease surveillance, and 5) systematic reviews related to these and other scientific topics relevant to source attribution. For the panel on lead, questions were categorized as: 1) mean blood levels; 2) dietary exposure, and 3) dietary patterns and food supply. Examples of seed questions are presented in Table 2. All seed questions are provided in (S2 File).
There were eight sets of seed questions with some overlapping questions for the biological hazards (S2 File), and a separate set for lead. This allows some consistency checks to be performed between panels on performance and scoring outcomes. The number of questions varied from ten to twelve per set of seed questions. Experts were asked to provide a central judgment in terms of the median value, and a 90% credible interval for each question. Seed questions were administered by facilitators through one-to-one telephone interviews. The experts were not presented with the seed questions before the interview and they were asked to provide estimates based on their experience, knowledge and judgment, without referring to other sources of information.
Target questions
Target questions are the substantive questions of interest. In this study, for all identified hazards, we enquired about the percentage of all human disease cases caused by exposure through each of a number of indicated exposure routes. The point of exposure was chosen as the point-of attribution i.e. the experts were asked to distribute the disease burden on the sources that were the direct cause of human exposure. So for example, someone with a Norovirus infection might be exposed by eating food contaminated with the virus, although the food may have been contaminated by waste water at an earlier stage.
In order to reduce the time and effort burden of the elicitation on expert panelists, the hazard-based Task Forces decided which hazard exposure routes were relevant for present purposes (Table 3). For example, human-to-human transmission was excluded as an exposure route for Brucella spp. However, the questionnaires did provide experts with an option to indicate additional routes of transmission in case they disagreed with the Task Force’s evaluation.
Hazards were grouped according to common exposure routes. na: not applicable, meaning that these exposure routes were considered not possible or extremely unlikely by the respective FERG task forces.
Experts were asked to complete a set of tables for each assigned hazard and subregion. They were provided with these tables at the end of the telephone interview during which the seed questions were asked, and the facilitator went through several target questions with the experts to ensure that they understood the task. For the target items (but not the seed questions), the experts were free to consult any information sources they felt appropriate in the four-week period they were given to return the target item spreadsheets.
As with the seed questions, the experts were asked to provide their 5th, 50th and 95th percentile values for each question. Technically, the median values of a joint distribution do not need to add up to 100%. It is the associated mean values that should respect this criterion. However, the Classical Model operates on the basis of probability quantiles and experts are asked to give median values as central tendency indicators. Because we included a category ‘Other’, we thus asked about a joint distribution that logically spanned all possible exposure routes and—to be coherent therefore—the experts’ median values for source attribution percentages for a hazard should, in total, produce a value that does not grossly deviate from about 100%. In individual cases, where these sums were found to differ significantly from 100% (i.e. outside 100% +/- 10%), the experts concerned were asked to review their responses.
Data analysis
Weights for individual experts were computed under the Classical Model formulation using the EXCALIBUR software by multiplying their calibration and informativeness scores, with the products then jointly normalized to sum to unity over all experts in the group. An expert was positively weighted only if his/her p-value was above a certain threshold, chosen to optimize the combined score across all seed items. In most applications of the Classical Model, a counterpart equally-weighted combination of experts’ distributions is also derived, typically for comparison with the corresponding performance-weighted solution. A detailed evaluation of the performance-based approach for this study has been described by Aspinall et al. [55]. See also the SI-1, Cooke [53], and Cooke et al. [56] for further details on the computation of expert performance weights.
The performance weighted attribution estimates were given in terms of the 5th, 50th, and 95th % percentiles. A minimally informative density was fit to the uniform background measure which complies with these quantile constraints. First, for each exposure category within a hazard-subregion combination, independent vectors of 10,000 random deviates were generated from a Uniform(0, 100) distribution. The quantiles corresponding to these random deviates were then obtained via linear interpolation. As all possible exposure routes were included in the target questions (per definition by including an ‘Other’ option) and the exposure routes were considered mutual exclusive, it was necessary to ensure that the random attribution proportions summed to 100%. Therefore, a ‘normalization’ step was applied at each iteration, in which each random value was divided by the sum of random values for each exposure pathway. More precisely, if f1, …, fk are relative frequencies which must sum to one, f1, …, fk are first sampled independently, and then on each sample, fi is replaced by . The 10,000 normalized random attribution proportions were then summarized by their median and a 95% uncertainty interval defined by the 2.5th and 97.5th percentiles. The resulting joint distributions (i.e., attributable proportion of illness per pathway, subregion and hazard) satisfy the ‘sum to 100%’ constraint which is closest to the product of the margins [57]. Data simulations were performed in R 3.1.1 [58] using functions available in the 'FERG' package [47].These distributions were applied by the hazard-specific Task Forces to estimate the burden of disease through the foodborne exposure route by multiplying the vectors of randomly-selected values for these parameters with a vector of randomly-selected values for the proportion that is foodborne, as described by Devleesshauwer et al. [47] and presented by Havelaar et al. [48].
Results
A total of 299 potential experts were asked by email of their interest to participate in the study. Of these154 replied positively and they were requested to forward their CV, a filled in expert sheet and a signed declaration of interest. A total of 103 did that. Of these, 3 were not included due to lack of experience (n = 1) or possible conflicts of interest (n = 2). Of the 100 experts enrolled, 78 completed interviews with facilitators and 73 returned their spreadsheets with their responses to the target questions and seed questions. The single main reason for not completing the interview and returning the spreadsheet was time constraints. All responses were reviewed (i.e. checked for missing estimates, that sums across pathways were close to 100%, and that the 5th percentile < 50th percentile < 95th percentiles, etc.) and some experts were contacted for clarification of the responses they had provided. One expert was dropped after not responding to requests for clarification. This resulted in the responses of 72 experts being included in the final dataset. Table 4 shows the distribution of experts across panels, and Fig 1 shows distribution of the experts by their geographical areas of expertise.
Expert performance
Due to the design of the study, having nine sets of seed questions, with 14 subregions and with some experts able to provide judgments only for certain regions or for some hazards, in the end there were 112 distinct datasets of expert elicitations (i.e. datasets that differed in membership or seed questions); in many cases, datasets differed only with respect to a few expert members. Overall, performance weight and equal weight combinations showed acceptable statistical accuracy. Only in the case of the dataset considering lead was the p-value of the performance-based combination small enough to cast doubt on the usual criterion for statistical accuracy, with p = 0.045 (i.e. less than 0.05 criterion). If the panels were independent (which they are not due to expert overlap) and statistically accurate, we should expect six panels’ p-values to fall below 5%.
Results obtained by applying equal weights pooling and performance weights pooling were compared. The equal weights solutions tended to have higher statistical accuracy than those produced by applying the performance weights. In contrast, the informativeness properties of the equal weights solutions were much lower than those provided by performance weights solutions (Fig 2). This ‘trade-off’ between accuracy and informativeness when applying equal weights or performance weights is often seen, because the least accurate experts are typically the most informative, and their narrow 90% confidence bands often have little or no overlap. Moreover, the combined score using performance-based weights was above that of the equal weights pooling in 62% of the cases. It was, therefore, decided to use the performance weights combinations for constructing the joint probability distributions for the pathway attribution estimates as long as the statistical accuracy was acceptable. It should also be noted that the weight attributed to an expert–comprising the normalized product of his or her two scores–is dominated by the accuracy term, so that high informativeness cannot buy down poor accuracy. A unique feature of the present study is that a large number of experts were assessed using very similar variables, thereby allowing their informativeness scores to be compared. However, an in-depth analysis of the experts’ performance is outside the scope of the present paper, but is described elsewhere [55].
Pathway attribution results
The collective results of the performance-based weighted expert responses are shown in Tables 5–9. We grouped the hazards according to the investigated transmission pathway to ease the presentation and discussion of results, following the order presented in Table 3. Individual expert responses can be found in (S3 File).
Overall, and for most estimates, there is considerable uncertainty, reflecting: 1) variations in uncertainty estimations between individual experts; 2) that, for some hazards, the values provided by experts having high performance weights in the analysis did not accord with one another, and 3) that, for some subregions or hazards, the number of contributing experts was small (<7). Thus, the broad uncertainty intervals are most likely reflecting current shortcomings in hard scientific evidence about the relative contribution to human disease from each of the transmission pathways.
Fig 3 shows the subregional estimates of the foodborne proportion for Campylobacter spp., non-typhoidal Salmonella spp., Shiga toxin-producing Escherichia coli (STEC), Brucella spp. and Shigella spp. For Salmonella spp. and Brucella spp., there is a clear pattern that the foodborne proportion is considered more important in the developed subregions (AMR A, EUR A and WPR A, see Fig 1 for subregions) compared to developing subregions. Although less distinct, this pattern can also be seen for Campylobacter spp. and STEC. For Campylobacter spp., Salmonella spp. and STEC the foodborne transmission route was assessed by the experts to be the most important route in all subregions, followed by direct animal contact, human-to-human transmission and waterborne transmission in varying order, but generally, with medians below 0.25 (Table 5). It is notable that in all subregions, over half of campylobacteriosis was deemed to be foodborne (Table 5). For Brucella spp., direct animal contact was considered equally or more important than foodborne transmission in developing subregions. Human-to-human transmission was considered the most important route for Shigella spp. in the majority of subregions. Proportion foodborne Shigella spp. infections ranged from 0.07 (95%CI: 0.00–0.46) in EUR A to 0.36 (95%CI: 0.01–0.70) in WPR A (Table 5). Overall, foodborne transmission was assessed to be more important in South-East Asian and Western Pacific subregions than in other parts of the world. Transmission through soil or other routes was recognised by the experts to be of minor importance for these five pathogens.
Indicated on the line plot are the 2.5th, 5th, 50th, 95th and 97,5th percentiles.
Fig 4 shows the subregional estimates of the proportion foodborne for enteropathogenic E. coli (EPEC), enterotoxigenic E. coli (ETEC), Cryptosporidium spp. and Giardia spp. The estimates for EPEC are seen to follow the same pattern as described above with the foodborne route being assessed to be more important in developed subregions. In developing subregions in the African, American and Eastern Mediterranean regions (AFR, AMR and EMR), water was identified as the most important transmission route. For ETEC, the estimated foodborne proportions were quite similar for all subregions with medians ranging from 0.33 to 0.43 (Table 6), but the foodborne route was only assessed by experts to be the more important route in European subregions. For Cryptosporidium spp. and Giardia spp., the foodborne proportions were also quite similar across subregions, but generally considered less important with medians below 0.20 (Table 6). Human-to-human and waterborne transmission were the more important routes for these infections in all subregions.
Indicated on the line plot are the 2.5th, 5th, 50th, 95th and 97,5th percentiles.
Fig 5 shows the subregional estimates of the proportion foodborne for Salmonella Typhi, Vibrio cholerae, Entamoeba histolytica, Norovirus, and Hepatitis A virus. Overall, foodborne infections were not assessed by the experts to be the more important routes in the majority of subregions. Exceptions were Hepatitis A infections, where foodborne and human-to-human transmission were evaluated equally important in most subregions, and S. Typhi, where foodborne and waterborne infections were assessed equally important in SEAR and WPR regions (Table 7). Human-to-human transmission was identified as the main exposure route for Norovirus and E. histolytica in most subregions, whereas waterborne transmission was estimated to be the main transmission route for V. cholerae infections (Table 7).
Indicated on the line plot are the 2.5th, 5th, 50th, 95th and 97,5th percentiles.
Fig 6 shows the subregional estimates of the proportion foodborne for Toxoplasma gondii, Echinococcus multilocularis, Echinococcus granulosus and Ascaris spp. The foodborne route was assessed by the experts to be the most important transmission route for T. gondii and Ascaris spp. in most subregions, but there was a clear tendency for soil to increase in relative importance in less developed subregions (subregions D and E) (Table 8). Specifically for Ascaris spp., the foodborne route was assessed to be particularly important in developed subregions (subregions A). There was only little geographical variation between the median estimates for each of the transmission pathways for the two Echinococcus species. For E. granulosus, animal contact was clearly believed to be the most important route with medians just over 0.50. For E. multilocularis, the foodborne route was considered most important with medians ranging from 0.43 in EMR B to 0.58 in AFR D and E, AMR B and D, and SEAR B and D, but the estimates had very large uncertainty ranges (Table 8).
Indicated on the line plot are the 2.5th, 5th, 50th, 95th and 97,5th percentiles.
Fig 7 shows the subregional estimates of the foodborne proportion for lead exposure. Water, food and air exposure were the main transmission routes indicated by the experts with some subregional differences (Table 9). The foodborne route was assessed to be the most important only in two subregions in Europe. Air was assessed to be the main exposure route in seven of the 14 subregions and water in four regions. Soil, paint, cookware/pottery/glassware and toys were in comparison only found to be of minor importance in the majority of subregions (Fig 8).
Indicated on the line plot are the 2.5th, 5th, 50th, 95th and 97,5th percentiles.
Discussion
In this study, we present the results of the first world-wide study on the contribution of contaminated food and other exposure routes to human disease caused by 18 major microbiological hazards and a chemical hazard. The study highlights the importance of the foodborne route of transmission for these hazards and–when combined with estimates of incidence, severity, duration and mortality–allows estimation of the burden of foodborne disease [48]. Attempting to estimate foodborne transmission at the subregional level is an ambitious goal. However, this was vital given the geographically localized nature of exposure to many pathogens. These new findings are important due to the global nature of the estimation, the number of experts participating, and the rigorous approach taken to assessing and including expert performance in the final estimates.
We were unable to identify epidemiological studies in the literature that delineate and quantify the importance of each transmission pathway as investigated in this study. This makes it difficult to formally validate the outcomes of the expert elicitation. Still, a discussion of summary findings in the context of other scientific knowledge may be of value.
The hazards can be grouped in several categories with respect to their major pathways. For Campylobacter spp., non-typhoidal Salmonella, STEC, T. gondii, and E. multilocularis, the foodborne route was considered the most important route in all subregions. These pathogens are all zoonotic and known to have one or more animal reservoirs. The zoonotic nature of these organisms is also reflected in experts’ judgments by the identification of direct contact with animals as an important transmission route as well. Other pathogens with animal reservoir include E. granulosus and Brucella spp., and here direct contact with animals was considered equally or more important than food as routes of transmission.
As described in the results section for several pathogens, there was a clear pattern that the experts considered the foodborne route less important in low- and middle-income subregions, where other routes (animal contact, water and soil) were believed to contribute relatively more when compared to high-income subregions. This is consistent with data showing lower levels of access to improved water and sanitation in less developed regions as compared to high-income countries. This ranking of subregions across different pathogens provides some confidence in the results, as the estimates were done independently and partly by different experts.
An expert elicitation was used to estimate source attribution parameters not only because of the lack of globally consistent data on which to base such estimates, but also a general lack of data and research on source attribution in most parts of the world. The generally wide uncertainty bounds provided by the expert elicitation in this study are presumed to reflect both uncertainty and variation, where uncertainty arises due to the epistemic sparseness of hard evidence data, or the presence of conflicting evidence, on the contribution of different transmission pathways, and variation reflects the experts’ beliefs on aleatory variations between countries within any given subregion. A study operating with smaller regions or at country level may have reduced the uncertainty due to variation.
There exist a few recent national studies that estimate the proportion of illnesses attributable to the foodborne route for specific infectious diseases [40, 41, 42, 44, 59, 60]. Table 10 provides the main results from these studies. Four of the six studies used some kind of formal expert elicitation, where enrolled experts were asked to provide a central estimate and their uncertainty bounds around this. The estimates published by Gkoga et al. [59] and Scallan et al. [60] were derived by the authors by a synthesis of data from different public health surveillance systems and the literature. Studies also differed on the following two aspects that may explain some of the differences seen: 1) whether the study only addressed domestically acquired cases, and 2) whether more exposure routes than just the foodborne route were addressed. For illnesses with a high proportion of travel-related illness, and where the relative importance of exposure routes differs between domestically acquired and travel-related illness (e.g. illnesses caused by S. Typhi and V. cholera), the attribution estimates must be expected to differ between studies including travel-related illness and those only addressing domestically acquired illness. Only two of the national studies included more exposure routes than the foodborne [40, 41]. Asking experts to assign attribution estimates to all potential exposure routes and not just the foodborne has been discussed to result in relative lower estimates for the foodborne route [40], because experts are faced with providing estimates (and uncertainty bounds around these) for exposure routes that they may not otherwise have considered important, if they had only been asked about the foodborne route. Although, no overall trend of this can be seen from Table 10, it may be the case for hazards in regions that typically are thought of as being associated primarily with foodborne exposure (e.g. non-typhoidal Salmonella spp. and Campylobacter spp.). Finally, this study differed from the national studies by eliciting regional estimates, which could be interpreted as a weighted average across all countries in the subregion and thereby differ from country-specific estimates from the same subregion. As all six national studies were conducted in developed countries, we compare the results only with the results from the relevant subregions (i.e. EUR A, AMR A and WPR A) in this study.
For the zoonotic pathogens, particularly non-typhoid Salmonella spp., the estimates are agreeing better and the uncertainty ranges tend to be relatively narrower than for pathogens with a human reservoir (e.g. Hepatitis A virus, S. Typhi and V. cholerae). As discussed above some of the differences observed may occur because we are comparing studies including only the foodborne route with studies including more exposure routes.
For Hepatitis A virus there is a strong divregence between the national studies and this study, where the proportion foodborne is estimated to be less, but at the same level in the four national studies that investigated this pathogen. This difference cannot be readily explained, but it should be noted that there was also disagreement between the experts in this study, where three of the six experts serving on the Hepatitis A virus panel provided estimates in line with those published for the national studies (see S3 File), whereas the remaining three experts provided considerably higher estimates. The differences could not be associated with the experts’ country of residence. The divergence between the experts is reflected in the uncertainty bounds for the estimates, which are quite wide and encompass the estimates from the national studies.
For S. Typhi and V. cholerae, the estimates from the national studies are higher than those found in this study. One explanation could be that the national studies [42, 59, 60] were only attributing domestically acquired cases. In the study by Scallan et al. [60], the proportion foodborne for S. Typhi was estimated based on data from 17 domestic outbreaks reported in a 19-year period, where 13 outbreaks were confirmed foodborne and 4 outbreaks were of unknown origin. The same study included also only data from domestically acquired cases of V. cholerae, and as around 70% of all V. cholerae and 67% of all S. Typhi cases were estimated to be travel related, including all cases could decrease the proportion of foodborne illness significantly. Infections with S. Typhi and V. cholerae are typically linked with contaminated water sources and poor hygiene in developing regions [24], thus these transmission routes are likely to be relatively more important among cases that have been travelling to these regions. This factor is probably reflected in the attribution results in this study.
The operational definition of different transmission routes, in particular the food and waterborne routes, will affect attribution estimates. Hazards transmitted by multiple routes can ‘change’ source or vehicle during the transmission from primary source to humans, meaning that the burden of illness caused by a particular hazard attributed to a specific transmission route may vary, depending on the point of attribution chosen [4, 7]. The choice of point of attribution seems particularly critical for delineating foodborne and waterborne diseases. This is because water is itself ingested just as a food, is used for irrigation of food plants, for washing and cleaning of food during preparation and constitutes an essential ingredient in many food products. In addition and particularly related to zoonotic diseases, the water source is often contaminated by an animal reservoir, including food-producing animals. Another situation relates to the consumption of water-based foods such as shellfish harvested from areas where the water contains pathogenic organisms, such as Vibrio spp. or enteric viruses. The burden related to the consumption of foods that have had contact with contaminated water at some stage of the production may, therefore, be attributed to either food or water depending on the point of attribution. It is clear that there exists no single ‘right’ way of delineating the foodborne route from other transmission routes; however, it is critical that point of attribution be clearly defined. That definition should depend on the objective and focus of the specific study, and could involve many factors, including the foodborne hazard in question, the food production systems and processing routines, the geographical occurrence of the hazard, sanitation and hygiene in the region, and consumption patterns. If the point of attribution is clearly defined, then additional modeling or further research can be used to adjust attribution to exposure at other points of interest in the transmission chain. If the point of attribution is not unambiguously defined, then not only are the results of the study unclear, but they will be difficult to use to model other relationships in the transmission chain.
FERG agreed that the point of human exposure was the most simple and understandable point of attribution to be used across all hazards for delineating the major transmission routes (Fig 9). FERG recognises that for other purposes, e.g. for risk management, other points of attribution may be more appropriate (e.g. primary production, processing and retail, or preparation). This said, the FERG definition of point of attribution for major transmission routes directly links attribution to disease incidence, as it is the end of the transmission chain. Further modeling can then be used to work backward from exposure to identify the important points of contamination.
In this study, we identified potential experts through peer-nomination, for the purpose of enumerating parameter uncertainties through structured expert judgment. It is important to recognize that the goal of a structured expert judgment is not to characterize the characteristics of the group of experts in some sense, but to obtain uncertainty quantifications of target variables which are statistically accurate and informative. The degree to which this goal is realized is assessed objectively by referencing elicited target uncertainties to the experts’ performances in judging similar, factual realizations of calibration variables in the subject matter field. This empirical validation of experts' (and combined experts') uncertainty quantifications is what distinguishes the Classical Model approach from surveys or statistical sampling [61]. When compared with the usual markers of professional qualifications, such as e.g. publication records [62], a formalized assessment of the uncertainty judgment capabilities of experts using calibration variables has also been demonstrated to be a better predictor of expert performance. Structured expert judgment studies are, therefore, not as sensitive to selection bias and low response rates as other types of surveys. In our study, we approached 299 potential experts and ended up with a final pool of 72 (24%), which is actually a fair response rate compared to population surveys, for example. The response rate of those specialists—confirmed as having appropriate expertise–who committed to participate in the study by forwarding their CV, DOI, etc., was 70%. The motives for some people declining to participate were not specifically asked for, but most of those giving a reason indicated lack of time. A few declined because they did not perceive themselves experts in the field. The reason given by the majority for not finally submitting responses to the target questionnaire, even though they had gone through the interview, was also time constraint. Although some selection bias cannot be ruled out absolutely, we do not believe that it has had major impact on the study results due to the formalized basis of the elicitation process and objective judgement pooling methodology.
Future research into source attribution should focus on developing more data-driven approaches, combining information from different sources in a single analytical framework. Ideally, this would include intervention studies examining the contribution of food to different diseases, along with source attribution studies based on highly specific genomic information from pathogens. To provide more consistent and comparable data, it is important that there is international harmonization of definitions, groupings, protocols, and data analysis methods.
Conclusion
We provide for the first time global estimates for the proportions of diseases due to selected hazards that are attributable to food and to other major transmission routes. These estimates are essential for quantifying disease burdens within the FERG framework, and facilitates the estimation of the global burden of foodborne diseases [48] While some gaps exist, we believe the estimates presented here are the best current source of guidance to support decision makers when allocating resources for control and intervention, and for future research initiatives.
Supporting Information
S1 File. Supporting information on the Cooke’s Classical Model.
2015 Jan 14.
https://doi.org/10.1371/journal.pone.0145839.s001
(PDF)
S2 File. Seed question for microbiological and chemical hazards.
https://doi.org/10.1371/journal.pone.0145839.s002
(PDF)
S3 File. Data file including anonymised responses from all experts.
The Excel file includes a sheet for each hazard.
https://doi.org/10.1371/journal.pone.0145839.s003
(XLSX)
Acknowledgments
The authors are very grateful to the specialists who contributed their expertise and time to this study. We also thank our facilitators, who conducted the expert interviews and collected the data: Charlene Bayemi, Anca Hanea, Taher Jamshidi, Vikram Rao, Marshall Kuypers, Alessandra Lafranconi, Anuj Jindal, Sarvnaz Modaressi, Lohendy Muñoz, Jessica Kratchman, and Fjordor Melnikov. We would also like to acknowledge the assistance of the WHO Secretariat over the life of the FERG initiative, particularly Amy Cawthorne, Claudia Stein, and Yuki Minato.
Author Contributions
Conceived and designed the experiments: TH WA RC AH HG PT MK FA RL SH. Performed the experiments: TH TC SH. Analyzed the data: TH WA RC BD NS SH. Wrote the paper: TH WA BD RC AH MK FA RL NS SH.
References
- 1. Stein C, Kuchenmuller T, Hendrickx S, Pruss-Ustun A, Wolfson L, Engels D, et al. The Global Burden of Disease assessments—WHO is responsible? PLoS neglected tropical diseases. 2007;1(3):e161. pmid:18160984. Pubmed Central PMCID: 2154395.
- 2. Tauxe RV, Doyle MP, Kuchenmuller T, Schlundt J, Stein CE. Evolving public health approaches to the global challenge of foodborne infections. International journal of food microbiology. 2010 May 30;139 Suppl 1:S16–28. pmid:19931203.
- 3. Chan M, 2014. Food safety must accompany food and nutrition security. The Lancet. Published online 2014 Nov 19. Available: http://dx.doi.org/10.1016/S0140-6736(14)62037-7.
- 4. Pires SM. Assessing the Applicability of Currently Available Methods for Attributing Foodborne Disease to Sources, Including Food and Food Commodities. Foodborne pathogens and disease. 2013; 10(3): 206–13. pmid:23489045
- 5. Mangen MJ, Batz MB, Käsbohrer A, Hald T, Morris JG Jr, Taylor M, Havelaar AH. Integrated Approaches for the Public Health Prioritization of Foodborne and Zoonotic Pathogens. Risk Analysis. 2010 May; 30(5):782–97. pmid:19765248
- 6. EFSA (European Food Safety Authority), 2008. Scientific Opinion of the Panel on Biological Hazards on overview of methods for source attribution for human illness from food-borne microbiological hazards. The EFSA Journal. 2008; 764: 1–43.
- 7. Pires SM, Evers EG, van Pelt W, Ayers T, Scallan E, Angulo FJ, Havelaar AH, Hald T. Attributing the human disease burden of foodborne infections to specific sources. Foodborne pathogens and diseases. 2009 May; 6(4): 417–424.
- 8. Havelaar AH, Bräunig J, Christiansen K, Cornu M, Hald T, Mangen MJM, Mølbak K, Pielaat A, Snary E, van Pelt W, Velthuis A, Wahlström H. Towards an Integrated Approach in Supporting Microbiological Food Safety Decisions. Zoonoses and Public Health. 2007; 54: 103–117. pmid:17456140
- 9. EFSA BIOHAZ Panel (EFSA Panel on Biological Hazards). Scientific Opinion on the evaluation of molecular typing methods for major food-borne microbiological hazards and their use for attribution modelling, outbreak investigation and scanning surveillance: Part 1 (evaluation of methods and applications). EFSA Journal. 2013;11(12):3502, 1–84.
- 10. Pires SM, Vieira AR, Hald T, Cole D. Source attribution of human salmonellosis: An overview of methods and estimates. Foodborne pathogens and disease. 2014 Sept; 11(9): 667–76. Epub 2014 Jun 2. pmid:24885917
- 11. Hald T, Vose D, Wegener HC, Koupeev T. A Bayesian Approach to Quantify the Contribution of Animal-Food Sources to Human Salmonellosis. Risk Analysis. 2004 Feb; 24(1): 251–65.
- 12. Evers E G, van der Fels-Klerx HJ, Nauta M J, Schijven JF, Havelaar AH. Campylobacter source attribution by exposure assessment. International Journal of Risk Assessment and Management. 2008; 8(1–2): 174–190.
- 13. Wilson DJ, Gabriel E, Leatherbarrow AJ, Cheesbrough J, Gee S, Bolton E, Fox A, Fearnhead P, Hart CA, Diggle PJ. Tracing the source of campylobacteriosis. PLoS Genetics. 2008 Sep 26; 4(9).
- 14. Mullner P, Jones G, Noble A, Spencer SE, Hathaway S, French NP. Source attribution of food-borne zoonoses in New Zealand: a modified Hald model. Risk Analysis. 2009; 29: 970–984. pmid:19486473
- 15. Pires SM, Vigre H, Makela P, Hald T. Using Outbreak Data for Source Attribution of Human Salmonellosis and Campylobacteriosis in Europe. Foodborne pathogens and disease. 2010; 7(11):1351–1361. pmid:20586609
- 16. Guo C, Hoekstra RB, Schroeder CM, Pires SM, Ong KL, Hartnett E, et al. Application of Bayesian techniques to model the burden of human salmonellosis attributable to US food commodities at the point of processing: Adaptation of a Danish model. Foodborne pathogens and disease. 2011; 8:509–516. pmid:21235394
- 17. Little CL, Pires SM, Gillespie IA, Grant K, Nichols GL. Attribution of human Listeria monocytogenes infections in England and Wales to ready-to-eat food sources placed on the market: adaptation of the Hald Salmonella source attribution model. Foodborne pathogens and disease. 2010; 7: 749–756. pmid:20156087
- 18. Pires SM, Vieira AR, Perez E, Lo Fo Wong DMA, Hald T. Attributing human foodborne illness to food sources and water in Latin America and the Caribbean using data from outbreak investigations. International Journal of Food Microbiology. 2012; 152 (3): 129–138. pmid:21570732
- 19. Domingues AR, Pires SM, Halasa T, Hald T. Source attribution of human salmonellosis using a meta-analysis of case-control studies of sporadic infections. Epidemiology and infection. 2012a; 140(6): 959–969.
- 20. Domingues AR, Pires SM, Halasa T, Hald T. Source attribution of human campylobacteriosis using a meta-analysis of case-control studies of sporadic infections. Epidemiology and infection. 2012b; 140(6), 970–981.
- 21. Painter JA, Hoekstra RM, Ayers T, Tauxe RV, Braden CR, Angulo FJ, Griffin PM. Attribution of Foodborne Illnesses, Hospitalizations, and Deaths to Food Commodities by using Outbreak Data, United States, 1998–2008. Emerging infectious diseases. 2013; 19(3), 407–415. pmid:23622497
- 22. Boysen L, Rosenquist H, Larsson JT, Nielsen EM, Sorensen G, Nordentoft S, Hald T. Source attribution of human campylobacteriosis in Denmark. Epidemiology and Infection. 2014; 142(8), 1599–1608. pmid:24168860
- 23. de Knegt LV, Pires SM, Hald T. Attributing foodborne salmonellosis in humans to animal reservoirs in the European Union using a multi-country stochastic model. Epidemiology and Infection. 2015;143(6):1175–86. pmid:25083551
- 24. Cairncross S, Hunt C, Boisson S, Bostoen K, Curtis V, Fung ICH, Schmidt W- P. Water, Sanitation and Hygiene for the Prevention of Diarrhoea. International journal of epidemiology. 2010; 39 (Suppl. 1): 193–205.
- 25. Hunter PR. Household Water Treatment in Developing Countries: Comparing Different Intervention Types Using Meta-Regression. Environmental Science and Technology. 2009; 43(23): 8991–8997. pmid:19943678
- 26. Aspinall W. A route to more tractable expert advice. Nature. 2010; 463(7279): 294–295. pmid:20090733
- 27. Arnell NW, Tompkins EL, Adger WN. Eliciting information from experts on the likelihood of rapid climate change. Risk Analysis. 2005; 25:1419–1432. pmid:16506972
- 28. Lempert RJ, Nakicenovic N, Sareqitz D, Schlesinger M. Characterizing climate change uncertainties for decision-makers. Climate Change 2004; 45:1–9.
- 29. Gale P, Brouwer A, Ramnial V, Kelly L, Kosmider R, Fooks AR, Snary EL. Assessing the impact of climate change on vector-borne viruses in the EU through the elicitation of expert opinion. Epidemiology and Infection. 2010; 138, pp 214–225. pmid:19580695
- 30. Bamber JL, Aspinall WP. An expert judgement assessment of future sea level rise from the ice sheets. Nature Climate Change. 2013; 3: 424–427.
- 31. Cooke RL, Goossens HJ. Expert judgement elicitation for risk assessment of critical infrastructures. Journal of risk research. 2004; 6:643–656.
- 32. Krayer von Krauss MP, Casman EA, Small MJ. Elicitation of expert judgments of uncertainty in the risk assessment of herbicide-tolerant oilseed crops. Risk Analysis. 2004; 24:1515–1528. pmid:15660608
- 33.
Aspinall W, Cooke RM. Expert judgement and the Montserrat Volcano eruption. Proceedings of the 4th International Conference on Probabilistic Safety Assessment and Management PSAM4.1998 Sept 13th -18th, New York City, USA (eds. Ali Mosleh and Robert A. Bari), 3: 2113–2118.
- 34.
Aspinall WP. Structured elicitation of expert judgment for probabilistic hazard and risk assessment in volcanic eruptions. In: Mader HM, Coles SG, Connor CB, Connor LJ, editors. Statistics in Volcanology. Special Publications of IAVCEI, 1. Geological Society, London, 2006; pp. 15–30.
- 35. Neri A, Aspinall WP, Cioni R, Bertagnini A, Baxter PJ, Zuccaro G, Andronicof D, Barsottia S, Coleg PD, Esposti Ongaroa T, Hincksh TK, Macedonioi G, Papalea P, Rosij M, Santacrocej R, Woo G. Developing an event tree for probabilistic hazard and risk assessment at Vesuvius. Journal of volcanology and geothermal research. 2008; 178(3): 397–415.
- 36. Baxter PJ, Aspinall WP, Neri A, Zuccaro G, Spence RJS, Cioni R, Woo G. Emergency planning and mitigation at Vesuvius: A new evidence-based approach. Journal of volcanology and geothermal research. 2008; 178(3): 454–473.
- 37. Tyshenko MG, ElSaadany S, Oraby T, Darshan S, Aspinall W, Cooke R, Catford A, Krewski D. Expert Elicitation for the Judgment of Prion Disease Risk Uncertainties. Journal of Toxicology and Environmental Health. 2011; Part A; 74: 261–285 pmid:21218351
- 38. Tyshenko MG, ElSaadany S, Oraby T, Darshan S, Catford A, Aspinall W, Krewski D. Expert judgement and re-elicitation for prion disease risk uncertainties. International Journal of Risk Assessment and Management. 2012; 16(1–3): 48–77.
- 39. Hoffmann S, Fischbeck P, Krupnick A, McWilliams M. Using Expert Elicitation to Link Foodborne Illnesses in the United States to Foods. Journal of Food Protection. 2007; 70(5): 1220–1229. pmid:17536683
- 40. Havelaar AH, Galindo AV, Kurowicka D, Cooke RM. Attribution of foodborne pathogens using structured expert elicitation. Foodborne pathogens and disease. 2008; 5: 649–659. pmid:18687052
- 41. Lake RJ, Cressey PJ, Campbell DM, Oakley E. Risk Ranking for Foodborne Microbial Hazards in New Zealand: Burden of Disease Estimates. Risk Analysis. 2010; 30(5): 743–52 pmid:19645753
- 42. Ravel A, Davidson VJ, Ruzante JM, Fazil A. Foodborne Proportion of Gastrointestinal Illness: Estimates from a Canadian Expert Elicitation Survey. Foodborne pathogens and disease. 2010; 7:1463–1472. pmid:20704505
- 43. Davidson VJ, Ravel A, Nguyen TN, Fazil A, Ruzante JM. Food-Specific Attribution of Selected Gastrointestinal Illnesses: Estimates from a Canadian Expert Elicitation Survey. Foodborne pathogens and disease. 2011; 8(9): 983–995. pmid:21561379
- 44. Vally H, Glass K, Ford L, Hall G, Kirk MD, Shadbolt C, Becker N. Proportion of Illness Acquired by Foodborne Transmission for Nine Enteric Pathogens in Australia: An Expert Elicitation. Foodborne pathogens and disease. 2014; 11(9): 727–733. pmid:25072416
- 45. van der Fels-Klerz IJ, Cooke RM, Nauta MN, Goossens LH, Havelaar AH. A structured expert judgment study for a model of Campylobacter transmission during broiler-chicken processing. Risk Analysis. 2005; 25:109–124. pmid:15787761
- 46. Cooke RM, Nauta MN, Havelaar AH, van der Fels-Klerx IJ. Probabilistic inversion for chicken processing lines. Reliability engineering and system safety. 2006; 91:1364–1372.
- 47. Devleesschauwer B, Haagsma J, Angulo FJ, Bellinger D, Cole D, Dopfer D, et al. Methodological framework for estimating the global burden of foodborne diseases. PLOS One. 2015;(accepted).
- 48. Havelaar AH, Kirk MD, Torgerson PR, Gibb HJ, Hald T, Lake RJ, et al. World Health Organization estimates and regional comparisons of the burden of foodborne disease, 2010. PLOS Medicine. 2015; (accepted).
- 49. Kirk MD, Pires SM, Black RE, Caipo M, Crump JA, Devleesschauwer B, et al. World Health Organization estimates of the global and regional disease burden of 22 foodborne bacterial, protozoal and viral diseases, 2010. PLOS Medicine. 2015; (accepted).
- 50. Torgerson PR, Devleesschauwer B, Praet N, Speybroeck N, Willingham AL, Kasuga F, et al. World Health Organization estimates of the global and regional disease burden of 11 foodborne parasitic diseases, 2010. PLOS Medicine. 2015; (accepted).
- 51. Gibb H, Devleesschauwer B, Bolger PM, Bellinger D, Wu F, et al. World Health Organization estimates of the global and regional disease burden of four foodborne chemicals and toxins, 2010. PLOS Medicine. 2015; (submitted).
- 52. Cooke RM, Goossens LHJ. Procedures Guide for Structured Expert Judgement in Accident Consequence Modeling. Radiation Protection Dosimetry. 2000; 90 (3): 303–311. ISBN 1 870965 64 7
- 53.
Cooke RM. Experts in uncertainty—Opinion and subjective probability in science. Environmental Ethics and Science Policy Series. Oxford: Oxford University Press. 1991.
- 54. Cooke RM, Goossens LHJ. TU Delft expert judgment data base. Reliability engineering and system safety. 2008; 93(5): 657–674.
- 55. Aspinall WP, Cooke RM, Havelaar AH, Hoffmann S, Hald T. Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases. PLOS One, 2015; (accepted).
- 56. Cooke RM, ElSaadany S, Huang X. On the performance of social network and likelihood-based expert weighting schemes. Reliability engineering and system safety. 2008; 93(5): 745–756.
- 57.
Rice J. Mathematical statistics and data analysis, 3rd edition. Belmond, CA, USA: Thomson Higher Education. 2007. p.353
- 58.
R Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. 2014. Available: http://www.R-project.org/.
- 59. Gkogka E, Reij MW, Havelaar AH, Zwietering MH, Gorris LGM. Risk-based Estimate of Effect of Foodborne Diseases on Public Health, Greece. Emerging infectious diseases. 2011; 17(9): 1581–1590. pmid:21888782
- 60. Scallan E, Hoekstra RM, Angulo FJ, Tauxe RV, Widdowson MA, Roy SL, et al. Foodborne Illness Acquired in the United States-Major Pathogens. Emerging infectious diseases. 2011; 17(1): 7–15. pmid:21192848
- 61. Cooke RM. Messaging climate change uncertainty with supplementary Online Material. Nature Climate Change. 2015; 5: 8–10.
- 62. Flandoli F, Giorgi E, Aspinall WP, Neri A. Comparison of a new expert elicitation model with the Classical Model, equal weights and single experts, using a cross-validation technique. Reliability Engineering and System Safety. 2011; 96: 1292–1310.
- 63. Ezzati M, Lopez A, Rodgers A, Vander Hoorn S, Murray C, and the Comparative Risk Assessment Collaborating Group. Selected major risk factors and global and regional burden of disease. Lancet. 2002; 360:1347–60. pmid:12423980