http://orcid.org/0000-0003-2146-8895
Figures
Abstract
In European regulations for the deliberate release into the environment of genetically modified organisms (GMO), the objective of General Surveillance in Post-Market Environmental Monitoring is defined as the identification of the occurrence of adverse effects of the GMO or its use which were not anticipated in the environmental risk assessment (ERA). Accompanying the commercial cultivation in the EU of maize event MON 810, General Surveillance was implemented by Monsanto, the authorization holder, on a voluntary basis. We carried out a statistical analysis on the pooled results of ten years of farmer questionnaires, which were a part of this General Surveillance, amounting to 2,627 farmer fields in eight European countries in the period 2006–2015. This analysis did not reveal any unexpected adverse effects associated with the cultivation of MON 810. Results from farmer questionnaires confirmed that the cultivation of MON 810 resulted in a significant reduction in the use of pesticides, efficient protection against the target pests, and healthier, higher yielding crops compared to conventional maize. MON 810 also had reduced susceptibility to disease and pests when compared to conventional maize. Monitoring characteristics related to environment and wildlife revealed no significant differences between MON 810 and conventional maize. Literature searches, that were also conducted as part of General Surveillance, identified a comprehensive set of publications addressing environmental safety as well as food and feed safety aspects of MON 810. None of the publications indicated any adverse effect of MON 810 that was not anticipated in the initial ERA, nor did they lead to a change in the conclusions of the initial risk assessment that demonstrated the safety of MON 810. The development of resistance by the target pests (Ostrinia nubilalis, ECB and Sesamia nonagrioides, MCB) was the only potential adverse effect identified in the ERA of MON 810 cultivation in the EU. The extensive safety data package for MON 810, the robust weight of evidence demonstrating both its safety and benefits, and the history of safe use of MON 810 for 15 years in the EU, indicates that focussing the General Surveillance of MON 810 on literature searches and farmer complaint systems would be appropriately protective. This will allow the identification of potential adverse effect not anticipated in the initial ERA without the intensive effort and organizational challenges of farmer questionnaires.
Citation: Bertho L, Schmidt K, Schmidtke J, Brants I, Cantón RF, Novillo C, et al. (2020) Results from ten years of post-market environmental monitoring of genetically modified MON 810 maize in the European Union. PLoS ONE 15(4): e0217272. https://doi.org/10.1371/journal.pone.0217272
Editor: Juan Luis Jurat-Fuentes, University of Tennessee, UNITED STATES
Received: August 10, 2018; Accepted: February 19, 2020; Published: April 24, 2020
Copyright: © 2020 Bertho et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript and its Supporting Information files.
Funding: This work was funded by Monsanto Company*, and the funder (Monsanto) accepted no outside funding for this research represented in this manuscript. This work was also supported by BioMath who played a role in defining the study design, data collection and analysis, and preparation of the manuscript. The funder provided financial support in the form of salaries [IB, RFC, GH, CN and LB] and/or compensation [KS and JS] for all authors, as well as scientific support. The funder also played a role in the design and preparation of the manuscript and the decision to publish. The specific roles of the authors are articulated in the ‘author contributions’ section. * Note that Monsanto has become a subsidiary of Bayer AG as of 21 August 2018. However, as the monitoring tasks described in this manuscript were conducted before the specified date, the name ‘Monsanto’ is kept throughout the documents.
Competing interests: This work was part of the regulatory programs of Monsanto Company*. All of the authors were employees of Monsanto [IB, RFC, GH, CN and LB] or BioMath [KS and JS] while engaged in the research project described in this publication. This does not alter our adherence to PLOS ONE policies on sharing data and materials. * Note that Monsanto has become a subsidiary of Bayer AG as of 21 August 2018. However, as the monitoring tasks described in this manuscript were conducted before the specified date, the name ‘Monsanto’ is kept throughout the documents.
Introduction
The European corn borer (Ostrinia nubilalis, ECB) and the Mediterranean corn borer (Sesamia nonagrioides, MCB) severely damage European maize production by foliage feeding and stalk tunneling in the field, and via secondary effects such as increased mycotoxin levels [1]. Conventional practices—not involving genetically modified (GM) hybrids—for controlling these pests include the use of synthetic insecticides in Integrated Pest Management (IPM) programs but these have variable efficacy.
In the 1990’s, Monsanto developed the GM maize event MON 810, expressing the naturally occurring Bacillus thuringiensis (Bt) protein Cry1Ab, which provides protection against certain lepidopteran insect pests, including both ECB and MCB. In 1998, an authorization for commercial cultivation of MON 810 in the EU was obtained for a period of 10 years [2]. In 2007, the renewal application for MON 810 cultivation was submitted and now, more than a decade later, it is still pending a decision at the European Commission.
European Directive 2001/18/EC [3] introduced the obligation to implement a monitoring plan to trace and identify any direct or indirect, immediate, delayed or unforeseen effects on human or animal health or the environment of GM organisms or products after they have been placed on the market. The objectives of a Post-Market Environmental Monitoring (PMEM) plan were defined as 1) confirming that assumptions in the environmental risk assessment (ERA) regarding the occurrence and impact of potential adverse effects of the GMO or its use are correct (“Case Specific” or CS), and 2) identifying any adverse effects of the GMO or its use on human or animal health or the environment which were not anticipated in the ERA (“General Surveillance” or GS). Since then, additional guidelines on PMEM and GS have been published [4–6].
The initial authorization for MON 810 cultivation in the EU was granted before Directive 2001/18/EC entered into force and included the requirement to conduct CS monitoring [2]. The development of resistance to the expressed Cry1Ab protein in the target pests was the only potential adverse effect identified in the ERA of MON 810 cultivation in the EU. While it has been debated whether development of resistance in insects is an agronomic/economic effect rather than an environmental one, CS monitoring focused on detecting resistant insect populations and on the implementation of, and compliance with, the Insect Resistance Management (IRM) plan. The results of the CS monitoring show no indication of reduced Cry1Ab susceptibility after 15 years of commercial planting for the two target pests in the EU, indicating that the established IRM plan for MON 810 has been a conservative, pro-active and effective tool in sustaining the efficacy of the product [8, 25–34]. It should be noted that several product characteristics and aspects of pest biology contribute to a low likelihood of changes in susceptibility of the target pests against the MON 810 Cry1Ab protein in the EU (e.g. use of a high-dose product and the mobility of ECB and MCB populations). Furthermore, resistance is more likely to develop when there is high adoption of the trait and, even though there are certain areas where the MON 810 adoption rate is high, in general it is cultivated on a relatively small-scale in the EU [7, 8].
The initial MON 810 authorization did not include a requirement to conduct GS. Nevertheless, the authorization holder, Monsanto, has implemented GS since 2006 on a voluntary basis following an approach developed by industry that combines farmer questionnaires, searches of relevant scientific publications, company stewardship programs, and surveillance networks where appropriate [9]. Questionnaires directed at farmers growing GM crops are considered one of the possible approaches for collecting first hand data on the performance and impacts of GM crops in comparison with conventional crop cultivation [10–14]. At the same time, searches of peer-reviewed literature can support and provide new context to the original ERA by identifying new publications related to MON 810 conducted worldwide including experimental research, developmental and advisory studies on crop cultivation, hybrid registration and hybrid performance trials.
In this paper, the results from ten years of farmer questionnaires and literature searches that were independently conducted as part of PMEM for commercial cultivation of MON 810 in the EU are described. A comprehensive statistical analysis of the pooled data from the ten years of farmer questionnaires was conducted, including an analysis of trends within the data. With MON 810 being the only GM crop cultivated commercially in the EU for nearly 15 years, this summary of GS information illustrates the experience gained through MON 810 cultivation and provides an opportunity to reflect on possible adjustments to the current PMEM approach. As far as we know, this is also the longest duration and most comprehensive PMEM data set ever developed for a GM crop.
Materials and methods
Farmer questionnaires
Design of the questionnaire.
The methodology for using farmer questionnaires in PMEM of GMO has been published [11–24]. For the MON 810 farmer questionnaires, three subjects were identified which might be influenced by and observed during cultivation, i.e. agronomic practices, plant characteristics in the field, and effects on environment and wildlife (Table 1). For each subject, practical endpoints (monitoring characteristics) suitable to detect emerging effects from the cultivation of MON 810 were defined.
For each monitoring characteristic, qualitative categories were defined to direct the assessment of the cultivation of MON 810 compared to conventional maize. For most characteristics, the possible categories of answers were “as usual” and “different” with the latter category subdivided into “plus” (e.g. later, delayed, higher, more) or “minus” (e.g. earlier, accelerated, lower or less). For some characteristics related to agronomic practices and performance of fed animals, a subdivision of the “different” category based on quantitative terms was not meaningful.
A farmer questionnaire was designed to obtain data on the monitoring characteristics. Because in an agricultural landscape other factors (like soil characteristics, cultivation methods or environmental factors) may influence the monitored characteristics, these were also taken into account in the questionnaire.
The questionnaire was adjusted yearly based on the experience gained in the previous season, on improvements of the statistical relevance of the collected data, and according to feedback received from the ESFA GMO Panel and the European Commission (DG Environment). The survey questions used for each year of the survey are provided in the published, annual PMEM reports [25–34].
Sampling.
An annual survey was performed for ten years. The total population consisted of all fields within the EU where MON 810 was cultivated within the ten-year monitoring period. The sample size for the survey was determined in relation to the statistical procedure (one-sided exact binomial test, comparison of a probability with a constant), the performance requirements (type-1 error rate α = 0.01, type-2 error rate β = 0.01, constant = 0.1 (10% “different” answers)) and minimum difference of practical interest (0.03) [14]. This resulted in a minimum sample size of 2,436 fields. To obtain this minimum sample size (e.g. allowing for the exclusion from the survey of low quality questionnaires), an accumulated multiyear sample size of at least 2,500 fields was set (more details are given in S1 File).
The total number of monitoring objects, i.e. the 2,500 fields, was equally subdivided into 250 per year (first stratum). MON 810 cultivation areas in various European countries varied from year to year, meaning that the population was distributed irregularly. To handle the variation across regions and population of MON 810 cultivating farmers from year to year, a quota was applied considering:
- the countries of MON 810 cultivation in the respective year,
- the magnitude of MON 810 cultivation (ha planted per country/ ha planted in the EU), and
- local situation (average field size in the country).
The fluctuating deployment of MON 810 was taken into account by assigning the 250 fields per year to the respective countries (second stratum). Cultivation areas with high uptake of MON 810 are represented in the sample by a large number of monitored fields, whereas countries with proportionally very low cultivation were excluded from the monitoring.
Within each stratum (per year and country), the determined number of monitoring units needed to be selected. Where publicly available, the “GMO cultivation register” was used to identify regions of cultivation, the county-specific sample size was distributed in proportion to the regional cultivation and MON 810 cultivating farmers were identified by the interviewers per region. Where information in the GMO cultivation register could not be used to identify the farmers (e.g. due to privacy provisions), the interviewers had to contact farmers identified via their own research or with support of the MON 810 seed selling companies. The identity of the contributing farmers was kept confidential.
The stratified approach for sampling was intended to ensure that the monitoring area was proportional to and representative of the total regional area under MON 810 cultivation.
Collection of information.
Information was collected by trained interviewers. The annual training addressed the background of the questions and experience gained during previous years’ surveys (uncertainties, misinterpretation of questions). To ensure consistent high data quality and to assist the interviewers in filling out the questionnaires with farmers, a 'user's manual' was developed.
Statistical analysis.
If there is no effect of MON 810 cultivation or other influencing factors, and the question is well formulated and unambiguous, farmers will assess the situation to be “as usual”. Uncertainty or environmental impacts may result in low frequencies of differing answers. These are expected to be balanced in both “minus” and “plus” directions. If the cultivation of MON 810 or any other influencing factor has a real impact, this would result in a greater proportion of “different” answers. To test for an effect, the proportions of “different “answers were compared with the threshold of 10% by a statistical test (one-sided, comparison of a probability with a constant). Because the “as usual” and “different” answers complement each other, a closed test procedure was applied: first the “as usual” proportion was compared with the threshold of 90%. If the “as usual” proportion exceeded this threshold, this was considered to indicate no effect. Otherwise, both “plus” and “minus” proportions were compared with the 10% threshold and an effect was indicated if the threshold was exceeded by one of the proportions (more details are given in S1 File).
The data from the farmer questionnaires were analysed annually, and annual reports of the results were published [25–34]. Data from the first ten years of surveys have been used to conduct a comprehensive analysis of the pooled data including an analysis of trends within the data. In this analysis of the pooled data (ten years of monitoring, all countries), simple proportions ps as well as weighted proportions pw of the “as usual”, “plus” and “minus” categories of the monitoring characteristics were estimated. The model proportions pmo (inclusive 99% confidence intervals) of all categories were estimated by applying a linear mixed model (more details are given in S1 File). The estimated 99% confidence intervals for the model proportions pmo were compared with the defined thresholds according to the closed test procedure described above.
For each monitoring characteristic, the year-specific model proportions of all categories were estimated and assessed for any linear trends over years by fitting linear regression models and subsequently testing the estimated slopes against zero (more details are given in S1 File).
Literature search
Since the start of the PMEM efforts, literature searches have been performed annually with the objective to detect any adverse effects of MON 810 or its use that were not anticipated in the ERA. The methodology for these literature searches was developed in-house in 2006 and further refined over the years based on experience. Considering the objective of the literature searches was not to address a specific question, they do not reflect the characteristics of a systematic literature review. Single or combinations of queries with free text terms as keywords and combinations with Booleans were used (S6 File). The keywords and keyword combinations were used to initiate searches, or collect updates, in the Web of Science database (WoS) and, since 2015, in the Centre for Agriculture and Biosciences International database (CABI). Over the years, the keywords used for the literature searches were adjusted taking into consideration newly established knowledge and experience. The keywords and keyword combinations that were used for each year of monitoring are available in the published annual PMEM reports [25–34] and are also provided in S6 File. An automated alert system was set up retrieving potentially relevant publications on a monthly basis from the Web of Science Core Collection database. Taking into account recommendations of the European Food Safety Authority (EFSA) GMO Panel [35] the CABI CAB Abstracts and Global Health database was included in the automated search starting in 2015.
All the retrieved publications were assessed for their potential relevance to the risk assessment of MON 810 based on the following inclusion/exclusion criteria: i) the objective(s) of the studies, i.e. assessment of potential effects on human and animal health or the environment of MON 810 and ii) the scope of the application, i.e. authorization for import, processing and all uses as any other maize, including the cultivation of MON 810 in the EU, and iii) the categories of information for risk assessment data requirements outlined in relevant guidelines and legislation. The list of different categories included food and feed safety (toxicity / animal feeding; molecular characterization; protein expression; toxicity in vitro; crop composition/nutrition; protein fate / DNA in the digestive tract; allergenicity; mycotoxins; others) or environmental safety (spillage and consequences thereof; agronomy; non target organisms (NTOs); pollen mediated gene flow; protein/DNA fate in soil; toxin fate in soil; insect resistance management (IRM); ecology; others). Because of this diversity of categories, studies with different experimental design, including review articles, were considered for inclusion in the annual PMEM reports.
A stepwise approach was followed by three different reviewers: two external reviewers with the company ToxMinds BVBA and one internal reviewer. This approach was conducted to assess the relevance of the retrieved publications to the risk assessment of MON 810. First, a rapid assessment was conducted based on the information in the title and abstract of all the retrieved publications following the inclusion/exclusion criteria described previously. In a second step, a more detailed assessment of the publications that were selected as potentially relevant within the rapid assessment was conducted. The full text of these selected, potentially relevant studies was further assessed based on the inclusion/exclusion criteria, leading to a final list of relevant publications included in the PMEM reports (provided in S5 File). In cases when the reviewers’ assessment of a retrieved study did not lead to a unanimous recommendation for relevance, the study was de facto included.
Using the information reported in the publications that were included in the PMEM reports (S5 File), the reviewers carried out an assessment to define if the results of the study impacted the initial risk assessment conclusions for MON 810. In accordance with the framework provided by the European Commission [36], all the included publications were assessed by summarizing the study results and conclusions, and by considering if there were any adverse effects of MON 810 reported in the studies related to human, animal or environmental safety. These assessments are available in the published annual PMEM reports [25–34] and determined whether the conclusions of the initial risk assessment that demonstrated the safety of MON 810 based on the comprehensive weight of evidence were still valid. The studies that suggested no adverse effects were considered as confirmatory studies to the initial risk assessment and no further action was taken. The studies that indicated potential adverse effects of MON 810 to human and animal health or the environment, relevant to the above-mentioned risk assessment categories, were subjected to further analysis by internal and external experts with a solid experience in the risk assessment of GM plants and by experts with technical experience in the specific area of the selected publication. This further assessment determined whether the methodological quality of the experiment described in the study adhered to the basic principles for conducting a scientific experiment (e.g. inclusion of the necessary control groups, characterization of test materials used, consideration of natural variability in study results, etc.). The results of the methodological quality assessment conducted on the studies indicating a potential adverse effect of MON 810 on human and animal health or the environment are provided in the published annual PMEM reports [25–34].
Results
Farmer questionnaires
MON 810 cultivation in the EU.
Since the introduction of MON 810 in the EU, the area of commercial cultivation has quickly reached a stable footprint focused on regions in Spain and Portugal where the target pests are abundant (Table 2).
Sample and survey response rate.
Over ten years of MON 810 PMEM in Europe, 2,627 farmer questionnaires were completed by 1,262 farmers in eight countries. The final totals (number of farmers per country and year included in the biometrical analysis) are described in Table 3.
The sampling procedure for the survey was impacted and complicated by several challenges:
- data on the total population of interest, i.e. the total number of fields (and the field sizes), were not known at the beginning of the monitoring project;
- the development of areas of MON 810 cultivation could not be predicted;
- country-specific promotions and bans of MON 810 led to a very irregular distribution across the strata of the sampling frame.
Since 2014 the programme has focused on Spain and Portugal in view of the area of MON 810 commercially grown.
The response rate was calculated as the number of farmers who completed the farmer survey with sufficient data quality to be included in the analysis, divided by the total number of farmers contacted. All farmer refusals were recorded, however, overall only 1% of farmers refused to participate. This unusually low refusal rate is likely due to how the surveys were presented and conducted. In advance of the review, the farmers were personally contacted and asked for their ability and willingness to participate. The 1% of farmers who refused to participate mostly did so for lack of available time to complete the survey. Less than 0.1% of farmers’ responses had to be excluded from the analysis for insufficient data quality reasons [25–34]. Therefore, the response rate was approximately 98%.
Longitudinal aspects.
For representativeness, taking into consideration variable cultivation areas and farmers' willingness to participate in the interview, new farmers were recruited each year. Consequently, each year the survey covered a mixture of newly identified and known farmers.
Table 4 indicates the number of times that an individual farmer was surveyed across the ten years. Fifty-two percent of the farmers were surveyed across multiple years, though not necessarily for the same field. Eighty-eight percent participated up to four times and three farmers were interviewed in all ten years.
Ten-year analysis.
Data on 25 monitoring characteristics covering agronomic practices, characteristics in the field, and environmental and wildlife aspects were collected (see Table 1 and S7 File for the results). The Supporting Information lists all simple proportions p (Tables A-C in S2 File), the descriptive weighted proportions pw (Tables D-F in S2 File); the results of ten-year analysis, model proportions pmo (Tables G-I in S2 File); and the slopes and confidence intervals of linear regression (Table J in S2 File) per monitoring characteristic. The results of the descriptive weighted proportions pw of each monitoring characteristic are also provided as figures in S3 File.
The model proportions pmo estimate the mean annual monitoring character values considering multiple participation of farmers. The estimated variances of the random effect 'multiple participation of farmers' were either zero or negligible, so an influence on the monitoring characteristics could not be identified. Consequently, multiple growing seasons may be considered quasi-independent as each year represents a different growing season with different climate and growing conditions. Moreover, across growing seasons, different fields were planted with MON 810 by the same farmer due to decisions on crop rotation measures.
The results of the analysis of characteristics relating to agronomic practices are presented in Fig 1.
In significantly more than 10% of surveyed MON 810 fields, insect control (20.04%; CI 20.03% - 20.06%) and maize borer control practices (20.79%; CI 20.77% - 20.81%) were changed (decreased) relative to conventional maize fields. MON 810 maize was harvested significantly later than conventional comparators at 16.70% (CI: 16.69% - 16.71%) of all monitored fields. All other monitored agronomic practice characteristics (crop rotation, time of planting, tillage and planting techniques, fertilizer application, weed and fungal control practices as well as irrigation practices) of MON 810 fields were not significantly different from conventional maize fields.
The results of the analysis of characteristics in the field are presented in Fig 2.
Six of the characteristics in the MON 810 fields that were monitored were significantly different from what was observed in conventional maize fields (Fig 2). As expected based on the insect protection provided by MON 810, yield (53.74%; CI 53.71% - 53.76%) was significantly increased, while incidence of stalk/ root lodging (34.03%; CI 34.01% - 34.05%) and occurrence of volunteers (11.14%; CI: 11.12% - 11.16%) were significantly reduced in MON 810 fields compared to conventional maize fields. In addition, germination vigor was significantly increased (10.83%; CI: 10.82% - 10.85%) in MON 810 fields, while time to maturity was delayed (19.30%; CI: 19.28% - 19.32%). Plant growth and development was significantly changed (as usual 87.76%; CI 87.75% - 87.77%): 3.68% delayed (CI: 3.67% - 3.69%, not significant), 8.56% accelerated (CI: 8.55–8.57, not significant).
Other characteristics, such as time to emergence and time to male flowering, were not significantly different in MON 810 fields compared to conventional maize fields.
The results of the analysis of characteristics relating to environment and wildlife are presented in Fig 3.
Two of the monitored characteristics showed a statistically significant difference between growing MON 810 and conventional maize: disease susceptibility (32.25%; CI: 32.24% - 32.27%) and pest susceptibility (18.20%; CI: 18.19% - 18.21%) were both decreased in the surveyed MON 810 fields relative to conventional maize fields. No changes were observed for weed pressure or occurrence of non-target insects, birds and mammals. Also, no difference was reported in the performance of animals fed with MON 810 compared to animals fed conventional maize (Fig 3).
Table 5 provides an overview of the results described above from the analysis for all the monitoring characteristics.
Trend analysis.
Table J in S2 File presents the results of the trend analysis over the ten years. A trend analysis can indicate if the findings remain stable over time. Although some parameters changed across years as described below, overall the changes were minor and did not modify the initial findings.
The ratio of fields reported to have changed crop rotation grew by 0.96% per year, indicating that MON 810 was cultivated in slightly more fields with different previous crops compared to conventional maize. There was also a minor increase in the proportion of fields with changed tillage and planting techniques (0.37% per year) as well as with later time of planting (0.39% per year). In contrast, the difference in fertilizer application and harvest time of MON 810 compared with conventional maize both diminished over years (resp. 0.11% and 0.18% per year). No trends were detectable for insect, weed and fungal control practices, irrigation practices and maize borer control practices.
Time to male flowering and yield showed no statistically significant trends over time, while the other observed characteristics in the field showed significant (though minor) trends. Most trends in field characteristics indicated an increase in the proportion of “as usual” responses.
For environment and wildlife monitoring characteristics no trends were observed for weed pressure, occurrence of birds and the performance of animals fed on MON 810. However, the proportion of surveyed MON 810 fields for which disease susceptibility and occurrence of non-target insects and mammals was assessed to be “as usual” increased per year.
Literature search
The use of the Web of Science Core Collection database and the CABI CAB Abstracts and Global Health database ensured a broad coverage as well as a high probability of retrieving relevant publications from several peer-reviewed and high impact journals. Over the ten years of PMEM, 375 publications and 48 review papers related to MON 810 and Cry1Ab published in peer-reviewed journals were identified (See S4 File). Among the retrieved publications, 102 were related to food and feed safety while 273 of the publications were related to environmental safety. An assessment of each of the relevant publications was conducted in the context of the yearly PMEM report. A full list of the retrieved relevant publications is provided in S5 File. None of them led to a change in the conclusions of the initial risk assessment that demonstrated the safety of MON 810.
Discussion
Farmer questionnaires use farmers' experience with a product and provide first-hand knowledge of agricultural environments and crop performance. With the challenges associated with the evolution of the cultivated area, a stratified sampling approach had to be used to ensure that the monitoring area was proportional to and representative of the total regional area cultivated with MON 810 in the EU. By doing so, it was confirmed that the cultivation of MON 810 in the EU resulted in a significantly reduced proportion of fields on which standard insect control practices, in particular maize borer control, were applied. Standard pest control practices require the use of synthetic insecticides. Due to the protection provided by the Cry1Ab protein expressed in MON 810, a decrease in these and other maize borer control practices is expected. The same effect has been observed in other regions of the world that have adopted insect-protected GM crops. Overall, a decrease of as much as 56.1% in insecticide active ingredient use globally from using GM insect-protected maize was reported based on data gathered from 1996 to 2016 [37–39]. Hutchinson et al. [40] showed that, as a result of area-wide pest suppression, economic benefits accrue not only to farmers planting Bt maize but also to those planting non-Bt maize, and that these suppression benefits can equal or exceed the benefits to Bt maize growers [41].
Adoption of MON 810 technology reduces the use of pesticides, provides efficient protection against the target pests and results in a healthier and better yielding crop. The significantly improved control of the targeted corn borers leads to improved plant growth and development in general, delayed time to maturity and consequently a delayed time to harvest for MON 810 compared to conventional maize. The results of the farmer questionnaire analysis confirmed a significantly later harvesting time of MON 810 maize plants compared to conventional maize. An indicator for farmers to determine time of maize harvest is the grain moisture. It has been demonstrated previously that Bt maize plants tend to have higher grain moisture compared to non-Bt maize [42], potentially caused by accelerated senescence of maize plants when infested with corn borers [43]. Therefore, the delayed time to maturity and later harvesting time that was observed for MON 810 is likely due to preserved grain moisture in the absence of corn borer infestations. Harvesting early also prevents further infestation and damage by the corn borer larvae and consequently reduces the amount of lodged plants or dropped ears [44]. Later harvesting also results in yield protection when farmers plant MON 810 instead of conventional maize. The availability of high quality germplasm used by MON 810 cultivating farmers likely adds to the observed increase in yield. This result is in line with studies conducted with MON 810 or Bt maize in different non-EU countries and regions that are subject to high pressure of certain maize insect pests. The conclusions consistently confirm an increased yield compared to the cultivation of conventional maize [39, 45–47].
Stalk lodging happens more frequently to maize plants infested with corn borers. Lodged plants may escape harvest at the end of the growing season and can therefore increase the occurrence of maize volunteers in the subsequent year. The data of the farmer questionnaires confirm that the occurrence of volunteers was reduced in a significant number of fields with MON 810. Increased germination vigor was observed for MON 810 compared to conventional maize, which is considered to be linked to the availability of high-quality germplasm and premium seed treatments when buying MON 810 seeds. This likely leads to the observation of an increased germination vigor for MON 810 seeds compared to conventional maize seeds.
Finally, MON 810 had reduced susceptibility to disease and pests when compared to conventional maize. Research has demonstrated that insect-damaged maize plants are an easy target for pathogens, such as Fusarium [48, 49]. The farmer questionnaires demonstrate that MON 810 maize with its plant-incorporated insect protection is less susceptible to these pathogens. Several reports [48–50] highlighted the beneficial effect of reduction of mycotoxins in the harvested crop, resulting from preventing secondary diseases such as Fusarium by limiting insect damage.
Other monitoring characteristics related to environment and wildlife revealed no significant difference between MON 810 maize fields and conventional maize fields, as expected given the nature of MON 810 maize.
The essential characteristics of changes in pest control practice, in particular for maize borer control, and increased yield remained stable over the ten-year period. The minor trends observed in certain other monitoring characteristics may have resulted from farmers adapting their agronomic practice based on experience with MON 810: confidence in efficient pest protection, improved plant development and reduced losses and may have resulted in a later planting. In addition, as farmers gain familiarity with the crop, the “as usual” category is expected to increase for many monitoring characteristics. In any case, the trends are minor and do not significantly influence the observed differences reported above.
The searches of peer-reviewed literature broaden the scope beyond the agricultural setting and the defined territory. The literature searches identified a comprehensive set of publications addressing environmental safety as well as food and feed safety aspects of MON 810. None of the identified publications challenged the original ERA conclusions, nor did they lead to a change in the conclusions of the initial risk assessment that demonstrated the safety of MON 810. All gathered information confirmed that MON 810 is as safe to human and animal health and the environment as conventional maize, and that there is no adverse impact from the cultivation of MON 810 on biodiversity, abundance, or survival of non-target species.
The EU regulatory framework foresees a maximum authorization period of ten years for the placing of GM products on the market, after which a renewal application needs to be submitted. During this period, PMEM required a substantial effort. MON 810 is the only GM product authorized for cultivation in the EU and therefore the first GM product in the EU that has completed a full ten-year cycle.
CS monitoring is intended to assess alleged development of resistance by the target pests, the only potential adverse effect of the GMO identified in the ERA. For the cultivation of MON 810 in the EU, CS monitoring has focused on detecting resistance in the target pests to Cry1Ab and the effectiveness of the implemented IRM plan. There have been no indications of reduced Cry1Ab susceptibility in the two target pests (ECB and MCB) in the EU. This can largely be attributed to the rather conservative, pro-active and effectively implemented IRM plan for MON 810 [7, 8] in combination with additional factors that further reduce the likelihood of resistance development of the target pests (i.e. use of a high-dose product, the mobility of ECB and MCB populations, and the relatively small-scale MON 810 cultivation in the EU, although there are certain areas in the region where the MON 810 adoption rate is high (Table 2)). The implemented IRM plan accompanying the cultivation of MON 810 is needed to sustain the efficacy of this Bt product in the EU.
GS is intended to identify adverse effects of the GMO or its use on human or animal health or the environment which were not anticipated in the ERA. It has been conducted voluntarily and summarized results provide a unique opportunity to review the validity of the current PMEM approach. In this paper, results were reported from implementing the farmer questionnaire and the searches of relevant scientific literature.
The literature searches constitute to be an important GS component. Automation of the identification step ensures a rapid discovery of new publications, whereas the critical review by internal and external experts ensures an evaluation in relation to the existing ERA and the information supporting it.
The farmer questionnaire component required establishing an intensive program with trained interviewers in order to guarantee the necessary response rate. Even then, the selection of stratified subsamples in an evolving market was a challenge. The farmer questionnaires have so far not identified a case of a MON 810 efficacy performance problem or an unanticipated adverse effect. With a great deal of effort, a large number of “as usual” observations have been collected. Based on these results and the efforts required to obtain them, the farmer questionnaire system is not an efficient option for future GS of MON 810 and is not proportionate to the risk.
An alternative tool to capture relevant data from farmers is a farmer complaint system. Companies that are selling MON 810 varieties have established robust farmer complaint systems, which are available to the entire farming community and consequently are not limited to a subsample. The farmer complaint systems serve as a primary tool to detect unexpected adverse effects such as insect susceptibility shifts, and allow reporting of broader issues related with performance criteria such as germination, crop development, damage due to pests and insects, effects on yield and any kind of unexpected finding. Farmers can report complaints to the seed suppliers about product-related topics via the local sales representatives or customer service routes. An example is the Technical User Guide that accompanies each bag of MON 810 and that includes a section with company and seed association contact details for the farmer’s reference. Although the specifics may differ between seed suppliers, in all cases an internal procedure for verification, analysis and follow up processes is initiated upon receipt of a validated product-specific complaint. This procedure includes on-site inspection by company representatives and additional testing of larval susceptibility to Cry1Ab. To collectively monitor the complaints each company received by farmers on a yearly basis, a system was put in place in 2016 by the member companies of the Asociación Nacional de Obtentores Vegetales (ANOVE) that commercialize MON 810 in Spain. A total of 1,556 farmer complaints were received in 2016, demonstrating its effectiveness. Six of the received complaints were related to efficacy of MON 810. After applying the follow-up procedure for these verified complaints, it was confirmed that none of them were linked to resistance development in the field. In addition, registration holders of any MON 810 variety in Spain are obliged to comply with a two-month monitoring plan that includes inspection of fields to monitor potential resistance development [51, 52]. In line with the results of the questionnaires and literature, the complaint systems have so far not identified a case where an in-depth further assessment of a received complaint was attributed to a MON 810 efficacy performance problem or a non-anticipated effect. The complaint systems allow focus on deviating observations that can be early indicators of adverse effects of MON 810 that were not anticipated in the initial ERA, thereby offering an effective, integrated (as used for all kind of complaints) and more cost-efficient option for MON 810 GS, proportionate to the risk.
Conclusion
The ten-year analysis of data reported by farmers on 2,627 fields in eight European countries in the period 2006–2015 did not reveal any adverse effects associated with the cultivation of MON 810. Statistically significant differences between the cultivation of conventional maize and of MON 810 observed in the farmer questionnaires can be directly attributed to the insect protection provided by Cry1Ab expressed in MON 810. MON 810 provided tangible and non-tangible benefits and allowed farmers to adapt their crop management as illustrated by some of the trends. It provided flexibility in management while ensuring that the crop was adequately protected against the target pests, a protection that likely spanned beyond the specific fields and provided benefits to non Bt-maize growing farmers.
Furthermore, the searches of peer-reviewed literature, guaranteeing a broad coverage of potential issues, did not reveal any adverse effects that changed the conclusions of the initial risk assessment that demonstrated the safety of MON 810. These results are in line with EFSA’s assessment of the 2007 renewal application, confirming the conclusions of the original safety assessment: “…MON 810 is as safe as its conventional counterpart with respect to potential effects on human and animal health. The EFSA GMO Panel also concludes that maize MON 810 is unlikely to have any adverse effect on the environment in the context of its intended uses…” [53].
When the concepts of PMEM were developed, the dynamic nature of the approach was stressed. All PMEM guidance documents indicate that monitoring plans should not be viewed as static and that it is fundamental that the monitoring plan and associated methodology are reviewed at appropriate intervals and may need to be adapted depending on the results of the monitoring information collected [4, 54]. The PMEM results for MON 810 offer the first opportunity to evaluate the contribution of different components. Even though no resistance of ECB or MCB to the Cry1Ab protein has been detected in the EU, continued IRM efforts are needed to sustain product efficacy and delay the development of resistance. On the other hand, while requiring an intensive effort and facing organizational challenges, the farmer questionnaires did not reveal any new finding that could not have been revealed via the farmer complaint system. The results presented in this publication are consistent with the history of safe use of MON 810 for nearly 15 years in the EU. Therefore, this publication suggests that focussing the GS component of the accompanying PMEM of MON 810 to literature searches and the farmer complaint systems would not weaken the level of protection for human or animal health and the environment.
Supporting information
S2 File. Results of the statistical analysis of the farmer questionnaires.
https://doi.org/10.1371/journal.pone.0217272.s002
(DOCX)
S3 File. Results of the statistical analysis of the farmer questionnaires (Graph bars).
https://doi.org/10.1371/journal.pone.0217272.s003
(DOCX)
S4 File. Results of the literature screening.
https://doi.org/10.1371/journal.pone.0217272.s004
(XLSX)
S6 File. Keywords used for the literature searches.
https://doi.org/10.1371/journal.pone.0217272.s006
(DOCX)
S7 File. Results of the individual farmer questionnaires.
https://doi.org/10.1371/journal.pone.0217272.s007
(XLSX)
Acknowledgments
We would like to thank Francesca Tencalla at ToxMinds for support in the preparation of the sections on the literature searches. We would also like to thank Pamela Bachman, Luis Burzio, Angela Culler, Tewodros Firdissa Duressa, Dan Kendrick and Steve Levine at Monsanto Company for their review of the manuscript, and Jamis Perrett for technical assistance.
References
- 1. Cordelo A, Malvar RA, Butrón A, Revilla P, Velasco P, Ordás A. Population dynamics and life-cycle of corn borers in South Atlantic European coast. Maydica. 1998;43:5–12.
- 2.
European Commission. Commission Decision (98/294/EC) of 22 April 1998 concerning the placing on the market of genetically modified maize (Zea mays. L. line MON810), pursuant to Council Directive 90/220/EEC: COMMISSION DECISION 98/294/EC; 1998.
- 3. Commission European. Directive 2001/18/EC of the European Parliament and of the Council on the deliberate release into the environment of genetically modified organisms and repealing Council Directive 90/220/EEC. Official Journal of the European Union. 2001;L 106:1–38.
- 4.
European Commission. Council Decision (2002/811/EC) of 3 October 2002 establishing guidance notes supplementing Annex VII to Directive 2001/18/EC of the European Parliament and of the Council on the deliberate release into the environment of genetically modified organisms and repealing Council Directive 90/20/EEC; 2002.
- 5. EFSA Panel on Genetically Modified Organisms. Opinion of the Scientific Panel on genetically modified organisms [GMO] on the Post Market Environmental Monitoring (PMEM) of genetically modified plants. EFSA Journal. 2006;4:319‐n/a.
- 6. EFSA Panel on Genetically Modified Organisms. Guidance on the Post-Market Environmental Monitoring (PMEM) of genetically modified plants. EFSA Journal. 2011;9:2316.
- 7. Thieme TGM, Buuk C, Gloyna K, Ortego F, Farinós GP. Ten years of MON 810 resistance monitoring of field populations of Ostrinia nubilalis in Europe. Journal of Applied Entomology. 2017;2:47.
- 8. Farinós GP, Hernández-Crespo P, Ortego F, Castañera P. Monitoring of Sesamia nonagrioides resistance to MON 810 maize in the European Union: Lessons from a long-term harmonized plan. Pest Manag Sci 2017.
- 9. Tinland B, Janssens J, Lecoq E, Legris G, Matzk A, Pleysier A, et al. Implementation of General Surveillance in Europe: The Industry Perspective. Journal fur Verbraucherschutz und Lebensmittelsicherheit. 2006;1:42–4.
- 10.
ACRE. Guidance on Best Practice in the Design of Post-market monitoring Plans in Submissions to the Advisory Committee on Releases to the Environment. 2004. http://webarchive.nationalarchives.gov.uk/20081110024907/http:/www.defra.gov.uk/environment/acre/postmarket/acre_postmarketmonitor-guidance.pdf.
- 11. Sanvido O, Widmer F, Winzeler M, Bigler F. A framework for the design of general surveillance of genetically modified crops based on a concept for environmental post-market monitoring. Journal fur Verbraucherschutz und Lebensmittelsicherheit. 2006;1:5–10.
- 12. Wilhelm R, Beißner L, Schmidt K, Schmidtke J, Schiemann J. Monitoring des Anbaus gentechnisch veränderter Pflanzen—Fragebögen zur Datenerhebung bei Landwirten: Monitoring of the cultivation of genetically modified plants—use of farmers' questionnaires. Nachrichtenblatt des Deutschen Pflanzenschutzdienstes (Braunschweig). 2004;56:184–8.
- 13. Schmidt K, Schmidtke J, Wilhelm R, Beißner L, Schiemann J. Biometrische Auswertung des Fragebogens zum Monitoring des Anbaus gentechnisch veränderter Maissorten: Statistische Beurteilung von Fragestellungen des GVO-Monitoring. Nachrichtenblatt des Deutschen Pflanzenschutzdienstes (Braunschweig). 2004;56:206–12.
- 14. Schmidt K, Wilhelm R, Schmidtke J, Beißner L, Mönkemeyer W, Böttinger P, et al. Farm questionnaires for monitoring genetically modified crops: a case study using GM maize. Environ Biosafety Res. 2008;7:163–79. pmid:18801326
- 15.
Berensmeier A, Schmidt K. “Good Monitoring Practice” Quality control measures for farm questionnaires. In: Bundesamt für Verbraucherschutz und Lebensmittelsicherheit, editor. Proceedings of the international workshop on "Post Market Environmental Monitoring of Genetically Modified Plant: Harmonisation and Standardisation—a Practical Approach; 26-27- April 2007; Berlin-Dahlem. 1st ed.: Birkhäuser Verlag; 2007. p. 56–58. https://doi.org/10.1007/s00003-007-0238-z
- 16.
Berensmeier A, Schmidt K, Beißner L, Schiemann J, Wilhelm R. Statistical analysis of farm questionnaires to search for differences between GM- and non-GM-maize. In: Bundesamt für Verbraucherschutz und Lebensmittelsicherheit, editor. Proceedings of the international workshop on "Post Market Environmental Monitoring of Genetically Modified Plant: Implementation of General Surveillance"; 27–28. April 2006; Berlin-Dahlem. 1st ed.: Birkhäuser Verlag; 2006. p. 80–84. https://doi.org/10.1007/s00003-006-0081-7
- 17.
Böttinger P, Schmidt K, Mönkemeyer W, Wilhelm R, Schmidtke J, Schiemann J. Focused data for post market monitoring purposes gathered by farm questionnaires. In: International Society for Biosafety Research, editor. 10th International Symposium on the Biosafety of Genetically Modified Organisms; 16–21. November 2008; Wellington, New Zealand; 2008. p. 183.
- 18.
Schiemann J, Wilhelm R, Beißner L, Schmidtke J, Schmidt K. Data acquisition by farm questionnaires and linkage to other sources of data. In: Bundesamt für Verbraucherschutz und Lebensmittelsicherheit, editor. Proceedings of the international workshop on "Post Market Environmental Monitoring of Genetically Modified Plant: Implementation of General Surveillance"; 27–28. April 2006; Berlin-Dahlem. 1st ed.: Birkhäuser Verlag; 2006. p. 26–29. https://doi.org/10.1007/s00003-006-0065-7
- 19.
Schiemann J, Wilhelm R, Schmidt K. Farm questionnaires for monitoring the cultivation of genetically modified maize. In: Romeis J, Meissle M, editors; 1–3 June 2005; Lleida (Catalonia), Spain; 2006. p. 151–158.
- 20.
Schmidt K, Beißner L, Schiemann J, Wilhelm R. Methodology and Tools for Data Acquisition and Statistical Analysis. In: Bundesamt für Verbraucherschutz und Lebensmittelsicherheit, editor. Proceedings of the international workshop on "Post Market Environmental Monitoring of Genetically Modified Plant: Implementation of General Surveillance"; 27–28. April 2006; Berlin-Dahlem. 1st ed.: Birkhäuser Verlag; 2006. p. 21–25.
- 21.
Schmidt K, Schiemann J, Wilhelm R. European-wide GMO-monitoring data management and analysis. In: Bundesamt für Verbraucherschutz und Lebensmittelsicherheit, editor. Proceedings of the international workshop on "Post Market Environmental Monitoring of Genetically Modified Plant: Harmonisation and Standardisation—a Practical Approach; 26-27- April 2007; Berlin-Dahlem. 1st ed.: Birkhäuser Verlag; 2007. p. 11–15. https://doi.org/10.1007/s00003-007-0290-8
- 22.
Schmidt K, Sweet J, Beißner L, Wilhelm R, Böttinger P, Mönkemeyer W, et al. Data surveys, analysis and tools for post-market environmental monitoring—the European approach. In: International Society for Biosafety Research, editor. 10th International Symposium on the Biosafety of Genetically Modified Organisms; 16–21. November 2008; Wellington, New Zealand; 2008. p. 93.
- 23.
Wilhelm R, Beißner L, Schmidt K, Schiemann J. Der Einsatz von Fragebögen zur Einbindung der Landwirte in ein GVO-Monitoring: Using Farmers questionnaires as a tool for the monitoring of GMO. In: Biologischen Bundesanstalt für Land- und Forstwirtschaft Berlin und Braunschweig, editor. 54. Deutsche Pflanzenschutztagung; 20.-23. September 2004; Hamburg. 396th ed.; 2004. p. 290.
- 24. Wilhelm R, Sanvido O, Castanera P, Schmidt K, Schiemann J. Monitoring the commercial cultivation of Bt maize in Europe—conclusions and recommendations for future monitoring practice. Environ Biosafety Res. 2009;8:219–25. pmid:20883660
- 25.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2006: Czech Republic, France, Germany, Portugal, Slovakia and Spain; 2007.
- 26.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2007: Czech Republic, France, Germany, Poland, Portugal, Romania, Slovakia and Spain; 2008.
- 27.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2008: Czech Republic, Germany, Poland, Portugal, Romania, Slovakia and Spain; 2009.
- 28.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2009: Czech Republic, Portugal, Slovakia, Poland, Romania and Spain; 2010.
- 29.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2010: Czech Republic, Poland, Portugal, Romania, Slovakia, and Spain; 2011.
- 30.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2011: Czech Republic, Poland, Portugal, Romania, Slovakia, and Spain; 2012.
- 31.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2012: Czech Republic, Portugal, Romania, Slovakia, and Spain; 2013.
- 32.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2013: Czech Republic, Portugal, Romania, Slovakia, and Spain; 2014.
- 33.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2014: Czech Republic, Portugal, Romania, Slovakia, and Spain; 2015.
- 34.
Monsanto Europe S.A. Annual monitoring report on the cultivation of MON 810 in 2015: Czech Republic, Portugal, Romania, Slovakia, and Spain; 2016.
- 35. EFSA Panel on Genetically Modified Organisms. Scientific Opinion on the Annual post-market environmental monitoring (PMEM) report on the cultivation of genetically modified maize MON 810 in 2014 from Monsanto Europe S.A. EFSA Journal 2016. pmid:31024636
- 36. European Commission. Commission Decision (2009/770/EC) establishing standard reporting formats for presenting the monitoring results of the deliberate release into the environment of genetically modified organisms, as or in products, for the purpose of placing on the market, pursuant to Directive 2001/18/EC of the European Parliament and of the Council. Official Journal of the European Union. 2009;L 275:9–27.
- 37.
VIB. Effect of genetically modified crops on the environment; 2016.
- 38. Pellegrino E, Bedini S, Nuti M, Ercoli L. Impact of genetically engineered maize on agronomic, environmental and toxicological traits: a meta-analysis of 21 years of field data. Sci Rep. 2018;8:3113. pmid:29449686
- 39. Brookes G, Barfoot P. GM crops: global socio-economic and environmental impacts 1996–2016; 2018.
- 40. Hutchison WD, Burkness EC, Mitchell PD, Moon RD, Leslie TW, Fleischer SJ, et al. Areawide suppression of European corn borer with Bt maize reaps savings to non-Bt maize growers. Science. 2010;330:222–5. pmid:20929774
- 41. Dively GP, Venugopal PD, Bean D, Whalen J, Holmstrom K, Kuhar TP, et al. Regional pest suppression associated with widespread Bt maize adoption benefits vegetable growers. Proc Natl Acad Sci U S A. 2018;115:3320–5. pmid:29531049
- 42. Dillehay BL, Roth GW, Calvin DD, Kratochvil RJ, Kuldau GA, Hyde JA. Performance of Bt Corn Hybrids, their Near Isolines, and Leading Corn Hybrids in Pennsylvania and Maryland. Agronomy Journal. 2004;96:818.
- 43. Graeber JV, Nafziger ED, Mies DW. Evaluation of Transgenic, Bt-Containing Corn Hybrids. Journal of Production Agriculture. 1999;12:659–63.
- 44.
Bessin R. Bt corn refuges for corn borer management. UK Cooperative Extension Service;Entfact-128.
- 45. Novillo C, Fernández-Anero J, Costa J. Resultados en España con variedades de maíz derivadas de la línea MON 810, protegidas genéticamente contra taladros. Bol. San. Veg. Plagas. 2003;29:427–39.
- 46. Riesgo L, Areal FJ, Rodríguez-Cerezo E. How can specific market demand for non-GM maize affect the profitability of Bt and conventional maize?: A case study for the middle Ebro Valley, Spain. Span J Agric Res. 2012;10:867.
- 47. Tefera T, Mugo S, Mwimali M, Anani B, Tende R, Beyene Y, et al. Resistance of Bt-maize (MON810) against the stem borers Busseola fusca (Fuller) and Chilo partellus (Swinhoe) and its yield performance in Kenya. Crop Prot. 2016;89:202–8. pmid:27812235
- 48. Folcher L, Delos M, Marengue E, Jarry M, Weissenberger A, Eychenne N, et al. Lower mycotoxin levels in Bt maize grain. Agron. Sustain. Dev. 2010;30:711–9.
- 49. Scarpino V, Reyneri A, Vanara F, Scopel C, Causin R, Blandino M. Relationship between European Corn Borer injury, Fusarium proliferatum and F. subglutinans infection and moniliformin contamination in maize. Field Crops Research. 2015;183:69–78.
- 50. Bakan B, Melcion D, Richard-Molard D, Cahagnier B. Fungal growth and Fusarium mycotoxin content in isogenic traditional maize and genetically modified maize grown in France and Spain. Journal of Agricultural and Food Chemistry. 2002;50:728–31. pmid:11829636
- 51.
ORDEN APA/520/2003; 27 of February 2003.
- 52.
ORDEN de 23 de marzo de 1998 por la que se modifica el Reglamento General del Registro de Variedades Comerciales; 23 of March 1998.
- 53. EFSA Panel on Genetically Modified Organisms. Scientific Opinion of the Panel on Genetically Modified Organisms on applications (EFSA-GMO- RX-MON810) for the renewal of authorisation for the continued marketing of (1) existing food and food ingredients produced from genetically modified insect resistant maize MON810; (2) feed consisting of and/or containing maize MON810, including the use of seed for cultivation; and of (3) food and feed additives, and feed materials produced from maize MON810, all under Regulation (EC) No 1829/2003 from Monsanto. EFSA Journal. 2009;1149:1–85.
- 54. EFSA Panel on Genetically Modified Organisms. Guidance on the risk assessment of genetically modified microorganisms and their products intended for food and feed use. EFSA Journal. 2011;9:2193.