Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Incidence rates of narcolepsy diagnoses in Taiwan, Canada, and Europe: The use of statistical simulation to evaluate methods for the rapid assessment of potential safety issues on a population level in the SOMNIA study

  • Caitlin N. Dodd ,

    Roles Conceptualization, Formal analysis, Methodology, Visualization, Writing – original draft, Writing – review & editing

    c.n.dodd@umcutrecht.nl

    Affiliations Erasmus Medical Center, Rotterdam, The Netherlands, Julius Center Global Health, University Medical Center Utrecht, Utrecht, The Netherlands

  • Maria de Ridder,

    Roles Formal analysis, Methodology, Supervision, Writing – review & editing

    Affiliation Erasmus Medical Center, Rotterdam, The Netherlands

  • Wan-Ting Huang,

    Roles Data curation, Formal analysis, Writing – review & editing

    Affiliation Taiwan Centers for Disease Control, Taipei, Taiwan

  • Daniel Weibel,

    Roles Project administration, Supervision, Writing – review & editing

    Affiliation Erasmus Medical Center, Rotterdam, The Netherlands

  • Maria Giner-Soriano,

    Roles Data curation, Writing – review & editing

    Affiliations Institut Universitari d’Investigació en Atenció Primària Jordi Gol (IDIAP Jordi Gol), Barcelona, Spain, Universitat Autònoma de Barcelona, Bellaterra (Cerdanyola del Vallès), Barcelona, Spain

  • Silvia Perez-Vilar,

    Roles Data curation, Writing – review & editing

    Affiliations Erasmus Medical Center, Rotterdam, The Netherlands, Fundación para el Fomento de la Investigación Sanitaria y Biomédica de la Comunitat (FISABIO), Vaccine Research, Valencia, Spain

  • Javier Diez-Domingo,

    Roles Data curation, Writing – review & editing

    Affiliation Fundación para el Fomento de la Investigación Sanitaria y Biomédica de la Comunitat (FISABIO), Vaccine Research, Valencia, Spain

  • Lawrence W. Svenson,

    Roles Data curation, Writing – review & editing

    Affiliation University of Alberta, Division of Preventative Medicine, Alberta, Canada

  • Salahddin M. Mahmud,

    Roles Data curation, Writing – review & editing

    Affiliation University of Manitoba, Rady Faculty of Health Sciences, Winnipeg, Manitoba, Canada

  • Bruce Carleton,

    Roles Data curation, Writing – review & editing

    Affiliation University of British Columbia, Faculty of Medicine, Vancouver, British Columbia, Canada

  • Monika Naus,

    Roles Data curation, Writing – review & editing

    Affiliation University of British Columbia, Faculty of Medicine, Vancouver, British Columbia, Canada

  • Jeffrey C. Kwong,

    Roles Writing – review & editing

    Affiliations Institute for Clinical Evaluative Sciences, Toronto, Ontario, Canada, Public Health Ontario, Toronto, Ontario, Canada, Department of Family & Community Medicine, University of Toronto, Toronto, Ontario, Canada, Dalla Lana School of Public Health, University of Toronto, Toronto, Ontario, Canada, University Health Network, Toronto, Ontario, Canada

  • Brian J. Murray,

    Roles Validation, Writing – review & editing

    Affiliation Division of Neurology, Department of Medicine, Sunnybrook Health Sciences Centre, Toronto, Canada

  • Lisen Arnheim-Dahlstrom,

    Roles Data curation, Writing – review & editing

    Affiliation Karolinska Institut, Department of Medical Epidemiology and Biostatistics, Stockholm, Sweden

  • Lars Pedersen,

    Roles Data curation, Writing – review & editing

    Affiliation Aarhus University, Department of Clinical Epidemiology, Aarhus, Denmark

  • Rosa Morros,

    Roles Data curation, Writing – review & editing

    Affiliations Institut Universitari d’Investigació en Atenció Primària Jordi Gol (IDIAP Jordi Gol), Barcelona, Spain, Universitat Autònoma de Barcelona, Bellaterra (Cerdanyola del Vallès), Barcelona, Spain

  • Francisco Javier Puertas,

    Roles Data curation, Writing – review & editing

    Affiliations Sleep Unit, Neurophysiology Department, La Ribera University Hospital, Valencia, Spain, Physiology Department, University of Valencia, Valencia, Spain

  • Steven Black,

    Roles Conceptualization, Funding acquisition, Project administration, Supervision, Writing – review & editing

    Affiliation Cincinnati Children’s Hospital, Center for Global Health, Cincinnati, Ohio, United States of America

  •  [ ... ],
  • Miriam Sturkenboom

    Roles Conceptualization, Funding acquisition, Methodology, Project administration, Supervision, Writing – review & editing

    Affiliation Julius Center Global Health, University Medical Center Utrecht, Utrecht, The Netherlands

  • [ view all ]
  • [ view less ]

Abstract

Background & objectives

Vaccine safety signals require investigation, which may be done rapidly at the population level using ecological studies, before embarking on hypothesis-testing studies. Incidence rates were used to assess a signal of narcolepsy following AS03-adjuvanted monovalent pandemic H1N1 (pH1N1) influenza vaccination among children and adolescents in Sweden and Finland in 2010. We explored the utility of ecological data to assess incidence of narcolepsy following exposure to pandemic H1N1 virus or vaccination in 10 sites that used different vaccines, adjuvants, and had varying vaccine coverage.

Methods

We calculated incidence rates of diagnosed narcolepsy for periods defined by influenza virus circulation and vaccination campaign dates, and used Poisson regression to estimate incidence rate ratios (IRRs) comparing the periods during which wild-type virus circulated and after the start of vaccination campaigns vs. the period prior to pH1N1 virus circulation. We used electronic health care data from Sweden, Denmark, the United Kingdom, Canada (3 provinces), Taiwan, Netherlands, and Spain (2 regions) from 2003 to 2013. We investigated interactions between age group and adjuvant in European sites and conducted a simulation study to investigate how vaccine coverage, age, and the interval from onset to diagnosis may impact the ability to detect safety signals.

Results

Incidence rates of narcolepsy varied by age, continent, and period. Only in Taiwan and Sweden were significant time-period-by-age-group interactions observed. Associations were found for children in Taiwan (following pH1N1 virus circulation) and Sweden (following vaccination). Simulations showed that the individual-level relative risk of narcolepsy was underestimated using ecological methods comparing post- vs. pre-vaccination periods; this effect was attenuated with higher vaccine coverage and a shorter interval from disease onset to diagnosis.

Conclusions

Ecological methods can be useful for vaccine safety assessment but the results are influenced by diagnostic delay and vaccine coverage. Because ecological methods assess risk at the population level, these methods should be treated as signal-generating methods and drawing conclusions regarding individual-level risk should be avoided.

Introduction

In August 2010, a safety signal of narcolepsy following AS03-adjuvanted pdm(09)H1N1 influenza vaccine Pandemrix® was reported in Finland and Sweden among children and adolescents [1]. Other rapid risk assessment studies conducted in the European Union (EU) did not show changes in incidence rates of narcolepsy diagnoses, except in Finland, Sweden, and Norway [2], all countries that achieved high coverage rates with Pandemrix. Subsequent hypothesis-testing studies showed associations; these had high within- and between-study variation [3]. In China, where vaccine coverage was very low, a 3-fold increase in narcolepsy onset was reported following the peak of the pandemic [4].

Narcolepsy is a rare disease with a long interval from onset of symptoms to diagnosis, especially in adults. Several possible explanations for the purported pdm(09)H1N1 and narcolepsy link have been proposed but none confirmed. Hypotheses range from a causal effect of the AS03 adjuvant, the manufacturing process, presence of nucleoproteins in Pandemrix, and molecular mimicry, to awareness and assessment biases, and residual confounding [510]. Based upon simulation studies conducted by Wijnans et al., in the absence of a causal association but in the presence of accelerated diagnosis due to awareness, we would expect to see an increased incidence of narcolepsy diagnosis following awareness of the association followed by a decrease, even to levels below the background incidence, due to depletion of cases[8]. This effect may be particularly important in conditions with a long delay to diagnosis such as in narcolepsy where the delay in diagnosis from initial symptoms can be 10–20 years [11].

The SOMNIA (Systematic Observational Method for Narcolepsy and Influenza Immunization Assessment) study was funded by the US Centers for Disease Control and Prevention (CDC) and used information from countries that used different types of adjuvanted pandemic influenza vaccines to assess whether the pdm(09)H1N1 influenza vaccine and specifically the MF59 and AS03 adjuvants were associated with narcolepsy.

One of the goals of SOMNIA was to assess patterns of incidence rates of narcolepsy in multiple geographic areas and to understand changes in incidence rates of narcolepsy diagnoses before, during, and after the pdm(09)H1N1 influenza pandemic by using electronic health care data, which may be rapidly available. In this paper, we explore whether assessment of safety signals based on ecological methods and population-based electronic health care data are suitable for vaccine safety risk assessment, by exploiting the heterogeneity in vaccine coverage, types of vaccines, and vaccination programs across countries. We assess what strength of signals can be detected using population-level data collected before and after a hypothetical targeted vaccination campaign.

Ecological studies can be defined as those that measure exposure and outcomes at the group level rather than at the individual level [12, 13]. In such a study, groups are defined by a naturally occurring difference in space or time such as a change in the vaccination schedule [14] or the beginning and end of a targeted vaccination campaign [15].

This study may serve as an example of the utility of ecological methods to assess vaccine safety signals, particularly regarding events with long onset-to-diagnosis intervals.

Materials and methods

Narcolepsy diagnosis incidence rates were evaluated in ten sites representing seven countries spanning three continents (Taiwan (TW), Canada (CA) [Manitoba, Alberta, and British Columbia], The Netherlands (NL), The United Kingdom (UK), Sweden (SE), Denmark (DK), and Spain (ES) [Valencia and Cataluña]) using population-based electronic healthcare databases originating from general practitioners (GPs) (UK, ES, NL) or claims/record linkage databases (SE, DK, TW [1618], and CA) (S1 Table).

Study population and follow-up

For data sources in which individual linkage can be made between population and diagnoses (all sites except Sweden and British Columbia, Canada), the study population comprised all individuals registered within each of the databases during the study period. Observation time began on the date of first registration of an individual in the database, the start of the study period (January 2003), or the start date of data collection for the database, whichever was the latest and ended on the date of death, the date registration was terminated, the end of data collection, or the end of the study period (December 2013), whichever was the earliest. Sweden and British Columbia, Canada used census data to calculate person-time denominators. We used a harmonized approach in which databases locally extracted their data into simple input files in a common format that could be locally analyzed and aggregated using SAS or JAVA-based software [2, 19].

Case identification and validation

Cases were persons with a new diagnosis of narcolepsy with or without cataplexy. Validation of the diagnostic codes using patient discharge letters and medical records was conducted in the GP databases in the Netherlands and Valencia, Spain. For these two sites, only validated cases were used in the analysis. The other sites used algorithms combining diagnostic codes for narcolepsy with claims for multiple sleep latency tests (MSLTs) to reduce the false positive rate. The same method was used at each site over the entire time period. No further validation was done in other sites (S1 Table).

Analysis

To investigate the purported narcolepsy-pandemic vaccine effect, incidence rates of narcolepsy diagnosis were calculated by calendar year and month and also categorized into three periods based on specific circulation/vaccination periods in each country: 1) pre-pandemic (from January 2003 until the start of the period of pH1N1 circulation); 2) during pH1N1 wild-type virus circulation until the start of the country’s pH1N1 vaccination campaign; and 3) from the start of the pH1N1 influenza vaccination campaign through the end of the study (S1 Table). Pandemic H1N1 virus circulation was defined as the period during which weekly influenza test positivity for pH1N1 infection exceeded 10%. Dynamic age groups were categorized as <5 years, 5–19 years, 20–59 years, and ≥60 years at the time of diagnosis. This categorization was motivated by differences in diagnosis for each age group, and particularly the challenges of differential diagnosis in young children and the elderly [20, 21]. Incidence rates of narcolepsy diagnoses were calculated by dividing the number of narcolepsy cases by the accumulated person-time. Ninety-five percent confidence intervals (CIs) were calculated assuming a negative binomial distribution. Following confirmation of homogeneity in incidence rates among databases within the same country, further analyses were conducted at the level of the country rather than the site.

Within each country, we estimated incidence rate ratios (IRRs) and 95% CIs for each time period using Poisson regression, with the pre-circulation period as a reference. We included terms for age strata, time periods, and an age*time period interaction using time periods as defined by pH1N1 circulation and vaccination campaign dates.

We conducted additional analyses restricted to European countries to estimate the impact of vaccine coverage and adjuvant among children and adolescents and separately among adults. For this analysis, a composite variable summarizing vaccine coverage classified as low (<20%) or high (≥20%) and adjuvant (MF59 or AS03) was created, and incidence in the period after vaccination had started was compared to the pre-pH1N1 circulation period. Because the composite adjuvant/coverage variable was collinear with database and country, neither database nor country was included in the European model.

Simulation

To better understand the utility of ecological methods for assessing vaccine safety signals and whether an association in one age group may be masked in a population-level analysis, we conducted statistical simulations. Each set consisted of 10,000 subjects aged 0 to 100 years, who were observed in the period January 1, 2003 to December 31, 2012. The period of the vaccination campaign was set from October to December 2009. In each simulated data set, vaccine coverage during this period was set at one out of 9 different values between 1 and 99 percent (see Table 1). Subjects assigned as vaccinated got a vaccination date during the campaign period. Baseline narcolepsy incidence was simulated based on reported estimates of incidence by decade of age [22].For vaccinated persons aged <20 years, during the six months following vaccination, the risk for narcolepsy onset was simulated using a relative risk varying from 0.5 to 10 compared to the baseline incidence. In subjects aged ≥20 years no increased risk was applied. The median time from onset to diagnosis was initially set at 4 years for adults and at 1.5 years for children (aged ≤18 years) based on SOMNIA data (not shown). The effect of the length of the interval from onset to diagnosis was also tested by varying a reduction rate parameter, with 0 removing the interval (i.e. immediate diagnosis following onset), 0.5 halving the interval, and 1 retaining the full simulated interval (see Table 1 for description of simulation parameters). Finally, in part of the sets the full 10 years was used as study period, in the others it was restricted to 6 months after the end of the vaccination campaign (7.5 years).

For each set of simulation parameters, 500 replications were run. The population-level incidence rate ratio for the period following the vaccination campaign vs. the period prior to the vaccination campaign was estimated using Poisson regression. The median incidence rate ratio from the 500 replications was calculated overall and by age group, and plotted against vaccine coverage. The percentage bias was calculated by subtracting the simulated relative risk from the estimated IRR and dividing by the simulated IRR.

Calibration

Using the simulation results, a model was constructed to determine how the underlying individual-level relative risk was related to the median estimated IRR and the appointed vaccine coverage and interval reduction, and the interaction of these terms (Model: Simulated individual-level true RR = Estimated IRR + Simulated vaccine coverage + Simulated onset-to-diagnosis reduction + Interactions). This model was then applied to results obtained from the observed data, using the IRR found in the SOMNIA study and the relevant vaccine coverage and diagnostic delay (S1 Table), in order to calculate the underlying relative risks in 5- to 19-year olds, that would have been necessary to produce this IRR in the absence of other sources of bias. This calibration was restricted to the 5–19 year age group as this group was the source of the safety signal and the only subjects for whom an increased risk was simulated.

This study was conducted under the principles of the Helsinki declaration and each site was responsible for obtaining appropriate ethical approvals. The overall study was approved by the Institutional Review Board at Cincinnati Children’s Hospital, Cincinnati, Ohio, USA. The study was also approved at each site by the following bodies: Health Research Ethics Board, Alberta, Canada; University of Manitoba Research Ethics Board, Manitoba, Canada; Ethics Committee for Clinical Research of the Directorate General of Public Health and Center for Advance Research in Public Health (CEIC DGSP/CSISP), Valencia, Spain; Comitè Ètic d'Investigació de l'IDIAP Jordi Gol, Cataluña, Spain; Regional Ethics Review Board, Stockholm, Sweden; The Institutional Review Board of Centers for Disease Control, Ministry of Health and Welfare, Taiwan; THIN Scientific Review Committee, United Kingdom; Erasmus Medical Center Medical Internal Review Board, The Netherlands. Due to use of aggregate data, British Columbia, Canada and Denmark conducted the study under rolling approval and study-specific approval was not sought.

Results

Observed incidence

Incidence rates of narcolepsy diagnoses ranged from 0.22 to 1.52 per 100,000 person-years by site (Table 2). Incidence rates in databases within the same country (Canada and Spain) were similar so for further analysis country-specific data were pooled.

Due to very low rates observed among the very young (<5 years) and the elderly (≥60 years) (S2 Table) and known difficulties in diagnosis in these age groups, these were not included in further analyses. In Fig 1, IRs are shown stratified by age group and time period. In investigation of age group and time period and the interaction of these factors, IRRs were significantly elevated in both age groups in Taiwan (where MF59-adjuvanted vaccine coverage was 59% for those <19 years and 11% for those ≥19 years) in the period during circulation of wild-type virus prior to vaccination. For those aged 5–19 years, the IRR was 2.50 (95%CI 1.46, 4.28), and for those aged 20–59 years, the IRR was 2.23 (95%CI 1.26, 3.94). This continued in the period after the vaccination campaign had started, with IRR 1.60 (95%CI 1.20, 2.13) for those aged 5–19 years and IRR 2.13 (95%CI 1.62, 2.79) for those aged 20–59 years (Table 3). In Sweden, where AS03-adjuvanted Pandemrix vaccine coverage was 60%, in the period after vaccination incidence rates among those aged 5–19 years and 20–59 years were elevated [IRR = 9.01 (95%CI 6.89, 11.80) and IRR = 1.69 (95%CI 1.46, 1.95)], respectively (Fig 1 & Table 3). None of the other countries showed significant time-period-by-age-group interactions.

thumbnail
Fig 1. Incidence rates (with 95% CIs) of narcolepsy pre-circulation and post-vaccination in children (05–19 yrs.) and adults (20–59 yrs.).

https://doi.org/10.1371/journal.pone.0204799.g001

In the analysis restricted to Europe and including a vaccine coverage/adjuvant composite variable, IRRs were elevated in the period following start of vaccination in the high-coverage AS03 (Sweden) and low-coverage AS03 groups for children and adolescents (S3 Table). In adults, an elevated incidence in the period following vaccination was detected in the AS03 high-coverage group, which was limited to Sweden. In this analysis, no changes in the incidence of narcolepsy in the post-vaccination period were seen in sites using MF59-adjuvanted vaccine, all of which had low coverage (S3 Table).

Simulation

The simulation study showed that in an analysis such as the one described above, the true RR is consistently underestimated when it is greater than one and overestimated when less than one (Fig 2). Underestimation of true relative risks greater than one is attenuated as vaccination coverage increases but the estimate remains about 6%-26% too low even with vaccination coverage as high as 99% with no delay from onset to diagnosis and a 10-year observation period (Fig 3). As the interval from onset to diagnosis increases, the estimated relative risk deviates further from the true relative risk; performance is improved if the observation time captures only the period of increased risk (data not shown). When the time from onset to diagnosis was not reduced no increased risk was detected for any set of simulation parameters (data not shown). Stratification by age group was effective in elucidating the group that was the source of the increased risk.

thumbnail
Fig 2. IRR estimates in simulated data, immediate diagnosis (interval reduction rate = 0).

Population-level incidence rate ratio estimated from simulated data with observation time equal to 3625 days and true individual-level relative risk equal to .05, 1, 2, 5, or 10 (columns). Gray horizontal reference lines represent the true simulated individual-level relative risk of narcolepsy diagnosis. The interval reduction rate parameter is set equal to zero (meaning immediate diagnosis following onset of symptoms). Vaccination coverage increases within each column along the x-axis. Colored lines represent age group-specific IRRs as noted in the legend and colored bands represent associated 95% confidence intervals.

https://doi.org/10.1371/journal.pone.0204799.g002

thumbnail
Fig 3. Percent bias of IRR estimates in simulated data, immediate diagnosis (interval reduction rate = 0).

Population-level percent bias estimated from simulated data with observation time equal to 3625 days and true individual-level relative risk equal to .05, 1, 2, 5, or 10 (columns). The (interval reduction rate parameter is set equal to zero (meaning immediate diagnosis following onset of symptoms). Vaccination coverage increases within each column along the x-axis. Colored lines represent age group-specific bias as noted in the legend.

https://doi.org/10.1371/journal.pone.0204799.g003

Calibration

When using a model derived from the results of the simulation study to predict which should have been the true (calibrated) IRR to produce the estimates found in the SOMNIA study, these true IRRs were always considerably higher than the estimates. Notably, according to the model, the required individual-level relative risk underlying a 9-fold IRR which was found in 5–19 year olds in Sweden should have been 36.04 (95% CI: 27.79, 46.90) (Table 4).

thumbnail
Table 4. IRRs calibrated using model derived from simulated data.

https://doi.org/10.1371/journal.pone.0204799.t004

Discussion

Evaluation of incidence rates on a population level can be done relatively quickly in countries/regions with accessible population-based electronic health care databases. This is useful for assessing potential vaccine safety signals. In order to calculate rates quickly in a standardized manner, harmonization of data into simple input files in a common format allowed for the pooling and sharing of data across three continents. The method was capable of identifying the signal in Sweden in 5- to 19-year olds.

In the analysis by country, elevated rates of narcolepsy were only detected in Taiwan during wild-type virus circulation through the period following vaccination with MF59-adjuvanted and non-adjuvanted vaccines, and in Sweden following vaccination with AS03-adjuvanted vaccines. The finding in Taiwan may be due to circulation of wild-type influenza virus prior to the start of the vaccination campaign [17]. This is consistent with the finding of a 3-fold increase in narcolepsy onset in China following the peak of the pH1N1 pandemic in a population with very low vaccine coverage [4]. Taiwan vaccinated children aged <1 year with MF59-adjuvanted vaccine and adults and school children with mainly non-adjuvanted vaccine. In Sweden, where the signal of a narcolepsy safety concern was originally detected [23] and where patients diagnosed with narcolepsy are being compensated [7], rates were much higher than in the other countries. This could be due to differential reporting due to increased awareness of the putative association, a true causal effect in this population with this vaccine, or some combination of these factors. As shown in simulations, reduction in the time from onset to diagnosis due to awareness of an association can lead to artificial inflations in risk estimates [8]. In Canadian provinces, with around 40% vaccine coverage of a different AS03-adjuvanted vaccine (ArepanrixTM), no effect was seen in any of the age groups or periods. This study, which is by necessity observational, has several limitations. Data were collected according to a shared protocol but using locally derived algorithms, which may have led to differences in sensitivity and specificity. Case validation in some sites revealed low specificity of the original extraction, which may be the case in other sites as well. In our analysis by adjuvant and vaccine coverage, high coverage with AS03-containing vaccine was only present in Sweden, making Sweden and this adjuvant/coverage group collinear. This makes it impossible to determine whether we are seeing the effects of the vaccine itself or of the reporting and detection patterns in each country. Additionally, the manufacturing process of ArepanrixTM differed from that of PandemrixTM, leading to vaccines containing different quantities of influenza virus components [24]. The potential effects of these differences in manufacturing cannot be differentiated from adjuvant specific-effects or from other country-specific effects using an ecological design such as the one presented here. Similarly, the countries in which MF59 was used were the same countries in which case validation was conducted. This limits comparability between these countries and others and, therefore, between MF59-containing vaccines and other pandemic vaccines.

Differences in case ascertainment could also have impacted our estimates. For example, it has been noted elsewhere that the safety signal originated in Sweden and that this, together with compensation for cases, may have impacted diagnosis patterns [6, 7]. Additionally, due to the healthcare system in Taiwan, children complaining of excessive daytime sleepiness are seen by a specialist quickly, making the interval from onset to diagnosis for these children shorter. A median time from symptom onset to MSLT referral of 60 days has been reported for pediatric narcolepsy cases in Taiwan[25]. While it is not possible to rule out a causal association, it is important to note that these factors undoubtedly contributed to the estimates obtained in this study. Differences in the prevalence of the underlying HLA-DQB1*06:02 risk allele for narcolepsy, which has been reported to vary widely by country, may affect the incidence at the population level but is unlikely to have affected relative risk estimates [2630].

Ecological methods, when applied to assessment of a signal association with a targeted vaccination campaign and a disease with a potentially long interval from onset to diagnosis, can provide an unbiased estimate of vaccine-associated risk in a very limited set of circumstances. Obtaining an unbiased estimate in the absence of an association is possible even with very low vaccine coverage and a long onset to diagnosis interval. However, in the presence of a true vaccine-associated risk, all estimates will be biased toward one; this bias is reduced when cases are detected quickly and vaccination coverage is high. Based upon simulations, the estimates we obtained in the current study appear to be underestimations of true relative risks greater than one. However, the simulations did not take into account the possibility of increased reporting due to an awareness of the association, which has been shown in previous simulations to inflate risk estimates [8].

The predicted underlying individual-level relative risk obtained using models derived from simulated data, given the low vaccine coverage attained in most sites, are remarkably high. It is very unlikely that the true relative risk in Sweden, for example, is 36-fold. Calibrated estimates for The United Kingdom, however, are only slightly lower than those reported by Miller et al in their study of narcolepsy following PandemrixTM vaccination in children aged 4–18 [31]. In general, these predicted relative risks are not in line with results found in the case-control study conducted within SOMNIA, in which no increased risk following pH1N1 vaccines was detected [32, 33]. It is important to note that while the case-control data in the SOMNIA study did not include any pediatric cases exposed to PandemrixTM, the case-coverage sub-study of pediatric cases in The Netherlands did not find an association with PandemrixTM [33]. These inconsistencies may be an illustration of the ecological fallacy, namely that associations detected at the population level may not be causal at the individual level [34]. In fact, as coverage in our simulations approached 100%, our population-level analysis also approached an individual-level analysis with accurate exposure data for all subjects, explaining why increased vaccine coverage in simulations leads to more accurate estimates of the simulated relative risk.

Previous simulations have shown that reduction in the time from onset to diagnosis following awareness of an association increase risk estimates [8]. Our simulation did not take into account factors that may have changed over the course of the study period such as awareness of narcolepsy and of the pH1N1-narcolepsy association as well as changes in diagnostic and coding practices. Each of these likely contributed to the IRR estimates we obtained. What our simulations do show is that in the absence of factors that increase case detection in the post-exposure period, detection of increased population-level risk of a disease with a long onset to diagnosis interval using ecological methods requires an extreme underlying individual-level risk.

The ecological approach fails to detect any increased risk unless the time from onset to diagnosis is short and both coverage and the true relative risk are high. Because of this, we recommend that population-level methods be used in assessment of outcomes with a delay from onset to diagnosis only to generate hypotheses or to strengthen signals when population-level exposure is high. Analysis of the full population when increased risk is only present in one age stratum performs as well as stratified analysis in terms of the magnitude of risk detected, but fails to identify the source of the increased risk. Therefore, if increased risk is suspected in a subset of the population, analyses should be stratified.

Conclusions

Ecological methods can be useful in assessment of vaccine safety but it is important for investigators to understand the impacts of masking by strata not at risk, patterns of onset and diagnosis, and vaccine coverage. What appears to be an estimate of no effect could be valid or, as shown in our simulations, could be an underestimation.

Supporting information

S1 Table. Characteristics of the Databases in this study.

* Linked Medical Records = Population based medical records (GP and specialist diagnoses), directly linked; Population-based registry = Population based registries (emergency room, in and out patient diagnoses); Medical Record diagnoses + Census Population = In and outpatient diagnoses, case counts and population counts (census);

1. http://www.epic-uk.org/our-data/our-data.shtml

2. http://www.erasmusmc.nl/med_informatica/research/555688/?lang=en#

3. http://www.kea.au.dk/en/ResearchRegistries.html

4. http://www.sidiap.org/index.php/en

5. http://fisabio.san.gva.es/en/fisabio;jsessionid=AFE38E9ACF0A380A692A9739E88F2FF4

6. http://www.socialstyrelsen.se/english

7. http://www.mohw.gov.tw/CHT/DOS/DM1.aspx?f_list_no=812 (Chinese)

8. http://www.health.alberta.ca/documents/Research-Health-Datasets.pdf

9. http://umanitoba.ca/faculties/health_sciences/medicine/units/chs/departmental_units/mchp/resources/repository/index.html

10. https://www.popdata.bc.ca/data.

https://doi.org/10.1371/journal.pone.0204799.s001

(DOCX)

S2 Table. Incidence Rates and Incidence Rate Ratios by continent, country, age, and period.

*Periods are as follows: Pre-Circulation = January 2003-the beginning of wild-type H1N1 circulation (defined per country); Circulation = Period from the beginning of wild-type H1N1 circulation until the start of the vaccination campaign (defined per country); Vaccination & Post = Period from the beginning of the vaccination campaign through December 2013.

† IRR comparing the period to the pre-circulation period, within the age group

‡The count has been suppressed either because (1) the observed number of events is very small (n ≤ 2) and not appropriate for publication; or (2) it could be used to calculate the number in a cell that has been suppressed.

https://doi.org/10.1371/journal.pone.0204799.s002

(DOCX)

S3 Table. IRRs of Post-Vaccination vs. Pre-Circulation in categories of Coverage and Adjuvant.

https://doi.org/10.1371/journal.pone.0204799.s003

(DOCX)

Acknowledgments

The authors would like to thank Tom Shimabukuro and Frank DeStefano for oversight and facilitation of the SOMNIA project.

Disclaimer: All inferences, opinions, and conclusions drawn in this publication are those of the author(s), and do not necessarily reflect the opinions or policies of the British Columbia Ministry of Health.

References

  1. 1. Nohynek H, Jokinen J, Partinen M, Vaarala O, Kirjavainen T, Sundman J, et al. AS03 adjuvanted AH1N1 vaccine associated with an abrupt increase in the incidence of childhood narcolepsy in Finland. PloS one. 2012;7(3):e33536. Epub 2012/04/04. pmid:22470453; PubMed Central PMCID: PMC3314666.
  2. 2. Wijnans L, Lecomte C, de Vries C, Weibel D, Sammon C, Hviid A, et al. The incidence of narcolepsy in Europe: before, during, and after the influenza A(H1N1)pdm09 pandemic and vaccination campaigns. Vaccine. 2013;31(8):1246–54. Epub 2012/12/19. pmid:23246544.
  3. 3. Control ECfDPa. Narcolepsy in association with pandemic influenza vaccination (a multi-country European epidemiological investigation). Stockholm2012.
  4. 4. Han F, Lin L, Warby SC, Faraco J, Li J, Dong SX, et al. Narcolepsy onset is seasonal and increased following the 2009 H1N1 pandemic in China. Annals of neurology. 2011;70(3):410–7. Epub 2011/08/26. pmid:21866560.
  5. 5. Bollaerts K, Shinde V, Dos Santos G, Ferreira G, Bauchau V, Cohet C, et al. Application of Probabilistic Multiple-Bias Analyses to a Cohort- and a Case-Control Study on the Association between Pandemrix and Narcolepsy. PloS one. 2016;11(2):e0149289. Epub 2016/02/24. pmid:26901063; PubMed Central PMCID: PMC4762678.
  6. 6. Verstraeten T, Cohet C, Dos Santos G, Ferreira GL, Bollaerts K, Bauchau V, et al. Pandemrix and narcolepsy: A critical appraisal of the observational studies. Human vaccines & immunotherapeutics. 2016;12(1):187–93. Epub 2015/09/18. pmid:26379011; PubMed Central PMCID: PMC4962758.
  7. 7. Sturkenboom MC. The narcolepsy-pandemic influenza story: can the truth ever be unraveled? Vaccine. 2015;33 Suppl 2:B6–B13. Epub 2015/05/30. pmid:26022571.
  8. 8. Wijnans L, Dodd C, de Ridder M, Romio S, Weibel D, Overeem S, et al. Pandemic influenza vaccine & narcolepsy: simulations on the potential impact of bias. Expert review of vaccines. 2016;15(5):573–84. Epub 2016/03/12. pmid:26967200.
  9. 9. Ahmed SS, Schur PH, MacDonald NE, Steinman L. Narcolepsy, 2009 A(H1N1) pandemic influenza, and pandemic influenza vaccinations: what is known and unknown about the neurological disorder, the role for autoimmunity, and vaccine adjuvants. Journal of autoimmunity. 2014;50:1–11. Epub 2014/02/25. pmid:24559657.
  10. 10. Ahmed SS, Volkmuth W, Duca J, Corti L, Pallaoro M, Pezzicoli A, et al. Antibodies to influenza nucleoprotein cross-react with human hypocretin receptor 2. Science translational medicine. 2015;7(294):294ra105. Epub 2015/07/03. pmid:26136476.
  11. 11. Taddei RN, Werth E, Poryazova R, Baumann CR, Valko PO. Diagnostic delay in narcolepsy type 1: combining the patients' and the doctors' perspectives. Journal of sleep research. 2016;25(6):709–15. Epub 2016/05/07. pmid:27149919.
  12. 12. Farrington CP. Control without separate controls: evaluation of vaccine safety using case-only methods. Vaccine. 2004;22(15–16):2064–70. Epub 2004/05/04. pmid:15121324.
  13. 13. Morgenstern H. Ecologic studies in epidemiology: concepts, principles, and methods. Annual review of public health. 1995;16:61–81. Epub 1995/01/01. pmid:7639884.
  14. 14. Shields WD, Nielsen C, Buch D, Jacobsen V, Christenson P, Zachau-Christiansen B, et al. Relationship of pertussis immunization to the onset of neurologic disorders: a retrospective epidemiologic study. The Journal of pediatrics. 1988;113(5):801–5. Epub 1988/11/01. pmid:3263484.
  15. 15. da Silveira CM, Salisbury DM, de Quadros CA. Measles vaccination and Guillain-Barre syndrome. Lancet. 1997;349(9044):14–6. Epub 1997/01/04. pmid:8988116.
  16. 16. Hsing AW, Ioannidis JP. Nationwide Population Science: Lessons From the Taiwan National Health Insurance Research Database. JAMA internal medicine. 2015;175(9):1527–9. Epub 2015/07/21. pmid:26192815.
  17. 17. Chuang JH, Huang AS, Huang WT, Liu MT, Chou JH, Chang FY, et al. Nationwide surveillance of influenza during the pandemic (2009–10) and post-pandemic (2010–11) periods in Taiwan. PloS one. 2012;7(4):e36120. Epub 2012/05/01. pmid:22545158; PubMed Central PMCID: PMC3335813.
  18. 18. Huang WT, Chen WW, Yang HW, Chen WC, Chao YN, Huang YW, et al. Design of a robust infrastructure to monitor the safety of the pandemic A(H1N1) 2009 vaccination program in Taiwan. Vaccine. 2010;28(44):7161–6. Epub 2010/09/02. pmid:20804804.
  19. 19. Trifiro G, Coloma PM, Rijnbeek PR, Romio S, Mosseveld B, Weibel D, et al. Combining multiple healthcare databases for postmarketing drug and vaccine safety surveillance: why and how? Journal of internal medicine. 2014;275(6):551–61. Epub 2014/03/19. pmid:24635221.
  20. 20. Hood BM, Harbord MG. Paediatric narcolepsy: complexities of diagnosis. Journal of paediatrics and child health. 2002;38(6):618–21. Epub 2002/11/02. pmid:12410881.
  21. 21. Chakravorty SS, Rye DB. Narcolepsy in the older adult: epidemiology, diagnosis and management. Drugs & aging. 2003;20(5):361–76. Epub 2003/04/17. pmid:12696996.
  22. 22. Silber MH, Krahn LE, Olson EJ, Pankratz VS. The epidemiology of narcolepsy in Olmsted County, Minnesota: a population-based study. Sleep. 2002;25(2):197–202. Epub 2002/03/21. pmid:11902429.
  23. 23. MP. S. Occurrence of narcolepsy with cataplexy among children and adolescents in relation to the H1N1 pandemic and Pandemrix vaccinations–results of a case inventory study by the MPA in Sweden during 2009–2010. 2011.
  24. 24. Vaarala O, Vuorela A, Partinen M, Baumann M, Freitag TL, Meri S, et al. Antigenic differences between AS03 adjuvanted influenza A (H1N1) pandemic vaccines: implications for pandemrix-associated narcolepsy risk. PloS one. 2014;9(12):e114361. Epub 2014/12/17. pmid:25501681; PubMed Central PMCID: PMC4266499.
  25. 25. Huang W-TH, Chung-Yao ; Chen Hsi-Chung; Lee Hsin-Chun; Lin Hui-Chen; Hsieh Cheng-Fang; Wu Meng-Ni; Yang Chin-Hui. Case-control Analysis of Narcolepsy after 2009 H1N1 Pandemic Vaccination in Taiwan. ICPE; 25 January 2017; Montreal, Canada2017.
  26. 26. Chen PL, Fann CS, Chu CC, Chang CC, Chang SW, Hsieh HY, et al. Comprehensive genotyping in two homogeneous Graves' disease samples reveals major and novel HLA association alleles. PloS one. 2011;6(1):e16635. Epub 2011/02/11. pmid:21307958; PubMed Central PMCID: PMC3030609.
  27. 27. Kotb M, Norrby-Teglund A, McGeer A, El-Sherbini H, Dorak MT, Khurshid A, et al. An immunogenetic and molecular basis for differences in outcomes of invasive group A streptococcal infections. Nature medicine. 2002;8(12):1398–404. Epub 2002/11/19. pmid:12436116.
  28. 28. Tafti M, Hor H, Dauvilliers Y, Lammers GJ, Overeem S, Mayer G, et al. DQB1 locus alone explains most of the risk and protection in narcolepsy with cataplexy in Europe. Sleep. 2014;37(1):19–25. Epub 2014/01/02. pmid:24381371; PubMed Central PMCID: PMC3865351.
  29. 29. Balas A, Garcia-Sanchez F, Vicario JL. Allelic and haplotypic HLA frequency distribution in Spanish hematopoietic patients. Implications for unrelated donor searching. Tissue antigens. 2011;77(1):45–53. Epub 2010/12/16. pmid:21155721.
  30. 30. Crespi C, Mila J, Martinez-Pomar N, Etxagibel A, Munoz-Saa I, Priego D, et al. HLA polymorphism in a Majorcan population of Jewish descent: comparison with Majorca, Minorca, Ibiza (Balearic Islands) and other Jewish communities. Tissue antigens. 2002;60(4):282–91. Epub 2002/12/11. pmid:12472657.
  31. 31. Miller E, Andrews N, Stellitano L, Stowe J, Winstone AM, Shneerson J, et al. Risk of narcolepsy in children and young people receiving AS03 adjuvanted pandemic A/H1N1 2009 influenza vaccine: retrospective analysis. BMJ. 2013;346:f794. Epub 2013/02/28. pmid:23444425.
  32. 32. Black S, Miriam Sturkenboom, Jan Bonhoeffer, Frank Destefano, Tom Shimabukuro, Caitlin Dodd, Maria De Ridder, Daniel Weibel, Angela Gentile, Noberto Giglio, Salahddin Mahmud. The Risk of Narcolepsy Following Receipt of Adjuvanted Pandemic 2009 The Subtype of Influenza A Virus (H1N1) Vaccines: Results of the SOMNIA Global Collaborative Study. Open Forum Infectious Diseases. 2016;3(suppl 1.):792.
  33. 33. Steven Black MS, Maria de Ridder, Caitlin Dodd, Daniel Weibel, Jan Bonhoeffer, Tom Shimabukuro, Frank Destefano, Angela Gentil, Norberto Giglio. The SOMNIA Study: Assessment of the Risk of Narcolepsy following Adjuvanted 2009 H1N1 Pandemic Vaccines IDWeek; October 27, 2016; New Orleans, LA2016.
  34. 34. Greenland S, Morgenstern H. Ecological bias, confounding, and effect modification. International journal of epidemiology. 1989;18(1):269–74. Epub 1989/03/01. pmid:2656561.