http://orcid.org/0000-0001-7594-0361
Figures
Abstract
Introduction
Timely adverse event following immunisation (AEFI) signal event detection is essential to minimise further vaccinees receiving unsafe vaccines. We explored the proportional reporting ratio (PRR) ability to detect two known signal events with influenza vaccines with the aim of providing a model for prospective routine signal detection and improving vaccine safety surveillance in Australia.
Methods
Passive AEFI surveillance reports from 2008–2017 relating to influenza vaccines were accessed from the Australian SAEFVIC (Victoria) database. Proportional reporting ratios were calculated for two vaccine-event categories; fever and allergic AEFI. Signal detection sensitivity for two known signal events were determined using weekly data; cumulative data by individual year and; cumulative for all previous years. Signal event thresholds of PRR ≥2 and Chi-square ≥4 were applied.
Results
PRR provided sensitive signal detection when calculated cumulatively by individual year or by all previous years. Known signal events were detected 15 and 11 days earlier than traditional methods used at the time of the actual events.
Conclusion
Utilising a single jurisdiction’s data, PRR improved vaccine pharmacovigilance and showed the potential to detect important safety signals much earlier than previously. It has potential to maximise immunisation safety in Australia. This study progresses the necessary work to establish national cohesion for passive surveillance signal detection and strengthen routine Australian vaccine pharmacovigilance.
Citation: Clothier HJ, Lawrie J, Russell MA, Kelly H, Buttery JP (2019) Early signal detection of adverse events following influenza vaccination using proportional reporting ratio, Victoria, Australia. PLoS ONE 14(11): e0224702. https://doi.org/10.1371/journal.pone.0224702
Editor: Eric HY Lau, The University of Hong Kong, CHINA
Received: May 30, 2019; Accepted: October 19, 2019; Published: November 1, 2019
Copyright: © 2019 Clothier et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of confidentiality and data privacy regulations. Data are available from the SAEFVIC Murdoch Childrens Research Institute Data Access Coordinator (contact via 1300 882 924) for researchers who meet the criteria for access to confidential data.
Funding: SAEFVIC is funded by the Department of Health and Human Services, Victoria. Hazel Clothier receives an Australian Government Research Training Program Scholarship. Authors HC, JL and JB are employees of SAEFVIC. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Criticism of adverse event following immunisation (AEFI) signal event detection in Australia has included that detection occurred relatively late, unnecessarily exposing further Australians to unsafe vaccines [1]. Typically, signals have been detected only following a reporter or surveillance team member describing an anecdotal impression of a change in AEFI presentations or reports [2].
The most notable event occurred in 2010 when increased incidence of fever and febrile seizures occurred following seasonal influenza vaccines [3, 4]. This was subsequently identified as arising from increased reactogenicity of one tri-valent influenza vaccine brand (Fluvax® bioCSL). The initial alert was made by Western Australia emergency department physicians. The vaccine was withdrawn 10 days later, but not before additional cases had occurred, including a prolonged febrile seizure resulting in profound disability in a previously healthy 11-month-old child [5]. The multi-million dollar compensation decision recognized the delays in reporting and state and federal response processes that were identified in an independent enquiry [1, 5].
A separate signal event occurred in 2015 when retrospective analyses of AEFI with seasonal influenza vaccines identified a probable increase in allergy-related AEFI [6]. In this instance, severity was fortunately insufficient to warrant modification of the program. Nevertheless, it highlighted continuing insensitivity of the Australian passive surveillance systems to detect AEFI signal events in a timely manner [2]. The welcome introduction of active surveillance systems has helped, however these can be costly, time consuming and usually focus on known potential risks (new vaccines, or high risk groups) [7]. The backbone of AEFI surveillance remains passive systems and it is critical they be utilised optimally for timely signal detection.
A “signal” can be defined as incidence of AEFI occurring at a higher level than is normally expected. Signal detection in vaccine vigilance requires a multi-faceted approach as AEFI range from a rare occurrence of a severe AEFI to increased incidence or increased severity of a known, often frequently occurring, AEFI [8, 9]. While rare but particularly serious events can be detected through review of each individual report or active surveillance, an increased incidence in a more common AEFI is often more difficult to detect [10, 11], and has been described as akin to “finding a needle in the haystack” [12].
If there is known high potential for a change in AEFI incidence, as demonstrated with febrile and allergy-related AEFI related to seasonal influenza vaccines [6, 13, 14], the extra resources and intensity of active surveillance may be warranted [7]. However, for unanticipated changes in reporting, passive AEFI surveillance can provide effective means of signal detection if routine monitoring systems are applied [9]. A benefit of passive surveillance is the lower resource intensity and the stability of routine systems. The best signal detection methodologies will be those that have equal low resource requirements, do not require extensive computational skills and are easy to interpret [15].
Statistical methods for pharmacovigilance signal detection are well established but their performance characteristics remain incompletely defined [9, 16, 17]. While there are various justifications for the benefits of individual methods [18–22], no single approach is optimal in all situations [23–25] and a range of frequentist and / or Bayesian measures of disproportionality are used internationally [9, 16, 19, 26]. In Australia, the national regulator Therapeutic Goods Administration (TGA) conducts bi-monthly proportional reporting ratios (PRR) for collated national AEFI surveillance data accompanied by weekly clinical review of line lists [27, 28]. However, there are no published reports on their routine use in Australia, or evaluation of their ability to detect signal events. Investigation of these signal detection tools and applicability within the resource structure of Australian systems is therefore timely.
Proportional reporting ratio (PRR)
This analysis focusses on proportional reporting ratio as a measure of disproportionality [29, 30]; chosen because it is an established signal detection algorithm (SDA) in pharmacovigilance [31, 32] and easy and practical to employ. Moreover it was an approach recommended in the response to national review into the management of a significant adverse event [33, 34].
The PRR is defined as the ratio between the frequency with which a specific adverse event is reported for the vaccine of interest (relative to all adverse events reported for the vaccine) and the frequency with which the same adverse event is reported for all vaccines in the comparison group (relative to all adverse events for vaccines in the comparison group). Although the PRR is an established measure, the definition of the comparator group used in calculations can vary, hence it is important to evaluate its performance prior to implementation in a new dataset.
We explore the PRR SDA as a simple to use, well defined SDA on passive vaccine AEFI surveillance data from the Victorian state vaccine safety service, SAEFVIC, using influenza vaccines as an example with the aim of providing a model for using an SDA and improving pharmacovigilance in Australia.
Objectives
To apply PRR to routine SAEFVIC surveillance data for specific AEFI with seasonal influenza vaccines and
- Determine the time period for data calculation sufficient to meet case count requirements
- Determine ability to detect known safety signals
- 2010 fever/febrile seizures (Fluvax® brand specific)
- 2015 allergic reactions
- Investigate impact of vaccine brand and age subgroup analyses on signal detection
- Ascertain generation of other potential / spurious signals
Methods
SAEFVIC is approved under the auspices of the Human Research Ethics Committee, Royal Childrens Hospital, Victoria to manage the clinical and public health surveillance database in compliance with national and jurisdictional privacy legislation (No. 37194). Data for this study were used within the public health surveillance functions for which it is approved. No patient identifyers were included in the extraction fields to generate the PRR.
Data source
Data from AEFI reports submitted to SAEFVIC database from January 2008 to December 2017 (as at 4 April 2018) were extracted by reported date, imported and analysed in Microsoft Power BI (Version 2.56.5023.2021, March 2018) [35].
Annual reporting periods for calendar week 10 to 40 were chosen to coincide with seasonal influenza vaccine program, although it was noted that in 2015, week 16 was the first week of reporting due to late commencement of the seasonal vaccination program [36].
Seasonal influenza vaccines
In Australia annual influenza vaccination is recommended for anyone six months of age and older, with persons aged 65 years and over, pregnant women and those who suffer chronic conditions eligible for free vaccine under the National Immunisation Program [37]. Seasonal influenza vaccines have changed across the years and may be funded differently for different target groups (e.g children / health care workers). Reports for all seasonal influenza vaccines were combined for analyses, except where signal investigation of brand specific reactogenicity were performed. Where junior formulations were reported, they were grouped with the parent brand for analysis. Monovalent pandemic influenza vaccines H1N1Pdm2009 were excluded.
AEFI reactions
AEFI reactions are categorised by SAEFVIC immunisation nurses according pre-determined case definitions based on symptom description [35]. We created two adverse event groups collating AEFI relating to fever or allergic reactions (below).
- Fever (fever unmeasured, fever ≥38<40, fever ≥40 and febrile seizure)
- Allergy-related (allergic reaction generalised, urticaria, anaphylaxis, angioedema)
Calculations
Proportional reporting ratio were calculated according to a 2x2 table (Table 1).
PRR = [a/(a+b)]/[c/(c+d)]
Chi-square (ChiSq) calculated as [12, 38]:
At 1 degree of freedom and a 5% error the tabulated value is 3.84.
Empirical thresholds were applied [12, 39], being:
- a minimum vaccine-event case-count (a) of 3
- signal alert threshold of PRR ≥2
- statistical significance test of ChiSq ≥4
Proportional reporting ratios were calculated for seasonal influenza vaccines with each of the adverse event-pair groups (fever and allergic reactions).
Three time period analyses were tested comparing the influenza vaccine-event ratio with the rationale that cumulative data would provide larger sample size for disproportionality assessment:
- By week reported in that week alone
- Cumulative by week in that individual year, and
- Cumulative by week with all previous years data.
Signal detection
Analyses were undertaken to ascertain if PRR would have assisted detection of known/suspected signal events (sensitivity):
Brand subgroup analysis
If a signal was suggested (PRR ≥ 2 and ChiSq ≥4), the PRR was calculated for influenza vaccines as a specific subgroup, whereby each of that seasons individual influenza brands (as a ratio to the other influenza vaccine brands used in that year) to ascertain brand specific contribution to the increased PRR [38].
Age subgroup analysis
Two age groups were defined as children (0–4 years) and 5 years and older reflecting the Australian National Immunisation Program [37]. PRR were calculated using single year and all previous year denominator data to ascertain impact of age subgroup on signal detection and signal magnitude.
Past signal subanalysis
It was considered possible that inclusion of increased reporting from signal events in the cumulative all-years denominator may reduce the proportional ratio and mask a signal in subsequent years. Therefore we re-examined the data by excluding 2010 and 2015 from the respective cumulative all-year denominator analyses for fever and allergic reaction AEFI in calculations for each year following the known signal event.
Results
A total of 12,152 AEFI reports were accessed of which 1649 (13.6%) related to administration of a seasonal influenza vaccine (Table 2). There was higher reporting for females receiving influenza vaccines (male:female ratio 1:2.2) than with all vaccines (male:female ratio 1:1.2) reflecting the funded influenza vaccine program for healthworkers, who are predominantly female. The mean time to report was 4 days with 95% of all reports received within 6 months. Reports by influenza vaccine brand are provided in supplementary data (S1 Table).
PRR calculation by time period
Reporting week.
Due to low case numbers reported in any one week, the threshold of a ≥3 was reached irregularly and in some years not at all, resulting in inconsistent reporting of PRR for both fever and allergic AEFI categories. We therefore progressed to cumulative analyses.
Cumulative by week and individual year.
Monitoring of data cumulatively in each year provided case numbers consistently above the calculation threshold (a ≥3) for PRR calculation. For Influenza vaccines and fever reactions, a PRR ≥2 was attained only in 2010 when in fact the PRR was >2 and ChiSq >4 in all of the 17 weeks of reporting (Fig 1).
For influenza vaccines and allergic reactions, signal threshold of PRR ≥2 and ChiSq ≥4 was attained in the first two weeks of reporting in 2011 before subsiding below threshold level as the number of reports received increased. Signal thresholds were reached consistently across all reporting weeks in 2015 (Fig 2).
Cumulative by week and all previous years data.
Both signals were detected using cumulative data for all years available (i.e., proportion of all other vaccines with the AEFI of interest in all previous years to the same reporting week time point) (Figs 3 and 4).
A comparison of PRR and ChiSq calculated using single year data or using all previous years data demonstrated variable effect on the signal magnitude (Table 3). When using all-years data in the denominator compared to the single year data, the magnitude of PRR and ChiSq significance more than doubled for the 2010 signal of influenza vaccines and fever reactions but—while still above signal thresholds—almost halved for the 2015 signal of influenza vaccines and allergy AEFI.
Sensitivity and subgroup analyses
2010 fever signal.
In 2010 a signal would reasonably have been indicated by at least week 13 (using single year data; likely earlier with cumulative all-years data), which equates to 28 March 2010. At this time point the PRR = 2.12 with ChiSq 5.32 (Fig 5). This was 15 days earlier than the alert raised through clinical observation (12 April) and 26 days before the child influenza program was suspended nationally (23 April) (Fig 6).
Sub-analysis comparing individual influenza vaccine brands with the other brands, for fever AEFI in 2010 shows that the proportional reporting of fever with Fluvax® was significantly higher than the other brands in use that year (Fig 7).
Sub-analysis by age group using single year data brought higher magnitude (PRR 5.74, ChiSq 15.88) and earlier detection (calendar week 12, March 21, 2010) in the 0–4 age group. However, case count threshold was not reached for PRR calculation in 2008 and 2009 and was delayed in the remaining years by between 3–8 weeks compared to analyses across all ages. In the 5 years and older age subgroup signal magnitude was increased (PRR = 3.28, ChiSq 13.22) but was delayed by 2 weeks till calendar week 15 (first week calculable). The increased magnitude was more pronounced using cumulative all previous year data PRR 8.72, ChiSq 24.15, however the 2 week delay remained.
2015 allergy signal.
In 2015 the PRR was consistently >2 from onset of reporting, with ChiSq reaching >4 (8.2) in week 16 (PRR 3.51) which equates to April 19 2015. This was 11 days earlier than the alert raised through clinical observation (1 May) (Fig 8).
Sub-analysis by age group did not facilitate signal detection in 2015. Case count in the 0–4 year age subgroup reached threshold for calculation in only three of the 10 years under investigation (2010, 2012 and 2016) and not at all in the signal event year of 2015. Signal threshold was reached in 2015 for the 5 years and older age subgroup, but one week later and at a lower magnitude (PRR = 2.48) compared to analysis across all ages. Cumulative analysis using all previous year data did not reach signal detection thresholds in either age subgroup.
Sub-analysis comparing individual influenza vaccine brands with the other brands, for allergic AEFI in 2015 shows that the proportional reporting of allergic AEFI with Vaxigrip®, Sanofi Pasteur and Influvac®, Solvay Pharmaceuticals were significantly higher than for Fluvax® and Fluarix®, GlaxoSmithKline that year (Fig 9).
Sub-analysis removing signal year data from the all previous years cumulative denominator when calculating PRR for subsequent years did not elicit any additional signal detections.
Signal hypothesis generation
This analysis intended to document the number of weeks in which PRR and ChiSq reached signal threshold levels of ≥2 and ≥4 respectively, and correlate the proportion that matched with known events (positive detection) as well as to indicate the proportion of weeks when an investigation could have been suggested (hypothesis generation). The data were calculated for each week and for each reporting year as single year data and as all previous years data. Signal thresholds, aside from the two known events, were reached only in the first two weeks (weeks 13 and 14) of seasonal surveillance in 2011.
The analysis for Influenza vaccines and fever AEFI was repeated for all subsequent years, excluding 2010 data, and did not produce any additional reporting weeks reaching PRR ≥2.
Outcomes
- ○. PRR identified the two known signals of influenza vaccines and fever / allergic reaction
- ○. Statistical evidence hypothesised a signal 15 and 11 days earlier than actual alert dates respectively
- ○. Cumulative data period influences signal magnitude depending on AEFI association to vaccine
- ○. PRR sub-analysis by vaccine brand supports hypothesis of brand specific signals
- ○. Using PRR generated one signal hypothesis requiring further investigation for signal validation
Discussion
This retrospective analysis demonstrated PRR as a feasible and applicable signal detection algorithm for two known influenza vaccine signal events. If cumulative PRR analysis had been in use at the time, we would have been able to identify the febrile signal in 2010 and allergy signal in 2015 much earlier than was possible by signal monitoring methods in place at those time points. Earlier signal detection and investigation implicitly enables earlier action if warranted, which in 2010 could have prevented many children receiving an unsafe vaccine. In 2015, it would have helped clarify the risk-benefit information and improve consumer confidence to continue with vaccination.
While the longevity of the SAEFVIC database with over a decade of data benefited cumulative calculations, a single year of data sufficed for effective signal detection. This is important for newly established datasets. It is also an important performance requirement if monitoring new vaccines (e.g Zostavax® Merck vaccines), or new brands or formulations (e.g Garadsil®9 Seqirus) or population subgroups (e.g in pregnancy) as it may not be appropriate to include disparate historical AEFI reporting data in the comparator group.
There was discordant impact on efficiency of signal detection when using cumulative data from all previous years compared to data from a single year. In 2010 with fever AEFI the signal was magnified, but in 2015 with allergy AEFI the signal diminished. This may occur as fever is a common AEFI with many vaccines but allergy-related AEFI occur more frequently with influenza vaccines due to the multiple antigens and the manufacturing process being in eggs with small amounts of egg albumin retained in influenza vaccines [40]. The inclusion of higher numbers of allergy-related reports from previous years in the denominator therefore diminishes the signal.
Analysis by population or categorical subgroups (e.g age, gender or geographic area) has potential to increase sensitivity for signal detection but may also mask signals [38]. Constraints can arise through small case numbers leading to a delay in reaching calculation threshold and consequently signal detection opportunities being equally delayed. However, jurisdictional funding in recent years as well as the July 2019 Pharmaceutical Benefits Advisory Committee recommendation for seasonal influenza vaccination of children aged 6 months to 5 years to be included in the National Immunisation Program [41], is likely to boost numbers in this age group and therefore increase the utility and sensitivity of age subgroup analyses [42].
The population under surveillance is all vaccinees and when operating prospectively it will not be known in which age group a signal may arise in, therefore it is appropriate to analyse across the whole population routinely, as well as by subgroup [38].
This study hypothesises that Vaxigrip® and Influvax® vaccine brands flagged in the 2015 allergy-related signal event, which is in contrast to the findings in the previous investigation, which concluded there was no implication of a specific brand[6]. The previous analysis however, was limited to calculating brand-specific reporting rate per 100,000 doses distributed; the rates of which did not differ significantly between brands. Furthermore, as Influvac® was not a Department of Health funded vaccine in that year, dose distribution data were not available and so Influvac® could not be included in the brand-specific analyses in that study.
The paucity of spurious signals in our study should be interpreted with caution as this study was limited to a narrow scope of influenza vaccines and only two AEFI categories. Other studies indicate the norm is to find a substantial fraction of signals, albeit for which no external supporting evidence can be found, even when a highly inclusive search for such evidence is conducted[43]. Spurious signals are likely to be detected if PRR were applied to all vaccine-AEFI event pairs occurring in the database, which would be extensive. This study used calculation and signal threshold determinants with precedence from earlier studies [12, 38]. Further exploration of these arbitrary detection thresholds may be required as changes in the standard threshold for the count of drug-event combinations can result in a substantial variation in efficiency of the signal detection process. The European Medicines Agency found a threshold of five compared to a threshold of three (as we used) gave a reduction of 25% in false positive signals in return for a loss of 12% in true signals detected early [39].
Strengths for this proof-of-principle study were the decade of data available and presence of two confirmed signal events of differing magnitude contained within. Limitations included the small number of case-event pairs, particularly in non-signal years; testing on only two signals, both involving the same vaccine group (influenza vaccines) and; the absence of documented false signals within the dataset and non-generation of spurious signals meaning an inability to examine the algorithm performance with false-positive signals. The potential for prospective analyses to inform signal detection must also be viewed commensurate with known limitations of passive surveillance systems: namely under-reporting—particularly of reactions perceived as mild—and the time lag from symptom onset to report submission. However, it is worth noting that in this study, simulating reality as data were extracted according to date reported, the safety signal was still generated well ahead of the historical clinical recognition.
To mature signal hypothesis generation, we plan to integrate PRR as a routine prospective signal detection platform with automated calculations updated weekly providing an “alert” table presenting any vaccine-event pairs reaching the pre-defined threshold and statistical significance levels for epidemiological review. This will provide signal hypothesis generation as a first step in comprehensive signal management process. Even though this will be performed on the SAEFVIC Victorian state-level data it has relevance for Australian vaccine safety nationally as SAEFVIC contributes approximately 45 per cent of all AEFI reporting to the national AEFI database [44].
Further studies to incorporate and compare performance characteristics with Bayesian algorithms and multivariate modeling techniques are required [9, 45], however the final choice of SDA should be made in the context of meeting the Innovative Medicines Initiative PROTECT (Pharmacoepidemiological Research on Outcomes of Therapeutics by a European ConsorTium) project recommendation that “Choice of a disproportionality statistic for signal detection should be primarily based on ease of implementation, interpretation and optimisation of resources” [15].
Conclusion
The PRR algorithm is relatively easy to implement and analyse. Utilising a single jurisdiction’s data, PRR identified a loud (febrile) and a quieter (allergy) signal in the SAEFVIC data set. There is no reason to think it would not be as successful prospectively as it has been shown to be retrospectively. We believe the PRR should be routine in SAEFVIC and in any national adverse event surveillance system that evolves.
The implementation of signal detection algorithms in routine Australian vaccine pharmacovigilance has the potential to detect important safety signals much earlier than previously, keeping immunisations in Australia as safe as possible and helping maintain community and provider confidence.
Supporting information
S1 Table. AEFI reports by Influenza vaccine brand and year reported, 2008–2017, SAEFVIC.
https://doi.org/10.1371/journal.pone.0224702.s001
(DOCX)
Acknowledgments
The authors wish to acknowledge the contribution of SAEFVIC immunisation nurses and paediatricians in the management of AEFI reports.
References
- 1.
Stokes B. Ministerial Review into the Public Health Response into the Adverse Events to the Seasonal Influenza Vaccine. In: Government of Western Australia Department of Health. Perth 2010. Available from: www.health.wa.gov.au/publications/documents/Stokes_Report.pdf [Accessed 20 July 2018].
- 2. Clothier HJ, Crawford NW, Russell M, Kelly H, Buttery JP. Evaluation of 'SAEFVIC', A Pharmacovigilance Surveillance Scheme for the Spontaneous Reporting of Adverse Events Following Immunisation in Victoria, Australia. Drug safety. 2017;40(6):483–95. pmid:28342074
- 3. Gold MS, Effler P, Kelly H, Richmond PC, Buttery JP. Febrile convulsions after 2010 seasonal trivalent influenza vaccine: implications for vaccine safety surveillance in Australia. The Medical journal of Australia. 2010;193(9):492–3. pmid:21034379
- 4.
Australian Governement Department of Health and Ageing Therapeutic Goods Administration. Overview of Vaccine Regulation and Safety Monitoring and Investigation into Adverse events Following 2010 Seasonal Influenza Vaccination in Young Children Canberra: 8 October 2010. [cited 20 July 2018] Available from: www.tga.gov.au/sites/default/files/alerts-medicine-seasonal-flu-101008.pdf
- 5. Moulton Emily. Parents of Saba Button who was victim of flu debacle received payout from WA Government. PerthNow 2014. [cited 12 November 2018] Available from: www.perthnow.com.au/news/wa/parents-of-saba-button-who-was-victim-of-flu-vaccine-debacle-receive-payout-from-wa-government-ng-bcbf1a145ec457cc170ef5692f3b691d
- 6. Clothier HJ, Crawford N, Russell MA, Buttery JP. Allergic adverse events following 2015 seasonal influenza vaccine, Victoria, Australia. Euro surveillance. 2017;22(20). pmid:28552101
- 7. Crawford NW, Clothier H, Hodgson K, Selvaraj G, Easton ML, Buttery JP. Active surveillance for adverse events following immunization. Expert review of vaccines. 2014;13(2):265–76. pmid:24350637
- 8. Waldman EA, Luhm KR, Monteiro SA, Freitas FR. Surveillance of adverse effects following vaccination and safety of immunization programs. Revista de saude publica. 2011;45(1):173–84. pmid:21181055
- 9. Harpaz R, DuMouchel W, LePendu P, Bauer-Mehren A, Ryan P, Shah NH. Performance of pharmacovigilance signal-detection algorithms for the FDA adverse event reporting system. Clinical pharmacology and therapeutics. 2013;93(6):539–46. pmid:23571771
- 10. Collet JP, MacDonald N, Cashman N, Pless R. Monitoring signals for vaccine safety: the assessment of individual adverse event reports by an expert advisory committee. Advisory Committee on Causality Assessment. Bull World Health Organ. 2000;78(2):178–85. pmid:10743282
- 11. Almenoff J, Tonning JM, Gould AL, Szarfman A, Hauben M, Ouellet-Hellstrom R, et al. Perspectives on the use of data mining in pharmaco-vigilance. Drug safety. 2005;28(11):981–1007. pmid:16231953
- 12. Evans SJ, Waller PC, Davis S. Use of proportional reporting ratios (PRRs) for signal generation from spontaneous adverse drug reaction reports. Pharmacoepidemiology and drug safety. 2001;10(6):483–6. Epub 2002/02/07. pmid:11828828
- 13.
Australian Government Department of Health and Ageing. Investigation into febrile reactions in young children following 2010 seasonal trivalent influenza vaccination. Canberra: Therapeutic Goods Administration, 2010. [cited 18 November 2018] Available from: www.tga.gov.au/safety/alerts-medicine-seasonal-flu-100702.htm#fullreport
- 14. Armstrong PK, Dowse GK, Effler PV, Carcione D, Blyth CC, Richmond PC, et al. Epidemiological study of severe febrile reactions in young children in Western Australia caused by a 2010 trivalent inactivated influenza vaccine. BMJ open. 2011;1(1):e000016. pmid:22021725
- 15. Wisniewski AF, Bate A, Bousquet C, Brueckner A, Candore G, Juhlin K, et al. Good Signal Detection Practices: Evidence from IMI PROTECT. Drug safety. 2016;39(6):469–90. pmid:26951233
- 16. Szarfman A, Machado SG, O'Neill RT. Use of screening algorithms and computer systems to efficiently signal higher-than-expected combinations of drugs and events in the US FDA's spontaneous reports database. Drug safety.2002;25(6):381–92. pmid:12071774
- 17. Hauben M. A brief primer on automated signal detection. The Annals of pharmacotherapy. 2003;37(7–8):1117–23. pmid:12841826
- 18. McClure DL, Xu S, Weintraub E, Glanz JM. An efficient statistical algorithm for a temporal scan statistic applied to vaccine safety analyses. Vaccine. 2012;30(27):3986–91. pmid:22531555
- 19. Martin D, Menschik D, Bryant-Genevier M, Ball R. Data mining for prospective early detection of safety signals in the Vaccine Adverse Event Reporting System (VAERS): a case study of febrile seizures after a 2010–2011 seasonal influenza virus vaccine. Drug safety.2013;36(7):547–56. pmid:23657824
- 20. Musonda P, Hocine MN, Andrews NJ, Tubert-Bitter P, Farrington CP. Monitoring vaccine safety using case series cumulative sum charts. Vaccine. 2008;26(42):5358–67. pmid:18723063
- 21. Nelson JC, Yu O, Dominguez-Islas CP, Cook AJ, Peterson D, Greene SK, et al. Adapting group sequential methods to observational postlicensure vaccine safety surveillance: results of a pentavalent combination DTaP-IPV-Hib vaccine safety study. Am J Epidemiol. 2013;177(2):131–41. pmid:23292957
- 22. Sakaeda T, Tamon A, Kadoyama K, Okuno Y. Data mining of the public version of the FDA Adverse Event Reporting System. International journal of medical sciences. 2013;10(7):796–803. pmid:23794943
- 23.
Shibata A, Hauben M. Pharmacovigilance, Signal Detection and Signal Intelligence Overview. 14th International Conference on Information Fusion; July 5–8; Chicago, Illinois, USA2011. Available from: http://isif.org/fusion/proceedings/Fusion_2011/data/papers/200.pdf
- 24. Hauben M. Early postmarketing drug safety surveillance: data mining points to consider. The Annals of pharmacotherapy. 2004;38(10):1625–30. pmid:15304626
- 25. Banks D, Woo EJ, Burwen DR, Perucci P, Braun MM, Ball R. Comparing data mining methods on the VAERS database. Pharmacoepidemiology and drug safety. 2005;14(9):601–9. pmid:15954077
- 26. Harpaz R, DuMouchel W, Shah NH, Madigan D, Ryan P, Friedman C. Novel data-mining methodologies for adverse drug event discovery and analysis. Clinical pharmacology and therapeutics. 2012;91(6):1010–21. pmid:22549283
- 27.
Horvath J.C. Review of the management of adverse effects associated with Panvax and Fluvax. 2017. Australian Government Department of health and Ageing. Available from: www.health.gov.au/resources/publications/review-of-the-management-of-adverse-events-associated-with-panvax-and-fluvax-0
- 28.
Australian Government Therpeutic Goods Administration. Response to the West Australian (Stokes) Review into the Handling of AEFIs following 2010 Seasonal Flu Vaccination. Office of Product Review; 2010. Available from: www.tga.gov.au/pdf/alerts-medicine-seasonal-flu-101018.pdf
- 29. Moore N, Thiessard F, Begaud B. The history of disproportionality measures (reporting odds ratio, proportional reporting rates) in spontaneous reporting of adverse drug reactions. Pharmacoepidemiology and drug safety. 2005;14(4):285–6. pmid:15782397
- 30. Rothman KJ, Lanes S, Sacks ST. The reporting odds ratio and its advantages over the proportional reporting ratio. Pharmacoepidemiology and drug safety. 2004;13(8):519–23. pmid:15317031
- 31.
Council for International Organizations of Medical Sciences. Practical aspects of signal detection in pharmacovigilance: Report of CIOMS Working Group VIII. Geneva: CIOMS; 2010.
- 32. Hauben M, Madigan D, Gerrits CM, Walsh L, Van Puijenbroek EP. The role of data mining in pharmacovigilance. Expert opinion on drug safety. 2005;4(5):929–48. pmid:16111454
- 33.
Working Party of Experts. Report on the response to selected recommendations from the ‘Review of the management of adverse events associated with Panvax and Fluvax’ [Unpublished report]. In press 2013.
- 34.
Australian Government Department of Health & Ageing. Review of the management of adverse events associated with Panvax and Fluvax (Horvath Review): Outcome of recommendations [cited 02 August 2018]. Available from: http://www.immunise.health.gov.au/internet/immunise/publishing.nsf/Content/AF9FE9D551D0879BCA257C0600029C4F/$File/Horvath-review-outcomes.pdf.
- 35. Clothier HJ, Crawford NW, Kempe A, Buttery JP. Surveillance of adverse events following immunisation: the model of SAEFVIC, Victoria. Communicable diseases intelligence quarterly report. 2011;35(4):294–8. pmid:22624490
- 36.
Australian Government Department of Health. 2015 seasonal influenza vaccines 2015. [cited 25 October 2018]. Available from: https://www.tga.gov.au/media-release/2015-seasonal-influenza-vaccines.
- 37.
Australian Techinical Advisory Group on Immunisation (ATAGI). Australian Immunisation Handbook. Canberra: Australian Government Department of Health; 2018. Available from: https://immunisationhandbook.health.gov.au/
- 38.
European Medicines Agency. Screening for adverse reactions in EudraVigilance. London, United Kingdom: European Medicines Agency, 2016. [cited 12 October 2017] Available from: http://www.ema.europa.eu/docs/en_GB/document_library/Other/2016/12/WC500218606.pdf
- 39. Slattery J, Alvarez Y, Hidalgo A. Choosing thresholds for statistical signal detection with the proportional reporting ratio. Drug safety. 2013;36(8):687–92. pmid:23754759
- 40. McNeil MM, DeStefano F. Vaccine-associated hypersensitivity. The Journal of allergy and clinical immunology. 2018;141(2):463–72. pmid:29413255
- 41.
Australian Government Department of Health—Pharmaceutical Benefits Scheme. July 2019 PBAC Meeting—Positive recommendations Canberra: Australian Government Department of Health; 2019 [cited 28 August 2019]. Available from: http://www.pbs.gov.au/industry/listing/elements/pbac-meetings/pbac-outcomes/2019-07/positive-recommendations-07-2019.pdf.
- 42. Beard FH, Hendry AJ, Macartney K. Early success with room for improvement: influenza vaccination of young Australian children. The Medical journal of Australia. 2019;210(11):484–6 e1. Epub 2019/05/08. pmid:31063595
- 43. Hochberg AM, Hauben M, Pearson RK, O'Hara DJ, Reisinger SJ, Goldsmith DI, et al. An evaluation of three signal-detection algorithms using a highly inclusive reference event database. Drug safety. 2009;32(6):509–25. Epub 2009/05/23. pmid:19459718
- 44. Dey A, Wang H, Quinn H, Cook J, Macartney K. Surveillance of adverse events following immunisation in Australia, 2015. Communicable diseases intelligence quarterly report. 2017;41(3):E264–E78.
- 45. Arnaud M, Begaud B, Thurin N, Moore N, Pariente A, Salvo F. Methods for safety signal detection in healthcare databases: a literature review. Expert opinion on drug safety. 2017;16(6):721–32. Epub 2017/05/12. pmid:28490262