https://orcid.org/0000-0003-4331-5785
Figures
Abstract
In December 2020, an interim recommendation for the use of Pfizer-BioNTech COVID-19 vaccine in persons aged ≥16 years was made under Food and Drug Administration’s Emergency Use Authorization. In preparation for Biologics License Application approval, we conducted a systematic review and meta-analysis to inform the U.S. Centers for Disease Control and Prevention’s Advisory Committee for Immunization Practice’s (ACIP) decision-making for a standard recommendation. We conducted a rapid systematic review and meta-analysis of Pfizer-BioNTech vaccine effectiveness (VE) against symptomatic COVID-19, hospitalization due to COVID-19, death due to COVID-19, and asymptomatic SARS-CoV-2 infection. We identified studies through August 20, 2021 from an ongoing systematic review conducted by the International Vaccine Access Center and the World Health Organization. We evaluated each study for risk of bias using the Newcastle-Ottawa Scale. Pooled estimates were calculated using meta-analysis. The body of evidence for each outcome was assessed using the Grading of Recommendations, Assessment, Development and Evaluation (GRADE) approach. We identified 80 articles, selected 35 for full-text review, and included 26. The pooled VE of Pfizer-BioNTech COVID-19 vaccine was 92.4% (95% CI: 87.5%–95.3%) against symptomatic COVID-19 with moderate evidence certainty (eight studies), 94.3% (95% CI: 87.9%–97.3%) against hospitalization due to COVID-19 with moderate certainty (eight studies), 96.1% (95% CI: 91.5%–98.2%) against death due to COVID-19 with moderate certainty (four studies), and 89.3% (88.4%–90.1%) against asymptomatic SARS-CoV-2 infection with very low certainty (two studies). The Pfizer-BioNTech COVID-19 vaccine demonstrated high effectiveness in all pre-specified outcomes and extended knowledge of the vaccine’s benefits to outcomes and populations not informed by the RCTs. Use of an existing systematic review facilitated a rapid meta-analysis to inform an ACIP policy decision. This approach can be utilized as additional COVID-19 vaccines are considered for standard recommendations by ACIP.
Citation: Wallace M, Collins JP, Moline H, Plumb ID, Godfrey M, Morgan RL, et al. (2022) Effectiveness of Pfizer-BioNTech COVID-19 vaccine as evidence for policy action: A rapid systematic review and meta-analysis of non-randomized studies. PLoS ONE 17(12): e0278624. https://doi.org/10.1371/journal.pone.0278624
Editor: Zemenu Yohannes Kassa, Hawassa University College of Medicine and Health Sciences, ETHIOPIA
Received: June 14, 2022; Accepted: November 18, 2022; Published: December 6, 2022
This is an open access article, free of all copyright, and may be freely reproduced, distributed, transmitted, modified, built upon, or otherwise used by anyone for any lawful purpose. The work is made available under the Creative Commons CC0 public domain dedication.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Abbreviations: ACIP, Advisory Committee on Immunization Practices; BLA, biologics license application; FDA, Food and Drug Administration; GRADE, Grading of Recommendations, Assessment, Development and Evaluation; IVAC, International Vaccine Access Center; WHO, World Health Organization
Background
In the United States, the Food and Drug Administration (FDA) is the regulatory authority for approval of new vaccines. Following regulatory allowance from the FDA (Emergency Use Authorization or licensure), the Centers for Disease Control and Prevention (CDC) makes public health recommendations for vaccine use under advice from the Advisory Committee for Immunization Practices (ACIP), with interim recommendations following Emergency Use Authorization (EUA) and standard recommendations following licensure. In December 2020, the FDA issued an EUA for the Pfizer-BioNTech COVID-19 vaccine and ACIP issued interim recommendations for the use of the vaccine in persons aged ≥16 years [1, 2]. At the time of the EUA, the available body of evidence of benefits and harms consisted of one phase I randomized controlled trial (RCT) [3] and one large endpoint-driven phase II/III RCT [4], which had met the pre-specified number of events and had a median 2 months of follow-up. In the ensuing months, hundreds of millions of doses of the Pfizer-BioNTech COVID-19 vaccine were administered in the United States and worldwide [5, 6].
On August 23, 2021, the FDA approved the biologics license application (BLA) for the Pfizer-BioNTech COVID-19 vaccine for the prevention of COVID-19 in persons aged ≥16 years [7]; subsequently, the ACIP met to consider a standard recommendation for use. The policy question under consideration was, “Should vaccination with Pfizer-BioNTech COVID-19 vaccine (2-doses, intramuscularly, IM) be recommended for persons 16 years of age and older?”. The ACIP COVID-19 Vaccines Work Group framed the policy question by defining the population, intervention, comparison, and outcomes (PICO) of interest. The Work Group defined the population as persons aged ≥16 years, the intervention as two doses of Pfizer-BioNTech COVID-19 vaccine, and the comparison as no COVID-19 vaccine. The outcomes of interest for potential benefits pre-specified by the work group were symptomatic laboratory-confirmed COVID-19, hospitalization due to COVID-19, death due to COVID-19, and asymptomatic SARS-CoV-2 infection. The pre-specified potential harms were serious adverse events and severe or potentially life threatening reactogenicity. To inform ACIP deliberations, a systematic review of the evidence for benefits and harms for Pfizer-BioNTech COVID-19 vaccine was presented to ACIP on August 30, 2021 [8]. The GRADE approach used by ACIP evaluates certainty in estimates of benefits and harms and incorporates data from both RCT and observational studies [9]. The overall methods and results of GRADE were disseminated with the publication of an ACIP policy note [10] shortly after the policy decision. However, detailed methods and results of the meta-analysis performed to support GRADE have not been published.
The assessment of the RCT evidence was previously described in the policy note [10]. Briefly, the available body of evidence from RCTs to inform the PICO question consisted of one Phase I RCT [3] and one large Phase II/III RCT with six months of follow up [11]. These studies provided evidence for most of the outcomes considered, with the exception of asymptomatic SARS-CoV-2 infection (Table 1) [10]. The initial vaccine roll-out under an EUA allowed for a large body of observational evidence to accrue before initial licensure. To incorporate available observational data in the GRADE assessment in the short timeframe required for the policy decision, we performed a rapid systematic review and meta-analysis to characterize the available Pfizer-BioNTech COVID-19 vaccine effectiveness (VE) data for each of the pre-specified beneficial outcomes (Table 1). Although harms were considered in the GRADE assessment, they were not included in the meta-analysis presented here because they were informed by the RCTs and two large post-authorization safety surveillance systems rather than global published and pre-print studies [3, 4, 11]. To provide additional transparency regarding the data synthesis considered by ACIP and to support future application of these methods, here we present the findings of the rapid review and meta-analysis of vaccine effectiveness studies, which was performed for the outcomes of symptomatic COVID-19, hospitalization due to COVID-19, death due to COVID-19, and asymptomatic SARS-CoV-2 infection.
Methods
Systematic review
In response to the rapidly changing pandemic, we implemented steps to facilitate timely identification and evaluation of the evidence. This systematic review was completed in accordance with the "Preferred Reporting Project for Systematic Reviews and Meta-Analysis (PRISMA) statement," and follows Cochrane rapid reviews methods (S1 Checklist) [12, 13]. Briefly, we identified studies through an ongoing systematic review and screened for eligibility and inclusion in the meta-analysis. We extracted data with a standardized and piloted tool and assessed risk of bias for each study, making final qualitative assessments of the risk of bias. When appropriate, we performed meta-analyses for each outcome and evaluated measures of consistency. We used sensitivity analyses to further explore influence of study characteristics and robustness of pooled VE estimates to inclusion decisions. The certainty of evidence was assessed using GRADE. The systematic review and meta-analysis took place over a three-week period. Article screening began on August 11, 2021 and we continued including available articles through August 20, 2021, with the results presented to ACIP on August 30, 2021.
To expedite the process of identifying relevant literature, we identified observational studies through an ongoing systematic review conducted by the International Vaccine Access Center and the World Health Organization (IVAC/WHO) [14]. Details of search terms and inclusion criteria are available on the IVAC Vaccine Information and Epidemiology Window (VIEW)-hub platform resources page [14]. Given the evolving nature of the COVID-19 pandemic, we included preprints in the available body of evidence for consideration. Thus, articles published in a peer-reviewed journal or posted to a preprint server and captured by the IVAC/WHO systematic review since real-world use of COVID-19 vaccines began in December 2020 through August 20, 2021 were considered for inclusion. In addition, efforts were made to obtain additional relevant data by hand-searching reference lists and consulting with subject matter experts. Following input from the ACIP COVID-19 Vaccines Work Group, we made a priori decisions to further restrict the studies retrieved from the IVAC/WHO systematic review and other sources to the population, intervention, comparison, and outcomes being considered by ACIP. We included studies of the general population and sub-populations (e.g., healthcare workers, persons aged ≥65 years). Inclusion of a given study required assessment of the pre-specified beneficial outcomes starting a minimum of 7 days after the 2nd dose of Pfizer-BioNTech. We included case-control, test-negative, and cohort study designs, and did not discriminate on the basis of SARS-CoV-2 variant or dose interval. We excluded estimates of VE that could not be verified as Pfizer-BioNTech specific (e.g., “mRNA vaccine VE”). Two reviewers cross-validated the IVAC summary tables independently and in duplicate for eligibility. Studies with conflicting reviewer opinion were adjudicated through group discussion.
We extracted data for all relevant outcomes (i.e., symptomatic COVID-19, hospitalization due to COVID-19, death due to COVID-19, and asymptomatic SARS-CoV-2 infection) from the eligible studies. One reviewer performed a full data extraction on each study with a standardized, pilot tested tool; each extraction was reviewed and confirmed by a second reviewer. Information extracted included country, dates, population, dosing interval (standard, defined as consistent with the intervals used in an RCT; or extended, defined as a recommendation for a longer interval than was evaluated in an RCT), circulating variants, study design, publication status, raw numerators and denominators when available, adjusted VE and 95% confidence intervals, and variables used for adjustment. VE was extracted as estimated by the authors using a variety of study designs and analytical approaches based on relative effects, generally defined as 100% x (1-relative risk); relative risk measures included risk ratio, rate ratio, odds ratio, and hazard ratio.
All observational studies were evaluated by two reviewers for study limitations (risk of bias) using the Newcastle-Ottawa Scale (NOS) [15], a 9-point instrument that assesses study limitations related to selection, comparability, and assessment of outcome (cohort) and ascertainment of exposure (case-control or test-negative design). We used the NOS evaluations to consider the entire body of evidence for risk of bias as is assessed in the GRADE process. Studies with scores <7 were considered to have serious study limitations that would have resulted in concerns for risk of bias for the body of evidence. We adjudicated conflicting NOS scores which were qualitatively different (i.e., one score <7 and one score ≥7) through group discussion to determine if a study would be classified as having serious study limitations.
Meta-analysis
We calculated pooled adjusted VE estimates and 95% confidence intervals using random effects (>3 studies) or fixed effects (≤2 studies) meta-analysis (R Version 4.1.1, meta package) [16]. When multiple studies provided estimates based on the same study population (or a subset), we selected the study with the broadest study population or longest duration of follow-up for inclusion in the pooled estimate. For studies that provided multiple or variant-specific estimates, the overall estimate was used when available. If an overall estimate was not available, we used the VE estimates specific to the Delta or Beta variants, as these were considered most likely to impact VE and Delta was the circulating variant at the time of the policy discussion. Studies with serious limitations as assessed by the NOS were excluded a priori from the primary pooled VE assessed used in GRADE. We generated forest plots of VE estimates from all included and excluded studies and forest plots of the pooled VE estimates along with VE estimates from studies included in the pooled estimate.
We performed a series of sensitivity analyses to assess the influence of study characteristics (e.g., special populations vs. general population, preprint vs. peer-reviewed manuscript, standard (3-week) vs. extended dosing interval, study design, study limitations, and circulating variants) on the pooled VE estimates, as indicated by the characteristics of the body of evidence for each outcome. Studies excluded from the primary pooled analysis were included in sensitivity analyses exploring heterogeneity due to study limitations, variant-specific estimates, and publication bias. Heterogeneity was additionally explored using I2 statistic. I2 values of 75% or greater are generally considered to indicate high heterogeneity [17]. We generated forest plots of the pooled VE estimate from the primary meta-analyses along with estimates generated from sensitivity analyses.
GRADE assessment
We performed a GRADE assessment of the quality of the evidence for each outcome. GRADE methods dictate that the body of evidence from observational studies is initially classified as low certainty, is then rated down or, if no reasons for lowering certainty, may be evaluated for factors which would increase certainty [18]. We evaluated the available body of evidence for factors that may decrease the evidence certainty (i.e., risk of bias, indirectness, inconsistency, imprecision, and publication bias), and other considerations such as large or very large effect, dose response gradient, and opposing residual confounding which may increase the evidence certainty.
Results
After screening 80 studies from the IVAC/WHO review, we identified 35 observational studies that met the PICO criteria for the policy question eligible for full-text review (Fig 1). Of these, we excluded one for having no primary data, four for assessing a different intervention (i.e., a different vaccine or a class of vaccines), and four for assessing a different outcome (e.g., any infection instead of symptomatic or asymptomatic infection). This left 26 observational VE studies eligible for inclusion in the meta-analyses [19–45].
Symptomatic, laboratory-confirmed COVID-19
Seventeen studies (ten peer-reviewed and seven preprint) reported VE estimates for symptomatic, laboratory-confirmed COVID-19 (Table 2, S1 Fig) [19–25, 27, 30–35, 37, 39, 43, 45].
Of the 17 studies, five were conducted in Israel [20, 21, 25, 30, 39]. The largest Israeli study, by Haas et al., was a nationwide retrospective cohort study evaluating VE from January 24 through April 4, 2021 [30]; this was the only Israeli study included in the primary meta-analysis. An additional retrospective cohort study reporting VE in the general population was excluded from the meta-analysis because it had an overlapping study population and less comprehensive study period (December 20, 2020 through February 14, 2021) [22, 25]. Three cohort studies conducted in specific populations (two in healthcare workers and one in pregnant women) were excluded from the primary meta-analysis because the substantial population and observation period overlapped with the included study [20, 21, 39].
We included two of three studies conducted in Canada, where COVID-19 vaccines were administered at extended dosing intervals (recommended at up to 4 months between doses) [23, 24, 35, 46]. One was a test-negative design study among healthcare workers in Quebec with an observation period from January 17 through June 5, 2021, which was included in the pooled analysis [23]. The other two studies evaluated the same population in Ontario, both using a test-negative design. [24, 35]. The first, Chung et al., had an observation period from December 14, 2020 through April 19, 2021 and provided an overall VE estimate [24]. The second, Nasreen et al, had an observation period form December 14, 2020 through May 2, 2021 and provided variant-specific VE estimates [35]. Given the availability of an overall VE estimate, we used Chung et al. in the primary meta-analysis, but included the variant specific VE estimates from Nasreen et al. in sensitivity analyses.
Of four studies conducted in the United Kingdom or England, we included two in the primary meta-analysis. The first was a community-based survey among a representative sample aged ≥18 years across the entire United Kingdom reporting variant-specific VE estimates, with an observation period from December 1, 2020 through August 1, 2021. We included the Delta-specific estimate from this study in the pooled analysis [37]. Three studies from England, where the recommended dosing interval was 3–12 weeks, reported on data from the National Health Service [32, 33, 45, 47]. We included one of these studies in the primary meta-analysis; a prospective cohort study reporting an overall VE estimate from December 7, 2020 through June 13, 2021 [45]. Two studies with a test-negative design were excluded from the primary analysis: one because it was a subpopulation of an included study [32], and the second because it reported only variant-specific VE estimates [33].
Three other studies with non-overlapping study settings were included in the meta-analysis. A test-negative design study included in the primary meta-analysis followed persons aged ≥65 years from eight European countries from December 10, 2020 through May 31, 2021 [31]. We included one prospective cohort study among close contacts of persons with COVID-19 in Spain, which had an observation period from January through April 2021 [34], and one retrospective cohort study among healthcare workers in Kuwait had an observation period from December 24, 2020 through June 15, 2021 [11]. One retrospective cohort study among the general population in Qatar was identified as having study limitations regarding selection and comparability and was included only in sensitivity analyses [43].
Once studies with duplicate populations (n = 8) or study limitations (n = 1) were excluded, the remaining eight studies included in the primary analysis had a range of VE estimates from 82%–97.0%. The resulting pooled VE estimate for the effectiveness of the Pfizer-BioNTech COVID-19 vaccine in prevention of symptomatic, laboratory-confirmed COVID-19 from a random effects meta-analysis was 92.4% (95% CI: 87.5%–95.3%) (Fig 2A). The VE estimates from all 17 studies available ranged from 56.1%–97% (Table 2, S1 Fig).
Forest plots showing individual-study and pooled vaccine effectiveness estimates against (A) symptomatic PCR-confirmed COVID-19; (B) hospitalization due to COVID-19; (C) death due to COVID-19; and (D) asymptomatic SARS-CoV-2 infection. Pooled vaccine effectiveness estimates were derived from random effects meta-analyses (A-C) or fixed effects meta-analysis (D). Values for pooled effectiveness estimates and heterogeneity are shown in supplementary tables.
Sensitivity analyses for symptomatic, laboratory-confirmed COVID-19 resulted in pooled VE estimates ranging from 81.2%–93.5% (Fig 3A, S1 Table). Most sensitivity analyses yielded VE estimates similar to the primary pooled VE estimate used for GRADE. However, a meta-analysis of the four studies that provided Delta-specific estimates resulted in a VE that was somewhat lower (VE: 81.2%; 95% CI: 50.2%–92.9%).
Forest plots showing primary pooled vaccine effectiveness estimates and estimates from sensitivity analyses for (A) symptomatic PCR-confirmed COVID-19; (B) hospitalization due to COVID-19; (C) death due to COVID-19; and (D) asymptomatic SARS-CoV-2 infection. Pooled vaccine effectiveness estimates were derived from random effects meta-analyses (A-C) or fixed effects meta-analysis (D). Values for pooled effectiveness estimates and heterogeneity are shown in supplementary tables. 1Three studies provided a Delta-specific estimates [29, 31, 40] and one provided an estimate for the Delta-dominant period [33].
Hospitalization due to COVID-19
We identified 13 observational studies that reported Pfizer-BioNTech VE estimates for hospitalization due to COVID-19 (Table 2, S1 Fig) [21, 25, 26, 28–30, 34–36, 38, 41, 42, 44]. Of these, six were peer-reviewed and seven were pre-print articles.
Five studies were from Israel [21, 25, 29, 30, 41], of which three [21, 25, 30] were previously described. Two additional, general population cohort studies provided VE estimates, one with an observation period from December 20, 2020 through March 20, 2021 and the other from January 18 through April 25, 2021 [29, 41]. As with the outcome of symptomatic COVID-19, we only included Haas et al., from the Israeli studies in the primary meta-analysis for COVID-19 hospitalization as it represented the broadest population and had substantial population overlap with the other studies.
Three studies provided data from the Unites States [36, 38, 44]. Two were retrospective cohort studies from the same large health network; Pawlowski et al., with an observation period from February 15 through April 20, 2021 [36] and Puranik et al., with an observation period from January through July 2021 [38]. Of these, we included Puranik et al., because of the longer duration of follow up and fewer concerns related to selection. Additionally, we included a case-control study among hospitalized adults aged ≥18 years with an observation period from March 11 through May 5, 2021[44].
We included one retrospective cohort study from Denmark, which reported on populations prioritized for vaccination, with an observation period from December 27, 2020 through April 11, 2021 [26]. A retrospective cohort from Italy provided a VE estimate from the general population aged ≥18 years with an observation period from January 2 through May 21, 2021 [28]. We included the VE estimate from the previously described study from Spain [34] and the Beta/Gamma specific VE estimates from a previously described study from Canada [34, 35]. One study from England, with a test-negative design and extended dosing interval provided variant-specific VE estimates from April 12, 2021 through June 4, 2021 [42]. We included the Delta-specific estimate in the meta-analysis.
The eight included studies had Pfizer-BioNTech COVID-19 VE estimates against hospitalization due to COVID-19, which ranged from 84.3%–99% (Table 2, S1 Fig); the pooled primary VE estimate for hospitalization due to COVID-19 was 94.3% (95% CI: 87.9–97.3) (Fig 2B). Five studies (VE estimate range: 87%–94.4%) were not included in the primary meta-analysis because they reported on the same study population as an included study. Sensitivity analyses resulted in pooled VE estimates ranging from 89.4%–95.7% (Fig 3B, S1 Table).
Death due to COVID-19
Six studies (two peer-reviewed, four preprint) reported estimates of VE by Pfizer-BioNTech vaccine against death due to COVID-19 (Table 2, S1 Fig) [26, 28–30, 38, 41]. We included previously described studies from Denmark, Italy, and the United States in the primary meta-analysis for this outcome [26, 28, 38]. Three previously described studies from Israel provided VE estimates, but only the estimate from Haas et al. was included because of population overlap (VE estimate range of excluded studies: 84.0–93.7). The four studies included in the primary meta-analysis for death due to COVID-19 had VE estimates ranging from 94%–100% (Table 2, S1 Fig). The resulting pooled VE estimate from a random effects meta-analysis was 96.1% (95% CI: 91.5%–98.2%) (Fig 2C). Sensitivity analyses resulted in pooled VE estimates ranging from 95.6%–96.8% (Fig 3C, S3 Table).
Asymptomatic SARS-CoV-2 infection
Five previously described studies reported Pfizer-BioNTech VE estimates for asymptomatic SARS-CoV-2 infections: three from Israel, one from Qatar, and one from the United Kingdom (Table 2, S1 Fig) [20, 30, 37, 39, 43].
Among the Israeli studies, one study [39] was considered to have serious study limitations due to selection of controls and comparability of asymptomatic cases and controls. The two other Israeli studies all had population overlap with Haas et al., which was included in the pooled analysis. The study from Qatar was not included in the pooled analysis because of study limitations related to selection of controls, and comparability of asymptomatic cases and controls [43]. The community-based survey from the United Kingdom reported VE estimates by variant dominant period [37]. We included the VE estimate for the Delta-dominant period in the meta-analysis. In total, we excluded three studies from the meta-analysis, two because they reported on the same study population as an included study and one due to study limitations.
A fixed effects meta-analysis using the remaining two studies resulted in a pooled VE estimate of 89.3% (95% CI: 88.4%–90.1%) (Fig 2D). In a sensitivity analysis which included the study considered to have study limitations, the pooled VE estimate was 88.1% (95% CI: 87.2%–89.0%) (Fig 3D).
GRADE
We did not downgrade for risk of bias because we required studies to have an NOS of ≥7 for inclusion in the primary meta-analyses. We did not downgrade for indirectness because our inclusion criteria specified that studies must address the PICO defined policy question for inclusion in the meta-analysis and that the follow-up duration was deemed adequate for all included studies. Regarding inconsistency, I2 values (see S1–S4 Tables) and methodologic differences raised concern for possible heterogeneity. However, estimates from the included studies all indicated a substantial benefit from vaccination, with overlapping confidence intervals, for all outcomes except asymptomatic SARS-CoV-2 infection; this was the only outcome downgraded for serious inconsistency. We did not downgrade for imprecision because the pooled analyses for each outcome consisted of hundreds of thousands of persons and thousands of events, resulting in precise estimates. The pooled VE estimates for symptomatic COVID-19, hospitalization due to COVID-19, and death due to COVID-19 had no concerns for rating down and showed large effect associations, and the body of evidence for each of these outcomes was rated up to moderate certainty. The certainty of evidence for asymptomatic SARS-CoV-2 infection was rated very low evidence due to serious concern for inconsistency (Table 1).
Discussion
In this rapid systematic review and meta-analysis performed to inform policy discussions for ACIP, the Pfizer-BioNTech COVID-19 vaccine demonstrated effectiveness in the prevention of symptomatic, laboratory-confirmed COVID-19 (VE: 92.4%), hospitalization due to COVID-19 (VE: 94.3%), death due to COVID-19 (VE: 96.1%), and asymptomatic SARS-CoV-2 infection (VE: 88.1%) when compared with receipt of no COVID-19 vaccine. These estimates each had a moderate level of evidence certainty based on a GRADE assessment, except for asymptomatic SARS-CoV-2 infection, which had very low evidence certainty. While this review only represents the data available at a single point of time (i.e., the body of evidence available for decision-making at the time of the August 20, 2021 data cutoff prior to the policy vote on August 30, 2021), the estimates were robust to a variety of sensitivity analyses, all of which supported vaccine benefit for each pre-specified outcome.
For the outcomes included in Phase II/III RCT, the pooled VE estimates from the meta-analyses were well aligned with the RCT estimates as shown in Table 1; additionally, the certainty was judged to be the same from RCT and observational evidence for hospitalization and death due to COVID-19 [10]. Furthermore, the observational data included in the GRADE assessment provided additional complementary, sequential and replacement evidence that we did not identify from the RCT data, including observation time in the Delta-dominant period and inclusion of populations excluded from the RCT such as long-term care facility residents and pregnant women [48]. The RCT did not provide data on the outcome of asymptomatic SARS-CoV-2 infection, and the outcomes of hospitalization and death due to COVID-19 had few events during the trial, leading to imprecision based on concerns with fragility in the estimates. The observational studies filled these gaps as they captured many events and provided data for all outcomes [10]. In addition, the observational studies provided ‘real-world’ evidence of the protective effect of vaccines in a variety of populations after routine introduction of the vaccine.
Waning immunity and variants of concern may impact VE over time. The phase II/III RCT showed lower efficacy against symptomatic, laboratory-confirmed COVID-19 ≥4 months post vaccination, however we were unable to calculate VE estimates by the time since vaccination from the available observational data [11]. The phase II/III RCT did not evaluate vaccine efficacy during the Delta-dominant period, and only half of the observational studies included in the primary pooled estimate for symptomatic COVID-19 had any follow up during the Delta-dominant period as identified by the authors. At the time of the systematic review, Omicron had not yet emerged as a variant of concern. Sensitivity analyses of observational studies, which provided Delta-specific VEs or VEs during the Delta-dominant period, showed the Delta-specific estimate for symptomatic COVID-19 was lower than the primary pooled estimate, although Delta variant-specific VE against symptomatic infection was still quite high at 81.2% (95% CI: 50.2–92.9) and had overlapping confidence intervals with the pooled estimate. A single Delta variant-specific VE estimate for hospitalization due to COVID-19 was available, which also showed continued high effectiveness (VE: 96%) [42]. Two studies provided Delta variant-specific VE estimates for asymptomatic SARS-CoV-2 infection with less consistent results (VE range: 36%–74%) [37, 43].
To perform this rapid systematic review and meta-analysis, we built upon existing ongoing efforts in response to the COVID-19 pandemic. The IVAC/WHO systematic review facilitated expeditious identification of relevant articles and the weekly updated summary tables enabled the inclusion of newly identified studies up to the week before the policy discussion. The increased availability of preprint articles aided the inclusion of timely data, which was particularly important in evaluating VE against the Delta variant. We implemented other steps to facilitate a timely systematic review and meta-analysis, including a threshold for the interpretation of the risk of bias, restricting data extraction to only data presented in included studies, and extracting a minimum number of key variables including the adjusted VE measures in each study.
This analysis was subject to several important limitations. The analysis was performed following streamlined methods for the purpose of informing ACIP policy discussions and the analytic decisions made were in that context. No formal protocol was developed and the review was not registered, although a priori decisions regarding inclusion and exclusion criteria and the general approach were discussed among an external ACIP Work Group and documented. The specificity of the policy question led to the exclusion of studies that might have provided additional evidence relevant for some of the outcomes. To provide ACIP with the most recent evidence available, we included preprint studies which had not yet been through the peer-review process; however, we assessed each study for limitations using NOS. Use of the NOS tool allowed for rapid assessment of risk of bias, which was further simplified by using NOS ≥7 as a qualitative cut off for serious study limitations leading to exclusion from the primary meta-analysis; however, use of the tool in this way limits the description of the heterogeneity in the risk of bias among the studies. Other, more exhaustive, risk-of-bias tools were not used due to the rapid timeline required for the review [49]. Because of the inclusion of preprints, we did not assess publication bias. To provide ACIP with the most comprehensive body of evidence, we did not exclude studies based on study design, population subgroup, dosing interval, or COVID-19 variant. Despite the limited exclusions, few studies provided estimates of VE against asymptomatic SARS-CoV-2 infection. Pooling estimates from different vaccine effectiveness studies might be limited because of differences between populations and in the way that vaccines were introduced. However, in this context the relative statistical homogeneity, similar effectiveness estimates for most outcomes, and robustness to sensitivity analyses strengthens the pooled estimates.
The publication of this meta-analysis provides transparency in the process and data that informed the ACIP policy decision to recommend Pfizer-BioNTech COVID-19 vaccine. This review evaluated the totality of the data available to provide VE estimates upon which a policy decision could be made. At the time of the ACIP vote, overwhelming evidence of benefits of the Pfizer COVID-19 vaccine was identified, with consistent results from RCTs and observational studies. This is a rapidly evolving field with new studies published weekly, and similar approaches might be beneficial as new observational evidence accrues to inform future COVID-19 vaccine policy decisions. Using the ongoing IVAC/WHO systematic review as the source of studies facilitated a timely review. This rapid meta-analysis process will be repeated as additional COVID-19 vaccines in use under EUAs are licensed and considered for standard recommendations, keeping the a priori decisions on inclusion and exclusion criteria and general methods consistent across vaccines and policy questions [50].
Supporting information
S1 Table. Sensitivity analysis for VE of the Pfizer-BioNTech COVID-19 vaccine against symptomatic laboratory-confirmed COVID-19.
https://doi.org/10.1371/journal.pone.0278624.s002
(DOCX)
S2 Table. Sensitivity analysis for VE of the Pfizer-BioNTech COVID-19 vaccine against hospitalization due to COVID-19.
https://doi.org/10.1371/journal.pone.0278624.s003
(DOCX)
S3 Table. Sensitivity analysis for VE of the Pfizer-BioNTech COVID-19 vaccine against death due to COVID-19.
https://doi.org/10.1371/journal.pone.0278624.s004
(DOCX)
S4 Table. Sensitivity analysis for VE of the Pfizer-BioNTech COVID-19 vaccine against asymptomatic SARS-CoV-2 infection.
https://doi.org/10.1371/journal.pone.0278624.s005
(DOCX)
S1 Fig.
Forest plots showing primary pooled vaccine effectiveness estimates and estimates from sensitivity analyses for (A) symptomatic PCR-confirmed COVID-19; (B) hospitalization due to COVID-19; (C) death due to COVID-19; and (D) asymptomatic SARS-CoV-2 infection. Pooled vaccine effectiveness estimates were derived from random effects meta-analyses (A-C) or fixed effects meta-analysis (D).
https://doi.org/10.1371/journal.pone.0278624.s006
(TIF)
Acknowledgments
We would like to acknowledge voting members of the Advisory Committee on Immunization Practices and members of the Advisory Committee on Immunization Practices COVID-19 Vaccines Work Group for their contributions to this work.
References
- 1. Oliver SE, Gargano JW, Marin M, Wallace M, Curran KG, Chamberland M, et al. The Advisory Committee on Immunization Practices’ Interim Recommendation for Use of Pfizer-BioNTech COVID-19 Vaccine—United States, December 2020. MMWR Morb Mortal Wkly Rep. 2020;69(50):1922–4. Epub 2020/12/18. pmid:33332292; PubMed Central PMCID: PMC7745957 Journal Editors form for disclosure of potential conflicts of interest. No potential conflicts of interest were disclosed.
- 2.
Food and Drug Administration. Pfizer-BioNTech COVID-19 Vaccine Emergency Use Authorization. Silver Spring, MD: US Department of Health and Human Services, Food and Drug Administration; 2020. https://www.fda.gov/emergency-preparedness-and-response/coronavirus-disease-2019-covid-19/pfizer-biontech-covid-19-vaccine
- 3. Walsh EE, Frenck RW Jr., Falsey AR, Kitchin N, Absalon J, Gurtman A, et al. Safety and Immunogenicity of Two RNA-Based Covid-19 Vaccine Candidates. N Engl J Med. 2020;383(25):2439–50. Epub 2020/10/15. pmid:33053279; PubMed Central PMCID: PMC7583697.
- 4. Polack FP, Thomas SJ, Kitchin N, Absalon J, Gurtman A, Lockhart S, et al. Safety and Efficacy of the BNT162b2 mRNA Covid-19 Vaccine. N Engl J Med. 2020;383(27):2603–15. Epub 2020/12/11. pmid:33301246; PubMed Central PMCID: PMC7745181.
- 5.
Hannah Ritchie EM, Lucas Rodés-Guirao, Cameron Appel, Charlie Giattino, Esteban Ortiz-Ospina, Joe Hasell, et al. Coronavirus (COVID-19) Vaccinations Published online at OurWorldInData.org2020 [cited 2021 October 6]. Available from: https://ourworldindata.org/covid-vaccinations.
- 6.
Centers for Disease Control and Prevention. CDC COVID Data Tracker Atlanta, GA: US Department of Health and Human Services, CDC; 2020 [cited 2021 October 6]. Available from: https://covid.cdc.gov/covid-data-tracker/#vaccination-trends_vacctrends-total-cum.
- 7.
Food and Drug Administration. Comirnaty. Silver Spring, MD: US Department of Health and Human Services, Food and Drug Administration; 2021. https://www.fda.gov/vaccines-blood-biologics/comirnaty
- 8.
Gargano JW. Grading of Recommendations, Assessment, Development, and Evaluation (GRADE): Pfizer-BioNTech COVID-19 Vaccine Atlanta, GA: US Department of Health and Human Services, CDC;; 2021 [cited November 11, 2021]. Available from: https://www.cdc.gov/vaccines/acip/meetings/downloads/slides-2021-08-30/07-COVID-Gargano-508.pdf.
- 9.
Ahmed F. U.S. Advisory Committee on Immunization Practices (ACIP) Handbook for Developing Evidence-based Recommendations. Atlanta, GA: Centers for Disease Control and Prevention (CDC); 2013.
- 10. Dooling K, Gargano JW, Moulia D, Wallace M, Rosenblum HG, Blain AE, et al. Use of Pfizer-BioNTech COVID-19 Vaccine in Persons Aged ≥16 Years: Recommendations of the Advisory Committee on Immunization Practices—United States, September 2021. MMWR Morb Mortal Wkly Rep. 2021;70(38):1344–8. Epub 2021/09/24. pmid:34555007.
- 11. Thomas SJ, Moreira ED, Kitchin N, Absalon J, Gurtman A, Lockhart S, et al. Six Month Safety and Efficacy of the BNT162b2 mRNA COVID-19 Vaccine. medRxiv. 2021:2021.07.28.21261159.
- 12. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, Ioannidis JP, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol. 2009;62(10):e1–34. Epub 2009/07/28. pmid:19631507.
- 13. Garritty C, Gartlehner G, Nussbaumer-Streit B, King VJ, Hamel C, Kamel C, et al. Cochrane Rapid Reviews Methods Group offers evidence-informed guidance to conduct rapid reviews. J Clin Epidemiol. 2021;130:13–22. Epub 2020/10/18. pmid:33068715; PubMed Central PMCID: PMC7557165.
- 14.
International Vaccine Access Center (IVAC) and Johns Hopkins Bloomberg School of Public Health. VIEW-hub [cited 2021 8/20/2021]. Available from: https://view-hub.org/sites/default/files/2021-07/COVID19%20VE%20%26%20Impact%20Lit%20Review%20Methods.pdf.
- 15. Wells GA SB, O’Connell D, Peterson J, Welch V, Losos M, Tugwell P,. The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses 2021. Available from: http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp.
- 16. Balduzzi S, Rucker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health. 2019;22(4):153–60. Epub 2019/09/30. pmid:31563865.
- 17. Higgins JP, Thompson SG, Deeks JJ, Altman DG. Measuring inconsistency in meta-analyses. BMJ. 2003;327(7414):557–60. Epub 2003/09/06. pmid:12958120; PubMed Central PMCID: PMC192859.
- 18. Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, et al. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. J Clin Epidemiol. 2011;64(4):383–94. Epub 2011/01/05. pmid:21195583.
- 19. Alali WQ, Ali LA, AlSeaidan M, Al-Rashidi M. Effectiveness of BNT162b2 and ChAdOx1 vaccines against symptomatic COVID-19 among Healthcare Workers in Kuwait: A retrospective cohort study. medRxiv. 2021:2021.07.25.21261083. pmid:34946418
- 20. Angel Y, Spitzer A, Henig O, Saiag E, Sprecher E, Padova H, et al. Association Between Vaccination With BNT162b2 and Incidence of Symptomatic and Asymptomatic SARS-CoV-2 Infections Among Health Care Workers. JAMA. 2021;325(24):2457–65. Epub 2021/05/07. pmid:33956048; PubMed Central PMCID: PMC8220476.
- 21. Balicer R, Dagan N, Barda N, Biron-Shental T, Makov-Assif M, Key C, et al. Effectiveness of the BNT162b2 mRNA COVID-19 Vaccine in Pregnancy. Research Square.
- 22. Barda N, Dagan N, Balicer RD. BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. Reply. N Engl J Med. 2021;384(20):1970. Epub 2021/04/22. pmid:33882227.
- 23. Carazo S, Talbot D, Boulianne N, Brisson M, Gilca R, Deceuninck G, et al. Single-dose mRNA vaccine effectiveness against SARS-CoV-2 in healthcare workers extending 16 weeks post-vaccination: a test-negative design from Quebec, Canada. medRxiv. 2021:2021.07.19.21260445.
- 24. Chung H, He S, Nasreen S, Sundaram ME, Buchan SA, Wilson SE, et al. Effectiveness of BNT162b2 and mRNA-1273 covid-19 vaccines against symptomatic SARS-CoV-2 infection and severe covid-19 outcomes in Ontario, Canada: test negative design study. BMJ. 2021;374:n1943. Epub 2021/08/22. pmid:34417165
- 25. Dagan N, Barda N, Kepten E, Miron O, Perchik S, Katz MA, et al. BNT162b2 mRNA Covid-19 Vaccine in a Nationwide Mass Vaccination Setting. N Engl J Med. 2021;384(15):1412–23. Epub 2021/02/25. pmid:33626250; PubMed Central PMCID: PMC7944975.
- 26. Emborg H-D, Valentiner-Branth P, Schelde AB, Nielsen KF, Gram MA, Moustsen-Helms IR, et al. Vaccine effectiveness of the BNT162b2 mRNA COVID-19 vaccine against RT-PCR confirmed SARS-CoV-2 infections, hospitalisations and mortality in prioritised risk groups. medRxiv. 2021:2021.05.27.21257583.
- 27. Fabiani M, Ramigni M, Gobbetto V, Mateo-Urdiales A, Pezzotti P, Piovesan C. Effectiveness of the Comirnaty (BNT162b2, BioNTech/Pfizer) vaccine in preventing SARS-CoV-2 infection among healthcare workers, Treviso province, Veneto region, Italy, 27 December 2020 to 24 March 2021. Euro Surveill. 2021;26(17). Epub 2021/05/01. pmid:33928898; PubMed Central PMCID: PMC8086247.
- 28. Flacco ME, Soldato G, Acuti Martellucci C, Carota R, Di Luzio R, Caponetti A, et al. Interim Estimates of COVID-19 Vaccine Effectiveness in a Mass Vaccination Setting: Data from an Italian Province. Vaccines. 2021;9(6):628. pmid:34200538
- 29. Goldberg Y, Mandel M, Woodbridge Y, Fluss R, Novikov I, Yaari R, et al. Protection of previous SARS-CoV-2 infection is similar to that of BNT162b2 vaccine protection: A three-month nationwide experience from Israel. medRxiv. 2021:2021.04.20.21255670.
- 30. Haas EJ, Angulo FJ, McLaughlin JM, Anis E, Singer SR, Khan F, et al. Impact and effectiveness of mRNA BNT162b2 vaccine against SARS-CoV-2 infections and COVID-19 cases, hospitalisations, and deaths following a nationwide vaccination campaign in Israel: an observational study using national surveillance data. Lancet. 2021;397(10287):1819–29. Epub 2021/05/09. pmid:33964222
- 31. Kissling E, Hooiveld M, Sandonis Martín V, Martínez-Baz I, William N, Vilcu A-M, et al. Vaccine effectiveness against symptomatic SARS-CoV-2 infection in adults aged 65 years and older in primary care: I-MOVE-COVID-19 project, Europe, December 2020 to May 2021. Eurosurveillance. 2021;26(29):2100670. pmid:34296676
- 32. Lopez Bernal J, Andrews N, Gower C, Robertson C, Stowe J, Tessier E, et al. Effectiveness of the Pfizer-BioNTech and Oxford-AstraZeneca vaccines on covid-19 related symptoms, hospital admissions, and mortality in older adults in England: test negative case-control study. BMJ. 2021;373:n1088. Epub 2021/05/15. pmid:33985964
- 33. Lopez Bernal J, Andrews N, Gower C, Gallagher E, Simmons R, Thelwall S, et al. Effectiveness of Covid-19 Vaccines against the B.1.617.2 (Delta) Variant. N Engl J Med. 2021;385(7):585–94. Epub 2021/07/22. pmid:34289274; PubMed Central PMCID: PMC8314739.
- 34. Martinez-Baz I, Miqueleiz A, Casado I, Navascues A, Trobajo-Sanmartin C, Burgui C, et al. Effectiveness of COVID-19 vaccines in preventing SARS-CoV-2 infection and hospitalisation, Navarre, Spain, January to April 2021. Euro Surveill. 2021;26(21). Epub 2021/05/29. pmid:34047271; PubMed Central PMCID: PMC8161727.
- 35. Nasreen S, He S, Chung H, Brown KA, Gubbay JB, Buchan SA, et al. Effectiveness of COVID-19 vaccines against variants of concern, Canada. medRxiv. 2021:2021.06.28.21259420.
- 36. Pawlowski C, Lenehan P, Puranik A, Agarwal V, Venkatakrishnan AJ, Niesen MJM, et al. FDA-authorized mRNA COVID-19 vaccines are effective per real-world evidence synthesized across a multi-state health system. Med (N Y). 2021;2(8):979–92 e8. Epub 2021/07/06. pmid:34223401; PubMed Central PMCID: PMC8238652.
- 37. Pouwels KB, Pritchard E, Matthews PC, Stoesser N, Eyre DW, Vihta K-D, et al. Impact of Delta on viral burden and vaccine effectiveness against new SARS-CoV-2 infections in the UK. medRxiv. 2021:2021.08.18.21262237.
- 38. Puranik A, Lenehan PJ, Silvert E, Niesen MJM, Corchado-Garcia J, O’Horo JC, et al. Comparison of two highly-effective mRNA vaccines for COVID-19 during periods of Alpha and Delta variant prevalence. medRxiv. 2021:2021.08.06.21261707. pmid:34401884
- 39. Regev-Yochay G, Amit S, Bergwerk M, Lipsitch M, Leshem E, Kahn R, et al. Decreased infectivity following BNT162b2 vaccination: A prospective cohort study in Israel. Lancet Reg Health Eur. 2021;7:100150. Epub 2021/07/13. pmid:34250518; PubMed Central PMCID: PMC8261633 personal relationships that could have appeared to influence the work reported in this paper.
- 40. Rosenberg ES, Holtgrave DR, Dorabawila V, Conroy M, Greene D, Lutterloh E, et al. New COVID-19 Cases and Hospitalizations Among Adults, by Vaccination Status—New York, May 3-July 25, 2021. MMWR Morb Mortal Wkly Rep. 2021;70(34):1150–5. Epub 2021/08/27. pmid:34437517.
- 41. Saciuk Y, Kertes J, Mandel M, Hemo B, Shamir Stein N, Zohar A. Pfizer-BioNTech Vaccine Effectiveness Against SARS-CoV-2 Infection: Findings From a Large Observational Study in Israel. SSRN. 2021. pmid:34974072
- 42. Stowe J AN, Gower C, Gallagher E UL, Simmons R, Thelwall S TE, Groves N, et al. Effectiveness of COVID-19 Vaccines Against Hospital Admission with the Delta (B.1.617.2) Variant. 2021. https://fpmag.net/wp-content/uploads/2021/06/Effectiveness-of-COVID-19-vaccines-against-hospital-admission-with-the-Delta-B_1_617_2variant.pdf.
- 43. Tang P, Hasan MR, Chemaitelly H, Yassine HM, Benslimane FM, Khatib HAA, et al. BNT162b2 and mRNA-1273 COVID-19 vaccine effectiveness against the Delta (B.1.617.2) variant in Qatar. medRxiv. 2021:2021.08.11.21261885.
- 44. Tenforde MW, Patel MM, Ginde AA, Douin DJ, Talbot HK, Casey JD, et al. Effectiveness of SARS-CoV-2 mRNA Vaccines for Preventing Covid-19 Hospitalizations in the United States. medRxiv. 2021:2021.07.08.21259776. pmid:34268515
- 45. Whitaker H, Tsang R, Byford R, Andrews N, Sherlock J, Pillai P, et al. Pfizer-BioNTech and Oxford AstraZeneca COVID-19 vaccine effectiveness and immune response among individuals in clinical risk groups. Knowledgehub. 2021. https://khub.net/documents/135939561/430986542/RCGP+VE+riskgroups+paper.pdf/a6b54cd9-419d-9b63-e2bf-5dc796f5a91f
- 46.
National Advisory Committee on Immunization (NACI). Extended dose intervals for COVID-19 vaccines to optimize early vaccine rollout and population protection in Canada in the context of limited vaccine supply Public Health Agency of Canada; 2021 [cited 2021 November 11, 2021]. Available from: https://www.canada.ca/content/dam/phac-aspc/documents/services/immunization/national-advisory-committee-on-immunization-naci/naci-summary-extended-dose-interval-covid-19-en.pdf.
- 47.
Joint Committee on Vaccination and Immunisation. Advice on priority groups for COVID-19 vaccination, 30 December 2020 2020 [cited 2021 November 2, 2021]. Available from: https://www.gov.uk/government/publications/priority-groups-for-coronavirus-covid-19-vaccination-advice-from-the-jcvi-30-december-2020/joint-committee-on-vaccination-and-immunisation-advice-on-priority-groups-for-covid-19-vaccination-30-december-2020.
- 48. Schunemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, et al. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Res Synth Methods. 2013;4(1):49–62. Epub 2013/03/01. pmid:26053539.
- 49. Sterne JA, Hernan MA, Reeves BC, Savovic J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ. 2016;355:i4919. Epub 2016/10/14. pmid:27733354; PubMed Central PMCID: PMC5062054
- 50. Wallace M, Moulia D, Blain AE, Ricketts EK, Minhaj FS, Link-Gelles R, et al. The Advisory Committee on Immunization Practices’ Recommendation for Use of Moderna COVID-19 Vaccine in Adults Aged >/ = 18 Years and Considerations for Extended Intervals for Administration of Primary Series Doses of mRNA COVID-19 Vaccines—United States, February 2022. MMWR Morb Mortal Wkly Rep. 2022;71(11):416–21. Epub 20220318. pmid:35298454; PubMed Central PMCID: PMC8942305.