Introduction

The COVID-19 pandemic has placed a significant toll on health systems and economies. With the development and roll-out of COVID-19 vaccines, policymakers in low- and middle-income countries (LMICs) now have an additional tool to control the pandemic, with the potential to ease lockdowns and other non-pharmaceutical interventions. Yet there is increasing evidence to suggest that vaccines are not a magic bullet, and policymakers will have to identify how to best use vaccines as part of a comprehensive set of interventions [1]. In the immediate term, vaccination programme constraints, both in terms of vaccine supply as well as the capacity of health programmes to deliver vaccine at an unprecedented scale, mean that policymakers must identify how best to target vaccines for greatest impact. In the longer-term, financial sustainability is likely to become an ever more pressing issue. Policymakers have been able to allocate emergency funding to finance COVID-19 prevention and control measures, and many financial institutions have unlocked access to grants and concessional loans to tackle the pandemic [2]. However, as more data become available on vaccine duration of protection, protection against transmission, and protection against COVID-19 variants, policymakers will have to decide which vaccination strategies are sustainable and most appropriate to implement in their context [3]. Already there are stark differences in COVID-19 vaccination coverage targets between countries, ranging from those aiming to vaccinate 30% of the population to those aiming for full population coverage [4].

To inform evidence-based policies on the rational use of COVID-19 vaccines, LMICs require real-world data on the effectiveness of vaccines in their context. Efficacy data from clinical trials are important for regulatory authorities to identify if a vaccine works and if it is safe. However, there are a number of limitations in using efficacy data for policy. Firstly, clinical trials use strict inclusion and exclusion criteria, which are not necessarily representative of all eligible populations for vaccination [5–7]. For COVID-19, a number of vaccines have been recommended for use with limited data on effectiveness in the elderly, pregnant women, and populations with comorbidities, despite these being priority target groups in many countries [8–11]. Second, the setting of clinical trials may not reflect local epidemiology. COVID-19 vaccine clinical trials have been conducted in settings with different circulating strains, diverse underlying population health, varying transmission dynamics and non-pharmaceutical interventions (NPIs), and measuring different outcomes [12]. Finally, due to their nature, efficacy studies are unable to address programmatic issues around health service utilization or off-label use [5]. For COVID-19 vaccines, this includes issues such as timely receipt of the second dose, modified vaccine schedules to address supply shortages or to align timing across vaccine products, vaccine acceptance and hesitancy (especially among specific population groups), interchangeability for mixed product schedules, cold chain excursions and other logistics issues, among others [13].

Real-world effectiveness studies are important for informing policy decisions, as an estimate of the context-specific performance of vaccines [13–15]. The results from real-world effectiveness studies not only monitor impact, but also give country-specific inputs for modelling future strategies for vaccination and relaxation of NPIs, as well as justifying budget allocation into, or away from, the COVID-19 vaccination programme. Due to the nature of real-world effectiveness studies, they can be subject to selection bias, confounding factors, and missing data, therefore requiring careful study design [5, 16, 17]. Important considerations for observational studies include sample size; methods to minimise selection bias; accurate measurement of exposures and outcomes; planning for, managing, and reporting on potential confounders and missing data; and planning appropriate analysis [16, 17].

The World Health Organization (WHO) has published an interim guidance for conducting vaccine effectiveness studies in LMICs, and is maintaining a landscape of observational study designs for COVID-19 vaccination effectiveness [13, 18]. Whilst many studies have synthesised COVID-19 vaccine effectiveness estimates from observational studies [19–24], to our knowledge, there is no systematic review of published real-world effectiveness study designs for COVID-19 vaccination, to support LMICs to understand which study designs are most feasible to implement in their settings, and the advantages and drawbacks of different approaches. This review was commissioned by the Thai government to summarise methodological approaches being used to study real-world COVID-19 vaccine effectiveness, to assess the quality of published literature, and to consider which best-practice approaches are most suitable for implementation in Thailand and other LMICs.

Methods

Results

We identified 5,933 articles through the database search. No additional articles were identified from searching reference lists. After removal of duplicates (2,606) and exclusion of studies based on screening the abstract (3,249) or the full text (42), 36 studies were identified. We included an additional 6 studies identified during expert consultation, resulting in 42 papers for inclusion (Fig 1). Of the 42 studies excluded during full text screening, 31 reported on an excluded outcome (not effectiveness) and 11 were an excluded study type (randomised control trial or modelling study). All studies were in English, except one study in Spanish.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Study profile.

https://doi.org/10.1371/journal.pone.0261930.g001

All 42 studies identified were published in 2021 and all but one study [28] were conducted in high-income countries (HICs) (Table 1). No studies were identified from Africa and only one from Asia [28]. Presence of circulating variants were reported in 12 (29%) studies [11, 29–39]. Most studies assessed effectiveness of mRNA vaccines (33 studies), followed by an mRNA and a viral vector vaccine (7 studies), and 1 study each for viral vector and inactivated vaccine. Ethical approval was required in 27 studies (64%), with 13 studies (31%) not reporting on ethical approval. Many studies (18, 43%) did not report on funding source; for the other studies, 11 (26%) were publicly funded, 2 (5%) funded through public and private funds, 3 (7%) through not-for-profit private funding, and 8 (19%) did not receive funding.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. General characteristics of articles on real-world effectiveness of COVID-19 vaccines.

https://doi.org/10.1371/journal.pone.0261930.t001

Table 2 summarises study characteristics. Most studies (32 of 42, 76%) reported on vaccine effectiveness against either COVID-19 infection, hospitalisation, or death, whereas 3 studies reported 2 outcomes (hospitalisation and infection [37, 66], hospitalisation and death [51]) and 7 studies reported on all 3 outcomes [31, 33, 35, 42, 54, 58, 59]. Of the 37 studies measuring vaccine effectiveness against infection, 31 are cohort studies, 4 test-negative design case control studies, and 2 screening method (Fig 2). The most common study type is retrospective cohort study, (22 studies), often employing immunisation registries and medical databases. Only five studies considered asymptomatic infection among patients under investigation, frontline workers and randomly selected individuals in the community [11, 37, 39, 61, 62]. Most cohort studies were conducted among healthcare workers undergoing routine RT-PCR testing as part of the hospital surveillance system. Sample size ranged from 189 to 10,187,720 (mean 443,697; median 6,904). For vaccine effectiveness against hospitalisation and/or death, we identified 12 cohort and 2 test negative design case control studies. Contrary to infection studies, none had healthcare workers as the population. All studies in the general population used national level surveillance data. Sample size ranged from 189 to 10,187,720 (mean 1,890,171; median 338,145). The test negative designs had small sample sizes compared to cohort studies.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Study design by outcome for COVID-19 vaccine effectiveness studies meeting inclusion criteria.

https://doi.org/10.1371/journal.pone.0261930.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of COVID-19 vaccine real-world effectiveness studies meeting inclusion criteria.

https://doi.org/10.1371/journal.pone.0261930.t002

Table 3 summarises methodology employed across included studies. Most studies assessed vaccination status by registry (31), with 2 studies using self- report [9, 41], 3 using a mixture of registry and self-report [35, 44, 60], and 6 studies not reporting on methods to ascertain vaccination status [28, 32, 47, 61, 64, 65]. For confirmation of COVID-19 infection, 39 studies confirmed diagnosis with reverse transcription polymerase chain reaction (RT-PCR); 2 studies used RT-PCR as the main method of confirming diagnosis, but either allowed rapid antigen test for symptomatic cases [37] or if RT-PCR was not available [54]; and 1 study did not mention method of confirmation of COVID-19 [28]. Of the studies reporting methods to reduce misclassification error, most restricted analysis to samples collected within a certain number of days from symptom onset, ranging up to 7 days before symptom onset and 7–14 days after symptom onset [9, 10, 33, 35, 37, 46, 50]. Other studies reported reducing misclassification error by restricting analysis to symptomatic cases [9, 42, 46], censoring the date of unreliable vaccination dates [11], and conducting sensitivity analysis removing days for possible misclassification [60]. Although not reported as a method to reduce misclassification error, an additional 12 studies only included symptomatic cases [10, 32, 34, 35, 39, 47, 50, 53, 54, 58]. There was considerable difference across studies in terms of when outcomes were assessed in vaccinated individuals: 10 studies only included outcomes more than 14 days after vaccination [31–33, 37, 39, 44, 49, 54, 57, 60]; 10 studies more than 7 days after vaccination [10, 34, 35, 42, 46, 47, 53, 55, 63, 66]; 9 studies included outcomes more than 14 days after vaccination for one of the two vaccine doses, and more than 7 days after the other vaccine dose [9, 11, 30, 38, 45, 50, 51, 56, 62]; 2 studies included outcomes either 14 days or 7 days after vaccination depending on vaccine type [48, 59]; 7 studies included outcomes any time after vaccination, but stratified outcomes by number of days after vaccination [36, 40, 41, 43, 52, 58, 64]; 2 studies included outcomes any time after vaccination [29, 65]; and 2 studies did not report on time between vaccination and outcome inclusion [28, 61]. 3 studies conducted sensitivity or sub-group analysis by days after vaccination [46, 49, 65].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Methodology of included studies.

https://doi.org/10.1371/journal.pone.0261930.t003

For the quality assessment using RELEVANT, 9 of the 42 studies (of which all were cohort studies) met less than half of the criteria [28, 34, 43, 47, 48, 53, 61, 63, 64]. Only 10 of the 43 studies reported registration or publication of the study protocol and 17 reported on potential conflicts of interest (Fig 3). Regarding study methods, there were a number of limitations across studies. Firstly, due to the short time since vaccine roll-out, follow-up time for all studies was very short (mean 6.3 weeks for studies with infection outcomes, 9.7 weeks for hospitalisation or death outcomes). Secondly, only 10 studies reported calculating a sample size a priori (Fig 3). Although studies with large national datasets do not need to calculate a minimum sample size, 3 out of 4 (75%) of the test negative case control designs with fewer than 5,000 participants did not report calculating a minimum sample size [9, 10, 41], and this was also the case for 6 out of 10 of the cohort studies with fewer than 5,000 participants [30, 47, 55, 57, 64, 65]. Thirdly, most studies did not clearly delineate inclusion/exclusion of study participants as a flowchart, although all studies were judged to be in a relevant population and setting. For the test-negative design case control studies, 2 studies were conducted in older adults [33, 41], whilst 2 studies were conducted in health workers ([9, 10]. However, 1 test-negative design case control study was in the general population [32], which may be subject to collider bias. Fourthly, due to the observational study design, selection bias and confounding effects were inevitable limitations. However, 22 studies did not report on assessment and mitigation of potential confounders (Fig 3). The most commonly reported confounders were age [9–11, 29–33, 37, 39–42, 44–46, 50, 51, 54–57, 59–62, 66], sex [9, 10, 29–33, 37, 39–42, 44–46, 50, 51, 54–57, 60–62, 66], socio-demographic factors (ethnicity/religion) [11, 33, 39, 41, 44, 50–52, 55, 60, 61, 66], geographical location [10, 11, 30, 33, 39, 41, 44, 51, 52, 54, 57, 62], chronic disease and/or comorbidities [9, 11, 31, 32, 37, 39, 40, 50, 51, 54, 60, 66], time [10, 33, 36, 37, 40–42, 50, 52, 57], occupation [10, 11, 39, 44, 45, 55, 56, 60], and socio-economic status [33, 39, 40, 54, 57, 66]. Methods reported to manage confounders include adjusted logistic regression model [10, 11, 29, 30, 38, 45, 57, 60], stratified analysis [42, 54], matching cases and controls [51], and excluding population groups with high variability in the probability of vaccination or outcome [51]. 4 studies reported adjusting for or conducting sensitivity analysis by different exposure or infection rates [40, 49, 55, 62]. No study in our review measured adherence to NPIs and none of the test-negative design studies measured respiratory viral infection, which could bias likelihood of individuals seeking COVID-19 tests. Previous SARS-CoV-2 infection was not measured (or not reported) in the majority of studies, participants with prior infection were excluded in 16 studies, and 2 studies included prior infection in sensitivity analysis [10, 33]. Finally, only 14 of 26 studies reported on the extent of missing data (Fig 3). Studies reported dealing with missing data by creating a separate group for individuals with missing data [40], not including missing variables in the analysis [50, 60], or by mean imputation [32].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Quality assessment of included studies using the Real Life Evidence AssessmeNt Tool (RELEVANT).

https://doi.org/10.1371/journal.pone.0261930.g003

Discussion

To our knowledge, this is the first systematic review of methodologies for COVID-19 vaccine effectiveness studies. Given the scale of COVID-19 vaccine roll-out thus far, our review identified relatively few studies assessing real-world vaccine effectiveness. All studies identified are from HICs, often utilising national databases (which may not exist or may be of poorer quality in LMICs), and the great majority assessed mRNA vaccines, which are more prevalent in HICs but only represent a third of the vaccines with WHO Emergency Use Listing (EUL) [67] and one-fifth of COVAX secured supply from legally binding agreements [68]. Whilst the WHO landscape of observational studies has identified pre-prints and registered studies being conducted in six middle income countries (Argentina, Brazil, India, Indonesia, Tunisia, Turkey) [18], between our review and the WHO landscape document there are few real-world effectiveness studies for vaccines that have received WHO EUL and no study in low-income countries. These findings underscore the importance of advocating for real-world effectiveness studies on all approved COVID-19 vaccines and across diverse LMIC settings.

Our review has highlighted several important components to consider at the outset of designing a real-world effectiveness study of COVID-19 vaccines, including the appropriate study design, study population, outcome, and time for follow-up. The most common study design identified in our review was a cohort approach, which may have been facilitated by the presence of large, reliable, and inter-linked databases in study countries. Test negative design case control studies were the second most common study design, but we did not identify any case-control studies in this review. We hypothesise that this finding may be because of the challenges in enrolling an unbiased comparison group: the low number of case-control registered studies and pre-prints suggests that we did not select against case-control studies by restricting our search to peer-reviewed articles [18].

In studies assessing symptomatic or asymptomatic infection as an outcome, healthcare workers were the most common study population. In many studies, healthcare workers were an opportune population due to routine symptomatic or RT-PCR screening activities undertaken within the health system. Conversely, we identified no studies using healthcare workers as the study population for the outcomes hospitalisation and death, which we hypothesise as being due to the low number of severe outcomes in this group [69]. Instead, studies either selected populations at high risk of disease (such as the elderly) or utilised large national databases to assess outcomes in the general population. If large-scale studies are not feasible, or rely on poor-quality databases, LMICs may find that test-negative designs are most feasible to implement, as recommended by the WHO interim guidance [13]. Regarding study population and outcome, we suggest that health workers may be the most appropriate population for studies measuring effectiveness against infection, whereas studies on hospitalisation/death may best focus on elderly populations or other high risk groups.

Given the short timeline since COVID-19 vaccine introduction, the duration of all studies was less than five months. As would be expected, studies looking at hospitalisation and death tended to have longer duration than those assessing infection. However, the short follow-up time may have underestimated vaccine effectiveness against severe outcomes, and means that studies were not able to consider duration of protection, which will be important in informing strategies for delivering booster doses among different populations. Studies of longer duration may also allow assessment of changing vaccine effectiveness with the emergence of new VOCs. Despite widespread concern on protection of COVID-19 vaccines against VOCs, many studies did not assess prevalence of variants and none reported on the delta strain. The WHO landscape of observational studies for vaccine effectiveness suggests that this is likely to remain a significant gap in the literature for future research to consider [18].

Our review highlights several gaps that merit further study, alongside opportunities to strengthen the quality of real-world vaccine effectiveness studies. Firstly, we identified a need for studies in LMICs, especially in Africa and Asia, as well as effectiveness studies with a longer duration and covering all vaccines with WHO EUL. Without information on vaccine effectiveness for all licensed products, governments may face diminishing public confidence towards the vaccines in use in their country. Second, most studies did not calculate (or report) the sample size a priori. Whilst this may be less relevant for retrospective cohort studies based on national databases, which often utilise thousands or millions of records, it is an important consideration for prospective study designs or smaller scale retrospective cohort studies. Since many LMICs are unlikely to be able to replicate the large-scale studies from HICs, calculating minimum sample size will be very important, and should account for differences in access to healthcare services and health seeking behaviour in LMICs, as compared with HICs. Third, we identified weaknesses across studies in identifying and mitigating against potential confounders, and in reporting on missing data. Missing data are likely to be a greater issue in LMICs and differences in healthcare utilisation are likely to be more pronounced than in many HICs, requiring a well-considered plan for identifying and dealing with confounders and missing data. In particular, we note that many studies either did not measure for previous SARS-CoV-2 infection or used this as an exclusion criterion. If the infrastructure exists, we recommend testing for previous infection and conducting sensitivity analysis including this group, to avoid selecting the sample based on exposure risk. Finally, most studies failed to report on the presence of VOCs or on conflict of interest, including funding source. The former is important to respond to changes in vaccine effectiveness with new variants, and the latter is important for credibility of studies for policymaking. Accordingly, we recommend a number of additions to the WHO interim guidance on evaluation of COVID-19 vaccine effectiveness. The document would benefit from further guidance on setting an appropriate time horizon for studies, alongside guidance on designing studies that can be conducted with limited resources. We also propose the inclusion of practical guidance on identifying important confounders for a given setting and management of missing data. Finally, we suggest the inclusion of managing and reporting conflict of interest, as a fundamental part of study design.

There are several limitations to our review. We conducted the review only seven months after the first COVID-19 vaccines were licensed, limiting the number of studies and timeframe, as well as skewing our search results towards HICs, which were the first to introduce COVID-19 vaccination. Restricting our search to peer-reviewed articles further limited the number of results and favoured earlier studies in HICs with limited outcomes based on available data. Because of these limitations, our review was unable to objectively compare approaches that may be more appropriate to LMIC settings. Furthermore, because of an urgent request from the Thai government, we employed rapid review methodology. Consultation with experts identified six additional papers that were not captured by our search terms, and there may be other studies which we missed. However, because the focus of our review is methodology of studies and not an estimate of vaccine effectiveness, we believe that this is acceptable. Particularly for the quality assessment of studies, we had to make assumptions based on reporting in the article, whereas contacting study authors for clarifications may have yielded further information to enhance our analysis.

Despite the importance of real-world effectiveness studies for informing national COVID-19 prevention and control policies in LMICs, existing studies tend to focus on settings, available vaccines, and VOCs specific to a handful of HICs. Although WHO recommends against conducting effectiveness studies in each country [13], in light of the heterogeneity between studies, we argue that there is benefit to each country designing and conducting effectiveness studies, subject to available resources. Considerable funding has been made available from the public sector for COVID-19 vaccine development and deployment. We therefore argue that it is imperative for the public sector to continue funding to the end of the product development continuum and finance studies on effectiveness and impact, not just domestically but across countries, given the global nature of the COVID-19 pandemic.

In summary, our review highlights the importance of local vaccine effectiveness data, and in providing further guidance on important confounders and methods for managing missing data. Most vaccine effectiveness studies to date have been conducted in HICs with access to reliable and interlinked databases for COVID-19 vaccination, diagnosis and treatment. Such databases often do not exist in LMICs, meaning that countries will be employing prospective study designs, requiring a priori calculation of sample size and a clear plan to manage and report on confounders and missing data. We highlight the limited experience conducting vaccine effectiveness in LMICs, but emphasise the importance of such studies for policymakers in LMICs to develop and monitor vaccination policies, as well as to enhance public confidence in vaccination. We call on the global community to support LMICs to lead and implement COVID-19 vaccine effectiveness studies in their settings, as a priority research area moving forward.

Supporting information

Acknowledgments

The authors would like to acknowledge Vice Public Health Minister Sopon Mekthon who commissioned this study. The Health Intervention and Technology Assessment Program (HITAP) is supported by the International Decision Support Initiative (iDSI) to provide technical assistance on health intervention and technology assessment to governments in low- and middle-income countries. iDSI is funded by the Bill & Melinda Gates Foundation, the UK’s Department for International Development, and the Rockefeller Foundation. HITAP is also supported by the Access and Delivery Partnership, which is hosted by the United Nations Development Programme and funded by the Government of Japan.