1 Introduction

The COVID-19 pandemic has caused the deaths of hundreds of thousands. Its economic fallout has upended the lives of billions, and caused trillions of dollars in financial losses. Life may not return to normal until a vaccine is found [1]. Despite the many candidates undergoing testing, an approved vaccine is not expected until 2021, even with substantially compressed development timelines [2], smooth proceeding of clinical trials, and not accounting for possible failures [3]. It is possible—though considered highly unlikely at the present time—that, like many non-influenza epidemics, the crisis may be over before a successful vaccine is developed [4].

Unlike typical therapeutics that are administered to sick patients, vaccines are intended for the healthy. Therefore, confirming the safety and effectiveness of a vaccine is of critical importance [5]. The two primary methods for demonstrating vaccine safety and efficacy are through either a vaccine efficacy randomized clinical trial (RCT) or a vaccine immunogenicity RCT. In the former, large numbers of recruited healthy volunteers are randomly selected to receive either the vaccine or a placebo/active control and then monitored for a period of time. At the end of the surveillance period, the difference in the proportion of infections between the treatment and control arms is computed to demonstrate the ability of the vaccine to prevent infection or disease. A phase 3 vaccine efficacy RCT typically takes five to ten years to complete [6].

In a vaccine immunogenicity RCT, the primary endpoint is an immunity measurement or surrogate marker which is known to correlate with protection against infection or a disease. This type of trial involves a smaller number of volunteers and requires a shorter follow-up period, and as a result, is quicker and less costly [7]. Given that SARS-CoV-2 is a novel pathogen for which we do not yet know how to determine whether a subject is protected, vaccine efficacy must be confirmed using the longer and more costly vaccine efficacy RCT. While there exists the possibility of an expedited (conditional) licensure based on immunogenicity results with post-approval commitments, we find it unlikely to occur given the latest information. The U.S. Food and Drug Administration (FDA) has also issued a guidance stating that “the goal of development programs should be to pursue traditional approval via direct evidence of vaccine efficacy” [8].

A human challenge trial (HCT), in which volunteers are randomized into either the vaccine or placebo arm and then infected deliberately with live virus in a controlled setting, has been proposed as an alternative to accelerate the vaccine development process. Upon challenge, HCTs can quickly demonstrate safety and efficacy of candidate vaccines in preventing infection or disease. Depending on sample size, HCTs could also help to establish functional immune correlates of protection to inform the design of future vaccines. Since an HCT allows comparison of immune responses in vaccinated and unvaccinated individuals, precise measurements of post-vaccination viral loads, characterization of immune responses (innate, adaptive, cell-mediated) and antibody titers, and close monitoring and care of patients, it can help establish the correlates of protection and prove vaccine efficacy concurrently. Moreover, a properly designed HCT can determine transmission risk of the infected in a controlled setting with minimal exposure to investigators and the public. While concerns have been raised regarding the ethics and morality of HCTs, it is generally accepted that HCTs are ethically permissible when the benefits to society outweigh acknowledged risks [9], and they have been deemed acceptable for developing vaccines for multiple infectious diseases such as influenza [10], malaria [11], typhoid [12], cholera [13], and dengue fever [14]. To the best of our knowledge, there have been no published studies of a quantitative analysis of the potential societal value of a COVID-19 HCT.

In this paper, we compare the costs and benefits—as measured by the number of deaths and infections avoided—of confirming the safety and efficacy of a COVID-19 vaccine using four clinical trial designs: a traditional vaccine efficacy RCT, a vaccine efficacy RCT with an optimized surveillance period that maximizes the benefits of the trial (ORCT), an adaptive vaccine efficacy RCT (ARCT), and an HCT. Although our framework applies more broadly to any vaccine candidate for any infectious disease, we calibrate our simulations using a set of estimated epidemiological models for the SARS-CoV-2 virus (one for each of the 50 states and Washington, D.C.) to determine attack rates and cumulative numbers of infections and deaths in the U.S under various scenarios.

A summary of our simulation framework is displayed in Fig 1. We first estimate baseline models from data and make assumptions for the evolution of the epidemic in order to predict the attack rates over the course of the clinical trials. We then combine the attack rates with the assumptions for the vaccine trial design to simulate the outcomes for the clinical trials. Conditioned on the vaccine being approved, we make assumptions on the vaccination schedule and simulate the path of the epidemic in order to compute the net infections and deaths prevented.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Simulation framework.

For each Monte Carlo simulation path, we simulate patient-level infections data based on input trial design assumptions and attack rates from the population epidemiological model (for an RCT, ORCT, and ARCT). At the end of the trial (or, at each interim analysis for an ARCT), we determine if the vaccine candidate is approved under superiority or superiority-by-margin testing. Finally, we compute the expected net value of the trial design over 100,000 simulation paths.

https://doi.org/10.1371/journal.pone.0244418.g001

Assuming that a clinical trial testing a vaccine with a true efficacy of 50% and using superiority tests starts on August 1, 2020, we estimate the date of licensure of the hypothetical vaccine candidate to be some time in November 2021 with a traditional RCT (476 days), between June and August 2021 with an ORCT (326 to 380 days), between April and June 2021 with an ARCT (246 to 306 days), and between March and June 2021 with an HCT (221 to 311 days). For specificity, we report estimated times to licensure using calendar dates and provide the corresponding number of days in parentheses. However, our simulations do depend on calendar dates in one respect: the epidemiological model used to estimate the attack rates depends on current data. Therefore, the estimates reported in this paper are all based on extrapolated conditions as of August 1, 2020, and may need to be revised for other start dates.

The ARCT provides the greatest expected net benefit among the three RCT designs in almost all scenarios. The utility of an HCT versus the RCTs, however, depends critically on the HCT set-up time and the course of the epidemic. The benefits of HCTs are greatest when trials are initiated as early in an epidemic as possible, and/or if the rate of infection is relatively low. Assuming a 30-day set-up time, a vaccine efficacy of 50%, a behavioral epidemiological model, and a population vaccination schedule of 10M doses per day, an HCT can reduce the time to licensure by one month, thus preventing approximately 1.1M incremental infections and 8,000 incremental deaths compared to the best performing alternative clinical trial design, the ARCT. We observe similar results when superiority-by-margin tests are used instead.

We review the designs and assumptions for the four vaccine trials considered in Section 2 and explain our cost/benefit calculations in Section 4. We present the epidemiological model used in our forecasts in Section 2 and report our simulation results in Section 4. We discuss our findings and some broader issues of COVID-19 clinical trials in Section 5 and conclude in Section 6.

2 Vaccine trial design

We begin by describing the assumptions and calibrations used in each of the four vaccine trial designs we considered in our simulations.

3 Epidemiological model

To estimate the attack rate encountered by subjects in a given clinical trial—a key component for our cost/benefit calculations—we require information about the spread of the COVID-19 epidemic in the U.S. We use the Susceptible-Infected-Resolving-Dead-ReCovered with social distancing (SIRDC-SD) model proposed by Fernandez-Villaverde and Jones [28], chosen because it is able to fit both the cumulative and daily number of deaths in all the states well despite being a simple model, to establish a baseline for the epidemic. The details of the model are described in Section A5 in the S1 Appendix.

We estimate the model for each of the 50 states in the U.S. and Washington, D.C., using the time series of deaths in the U.S. obtained from the John Hopkins Center for Systems Science and Engineering (CSSE) COVID-19 repository [29, 30]. Our data were downloaded on June 16, 2020. We do not scale the number of deaths but continue to perform a centered moving average smoothing on the daily number of deaths, as described in Fernandez-Villaverde and Jones [28]. Our estimation method is detailed in Section A6 in the S1 Appendix and the estimated parameters are reported in Table A.3 in S1 Appendix.

The estimated models show how the epidemic has played out thus far but we will need to predict how it will evolve in the future after the lockdowns are relaxed and/or vaccines are developed. To do so, we extend the SIRDC-SD model to take into account semi-effective vaccination. The new model, which we shall name Susceptible-Infected-Resolving-Dead-ReCovered-Vaccinated with social distancing (SIRDCV), is explained in Section A8 in the S1 Appendix.

4 Results

Given the parameters for each trial design and an epidemiological model, we simulate the outcome of hypothetical clinical trials for all four designs and measure their incremental differences.

4.2 Simulation results

We compute the expected net value of different trial designs using Monte Carlo simulations and asymptotic distributions of the efficacy test statistics (see Section A1 in the S1 Appendix). Fig 1 illustrates the inputs, computations, and outputs of our simulation framework. We assume that all trials start on August 1, 2020, and simulate the epidemiological models until December 31, 2022. We perform sensitivity analysis over a wide range of trial design, epidemiological model, and population vaccination schedule assumptions (see Table 1), covering 756 different scenarios. We summarize our results in Table 2 and Section A11 in the S1 Appendix. In addition to our results, we release an open-source version of our simulation software, and encourage readers to rerun our simulations with their own preferred set of assumptions and inputs.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Sensitivity analysis with respect to trial design, epidemiological model, and population vaccination schedule assumptions.

https://doi.org/10.1371/journal.pone.0244418.t001

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Expected number of incremental infections and deaths avoided in the U.S. under different trial designs, vaccine efficacies, and epidemiological scenarios, assuming trials start on August 1, 2020, superiority testing, and 10M doses of a vaccine per day are available after licensure, compared to the baseline case in which no vaccine is ever approved.

https://doi.org/10.1371/journal.pone.0244418.t002

Assuming superiority testing and a vaccine efficacy of 50%, we estimate the date of licensure of the hypothetical vaccine candidate to be some time in November 2021 under an RCT (476 days), between June and August 2021 under an ORCT (326 to 380 days), between April and June 2021 under an ARCT (246 to 306 days), and between March and June 2021 under an HCT (221 to 311 days). Apart from an RCT which has a fixed trial duration, the dates of licensure from the ORCT and ARCT depend largely on the status of the epidemic during the clinical trial. If the transmission rate of the disease is low (e.g., due to social distancing or other non-pharmaceutical interventions), an extended surveillance period is required to accrue enough natural infections in order to observe a statistically significant difference in infection risk between the treatment arm and the control arm. Conversely, when the transmission rate is high, a short surveillance period is sufficient to observe significant results. We note that an HCT, on the other hand, does not depend on the epidemic situation but is instead limited by the time required to set up the challenge model. In general, we find that the time to licensure under ORCT and ARCT decreases with increasing vaccine efficacy: the greater the efficacy, the easier it is to observe a significant treatment effect.

We find that the ARCT provides the greatest expected net benefit among the three RCT designs in almost all scenarios. The utility of an HCT versus the RCTs, however, depends critically on the set-up time and the dynamics of the epidemic. For example, assuming superiority testing, a vaccine efficacy of 50%, the behavioral epidemiological model, and a population vaccination schedule of 10M doses per day, we estimate that the ARCT can help accelerate licensure by almost 8 months versus the RCT, thus preventing approximately 2.9M incremental infections and 23,000 incremental deaths from COVID-19 in the U.S. versus the latter.

Under the same set of assumptions, an HCT that requires 30 days to set up can further reduce the time to licensure by a month, thus preventing approximately 1.1M more infections and 8,000 more deaths versus the ARCT. However, the advantage of the HCT vanishes when its set-up time is long: an HCT that requires 90 days to set up takes about one month longer to reach licensure as compared to the ARCT, leading to around 1.0M more infections and 8,000 more deaths versus the latter (see Fig 2a). Under such circumstances, the use of an HCT is worthwhile only when the prevalent transmission rate is low. If we consider the status quo scenario instead of the behavioral epidemiological model, the time to licensure is about one month shorter under the HCT than under the ARCT even with a 90 day set-up period (see Fig 2b). In this case, the HCT prevents approximately 60,000 incremental infections and 500 incremental deaths versus the ARCT. We observe similar trends under superiority-by-margin testing at a threshold of 50%.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Dates of licensure under RCT, ORCT, ARCT, HCT (30-day set-up time), and HCT (90-day set-up time), assuming superiority testing, a vaccine efficacy of 50%, and a population vaccination schedule of 10M doses per day.

(a) Under the behavioral epidemiological model. (b) Under the status quo epidemiological model.

https://doi.org/10.1371/journal.pone.0244418.g002

5 Discussion

There has been a plethora of papers highlighting various ethical considerations for conducting HCTs [40, 41], some specifically for COVID-19 [9, 42–46]. Some of the main ethical concerns are: (1) what is the explicit scientific rationale for, and societal value of, an HCT; (2) whether the risks of harm to the subjects and the public at large are understood by the scientists and have been minimized; (3) whether informed consents have been obtained from subjects after they are given full disclosures of the risks involved; and (4) whether the subjects have been selected fairly and given appropriate compensation for both the risk and actual harm brought on by HCTs. Most bioethicists generally accept that these concerns can be addressed within the existing ethical framework for human medical research.

Our paper addresses the first and second of these ethical concerns. We provide scientific justifications for COVID-19 HCTs by considering how conducting them can allow companies to learn about the protection curves and accelerate the development of vaccines against SARS-CoV-2.

However, our analysis does not address the latter two ethical considerations as they concern the execution of HCTs, which is beyond the scope of this paper. Nonetheless, companies and scientists seeking to perform HCTs, and especially regulators, will have to address those concerns to preserve public trust and avoid a public backlash that could jeopardize other important medical research critical to addressing the current epidemic.

Some scientists argue that “a single death or severe illness in an otherwise healthy volunteer would be unconscionable” [46]. However, it can be argued that allowing tens of thousands of individuals to die by denying the consent of an informed individual to take a calculated risk is equally unconscionable. In this paper, we adopt the Benthamite approach [47], where every individual’s utility is weighted equally in the aggregate utility function, as is the common convention in public economics analyses. Within this ethical perspective, our calculations show that an HCT can potentially provide substantial public health benefits in terms of accelerating vaccine development and reducing the burden of coronavirus-related mortality and morbidity in the U.S.—in some cases, by more than 1.1M infections and 8,000 deaths compared to the best performing RCT—when conducted early in the pandemic’s life cycle and in cases where the spread of COVID-19 in the population is muted due to non-pharmaceutical interventions.

We also expect the financial costs of an HCT—which includes the cost of liability protection—to be lower than those of a traditional vaccine efficacy RCT, adding further support for a challenge design (see Section A4 in the S1 Appendix for further discussion). While we have focused on public health outcomes here, it is clear that accelerated vaccine development provides tremendous societal and economic benefits as well—e.g., savings in insured medical costs, direct medical expenditures, and hospitalization costs, and accelerated economic recovery from an earlier reopening.

We emphasize that the expected costs and benefits of a clinical trial depend critically on many assumptions about existing conditions. For example, recruiting subjects in sufficient numbers and diversity can sometimes present a challenge for clinical trials involving experimental vaccines (although, in the case of HCTs for COVID-19, the organization 1Day Sooner reports over 32,000 registered volunteers as of July 27, 2020). Also, we do not include set-up time for non-challenge RCTs because phase 3 vaccine efficacy trials are already imminent as of now. Moreover, we assume a relatively short set-up time for HCTs because a challenge study can be set up relatively quickly using a wild-type strain [24], and the National Institute of Allergy and Infectious Diseases (NIAID) appears to have already made some headway in manufacturing challenge doses [48–50]. If, instead, we assume comparable set-up times (e.g., two months) and start dates for both an HCT and non-challenge RCTs, we expect that an HCT can accelerate licensure by two months when compared to an adaptive RCT, assuming superiority testing, a vaccine efficacy of 50%, and the behavioral epidemiological model. Some have argued that at least one to two years is required to develop a robust model from scratch [46]. In this case, our results indicate that an ARCT will almost always be faster than an HCT. However, even if an HCT with a long set-up time does not lead to faster vaccine licensures over an ARCT given current conditions, the creation of a standing HCT agent and setting up an HCT now can provide a hedge against potential failures in the current crop of vaccine candidates. By having an approved, ready-to-go challenge virus and ready-to-go HCT sites that vaccine developers can access immediately, the approval process for as-yet-untested SARS-CoV-2 vaccine candidates can be accelerated when required. For a pandemic like COVID-19, such a hedge will almost always show substantial net benefits relative to its costs.

HCTs have several other benefits that will be more obvious as the pandemic progresses. They require many fewer eligible volunteers, whose numbers will dwindle as the pandemic progresses. They do not depend on attack rates at clinical trial sites which are notoriously difficult to estimate and highly dependent on non-pharmaceutical interventions such as lockdowns and other social-distancing policies. They also avoid logistical problems such as identifying subjects, obtaining subjects’ consent, obtaining institutional review board’s approval or tracking subjects, particularly when attempting large-scale clinical trials in places where contract research organizations (CROs) have little experience.

It is conceivable that multiple vaccines—instead of the single vaccine in our simulation study—are tested concurrently in a single trial design [51]. For example, five vaccines, such as those selected by Operation Warp Speed [52], could be tested concurrently in a six-arm trial (five vaccine arms and a control arm), requiring 40% fewer test subjects, thereby reducing in-trial expected morbidity and mortality costs by the same amount. The benefits can be increased if an adaptive platform clinical trial—designed to eliminate ineffective vaccines at the first signs of futility—is adopted. A clinical trial testing multiple vaccines can also reduce competition for volunteers, a problem that continues to plague vaccine developers [53].

We choose to quantify the cost and benefits of the clinical trials by measuring the number of infections and deaths avoided, and refrain from performing a traditional health technology assessment, such as comparing the economic value of an HCT versus an RCT using quality-adjusted life years measures or willingness to pay estimates such as the value of a statistical life. Performing such computations is straightforward given the output of our simulations, but we have refrained from doing so in deference to non-economist stakeholders who find it offensive to use any pecuniary measures when discussing the loss of human life.

Finally, our analysis focuses mainly on the U.S. for practical reasons involving access to data with which to calibrate our simulations and the broader goal of informing U.S. public health officials and policymakers as the country enters the final stages of vaccine development. However, we note that vaccine companies such as Pfizer/BioNTech are also looking at recruitment in the Southern Hemisphere (e.g., Brazil [54]), which can affect the rate at which these events accumulate in trials, depending on the spread of COVID-19 in those countries. In addition, a vaccine licensure may apply internationally. Given that the U.S. currently comprises 25% of all confirmed COVID-19 cases (as of July 7, 2020) [29], if the assumptions made in our study also hold internationally, the net benefits for all the clinical trials will scale by a factor of 4, in which case HCTs can save an additional 4.4M infections and 32,000 deaths compared to the best performing RCT in certain situations.

We highlight that these figures depend heavily on the development of the epidemic in the U.S. moving forward. We have considered three simple scenarios, status quo, ramp, and behavioral, corresponding to low transmission, moderate transmission, and behavioral-based response, respectively. There are clearly many other sources of uncertainty that are not reflected here. For example, non-adherence to social distancing advisories and/or resistance to precaution recommendations such as wearing a mask in public will lead to an uncontrolled outbreak, which will help to accelerate non-challenge RCTs, making them attractive even when compared to an HCT with a short set-up time. We have found it difficult and impractical to incorporate these uncertainties in our assumptions due to the speed at which things are evolving and the unpredictability of public reaction. In addition, studies that have attempted to incorporate such uncertainties in their epidemic model report huge error bounds in their projections [55]. The wide confidence intervals prevent us from drawing any useful conclusions, which severely limit the usefulness of such models. Therefore, we recommend readers not to take our results as final or definitive, but to re-run our simulations with their own preferred set of assumptions, calibrated using the most current epidemiological data.

6 Conclusion

Our paper presents a systematic framework for quantitatively accessing the in-trial and societal cost/benefit trade-offs of various clinical trial designs in terms of infections and deaths averted. We hope that this framework will allow stakeholders to make more informed practical and ethical decisions regarding accelerating COVID-19 vaccine development in the ongoing pandemic.

Supporting information

Acknowledgments

We thank Arthur Caplan, Norman Baylor, and Frederick Hayden for helpful comments and discussion, Amanda Hu for research assistance, and Jayna Cummings for editorial support. The views and opinions expressed in this article are those of the authors only and do not necessarily represent the views and opinions of any other organizations, any of their affiliates or employees, or any of the individuals acknowledged above.