Patients and study design

A convenience sample of patients from 10 sites was collected. The study sites span the whole spectrum of sizes from large university hospitals in major cities with multiple dedicated Covid-19 wards (Thomayer University Hospital in Prague, Motol University Hospital in Prague, Kralovské Vinohrady University Hospital in Prague, General University Hospital in Prague, University Hospital in Pilsen) through major regional/specialized hospitals (Na Homolce Hospital in Prague, Military Hospital Olomouc) as well as smaller hospitals caring for just several Covid-19 patients at a time (AGEL Hospital Nový Jičín, Hořovice Hospital, Třebíč Hospital). The sites were chosen based on availability and willingness of the personnel to participate in data collection. None of the study sites was exclusively dedicated to treating Covid-19 patients. For each site, the dataset contains all patients hospitalized in the participating wards over the data collection period. The data collection started at the onset of the Covid-19 pandemic in March 2020 (except for one site where some older records were inaccessible), but the end date for collection differs between sites due to time constraints of the participating physicians. Three sites included total of 23 patients that could be considered part of “second wave” (admitted after September 1st). Last patient included in the dataset was admitted on October 12th. See S1 Fig for per-site data collection periods. Patients over the age of 18 were included if they had PCR-confirmed infection of SARS-CoV-2 and were not participating in a clinical trial of any Covid-19 pharmacotherapy.

Not all patients developed pneumonia or other symptoms of Covid-19. All patients received the standard of care which could include supplemental oxygen and ventilation and antibiotics for bacterial superinfections, as determined by the attending physician. Some patients were not indicated for all treatment modalities (especially mechanical ventilation) based on decision of the attending physician and underlying patient condition. We note that the participating sites were not homogeneous in either patient population or treatment protocols. The choice of pharmacological treatment was based on the decision of the attending clinician and its availability.

The study was approved by the Ethical committees of General University Hospital, Hospital Nový Jíčín, Motol University Hospital, Thomayer Hospital, University Hospital Vinohrady, Military Hospital Olomouc, Na Homolce Hospital, University Hospital in Pilsen, Hořovice Hospital, Jihlava Hospital, all data were collected in fully anonymized form. Data was collected between June and October 2020 for patients that were treated between March and October 2020.

Data collection

We collected data on comorbidities and information about disease progression on daily resolution including breathing support required, oxygen flow rate, experimental anti-Covid-19 and antimicrobial drugs taken and several laboratory markers (PCR positivity for SARS-CoV-2, C-reactive protein, D-dimer, Interleukin 6, Ferritin, lymphocyte count). Full protocol for data collection is attached in S1 File and the data collection table in S2 File. Due to very low number of patients using extra-corporeal membrane oxygenation (N = 1) or non-invasive positive pressure ventilation (N = 6) in our sample, we merged those categories with mechanical ventilation.

Statistical analysis

The character of the convenience sample does not allow for a proper assessment of the association between treatments and patient outcomes, because the treatments had not been assigned to patients at random but were only observed retrospectively. This can be partially remedied by adjusting for patient characteristics in the analysis, but such adjustments will always be imperfect and the analysis needs to be treated as exploratory and interpreted cautiously.

Since many details of analysis may influence the conclusions made, we performed multiverse analysis [15] and report results for all the hypothesis tested across multiple different models using both frequentist and Bayesian paradigms. For each model class we worked with several possible sets of adjustments. All analyses were performed in the R language [16], visualization and data cleaning was run with the tidyverse package [17].

First class of models are frequentist survival and multistate models under the proportional hazards assumption as implemented in the coxph function from the survival package [18]. We primarily use a model with competing risks for death and discharge from hospital (see Fig 1a).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. States used in the competing risk model (a) and in the two hidden Markov model variants (b,c).

AA = Ambient air, Oxygen = Nasal oxygen, Ventilated = any form of ventilation (non-invasive positive-pressure ventilation, mechanical ventilation and extra-corporeal membrane oxygenation). In all models the ‘Death’ and ‘Discharged’ states are terminal. In the second hidden Markov model (c), the ‘Improving’ and ‘Worsening’ variants of each non-terminal state are not observable—only the breathing support is observed and improving/worsening is inferred from progression of the disease.

https://doi.org/10.1371/journal.pone.0245103.g001

Second class of models are Bayesian hidden Markov models (HMM) of disease progression implemented via a custom extension to the brms package [19]. The parametrization of the HMM is inspired by Williams et al. [20]: the actual process of disease is assumed to be continuous and allow only for transitions between neighboring states (as shown in Fig 1b and 1c). The total probability of transition between any two states over the period of a day is then computed as the total probability of transition across all possible paths. This class of models does not satisfy the proportional hazards assumption, instead, it is assumed the process has the Markov property—i.e., that the (potentially unobserved) state and the covariates at a given day contain all the information to determine probabilities of the states on the next day. We use two versions of such models, one working solely with the observed breathing support and one assuming a hidden improving/worsening distinction. All of the hidden Markov models take into account whether best supportive care was initiated and a patient was thus not indicated to progress to more intensive treatment modalities.

Finally, we used a set of Bayesian regression models implemented with the brms package [19]. Those included overall survival, state at day 7 or 28 as either binary or categorical outcome and a Bayesian version of the Cox proportional-hazards model.

Except for age, sex and comorbidities, all covariates are treated as time-varying, e.g., the effect of taking a drug is only included for the days after the drug was taken. More details on the exact model formulations can be found in the supplementary statistical analysis in S3 File.

Evaluating prognostic models

We searched the living systematic review of Covid-19 prognostic models [14] for those that could be applied to our dataset (i.e., where we have gathered all the input features). We primarily focused on the Area Under Receiver Operating Characteristic Curve (AUC), and its bootstrapped 95% confidence intervals which we computed using the pROC package [21]. When there were multiple reasonable ways to evaluate the outcome or a predictor in our dataset, we computed and reported all of those options. We used two simple scores with age or the decade of age as the sole predictor to have a baseline to compare the scores against.

Complete code for all analyses is available at https://github.com/cas-bioinf/covid19retrospective/.

Baseline characteristics

In total, we were able to gather data for 213 patients, see Table 1 for the overall characteristics of the patient sample and several subgroups we used in the analysis, including treatments taken. Counts of all treatment combinations are shown in S2 and S3 Figs shows outcomes by study site, demonstrating quite large hospital-specific differences. The dataset includes 19 patients already reported in a study of inflammatory signatures of Covid-19 [22].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Patient characteristics for the overall sample and treatment subgroups.

Note that the favipiravir subgroup is not exclusive with either the HCQ or No HCQ group.

https://doi.org/10.1371/journal.pone.0245103.t001

Disease progression

In Fig 2 we show the overall disease progression for all patients and in Fig 3 we show the time-course of a subset of the markers we have measured. The data show some interesting patterns: patients with low Interleukin-6 or D-dimer values are overrepresented among patients with better outcomes, most patients had high CRP upon admission and for many the CRP levels stayed elevated over the whole hospitalization. However, the limited nature of the data does not allow for any statistically robust conclusions. We also see that the marker levels were not substantially stratified by study site. Those patterns should however be interpreted with care due to systematic missingness of the data—in particular, patients that fared worse were probably more likely to have the markers measured. However, we believe this kind of patient-level view is useful to appreciate the extent of both between-patient and within-patient variability.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Disease progression for all patients included in the study as determined by breathing support required.

Each vertical strip is a single patient, the ordering on the horizontal axis is by disease severity. Ventilated = any form of ventilation (non-invasive positive-pressure ventilation, mechanical ventilation and extra-corporeal membrane oxygenation).

https://doi.org/10.1371/journal.pone.0245103.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Values of selected markers over the course of the disease.

Each line represents a patient, stratified by the worst breathing support required over the hospitalization. Color indicates study sites. The vertical scale is logarithmic. Ventilated = any form of ventilation (non-invasive positive-pressure ventilation, mechanical ventilation and extra-corporeal membrane oxygenation), CRP = C-reactive protein [mg/l], D-dimer [ng/ml DDU], Ly = lymphocyte count [10⁹/l], IL-6 = Interleukin 6 [ng/l].

https://doi.org/10.1371/journal.pone.0245103.g003

The between-patient variability is notable even across outcomes—when ordering the patients by the highest CRP levels experienced throughout the hospital stay, the top 20% of patients that breathed ambient air for the whole hospitalization experienced higher levels than the bottom 20% of patients that required ventilation or died. This overlap is even larger when comparing only against the patients that died and D-dimer, Interleukin-6 and lymphocyte count also show a notably larger overlap than CRP (S4 Fig).

Association between patients’ characteristics and treatments

As noted above, the nature of the convenience sample did not enforce random assignment of treatments to patients. In fact, patients with worse baseline characteristics, which lead to worse outcomes, were less likely to receive hydroxychloroquine (see Fig 4). This clearly creates a bias towards a positive effect of hydroxychloroquine on the outcome (and potentially for other treatments as well—most were used in combination with hydroxychloroquine), which, however, could be false.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. The choice of treatment with hydroxychloroquine seemed to be associated with the status of patients upon hospitalization.

Comorbidities were associated with both worse outcome (black) and lower chance of treatment with hydroxychloroquine (red). Dots and lines represent the estimates and the 95% confidence intervals of the log odds ratio of the respective outcome. HCQ: hydroxychloroquine, IHD: ischemic heart disease, HD: hypertension drugs, HF: heart failure history, COPD: chronic obstructive pulmonary disease, LungD: other lung disease, Dia: diabetes, RenalD: renal disease, LiverD: liver disease, HighCr: creatinin above 115 for males or above 97 for females, HighInr: Prothrombin time (Quick test) as International Normalized Ratio above 1.2, LowAlb: albumin in serum/plasma below 36 g/l.

https://doi.org/10.1371/journal.pone.0245103.g004

Taken quantitatively, the comorbidities known upon hospitalization were informative with respect to the future hydroxychloroquine treatment: the score representing the cumulative presence of ischemic heart disease, hypertension drugs, former heart failure, COPD, other lung diseases, renal disease, or high creatinine was associated with a lower chance of taking hydroxychloroquine over the course of the hospitalization (the chance was only 79.9%, 95% confidence interval (65.3, 97)%, Chi-square test in the logistic regression model, χ² = 5.18, df = 1, P = 0.023).

Association between treatments and outcomes

Here, we focus on hydroxychloroquine and azithromycin as those are the only treatments with larger sample size. We also investigate favipiravir as it is less well reported in the literature. Hydroxychloroquine was dosed almost exclusively in a 5-day regime starting with a loading dose of 800mg on the first day and followed by 400mg. Majority of patients complemented hydroxychloroquine with azithromycin while azithromycin was rarely used alone (see Table 1). Azithromycin was most frequently dosed 250 or 500mg/day, but doses ranging from 100mg/day to 1500 mg/day were observed. Favipiravir was used only at one site with a loading dose of 3600mg on the first day, followed by at most 9 days with a 1600mg dose. All but one of the patients receiving favipiravir also received hydroxychloroquine. Treatment was initiated mostly within two days of admission (see S5 Fig).

The results of the multiverse analysis for association between hydroxychloroquine, azithromycin and favipiravir usage and death is shown in Fig 5—here, we only show models that were not found to have immediate problems representing the data well or computational issues (see S3 File for details). Results for all models we tested are reported in S6–S8 Figs, with additional details in S3 File. The results do not change noticeably when only patients from the first wave are included (S6–S8 Figs).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Estimates of model coefficients for association between treatments and main outcomes.

Each row represents a model—Categorical 7/28 = Bayesian categorical regression for state at day 7/28, Bayes Cox = Bayesian version of the Cox proportional hazards model with a binary outcome, Cox (single) = frequentist Cox model with a binary outcome, Cox (competing) = frequentist Cox model using competing risks (as in Fig 1a), HMM A = Bayesian hidden-Markov model as in Fig 1b with predictors for rate groups, HMM B = Bayesian hidden-markov model as in Fig 1b with predictors for individual rates, HMM C = Bayesian hidden-Markov model as in Fig 1c. For frequentist models, we show maximum likelihood estimate and 95% confidence interval, for Bayesian models we show posterior mean and 95% credible interval. The estimands are either log odds-ratio (Categorical, HMM) or log hazard ratio (Cox variants). In all cases coefficient <0 means better patient outcome in the treatment group. Vertical lines indicate zero (blue) and substantial increase or decrease with odds or hazard ratio of 3:2 or 2:3 (green). Additionally the factors the model adjusted for are listed—Site = the study site, admitted = Admitted for Covid-19, Supportive = best supportive care initiated, Comorb. = total number of comorbidities, AZ = took azithromycin, HCQ = took hydroxychloroquine, FPV = took favipiravir, C. plasma = received convalescent plasma.

https://doi.org/10.1371/journal.pone.0245103.g005

Most models report that using hydroxychloroquine is associated with lower risk of death. We must however bear in mind the potential bias noted in the previous section. Also, we see that for the HMM models, as we add adjustments the credible intervals do not widen but instead shift towards zero. This is a weak indication that further adjustments could drive the effect towards zero. We did not attempt to model additional adjustments as the models became computationally unstable. The case of hydroxychloroquine serves as a “control group” for our other results—since randomized trials give us high confidence that hydroxychloroquine does not substantially reduce mortality, we can be quite certain the associations we observe for hydroxychloroquine are just a measure of bias in the data. Additionally, our models either cannot determine the sign of association between azithromycin and risk of death or even show an increase in risk of death. This serves as a weak evidence that a substantial improvement in mortality from azithromycin is unlikely.

Most models exclude very strong association between increased risk of death and using favipiravir, but our data are necessarily quite limited, which is reflected in the very wide uncertainties around estimates. We also cannot put strict bounds on the association between favipiravir and length of hospitalization.

We also examined the association between treatments and length of hospital stay for all the patients that survived. Almost all models cannot discern the sign of the association for all treatments examined (S6–S8 Figs). Similarly, we studied the association between D-dimer and Interleukin 6 and outcomes, with unconclusive results as well (S9 Fig).

Published prognostic models are not better than using age as the sole predictor of outcome

Following Wynants et al. [14] we found five prediction models we were able to recompute: Li et al. report the ACP index [23] combining CRP and age to form 3 grades, Chen & Liu [24] report a continuous score using age, CRP, D-dimer and lymphocyte count, Shi et al. [25] use age, sex and hypertension to form 4 grades, Caramelo et al. use age, sex, hypertension, diabetes, cardiac disease, chronic respiratory disease and cancer to form a continuous score [26], Bello-Chavolla et al. [27] use age, diabetes, obesity, pneumonia, chronic kidney disease, COPD and immunosuppression to build a score ranging from -6 to 22. For the latter two scores we had to impute some of the predictors as they had no immediate equivalent in our dataset. The outcomes present in the studies were: 12-day mortality, 30-day mortality and mortality without any further details, here we report results for both 12-day and 30-day mortality. Full details on the scores and how we used our dataset to compute them is given in the S3 File.

All prognostic models we tested performed similarly to or notably worse than using age as the only predictor and also worse than originally reported (Fig 6). Additionally, some publications did not provide enough detail to unambiguously reconstruct how the score and/or outcome was assessed. We thus concur with Wynants et al. [14] that reported prediction scores are at high risk of bias and need additional careful evaluation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Performance of tested prediction scores as measured by AUC.

AUC = 1 means perfect prediction while AUC ≤ 0.5 means that the score is worse than random guess and a better prediction would be obtained by reversing the score (marked by thick blue line). The line ranges represent the bootstrapped 95% confidence intervals. Red dots show results computed in present study—model variants (horizontal axis) vary in the outcome measured (12-day or 30-day mortality, severe disease) and potentially on how ambiguities in score computation were resolved, although this rarely makes a big difference—see S3 File for details. Cyan triangles show AUC as reported by the original authors or recomputed based on their published data. When the confidence interval or the AUC of the original study is not shown, it means that the value was not reported by the authors and not enough information to recompute it was given.

https://doi.org/10.1371/journal.pone.0245103.g006