Data: The PRospective, Observational, Multi-center Major Trauma Transfusion study

The PRospective, Observational Multi-center Major Trauma Transfusion (PROMMTT) study enrolled 1,245 individuals at ten level I trauma centers from around the United States [8]. Patients had to survive at least 30 minutes and receive at least one unit of red blood cells within 6 hours of arrival in the emergency department (ED) [8]. Once enrolled they were followed for diagnostic and therapeutic procedures and subsequent outcome data were collected [8]. The PROMMTT study was approved at each study site and the Data Coordinating Center by the local institutional review boards. The US Army Human Research Protections Office also provided secondary level review and approval. Patient records were de-identified prior to their use in this analysis.

For the purposes of this analysis, we partitioned centers and defined ‘volume’ based on the number of patients massively transfused, or receiving ten or more units of red blood cells in the first 24 hours, at each site. The top three massive transfusers were defined as large-volume centers while the others were defined as small-volume centers. We note that the hospitals could have been dichotomized by volume in many ways, and so the choice based upon volume of MT is somewhat arbitrary, and used for illustrative purposes only. Volume varies from a low of 4 patients in one hospital during the study period, to a high of around 80 patients, with no obvious inflection in MT volume between “large” and “small” groups. Thus, for the remainder of the paper, “large” versus “small” only refers to relative number of MT patients in this arbitrary grouping, with the split point placed between 32 and 33 MT patients, noting the analysis could be equivalently done for other categorizations, which, in turn, could have led to different estimates.

Only 1,242 of 1,245 PROMMTT patients had sufficient data to analyze and were used for the analysis. Table 1 shows the distributions of the outcomes of interest and other cohort characteristics in our sample. The three large-volume sites treated 551 individuals while 691 were treated at the seven small-volume sites.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary statistics.

Summaries are presented as percent where indicated, and mean (standard deviation) otherwise.

https://doi.org/10.1371/journal.pone.0136438.t001

Model and Parameter of Interest

The goal was to estimate the relationship between patient outcomes and site volume type, adjusting for a large number of covariates. If we define Y as the outcome, S (large vs. small-volume site) as the explanatory variable of interest, and C as all confounding variables, we could predict, on average, what would happen to a patient if they went to a site of a specific size, if we knew the model: mean(Y | S = s, C). We discuss below how to estimate this prediction function, going beyond traditional regression approaches and utilizes machine-learning tools. First, however, we need to define a parameter that is not just a prediction for a specific patient (defined by their covariates S and C), but an association that summarizes the impact for each patient across the patient population. For instance, we could define the mean of these predictions, mean(Y(s)) = mean{mean(Y | S = s, C)}, where the outer mean is taken over all the patients (and the “natural” distribution of confounders). If the outcome prediction model is correct, along with important identifiability assumptions outlined below, we could then compare patient outcomes if, keeping their covariates (C) the same, they went to all large vs. small volume sites. This parameter, mean(Y(large))−mean(Y(small)), would be a measure of the average causal association of site size. We chose a slightly different parameter, which examines this difference only among the people who went to larger sites, or mean{Y(large)−Y(small) | S = large}. In what sub-group one estimates the causal association is somewhat arbitrary, but in this case we chose such a parameter because 1) these populations are often quite different, and a parameter over the whole study population has little meaning in some theoretical mixed target population of large and small hospitals and 2) we wish to have an effect estimate for which one could compare to a matching procedure discussed below, in which patients at large sites are paired to patients at small sites based on their characteristics. Our estimand of interest (based on explicit identifiability assumptions discussed below) is an effect of treatment among the treated (ETT): (1)

Estimation: Regression Models

As mentioned above, we do not know the outcome prediction model mean(Y | S = s, C), nor do we know how the distribution of center types differs by patient characteristics, Prob(S = large | C), sometimes called a propensity score, and thus must estimate both functions from the data. While there are many options for the choice of prediction model to use, in order to make the most efficient use of available information and to estimate this prediction function using a theoretically optimal procedure, we advocate the use of the ensemble machine-learning algorithm called SuperLearner (SL). This prediction algorithm consists of a weighted combination of a library of prediction procedures [9]. We used 10-fold cross-validation where the data were divided randomly into 10 equal validation samples. One at a time, these validation samples were “removed” from the data, and prediction algorithms were fit on the remaining 9/10 of the data (the so-called training sample). Then, for each of these 10 overlapping training samples, the procedure predicted, for each algorithm, on the independent (independent of the corresponding training sample) validation sample. Finally, the SL procedure found the optimal weighted average of these predictors by comparing the observed outcomes to those predicted based on the models from the corresponding training sample. The procedure is more than an intuitive way to build a prediction model, as theory shows that the SL is guaranteed to perform as well as or better than the best prediction algorithm in the supplied library [9]. Thus, in essence one cannot beat SL, because any competing algorithm can be added as one of the candidates. Another of the great advantages of this approach is that it takes the art (and thus potential bias) out of model selection, as the user does not have to choose a particular prediction model but can allow all of them to be competitors, including very simple algorithms, very complicated machine learners, and even particular models that the researchers prefer. In this case, the SL builds the final model based on which of them does the best job of predicting a “new” set of data, so one need not engage in sterile arguments about the best approach to model such data—in this case “proof is in the pudding”.

Finally, though there was almost no missing values for the outcomes (see Table 1), some of the covariates have missing values for many of the observations. Instead of using an imputation approach, we create new basis functions: indicators of whether the variable is not missing (1 if not missing, 0 if missing, denoted by Δ) and the interaction of each indicator with its respective covariate (ΔC). The advantage of this approach is that this missingness information is used directly by the procedure to find the prediction model, so that if the missingness is informative, and there is sufficient information in the remaining measured variables to explain it relative to the outcome, then the procedure will find an optimal prediction model. Also, the necessary identifiability assumptions to get consistent estimates of causal effects under this strategy are very transparent—that is, the combination of observed covariates and missing data information must be sufficient such that there are no further unmeasured pathways related to the set of explanatory variables we include in the model.

In an observational clinical study like PROMMTT, some covariates of interest almost certainly remain unobserved due to life-saving measures trumping data collection. Since a patient’s condition could dictate whether or not information on that patient’s condition was recorded, missingness not at random (MNAR) cannot be ruled out here. Just as is the case with the assumption of no unmeasured confounding in observational studies, there is no single solution and in some circumstances one might conclude that the estimation is hopelessly biased due to the missing covariates, regardless of the missing data strategy employed. Here, our goal is to optimally estimate determinants of patient outcomes in the context of the available information without manufacturing additional information under potentially unsupported parametric assumptions. We therefore choose to employ the “indicator method” described above for any covariates with missing values, reported in Table 1.

Estimation: Effect Estimates

We used several methods to estimate the ETT, the first being a simple substitution estimator [10], where one first predicts the outcome of interest based on predictors (S and C) to derive an estimate of mean{Y | S, C} and subsequently applies this model to predicting the outcome for subjects if one kept the C fixed, but changed the S to “large.” The average difference among these predicted values and the observed outcomes in large volume sites defines the estimate of the ETT. We did this approach both using an estimated main terms logistic regression model for mean{Y | S, C} (“Simple Subs.(regression)” in Table 2) as well as the same estimator based upon a SL fit using the SuperLearner [9,11] package in R [12]. For both, we derived the inference (the standard error) using the nonparametric bootstrap [13]. Though there is no theory that guarantees this estimator based upon SL has a normal limit distribution, we estimated the standard error using the nonparametric bootstrap and generated Wald-style confidence intervals (CI). For the estimator based upon main terms logistic regression, we can assume asymptotic normality but the CI is still not valid due to bias.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Difference in means, and adjusted ETT estimates (simple substitution, TMLE, and propensity score matching).

95% confidence intervals are included in parentheses after each estimate. Asymptotic (normal-based) confidence intervals were calculated using the standard error of the Unadjusted, Targeted Maximum-Likelihood, and Matching estimators. The nonparametric bootstrap was used to generate 95% confidence intervals in the case of the Simple Substitution estimator.

https://doi.org/10.1371/journal.pone.0136438.t002

This substitution estimator can be updated to achieve additional bias reduction via a so-called targeted maximum-likelihood estimation (TMLE) step. The targeting procedure updates the estimated conditional mean functions mean{Y | S, C} and Prob(S = large | C) by adding “clever covariates” to the original respective SL regression fits. In each case, a logistic regression of the outcome (Y or S) on its respective clever covariate is estimated, with the log-odds of the previously predicted outcome for each patient serving as an offset. When the procedure converges (meaning that the resulting logistic regression coefficients no longer change measurably between iterations), the resulting updated function mean{Y | S, C} can be evaluated at each level of the treatment to yield a targeted estimate of mean{Y | S = large, C}−mean{Y | S = small, C}, evaluated only among those treated at a large volume site. This estimator has additional desirable statistical properties that the original, non-targeted substitution estimator does not; the additional step can further reduce bias and results in normally distributed estimators, whereas the original substitution estimator based upon SL alone is not guaranteed to have a normal limit distribution (see chapter 8 in [14]).

Finally, the matching estimator uses the SuperLearner to estimate the propensity score, Prob(S = large | C), and matches each large volume site patient to the closest small volume site patient based on this score (in this case, with replacement and allowing ties). Any small volume site patient whose propensity score was not within 0.05 standard deviations of any large volume site patient’s score was discarded for the purposes of this estimator. Here, the ETT is estimated by the average of the difference in the outcomes among these matched sites [15,16].

We note that the substitution estimator and the matching estimator rely on consistently estimating the prediction model and propensity score model, respectively. However, the TMLE relies on only estimating one of these consistently, and thus has greater robustness to model misspecification than either of these competitors.

Diagnostics

One of the challenges of using flexible, machine-learning tools is how to examine the fit of such models, and also how to explore which variables have the greatest influence on the resulting estimators. Though having a simple regression model that one can examine relatively easily in more traditional approaches seems an advantage, it is generally illusory. Because a restricted parametric model is likely to be misspecified, the resulting coefficients, for instance, are likely to have unclear interpretations, so their attractiveness is more the appearance of simplicity than the actual utility of interpretation [17]. An estimator like the TMLE, on the other hand, allows for use of more flexible, nonparametric prediction models, is protected against misspecification of either the treatment mechanism or the conditional distribution of the outcome, and uses the information in the data in the most efficient manner for estimating a target parameter. One could consider as a downside of these techniques their reliance on “black-box” algorithms, which, without further exploration, are not easily interpreted. However, below, we provide some example tools for exploring the results reported in Table 2, which help to explain how this automated, data-adaptive procedure results in the final estimates.

Effect of patient differences on overall mortality

We demonstrate these tools by examining the results for overall mortality. The goal is to identify which patients have the biggest impact on the estimates of the ETT, and perhaps are those most sensitive to any differences in practice at these clinics. To do so, we calculated “counterfactual” residuals for each observation among those in the S = large clinics: Y—predicted(Y | S = small, C), or the difference of the observed outcome and that predicted according to the resulting model had the C stayed the same, but the clinic size switched from its observed value to “small.” Positive residuals identify those subjects who have an estimated increase in outcome possibly due to hospital-volume, while those with negative values display the opposite. One could group or cluster the patients in various ways to examine if there are some patients that appear to have been more affected by going to a small versus large-volume hospital. We did so by ordering the patients using their predicted probability of death based on our SL model; this results in a meaningful ranking of patients from those are lower to higher risk of the outcome based upon the constellation of clinical and demographic factors. Specifically, a loess smooth of these residuals versus simply their predicted outcome (in this case, predicted(Y | S = large, C)) was examined. This allows one to estimate the difference in mortality due to volume versus patients on a scale from lower to higher estimated risk of death. We repeat the converse of this for subjects from smaller sites. The full results are presented in S1 and S2 Figs, for binary and continuous outcomes, respectively, while Fig 1 displays a subset of the results contained in S1 Fig.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. “Counterfactual” residual plot for binary outcomes.

Residuals (see Diagnostics) for each outcome were plotted against probabilities of overall mortality for large (dotted line) and small-volume (solid line) site patients for binary outcomes using loess smooths.

https://doi.org/10.1371/journal.pone.0136438.g001

The top inset of Fig 1 gives schematics for what these plots might look like under three scenarios: 1) there is no difference in the estimated ETT for the outcome (two horizontal lines close to the null line of 0), 2) there is a significant difference, but all patients are affected equally (two horizontal lines away from the null line), and 3) some patients (in this case, patients at a higher risk of death) are more affected by whether they end up at a large versus small volume hospital. For those outcomes where the estimate of ETT was close to the null, both the large and small volume hospital curves are near the null line at 0 (e.g., complications). For overall mortality, among those individuals in smaller hospitals, those with greater estimated probability of overall mortality appear to contribute the most to the estimate of greater increase in mortality among smaller sites, though this difference gets smaller for the most severe (highest probability of death) patients. Though it is not appropriate to reference the 95% confidence bands as reliable inference (given this is post-hoc), these exploratory plots provide a way to decompose the estimated differences seen in Table 2.

In order to characterize the individuals whose mortality would have been most affected by a site type change, we compared patients on either side of the point at which the loess curves started to diverge from 0, an area highlighted in blue in S4 Fig. This identified 148 individuals with negative residuals (survived) and 68 with positive residuals (died) who were treated at a large volume site and were estimated to be most affected by a theoretical change from a small to large volume site. These patients were then compared to the remaining large volume site patients in the white area of S4 Fig —those on whom the site volume appeared to have, on average, less of an impact. Patients who were estimated to be most affected by the site change were significantly more likely to be Black, had higher blunt injury rates, were more severely injured, and were further from consciousness (S1 Table). They also had significantly longer partial thromboplastin times, more severe base deficits, smaller INRs, and lower platelet counts (S1 Table). They were more likely to experience multiple organ failure and had higher transfusion rates, as well as lower rates of transfused red blood cells relative to other blood products (S2 Table).

To further illuminate which types of patients were driving the estimated difference due to hospital volume, we constructed a histogram of counterfactual predicted probabilities of mortality, by site (S3 Fig). Patients from all sites span the entire range of probabilities, suggesting that the curves fit to the residuals (Fig 1, S1 and S2 Figs) are not driven by only a few individuals.

Propensity Score Matching Follow-up

In addition to the supplementary analysis on the subset of patients whose mortality probabilities were estimated to be most impacted by a site type change, we explored the matched data set generated by the propensity score procedure to determine whether the procedure achieved balance in the covariates in the large versus small volume comparisons. The heatmap in Fig 2 is a visualization of the entire matched data set where the data have been scaled to be comparable across different variables. This procedure provides an exploration of the joint distribution of variables across individuals to see if balance was achieved across the multivariate distribution of adjustment variables. If the matching had not been successful, we would expect to see clusters of individuals corresponding to groupings based on the hospital volume (large versus small). While we see distinct clusters of individuals based on their covariate values, this appears not to coincide with being in a large versus small volume site, suggesting that reasonable balance in the covariates (via propensity score matching) occurred.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Heatmap of matched data set with hierarchical clustering.

A visualization of the scaled covariate values in the matched data set with hierarchical clustering of the individual revealed little clustering of the individuals by site type, indicated in the bar on the left, suggesting that balance in the covariates was achieved. The dendrogram on the right is the result of hierarchical clustering of the individuals based on their covariate values. The color bar on the left indicates the site size where each individual was treated (purple corresponds to small-volume sites and blue to large-volume sites).

https://doi.org/10.1371/journal.pone.0136438.g002

PRospective Observational Multicenter Major Trauma Transfusion (PROMMTT) Study Group:

Data Coordinating Center, University of Texas Health Science Center at Houston: Mohammad H. Rahbar, PhD (principal investigator); John B. Holcomb, MD (co-investigator); Erin E. Fox, PhD (co-investigator and study coordinator); Deborah J. del Junco, PhD (co-investigator); Bryan A. Cotton, MD, MPH (co-investigator); Charles E. Wade, PhD (co-investigator); Jiajie Zhang, PhD (co-investigator); Nena Matijevic, PhD (co-investigator); Yu Bai, MD, PhD (co-investigator); Weiwei Wang, PhD (co-investigator); Jeanette Podbielski, RN (study coordinator); Sarah J. Duran, MSCIS (data manager); Ruby Benjamin-Garner, PhD (data manager); Robert J. Reynolds, MPH (data manager).

PROMMTT Clinical Sites:

Brooke Army Medical Center: Christopher E. White, MD (principal investigator); Kimberly L. Franzen, MD (co-investigator); Elsa C. Coates, MS, RN (study coordinator).

Medical College of Wisconsin: Karen J. Brasel, MD, MPH (principal investigator); Pamela Walsh (study coordinator).

Oregon Health and Sciences University: Martin A. Schreiber, MD (principal investigator); Samantha J. Underwood, MS (study coordinator); Jodie Curren, RN, BSN (study coordinator).

University of California, San Francisco: Mitchell J. Cohen, MD (principal investigator); M. Margaret Knudson, MD (co-investigator); Mary Nelson, RN, MPA (study coordinator); Mariah S. Call, BS (study coordinator).

University of Cincinnati: Peter Muskat, MD (principal investigator); Jay A. Johannigman, MD (co-investigator); Bryce RH Robinson, MD (co-investigator); Richard Branson (co-investigator); Dina Gomaa, BS, RRT (study coordinator); Cendi Dahl (study coordinator).

University of Pittsburgh Medical Center: Louis H. Alarcon, MD (principal investigator); Andrew B. Peitzman, MD (co-investigator); Stacy D. Stull, MS, CCRC (study coordinator); Mitch Kampmeyer, MPAS, CCRC, PA-C (study coordinator); Barbara J. Early, RN, BSN, CCRC (study coordinator); Helen L. Shnol, BS, CRC (study coordinator); Samuel J. Zolin, BS (research associate); Sarah B. Sears, BS (research associate).

University of Texas Health Science Center at Houston: John B. Holcomb, MD (co-principal investigator); Bryan A. Cotton, MD, MPH (co-principal investigator); Marily Elopre, RN (study coordinator); Quinton M. Hatch, MD (research associate); Michelle Scerbo (research associate); Zerremi Caga-Anan, MD (research associate).

University of Texas Health Science Center at San Antonio: John G. Myers, MD (co-principal investigator); Ronald M. Stewart, MD (co-principal investigator); Rick L. Sambucini, RN, BS (study coordinator); Marianne Gildea, RN, BSN, MS (study coordinator); Mark DeRosa CRT (study coordinator); Rachelle Jonas, RN, BSN (study coordinator); Janet McCarthy, RN (study coordinator).

University of Texas Southwestern Medical Center: Herbert A. Phelan, MD, MSCS (principal investigator); Joseph P. Minei, MD (co-investigator); Elizabeth Carroll, BS, BA (study coordinator).

University of Washington: Eileen M. Bulger, MD (principal investigator); Patricia Klotz, RN (study coordinator); Keir J. Warner, BS (research coordinator).