http://orcid.org/0000-0002-3252-8972
Figures
Abstract
Background
Acute kidney injury (AKI) is a common complication after surgery that is associated with increased morbidity and mortality. The majority of existing perioperative AKI risk prediction models are limited in their generalizability and do not fully utilize intraoperative physiological time-series data. Thus, there is a need for intelligent, accurate, and robust systems to leverage new information as it becomes available to predict the risk of developing postoperative AKI.
Methods
A retrospective single-center cohort of 2,911 adults who underwent surgery at the University of Florida Health between 2000 and 2010 was utilized for this study. Machine learning and statistical analysis techniques were used to develop perioperative models to predict the risk of developing AKI during the first three days after surgery, first seven days after surgery, and overall (after surgery during the index hospitalization). The improvement in risk prediction was examined by incorporating intraoperative physiological time-series variables. Our proposed model enriched a preoperative model that produced a probabilistic AKI risk score by integrating intraoperative statistical features through a machine learning stacking approach inside a random forest classifier. Model performance was evaluated using the area under the receiver operating characteristic curve (AUC), accuracy, and Net Reclassification Improvement (NRI).
Results
The predictive performance of the proposed model is better than the preoperative data only model. The proposed model had an AUC of 0.86 (accuracy of 0.78) for the seven-day AKI outcome, while the preoperative model had an AUC of 0.84 (accuracy of 0.76). Furthermore, by integrating intraoperative features, the algorithm was able to reclassify 40% of the false negative patients from the preoperative model. The NRI for each outcome was AKI at three days (8%), seven days (7%), and overall (4%).
Citation: Adhikari L, Ozrazgat-Baslanti T, Ruppert M, Madushani RWMA, Paliwal S, Hashemighouchani H, et al. (2019) Improved predictive models for acute kidney injury with IDEA: Intraoperative Data Embedded Analytics. PLoS ONE 14(4): e0214904. https://doi.org/10.1371/journal.pone.0214904
Editor: Gaetano Valenza, Universita degli Studi di Pisa, ITALY
Received: May 29, 2018; Accepted: March 18, 2019; Published: April 4, 2019
Copyright: © 2019 Adhikari et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of presence of identifiable personal information (date and time of services and tests). Data are available from the University of Florida Institutional Data Access / Ethics Committee (contact via Gigi Lipori pflugg@shands.ufl.edu for researchers who meet the criteria for access to confidential data.
Funding: This work was supported by the National Institute of General Medical Sciences [grant numbers R01 GM110240, P50 GM-111152]; the Clinical and Translational Science Institute, University of Florida [grant number 97071]; the Gatorade grant 127900 from University of Florida; and the NIH/NCATS Clinical and Translational Sciences Award [UL1 TR000064]. Part of the cost for data acquisition for this study was supported by I Heermann Anesthesia Foundation, Inc and Vision Grant by Society of Critical Care Medicine.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Acute kidney injury (AKI) is one of the most common, yet underdiagnosed, postoperative complications with lasting consequences [1, 2]. It is associated with an increase in mortality, short- and long-term morbidity, chronic kidney disease, and cardiovascular disease [3–7]. An episode of postoperative AKI imposes an average hospital cost increase of $9000, even after adjusting for all other complications [8, 9]. The implementation of existing clinical guidelines for prevention and treatment of AKI is often hindered by the inability to accurately and timely assess the risk for AKI while accounting for the dynamic nature of the pathophysiological events during surgery.
With the advancement of digitalization in clinical medicine and the widespread availability of electronic health records (EHR), a number of predictive models have been developed to estimate the risk for AKI in different clinical settings, including after surgery [10, 11]. A majority of the existing AKI risk models are limited to preoperative factors [12], applicable only to a specific surgery type [13, 14]. Some of the available online prognostic calculators [15] are designed for intensive care unit (ICU) patients only, and do not take any surgical features into account. While preoperative models for AKI mainly rely on patients’ pre-existing health conditions and general risks associated with the type of surgical procedure, there is a wealth of intraoperative data reflecting the acute physiological responses to the stresses of surgery that is being ignored. Although recent studies have demonstrated an association between AKI and low intraoperative hemoglobin and hypotension [7, 16, 17], there is still a lack of comprehensive postoperative AKI prediction models that dynamically integrate physiologic intraoperative data with preoperative information.
Thus, there is a need for intelligent, accurate, and robust systems that are able to leverage temporal information related to patients’ physiological changes during surgery. We have recently developed and validated a machine learning algorithm, MySurgeryRisk, which predicts preoperative risk for major postoperative complications, including AKI, using EHR data [18]. The aim of this study was to develop and validate a dynamic machine-learning algorithm that readjusts the preoperative risk for AKI, using physiological time series and other data collected during surgery, to provide a personalized risk panel for AKI with both preoperative and immediate postoperative risk assessments.
Materials and methods
This study was approved by the University of Florida Institutional Review Board and Privacy Office as an exempt study with a waiver of informed consent. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) recommendations were followed under the Type 2a analysis category (random split sample development and validation) (S1 Table) [19].
Data source
The University of Florida Integrated Data Repository was used as an honest broker to assemble a single center longitudinal perioperative cohort for all patients admitted to the University of Florida Health for longer than 24 hours following any type of operative procedure between January 1, 2000 and November 30, 2010 by integrating electronic health records with other clinical, administrative, and public databases as previously described [12]. The resulting dataset included detailed information on patient demographics, diagnoses, procedures, outcomes, comprehensive hospital charges, hospital characteristics, insurance status, laboratory, pharmacy, and blood bank data as well as detailed intraoperative physiologic and monitoring data for the cohort.
Participants
We identified patients 18 years of age or older that were admitted to the hospital for longer than 24 hours following any type of inpatient operative procedure. If patients underwent multiple surgeries, only the first surgery was used in our analysis. Patients with end stage renal disease prior to admission (n = 1,935) and patients with missing serum creatinine values during hospitalization (n = 6,636) were excluded from our analysis. From the remaining cohort there were 2,911 patients who had complete intraoperative data for all vital signs, laboratory values, and medications.
Outcomes
The main outcome was the development of postoperative AKI within the first seven days after surgery. The secondary analysis modeled a) the risk for the development of AKI within the first three days after surgery, and b) the risk of developing postoperative AKI at any point during the hospitalization for the index surgery. AKI was defined using the consensus Kidney Disease: Improving Global Outcomes (KDIGO) criteria as at least a 50% or 0.3 mg/dl increase in serum creatinine relative to the reference creatinine [20]. Reference creatinine was determined based on the availability of measured creatinine prior to admission. The minimum serum creatinine value was used if results were available within seven days of the index hospitalization. If not available, the median serum creatinine value obtained within 8–365 days prior to admission was used. For patients without a prior creatinine value within the year prior to admission and no history of chronic kidney disease, an estimated reference serum creatinine was used [21–23]. The estimated reference serum creatinine was calculated by solving the abbreviated “Modification of Diet in Renal Disease” equation for creatinine, assuming a glomerular filtration rate of 75 ml/minute/1.73 m2. After the first seven days of the index hospitalization, the minimum serum creatinine from the preceding seven days was used as the reference creatinine [12, 24]. Patients with chronic kidney disease and end stage renal disease prior to admission were identified using the validated combination of The International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes [25, 26]. Exact dates were used to calculate the duration of mechanical ventilation and intensive care unit stay. A set of previously described criteria was used to annotate the remaining clinical outcomes [8].
Predictor features
The preoperative risk assessment used demographic, socio-economic, administrative, clinical, pharmacy, and laboratory data available prior to surgery to derive 285 preoperative predictor features from 69 preoperative variables (S2 Table). Preoperative comorbidities were derived using up to 50 ICD-9-CM codes as binary variables and the Charlson Comorbidity Index [7, 18, 27, 28]. A reference estimated glomerular filtration rate (eGFR) was calculated for all patients using the reference serum creatinine, sex, race, and age [29]. The immediate postoperative risk reassessment used the following intraoperative variables: five physiologic time series (mean arterial blood pressure (MAP), systolic blood pressure, diastolic blood pressure, minimum alveolar concentration (MAC) of inhaled anesthetics, and heart rate (HR)), 21 repeated laboratory measures, and other discrete variables (intraoperative medications, duration of the operation, anesthesia type, etc.) (S2 Table).
Sample size
Two thousand nine hundred eleven (2,911) patients were included in the cohort. The data was randomly split into 70% for training and 30% for validation. The algorithm was trained on the development cohort while results were reported from the validation cohort. By using 30% of the cohort for validation (n = 873), the overall sample size allows for a maximum width of the 95% confidence interval for the area under the receiver operating characteristic curve (AUC) of 0.08, when prevalence of AKI is between 30% and 40%.
Predictive analytics workflow
The proposed IDEA (Intraoperative Data Embedded Analytics) algorithm is conceptualized as a dynamic model that readjusts the preoperative risk for AKI using physiological time series and other data collected during surgery. The resulting adjusted postoperative risk is assessed immediately at the end of surgery. This flow simulates the clinical task faced by physicians involved in perioperative care where patients’ preoperative information is subsequently enriched by the influx of new data from the operating room. The final output produces IDEA, a personalized risk panel for AKI after surgery (Fig 1) with both preoperative and immediate postoperative risk assessments. The IDEA algorithm consists of two main layers, preoperative and intraoperative, (Fig 2) each containing two cores, data transformer and data analytics.
Phase I, all available health care data from the electronic health record and other public datasets is fed into the preoperative model. The preoperative model calculates a risk score for postoperative acute kidney injury (AKI) and shares the risk score with the clinical team before the surgery. Phase II, the preoperative data is combined with intraoperative data and fed into the IDEA algorithm which calculates a risk score for postoperative acute kidney injury and shares the risk score with the clinical team after surgery.
This diagram shows the aggregation of the data transformer, data engineering, and data analytics modules in the preoperative and intraoperative layers. The two layers can be integrated by either (1) stacking the preoperative prediction scores with the cleaned and feature engineered intraoperative data (blue arrow) or (2) obtaining the full perioperative dataset by merging all the clean features from both layers (orange arrow).
In the data transformer core, the algorithm transforms data from its native format into new complex variables that are optimized for use in predictive models (S2 Table). Data preprocessing was done using a set of automated rules to remove errors and outliers. Time series variables in the intraoperative layer were truncated to match the corresponding surgery start and stop times for each patient. Extreme values in the physiological time series data, defined as values outside the allowable range using expert opinion and medical knowledge [30], were replaced by the average of their five nearest neighbors. Outlier detection and removal was performed on laboratory results by replacing the top and bottom 1% of data using random uniform values generated from the 95%-99.5% and 0.5%-5% percentiles, respectively. Missing nominal variables were replaced with a distinct ‘‘missing” category, whereas missing continuous variables were replaced by the median value for a given variable [18, 31]. Categorical and nominal variables, with more than two levels, were further optimized by calculating conditional probabilities for a particular variable value (such as each surgeon’s ID number or each zip code in the dataset) to be associated with the occurrence of the complication using a separate dataset. The probabilities were calculated as the log of the ratio of the prevalence of a particular variable value among cases with one or more complications to cases without complications [28]. Surgical procedure codes were optimized using a forest of trees approach to reduce the 4-digit primary procedure ICD-9-CM codes that correspond to the anatomical location of surgery. Each node represents a group of procedures, with roots representing the most general groups of procedures and leaf nodes representing the specific procedures [28]. This grouping method reduces the number of discrete procedure codes from 318 to 174 and improves the analysis of low frequency procedures (S1 Methods). Statistical features were derived from the physiological time series data, which included minimum, maximum, mean, short- and long-term variability [32], time spent between different value ranges (determined by the number of standard deviations away from the mean value), and the percentage of time spent in each of the previously mentioned value ranges. Short- and long-term variability were computed using the base and residual signals [32]. The base signal represents the smoothed out version of the time series computed using a convolution filter, while the residual signal represents the difference between the original signal and base signal. The long-term variability is the standard deviation of the base signal and the short-term variability is the standard deviation of the residual signal. For repeated laboratory measurements, the percentage of abnormal values (the percentage of values outside the normal ranges from the Logical Observation Identifiers Names and Codes (LOINC) data table [https://loinc.org/downloads/]), value counts (number of laboratory results obtained during the surgery), and variances were derived (S1 Methods).
In the data analytics core, the IDEA algorithm was trained to calculate patient-level preoperative and immediate postoperative risk probabilities for AKI. In the first stage, the algorithm was trained to calculate preoperative risk probability for AKI using preoperative data only. Subsequently, the intraoperative data available at the end of surgery was used as an additional input for the algorithm to recalculate the postoperative risk probability for AKI. The preoperative risk probability was calculated using a generalized additive model (GAM) with logistic link function [18, 28, 31]. All models were adjusted for non-linearity of covariates using nonlinear risk functions estimated with thin plate regression splines [18, 28, 33]. The best GAM model was picked using a five-fold cross validation technique on 70% of the randomly selected training data cohort and the preoperative prediction scores were generated as the output.
Intraoperative Data Embedded Analytics (IDEA), employing a random forest classifier, was used to enrich the preoperative risk model with intraoperative data. Two approaches were compared, one where risk probability output from the GAM model was combined with the intraoperative data and used as input for the random forest (stacked model), and the other where all preoperative data was embedded with the intraoperative data as input for the random forest (full model). All intraoperative statistical features, along with the preoperative prediction scores, underwent a univariate analysis and only statistically significant (based on the F-test statistic) features were considered for the random forest model [34]. The feature selection and other hyper parameters in scikit-learn [35] random forest classifier (i.e., number of trees, maximum features for the best split, minimum number of samples required to be at a leaf node) were tuned simultaneously using a grid search technique with five-fold cross validation on the same 70% training data cohort (S1 Fig).
Model validation
All of the models were validated using the 30% validation cohort of 873 patients (S1 Fig). The results were reported using 1000 nonparametric bootstrap replicates generated from the R boot package [36]. Using the prediction results obtained from the 1000 bootstrap cohorts, nonparametric confidence intervals for each of the performance metrics were calculated.
Model performance
Each model’s discrimination was assessed using the area under receiver operating characteristic curve (AUC) and model accuracy by determining the fraction of correct classifications for each model. Stratification into high- and low-risk groups was done by calculating the optimal cut-off point based on the maximum Youden Index [37] computed during the training process for each outcome. Using the optimal thresholds for risk probabilities, a classification table was built from which sensitivity, specificity, and positive and negative predictive values were calculated for each model. Absolute risk was calculated as the percentage of cases for which acute kidney injury (AKI) occurred in low- and high-risk groups, respectively. Relative risk was calculated as the ratio of the absolute risk of AKI between high- and low-risk groups. The absolute risk was calculated for high- and low-risk groups for all three models and were compared using the R package “DTComPair” adjusting for multiple comparisons using the Bonferroni method. The Net Reclassification Improvement (NRI) index [38] was used to quantify how well the postoperative model reclassifies AKI patients compared to the preoperative model. Model calibration was tested using the Hosmer-Lemeshow statistic. Bootstrap sampling and nonparametric methods were used to obtain 95% confidence intervals for all performance measures. All analyses were performed using Python 2.7 [39], SciPy 1.0.0 [40], R 3.4 [41], and SAS 9.4 (Cary, NC) software.
Results
Participant baseline characteristics and outcomes
Among 2,911 patients who underwent inpatient surgery requiring a hospital admission of at least 24 hours in a quaternary-care academic center, 1,339 (46%) developed postoperative AKI prior to discharge. Of the patients that developed postoperative AKI, 1,163 (87%) had AKI onset within seven days of surgery. Only 176 (13%) of the AKI patients had late AKI onset (more than seven days after surgery), while 75% of the AKI patients had AKI onset within the first 72 hours after surgery (S3 Table).
The cohort included data from 129 surgeons with an average of 149 procedures per surgeon (Table 1). The acuity of the patient population was high, as 46% of surgeries were categorized as either non-elective or were associated with emergent or urgent hospital admission. Among the cohort, 62% of patients required ICU admission > 48 hours. The cohort had a median ICU length of stay of four days (25th-75th percentiles two-nine days) and a median hospital length of stay of eleven days (25th-75th percentiles seven-twenty days). The overall mortality was 4.8% at thirty days and 16% at one year after index admission. A wide range of comorbidities was documented on admission with cancer and diabetes mellitus being most prevalent. One fourth of the patients were from rural areas and 12% of the patients resided in neighborhoods with a household income below the poverty level [42]. The prevalence of examined complications ranged from 3% for wound complication to 62% for intensive care unit admission > 48 hours (Table 1 and S3 Table). Acute kidney injury, admission to ICU for > 48 hours, and mechanical ventilation for > 48 hours were the most common complications among all surgeries. The distribution of outcomes and preoperative clinical characteristics did not differ between training and validation cohorts.
Intraoperative physiological time series and risk of acute kidney injury
Patients with AKI had more profound and persistent intraoperative hemodynamic changes. A comparison of statistical variables extracted from intraoperative physiological time series data showed that patients with AKI had significantly higher maximum values, greater short- and long-term variability, lower minimum values, and lower average base signals for systolic, diastolic, and mean arterial pressure (MAP) (Table 2). Similar trends were observed for heart rate, except for average base signal, which was higher in patients with AKI. The only significant variables for minimum alveolar concentration were maximum value and long-term variability, which were higher in AKI patients. There were similar patterns for the secondary outcomes with AKI onset during the first three days after surgery and postoperative AKI onset during any point of the hospitalization for the index surgery (S3 Table).
Patients with AKI had greater variation in physiological parameters measured during surgery (Fig 3).
Intraoperative physiological time series variations for 100 randomly selected patients during the first 200 minutes of surgery stratified by the occurrence of acute kidney injury (AKI) within the first seven days after surgery. Mean and 95% Confidence Interval (CI) stratified by the occurrence of AKI are shown for (A) intraoperative mean arterial blood pressure, MAP (mmHg), (B) intraoperative heart rate, HR (beats/minute), and (C) intraoperative mean alveolar concentration of anesthetics, MAC.
The severity and duration of intraoperative hypotension was directly correlated with the risk of developing postoperative AKI (Fig 4). Patients with prolonged periods of blood pressure within the low range of normal values, yet above the traditional threshold for hypotensive treatment (60–65 mmHg), still carried a higher risk for AKI (Fig 4A). Similarly, persistently elevated heart rate was associated with the risk of AKI (Fig 4D).
The risk for the development of postoperative acute kidney injury (AKI) within the first seven days after surgery, as predicted by the postoperative stacked model, was aggregated across the entire cohort. The color represents the risk of developing postoperative AKI, where red is high-risk, and green is low-risk. The y-axis is time (minutes) and the x-axes are (A) diastolic blood pressure (mmHg), (B) systolic blood pressure (mmHg), (C) Mean arterial blood pressure (mmHg), and (D) Heart rate (beats/minute).
Other intraoperative variables and risk of acute kidney injury
The following intraoperative arterial blood gas panel measurements were found to be significantly higher in patients that developed AKI and were correlated with increased risk of developing AKI: partial pressure of carbon dioxide, percentage of carboxyhemoglobin, percentage of methemoglobin, and variance in bicarbonate. Additionally, arterial oxygen content was found to be significantly lower in AKI patients (Table 2 and S3 Table). Among other intraoperative laboratory tests, white blood cell count, red cell distribution width, and lactic acid were significantly higher in patients who developed AKI, while platelet count, red blood cell count, hemoglobin, and hematocrit were significantly lower in AKI patients (Table 2 and S3 Table). The correlation between important features and AKI risk probability was not linear (Fig 5). The total amount of blood products administered during surgery was significantly associated with progressive increases in AKI risk probability (Fig 5G). Patients who developed AKI were more likely to receive diuretics (18% vs 7%) and vasopressors (74% vs 62%) during surgery and were administered less intravenous fluid intraoperatively (Table 2 and S3 Table). The duration of surgery was significantly higher in patients with AKI and was highly correlated with the risk for AKI (Fig 5H). Interestingly, patients with AKI were more likely to have their operation performed between 7 pm and 7 am. Similar trends were observed for the secondary outcomes: AKI with the onset in the first three days of surgery and postoperative AKI occurring at any time during the hospitalization for the index surgery (S3 Table).
Each blue dot represents a patient that developed postoperative AKI. The y-axis represents the risk probability for postoperative acute kidney injury, which ranges from 0 to 1. The red line represents the median acute kidney injury risk score given the values of x. The x-axes represent (A) maximum intraoperative percentage of methemoglobin in arterial blood gasses, (B) intraoperative variance of bicarbonate (mmol/L)2 in arterial blood gasses, (C) intraoperative mean of lactic acid (mmol/L), (D) minimum intraoperative platelet count (thou/mm3), (E) intraoperative variance of mean corpuscular hemoglobin, MCH (pg/cell)2, (F) intraoperative mean of red cell distribution (%), (G) total of blood products administered during surgery (mL), and (H) duration of surgery (minutes).
Risk score stratification and model performance
The IDEA algorithm calculated the risk for developing AKI (ranging from 0 to 1) at two distinct time points: once preoperatively using only preoperative data and once immediately after surgery after enriching the preoperative data with physiological responses to surgery and intraoperative events. The algorithm automatically determined the optimal threshold for stratifying patients into low- and high-risk groups (Fig 2 and S2 Fig). The predictive performance for all three models in the validation dataset was very good (Fig 6) and both of the postoperative models (stacked model AUC 0.86, 95% CI 0.84–0.89 and full model AUC 0.87, 95% CI 0.85–0.90) performed better than the preoperative model (AUC 0.84, 95% CI 0.82–0.87). The sensitivity increased from 0.68 (95% CI 0.64–0.73) in the preoperative model to 0.81 (95% CI 0.76–0.84) in the full postoperative model. Even though the positive predictive values were in the same range for the preoperative and the postoperative models (0.68 to 0.78), the negative predictive value of the postoperative models (0.85, 95% CI 0.82–0.88) was significantly improved compared to the preoperative model (0.79, 95% CI 0.75–0.82). The data transformation step required 48 hours, training the postoperative full model required 20 hours, and training the postoperative stacked model required 8 hours. The data transformation and training were done on an Ubuntu PC with an Intel Xeon 3.7GHz Processor with 8 cores and 32GB RAM. The predictive performance for the two secondary outcomes (AKI onset within first three postoperative days and AKI onset before discharge) were similar and demonstrated the same trends as the primary outcome postoperative models, with AUC values ranging between 0.82 and 0.88 (S4 Table).
The graph at the top of the figure is the receiver operating characteristic curves for prediction of postoperative acute kidney injury (AKI) within the first seven days after surgery for the preoperative model (pink), the postoperative stacked model (green), and the postoperative full model (blue). The diamonds on each curve represent the cutoffs calculated by maximizing the Youden Index. The y-axis is sensitivity, which ranges from zero to one. The x-axis is one minus specificity, which ranges from zero to one. The table in the bottom of the figure contains the following performance metrics for the preoperative and both postoperative models: (A) accuracy (percentage of patients correctly classified), (B) sensitivity (percentage of patients that developed AKI that were classified as high-risk), (C) positive predictive value (percentage of high-risk patients that developed AKI), (D) negative predictive value (percentage of low-risk patients that did not develop AKI), (E) net reclassification improvement for patients that developed postoperative AKI (F) net reclassification improvement for patients that did not develop postoperative AKI, (H) overall net reclassification improvement.
Most significantly, according to the feature importance scores from the trained postoperative random forest models (S5 Table), only three of the top ten important features were preoperative variables (zip code, chronic kidney disease, and attending surgeon) while all other important features were derived from the following intraoperative variables: lactic acid, total blood products administered during surgery, red cell distribution width, and diastolic blood pressure (S5 Table).
Reclassification of risk groups with intraoperative data
The observed absolute risk (the percentage of patients in a risk group that developed AKI within the first seven days after surgery) was distinctly different between the low- and high-risk groups for the preoperative model and both postoperative models (Table 3). The relative risk (the ratio of the absolute risk between the high- and low-risk groups) was significantly higher for the postoperative stacked and full models, 4.6 (95% CI 3.7–5.7) and 5.1 (95% CI 4.1–6.4) respectively, compared to the preoperative model, 3.4 (95%CI 2.8–4.0).
To assess the incremental improvement of classifying patients into the correct risk groups by the addition of intraoperative data, the net reclassification improvement (NRI) was calculated for both the stacked and full postoperative models. The calculated NRI (net percentage of correctly reclassified cases after addition of the intraoperative data) showed a statistically significant improvement for both the stacked and the full postoperative models with 7% (95% CI (3%, 12%), p < 0.005) and 11% (95% CI (5%, 16%), p < 0.0005) of cases correctly reclassified, respectively (Fig 6). The NRI for patients that developed postoperative AKI within seven days after surgery was approximately 12% (95% CI (8%, 15%), p <0.0001) for both postoperative models, while the NRI for patients that did not develop postoperative AKI was -4% (95% CI (-4%, -7%), p<0.001) and -2% (95% CI (-5%, 1%), p>0.5) for the stacked and full models, respectively. Both postoperative models were able to reclassify approximately 40% of the false negative patients from the preoperative model correctly into the high-risk group (Fig 7). This demonstrates that intraoperative data can be used as a significant predictor to correctly assess patients’ increased risk for postoperative AKI within seven days of surgery (compared to preoperative risk). However, the addition of intraoperative data is unlikely to accurately lower patients’ risk for the development of postoperative AKI within seven days of surgery. The secondary outcomes, postoperative AKI onset within three days of surgery and postoperative AKI onset before discharge, also showed statistically significant NRIs comparing the postoperative models to the preoperative model. However, for the outcome predicting development of postoperative AKI within three days of surgery, both postoperative models were more effective at reclassifying false positive patients from the preoperative model compared to false negative patients (S3 Fig and S4 Table).
The postoperative risk scores in the first column are from the stacked model, while the postoperative risk scores for the second column are from the full model. The patients in the first row developed postoperative AKI within seven days of surgery, whereas the patients in the second row did not develop postoperative AKI within seven days of surgery. The y-axis on each plot is the postoperative model acute kidney injury risk score, which ranges from zero to one. The x-axis on each plot is the preoperative risk group. The red dots are patients at high-risk for AKI according to the postoperative model, whereas the green dots are patients at low-risk for AKI according to the postoperative model. (A & B) The proposed postoperative stacked and full models effectively reclassified false negative patients from the preoperative model as high AKI risk patients.
Furthermore, the patients in the low preoperative risk group for postoperative AKI who developed AKI within seven days of surgery that were correctly reclassified into the high-risk group by the postoperative models, had lower intraoperative mean arterial blood pressure values and higher intraoperative blood product volumes compared to those who remained in the low-risk group (Fig 8). Conversely, the patients in the high preoperative risk group for postoperative AKI who did not develop AKI within seven days of surgery that were correctly reclassified into the low-risk group by the postoperative models, had higher intraoperative mean blood pressure values and lower intraoperative blood product volumes compared to those who remained in the high-risk group.
The first column represents patients that developed postoperative acute kidney injury (AKI), yet were classified as low-risk by the preoperative model. The second column represents patients that did not develop postoperative AKI, but were classified as high-risk by the preoperative model. The boxes represent 95% confidence intervals for the given variables. (A) Patients correctly classified as having high-risk for AKI by the postoperative stacked model tended to have lower mean arterial pressure. (B) Patients correctly classified as having low-risk for AKI by the postoperative stacked model tended to have higher mean arterial pressure. (C) Patients correctly classified as having high-risk for AKI by the postoperative stacked model tended to have greater volumes of administered intraoperative blood products. (D) Patients correctly classified as having low-risk for AKI by the postoperative stacked model tended to have lesser volumes of administered intraoperative blood products.
Discussion
Using a large single-center cohort of surgical patients, we developed and validated a dynamic machine-learning algorithm that readjusts the preoperative risk for postoperative AKI using physiological time series data and other data collected during surgery to provide a personalized risk panel for acute kidney injury with both preoperative and immediate postoperative risk assessments. This work expands on our previously validated MySurgeryRisk algorithm which predicts preoperative risk for major postoperative complications, including AKI [18], to leverage temporal enrichment of the preoperative model with the new information related to patients’ changes in physiological status during surgery. The advantages of the algorithm include a) prediction entirely based on routinely available preoperative and intraoperative data, b) universal applicability to any surgical context, c) exportability to other EHR systems, and d) the ability to handle any data type in EHR (including time series and sparse data). Most importantly, the dynamic reassessment of the risk for postoperative AKI using temporal enrichment with intraoperative data allows for more precise reclassification of AKI risk based on how patients’ clinical trajectory progresses. While preoperative models mainly asses risk based on patients’ pre-existing health conditions and general risks associated with the type of planned procedures, the addition of intraoperative time series data reflects acute physiological responses to the stresses of surgery, which provides a better risk assessment for postoperative AKI to a physician who would want to use the model. For example, a patient without significant comorbidities who undergoes a moderate risk surgery would have low-risk for postoperative AKI based on a preoperative model. If that patient was to develop one or more complications during surgery, such as severe bleeding, adverse reaction to anesthetics, or treatment with nephrotoxic drugs, his/her physiological responses captured in the intraoperative data would reclassify him/her to the high-risk group. A change in classification such as this would be extremely valuable for a physician. The IDEA algorithm demonstrated the ability to integrate intraoperative data that not only resulted in an improved AUC compared to the preoperative model, but resulted in effectively reclassifying up to 40% of patients from the preoperative model into a new risk category based on intraoperative events.
While several preoperative factors, such as age, comorbidity, chronic kidney disease, and type of surgery have been identified as risk factors for AKI based on previous studies [43], the recognition that certain types of admission medications, or even the timing of the operation itself, are risk factors for AKI confirms some recent observations [44–46]. Furthermore, the dynamic changes in intraoperative blood pressure and heart rate were among the most important features contributing to the risk of developing postoperative AKI in our model. This indicates that the duration and magnitude of intraoperative hypotension as well as concomitant changes in heart rate significantly increase the risk for postoperative AKI. Intraoperative hypotension is a well-recognized risk factor for multiple postoperative complications [47–49], with biological plausibility that is well accepted among anesthesiologists and surgeons, yet no consensus exists regarding the optimal blood pressure target to support the perfusion of critical organs during surgery [50, 51]. The red cell distribution width, platelet count, and the duration of operation were among the most important intraoperative predictors for postoperative AKI, and while some of them have been previously reported as risk factors for AKI in other patient populations, this is the first report for surgical patients [52–54].
Most of the existing risk models for acute kidney injury are focused on the general hospital population [10, 11] or are designed for critically ill patients [15]. Many of the previous risk models for surgical AKI were limited to either a specific type of surgery or the use of preoperative risk factors for statistical modeling (mainly logistic regression) with reported AUCs between 0.77 and 0.84 [1, 2, 55, 56]. The risk prediction models that have incorporated intraoperative variables are scarce and mainly focused on patients undergoing cardiac surgery with reported AUCs ranging between 0.72 and 0.81 [13, 14, 57]. The fact that these models did not fully utilize the available time-varying physiological data during the surgery, or machine learning approach, may have contributed to their lower performance compared to our model.
This study has some limitations. Firstly, only the first surgery was used for patients that underwent multiple surgeries for building our proposed predictive models. Secondly, the algorithm was only provided with the data and outcomes in the training dataset, without explicit definitions of features. Thirdly, because the algorithm ‘‘learned” the features that were most predictive for the risk of developing postoperative AKI implicitly, it is possible that the algorithm is using features previously unknown to or ignored by physicians. Fourthly, the expansion of input features to include operative notes may increase the accuracy, but will require more elaborate computational approaches. Lastly, the algorithm has been trained to capture practice patterns for individual providers in the referral population of a large academic medical center in North Central Florida. Further training and validation of the algorithm is necessary in a dataset with different population characteristics and practice patterns.
Conclusions
In a large single-center cohort of surgical patients, our proposed Intraoperative Data Embedded Analytics (IDEA) algorithm employed a machine learning approach based on a random forest classifier to improve patients’ postoperative acute kidney injury (AKI) risk prediction with high sensitivity and specificity by utilizing intraoperative data. The IDEA algorithm was able to correctly reclassify approximately 40% of patients who were considered low-risk for postoperative AKI by preoperative model to high-risk. This illustrates the importance of intraoperative data in AKI risk stratification. Given the association between AKI and increased morbidity, mortality, and cost, it is important for clinicians to have dynamic AKI risk prediction algorithms capable of adjusting AKI risk as new information becomes available. Further research can address other post-surgical complications as well as validation of the proposed algorithm on external datasets.
Supporting information
S2 Table. Characteristics of input variables.
https://doi.org/10.1371/journal.pone.0214904.s003
(DOCX)
S3 Table. Summary of clinical characteristics of the cohort and outcomes stratified by each of the acute kidney injury outcomes.
https://doi.org/10.1371/journal.pone.0214904.s004
(DOCX)
S4 Table. Model performances for the secondary acute kidney injury outcomes.
https://doi.org/10.1371/journal.pone.0214904.s005
(DOCX)
S5 Table. Important features from the postoperative models.
https://doi.org/10.1371/journal.pone.0214904.s006
(DOCX)
S1 Fig. Model training flow diagram.
The cohort of size 2,911 with 231 features was randomly split into training (70%) and testing (30%) cohorts. A random forest classifier was used to train the AKI prediction model (we used 5-fold cross validation for hyperparameter tuning and feature selection) for all three outcomes separately and performance was tested using the testing (validation) cohort.
https://doi.org/10.1371/journal.pone.0214904.s007
(TIF)
S2 Fig. Performance plots for the postoperative stacked model.
The first column shows the optimization of the cutoff threshold by maximizing the Youden Index. The second column shows the relationship between the performance metrics (accuracy, positive predictive value, negative predictive value, and Youden Index) and threshold. (A & B) The first row is for the prediction of postoperative acute kidney Injury (AKI) within three days of surgery outcome. (C & D) The second row is for the prediction of postoperative AKI within seven days of surgery outcome. (E & F) The third row is for the prediction of postoperative AKI prior to discharge outcome.
https://doi.org/10.1371/journal.pone.0214904.s008
(TIF)
S3 Fig. Reclassification performance for the secondary outcomes.
The first row represents the reclassification of (A) patients that developed postoperative acute kidney injury (AKI) within three days of surgery and (B) patients that did not develop postoperative AKI within three days of surgery for the three day secondary outcome. The second row represents the reclassification of (C) patients that developed postoperative AKI before discharge and (D) patients that did not develop postoperative AKI before discharge for the overall secondary outcome.
https://doi.org/10.1371/journal.pone.0214904.s009
(TIF)
Acknowledgments
We would like to acknowledge Shivam Mittal, MS, Swati Sisodia, MS, Meixian Wang, MS, Paul Thottakkara, MS, Amir Motaei, PHD, Ashkan Ebadi, PHD, for the work done on data cleaning and coding, and George Omalay for project management. The authors Lasith Adhikari and Tezcan Ozrazgat-Baslanti contributed equally to this manuscript.
References
- 1. Hobson C, Singhania G, Bihorac A. Acute Kidney Injury in the Surgical Patient. Crit Care Clin. 2015;31(4):705–23. Epub 2015/09/28. pmid:26410139; PubMed Central PMCID: PMCPMC4584402.
- 2. Hobson C, Lysak N, Huber M, Scali S, Bihorac A. Epidemiology, outcomes, and management of acute kidney injury in the vascular surgery patient. J Vasc Surg. 2018;68(3):916–28. Epub 2018/08/28. pmid:30146038; PubMed Central PMCID: PMCPMC6236681.
- 3. Bihorac A, Yavas S, Subbiah S, Hobson CE, Schold JD, Gabrielli A, et al. Long-term risk of mortality and acute kidney injury during hospitalization after major surgery. Annals of Surgery. 2009;249(5):851–8. Epub 2009/04/24. pmid:19387314.
- 4. Hobson C, Ozrazgat-Baslanti T, Kuxhausen A, Thottakkara P, Efron PA, Moore FA, et al. Cost and Mortality Associated With Postoperative Acute Kidney Injury. Ann Surg. 2015;261(6):1207–14. Epub 2014/06/03. pmid:24887982; PubMed Central PMCID: PMCPMC4247993.
- 5. Huber M, Ozrazgat-Baslanti T, Thottakkara P, Efron PA, Feezor R, Hobson C, et al. Mortality and Cost of Acute and Chronic Kidney Disease after Vascular Surgery. Ann Vasc Surg. 2016;30:72–81 e1-2. pmid:26187703; PubMed Central PMCID: PMCPMC4691411.
- 6. Huber M, Ozrazgat-Baslanti T, Thottakkara P, Scali S, Bihorac A, Hobson C. Cardiovascular-specific mortality and kidney disease in patients undergoing vascular surgery. JAMA surgery. 2016;151(5):441–50. pmid:26720406
- 7. Ozrazgat-Baslanti T, Thottakkara P, Huber M, Berg K, Gravenstein N, Tighe P, et al. Acute and Chronic Kidney Disease and Cardiovascular Mortality After Major Surgery. Annals of surgery. 2016.
- 8. Hobson C, Ozrazgat-Baslanti T, Kuxhausen A, Thottakkara P, Efron PA, Moore FA, et al. Cost and Mortality Associated With Postoperative Acute Kidney Injury. Annals of Surgery. 2015;261(6):1207–14. 00000658-201506000-00027. pmid:24887982
- 9. Lysak N, Bihorac A, Hobson C. Mortality and cost of acute and chronic kidney disease after cardiac surgery. Curr Opin Anaesthesiol. 2017;30(1):113–7. Epub 2016/11/15. pmid:27841788; PubMed Central PMCID: PMCPMC5303614.
- 10. Koyner JL, Adhikari R, Edelson DP, Churpek MM. Development of a Multicenter Ward-Based AKI Prediction Model. Clin J Am Soc Nephrol. 2016;11(11):1935–43. Epub 2016/09/17. pmid:27633727; PubMed Central PMCID: PMCPMC5108182.
- 11. Hodgson LE, Sarnowski A, Roderick PJ, Dimitrov BD, Venn RM, Forni LG. Systematic review of prognostic prediction models for acute kidney injury (AKI) in general hospital populations. BMJ Open. 2017;7(9):e016591. Epub 2017/10/01. pmid:28963291; PubMed Central PMCID: PMCPMC5623486.
- 12. Bihorac A, Brennan M, Ozrazgat-Baslanti T, Bozorgmehri S, Efron PA, Moore FA, et al. National surgical quality improvement program underestimates the risk associated with mild and moderate postoperative acute kidney injury. Crit Care Med. 2013;41(11):2570–83. Epub 2013/08/10. pmid:23928835; PubMed Central PMCID: PMCPMC3812338.
- 13. Birnie K, Verheyden V, Pagano D, Bhabra M, Tilling K, Sterne JA, et al. Predictive models for kidney disease: improving global outcomes (KDIGO) defined acute kidney injury in UK cardiac surgery. Crit Care. 2014;18(6):606. Epub 2015/02/13. pmid:25673427; PubMed Central PMCID: PMCPMC4258283.
- 14. Ng SY, Sanagou M, Wolfe R, Cochrane A, Smith JA, Reid CM. Prediction of acute kidney injury within 30 days of cardiac surgery. J Thorac Cardiovasc Surg. 2014;147(6):1875–83, 83 e1. Epub 2013/09/03. pmid:23993316.
- 15. Flechet M, Guiza F, Schetz M, Wouters P, Vanhorebeek I, Derese I, et al. AKIpredictor, an online prognostic calculator for acute kidney injury in adult critically ill patients: development, validation and comparison to serum neutrophil gelatinase-associated lipocalin. Intensive Care Med. 2017;43(6):764–73. Epub 2017/01/29. pmid:28130688.
- 16. Haase M, Bellomo R, Story D, Letis A, Klemz K, Matalanis G, et al. Effect of mean arterial pressure, haemoglobin and blood transfusion during cardiopulmonary bypass on post-operative acute kidney injury. Nephrol Dial Transplant. 2012;27(1):153–60. Epub 2011/06/17. pmid:21677302.
- 17. Ozrazgat-Baslanti T, Blanc P, Thottakkara P, Ruppert M, Rashidi P, Momcilovic P, et al. Preoperative assessment of the risk for multiple complications after surgery. Surgery. 2016;160(2):463–72. Epub 2016/05/31. pmid:27238354; PubMed Central PMCID: PMCPMC5114020.
- 18. Bihorac A, Ozrazgat-Baslanti T, Ebadi A, Motaei A, Madkour M, Pardalos PM, et al. MySurgeryRisk: Development and Validation of a Machine-learning Risk Algorithm for Major Complications and Death After Surgery. Ann Surg. 2018. Epub 2018/03/01. pmid:29489489; PubMed Central PMCID: PMCPMC6110979.
- 19. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): the TRIPOD statement. Annals of internal medicine. 2015;162(1):55–63. Epub 2015/01/07. pmid:25560714.
- 20. Mehta RL, McDonald B, Gabbai F, Pahl M, Farkas A, Pascual MT, et al. Nephrology consultation in acute renal failure: does timing matter? Am J Med. 2002;113(6):456–61. Epub 2002/11/13. pmid:12427493.
- 21. Kidney Disease: Improving Global Outcomes (KDIGO) Acute Kidney Injury Work Group—KDIGO Clinical Practice Guideline for Acute Kidney Injury. Kidney Int. 2012;suppl (2):1–138.
- 22.
Program UNAKI. Acute Kidney Injury (AKI) Algorithm. 2014.
- 23. Holmes J, Roberts G, Meran S, Williams JD, Phillips AO, Welsh AKISG. Understanding Electronic AKI Alerts: Characterization by Definitional Rules. Kidney Int Rep. 2017;2(3):342–9. pmid:29142963; PubMed Central PMCID: PMCPMC5678680.
- 24. Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P, Acute Dialysis Quality Initiative w. Acute renal failure—definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care. 2004;8(4):R204–12. Epub 2004/08/18. pmid:15312219; PubMed Central PMCID: PMCPMC522841.
- 25. Wald R, Waikar SS, Liangos O, Pereira BJ, Chertow GM, Jaber BL. Acute renal failure after endovascular vs open repair of abdominal aortic aneurysm. J Vasc Surg. 2006;43(3):460–6; discussion 6. Epub 2006/03/08. pmid:16520155.
- 26. Wald R, Quinn RR, Luo J. Chronic dialysis and death among survivors of acute kidney injury requiring dialysis. JAMA. 2009;302:1179–85. pmid:19755696
- 27. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. Journal of Chronic Diseases. 1987;40(5):373–83. Epub 1987/01/01. pmid:3558716.
- 28. Korenkevych D, Ozrazgat-Baslanti T, Thottakkara P, Hobson CE, Pardalos P, Momcilovic P, et al. The Pattern of Longitudinal Change in Serum Creatinine and 90-Day Mortality After Major Surgery. Ann Surg. 2016;263(6):1219–27. Epub 2015/07/17. pmid:26181482; PubMed Central PMCID: PMCPMC4829495.
- 29. Levey AS, Stevens LA, Schmid CH, Zhang YL, Castro AF 3rd, Feldman HI et al. A new equation to estimate glomerular filtration rate. Annals of internal medicine. 2009;150(9):604–12. Epub 2009/05/06. 150/9/604 [pii]. pmid:19414839; PubMed Central PMCID: PMC2763564.
- 30. Shickel B, Loftus TJ, Adhikari L, Ozrazgat-Baslanti T, Ebadi A, Bihorac A, et al. DeepSOFA: A Real-Time Continuous Acuity Score Framework using Deep Learning. arXiv preprint arXiv:180210238. 2018.
- 31. Thottakkara P, Ozrazgat-Baslanti T, Hupf BB, Rashidi P, Pardalos P, Momcilovic P, et al. Application of Machine Learning Techniques to High-Dimensional Clinical Data to Forecast Postoperative Complications. PLoS One. 2016;11(5):e0155705. pmid:27232332; PubMed Central PMCID: PMCPMC4883761.
- 32. Saria S, Rajani AK, Gould J, Koller D, Penn AA. Integration of early physiological responses predicts later illness severity in preterm infants. Sci Transl Med. 2010;2(48):48ra65. Epub 2010/09/10. pmid:20826840; PubMed Central PMCID: PMCPMC3564961.
- 33. Wood SN. Thin plate regression splines. Journal of the Royal Statistical Society Series B-Statistical Methodology. 2003;65:95–114. WOS:000180996800005.
- 34. Breiman L. Random forest. Machine Learning. 2001;45(1):5–32. WOS:000170489900001.
- 35. Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research. 2011;12:2825–30. WOS:000298103200003.
- 36.
Angelo Canty BR. Boot: Functions and datasets for bootstrapping from the book “Bootstrap Methods and Their Application'' by A. C. Davison and D. V. Hinkley (1997, CUP), originally written by Angelo Canty for S. 2017.
- 37. Youden WJ. Index for rating diagnostic tests. Cancer. 1950;3(1):32–5. Epub 1950/01/01. pmid:15405679.
- 38. Pencina MJ, D'Agostino RB Sr., Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30(1):11–21. Epub 2011/01/05. pmid:21204120; PubMed Central PMCID: PMCPMC3341973.
- 39.
Foundation PS. Python Language Reference, version 2.7. Available from: http://www.python.org.
- 40. Jones E OE, Peterson P, et al. SciPy: Open Source Scientific Tools for Python. 2001.
- 41.
team Rc. R: A language and environment for statistical computing. R Foundation for Statistical Computing 2017. Available from: http://www.R-project.org/.
- 42.
United States Census Bureau. American FactFinder 2010 [05/16/2017]. Available from: http://www2.census.gov/.
- 43. Kane-Gill SL, Sileanu FE, Murugan R, Trietley GS, Handler SM, Kellum JA. Risk factors for acute kidney injury in older adults with critical illness: a retrospective cohort study. Am J Kidney Dis. 2015;65(6):860–9. Epub 2014/12/10. pmid:25488106; PubMed Central PMCID: PMCPMC4442750.
- 44. Coca SG, Garg AX, Swaminathan M, Garwood S, Hong K, Thiessen-Philbrook H, et al. Preoperative angiotensin-converting enzyme inhibitors and angiotensin receptor blocker use and acute kidney injury in patients undergoing cardiac surgery. Nephrol Dial Transplant. 2013;28(11):2787–99. pmid:24081864; PubMed Central PMCID: PMCPMC3811062.
- 45. Roshanov PS, Rochwerg B, Patel A, Salehian O, Duceppe E, Belley-Côté EP, et al. Withholding versus Continuing Angiotensin-converting Enzyme Inhibitors or Angiotensin II Receptor Blockers before Noncardiac SurgeryAn Analysis of the Vascular events In noncardiac Surgery patIents cOhort evaluatioN Prospective Cohort. Anesthesiology: The Journal of the American Society of Anesthesiologists. 2017;126(1):16–27.
- 46. Roncero LMV, Poveda DS, García JJV, Barrado MES, Vecino JMC. Perioperative use of angiotensin-converting-enzyme inhibitors and angiotensin receptor antagonists. Journal of Clinical Anesthesia. 2017;40:91–8. pmid:28625460
- 47. Li S, Krawczeski CD, Zappitelli M, Devarajan P, Thiessen-Philbrook H, Coca SG, et al. Incidence, risk factors, and outcomes of acute kidney injury after pediatric cardiac surgery: A prospective multicenter study. Critical Care Medicine. 2011;39(6):1493–9. WOS:000290715000037. pmid:21336114
- 48. Aronson S, Phillips-Bute B, Stafford-Smith M, Fontes M, Gaca J, Mathew JP, et al. The association of postcardiac surgery acute kidney injury with intraoperative systolic blood pressure hypotension. Anesthesiol Res Pract. 2013;2013:174091. pmid:24324489; PubMed Central PMCID: PMCPMC3845409.
- 49. Sun LY, Wijeysundera DN, Tait GA, Beattie WS. Association of intraoperative hypotension with acute kidney injury after elective noncardiac surgery. Anesthesiology. 2015;123(3):515–23. pmid:26181335.
- 50. Bihorac A, Hobson CE. Acute kidney injury: Precision perioperative care protects the kidneys. Nat Rev Nephrol. 2017;14(1):8–10. Epub 2017/12/14. pmid:29234162.
- 51. Futier E, Lefrant JY, Guinot PG, Godet T, Lorne E, Cuvillon P, et al. Effect of Individualized vs Standard Blood Pressure Management Strategies on Postoperative Organ Dysfunction Among High-Risk Patients Undergoing Major Surgery: A Randomized Clinical Trial. JAMA. 2017;318(14):1346–57. Epub 2017/10/04. pmid:28973220; PubMed Central PMCID: PMCPMC5710560.
- 52. Kertai MD, Zhou S, Karhausen JA, Cooter M, Jooste E, White WD, et al. Platelet Counts, Acute Kidney Injury, and Mortality after Coronary Artery Bypass Grafting Surgery Reply. Anesthesiology. 2016;125(2):438–9. WOS:000385022200043. pmid:27433767
- 53. Oh HJ, Park JT, Kim JK, Yoo DE, Kim SJ, Han SH, et al. Red blood cell distribution width is an independent predictor of mortality in acute kidney injury patients treated with continuous renal replacement therapy. Nephrology Dialysis Transplantation. 2012;27(2):589–94. WOS:000300421300020. pmid:21712489
- 54. Borthwick E, Ferguson A. Perioperative acute kidney injury: risk factors, recognition, management, and outcomes. Bmj. 2010;341:c3365. pmid:20603317.
- 55. Huen SC, Parikh CR. Predicting acute kidney injury after cardiac surgery: a systematic review. Ann Thorac Surg. 2012;93(1):337–47. S0003-4975(11)02155-2 [pii] pmid:22186469; PubMed Central PMCID: PMCPMC3286599.
- 56. Hobson C, Ruchi R, Bihorac A. Perioperative Acute Kidney Injury: Risk Factors and Predictive Strategies. Crit Care Clin. 2017;33(2):379–96. pmid:28284301.
- 57. Kashani K, Steuernagle JHt, Akhoundi A, Alsara A, Hanson AC, Kor DJ. Vascular Surgery Kidney Injury Predictive Score: A Historical Cohort Study. J Cardiothorac Vasc Anesth. 2015;29(6):1588–95. Epub 2015/07/15. pmid:26159745.