Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

APACHE III Outcome Prediction in Patients Admitted to the Intensive Care Unit with Sepsis Associated Acute Lung Injury

  • Zhongheng Zhang ,

    Contributed equally to this work with: Zhongheng Zhang, Kun Chen

    zh_zhang1984@hotmail.com

    ‡ Z.Z. and K.C. are co-first authors on this work.

    Affiliation Department of critical care medicine, Jinhua municipal central hospital, Jinhua hospital of Zhejiang university, Zhejiang, P. R. China

    ⨯
  • Kun Chen ,

    Contributed equally to this work with: Zhongheng Zhang, Kun Chen

    ‡ Z.Z. and K.C. are co-first authors on this work.

    Affiliation Department of critical care medicine, Jinhua municipal central hospital, Jinhua hospital of Zhejiang university, Zhejiang, P. R. China

    ⨯
  • Lin Chen

    Affiliation Department of critical care medicine, Jinhua municipal central hospital, Jinhua hospital of Zhejiang university, Zhejiang, P. R. China

    ⨯

Abstract

Background and objective

Acute Physiology and Chronic Health Evaluation (APACHE) III score has been widely used for prediction of clinical outcomes in mixed critically ill patients. However, it has not been validated in patients with sepsis-associated acute lung injury (ALI). The aim of the study was to explore the calibration and predictive value of APACHE III in patients with sepsis-associated ALI.

Method

The study was a secondary analysis of a prospective randomized controlled trial investigating the efficacy of rosuvastatin in sepsis-associated ALI (Statins for Acutely Injured Lungs from Sepsis, SAILS). The study population was sepsis-related ALI patients. The primary outcome of the current study was the same as in the original trial, 60-day in-hospital mortality, defined as death before hospital discharge, censored 60 days after enrollment. Discrimination of APACHE III was assessed by calculating the area under the receiver operating characteristic (ROC) curve (AUC) with its 95% CI. Hosmer-Lemeshow goodness-of-fit statistic was used to assess the calibration of APACHE III. The Brier score was reported to represent the overall performance of APACHE III in predicting outcome.

Main results

A total of 745 patients were included in the study, including 540 survivors and 205 non-survivors. Non-survivors were significantly older than survivors (59.71±16.17 vs 52.00±15.92 years, p<0.001). The primary causes of ALI were also different between survivors and non-survivors (p = 0.017). Survivors were more likely to have the cause of sepsis than non-survivors (21.2% vs. 15.1%). APACHE III score was higher in non-survivors than in survivors (106.72±27.30 vs. 88.42±26.86; p<0.001). Discrimination of APACHE III to predict mortality in ALI patients was moderate with an AUC of 0.68 (95% confidence interval: 0.64–0.73).

Conclusion

this study for the first time validated the discrimination of APACHE III in sepsis associated ALI patients. The result shows that APACHE III score has moderate predictive value for in-hospital mortality among adults with sepsis-associated acute lung injury.

Citation: Zhang Z, Chen K, Chen L (2015) APACHE III Outcome Prediction in Patients Admitted to the Intensive Care Unit with Sepsis Associated Acute Lung Injury. PLoS ONE 10(9): e0139374. https://doi.org/10.1371/journal.pone.0139374

Editor: Lynn M. Schnapp, Medical University of South Carolina, UNITED STATES

Received: April 23, 2015; Accepted: September 11, 2015; Published: September 30, 2015

Copyright: © 2015 Zhang et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited

Data Availability: All data files are available from the NHLBI Biologic Specimen and Data Repository Information Coordinating Center database (https://biolincc.nhlbi.nih.gov/studies/sails/?q=rosuvastatin).

Funding: The authors have no support or funding to report.

Competing interests: The authors have declared that no competing interests exist.

Abbreviations: ICU, Intensive care unit; APACHE, Acute Physiology and Chronic Health Evaluation; SOFA, sequential organ failure score; ALI, acute lung injury; ROC, receiver operating characteristic; NHLBI, National heart, lung, blood institute

Introduction

Intensive care unit (ICU) patients are at high risk of death and many risk stratification scores have been developed for outcome prediction. The most widely used scores include Acute Physiology and Chronic Health Evaluation (APACHE) from I to IV, sequential organ failure score (SOFA) and simplified acute physiological score (SAPS). These risk assessment scores are thought to play an important role in evaluating new therapies, triaging patients, improving quality assessment and monitoring resource utilization. However, these scores are mostly developed in unselected ICU patients. For instance, the APACHE III score was derived from 17,440 unselected medical/surgical ICU adult patients [1]. Although the APACHE III score has been well validated in unselected ICU patients [2], it has been found to be less reliable in specific subgroups of critically ill patients. For example, APACHE III predicted moderately between survivors and non-survivors of patients after orthotopic liver transplantation [3], with an area under curve (AUC) of 0.65. Similar results were found with other versions APACHE scores [4, 5], and these scores usually require modification to improve their discriminative power in specific subgroups of ICU patients.

Acute lung injury (ALI) is common in the ICU and is associated with adverse outcomes such as prolonged length of stay in ICU, increased financial cost and the risk of death [6, 7]. Diagnosis and prevention of ALI is an active area of research in critical care medicine. More importantly, the prognostication and triage of ALI patients may help to better utilize medical resources, inform patients and their families, and early use of certain interventions. However, there is currently no prediction score developed specifically for ALI patients. APACHE III is widely used in clinical practice and research for ALI patients, but its validity in sepsis-associated ALI is not systematically investigated. Therefore, the aim of the study was to investigate the discrimination and calibration of APACHE III in predicting mortality in sepsis associated ALI patients.

Methods

Data acquisition

The study was a secondary analysis of a prospective randomized controlled trial investigating the efficacy of rosuvastatin in sepsis-associated ALI (NCT00979121)[8]. The study was also known as Statins for Acutely Injured Lungs from Sepsis (SAILS) and performed from 2010 to 2013. The original study was approved by institutional review board at each of the 44 enrolling hospitals. The study stopped after enrollment of 745 of an estimated 1000 patients because of the futility of the intervention. Dataset for this trial was obtained from National heart, lung, blood institute (NHLBI) Biologic Specimen and Data Repository Information Coordinating Center (https://biolincc.nhlbi.nih.gov/studies/sails/?q=rosuvastatin). The secondary analysis was approved by the ethics committee of Jinhua municipal central hospital and informed consent was waived due to retrospective nature of the study. Patient records/information was anonymized and de-identified prior to analysis.

Study population

Patients were eligible for enrollment if they fulfilled the criteria of sepsis and ALI [8]. Sepsis was defined as documented or suspected infection and two or more of the following criteria for a systemic inflammatory response: 1) a white-cell count of more than 12,000 per cubic millimeter or less than 4000 per cubic millimeter or a differential count with more than 10% band forms; 2) a core body temperature of more than 38 or less than 36 degree centigrade; 3) Heart rate (> 90 beats/min) or receiving medications that slow heart rate or paced rhythm [9]. ALI was defined as acute onset of the illness and all of the three criteria: 1) a ratio of the partial pressure of arterial oxygen (Pao2) to the fraction of inspired oxygen (Fio2) of 300 or less; 2) bilateral infiltrates on chest radiography that were consistent with pulmonary edema, without evidence of left atrial hypertension; and 3) receiving positive-pressure mechanical ventilation through an endotracheal tube [10]. Study coordinators screened participating ICUs daily to identify potential candidates for enrollment.

Data extraction

Because the original study showed neutral effect of rosuvastatin on mortality outcome, data on the use of rosuvastatin was not considered in our analysis. The following data were extracted: demographics such as gender, age and ethnicity; the type of ICU; the number of quadrants with infiltrates on chest X-ray; suspected or documented infection site; the primary causes of lung injury; and APACHE III score. Variables employed to calculate APACHE III score were obtained 24 hours preceding randomization. Intraoperative values and values associated with death or cardio/respiratory arrest situations were not used.

Study endpoint

The primary outcome of the current study was the same as in the original trial, 60-day in-hospital mortality, defined as death before hospital discharge, censored 60 days after enrollment.

Statistical analysis

Descriptive data were summarized as mean (SD) or frequency (percentage) as appropriate. The difference between survivors and non-survivors were compared using student t test for continuous variables and using Pearson’s Chi-square test for discrete variables. If any continuous variables were non-parametric, Mann-Whitney U test was employed.

Discrimination of APACHE III was assessed by calculating the area under the receiver operating characteristic (ROC) curve (AUC) with its 95% CI. Discrimination was classified into categories of excellent, very good, good, moderate, and moderate for respective AUCs of 0.9 to 0.99, 0.8 to 0.89, 0.7 to 0.79, 0.6 to 0.69, and less than 0.6 [11]. To exclude the possibility that the lower predictive performance of APACHE III may indeed stem, in part, from a beneficial effect of rosuvastatin, we performed sensitivity analysis by restricting to the control arm. Hosmer-Lemeshow goodness-of-fit statistic was used to assess the calibration of APACHE III in overall population and subgroups [12, 13]. The “pROC” package was used in R platform [14]. A non-significant P value was considered evidence of good calibration. Overall model performance was evaluated by using the Brier score. It measures the average squared deviation between predicted probabilities for a set of events and their outcomes. A lower score represents higher accuracy. The Brier score represents the overall performance of APACHE III in predicting outcome, involving both discrimination and calibration [15, 16].

The predictive value of APACHE III score might be moderate in ALI patients. To improve its predictive value we included other potential confounding variables that were found to be statistically significant in bivariate analysis into multivariable regression model. Furthermore, the primary cause of ALI was forced into the model because empirical evidence had suggested its association with mortality outcome in ALI patients [17].

Statistical analysis was performed by using STATA 13.1 (StataCorp, College Station, Texas 77845 USA) and R packages (version 3.1.1). Statistical significance was considered at p<0.05.

Results

A total of 745 patients were included in the study, including 540 survivors and 205 non-survivors (Table 1). There was no statistical difference in gender, ethnicity, types of ICU, number of quadrants with infiltrates, and site of infection between survivors and non-survivors. Non-survivors were significantly older than survivors (59.71±16.17 vs 52±15.92 years, p<0.001). The primary causes of ALI were also different between survivors and non-survivors (p = 0.017). Survivors were more likely to have the cause of sepsis than non-survivors (21.2% vs. 15.1%). As expected, APACHE III score was higher in non-survivors than in survivors (106.72±27.30 vs. 88.42±26.86; p<0.001).

thumbnail
Download:
Table 1. Comparisons of baseline characteristics between survivors and non-survivors.

https://doi.org/10.1371/journal.pone.0139374.t001

Discrimination of APACHE III to predict mortality in ALI patients was moderate with an AUC of 0.68 (95% CI: 0.64–0.73; Fig 1). The model calibration was good with a non-significant p value for Hosmer-Lemeshow statistic (p = 0.61). Fig 2 shows calibration and discriminating of the APACHE III score in predicting mortality. Sensitivity analysis restricting to patients in control arm (n = 366) showed similar result with an AUC of 0.68 (95% CI: 0.62–0.75; red line in Fig 1). The Brier score was 0.1824, indicating that APACHE III had moderate predictive performance in ALI patients. The Hosmer-Lemeshow statistic was 6.31 (p = 0.61).

thumbnail
Download:
Fig 1. Assessment of discrimination with area under the receiver operating characteristic (ROC) curve.

Discrimination of APACHE III to predict mortality in overall ALI patients was moderate with an AUC of 0.68 (95% CI: 0.64–0.73). The red line shows sensitivity analysis by restricting to patients in the control arm. The results showed an AUC of 0.68 (95% CI: 0.62–0.75).

https://doi.org/10.1371/journal.pone.0139374.g001

thumbnail
Download:
Fig 2. Predicted versus observed probability of death.

The observed probability was calculated by categorizing predicted probability of death into eight subgroups. It is obvious that the observed probability of death increases monotonically.

https://doi.org/10.1371/journal.pone.0139374.g002

To adjust for confounding factors, we built a multivariable logistic regression model. The model initially incorporated all variables as listed in Table 1. Variables already included in calculating APACHE III score were omitted. However, all of the factors except for APACHE III were not statistically significant even at p = 0.1. Variable selection was performed by using stepwise procedure and only APACHE III was remained in the model. Thus, the result was actually univariate analysis as had been presented previously.

Discussion

Although two studies have reported the discrimination of APACHE III in patients with ALI/ARDS, our study for the first time focused on patients with sepsis-associated ALI [18, 19]. The prior validations of APACHE III score included ARDS patients with heterogeneous etiologies and our study is novel in that ALI population was more homogenous. The result showed that the APACHE III predicts moderately for mortality outcome in ALI patients.

As the APACHE III was originally developed to predict in-hospital death [1], censored 60 days after enrollment for follow-up in our study deviates from the initial prognostic methodology, which may underlie the discrepancy between the original AUC and those presented within the current trial. The original APACHE III was developed almost 25 years ago. Thus, the advancements in critical care over this extended period (e.g. early goal directed therapy, low tidal volume ventilation and bundled care) and general reduction in overall ICU death many be another contribution to the discrepancy. It might also be worth noting that although the APACHE III score was developed in an “unselected” group of critically ill patients, its intended use was via the “APACHE III first-day hospital risk equation” in which a patient’s diagnosis (example “non-cardiogenic pulmonary edema”) modified the risk of death predicted (acknowledging that the score by itself would perform differently in patients with ARDS than unselected ICU patients).

Although it is still controversial, clinical outcomes of ALI may differ between pulmonary and extrapulmonary causes. Sepsis causes of ALI is a typical form of extrapulmonary cause, while aspiration and/or pneumonia are regarded as direct causes. Some studies reported adverse impact of sepsis, while others showed neutral effect [20–22]. In a cohort of ALI patients in MICU, Zilberberg MD and colleagues reported that sepsis was associated with increased risk of death. In our study, there was no evidence that sepsis was associated with increased risk of death (p>0.05). On the other hand, the definition of primary cause of ALI was different in our study. All patients had sepsis-associated ALI, but there were different priamry causes. For example, a patient with sepsis presents no sign of ALI at the very beginning. Then he or she may have aspiration because of vomiting during ICU stay and subsequently develops ALI. At this stage, he has sepsis-associated ALI, but the primary cause is aspiration. This distinct definition in our study may partly explain the conflicting results. Age was demonstrated to be an important risk factor for mortality, which is consistent with other reports [23, 24]. However, older patients generally have more medical comorbidities that contribute to their poor outcome.

The advantage of the study is the use of sepsis associated ALI subjects, enabling the minimization of clinical heterogeneity. The problem of heterogeneity is a critical issue in clinical researches involving ICU patients [25]. For example, the idiosyncrasy of one dataset collected from an ICU may cause a model developed with current dataset to be poorly validated in dataset collected in another ICU. There are substantial differences among ICUs from different hospitals in the type of patients, treatment protocols, and some other administration-related factors. By restricting to study population to sepsis associated ALI, we can minimize the problem of heterogeneity as much as possible. Secondly, the dataset we used was prospectively collected, which was more accurate and reliable than those collected retrospectively. For example, to determine the primary causes of ALI, we need prospective clinical observation that cannot be obtained with retrospective dataset.

Limitations of the study need to be acknowledged in the study. The treatment strategy and resuscitation protocol for those with sepsis were not incorporated in our analysis. Although these aspects have significant impact on clinical outcomes, they are not timely. The primary aim of prediction model is to provide prognostic information as early as possible. If we consider treatment protocols, the timeliness of the model will be compromised.

Conclusion

In conclusion, the study for the first time validated the discrimination of APACHE III in sepsis associated ALI patients. The result showed that the APACHE III predicted moderately for mortality outcome in ALI patients.

Key messages

  1. - APACHE III predicted moderately for mortality outcome in ALI patients.
  2. - The advantage of the study is the use of sepsis associated ALI subjects, enabling the minimization of clinical heterogeneity.

Acknowledgments

This Manuscript was prepared using ALTA Research Materials obtained from the NHLBI Biologic Specimen and Data Repository Information Coordinating Center and does not necessarily reflect the opinions or views of the ALTA or the NHLBI.

Author Contributions

Conceived and designed the experiments: ZZ KC. Performed the experiments: ZZ LC. Analyzed the data: ZZ KC. Wrote the paper: ZZ LC.

References

  1. 1. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest. 1991;100(6):1619–36. Epub 1991/12/01. pmid:1959406.
  2. 2. Keegan MT, Gajic O, Afessa B. Comparison of APACHE III, APACHE IV, SAPS 3, and MPM0III and influence of resuscitation status on model performance. Chest. 2012;142(4):851–8. Epub 2012/04/14. pmid:22499827; PubMed Central PMCID: PMC3465106.
  3. 3. Keegan MT, Gali B, Findlay JY, Heimbach JK, Plevak DJ, Afessa B. APACHE III outcome prediction in patients admitted to the intensive care unit after liver transplantation: a retrospective cohort study. BMC surgery. 2009;9:11. Epub 2009/07/31. pmid:19640303; PubMed Central PMCID: PMC2726122.
  4. 4. Chen YX, Li CS. Arterial lactate improves the prognostic performance of severity score systems in septic patients in the ED. The American journal of emergency medicine. 2014;32(9):982–6. Epub 2014/07/26. pmid:25059886.
  5. 5. Cui Y, Wang T, Bao J, Tian Z, Lin Z, Chen D. Comparison of Charlson's weighted index of comorbidities with the chronic health score for the prediction of mortality in septic patients. Chinese medical journal. 2014;127(14):2623–7. Epub 2014/07/22. pmid:25043078.
  6. 6. Treggiari MM, Hudson LD, Martin DP, Weiss NS, Caldwell E, Rubenfeld G. Effect of acute lung injury and acute respiratory distress syndrome on outcome in critically ill trauma patients. Critical care medicine. 2004;32(2):327–31. Epub 2004/02/06. pmid:14758144.
  7. 7. Shah CV, Localio AR, Lanken PN, Kahn JM, Bellamy S, Gallop R, et al. The impact of development of acute lung injury on hospital mortality in critically ill trauma patients. Critical care medicine. 2008;36(8):2309–15. Epub 2008/07/31. pmid:18664786.
  8. 8. National Heart L, Blood Institute ACTN, Truwit JD, Bernard GR, Steingrub J, Matthay MA, et al. Rosuvastatin for sepsis-associated acute respiratory distress syndrome. The New England journal of medicine. 2014;370(23):2191–200. Epub 2014/05/20. pmid:24835849; PubMed Central PMCID: PMC4241052.
  9. 9. Bone RC, Balk RA, Cerra FB, Dellinger RP, Fein AM, Knaus WA, et al. Definitions for sepsis and organ failure and guidelines for the use of innovative therapies in sepsis. The ACCP/SCCM Consensus Conference Committee. American College of Chest Physicians/Society of Critical Care Medicine. Chest. 1992;101(6):1644–55. Epub 1992/06/01. pmid:1303622.
  10. 10. Bernard GR, Artigas A, Brigham KL, Carlet J, Falke K, Hudson L, et al. Report of the American-European consensus conference on ARDS: definitions, mechanisms, relevant outcomes and clinical trial coordination. The Consensus Committee. Intensive care medicine. 1994;20(3):225–32. Epub 1994/01/01. pmid:8014293.
  11. 11. Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143(1):29–36. Epub 1982/04/01. pmid:7063747.
  12. 12. Zhou X, Ben S, Chen H, Ni S. A comparison of APACHE II and CPIS scores for the prediction of 30-day mortality in patients with ventilator-associated pneumonia. International journal of infectious diseases: IJID: official publication of the International Society for Infectious Diseases. 2014;30C:144–7. Epub 2014/12/03. pmid:25461659.
  13. 13. Li Y, Song B, Fang H, Gao Y, Zhao L, Xu Y. External validation of the A2DS2 score to predict stroke-associated pneumonia in a Chinese population: a prospective cohort study. PloS one. 2014;9(10):e109665. Epub 2014/10/10. pmid:25299226; PubMed Central PMCID: PMC4192306.
  14. 14. Robin X, Turck N, Hainard A, Tiberti N, Lisacek F, Sanchez JC, et al. pROC: an open-source package for R and S+ to analyze and compare ROC curves. BMC bioinformatics. 2011;12:77. Epub 2011/03/19. pmid:21414208; PubMed Central PMCID: PMC3068975.
  15. 15. Ikeda M, Ishigaki T, Yamauchi K. Relationship between Brier score and area under the binormal ROC curve. Computer methods and programs in biomedicine. 2002;67(3):187–94. Epub 2002/02/21. pmid:11853944.
  16. 16. Rufibach K. Use of Brier score to assess binary predictions. Journal of clinical epidemiology. 2010;63(8):938–9; author reply 9. Epub 2010/03/02. pmid:20189763.
  17. 17. Zilberberg MD, Epstein SK. Acute lung injury in the medical ICU: comorbid conditions, age, etiology, and hospital outcome. American journal of respiratory and critical care medicine. 1998;157(4 Pt 1):1159–64. Epub 1998/05/01. pmid:9563734.
  18. 18. Cooke CR, Shah CV, Gallop R, Bellamy S, Ancukiewicz M, Eisner MD, et al. A simple clinical predictive index for objective estimates of mortality in acute lung injury. Critical care medicine. 2009;37(6):1913–20. Epub 2009/04/23. pmid:19384214; PubMed Central PMCID: PMC2731230.
  19. 19. Damluji A, Colantuoni E, Mendez-Tellez PA, Sevransky JE, Fan E, Shanholtz C, et al. Short-term mortality prediction for acute lung injury patients: external validation of the Acute Respiratory Distress Syndrome Network prediction model. Critical care medicine. 2011;39(5):1023–8. Epub 2011/07/16. pmid:21761595; PubMed Central PMCID: PMC3129473.
  20. 20. Agarwal R, Aggarwal AN, Gupta D, Behera D, Jindal SK. Etiology and outcomes of pulmonary and extrapulmonary acute lung injury/ARDS in a respiratory ICU in North India. Chest. 2006;130(3):724–9. Epub 2006/09/12. pmid:16963669.
  21. 21. Bhadade RR, de Souza RA, Harde MJ, Khot A. Clinical characteristics and outcomes of patients with acute lung injury and ARDS. Journal of postgraduate medicine. 2011;57(4):286–90. Epub 2011/11/29. pmid:22120856.
  22. 22. Irish Critical Care Trials G. Acute lung injury and the acute respiratory distress syndrome in Ireland: a prospective audit of epidemiology and management. Critical care. 2008;12(1):R30. Epub 2008/03/04. pmid:18312626; PubMed Central PMCID: PMC2374618.
  23. 23. Chetcuti Zammit S, Azzopardi N, Sant J. Mortality risk score for Klebsiella pneumoniae bacteraemia. European journal of internal medicine. 2014;25(6):571–6. Epub 2014/05/13. pmid:24814431.
  24. 24. Zhang Z, Ni H. Prediction model for critically ill patients with acute respiratory distress syndrome. PloS one. 2015;10(3):e0120641. Epub 2015/03/31. pmid:25822778; PubMed Central PMCID: PMC4378988.
  25. 25. Yun C, Hui Y. Heterogeneous postsurgical data analytics for predictive modeling of mortality risks in intensive care units. Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society IEEE Engineering in Medicine and Biology Society Annual Conference. 2014;2014:4310–4. Epub 2015/01/09. pmid:25570946.