Figures
Abstract
Background
Application of illness-severity scores in Intermediate Care Units (ImCU) shows conflicting results. The aim of the study is to design a severity-of-illness score for patients admitted to an ImCU.
Methods
We performed a retrospective observational study in a single academic medical centre in Pamplona, Spain. Demographics, past medical history, reasons for admission, physiological parameters at admission and during the first 24 hours of ImCU stay, laboratory variables and survival to hospital discharge were recorded. Logistic regression analysis was performed to identify variables for mortality prediction.
Results
A total of 743 patients were included. The final multivariable model (derivation cohort = 554 patients) contained only 9 variables obtained at admission to the ImCU: previous length of stay 7 days (6 points), health-care related infection (11), metastatic cancer (9), immunosuppressive therapy (6), Glasgow comma scale 12 (10), need of non-invasive ventilation (14), platelets 50000/mcL (9), urea 0.6 g/L (10) and bilirubin 4 mg/dL (9). The ImCU severity score (ImCUSS) is generated by summing the individual point values, and the formula for determining the expected in-hospital mortality risk is: eImCUSS points*0.099 – 4,111 / (1 + eImCUSS points*0.099 – 4,111). The model showed adequate calibration and discrimination. Performance of ImCUSS (validation cohort = 189 patients) was comparable to that of SAPS II and 3. Hosmer-Lemeshow goodness-of-fit C test was χ2 8.078 (p=0.326) and the area under receiver operating curve 0.802.
Citation: Alegre F, Landecho MF, Huerta A, Fernández-Ros N, Martínez-Urbistondo D, García N, et al. (2015) Design and Performance of a New Severity Score for Intermediate Care. PLoS ONE 10(6): e0130989. https://doi.org/10.1371/journal.pone.0130989
Editor: Lyle L. Moldawer, University of Florida College of Medicine, UNITED STATES
Received: January 30, 2015; Accepted: May 26, 2015; Published: June 29, 2015
Copyright: © 2015 Alegre et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper.
Funding: These authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Worldwide health care institutions try to give care based on best-practice models with cost-effectiveness. However, hospitalized patients are becoming complex and many of them exceed monitoring and nursing care available in conventional hospitalization wards. In this scenario, intermediate care units (ImCU) may provide a rational and proportional treatment between the intensive care units and the general ward [1–11].
The characteristics of ImCUs depend on resource availability, institutional infrastructure and the parent health care system. They can function as step-up or step-down units, or provide specialty care for cardiac, neurologic, respiratory or surgical conditions. Multipurpose ImCUs need characterization of the admitted patients in order to assess their illness severity and prognosis. Multiple severity scores have been designed and widely described in patients admitted to intensive care units (ICU) [12–21], but information in the setting of ImCU is scarce [1,5,8–9,11]. Moreover, the performance of the Simplified Acute Physiology Score II (SAPS II) [13] and the Simplified Acute Physiology Score 3 (SAPS 3) [14] showed conflicting results in intermediate care [5]. In addition, the collection of data for the calculation of these illness severity scores is time consuming.
The purpose of the present study is to design, based on simple variables, a new severity-of-illness scoring system for intermediate care, and to assess its performance in a single centre ImCU.
Patients and Methods
In order to develop the ImCU Severity Score (ImCUSS) we performed a retrospective observational study, with data collected from April 2006 to December 2013 in a single academic medical centre in Pamplona, Spain. The ImCU is a 9-bed unit adjacent to, but independent from, the mixed ICU. Each bed is equipped with a continuous telemetry, pulse oximetry, non-invasive arterial blood pressure, central venous pressure monitoring, and non-invasive pressure support ventilation. The signals are relayed to a central monitoring station and the nurse:patient ratio is 1:3. The ImCU infrastructure (beds, technical resources and nursing staff) is shared with the Stroke Unit and the Coronary Care Unit. The ImCU rounding team involves a nurse, the hospital pharmacist, the ImCU resident, the specialist or surgeon and the attending hospitalist. The hospitalist is responsible for admission and discharge of all ImCU patients. Admission and discharge criteria were set according to previous guidelines defined by The American College of Critical Care Medicine [22]. Exclusion criteria included age less than 18 years old, severe respiratory failure at imminent risk of requiring intubation, status epilepticus, and catastrophic brain illness.
From April 2006 to December 2013, every consecutive patient admitted to the ImCU was evaluated. ImCU readmissions and low risk patients admitted exclusively for drug administration and desensitization were excluded from data analysis. Demographics, past medical history, reasons for admission, physiological parameters at the time of admission and during the first 24 hours of ImCU stay, laboratory variables, location at discharge and survival to hospital discharge were recorded. SAPS II and SAPS 3 were also calculated according to standard coefficients.
Statistical analysis was performed using SPSS for Windows, version 20.0 (SPSS Inc, Chicago; IL). Continuous variables were reported as mean ± standard deviation or median (25%-75% interquartile range).
Logistic regression was used to develop an in-hospital mortality prediction model for patients admitted to our ImCU. Variables to be included in the multivariable model were determined by using univariable logistic regression (p value in the univariable model ≤0.20). The maximum likelihood method was applied to estimate model coefficients, and these were then used to develop our new severity score. We assigned integer point values to each predictor variable from the multivariable logistic regression model (the coefficients of these variables were transformed by multiplying them by 10 and rounding to the nearest integer), thus generating the ImCUSS. The expected mortality rates were calculated using the ImCUSS as the only variable in another logistic regression model.
The ImCUSS was derived using a random sample of 75% subjects from the cohort, and then validated in the remaining 25%. For both instances, the calibration and discrimination performances were evaluated with the Hosmer-Lemeshow goodness-of-fit (GOF) C test and the area under the receiver operator curve (AUROC), respectively.
Ethics statement
The study protocol was approved by the Institutional Review Board (IRB) at the Clínica Universidad de Navarra (ref. 129/2010). The IRB waived the need for informed consent, because it is an observational non-interventional study, and also because it did not interfere with decisions related to patient’s care. The study has been performed in accordance with the ethical standards laid down in the 1964 Declaration of Helsinki and its later amendments. Patient information was anonymized and de-identified prior to analysis.
Results
During the study period, 1112 patients were admitted to the ImCU. Of these, 369 were excluded: 70 low-risk patients (drug administration and desensitization), 200 readmissions, and 99 patients for missing variables. Therefore, 743 patients were included in data analysis.
Patient characteristics, reasons for admission, and relevant data regarding main differences between survivors and deaths are summarized in Table 1. The mean age was 67 years with 62% male. The patients were admitted from the general ward (50.4%), the emergency room (27.1%), the ICU (13.1%), the operating room (6.0%) and from other hospital wards (3.4%). Reasons for admission were primarily medical (88.7%), with respiratory failure (34.2%) and sepsis (21.9%) as the leading causes. The median length of ImCU stay was 4 (2–7) days.
As summarized in Table 1, factors significantly related to in-hospital mortality were SAPS II and 3 scores, the presence of metastatic cancer (includes solid organ metastatic cancer and haematological cancer), immunosuppressive therapy, do-not-resuscitate (DNR) orders, ECOG status, health-care related infection, need of non-invasive mechanical ventilation, impaired neurologic status assessed by Glasgow comma scale, cirrhosis, acute renal failure measured by RIFLE score, tachycardia, decreased urine output in the first 24 hours of ImCU admission, and need of vasoactive drugs and blood transfusions in the same period. In addition, some laboratory variables like decreased pCO2, haemoglobin, platelets and albumin, prolonged pro-thrombin time and increased lactate, C-reactive protein, urea and bilirubin were also related with in-hospital mortality.
Design and performance of the ImCUSS
The final multivariable model obtained from the derivation cohort (554 patients) contained only 9 variables that were obtained at admission to the ImCU: previous length of stay ≥7 days, health-care related infection, metastatic cancer, immunosuppressive therapy (includes steroids and other immunosuppressive drugs), Glasgow comma scale ≤12, need of non-invasive mechanical ventilation, platelets ≤50000/mcL, urea ≥0.6 g/L and bilirubin ≥4 mg/dL. No continuous variables were included, as some of them were transformed into dichotomous variables to simplify the model (conversion of polychotomus and continuous variables from the univariable analysis to dichotomous ones in the final multivariable analysis was made based on sensitivity analysis regarding different groups obtained for each variable, and favouring previous cut-offs described in the literature when possible). Parameters that were measured in the 24 hour interval after admission to the ImCU were also excluded, in order to make the score able to be calculated at admission to the ImCU and to avoid the potential Boyd and Grounds effect [23]. Additionally, we did not include variables that could have in our data a subjective component, such as ECOG scale (it was collected retrospectively in our series). Both DNR orders and the presence of solid organ metastatic and/or haematological cancer were significantly related to in-hospital mortality in the univariate analysis. Nonetheless, in order to avoid collinearity (65.4% of patients with DNR orders had also metastatic cancer), we excluded DNR for the final multivariate model.
The logistic regression model coefficients, SE, odds ratios, the 95% CI for the odds ratios, p values, and the point value for each of the predictor variables are shown in Table 2. Model performance was adequate, with Hosmer-Lemeshow GOF C test showing χ2 13.039, degrees of freedom 8, p = 0.111. p-value is greater than 0.05, suggesting that the model is well calibrated and predicts probabilities that reflect the true mortality experience of the data. The bootstrapping simulation validates that the developed model has good calibration as 974 of the 1000 Hosmer-Lemeshow GOF p values (97.4%) were greater than 0.05.
The ImCUSS is generated by summing the individual point values (Table 2) based on the patients’ status at ImCU admission, resulting in a minimal score of 0 and a maximal score of 84. The mean ± standard deviation score in the development group was 22.1 ± 15.2.). The AUROC was 0.843 (95% CI 0.805–0.881), showing good discrimination.
Probability of in-hospital mortality based on the ImCUSS was estimated using logistic regression. The scale of the score was determined to be linear in the logit using fractional polynomials. The coefficient for the score is 0.099 (standard error 0.010, OR 1.10 with 95% CI 1.08–1.13, p<0.001), and the model intercept -4.111 (standard error 0.342, p<0.001). The Hosmer-Lemeshow GOF p value is 0.111. In order to estimate the probability of in-hospital mortality, one would multiply the obtained ImCUSS by the coefficient and then add the intercept. This sum would be exponentiated and then divided by the addition of one plus the exponentiated sum.
Fig 1A compares observed and predicted mortality (based on the ImCUSS) for all subjects in the development cohort when using deciles of in-hospital mortality risk. These results indicate that estimated and observed hospital mortality pairs were very similar.
Performance of the obtained score was studied in the validation cohort (189 patients). The Hosmer-Lemeshow goodness-of-fit showed χ2 8.078 with 7 degrees of freedom and p = 0.326, indicating good calibration. Ability to discriminate in-hospital mortality was also good, with area under receiver operating curve of 0.802 (95% CI 0.733–0.871) (Fig 2). ImCUSS calibration curve for the validation cohort is shown in Fig 1B.
Comparison of ImCUSS with SAPS II and SAPS 3
The mean ImCUSS (in the validation cohort), SAPS II and SAPS 3 (in the whole cohort) were 21±16.1, 33.1±12.9 and 60.3±14.0 respectively. The expected mortality rates derived from these scores were 20.7±21.9%, 19.2±18.1% and 38.3±23.0%. The observed in-hospital mortality was 18.5% in the validation cohort (35/189) and 19.7% in the whole cohort (146/743), resulting in a standardized mortality ratio (SMR) of 0.89 (95% CI 0.59–1.19) for ImCUSS, 1.03 (95% CI 0.86–1.20) for SAPS II and 0.51 (95% CI 0.43–0.59) for SAPS 3. Interestingly, SAPS 3 clearly overestimated mortality, while SAPS II and ImCUSS showed results closer to the observed mortality.
Performance of the models is presented in Table 3. The scores showed acceptable discrimination, with an AUROC of 0.802 (95% CI 0.733–0.871) for ImCUSS, 0.777 (95% CI 0.737–0.817) for SAPS 2 and 0.793 (95% CI 0.756–0.831) for SAPS 3. The scores also showed similar calibration performance based on the Hosmer-Lemeshow GOF C-test: χ2 = 8.708 and p = 0.326 for ImCUSS, χ2 = 10.045 and p = 0.262 for SAPS II, and χ2 = 4.825 and p = 0.776 for SAPS 3. Integrated discrimination improvement analysis showed p values of 0.898 and 0.231 when comparing performance of ImCUSS versus SAPS II and SAPS 3 scores.
Clinical application of ImCUSS
In an effort to simplify the clinical application, sensitivity analysis was performed to find critical values which could classify patients in statistically different mortality groups. ImCUss was divided in 5 mortality groups; their cut-off values and mean probabilities of in-hospital death are shown in Table 4.
Additionally, we decided to study whether ImCUSS could also be useful to identify patients who may benefit of a direct ICU admission in those subjects suitable for transfer to a higher level of care in case of worsening (thus excluding patients with DNR orders at ImCU admission). Transfer to a higher level of care in the first 48 hours after ImCU admission may reflect inappropriate triage, and obviously warrants further consideration. Only 45/597 patients in our series were discharged to the ICU in the first 48 hours after ImCU admission, thus reflecting adequate global triage. Nevertheless, when studying this endpoint in the different mortality groups previously described (Table 4), we observe that patients with an ImCUSS ≥42 should be considered for direct ICU admission (eventually, 38.5% of them would need transfer to this unit in the following 48 hours in case they are admitted to the ImCU).
Discussion
ImCUs are an attractive alternative for the management of complex patients, in need of special monitoring and nursing care. Concerning the high costs of the traditional ICU and the limited resources of the general wards, multipurpose ImCUs were developed to try to give rational and proportional care. Description of this population is necessary and must be based on accurate severity scoring. Multiple severity scores have been designed and widely described in ICU patients: APACHE II, APACHE III, SAPS II, SAPS 3, SOFA, APACHE IV, MPM II (0 and 24), among others [12–21]. However, the information is scarce and the ability of these scores to accurately and reliably predict mortality in ImCU is inconsistent.
In 1998, Auriant et al [8] described the performance of SAPS II in a cohort of 433 patients, showing good discriminant power (AUROC 0.85) and calibration (C = 2.4, p<0.5), with an SMR of 0.93. Posteriorly, Ip et al [9] described the application of the APACHE II and SAPS scores in a geriatric ImCU cohort of 150 patients. Both scores showed acceptable performance regarding observed to expected mortality ratios (correlation coefficients of 0.97 and 0.92 respectively), but there is no data about calibration. In 2006, Torres et al [11] described the mortality risk of a cohort of 412 patients, using the APACHE II and TISS-28 scores. Mortality discrimination was acceptable (AUROC 0.77 and 0.88 respectively), and the scores showed statistical significant correlation with mortality in the logistic regression analysis. Unfortunately, the scores were not calibrated. Recently, our group described the performance of SAPS II and SAPS 3 in a cohort of 607 patients [5]. Both scores showed acceptable discrimination (AUROC 0.76 and 0.75 respectively) and calibration (χ2 = 12.9, p = 0.113 for SAPS II and χ2 = 4.07, p = 0.851for SAPS 3). Nonetheless, in this study SAPS 3 clearly overestimated mortality (SMR = 0.56), while the oldest version SAPS II showed better discrimination in terms of SMR, with results closer to the observed mortality (SMR = 0.87). These conflicting results, and the insufficient information, emphasize the need to find more reliable and accurate scores for the ImCU setting. In this context, the present study describes the design and performance of a new severity-of-illness model, which was developed using simple variables, easily obtained at patient admission to the ImCU. To the best of our knowledge, this is the first description of a new score, specifically designed for intermediate care. Moreover, it included the largest population in this setting.
Data regarding patient characteristics and overall mortality are similar to those previously published by our group [1,5]. Observed in-hospital mortality was 19.7%, different to that of other ImCUs and similar to that observed in ICU population, with very high risk for major complications and mortality. The contribution of oncologic patients (284/753), most of them with advanced disease (194/284 had metastatic cancer and/or haematological cancer) and elevated severity-of-illness scoring system predicted risk of death (with mean expected mortality rates of 24.5, 49.5 and 31.1% based on SAPS II, SAPS 3 and ImCUSS scores) probably contributed to the high acuity of our population. Additionally, a total of 146/793 patients had DNR-orders at admission to the ImCU.
The ImCUSS was designed based on 554 patients, and internally validated in 189 additional patients. Its simplicity is remarkable, as it only contains 9 dichotomic variables that are easily obtained in common practice. Four of them are related to previous history (length of in-hospital stay ≥7 days, health-care related infection, metastatic cancer and immunosuppressive therapy), two of them are bed-side obtained (Glasgow Comma Scale ≤12 and need of non-invasive mechanical ventilation), and the remaining three variables depend on routine blood tests (platelets ≤50000/mcL, urea ≥0.6 g/L and bilirubin ≥4 mg/dL) (Table 2). Consequently, its application could reduce the collection burden and potential errors associated to the time-consuming traditional scores. Despite the small number of patients, the ImCUSS showed adequate calibration and discrimination (Figs 1 and 2, and Table 3). Moreover, limiting acquisition of data to patient admission should minimise the impact of mortality overestimation associated with the occurrence of more abnormal physiologic values during the first 24 hours of ImCU stay. This, so-called Boyd and Grounds effect [23], might affect mortality prediction in scores like SAPS II, because the increase in computed severity illness and predicted mortality could be due to suboptimal care, more than to an increase in disease severity.
When we analyse the performance of the traditional scores in our population, it is also noteworthy the mortality overprediction based on SAPS 3 score (SMR = 0.51) compared with the ImCUSS (SMR = 0.89) and SAPS II (SMR = 1.03). In this setting, the differences in the case-mix profiles with the original cohort (SAPS 3), and the simplicity of the two other scores could explain this potential mortality overestimation [24]. Recent external validation studies showed similar results for SAPS 3 [5,25–26]. Nonetheless, although the global performance of the ImCUSS, SAPS II and 3 did not revealed meaningful differences, three aspects must be highlighted. First of all, ImCUSS is specifically designed for multipurpose ImCUs. Secondly, collection of data is made at admission to the ImCU (SAPS II includes variables collected in the first 24 hours after admission). Finally, its simplicity could be a remarkable advantage for ImCU mortality prediction.
Clinical application of ImCUSS is evident, as it provides prognostic information and an estimated in-hospital mortality rate. In addition, it may also be useful to identify patients who may benefit of a direct ICU admission, as we have previously observed in Table 4.
However, in the present study some limitations must be addressed. Lack of data regarding post-ImCU care and its impact in hospital-derived outcomes is shared by all previous scoring systems [27]. The limited number of patients derived from a single centre multipurpose ImCU, and even more restricted samples of various subgroups of the population (patients with AIDS were absent), could interfere with the evaluation of the uniformity-of-fit among different expected mortality subgroups and also limit the applicability of the new score in these specific populations. Furthermore, as previously explained, the case-mix of our cohort (advanced age, high prevalence of cancer, deteriorated functional status and DNR orders) may differ from that in other ImCUs, limiting the extrapolation of the results. Accordingly, we need larger, prospective and well-designed studies with external validation cohorts, before the routine application of the ImCUSS.
Conclusions
ImCUSS is a new score specially designed for intermediate care, based on simple and easy to calculate variables at admission, and with a notable performance in terms of calibration and mortality discrimination.
Acknowledgments
The authors thank the Nursing Staff of the ImCU for their unconditional support in patient care.
Author Contributions
Conceived and designed the experiments: FA MFL JQ JFL. Performed the experiments: FA MFL AH NFR DMU JFL. Analyzed the data: FA MFL DM NG JQ JFL. Contributed reagents/materials/analysis tools: FA MFL AH NFR JFL. Wrote the paper: FA MFL NG JQ JFL.
References
- 1. Lucena JF, Alegre F, Rodil R, Landecho MF, García-Mouriz A, Marqués M, et al. Results of a restrospective observational study of intermediate care staffed by hospitalists: impacto n mortality, co-managemente, and teaching. J Hosp Med. 2012; 7: 411–415.
- 2. Franklin CM, Rackow EC, Mamdani B, Nightingale S, Burke G, Weil MH. Decreases in mortality on a large urban medical service by facilitating Access to critical care. An alternative to rationing. Arch Intern Med. 1988; 148: 1403–1405. pmid:3132123
- 3. Byrick RJ, Mazer CD, Caskennette GM. Closure of an intermediate care unit. Impact on critical care utilization. Chest. 1993; 104: 876–881. pmid:8365304
- 4. Heras A, Abizanda R, Belenguer A, Vidal B, Ferrándiz A, Micó ML, et al. [Intermediate care units. Health care consequences in a reference hospital] Med Intensiva. 2007; 31: 353–360.
- 5. Lucena JF, Alegre F, Martínez-Urbistondo D, Landecho MF, Huerta A, García-Mouriz A, et al. Performance of SAPS II and SAPS 3 in intermediate care. PLoS One. 2013; 8: e77229. pmid:24130860
- 6. Byrick RJ, Power JD, Ycas JO, Brown KA. Impact of an intermediate care area on ICU utilization after cardiac surgery. Crit Care Med. 1986; 14: 869–872. pmid:3757527
- 7. Elpern EH, Silver MR, Rosen RL, Bone RC. The noninvasive respiratory care unit. Patterns of use and financial implications. Chest. 1991; 99: 205–208. pmid:1898646
- 8. Auriant I, Vinatier I, Thaler F, Tourneur MR, Loirat P. Simplified acute physiology score II for measuring severity of illness in intermediate care units. Crit Care Med. 1998; 26: 1368–1371. pmid:9710096
- 9. Ip SP, Leung YF, Ip CY, Mak WP. Outcomes of critically ill elderly patients: is high-dependency care for geriatric patients worthwhile? Crit Care Med. 1999; 27: 2351–2357. pmid:10579247
- 10. Bertolini G, Confalonieri M, Rossi C, Rossi G, Simini B, Gorini M, et al. Costs of the COPD. Differences between intensive care unit and respiratory intermediate care unit. Respir Med. 2005; 99: 894–900. pmid:15939252
- 11. Torres OH, Francia E, Longobardi V, Gich I, Benito S, Ruiz D. Short- and long-term outcomes of older patients in intermediate care units. Intensive Care Med. 2006; 32: 1052–1059. pmid:16791668
- 12. Gunning K, Rowan K. ABC of intensive care: outcome data and scoring systems. BMJ. 1999; 319: 241–244. pmid:10417092
- 13. Le Gall JR, Lemeshow S, Saulnier F. A new simplified acute physiology score (SAPS II) based on a European/North American multicenter study. JAMA. 1993; 270: 2957–2963. pmid:8254858
- 14. Moreno RP, Metnitz PG, Almeida E, Jordan B, Bauer P, Campos RA, et al. SAPS 3 –From evaluation of the patient to evaluation of the intensive care unit. Part 2: development of a prognostic model for hospital mortality at ICU admission. Intensive Care Med. 2005; 31: 1345–1355. pmid:16132892
- 15. Vincent JL, Moreno R, Takala J, Willatts S, de Mendoça A, Bruining H, et al. The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. Intensive Care Med. 1996; 22: 707–710. pmid:8844239
- 16. Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit Care Med. 1985; 13: 818–829. pmid:3928249
- 17. Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest. 1991; 100: 1619–1636. pmid:1959406
- 18. Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute Physiology and Chronic Health Evaluation (APACHE) IV: hospital mortality assessment for today’s critically ill patients. Crit Care Med. 2006; 34: 1297–1310. pmid:16540951
- 19. Lemeshow S, Teres D, Klar J, Avrunin JS, Gehlbach SH, Rapoport J. Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients. JAMA. 1993; 270: 2478–2486. pmid:8230626
- 20. Higgins TL, Teres D, Copes WS, Nathanson BH, Stark M, Kramer AA. Assessing contemporary intensive care unit outcome: an updated mortality probability admission model (MPM0-III). Crit Care Med. 2007; 35: 827–835. pmid:17255863
- 21. Breslow MJ, Badawi O. Severity scoring in the critically ill. Part 1 –interpretation and accuracy of outcome prediction scoring systems. Chest. 2012; 78: 1261–1269.
- 22. Nasraway SA, Cohen IL, Dennis RC, Howenstein MA, Nikas DK, Warren J, et al. Guidelines on admission and discharge for adult intermediate care units. American College of Critical Care Medicine of the Society of Critical Care Medicine. Crit Care Med. 1998; 26: 607–610. pmid:9504593
- 23. Boyd O, Grounds RM. Physiological scoring systems and audit. Lancet. 1993; 341: 1573–1574. pmid:8099649
- 24. Metnitz PG, Lang T, Vesely H, Valentin A, Le Gall JR. Ratios of observed to expected mortality are affected by differences in case mix and quality of care. Intensive Care Med. 2000; 26: 1466–1472. pmid:11126258
- 25. Strand K, Søreide E, Aardal S, Flaaten H. A comparison of SAPS II and SAPS 3 in a Norwegian intensive care unit population. Acta Anaesthesiol Scand. 2009; 53: 595–600.
- 26. Poole D, Rossi C, Latronico N, Rossi G, Finazzi S, Bertolini G. Comparison between SAPS II and SAPS 3 in predicting hospital mortality in a cohort of 103 Italian ICUs. Is new always better? Intensive Care Med. 2012; 38: 1280–1288. pmid:22584793
- 27. Keegan MT, Gajic O, Afessa B. Severity of illness scoring systems in the intensive care unit. Crit Care Med. 2011; 39: 163–169. pmid:20838329