Figures
Abstract
Background
Delayed nephrology consultation (NC) seems to be associated with worse prognosis in critically ill acute kidney injury (AKI) patients.
Design, Setting, Participants, & Measurements
The aims of this study were to analyze factors related with timing of NC and its relation with AKI patients' outcome in intensive care units of a tertiary hospital. AKI was defined as an increase ≥50% in baseline serum creatinine (SCr). Early NC and delayed NC were defined as NC performed before and two days after AKI diagnosis day. Multivariable logistic regression and propensity scores (PS) were used to adjust for confounding and selection biases. Hospital mortality and dialysis dependence on hospital discharge were the primary outcomes.
Results
A total of 366 AKI patients were analyzed and NCs were carried out in 53.6% of the patients. Hospital mortality was 67.8% and dialysis required in 31.4% patients (115/366). Delayed NCs (34%) occurred two days after AKI diagnosis day. This group presented higher mortality (OR: 4.04/CI: 1.60–10.17) and increased dialysis dependence (OR: 3.00/CI: 1.43–6.29) on hospital discharge. Four variables were retained in the PS model for delayed NC: diuresis (1000 ml/24 h - OR: 1.92/CI: 1.27–2.90), SCr (OR: 0.49/CI: 0.32–0.75), surgical AKI (OR: 3.67/CI: 1.65–8.15), and mechanical ventilation (OR: 2.82/CI: 1.06–7.44). After correction by PS, delayed NC was still associated with higher mortality (OR: 3.39/CI: 1.24–9.29) and increased dialysis dependence (OR: 3.25/CI: 1.41–7.51). Delayed NC was associated with increased mortality either in dialyzed patients (OR: 1.54/CI: 1.35–1.78) or non-dialyzed patients (OR: 2.89/CI: 1.00–8.35).
Citation: Costa e Silva VT, Liaño F, Muriel A, Díez R, de Castro I, Yu L (2013) Nephrology Referral and Outcomes in Critically Ill Acute Kidney Injury Patients. PLoS ONE 8(8): e70482. https://doi.org/10.1371/journal.pone.0070482
Editor: Emmanuel A. Burdmann, University of Sao Paulo Medical School, Brazil
Received: April 16, 2013; Accepted: June 20, 2013; Published: August 2, 2013
Copyright: © 2013 Costa e Silva et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Acute kidney injury (AKI) occurs with an incidence ranging between 30% and 70% in critically ill patients [1]. Despite several recent advances in AKI assistance, such as refined dialysis support, better dialysis adequacy, volume and fluid management, improved diagnosis and treatment of AKI-related distant organ dysfunction, AKI morbidity and mortality rates remain unacceptably elevated [2]–[9].
An important contribution for the understanding of AKI was the introduction of RIFLE and AKIN criteria [10], [11]. These indices provide a universal AKI definition that allows comparisons among clinical studies worldwide [12]. In addition, the search for earlier and more sensitive renal biomarkers has contributed to a new perspective on early AKI diagnosis and timely interventions [13], [14].
In view of this new AKI clinical approach, early nephrology consultation could provide opportunities for intervention and modification of AKI patients' outcomes. In a pioneer study, Mehta et al. reported that delayed nephrology consultation (NC) was associated with increased mortality in critically ill AKI patients but this result was not sustained after adjustment for confounding variables [15]. Recent reports have indicated that early NC may improve hospital-acquired AKI prognosis, although none of these studies have assessed a large group of AKI intensive care unit (ICU) patients [16]–[19]. Additionally, other important aspects, such as determining factors for nephrology referral and optimal time for renal consultation remain to be determined [20].
Therefore, the aims of this study were to analyze the determining factors for nephrology referral and evaluate the hypothesis that timing of nephrology consultation may influence the prognosis of AKI patients in the ICU. We hypothesized that delayed NC could be associated with higher mortality and dialysis dependence at hospital discharge.
Materials and Methods
Study participants: A prospective, observational study was conducted through an active search for AKI cases by daily visits to six ICUs (pulmonary, internal medicine, trauma, surgical, infectious diseases and emergency) comprising 53 beds at Hospital das Clinicas of the University of São Paulo, a tertiary academic institution in Brazil.
The study protocol was approved by the Hospital das Clinicas of University of Sao Paulo ethics and research committee. An informed consent, either verbal or written, was not required because of the observational nature of the study with critically ill patients in the ICU. The CAPPesq - Comissao de Etica para Analise de Projetos de Pesquisa do Hospital das Clinicas da Faculdade de Medicina da Universidade de Sao Paulo, local ethics and review board waived the need for written informed consent from the participants of the study.
All patients admitted to the selected ICUs were evaluated for AKI during ICU stay, between November 2003 and June 2005. All AKI patients were prospectively followed since AKI diagnosis day until hospital discharge or death. AKI was defined as an increase ≥50% from baseline serum creatinine (SCr), according to the RISK stage of the RIFLE system [10]. Sepsis was defined according to established criteria [21]. Chronic kidney dysfunction (CKD) was considered with an estimated Cr clearance <60 ml/min/1.73 m2 (NKF stage ≥III). Surgical AKI was considered as an AKI related to surgical procedures. The exclusion criteria were baseline SCr≥3.0 mg/dL, dialysis treatment, age <18 years old, kidney transplantation, ICU stay shorter than 48 h, AKI diagnosis established more than 24 h before ICU admission, urinary tract obstruction, and hypovolemia responsive to fluids. Nephrology consultation (NC) was classified as early or delayed NC if they occurred before or more than 48 h after the AKI diagnosis day, respectively [15].
In order to guarantee the observational character of the study, data were collected by a single observer, independent of the ICU and nephrology staff, who was not involved in patient care. Nephrology referral was kept by the ICU attending physician discretion. The attending nephrologists run daily rounds and are available 24 h/day. Thus, NC was usually performed immediately after ICU call. Nephrology service is responsible for dialysis support but decisions on dialysis indications are taken together by the ICU and nephrology physicians. The primary outcomes were in-hospital mortality and dialysis dependence at hospital discharge. Hospital and ICU length of stay were also recorded.
The following variables were collected: age, sex, race, hospital and ICU admission days, ICU origin, co-morbidities (assessed by Charlson co-morbidity Index) [22], baseline renal function, presumed AKI etiologies, vital signs, urine output, laboratory tests, vasoactive drug use and mechanical ventilation. Intermittent hemodialysis was defined as dialysis performed for up to 8 hours of duration (Fresenius 4008S, Germany) and continuous renal replacement therapy as continuous venous-venous hemodialysis (CVVHD), prescribed for at least 24 h (Diapact BBraun, Brazil). Illness severity was assessed with the Simplified Acute Physiology Score (SAPS) 3 with the customized equation for the Central and South America (CSA) region [23]. Organ failure was defined as follows: a) respiratory – need of mechanical ventilation; b) central nervous system (CNS) – Glasgow coma scale ≤8; c) hepatic – total bilirubin ≥2.0 mg/dL and/or enzymatic activity (prothrombin time) ≤50%; d) hematological – leukocytes ≤1,000/mm3 and/or platelets ≤20,000 and/or hematocrit ≤20%; and e) cardiovascular – cardiovascular SOFA score ≥3 [24].
Statistical analysis: Data were expressed as mean ± SD or median with 25th and 75th interquartile ranges (IQR) as appropriate. Categorical variables were expressed as proportions and were analyzed with Pearson's χ2 test for independent groups. Fisher's test was used where appropriate. Multivariable logistic regression models were constructed with backwards variable selection, using P-value <0.05 for variable retention. Candidate variables were those with a likelihood ratio of significance <0.05 upon bivariate analysis. The colinearity of the maximal models was evaluated using the criteria proposed by Belsley [25]. Discrimination was assessed using the AUCROC [26]. Calibration was assessed using the Hosmer-Lemeshow (HL) goodness-of-fit test comparing observed vs. expected mortality across deciles of risk [27]. A high P-value (>0.05) indicated a good fit of the model.
To address the influence of confounding variables, covariate adjusted propensity scores (PS) were utilized [28]–[30]. A PS for delayed versus early NCs was created with the variables obtained on the AKI diagnosis day. Additionally, a PS for nephrology consultation (yes or no) was created with the variables obtained on the day of the NC or on the third day after AKI diagnosis for those not evaluated by nephrology. A multivariable logistic regression model was generated using factors that differed among groups (P<0.05) and a model predicting the likelihood or “propensity” of consultation was fitted. Propensity scores were then incorporated as covariates in a logistic regression model, using mortality as the dependent variable.
A two-tailed P-value <0.05 was considered significant. Statistical analysis was carried out using SPSS for Windows, version 18.0 (Chicago, IL, USA), and SAS (SAS Institute, Cary, NC, USA).
Results
During the study period, a total of 2,998 patients were admitted to the selected ICUs. Among these patients, AKI was diagnosed in 400 patients. Only 34 patients (8.5%) were excluded because of missing data, resulting in 366 AKI patients suitable for analysis. NC was performed in 53.6% (196/366) of the AKI patients and occurred 3.0 (1.0–4.0) days after the AKI diagnosis day (DD). Most patients referred to nephrologists were subjected to early NC (65.3%) (Figure 1). The overall hospital mortality was 67.8% (248/366) and dialysis was required in 31.4% (115/366) of patients.
ICU: intensive care unit; AKI: acute kidney injury; NC: nephrology consultation.
Nephrology Consultation
Patients referred to nephrologists presented higher mortality (OR: 2.96/CI: 1.87–4.67) and decreased hospital length of stay compared with patients not evaluated by nephrologists (18.5 [11.0–36.0] vs. 27.50 [14.0–37.0] days, P<0.0001). When these two groups (NC and non-NC) were compared on the day of nephrology consultation (day 3), several differences were observed. NC patients demonstrated increased illness severity assessed by organ failures and SAPS3 scores, as well as with parameters related to AKI severity (reduced diuresis and serum bicarbonate [SBic]); increased SCr, blood urea nitrogen and potassium [SK] (Table 1). A propensity score (PS) for NC was created. The five variables retained in the model were: SAPS 3, CSA score, diuresis, SCr, SK, and SBic (Table 2). After correction by the PS, NC was no longer associated with increased mortality (OR: 1.21/CI: 0.66–2.19).
Nephrology Consultation Timing – Factors associated with Delayed Nephrology Consultation
AKI diagnosis occurred 1.0 (0.00–4.00) day after ICU admission while NC occurred 3.0 (1.0–4.0) days after the AKI DD. Median time for nephrology referral was 5.5 (4.0–9.0) and 1.0 (0.0–2.0) days in delayed and early NC groups, respectively.
The early and delayed NC groups presented differences, especially regarding AKI origin and renal function on the AKI diagnosis day. There were no differences in demographic (age, co-morbidities) or illness status characteristics (sepsis, organ failure – except mechanical ventilation, severity scoring) between the groups (Table 3). A propensity score for delayed NC was created. The variables retained in the model were diuresis, SCr, surgical AKI and mechanical ventilation (Table 4).
Influence of Delayed Nephrology Consultation on patient outcomes
The delayed NC group presented higher mortality (OR: 4.04/CI: 1.60–10.17) and increased dialysis dependence at hospital discharge (OR: 3.00/CI: 1.43–6.29). After correction by the PS, delayed NC was still associated with higher mortality (OR: 3.39/CI: 1.24–9.29) and greater dialysis dependence (OR: 3.25/CI: 1.41–7.51).
Delayed NC patients presented increased ICU length of stay (LOS, days) (19.00 [12.50–34.50] compared with the early NC group (13.5 [9.00–23.00], P = 0.03). Delayed NC patients needed longer mechanical ventilation support (MV, days)) (16.5 [11.0–32.0] compared with the early NC group (10.0 [6.0–15.0], P: 0.001, Table 3). Prolonged time on MV was associated with mortality (8.0 [0.0–18.0] vs. 12.0 [8.0–20.2] for survival and non-survival patients, respectively, P = 0.002).
Dialysis Characteristics Associated with Delayed Nephrology Consultation
Renal replacement therapy (RRT) was provided exclusively by intermittent hemodialysis (14.7%, 17/115) and CVVHD (87.8%, 104/115). Dialysis was initiated around 4.0 (2.0–7.0) days after AKI DD and 1.0 (0.0–3.0) day after NC. RRT was associated with increased mortality (OR:2.26/CI:1.13–4.53) but RRT requirement was not different in both early and delayed NC groups (P = 0.072).
Time period to start dialysis after NC was not different between early and delayed NC groups (P = 0.054). However, time duration to start dialysis after AKI DD was longer in delayed NC patients (7.0 days [5.0–11.0] than in early NC group (2.0 days [1.0–4.0], P<0.0001). Prolonged time for dialysis initiation after AKI diagnosis was associated with mortality (1.00 [1.0–2.0] vs. 4.0 [2.0–8.0] in survival and non-survival patients, respectively, P = 0.001).
Delayed NC was associated with increased mortality in both dialyzed (31.4%, OR: 1.54/CI: 1.35–1.78) and non-dialyzed patients (OR: 2.89/CI: 1.00–8.35). In addition, delayed NC patients needed longer periods of MV regardless the RRT requirement (Table 5). When dialysis and non-dialysis patients were analyzed on AKI DD, delayed and early NC groups presented similar results: no differences in demographics (age, co-morbidities) or illness status characteristics (sepsis, organ failure, severity scoring). However, early dialyzed NC patients presented worse parameters related to AKI severity such as reduced diuresis and increased SCr (Table 5).
Discussion
In this observational study of critically ill patients, it was observed that around half of AKI patients were referred to nephrologists These patients presented higher mortality which probably resulted from patients' worse illness characteristics (higher SAPS3 score, increased number of organ failures and more severe AKI). However, mortality rate was no longer different between NC and non-NC groups when corrected by the propensity score.
In contrast, a strong association was demonstrated between delayed NC and worse prognosis in NC patients. Mehta et al. [15] assessed the impact of NC timing on the prognosis of critically ill AKI patients. Delayed NC was associated with increased mortality in the bivariate analysis but this effect was not sustained after correction by PS. There are similarities compared with our study: ICU setting, similar sample size and PS variables (SCr and diuresis). Conversely, remarkable differences are also present: different AKI definition was utilized because patients were recruited between 1989 and 1995; only patients with AKI diagnosed on ICU admission were enrolled and missing data comprised almost one-third of the initial sample, resulting in only 215 patients out of 822 original AKI NC patients.
Perez-Valdivieso et al. [17] published a study with 646 AKI hospital patients using RIFLE definition. They reported that patients who had an increment in SCr greater than 101% at the time of NC presented increased mortality and worse renal function at hospital discharge. However, neither PS nor further statistical analysis was performed, remaining uncertain about the influence of residual confounding factors.
Balasubramanian et al. [16] demonstrated that early NC was associated with reduced risk of further decrease in kidney function (RIFLE criteria) in AKI hospital patients, considering a NC timing of 18 hours. For the first time, NC was assessed as an intervention in a randomized trial which probably provided a more powerful statistical analysis. However, informed consent was required for the patients, which might have induced selection bias for the attending physicians and nephrologists. Other limitations of this study were: pre-renal AKI was present in half of the sample; sepsis was the etiology in only one-third of the patients and dialysis requirement was observed in only 1% of the cases. These factors may explain the lack of association of early NC and mortality.
Recently, Ponce et al. [18] have published an observational, prospective study assessing 148 AKI ICU patients, in which only 77 patients were evaluated by nephrologists. Delayed NC, using a two-day interval, similarly was associated with increased mortality after adjustment in a multivariable analysis.
In our study, extended time to start RRT after AKI diagnosis was strongly associated with increased mortality. Although this study was not designed to assess decisions about dialysis indications, these data suggest that late RRT initiation may affect patients' prognosis. Decisions about dialysis initiation in the ICU setting are influenced by several factors: time period after AKI diagnosis, AKI severity, AKI etiology, presence of chronic kidney dysfunction and other organ failures, illness severity, volume overload and nutritional status, mechanical ventilation demands, functional reserve (age, co-morbidities) and attending physicians' preferences. An important issue of the present study is that RRT cannot be initiated without nephrology referral. Herein, ICU attending physician's perception about AKI severity and dialysis indication directly influenced nephrology referral and the timing of dialysis initiation.
Patients with delayed NC, regardless of dialysis requirement, presented higher mortality and longer mechanical ventilation support which could indicate either late nephrology intervention or worse patient's clinical status. Although propensity score for delayed NC should have corrected for these biases, we cannot propose that timing itself was responsible for the increased mortality risk.
Propensity score is an important statistical tool but it also has limitations [31]–[34]. Variables included in the model were those statistically significant in the bivariate analysis but other important non-observed variables might have been excluded. Furthermore, this is a single center study and NC was performed by an experienced academic AKI-oriented group with great expertise in AKI management, with all available dialysis techniques that might not be available elsewhere.
The present study has also some strength. An association between delayed NC patients and increased mortality and worse renal function was demonstrated, even after correction for confounding variables in a relatively large group of critically ill AKI patients. Sepsis, the main AKI etiology, was present in almost 70% of patients. Data were collected by a single observer, avoiding inter-observer variability, and only a small group of patients (8.5%) was excluded because of missing data, minimizing selection and analysis biases. Despite of limitations of the propensity score model, it might be a valuable alternative to evaluate the influence of NC on AKI outcome. Alternatively, evaluation of nephrology intervention in AKI patients' outcome in large randomized studies is hard to perform and to reproduce. Indeed, a recent pilot study has not brought better results [16]. Furthermore, our study represents the situation of a routine medical practice in several countries worldwide in which early nephrologists' interventions, including dialysis support, is dependent on ICU physicians' referral to medical specialists and their perception of illness severity and priority.
In conclusion, in this group of critically ill AKI patients, around half of the patients were referred to nephrologists. Despite of the greater severity of NC patients compared to non-NC group, mortality rate was similar when adjusted by the propensity score. In contrast, delayed NC occurred in 34% of AKI patients and it was associated with increased mortality and dialysis dependence at hospital discharge, even after adjustment for confounding factors. A direct causal relationship could not be definitely established because some other factors, such as delayed recognition of severe AKI cases, residual confounding effects and the interface between nephrology and intensive care practice may have influenced these results. Herein, the best medical approach seems to be an early collaborative work between ICU physicians and nephrologists to share decisions and join experiences which could lead to interventions that may influence the critically ill AKI patients' prognosis.
Author Contributions
Conceived and designed the experiments: VTCS FL LY. Performed the experiments: VTCS IC. Analyzed the data: IC AM RD. Wrote the paper: VTCS FL LY.
References
- 1. Hoste EA, Schurgers M (2008) Epidemiology of acute kidney injury: how big is the problem? Crit Care Med 36(suppl 4): S146–S151.
- 2. Cerda J, Ronco C (2009) Modalities of continuous renal replacement therapy: technical and clinical considerations. Semin Dial 22(2): 114–122.
- 3. Prowle JR, Bellomo R (2010) Continuous renal replacement therapy: recent advances and future research. Nat Rev Nephrol 6(9): 521–529.
- 4. Bouchard J, Soroko SB, Chertow GM, Himmelfarb J, Ikizler TA, et al. (2009) Fluid accumulation, survival and recovery of kidney function in critically ill patients with acute kidney injury. Kidney Int 76(4): 422–427.
- 5. Cerda J, Sheinfeld G, Ronco C (2010) Fluid overload in critically ill patients with acute kidney injury. Blood Purif 29(4): 331–338.
- 6. Jun M, Heerspink HJ, Ninomiya T, Gallagher M, Bellomo R, et al. (2010) Intensities of renal replacement therapy in acute kidney injury: a systematic review and meta-analysis. Clin J Am Soc Nephrol 5(6): 956–963.
- 7. Koyner JL, Murray PT (2008) Mechanical ventilation and lung-kidney interactions. Clin J Am Soc Nephrol 3(2): 562–570.
- 8. Liu KD, Matthay MA (2008) Advances in critical care for the nephrologist: acute lung injury/ARDS. Clin J Am Soc Nephrol 3(2): 578–586.
- 9. Schrier RW (2010) Fluid administration in critically ill patients with acute kidney injury. Clin J Am Soc Nephrol 5(4): 733–739.
- 10. Bellomo R, Ronco C, Kellum JA, Mehta RL, Palevsky P (2004) Acute renal failure - definition, outcome measures, animal models, fluid therapy and information technology needs: the Second International Consensus Conference of the Acute Dialysis Quality Initiative (ADQI) Group. Crit Care 8(4): R204–R212.
- 11. Mehta RL, Kellum JA, Shah SV, Molitoris BA, Ronco C, et al. (2007) Acute Kidney Injury Network: report of an initiative to improve outcomes in acute kidney injury. Crit Care 11(2): R31.
- 12. Ricci Z, Cruz D, Ronco C (2008) The RIFLE criteria and mortality in acute kidney injury: A systematic review. Kidney Int 73(5): 538–546.
- 13. Devarajan P (2011) Biomarkers for the early detection of acute kidney injury. Curr Opin Pediatr 23(2): 194–200.
- 14. Koyner JL, Vaidya VS, Bennett MR, Ma Q, Worcester E, et al. (2010) Urinary biomarkers in the clinical prognosis and early detection of acute kidney injury. Clin J Am Soc Nephrol 5(12): 2154–2165.
- 15. Mehta RL, McDonald B, Gabbai F, Pahl M, Farkas A, et al. (2002) Nephrology consultation in acute renal failure: does timing matter? Am J Med 113(6): 456–461.
- 16. Balasubramanian G, Al-Aly Z, Moiz A, Rauchman M, Zhang Z, et al. (2011) Early nephrologist involvement in hospital-acquired acute kidney injury: a pilot study. Am J Kidney Dis 57(2): 228–234.
- 17. Perez-Valdivieso JR, Bes-Rastrollo M, Monedero P, De IJ, Lavilla FJ (2007) Prognosis and serum creatinine levels in acute renal failure at the time of nephrology consultation: an observational cohort study. BMC Nephrol 8: 14.
- 18. Ponce D, Zorzenon CD, Santos NY, Balbi AL (2011) Early nephrology consultation can have impact on outcome of acute kidney injury patients. Nephrol Dial Transplant 26: 3202–3206.
- 19. Meier P, Bonfils RM, Vogt B, Burnand B, Burnier M (2011) Referral patterns and outcomes in noncritically ill patients with hospital-acquired acute kidney injury. Clin J Am Soc Nephrol 6(9): 2215–2225.
- 20. Mehta RL (2011) Management of Acute Kidney Injury: It's the Squeaky Wheel That Gets the Oil!. Clin J Am Soc Nephrol 6(9): 2102–2114.
- 21. Levy MM, Fink MP, Marshall JC, Abraham E, Angus D, et al. (2003) 2001 SCCM/ESICM/ACCP/ATS/SIS International Sepsis Definitions Conference. Crit Care Med 31(4): 1250–1256.
- 22. Charlson ME, Pompei P, Ales K, Mackenzie CR (1987) A New Method of Classifying Co-Morbidity in Longitudinal-Studies: Development and Validation. J Chronic Dis 40(5): 373–383.
- 23. Costa e Silva VT, de Castro I, Liano F, Muriel A, Rodriguez-Palomares JR, Yu L (2011) Performance of the third generation models of severity scorings systems (APACHE IV, SAPS 3, MPM-III) in acute kidney injury critically ill patients. Nephrol Dial Transplant 26(12): 3894–3901.
- 24. Vincent JL, Moreno R, Takala J, Willatts S, De MA, et al. (1996) The SOFA (Sepsis-related Organ Failure Assessment) score to describe organ dysfunction/failure. On behalf of the Working Group on Sepsis-Related Problems of the European Society of Intensive Care Medicine. Intensive Care Med 22(7): 707–710.
- 25.
Belsley DA (1991) Conditioning Diagnostics: Colinearity and Weak Data in Regression. Wiley Interscience.
- 26. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143(1): 29–36.
- 27. Hosmer DW, Hosmer T, Le CS, Lemeshow S (1997) A comparison of goodness-of-fit tests for the logistic regression model. Stat Med 16(9): 965–980.
- 28. Austin PC, Grootendorst P, Normand SL, Anderson GM (2007) Conditioning on the propensity score can result in biased estimation of common measures of treatment effect: a Monte Carlo study. Stat Med 26(4): 754–768.
- 29. Austin PC (2011) An Introduction to Propensity Score Methods for Reducing the Effects of Confounding in Observational Studies. Multivariate Behav Res 46(3): 399–424.
- 30. Rubin DB (2010) Propensity score methods. Am J Ophthalmol 149(1): 7–9.
- 31. Austin PC (2008) A critical appraisal of propensity-score matching in the medical literature between 1996 and 2003. Stat Med 27(12): 2037–2049.
- 32. Austin PC (2009) Type I Error Rates, Coverage of Confidence Intervals, and Variance Estimation in Propensity-Score Matched Analyses. Int J Biostat 5(1): Article 13.
- 33. Barnieh L, James MT, Zhang J, Hemmelgarn BR (2011) Propensity score methods and their application in nephrology research. J Nephrol 24(3): 256–262.
- 34. Rubin DB (2004) On principles for modeling propensity scores in medical research. Pharmacoepidemiol Drug Saf 13(12): 855–857.