Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Prediction of clinically relevant adverse drug events in surgical patients

  • Jacqueline M. Bos ,

    Roles Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review & editing

    jm.bos@cwz.nl

    Affiliation Department of clinical pharmacy, Canisius Wilhelmina hospital, Nijmegen, the Netherlands

  • Gerard A. Kalkman,

    Roles Formal analysis, Methodology, Writing – review & editing

    Affiliation Department of clinical pharmacy, Canisius Wilhelmina hospital, Nijmegen, the Netherlands

  • Hans Groenewoud,

    Roles Conceptualization, Investigation, Methodology, Software, Validation, Writing – review & editing

    Affiliation Department of health evidence, Radboud university medical center, Nijmegen, the Netherlands

  • Patricia M. L. A. van den Bemt,

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    Affiliation Department of hospital pharmacy, Erasmus university medical center, Rotterdam, the Netherlands

  • Peter A. G. M. De Smet,

    Roles Conceptualization, Methodology, Supervision, Writing – review & editing

    Affiliations Department of pharmacy, Radboud university medical center, Nijmegen, the Netherlands, Department Scientific Institute for Quality of Healthcare, Radboud university medical center, Nijmegen, the Netherlands

  • J. Elsbeth Nagtegaal,

    Roles Conceptualization, Methodology, Writing – review & editing

    Affiliation Department of clinical pharmacy, Meander Medical Center, Amersfoort, the Netherlands

  • Andre Wieringa,

    Roles Conceptualization, Writing – review & editing

    Affiliation Department of clinical pharmacy, Isala hospital, Zwolle, the Netherlands

  • Gert Jan van der Wilt,

    Roles Conceptualization, Formal analysis, Investigation, Methodology, Software, Supervision, Validation, Writing – review & editing

    Affiliation Department of health evidence, Radboud university medical center, Nijmegen, the Netherlands

  • Cornelis Kramers

    Roles Conceptualization, Formal analysis, Funding acquisition, Investigation, Methodology, Supervision, Validation, Writing – review & editing

    Affiliations Department of clinical pharmacy, Canisius Wilhelmina hospital, Nijmegen, the Netherlands, Department of clinical pharmacology and toxicology, Radboud university medical center, Nijmegen, the Netherlands

Abstract

Background

Risk stratification of hospital patients for adverse drug events would enable targeting patients who may benefit from interventions aimed at reducing drug-related morbidity. It would support clinicians and hospital pharmacists in selecting patients to deliver a more efficient health care service. This study aimed to develop a prediction model that helps to identify patients on the day of hospital admission who are at increased risk of developing a clinically relevant, preventable adverse drug event during their stay on a surgical ward.

Methods

Data of the pre-intervention measurement period of the P-REVIEW study were used. This study was designed to assess the impact of a multifaceted educational intervention on clinically relevant, preventable adverse drug events in surgical patients. Thirty-nine variables were evaluated in a univariate and multivariate logistic regression analysis, respectively. Model performance was expressed in the Area Under the Receiver Operating Characteristics. Bootstrapping was used for model validation.

Results

6780 admissions of patients at surgical wards were included during the pre-intervention period of the PREVIEW trial. 102 patients experienced a clinically relevant, adverse drug event during their hospital stay. The prediction model comprised five variables: age, number of biochemical tests ordered, heparin/LMWH in therapeutic dose, use of opioids, and use of cardiovascular drugs. The AUROC was 0.86 (95% CI 0.83–0.88). The model had a sensitivity of 80.4% and a specificity of 73.4%. The positive and negative predictive values were 4.5% and 99.6%, respectively. Bootstrapping generated parameters in the same boundaries.

Conclusions

The combined use of a limited set of easily ascertainable patient characteristics can help physicians and pharmacists to identify, at the time of admission, surgical patients who are at increased risk of developing ADEs during their hospital stay. This may serve as a basis for taking extra precautions to ensure medication safety in those patients.

Introduction

Pharmacotherapy is one of the most commonly applied interventions in hospital healthcare. In addition to the beneficial effects, prescribing medication also introduces risks of medication errors and adverse drug events that can lead to potentially preventable morbidity, mortality, and costs [13]. Patients on surgical wards are particularly at risk, due to their need for pain medication and antibiotics, frequent adjustments of antithrombotic regimens, and blood and fluid loss. In the case of elderly surgical patients, multiple co-morbidities requiring multiple drugs contribute even more to potential drug-related problems [4].

Risk prediction is a routine component in daily care practice in both specific areas (e.g. approaches used to determine stroke risk in patients with atrial fibrillation) as well as more generally, for example in identifying patients likely to require hospital admission. Risk stratification of hospital patients for adverse drug events (ADE) can target a population that can benefit from interventions aimed to reduce drug-related morbidity, as a form of personalized medicine. It can support clinicians and hospital pharmacists in patient prioritization to deliver more efficient health care service [5]. In a systematic review Yourman et al. emphasized that failure to consider risk prediction in a clinical setting can result in poor patient outcomes [6].

A recent review identified four studies that developed and validated ADE risk-prediction tools for use in adults over 65 years of age [4, 5, 79]. These prediction models had poor to modest performance and did not address clinical impact, thereby limiting clinical usefulness. Because a large number of variables contribute to ADE occurrence in patients, it is impossible to precisely predict every ADE in every patient. Therefore, Stevenson et al. suggested that these risk prediction strategies should focus either on one specific harmful ADE (e.g. gastrointestinal or intracranial bleeding) or ADEs in patients with a particular illness or clinical characteristic, for instance surgical patients [5]. In addition, since the aim is to prevent patient harm, it seems more rational to predict clinically relevant, potentially preventable adverse events, instead of adverse reactions in general.

The P-REVIEW study (Pharmacist-led Risk patients medication EValuation to Initiate Event reduction on surgical Wards) was designed to determine whether a multifaceted intervention comprised of educating the prescriber combined with medication review and pharmaceutical visits to the ward by the hospital pharmacist could lead to a reduction of adverse drug events among surgical patients [10]. In this study, experts assessed clinically relevant, potentially preventable ADE in a cohort of 13,264 admissions of surgical patients. An ADE was classified as clinically relevant when it was fatal, life-threatening, disabling, incapacitating, or when it resulted in or prolonged hospitalisation. When an ADE resulted from suboptimal care, it was classified as potentially preventable. The study showed a significant benefit for patients in the intervention period. To improve the cost effectiveness of medication review and other measures to prevent avoidable harm, it would be useful to identify patients at risk of clinically relevant ADEs in the surgical ward. For this purpose, we used the P-REVIEW data to develop a risk-prediction model that could identify patients at risk of a clinically relevant, potentially preventable adverse drug event during admission to the surgical ward, on the day of hospital admission.

Materials and methods

Study setting and population

The P-REVIEW study is a prospective open intervention study designed to investigate whether a multifaceted educational intervention could lead to a reduction of clinically relevant, potentially preventable adverse drug events among patients in surgical wards. The study was performed in two large general teaching hospitals in the Netherlands and has been described in detail elsewhere [10]. Patients who were admitted to the surgical, urological, or orthopaedic ward of one of the two hospitals during a period of six months were included. In case of readmission, patients could be included more than once. Day care patients were excluded. For the development of the prediction model, data were used from patients during the pre-intervention period.

P-REVIEW data set

Data available in the P-REVIEW dataset were collected for each admission, including patient characteristics, drug history, and biochemical, haematological, and microbiological markers. The major part of the data collection was performed automatically from the EHR (Electronic Health Record). In addition, some of the data were collected manually by a review of medical records.

Data from the day of admission of each patient was extracted. If more laboratory values were available of the same variable, the last one, being the most recent value available, was extracted.

Assessment of clinically relevant, potentially preventable adverse drug events that led to death, temporary or permanent disability, increased length of hospital stay, or readmission within 30 days was performed in the P-REVIEW study by teams of experts consisting of a hospital pharmacist and a hospital-based physician, who were not affiliated with the study hospitals. To classify seriousness the National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) index was used. Categories E to I of this index were considered to be clinically relevant. The causality between prescription error and the drug related problem was assessed using the algorithm by Kramer et al. The potential preventability was assessed using the algorithm according to Schumock et al., modified by Lau et al. [10].

Predicting variables

From the data that were collected in the P-REVIEW study, candidate model parameters were selected on the basis of reports in the literature of their association with ADEs [14, 1118]. Thirty-nine risk factors were identified, including patient characteristics (age, gender), department of admission (general surgery versus orthopaedic surgery and urology), type of admission (emergency versus elective); medication (number of medications, use of gastrointestinal drugs, hypoglycemic drugs, vitamin K antagonists, heparin or low molecular weight heparin (LMWH) in therapeutic dose, thrombocyte aggregation inhibitors, cardiovascular drugs in general, cardiac drugs, diuretics, betablockers, renin angiotensin system (RAS) inhibitors, antilipaemica, corticosteroids, antimicrobials, chemotherapy, non-steroidal anti-inflammatory drugs (NSAIDs), opioids, antiepileptics, central nervous system (CNS) agents in general, antipsychotics, anxiolytics, antidepressants, and serious drug-drug interactions); laboratory test results (albumin, glucose, hemoglobin, international normalized ratio (INR), potassium, sodium, leucocytes, chronic kidney disease epidemiology (CKD-EPI), oxygen saturation, positive microbiological blood culture, number of biochemical tests ordered (<20 versus ≥20)) [14, 1118]. Drugs were coded according to the Anatomical Therapeutic and Chemical codes [19]. The glomerular filtration rate was computed by the Chronic Kidney Disease Epidemiology Collaboration formula (CKD-EPI)[20]. Missing data where imputed using the ‘multiple imputation’ procedure from SPSS version 22. SPSS uses a Markov Chain Monte Carlo (MCMC) algorithm known as Fully Conditional Specification (FCS). This method can be used when the pattern of missing data is arbitrary. For each iteration and for each variable in the order specified in the variable list, the FCS method fits a univariate (single dependent variable) model using all other available variables in the model as predictors. Linear regression was used to predict a scale variable and logistic regression to predict categorical variables. Variables of which more than 60% of data were missing were left out of the analysis.

Model development

Model development consisted of two stages [21]. In the first stage, possible predictors were tested using a univariate binary logistic regression model. Variables that were found to be statistically significant (P<0.05) were taken forward to the next stage of multivariate analysis.

In case of variables that in clinical practice can be either too high or too low and confer risk in both situations, categorization of variables was performed (for instance, low potassium values < 3.5 mmol/l, normal values between 3.5 and 5.0 mmol/l and high values > 5.0 mmol/l).

In the second stage, backward and forward elimination procedures were used in multivariate logistic regression analysis in order to detect the best predictors. The removal criterion was set at p = 0.10.

Results from the univariate and multivariate logistic regression models were expressed in terms of the odds ratio with 95% confidence intervals and p-values.

Standardized odd ratios were computed to allow for comparison of the strength of the association between the various continuous variables and the probability of an ADE. Standardization was achieved through Z-transformation.

Model performance

Model performance of the logistic regression model was expressed in the Area Under the Curve (AUC) as computed by a Receiver Operating Characteristics curve analysis (ROC analysis) using the probability as predicted by the regression model and the real outcome (ADE).

Model validation

Bootstrapping was used to assess the internal validity of the model. Two hundred bootstrap samples were drawn to assess the reliability of the model expressed in over-optimism and the uniform shrinkage factor.

Statistical analyses were performed using SPSS Statistics version 22 (IBM Software, New York). Bootstrapping was performed in SAS version 9.4.

Ethics

For all stages of this research, patient records were anonymized prior to analysis in accordance with prevailing privacy regulations.

The institutional review boards of the Isala Hospital and the Meander Medical Center in the Netherlands stated that the study was exempt from ethical approval.

Results

The pre-intervention period of the P-REVIEW dataset study population comprised 6780 admissions of 5940 patients at surgical wards. A clinically relevant, potentially preventable adverse drug event during hospital stay, which led to death, temporary or permanent disability, increased length of hospital stay or readmission within 30 days was determined in 102 patients. The most frequent types of events were haemorrhage (19), arterial or venous thrombosis(7), renal insufficiency, dehydration or electrolyte related events (13), drug intoxication in renal insufficiency (4), central nervous system events (48) and faecal impaction (11). Characteristics of patients who did, and those who did not experience an ADE during hospital stay are shown in Table 1.

Univariate analysis

The candidate predictive variables of a clinically relevant ADE during hospital stay and results of the univariate analysis are reported in Table 2.

Multivariate analysis

The significant variables (p<0.05) were identified from the univariate analysis and taken forward to the next stage.

In all 20 imputed datasets, both backward and forward elimination procedure in multivariate analysis identified the same 5 variables that were significantly associated with the risk of developing a clinically relevant adverse drug event. It appeared that these 5 predictor variables did not have missing values. Therefore, the final analysis could be performed with the original, complete dataset.

The coefficients (ß), standard errors (SE), and odds ratios (OR) with 95% confidence intervals of the variables are shown in Table 3. The final model comprised age, number of biochemical tests ordered, heparin/LMWH in therapeutic dose, use of opioids, and use of cardiovascular drugs at the time of hospital admission.

thumbnail
Table 3. Coefficients, standard errors and Odds Ratios (OR) with 95% confidence intervals (CI) of the five variables of the final model.

The OR as found after applying the shrinkage factor is also given.

https://doi.org/10.1371/journal.pone.0201645.t003

The AUC of the model was 0.858 (95% CI 0.831–0.884). The ROC curve is shown in Fig 1. The validation of the model using bootstrap samples showed an over-optimism of 0.008 and a shrinkage factor of 0.960, leading to a corrected performance of the model of 0.85 (Table 3).

When the cut-off point for a high risk of developing an ADE was set at 1.6%, the model showed a sensitivity of 80.4% and a specificity of 73.4%. At this cut-off level, positive and negative predictive value were 4.5% and 99.6%, respectively.

The formula of the logistic regression model that allows for calculating the individual risk of a clinically relevant adverse drug event to a surgical patient is shown below.

Formula individual risk = 1/(1 + exp(-1*LP))

Where LP is the so called Linear Predictor which for our model is defined as:

Linear Predictor = 10.082 +

0.060 * age (in years) +

0.770 (in case number of biochemical tests ordered ≥ 20) +

0.604 * heparin/LMWH (in therapeutic dose) +

1.261 (in case the patient uses cardiovascular drugs) +

0.785 (in case the patient uses opioids)

Discussion

The risk prediction model resulting from this study helps to identify surgical patients that are at increased risk of sustaining a clinically relevant potentially preventable adverse drug event. This model uses five clinical variables that can be obtained routinely on hospital admission, and that can be incorporated into clinical practice as a tool to target patients that can benefit from interventions aimed at reducing potentially preventable clinically relevant adverse drug events during hospital stay in surgical wards.

The model was developed using data from the pre-intervention phase of the P-REVIEW study, which assessed clinically relevant, potentially preventable ADE among patients at surgical wards [10]. This is the first study that developed a risk prediction model focusing on clinically relevant potentially preventable ADE instead of adverse drug reactions in general. This study only targeted surgical patients, which leads to a better model performance. We only included variables that are available on the first day after hospital admission to be able to operationalize a model to predict risk during hospital stay, immediately after hospital admission. Therefore, variables such as length of hospital stay, interdisciplinary consultation or admission to the intensive care unit during hospital stay, were not included in our analysis.

Our final model contains five variables; age of the patient, number of biochemical tests ordered, treatment with heparin/LMWH in therapeutic dose, treatment with opioids, and treatment with cardiovascular drugs. Our model shows an acceptable level of fit and discrimination performance. We can use the model not only to label patients at risk of experiencing a drug-related event but also to label patients that are very unlikely to experience an event. The negative predictive value of our model is very high. Therefore, one could use this model to also identify patients for whom automated computerized clinical decision support without surveillance by the hospital pharmacist is sufficient.

An overview of studies of development and validation of risk prediction models for ADR or ADE is shown in Table 4. Other studies showed different predictive variables and found an area under the receiver operator characteristic curve (AUROC), varying between 0.70–0.74, with relatively low sensitivity and specificity scores. We hypothesized that focusing on patient cohorts with restricted clinical characteristics (surgical patients) and focusing on clinically relevant, potentially preventable adverse drug events, instead of adverse drug reactions in general, would lead to a better performing prediction model.

thumbnail
Table 4. Overview studies of development and validation of risk prediction models for ADR or ADE.

https://doi.org/10.1371/journal.pone.0201645.t004

In our study, the strongest independent risk factor was the age of the patients. As shown in many other studies, advancing age is associated with an increased number of comorbidities in association with polypharmacy resulting in an increased risk of ADEs [2, 3, 13].

This is the first study that explores the number of biochemical tests ordered as a predictive factor for drug related events. We hypothesized that besides the use of specific laboratory values as electronic triggers, the number of biochemical tests might also be useful in identifying a risk of ADE during hospital admission. Before diagnosing the patient’s condition, the physician may suspect a possible adverse outcome. This suspicion alerts the physician and leads to increased laboratory orders to clarify the patient’s condition and to prevent serious problems. Moreover, deviant laboratory values will lead to further monitoring [22]. Consequently, the number of biochemical tests might be a useful electronic trigger to identify a patient at risk of ADE, pointing out an acute unstable clinical situation [16, 23]. In our study this factor has proven to be a strong predictor for drug related problems in surgical wards.

Surgery in patients who use anticoagulation therapy, is a challenge. Guidelines on perioperative management of anticoagulation are complicated and people are at risk of either bleeding or thrombosis as a result of the surgical procedures. For patients with a high risk of thromboembolism, measured with a CHA2DS2-VASC score, needing certain surgical procedures, it is necessary to perform bridging with LMWH at therapeutic dose during the perioperative period. If this procedure is not correctly performed, these patients are at high risk of serious ADEs. Pardo Carbello et al. already demonstrated that the most frequent fatal ADE is haemorrhage [24].

We found that use of cardiovascular drugs and opioids are significant risk factors in surgical patients. Cardiovascular drugs are generally used by a vulnerable older population with multiple co-morbidities. Use of opioids often indicates (temporary) severe morbidity. The literature has shown that patients using these drugs have an increased risk of experiencing ADEs [15, 25].

This study has several strengths. The P-REVIEW database contains information on clinically relevant potentially preventable drug related events (leading to death, temporary or permanent disability, increased hospital stay or readmission) in a very large cohort. Aiming at prevention of patient harm, it seems more rational to focus on clinically relevant, potentially preventable adverse events, instead of adverse reactions in general. We focused on patients in surgical wards. By studying patient cohorts with restricted clinical characteristics the performance of risk prediction models will improve. Furthermore, we used variables that can be routinely obtained on hospital admission, so implementation can easily be operationalized.

There are some limitations to our study. In our analysis, we used potentially predictive variables that were available in the P-REVIEW database. Mostly, these data were automatically obtained from hospital databases [10]. The lack of automated coded information for some variables limited the use of these variables, e.g. information on co-morbidities using the International Classification of Diseases (ICD-10), frailty or cognitive impairment, reason for hospital admission or type of surgery performed. We internally validated our model by bootstrapping but we have not yet been able to externally validate this model in clinical practice. Because patients could be included more than once, in case of readmission, this may have led to an immortal time bias. However, since only a small patients were admitted several times in the study, we think this bias was very limited.

The development of our risk prediction model has potentially important implications for clinical practice and research. By using this model, lower-risk patients could be managed less extensively (for instance, only automatically using CPOE/CDSS), whereas higher risk patients could receive more intensive interventions, such as medication review, aimed at reducing drug-related adverse outcomes. Such selective use of ancillary precautions could also help to improve the cost-effectiveness of medication safety interventions. In that way, this risk model, which combines clinical and medication related variables can guide clinical intervention, delivered as part of an integrated system built on the principles of medication safety [5]. Future research should confirm whether intensive pharmacovigilance of high risk patients really leads to less adverse drug events in hospitalized patients. Our risk model can be incorporated into a CPOE system and thereby generate automatic risk-evaluation based on patients’ medical records upon hospital admission. Above a pre-specified cut-off point, the score can assist hospital pharmacists or prescribing physicians in their decisions to review the patient’s medication or to perform other relevant interventions. Under the cut-off point (not necessarily the same cut-off point), it may be possible for the hospital pharmacist to prioritize efforts in medication safety interventions and rely, when possible, on an automatic medication safety system.

Conclusions

A risk prediction model was developed to identify surgical patients at risk of experiencing a clinically relevant, potentially preventable adverse drug event during hospital admission. The resulting model contains five variables: age of the patient, number of biochemical tests ordered, treatment with heparin/LMWH in therapeutic dose, treatment with opioids, and treatment with cardiovascular drugs. This model can be used to guide the hospital pharmacist and the physician to effectively and efficiently implement clinical interventions to improve medication safety.

References

  1. 1. Davies EC, Green CF, Taylor S, Williamson PR, Mottram DR, Pirmohamed M. Adverse drug reactions in hospital in-patients: a prospective analysis of 3695 patient-episodes. PLoS One. 2009;4(2):e4439. pmid:19209224
  2. 2. Hoonhout LHF, de Bruijne MC, Wagner C, Asscheman H, van der Wal G, van Tulder MW. Nature, occurrence and consequences of medication-related adverse events during hospitalization: a retrospective chart review in the Netherlands. Drug safety. 2010;33(10):853–64. pmid:20812770
  3. 3. Krahenbuhl-Melcher A, Schlienger R, Lampert M, Haschke M, Drewe J, Krahenbuhl S. Drug-related problems in hospitals: a review of the recent literature. Drug safety. 2007;30(5):379–407. pmid:17472418
  4. 4. Onder G, Petrovic M, Tangiisuran B, Meinardi MC, Markito-Notenboom WP, Somers A, et al. Development and validation of a score to assess risk of adverse drug reactions among in-hospital patients 65 years or older: the GerontoNet ADR risk score. Archives of internal medicine. 2010;170(13):1142–8. pmid:20625022
  5. 5. Stevenson JM, Williams JL, Burnham TG, Prevost AT, Schiff R, Erskine SD, et al. Predicting adverse drug reactions in older adults; a systematic review of the risk prediction models. Clin Interv Aging. 2014;9:1581–93. pmid:25278750
  6. 6. Yourman LC, Lee SJ, Schonberg MA, Widera EW, Smith AK. Prognostic indices for older adults: a systematic review. JAMA. 2012;307(2):182–92. pmid:22235089
  7. 7. McElnay JC, McCallion CR, Al-Deagi F, Scott MG. Development of a risk model for adverse drug events in the elderly. Clin Drug Invest. 1997;13(1):47–55.
  8. 8. Tangiisuran B, Scutt G, Stevenson J, Wright J, Onder G, Petrovic M, et al. Development and validation of a risk model for predicting adverse drug reactions in older people during hospital stay: Brighton Adverse Drug Reactions Risk (BADRI) model. PLoS One. 2014;9(10):e111254. pmid:25356898
  9. 9. Trivalle C, Burlaud A, Ducimetiere P. Risk factors for adverse drug events in hospitalized elderly patients: a geriatric score. Eur Geriatric Med. 2011;2:284–9.
  10. 10. Bos JM, van den Bemt PM, Kievit W, Pot JL, Nagtegaal JE, Wieringa A, et al. A multifaceted intervention to reduce drug-related complications in surgical patients. British journal of clinical pharmacology. 2017;83(3):664–77. pmid:27670813
  11. 11. Bates DW, Miller EB, Cullen DJ, Burdick L, Williams L, Laird N, et al. Patient risk factors for adverse drug events in hospitalized patients. ADE Prevention Study Group. Archives of internal medicine. 1999;159(21):2553–60. pmid:10573045
  12. 12. Dequito AB, Mol PG, van Doormaal JE, Zaal RJ, van den Bemt PM, Haaijer-Ruskamp FM, et al. Preventable and non-preventable adverse drug events in hospitalized patients: a prospective chart review in the Netherlands. Drug safety. 2011;34(11):1089–100. pmid:21981436
  13. 13. Evans RS, Lloyd JF, Stoddard GJ, Nebeker JR, Samore MH. Risk factors for adverse drug events: a 10-year analysis. The Annals of pharmacotherapy. 2005;39(7–8):1161–8. pmid:15897265
  14. 14. Fijn R, Van den Bemt PM, Chow M, De Blaey CJ, De Jong-Van den Berg LT, Brouwers JR. Hospital prescribing errors: epidemiological assessment of predictors. British journal of clinical pharmacology. 2002;53(3):326–31. pmid:11874397
  15. 15. Suggett E, Marriott J. Risk Factors Associated with the Requirement for Pharmaceutical Intervention in the Hospital Setting: A Systematic Review of the Literature. Drugs—real world outcomes. 2016;3(3):241–63. pmid:27747829
  16. 16. Tegeder I, Levy M, Muth-Selbach U, Oelkers R, Neumann F, Dormann H, et al. Retrospective analysis of the frequency and recognition of adverse drug reactions by means of automatically recorded laboratory signals. British journal of clinical pharmacology. 1999;47(5):557–64. pmid:10336581
  17. 17. van den Bemt PM, Egberts AC, Lenderink AW, Verzijl JM, Simons KA, van der Pol WS, et al. Risk factors for the development of adverse drug events in hospitalized patients. Pharmacy world & science: PWS. 2000;22(2):62–6. pmid:10849925
  18. 18. Wilmer CM HV, Natsch S, Rennings AJM, van den Bemt, Bos JM. Drug-related problems in a clinical setting: a literature review and cross-sectional study evaluating factors to identify patients at risk. European Journal of Hospital Pharmacy. 2015;22(4):229–35.
  19. 19. Pahor M, Chrischilles EA, Guralnik JM, Brown SL, Wallace RB, Carbonin P. Drug data coding and analysis in epidemiologic studies. Eur J Epidemiol. 1994;10(4):405–11. pmid:7843344
  20. 20. Teruel Briones JL, Gomis Couto A, Sabater J, Fernandez Lucas M, Rodriguez Mendiola N, Villafruela JJ, et al. Validation of the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) equation in advanced chronic renal failure. Nefrologia. 2011;31(6):677–82. pmid:22130283
  21. 21. Steyerberg E.W. Clinical Prediction Models. A practical approach to development, validation and updating. Springer 2009.
  22. 22. Stolper E, van Bokhoven M, Houben P, Van Royen P, van de Wiel M, van der Weijden T, et al. The diagnostic role of gut feelings in general practice. A focus group study of the concept and its determinants. BMC Fam Pract. 2009;10:17. pmid:19226455
  23. 23. Ramirez E, Carcas AJ, Borobia AM, Lei SH, Pinana E, Fudio S, et al. A pharmacovigilance program from laboratory signals for the detection and reporting of serious adverse drug reactions in hospitalized patients. Clin Pharmacol Ther. 2010;87(1):74–86. pmid:19890254
  24. 24. Pardo Cabello AJ, Gonzalez Contreras LG, Manzano Gamero MV, Gomez Jimenez FJ, Puche Canas E. Prevalence of fatal adverse drug reactions in hospitalized patients. Int J Clin Pharmacol Ther. 2009;47(10):596–602. pmid:19825322
  25. 25. Saedder EA, Brock B, Nielsen LP, Bonnerup DK, Lisby M. Identifying high-risk medication: a systematic literature review. Eur J Clin Pharmacol. 2014;70(6):637–45. pmid:24671697