https://orcid.org/0000-0002-6651-3816
Figures
Abstract
Aim
Real-world predictors of major bleeding (MB) have been well-studied among warfarin users, but not among all direct oral anticoagulant (DOAC) users diagnosed with atrial fibrillation (AF). Thus, our goal was to build a predictive model of MB for new users of all oral anticoagulants (OAC) with AF.
Methods
We identified patients hospitalized for any cause and discharged alive in the community from 2011 to 2017 with a primary or secondary diagnosis of AF in Quebec’s RAMQ and Med-Echo administrative databases. Cohort entry occurred at the first OAC claim. Patients were categorized according to OAC type. Outcomes were incident MB, gastrointestinal bleeding (GIB), non-GI extracranial bleeding (NGIB) and intracranial bleeding within 1 year of follow-up. Covariates included age, sex, co-morbidities (within 3 years before cohort entry) and medication use (within 2 weeks before cohort entry). We used logistic-LASSO and adaptive logistic-LASSO regressions to identify MB predictors among OAC users. Discrimination and calibration were assessed for each model and a global model was selected. Subgroup analyses were performed for MB subtypes and OAC types.
Results
Our cohort consisted of 14,741 warfarin, 3,722 dabigatran, 6,722 rivaroxaban and 11,196 apixaban users aged 70–86 years old. The important MB predictors were age, prior MB and liver disease with ORs ranging from 1.37–1.64. The final model had a c-statistic of 0.63 (95% CI 0.60–0.65) with adequate calibration. The GIB and NGIB models had similar c-statistics of 0.65 (95% CI 0.63–0.66) and 0.67 (95% CI 0.64–0.70), respectively.
Conclusions
MB and MB subtype predictors were similar among DOAC and warfarin users. The predictors selected by our models and their discriminative potential are concordant with published data. Thus, these models can be useful tools for future pharmacoepidemiologic studies involving older oral anticoagulant users with AF.
Citation: Qazi JZ, Schnitzer ME, Côté R, Martel M-J, Dorais M, Perreault S (2021) Predicting major bleeding among hospitalized patients using oral anticoagulants for atrial fibrillation after discharge. PLoS ONE 16(3): e0246691. https://doi.org/10.1371/journal.pone.0246691
Editor: Gianluigi Savarese, Karolinska Institutet, SWEDEN
Received: November 4, 2020; Accepted: January 23, 2021; Published: March 3, 2021
Copyright: © 2021 Qazi et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: Data cannot be shared publicly because of privacy/ethical restrictions on account of provincial “Commission d’accès à l’information du Gouvernement du Québec” law. They can be contacted via cai.communications@cai.gouv.qc.ca for details. The same restriction applies to any minimal data set. However, the global prediction model was programmed into an interactive web application using the open-source R package “Shiny” and deployed for public use on https://jakubqazi.shinyapps.io/ShinyApp/?fbclid=IwAR33Slu0Zp-8JDs8qPWDFJPTbWM5qTI-3--b5sjIR75_dNWk44lURIS-vIk. The code base for this application is available at: https://github.com/jqazi95/Major_Bleeding_Risk_Calculator_For_Real-World_Oral_Anticoagulant_Users_With_AF/blob/main/MBriskApp.Rmd.
Funding: The study was supported by the Heart and Stroke Foundation of Canada (G-17-0018326) and the Réseau Québécois de Recherche sur les Médicaments (RQRM). Please refer to the following URL: https://www.heartandstroke.ca/. Jakub Qazi’s training was supported by the Fond de Recherche du Quebec - Santé Master’s Training bursary (bursary number 272652). Please refer to the following URL: http://www.frqs.gouv.qc.ca/en/. The study findings were presented with posters at the 2020 Pharmaceutical Sciences World Congress and 2020 International Conference on Pharmacoepidemiology (ICPE) – all access, both of which were funded by conference-specific bursaries. MJM works for Xcenda UK Ltd. MD, the sole founder and representative of StatSciences Inc., provided us with independent biostatistics consultancy. These funders provided support in the form of salaries for authors MJM and MD. The specific roles of these authors are articulated in the ‘author contributions’ section. All funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: MJM works for Xcenda UK Ltd and MD is the sole founder and representative of StatSciences Inc. There are no patents, products in development or marketed products associated with this research to declare. This does not alter our adherence to PLOS ONE policies on sharing data and materials.
Introduction
Atrial fibrillation (AF) is the most common cardiac arrhythmia worldwide with increasing incidence due to the aging population [1–3]. It is associated with 5-fold and 3-fold increases in the risk of stroke and systemic embolism, respectively, with AF-associated stroke showing twice the risk of thirty-day all-cause mortality relative to non-AF associated stroke [4–6]. Before 2010, the vitamin K antagonist, warfarin, was the only medication used for stroke and systemic embolism prevention for AF patients at moderate and high risk of these outcomes [7–9]. However, warfarin is associated with a high risk of major bleeding (MB; 7.2 per 100 person-years), of which the most common type is gastrointestinal bleeding (GIB) and the most lethal type, intracranial hemorrhage (ICH) [9, 10]. In 2010, the first of the direct oral anticoagulants (DOAC) received approval from the US Food and Drug Administration for stroke prevention in patients diagnosed with atrial fibrillation (AF). In addition to circumventing the need for INR, the DOACs (dabigatran, rivaroxaban, apixaban and edoxaban) presented pharmacokinetic, pharmacodynamic and safety advantages over warfarin [9].
The four DOAC clinical trials for AF, namely RE-LY, ROCKET-AF, ARISTOTLE and ENGAGE-AF, concluded non-inferior (or superior, in the case of ARISTOTLE) efficacy in reducing stroke, systemic embolism and all-cause mortality rates for each DOAC relative to warfarin and a lower risk of MB for all DOACs [11–16]. Given that randomized clinical trials (RCTs) do not account for real-world patient characteristics, pharmacoepidemiologic studies were required to complement and confirm RCT findings. According to meta-analyses of observational studies, DOAC effectiveness and safety with respect to MB risk was equivalent to warfarin’s [17, 18]. Additionally, pooled DOAC analyses were associated with a greater GIB risk and lower ICH risk in patients over 75 years old [17, 18]. However, apixaban was the only DOAC with an associated lower risk of MB, GIB, and ICH relative to warfarin. It also had an associated lower risk of MB relative to the other DOACs [17, 19, 20]. Within each DOAC subgroup, significant heterogeneity existed in at least one of the bleeding outcomes (MB, ICH or GIB) [17, 21, 22].
To ensure oral anticoagulant (OAC) safety, the risk-benefit profile needs to be carefully assessed while taking into account factors associated with a predisposition to bleeding [9]. The HAS-BLED, a scoring system used to identify patients at risk of bleeding, was developed based on warfarin user data and validated among rivaroxaban users [23, 24]. Since then, other MB prediction scores have been developed to improve bleeding prediction within this population. The HEMORR2AGES and ATRIA scores were derived from warfarin user data, while the ORBIT-AF also accounted for dabigatran user data. Ultimately, the ABS score was derived from DOAC and warfarin user data [9, 25–28]. However, given that the HAS-BLED is still the most commonly used score, a user-friendly MB prediction tool derived from a recent population of OAC users is essential.
Moreover, the HAS-BLED and other prediction models were developed to predict any MB, but it is also of interest to establish risk factors for specific MB subtypes, GIB, non-GI extracranial bleeding (NGIB) and ICH [9, 25–28]. The lack of prediction models for MB subtypes, and the lack of studies identifying MB subtype-specific predictors makes it difficult to accurately monitor MB and actively engage in their prevention [29, 30]. Specifically, we aimed to develop predictive models for MB and for the most prevalent MB subtypes (GIB and NGIB) based on data from real-world patients with AF taking any type of OAC. Therefore, our primary objective is to establish a model to predict MB in a population of all OAC users with AF. Our second objective is to identify important predictors of the most prevalent MB subtypes (GIB and NGIB). Our third objective is to compare the predictors of MB between warfarin and DOAC users as well as doing so with the MB subtypes. Our final objective is to evaluate the discriminative potential of the MB model fit to all OAC users for GIB and NGIB.
Methods
Data source
Administrative databases have proven to be a widely available and useful tool for pharmacoepidemiologic studies [31, 32]. The data for our study were compiled from a subset of the Régie de l’Assurance Maladie du Québec (RAMQ) drug and medical services database linked to the Med-Echo hospitalization database using encrypted patient healthcare insurance numbers [31, 33–36]. Quebec prescription and hospitalization data have been shown to have a high degree of completeness (with only 0 to 0.4% of data that was missing) and accuracy [31]. Thus, our cohort did not have any missing data.
Population-based cohort definition
We conducted a cohort study using drug claims and diagnostic coding data from the Quebec RAMQ and Med-Echo administrative databases. We identified adult patients who were hospitalized for all cause and discharged alive in the community from January 1, 2011 to December 31, 2017 with a primary or a secondary diagnosis of AF. They were identified using ICD-9 (427.3, 427.31 or 427.32) or ICD-10 (I48) codes [37, 38]. For patients with more than one admission with an AF diagnosis, we used the first date of admission. The ICD-9 codes displayed median positive predictive values of 89% and 95.7% in two distinct validation studies [37, 38].
Patients included in the cohort had to have a filled prescription of at least one of the DOACs (dabigatran, rivaroxaban and apixaban) or warfarin in the year following hospitalization, but could not have used any OAC one year prior to this claim. For this reason, they also had to have continuous RAMQ drug plan coverage for at least one year prior to cohort entry (see Fig 1). The date of cohort entry (or study index) was defined as the first filled OAC prescription after hospital discharge.
AF: atrial fibrillation; OAC: oral anticoagulant, DOAC: direct oral anticoagulant, RAMQ: Régie d’Assurance-Maladie du Québec.
We excluded patients with OAC contraindications (end‐stage chronic renal disease [ESRD] or dialysis for a minimum of 3 months) followed by kidney transplantation within 3 years before cohort entry. We also excluded patients with a non-AF indication for DOAC anticoagulation such as post-orthopedic surgery (hip or knee replacement 6 weeks before cohort entry) and a diagnosis of venous thromboembolism (defined as either deep vein thrombosis or pulmonary embolism) during the hospitalization period. Finally, we excluded those having undergone cardiac valve replacement up to 5 years prior to cohort entry.
Oral anticoagulant exposure
OAC exposure was defined as filing a new claim for warfarin or a DOAC (all dosages approved in Canada included) after hospital discharge. Given that the database had very few users of edoxaban, these patients were not included in our cohort. Patient treatment initiation was determined using dispensation dates of the OAC prescriptions. All individuals were new users, i.e., individuals who had not been exposed to any OAC at least one year prior to cohort entry.
Study outcomes
The primary outcomes were MB including GIB, NGIB and ICH. MB, GIB, NGIB and ICH were defined as the first instance of each respective bleeding event leading to a hospitalization during follow-up and identified using ICD-9 and ICD-10 codes from inpatient claims (S1 Table). These outcomes were defined using 6 distinct observational studies [39–45]. When multiple of either MB subtypes occurred, only the first of that respective MB subtype was evaluated as the primary outcome (e.g. GIB was defined as the first GIB during the follow-up period). These codes have been externally validated with positive predictive value ranging from 85% to 95% [46–48]. Patient follow-up began from the first OAC claim until the earliest occurrence of one of the following events: MB event, end of coverage of the RAMQ drug insurance, date of death, 1 year of follow-up or end of the study.
Baseline characteristics and predictor candidates
Sociodemographic variables (age, sex, and material and social deprivation indices) were defined at cohort entry [49]. Associated morbidities were assessed up to 3 years prior to cohort entry. They included stroke/transient ischemic attack, hypertension, dyslipidemia, cardiomyopathy, coronary artery disease, acute myocardial infarction, peripheral vascular disease (PVD), chronic heart failure, anemia, chronic kidney disease (CKD), severe kidney disease (creatinine clearance < 30 ml /min), acute renal failure, liver disease, diabetes mellitus, asthma and chronic obstructive pulmonary disease (COPD), history of MB, and prior Helicobacter Pylori infection [40, 50, 51]. The CHA2DS2-VASc score (stroke risk), a modified HAS-BLED (bleeding risk) excluding labile INR, and the Charlson-Deyo comorbidity index, were assessed up to 3-years prior to cohort entry (S2 and S3 Tables for coding algorithms). Finally, we documented baseline medication use, which included antiplatelets, proton pump inhibitors (PPIs), non-steroidal anti-inflammatory agents (NSAIDs), digoxin, amiodarone, antidepressants, β-blockers, calcium channel blockers, inhibitors of renin-angiotensin system, diuretics, loop diuretics, antidiabetics up to 2 weeks prior to cohort entry.
Statistical analyses
First, we generated descriptive data for warfarin, DOAC and OAC new users with and without GIB, NGIB and MB. We calculated percentages for binary and categorical variables and means with standard deviations for continuous ones.
We determined the cumulative incidence of MB, GIB, NGIB and ICH (events per 100 person-years), respectively. We then generated Kaplan-Meier curves for each dose-stratified OAC treatment group to assess cumulative MB, GIB and NGIB incidences within the first year after cohort entry. We used the log rank test to compare each of the MB, GIB and NGIB cumulative incidences of each DOAC treatment group to those of warfarin users.
We selected candidate variables to be evaluated as predictors of any MB or MB subtypes based on availability in our dataset and clinical relevance, which was defined as inclusion in bleeding scores, significant differences in baseline measurements, or a strong association with MB based on narrative review [25, 29, 52]. We used the Least Absolute Shrinkage and Selection Operator (LASSO) method, which introduces a penalty/bias to each coefficient of a regression model to select relevant predictors and to minimize overfitting, and the adaptive LASSO (adaLASSO), which uses the same principle while applying a larger penalty to smaller coefficients than to larger ones [53, 54].
Both LASSO and adaLASSO penalties can be incorporated into logistic regression (logistic-LASSO and logistic-adaLASSO, respectively), which perform well when the true model is sparse [53, 54]. Given that the 10 events per predictor rule, proposed to be too conservative for penalty-based regression, was respected for each outcome in the OAC models, we deemed the sample size of this cohort to be sufficiently large to derive robust prediction models (S4 Table) [55]. Most notably, all available data were used to maximize the power and generalizability of the results.
For each outcome, we calculated odds ratios (ORs) for each covariate for the warfarin, DOAC and OAC treatment groups using logistic-LASSO and logistic-adaLASSO regressions (R v3.6.2, package “glmnet”). We did not include 95% confidence intervals (CIs) as it is challenging to interpret them in log-LASSO and log-adaLASSO modelling. We calculated cross-validated concordance statistics (c-statistics) and their 95% CIs using the area under Receiving Operator Curves (auROC) to determine model discrimination (R v3.6.2, package cvAUC) [56]. Finally, the calibration of each model was quantitatively and qualitatively characterized using Hosmer-Lemeshow tests, a chi-squared test of mean squared differences of true and predicted outcomes between quantiles of outcome measurements, and their corresponding calibration plots (R v3.6.2, packages “generalhoslem” and “PredictABEL”) [56]. We then identified the best model, defined as having the best discrimination value, adequate calibration and having selected the least variables within each OAC subgroup (warfarin, DOAC and OAC). Ultimately, we evaluated the final MB model’s performance and evaluated its ability to detect MB subtypes (GIB and NGIB) via discrimination and calibration testing using the previously discussed methods.
Results
Demographic and clinical characteristics
The cohort of OAC new users diagnosed with AF that have met all inclusion and exclusion criteria comprised of 36,381 patients. The two treatment subgroups consisted of warfarin users (n = 14,741) and DOAC users (n = 21,640). The mean age of patients who experienced bleeding during follow-up and those that did not ranged from 78.9 to 80.9 years old as shown in Table 1. Whether or not they experienced MB, OAC users were more likely to be over the age of 75 (68.3% to 77.4%), had numerous comorbidities (Charlson-Deyo co-morbidity scores from 4.5±3.4 to 5.9±3.9), had a high stroke risk (CHA2DS2-VASc scores from 3.7±1.4 to 4.0±1.3) and had a high bleeding risk (HAS-BLED scores from 3.1±1.3 to 3.5±1.3), as shown in Table 1. Patients who experienced MB within the year of follow-up were more likely to be over 75 years old (76.1%), had over 5 comorbidities on average (Charlson-Deyo score: 5.3 ± 3.6), a high bleeding risk (HAS-BLED: 3.4 ± 1.2) and a high stroke risk (CHA2DS2-VASc: 4.0 ± 1.3). Warfarin and DOAC users had a total of 499 and 528 MB events, respectively (Table 1; S5 and S6 Tables).
Treatment-specific cumulative incidence measurements
Including both approved dosages, DOAC users had cumulative ICH, GIB, NGIB, and MB incidences ranging from 0.35 to 0.92, 0.89 to 1.80, 0.64 to 1.77 and 2.11 to 4.27 events per 100 person-years, respectively (Table 2). Warfarin users had cumulative ICH, GIB, NGIB and MB incidences of 1.05, 1.57, 1.28 and 2.84 events per 100 person-years, respectively (Table 2). As shown in Figs 2 and 3, apixaban users had lower incidences of all bleeding subtypes relative to warfarin users for both dosages (log rank p<0.05).
Warfarin, dabigatran, rivaroxaban and apixaban are shown in black, red, blue and purple, respectively. Gastrointestinal, non-gastrointestinal and all major bleeding are shown from left to right. * statistically significant difference relative to warfarin (p<0.05).
Warfarin, dabigatran, rivaroxaban and apixaban are shown in black, red, blue and purple, respectively. Gastrointestinal, non-gastrointestinal and all major bleeding are shown from left to right. * statistically significant difference relative to warfarin (p<0.05).
Logistic-LASSO and logistic-adaLASSO prediction models
The ORs of the selected predictors for the warfarin, DOAC and OAC models assessing GIB, NGIB and MB under the logistic-LASSO and logistic-adaLASSO regressions are presented in S7 and S8 Tables, respectively. The models for GIB, NGIB and MB had concordance statistics ranging from 0.60 (95% CI 0.58–0.62) to 0.66 (95% CI 0.63–0.70) with no statistically significant difference between logistic-LASSO and logistic-adaLASSO models (S7 and S8 Tables, S2 Fig). All models were adequately calibrated (Hosmer Lemeshow test: p>0.05) except for the logistic-LASSO selected OAC model for NGIB (S7 and S8 Tables, S1 Fig). There was little difference in discrimination or calibration between logistic-LASSO selected models and their logistic-adaLASSO counterparts. This was the case for all treatment groups and outcomes (S7 and S8 Tables, S1 and S2 Figs).
With the exception of NGIB, the predictors of each bleeding outcome were similar between the DOAC and warfarin treatment groups. Since the logistic-LASSO MB model derived from OAC user data selected marginally less variables than the logistic-adaLASSO MB model and the performance of the models did not differ significantly across methods, we chose the former as the final model fit. The most important MB predictors in our final MB model were liver disease (OR = 1.64), MB history (OR = 1.57), age ≥ 75 vs < 75 (OR = 1.37) antiplatelet use (OR = 1.28), cardiomyopathy (OR = 1.22), PVD (OR = 1.21) and COPD (OR = 1.21).
The selected model had a c-statistic of 0.63 (95% CI 0.61–0.65) and was well-calibrated (Table 3). The formula representing this model can be seen in Table 3. The final MB model performed just as well in detecting GIB and NGIB as it did for MB (GIB c-statistic: 0.65, 95% CI 0.63–0.66; NGIB c-statistic: 0.67, 95% CI 0.64–0.70; Table 3). However, with regards to calibration, the model underpredicted GIB and NGIB among patients at moderate and high risk of each respective MB subtype (see S3 Fig). To understand how to apply and interpret the selected model, you may refer to the formula for the risk of major bleeding in the year following OAC initiation derived for any OAC new user with AF (Table 3).
Discussion
Our study is the first to derive prediction models for MB and MB subtypes from a cohort of DOAC and warfarin new users with AF. It did so using a robust statistical prediction tool. Our MB and MB subtype models were well-calibrated and performed similarly to previously published MB scores. Warfarin and DOAC users presented similar predictors of MB and GIB, not NGIB. This was likely due to the variable locations of bleeding included in the definition of NGIB. We then built a final MB model derived from data from all OAC users. Due to the marginally superior discrimination of the OAC model relative to the warfarin model, it was deemed that the OAC model was more useful than having separate models for DOAC and warfarin users. The most important MB predictors in our final MB model were liver disease, MB history, age≥75, antiplatelet use, cardiomyopathy, PVD and COPD with ORs ranging from 1.21 to 1.64. Notably, the selection of apixaban as a protective factor (OR = 0.69) relative to warfarin corroborates previous observational studies [57, 58]. These findings may be attributable to the superior bleeding profile of apixaban relative to warfarin.
The OR values for the most important predictors of our final model were largely similar to those reported in the analyses used to derive existing MB scores. For the ABS, the population had a similar stroke risk, but was younger (mean age ranging from 68.1 to 73.7) and less at risk of bleeding (mean HAS-BLED ranging from 2.1 to 2.8). The ABS score, which, like us, was derived from OAC users, selected analogous predictors to our model, including prior MB (HR = 1.27, 95% CI 1.18–1.36), antiplatelet therapy (HR = 1.25, 95% CI 1.16–1.35), and COPD (HR = 1.21, 95% CI 1.13–1.30). The most important difference between our model and the ABS score is their selection of CKD. This difference is most likely due to the continuous definition of age given the association between our age categories, kidney function as well as OAC prescription guidelines.
Furthermore, the ORBIT-AF population had a similar age to ours, but a higher stroke risk (a median CHA2DS2-VASC ranging from 4.0 to 5.0) and lower bleeding risk (a median HAS-BLED of 2.0). The analyses used to create the ORBIT-AF score used warfarin and dabigatran user data, provided similar point estimates and predictors such as age≥75 (HR = 1.38, 95% CI 1.17–1.61), any prior bleeding excluding NGIB (HR = 1.73, 95% CI 1.34–2.23), and antiplatelet therapy (HR = 1.51, 95% CI 1.30–1.75). Like with the ABS score, the selection of CKD is a major distinction to our model. This may be due to their prediction method, the omission of NGIB in the MB history definition or the lower bleeding risk of the derivation cohort.
On the other hand, for each existing MB score, we found differences between some of their OR values and our own. Most notably, the HAS-BLED study presented a significantly different OR estimate for prior MB (OR = 7.51, 95% CI 3.00–18.78), while all other models selected CKD and omitted liver disease. The CKD discrepancy is most likely due to the contraindication of DOAC use among patients with renal dysfunction in our cohort. Moreover, the high prior MB point estimate may be attributable to the small sample size or selection bias attributable to the substantial missing data. However, despite these differences to our model, the HAS-BLED similarly incorporated age≥65 (OR = 2.66, 95% CI 1.33–5.32). Given that the HAS-BLED was derived from warfarin data, it may exclude important MB predictors among DOAC users, hence the need for a score that is derived from a cohort encompassing all types of OAC users.
Our model performed similarly to other MB scores in the literature with a c-statistic of 0.63 (0.60–0.65) and had adequate calibration. The HAS-BLED, (c-statistic: 0.65 [0.61‐0.69]) performed better than existing scores in a meta-analysis of observational studies (c-statistics of 0.63 (0.61‐0.66) and 0.63 (0.56‐0.72) for HEMORR2AGES and ATRIA, respectively) with Net Reclassification and Integrated Discrimination Improvement values exceeding 7% (p<0.001) [24, 59–62]. However, unlike our model, few of the studies used cross-validation or bootstrapping to evaluate model performance, which may have led to overconfident assessments if the models were not independently validated [24, 59–63]. Although our model performed similarly to the HAS-BLED, we evaluated its discrimination more robustly and the HAS-BLED was inadequately calibrated [64]. MB prediction scores, such as the ORBIT score and the ABS, which included DOAC user data in their derivation cohort, have performed similarly or slightly better than our model with c-statistics of 0.65 (0.64–0.66) and 0.68 (0.67–0.69), respectively [27, 28].
Our study was one of the few to have tested the ability of its MB prediction model to detect MB subtypes. A real-world study compared the HAS-BLED’s ability to discriminate MB subtypes to that of the Age Biomarker Clinical history score and found that the HAS-BLED performed better in detecting MB (c-statistics: 0.583 and 0.518, respectively) and GIB (c-statistics: 0.596 and 0.519, respectively) [65]. However, these findings were neither cross-validated, nor externally validated [60, 65]. Our own MB risk score overperformed relative to the HAS-BLED in this study (c-statistic: 0.65 95% CI 0.63–0.66), but further research is needed for confirmation. Furthermore, while the HAS-BLED outperformed other scores in predicting ICH, we were unable to evaluate this outcome due to a paucity of events-per-predictors [60, 65]. Finally, despite encompassing approximately half of MB cases, NGIB, which predominantly included genitourinary bleeding and gross hematuria, has been poorly studied [66–68]. Our model predicted NGIB as well as it did MB (c-statistic: 0.67 95% CI 0.64–0.70). Thus, one of the advantages of our MB model is that it also had a good discrimination in terms of GIB and NGIB. Nonetheless, these findings need to be validated with inpatient data.
Furthermore, no study has identified the predictors for the most prevalent MB subtypes among DOAC and warfarin users. Two prediction schemes (the Qbleed models) and one observational study evaluated predictors of upper GIB and ICH as well as all GIB, respectively. However, neither model accounted for all DOAC users [69, 70]. Our study is the first to identify predictors of GIB and NGIB using a derivation cohort of DOAC and warfarin users. Our final model identified similar predictors to existing MB scores, but may be more robust. Clinical scores that effectively predict common MB subtypes like GIB are essential as they can significantly impact patient quality of life, DOAC adherence, and mortality [29, 71].
Our study has several advantages. Firstly, it is the only study to have developed MB and MB subtype prediction models derived from DOAC and warfarin user data. Secondly, this is one of the few studies to calculate cumulative incidence of MB, GIB, ICH and NGIB stratified by dosage for all DOACs. Thirdly, we used a prediction method that minimized the likelihood of overfitting the regression to its derivation dataset, theoretically leading to a more robust model than existing ones [24, 27, 28, 60–62, 64, 72]. Fourth, unlike previous studies, our model’s performance indices have been cross-validated to avoid inflated c-statistics [24, 27, 60–62, 64, 72]. Fifth, we used a dataset large enough to establish models in each treatment subgroup. Sixth, our predictor candidates were well-defined and clinically useful (non-redundant) variables with externally validated coding algorithms. Moreover, we made sure that our outcome definitions were consistent with previous claims-based observational studies. Seventh, patient loss-to-follow-up (mainly death), OAC non-adherence and OAC switching during follow-up could limit model performance. However, our sensitivity analyses suggested that none of these factors have hindered model performance (S9 Table). Ultimately, the observational nature of our data allowed us to characterize real-world predictors of our outcomes.
Our findings presented some limitations. Firstly, prediction modelling is not designed for causal inference, thereby precluding conclusions regarding the impacts of hypothetical interventions on the risk factors. Secondly, due to the nature of our prediction models, these findings are not directly generalizable to any other common OAC indications or edoxaban users. Thirdly, important candidate predictors may not have been evaluated in our models. Specifically, our source data does not include information on alcohol use, tobacco use, ethnicity, over-the-counter aspirin use or labile INR (factors highly associated with bleeding) [24, 73, 74]. Despite the large populational data source, our sample size constrained our ability to identify ICH predictors. Fourth, some patients with prior cardiovascular diseases may not have been identified due to errors in diagnostic coding. Fifth, medication dispensation does not necessarily amount to medication use, resulting in a potential misclassification bias in our cumulative incidence findings and prediction error in our prediction model. Sixth, given our use of real-world data, our findings require external validation using inpatient data [28]. Seventh, our comparisons to published MB models were only speculative given the differences in MB and predictor definitions between models derived from administrative claims data and those derived from inpatient data. Lastly, given our selection of patients who were hospitalized, it is likely that our cohort was older, sicker and used more medications than the general population of anticoagulant users with AF. External validation will be required to ensure the generalizability of our findings to this population.
Our findings have several implications. Due to the overall similarity of MB predictors across treatment groups, our findings suggest that it would be ideal to create an MB risk score that groups together all OAC users rather than generating separate scores for DOACs and warfarin. Moreover, the paucity of RCT and observational data pertaining to GIB and NGIB predictors within an AF population of OAC users makes it difficult to assess whether existing prediction models, such as the HAS-BLED takes into account risk factors for the most prevalent MB subtypes in a real-world population. Thus, although it requires further validation using clinical data and real-world data from other AF patient populations, this study may inform the development of a much-needed monitoring tool that encompasses a more diverse range of MB risk factors adapted to the heterogeneity of OAC user and MB subtype characteristics. Ultimately, our derivation model is well-calibrated and has a similar discriminative potential relative to the other MB scores in the literature (most notably, the HAS-BLED, ABS, and ORBIT-AF), but will require further validation. Future studies will involve using inpatient data to compare our model to the HAS-BLED using adequate comparative performance metrics and seeing how well it stratifies the risk for each MB subtype relative to the HAS-BLED.
Supporting information
S1 Fig. Calibration plots.
Calibration plots of LASSO (red) and adaptive LASSO (blue) logistic regression models for GIB among users of A) Warfarin, B) DOACs, C) all OACs; NGIB among users of D) Warfarin, E) DOACs, F) all OACs; and MB among users of G) Warfarin, H) DOACs, I) all OACs.
https://doi.org/10.1371/journal.pone.0246691.s001
(TIF)
S2 Fig. Cross-validated ROC curves.
Cross-validated ROC curves of LASSO (red) and adaptive LASSO (blue) logistic regression models for GIB among users of A) Warfarin, B) DOACs, C) all OACs; NGIB among users of D) Warfarin, E) DOACs, F) all OACs; and MB among users of G) Warfarin, H) DOACs, I) all OACs.
https://doi.org/10.1371/journal.pone.0246691.s002
(TIF)
S3 Fig. Calibration of plots of the global MB model tested for MB subtypes.
Calibration plots of the global MB model tested for its ability to predict A. GIB and B. NGIB.
https://doi.org/10.1371/journal.pone.0246691.s003
(TIF)
S1 Table. Major bleeding outcome definition.
ICD-9 and ICD-10 codes for GIB, NGIB, MB. ICD-9 and ICD-10 codes for GIB, NGIB, ICH and MB. These outcomes were defined on the basis of 6 observational studies [39, 41–45].
https://doi.org/10.1371/journal.pone.0246691.s004
(DOCX)
S2 Table. Definition of CHADS2-VASc2, modified HAS-BLED, ATRIA, HEMORR₂HAGES and ORBIT-AF risk scores along with their scoring algorithms.
https://doi.org/10.1371/journal.pone.0246691.s005
(DOCX)
S3 Table. Definition of co-morbidity and concomitant medication variables used for CHA2DS2-VASc and HAS-BLED risk score calculation according to ICD-9 and ICD-10 codes from the Med-Echo databases.
https://doi.org/10.1371/journal.pone.0246691.s006
(DOCX)
S4 Table. Sample size justification.
Assuming 28 candidate predictors, these are the event requirements for each subgroup. a The number of outcomes in these groups would be sufficient to yield robust prediction models. b In a simulation study, it was found that under the assumption that outcomes are rare and that noise predictors (predictors presenting redundant information) are present, LASSO regression was shown to yield stable predictions (neither overfitted, nor underfitted models) with an events per candidate predictor ratio of 5.
https://doi.org/10.1371/journal.pone.0246691.s007
(DOCX)
S5 Table. Baseline characteristics of OAC new user with specific types of major bleeds in the year of follow-up from 2011 to 2018.
a Non-GI extracranial major bleeding as an outcome or a predictor includes vitreous, urogenital, hemoperitoneal and unspecified major bleeding as well as hemoarthrosis, hemopericardium, hemoptysis, hematuria and post-bleeding anemia. All major bleedings included GI, Non-GI extracranial major bleeding and intracranial bleeding. b DOAC users include all doses of dabigatran, rivaroxaban and apixaban. c OAC users include all doses of warfarin, dabigatran, rivaroxaban and apixaban. d Represents a history of at least one of the bleeding subcategories OR at least one prescription of antiplatelet subcategory. Although each subcategory is mutually exclusive, the totals will not add up to the parent variable.
https://doi.org/10.1371/journal.pone.0246691.s008
(DOCX)
S6 Table. Baseline characteristics of OAC new users without specific types of major bleeds in the year of follow-up from 2011 to 2018.
a Non-GI extracranial major bleeding as an outcome or a predictor includes vitreous, urogenital, hemoperitoneal and unspecified major bleeding as well as hemoarthrosis, hemopericardium, hemoptysis, hematuria and post-bleeding anemia. b DOAC users include all doses of dabigatran, rivaroxaban and apixaban. c OAC users include all doses of warfarin, dabigatran, rivaroxaban and apixaban. d Represents a history of at least one of the bleeding subcategories OR at least one prescription of antiplatelet subcategory. Although each subcategory is mutually exclusive, the totals will not add up to the parent variable.
https://doi.org/10.1371/journal.pone.0246691.s009
(DOCX)
S7 Table. Logistic regression LASSO analyses of major bleeding subtype predictors among OAC new users from 2011 to 2018.
All values are ORs. a In the DOAC group, the rivaroxaban and apixaban variables are compared to dabigatran. In the OAC group, dabigatran, rivaroxaban and apixaban are compared to warfarin. b DOAC users include all doses of dabigatran, rivaroxaban and apixaban. c OAC users include all doses of warfarin, dabigatran, rivaroxaban and apixaban.
https://doi.org/10.1371/journal.pone.0246691.s010
(DOCX)
S8 Table. Logistic regression adaptive LASSO analyses of major bleeding subtype predictors among OAC new users from 2011 to 2018.
All values are ORs. a In the DOAC group, the rivaroxaban and apixaban variables are compared to dabigatran. In the OAC group, dabigatran, rivaroxaban and apixaban are compared to warfarin. b DOAC users include all doses of dabigatran, rivaroxaban and apixaban. c OAC users include all doses of warfarin, dabigatran, rivaroxaban and apixaban.
https://doi.org/10.1371/journal.pone.0246691.s011
(DOCX)
S9 Table. Sensitivity analyses of the global MB model for all OAC users.
Discrimination values for the global score in patients who did not die during follow-up, adherent patients (PDC≥0.80), non-adherent patients (PDC<0.80), patients who did not switch OAC in the year of follow-up and patients who did not switch OAC or die during follow-up.
https://doi.org/10.1371/journal.pone.0246691.s012
(DOCX)
Acknowledgments
Jakub Qazi was the first author. He wrote the article, ran the analyses, contributed to part of the conceptualization of the project and participated in interpreting the data in conjunction with the members of the advisory committee (Dr. Sylvie Perreault, Dr. Mireille Schnitzer and Dr. Marie-Josée Martel) as well as Dr. Robert Côté. Sylvie Perreault is the supervising author. In addition to providing training, revisions and final approval of the published article, she created a conceptual framework for the study design and data analysis and provided access to her data. Mireille Schnitzer provided a statistical framework and guidance for the prediction modelling. Marc Dorais provided database training, programming guidance and performed preliminary statistical analyses. Robert Côté contributed clinical validation of the study’s findings. Marie-Josée Martel provided guidance regarding improvements to the study during advisory committee meetings. Marie-Josee Martel is also an employee of Xcenda UK Ltd.
References
- 1. Lippi G, Sanchis-Gomar F, Cervellin G. Global epidemiology of atrial fibrillation: An increasing epidemic and public health challenge. International Journal of Stroke. 2020. pmid:31955707
- 2. Heeringa J, van der Kuip DA, Hofman A, Kors JA, van Herpen G, Stricker BHC, et al. Prevalence, incidence and lifetime risk of atrial fibrillation: The rotterdam study. European heart journal. 2006; 27: 949–953. pmid:16527828
- 3. Morillo CA, Banerjee A, Perel P, Wood D, Jouven X. Atrial fibrillation: The current epidemic. Journal of geriatric cardiology: JGC. 2017; 14: 195. pmid:28592963
- 4. Wolf PA, Abbott RD, Kannel WB. Atrial fibrillation as an independent risk factor for stroke: The framingham study. Stroke. 1991; 22: 983–988. pmid:1866765
- 5. Lin H-J, Wolf PA, Kelly-Hayes M, Beiser AS, Kase CS, Benjamin EJ, et al. Stroke severity in atrial fibrillation: The framingham study. Stroke. 1996; 27: 1760–1764. pmid:8841325
- 6. Ruddox V, Sandven I, Munkhaugen J, Skattebu J, Edvardsen T, Otterstad JE. Atrial fibrillation and the risk for myocardial infarction, all-cause mortality and heart failure: A systematic review and meta-analysis. European journal of preventive cardiology. 2017; 24: 1555–1566. pmid:28617620
- 7. Sato H, Ishikawa K, Kitabatake A, Ogawa S, Maruyama Y, Yokota Y, et al. Low-dose aspirin for prevention of stroke in low-risk patients with atrial fibrillation: Japan atrial fibrillation stroke trial. Stroke. 2006; 37: 447–451. pmid:16385088
- 8. Bai Y, Deng H, Shantsila A, Lip GY. Rivaroxaban versus dabigatran or warfarin in real-world studies of stroke prevention in atrial fibrillation: Systematic review and meta-analysis. Stroke. 2017; 48: 970–976. pmid:28213573
- 9. Hellenbart EL, Faulkenberg KD, Finks SW. Evaluation of bleeding in patients receiving direct oral anticoagulants. Vascular Health and Risk Management. 2017; 13: 325–342. pmid:28860793
- 10. Shoeb M, Fang MC. Assessing bleeding risk in patients taking anticoagulants. Journal of Thrombosis and Thrombolysis. 2013; 35: 312–319. pmid:23479259
- 11. Loffredo L, Perri L, Violi F. Impact of new oral anticoagulants on gastrointestinal bleeding in atrial fibrillation: A meta-analysis of interventional trials. Digestive & Liver Disease. 2015; 47: 429–431.
- 12. Connolly SJ, Ezekowitz MD, Yusuf S, Eikelboom J, Oldgren J, et al. Dabigatran versus warfarin in patients with atrial fibrillation.[erratum appears in n engl j med. 2010 nov 4;363(19):1877]. New England Journal of Medicine. 2009; 361: 1139–1151. pmid:21059484
- 13. Patel MR, Mahaffey KW, Garg J, Pan G, Singer DE, Hacke W, et al. Rivaroxaban versus warfarin in nonvalvular atrial fibrillation. New England Journal of Medicine. 2011; 365: 883–891. pmid:21830957
- 14. Granger CB, Alexander JH, McMurray JJ, Lopes RD, Hylek EM, Hanna M, et al. Apixaban versus warfarin in patients with atrial fibrillation. New England Journal of Medicine. 2011; 365: 981–992. pmid:21870978
- 15. Giugliano RP, Ruff CT, Braunwald E, Murphy SA, Wiviott SD, Halperin JL, et al. Edoxaban versus warfarin in patients with atrial fibrillation. New England Journal of Medicine. 2013; 369: 2093–2104. pmid:24251359
- 16. Makam RCP, Hoaglin DC, McManus DD, Wang V, Gore JM, Spencer FA, et al. Efficacy and safety of direct oral anticoagulants approved for cardiovascular indications: Systematic review and meta-analysis. PloS one. 2018; 13. pmid:29795629
- 17. Ntaios G, Papavasileiou V, Makaritsis K, Vemmos K, Michel P, et al. Real-world setting comparison of nonvitamin-k antagonist oral anticoagulants versus vitamin-k antagonists for stroke prevention in atrial fibrillation: A systematic review and meta-analysis. Stroke. 2017; 48: 2494–2503. pmid:28716982
- 18. Mitchell A, Watson MC, Welsh T, McGrogan A. Effectiveness and safety of direct oral anticoagulants versus vitamin k antagonists for people aged 75 years and over with atrial fibrillation: A systematic review and meta-analyses of observational studies. Journal of clinical medicine. 2019; 8: 554. pmid:31022899
- 19. Fralick M, Colacci M, Schneeweiss S, Huybrechts KF, Lin KJ, Gagne JJ. Effectiveness and safety of apixaban compared with rivaroxaban for patients with atrial fibrillation in routine practice: A cohort study. Annals of internal medicine. 2020; 172: 463–473. pmid:32150751
- 20.
Li G, Lip GY, Holbrook A, Chang Y, Larsen TB, Sun X, et al. Direct comparative effectiveness and safety between non-vitamin k antagonist oral anticoagulants for stroke prevention in nonvalvular atrial fibrillation: A systematic review and meta-analysis of observational studies. Springer; 2019.
- 21. Almutairi AR, Zhou L, Gellad WF, Lee JK, Slack MK, Martin JR, et al. Effectiveness and safety of non–vitamin k antagonist oral anticoagulants for atrial fibrillation and venous thromboembolism: A systematic review and meta-analyses. Clinical therapeutics. 2017; 39: 1456–1478. e1436. pmid:28668628
- 22.
Li G, Holbrook A, Jin Y, Zhang Y, Levine MA, Mbuagbaw L, et al. Comparison of treatment effect estimates of non-vitamin k antagonist oral anticoagulants versus warfarin between observational studies using propensity score methods and randomized controlled trials. Springer; 2016.
- 23. Gorman E, Perkel D, Dennis D, Yates J, Heidel R, Wortham D. Validation of the has-bled tool in atrial fibrillation patients receiving rivaroxaban. Journal of atrial fibrillation. 2016; 9. pmid:27909541
- 24. Pisters R, Lane DA, Nieuwlaat R, De Vos CB, Crijns HJ, Lip GY. A novel user-friendly score (has-bled) to assess 1-year risk of major bleeding in patients with atrial fibrillation: The euro heart survey. Chest. 2010; 138: 1093–1100. pmid:20299623
- 25. Cheung K-S, Leung WK. Gastrointestinal bleeding in patients on novel oral anticoagulants: Risk, prevention and management. World journal of gastroenterology. 2017; 23: 1954. pmid:28373761
- 26. Barada K, Abdul-Baki H, El Hajj II, Hashash JG, Green PH. Gastrointestinal bleeding in the setting of anticoagulation and antiplatelet therapy. Journal of clinical gastroenterology. 2009; 43: 5–12. pmid:18607297
- 27. O’Brien EC, Simon DN, Thomas LE, Hylek EM, Gersh BJ, Ansell JE, et al. The orbit bleeding score: A simple bedside score to assess bleeding risk in atrial fibrillation. European heart journal. 2015; 36: 3258–3264. pmid:26424865
- 28. J’Neka SC, MacLehose RF, Lutsey PL, Norby FL, Chen LY, O’Neal WT, et al. A new model to predict major bleeding in patients with atrial fibrillation using warfarin or direct oral anticoagulants. PloS one. 2018; 13. pmid:30550837
- 29. Lauffenburger JC, Rhoney DH, Farley JF, Gehi AK, Fang G. Predictors of gastrointestinal bleeding among patients with atrial fibrillation after initiating dabigatran therapy. Pharmacotherapy: The Journal of Human Pharmacology & Drug Therapy. 2015; 35: 560–568.
- 30. Yokoyama S, Tanaka Y, Nakagita K, Hosomi K, Takada M. Bleeding risk of warfarin and direct oral anticoagulants in younger population: A historical cohort study using a japanese claims database. International journal of medical sciences. 2018; 15: 1686. pmid:30588192
- 31. Tamblyn R, Lavoie G, Petrella L, Monette J. The use of prescription claims databases in pharmacoepidemiological research: The accuracy and comprehensiveness of the prescription claims database in quebec. Journal of clinical epidemiology. 1995; 48: 999–1009. pmid:7775999
- 32. Cadarette SM, Wong L. An introduction to health care administrative data. The Canadian journal of hospital pharmacy. 2015; 68: 232. pmid:26157185
- 33. Perreault S, de Denus S, White-Guay B, Côté R, Schnitzer ME, Dubé MP, et al. Oral anticoagulant prescription trends, profile use, and determinants of adherence in patients with atrial fibrillation. Pharmacotherapy. 2020; 40: 40–54. pmid:31758592
- 34. Eguale T, Winslade N, Hanley JA, Buckeridge DL, Tamblyn R. Enhancing pharmacosurveillance with systematic collection of treatment indication in electronic prescribing: A validation study in canada. Drug Saf. 2010; 33: 559–567. pmid:20553057
- 35. Tamblyn R, Reid T, Mayo N, McLeod P, Churchill-Smith M. Using medical services claims to assess injuries in the elderly: Sensitivity of diagnostic and procedure codes for injury ascertainment. J Clin Epidemiol. 2000; 53: 183–194. pmid:10729691
- 36. Wilchesky M, Tamblyn RM, Huang A. Validation of diagnostic codes within medical services claims. J Clin Epidemiol. 2004; 57: 131–141. pmid:15125622
- 37. Jensen PN, Johnson K, Floyd J, Heckbert SR, Carnahan R, Dublin S. A systematic review of validated methods for identifying atrial fibrillation using administrative data. Pharmacoepidemiology and drug safety. 2012; 21: 141–147. pmid:22262600
- 38. Navar-Boggan AM, Rymer JA, Piccini JP, Shatila W, Ring L, Stafford JA, et al. Accuracy and validation of an automated electronic algorithm to identify patients with atrial fibrillation at risk for stroke. American heart journal. 2015; 169: 39–44. e32. pmid:25497246
- 39. Villines TC, Schnee J, Fraeman K, Siu K, Reynolds MW, Collins J, et al. A comparison of the safety and effectiveness of dabigatran and warfarin in non-valvular atrial fibrillation patients in a large healthcare system. Thromb Haemost. 2015; 114: 1290–1298. pmid:26446456
- 40. Perreault S, Shahabi P, Côté R, Dumas S, Rouleau-Mailloux É, Feroz Zada Y, et al. Rationale, design, and preliminary results of the quebec warfarin cohort study. Clinical Cardiology. 2018; 41: 576–585. pmid:29542828
- 41. Yao X, Abraham NS, Sangaralingham LR, Bellolio MF, McBane RD, Shah ND, et al. Effectiveness and safety of dabigatran, rivaroxaban, and apixaban versus warfarin in nonvalvular atrial fibrillation. J Am Heart Assoc. 2016; 5.
- 42. Lauffenburger JC, Farley JF, Gehi AK, Rhoney DH, Brookhart MA, Fang G. Effectiveness and safety of dabigatran and warfarin in real-world us patients with non-valvular atrial fibrillation: A retrospective cohort study. J Am Heart Assoc. 2015; 4. pmid:25862791
- 43. Maura G, Blotière PO, Bouillon K, Billionnet C, Ricordeau P, Alla F, et al. Comparison of the short-term risk of bleeding and arterial thromboembolic events in nonvalvular atrial fibrillation patients newly treated with dabigatran or rivaroxaban versus vitamin k antagonists: A french nationwide propensity-matched cohort study. Circulation. 2015; 132: 1252–1260. pmid:26199338
- 44. Graham DJ, Reichman ME, Wernecke M, Zhang R, Southworth MR, Levenson M, et al. Cardiovascular, bleeding, and mortality risks in elderly medicare patients treated with dabigatran or warfarin for nonvalvular atrial fibrillation. Circulation. 2015; 131: 157–164. pmid:25359164
- 45. Outcomes of dabigatran and warfarin for atrial fibrillation in contemporary practice. Annals of internal medicine. 2017; 167: 845–854. pmid:29132153
- 46. Thigpen JL, Dillon C, Forster KB, Henault L, Quinn EK, Tripodis Y, et al. Validity of international classification of disease codes to identify ischemic stroke and intracranial hemorrhage among individuals with associated diagnosis of atrial fibrillation. Circ Cardiovasc Qual Outcomes. 2015; 8: 8–14. pmid:25587093
- 47. Cunningham A, Stein CM, Chung CP, Daugherty JR, Smalley WE, Ray WA. An automated database case definition for serious bleeding related to oral anticoagulant use. Pharmacoepidemiol Drug Saf. 2011; 20: 560–566. pmid:21387461
- 48. Arnason T, Wells PS, van Walraven C, Forster AJ. Accuracy of coding for possible warfarin complications in hospital discharge abstracts. Thromb Res. 2006; 118: 253–262. pmid:16081144
- 49.
Pampalon R, Gamache P., Hamel D. The québec index of material and social deprivation: Methodological follow-up, 1991 through 2006. INSPQ. 2011.
- 50.
Blais CL, Hamel D, Brown K, Rinfret S, Cartier R, Giguère M, et al. Évaluation des soins et surveillance des maladies cardiovasculaires de santé publique du québec et de l’institut national d’excellence en santé et services sociaux. Gouvernement du Québec, Institut national de santé publique, Institut national d’excellence en santé et des services sociaux. 2012: 1–9.
- 51. Roy L, Zappitelli M, White-Guay B, Lafrance J-P, Dorais M, Perreault S. Agreement between administrative database and medical chart review for the prediction of chronic kidney disease g category. Canadian Journal of Kidney Health and Disease. 2020; 7: 2054358120959908. pmid:33101698
- 52. Friberg L, Rosenqvist M, Lip GY. Evaluation of risk stratification schemes for ischaemic stroke and bleeding in 182 678 patients with atrial fibrillation: The swedish atrial fibrillation cohort study. European heart journal. 2012; 33: 1500–1510. pmid:22246443
- 53. Zou H. The adaptive lasso and its oracle properties. Journal of the American statistical association. 2006; 101: 1418–1429.
- 54. Tibshirani R. Regression shrinkage and selection via the lasso: A retrospective. Journal of the Royal Statistical Society: Series B (Statistical Methodology). 2011; 73: 273–282.
- 55. Pavlou M, Ambler G, Seaman S, De Iorio M, Omar RZ. Review and evaluation of penalised regression methods for risk prediction in low-dimensional data with few events. Statistics in Medicine. 2016; 35: 1159–1177. pmid:26514699
- 56. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: A framework for some traditional and novel measures. Epidemiology (Cambridge, Mass). 2010; 21: 128. pmid:20010215
- 57. Tepper PG, Mardekian J, Masseria C, Phatak H, Kamble S, Abdulsattar Y, et al. Real-world comparison of bleeding risks among non-valvular atrial fibrillation patients prescribed apixaban, dabigatran, or rivaroxaban. PloS one. 2018; 13. pmid:30383768
- 58. Ntaios G, Papavasileiou V, Makaritsis K, Vemmos K, Michel P, Lip GY. Real-world setting comparison of nonvitamin-k antagonist oral anticoagulants versus vitamin-k antagonists for stroke prevention in atrial fibrillation: A systematic review and meta-analysis. Stroke. 2017; 48: 2494–2503. pmid:28716982
- 59. Chao T-F, Lip GY, Lin Y-J, Chang S-L, Lo L-W, Hu Y-F, et al. Incident risk factors and major bleeding in patients with atrial fibrillation treated with oral anticoagulants: A comparison of baseline, follow-up and delta has-bled scores with an approach focused on modifiable bleeding risk factors. Thrombosis and haemostasis. 2018; 47: 768–777. pmid:29510426
- 60. Apostolakis S, Lane DA, Guo Y, Buller H, Lip GY. Performance of the hemorr2hages, atria, and has-bled bleeding risk–prediction scores in patients with atrial fibrillation undergoing anticoagulation: The amadeus (evaluating the use of sr34006 compared to warfarin or acenocoumarol in patients with atrial fibrillation) study. Journal of the American College of Cardiology. 2012; 60: 861–867. pmid:22858389
- 61. Fang MC, Go AS, Chang Y, Borowsky LH, Pomernacki NK, Udaltsova N, et al. A new risk scheme to predict warfarin-associated hemorrhage: The atria (anticoagulation and risk factors in atrial fibrillation) study. Journal of the American College of Cardiology. 2011; 58: 395–401. pmid:21757117
- 62. Gage BF, Yan Y, Milligan PE, Waterman AD, Culverhouse R, Rich MW, et al. Clinical classification schemes for predicting hemorrhage: Results from the national registry of atrial fibrillation (nraf). American heart journal. 2006; 151: 713–719. pmid:16504638
- 63. LeDell E, Petersen M, van der Laan M. Computationally efficient confidence intervals for cross-validated area under the roc curve estimates. Electronic journal of statistics. 2015; 9: 1583. pmid:26279737
- 64. Zhu W, He W, Guo L, Wang X, Hong K. The has‐bled score for predicting major bleeding risk in anticoagulated patients with atrial fibrillation: A systematic review and meta‐analysis. Clinical Cardiology. 2015; 38: 555–561. pmid:26418409
- 65. Esteve-Pastor MA, Rivera-Caravaca JM, Roldan V, Vicente V, Valdes M, Marin F, et al. Long-term bleeding risk prediction in ‘real world’patients with atrial fibrillation: Comparison of the has-bled and abc-bleeding risk scores. Thrombosis and haemostasis. 2017; 117: 1848–1858. pmid:28799620
- 66. Tepper PG, Mardekian J, Masseria C, Phatak H, Kamble S, Abdulsattar Y, et al. Real-world comparison of bleeding risks among non-valvular atrial fibrillation patients prescribed apixaban, dabigatran, or rivaroxaban. PLoS ONE [Electronic Resource]. 2018; 13 (11) (no pagination). pmid:30383768
- 67. Caro Martinez C, Andreu Cayuelas JM, Flores Blanco PJ, Valdes M, Lorenzo B, Luis J, et al. Comparison of bleeding risk scores in patients with nonvalvular atrial fibrillation starting direct oral anticoagulants. Revista Española de Cardiología. 2017; 70: 878–880. pmid:28238647
- 68. Kodani E. Genitourinary hemorrhagic complications and malignancies in patients receiving anticoagulation therapy. Circulation Journal. 2017; 81: 149–150. pmid:27980296
- 69. Lauffenburger JC, Rhoney DH, Farley JF, Gehi AK, Fang G. Predictors of gastrointestinal bleeding among patients with atrial fibrillation after initiating dabigatran therapy. Pharmacotherapy: The Journal of Human Pharmacology and Drug Therapy. 2015; 35: 560–568.
- 70. Hippisley-Cox J, Coupland C. Predicting risk of upper gastrointestinal bleed and intracranial bleed with anticoagulants: Cohort study to derive and validate the qbleed scores. bmj. 2014; 349. pmid:25069704
- 71. Vaduganathan M, Bhatt DL. Gastrointestinal bleeding with oral anticoagulation: Understanding the scope of the problem. Clinical Gastroenterology and Hepatology. 2017; 15: 691–693. pmid:28089849
- 72. Gallego P, Roldán V, Torregrosa JM, Gálvez J, Valdés M, Vicente V, et al. Relation of the has-bled bleeding risk score to major bleeding, cardiovascular events, and mortality in anticoagulated patients with atrial fibrillation. Circulation: Arrhythmia and Electrophysiology. 2012; 5: 312–318. pmid:22319005
- 73. Langsted A, Nordestgaard BG. Smoking is associated with increased risk of major bleeding: A prospective cohort study. Thrombosis and haemostasis. 2019; 119: 039–047. pmid:30597498
- 74. Gibson CM, Yuet WC. Racial and ethnic differences in response to anticoagulation: A review of the literature. Journal of Pharmacy Practice. 2019: 0897190019894142. pmid:31875763