Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Open Access

Peer-reviewed

Research Article

Challenges of developing a cardiovascular risk calculator for patients with rheumatoid arthritis

Correction

7 Apr 2017: Crowson CS, Rollefstad S, Kitas GD, van Riel PLCM, Gabriel SE, et al. (2017) Correction: Challenges of developing a cardiovascular risk calculator for patients with rheumatoid arthritis. PLOS ONE 12(4): e0175605. https://doi.org/10.1371/journal.pone.0175605 View correction

Abstract

Objective

Cardiovascular disease (CVD) risk calculators designed for use in the general population do not accurately predict the risk of CVD among patients with rheumatoid arthritis (RA), who are at increased risk of CVD. The process of developing risk prediction models involves numerous issues. Our goal was to develop a CVD risk calculator for patients with RA.

Methods

Thirteen cohorts of patients with RA originating from 10 different countries (UK, Norway, Netherlands, USA, Sweden, Greece, South Africa, Spain, Canada and Mexico) were combined. CVD risk factors and RA characteristics at baseline, in addition to information on CVD outcomes were collected. Cox models were used to develop a CVD risk calculator, considering traditional CVD risk factors and RA characteristics. Model performance was assessed using measures of discrimination and calibration with 10-fold cross-validation.

Results

A total of 5638 RA patients without prior CVD were included (mean age: 55 [SD: 14] years, 76% female). During a mean follow-up of 5.8 years (30139 person years), 389 patients developed a CVD event. Event rates varied between cohorts, necessitating inclusion of high and low risk strata in the models. The multivariable analyses revealed 2 risk prediction models including either a disease activity score including a 28 joint count and erythrocyte sedimentation rate (DAS28ESR) or a health assessment questionnaire (HAQ) along with age, sex, presence of hypertension, current smoking and ratio of total cholesterol to high-density lipoprotein cholesterol. Unfortunately, performance of these models was similar to general population CVD risk calculators.

Conclusion

Efforts to develop a specific CVD risk calculator for patients with RA yielded 2 potential models including RA disease characteristics, but neither demonstrated improved performance compared to risk calculators designed for use in the general population. Challenges encountered and lessons learned are discussed in detail.

Citation: Crowson CS, Rollefstad S, Kitas GD, van Riel PLCM, Gabriel SE, Semb AG, et al. (2017) Challenges of developing a cardiovascular risk calculator for patients with rheumatoid arthritis. PLoS ONE 12(3): e0174656. https://doi.org/10.1371/journal.pone.0174656

Editor: Graham R. Wallace, University of Birmingham, UNITED KINGDOM

Received: October 28, 2016; Accepted: March 13, 2017; Published: March 23, 2017

Copyright: © 2017 Crowson et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: The ATACC-RA consortium legal contract does not allow full access of patient information to be provided to a third party without prior approval from the ATACC-RA steering committee. However, acess to the complete de-identified data can be made available following approval. Requests for approval can be sent to the corresponding author (Cynthia S. Crowson at Crowson@mayo.edu), who will forward them to the ATACC-RA Steering Committee for approval.

Funding: This work was supported by a collaborative agreement for independent research from Eli Lilly and a grant from the National Institutes of Health, NIAMS (R01AR46849). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Competing interests: I have read the journal's policy and the authors of this manuscript have the following competing interests: CSC, SR, SEG: nothing to disclose. GDK: MSD, UCB, Abbvie, Pfizer, BMS, Novartis. PLCMR: Abbott, BMS, MSD, Pfizer, Roche, Eli-Lilly, Galapagos, UCB, Celltrion. AGS: Merck/Schering- Plough, UCB, Abbott, Pfizer/Wyeth, BMS, Eli-Lilly. This does not alter our adherence to PLOS ONE policies on sharing data and materials.

Introduction

Patients with rheumatoid arthritis (RA) have a demonstrated increased risk of cardiovascular disease (CVD) of 1.5–2 fold compared to their peers without RA [1]. The elevated risk has been shown to be attributable to both traditional and RA-specific factors. Decision-making regarding indication for cardio-protective medication is supported by use of various CVD risk prediction algorithms. However, risk calculators designed for use in the general population do not accurately estimate the risk of CVD in patients with RA [2, 3].

By using risk prediction models, clinical care decisions are made every day based on the probability of disease or the probability of future events [4, 5]. The methodology for deriving and assessing risk calculators has received much attention in the statistical literature [6–10]. In fact, guidelines for reporting multivariable prediction models have recently been published [11].

Several attempts have been made to improve CVD risk prediction in patients with RA. The European League Against Rheumatism (EULAR) advocate to apply a 1.5 multiplier to the calculated risk by the nationally recommended CVD risk algorithm, QRISK2 have included a factor of 1.4 for patients with RA, and the Expanded cardiovascular Risk Score for RA (ERS-RA), which is the first published RA specific CVD risk calculator. However, these approaches had significant limitations [1, 12, 13]. A Trans-Atlantic Cardiovascular Consortium for Rheumatoid Arthritis (ATACC-RA) was formed in 2012 with the goal of building a risk calculator that would accurately predict risk of future CVD among patients with RA. This paper will describe general challenges involved in development of a risk calculator, and more specifically, bottlenecks in derivation of a risk calculator to optimize prediction of future CVD in RA patients.

Methods

Study populations

Thirteen cohorts of patients with RA originating from 10 different countries (UK, Norway, Netherlands, USA, Sweden, Greece, South Africa, Spain, Canada and Mexico) were combined for the purpose of developing an RA-specific CVD risk algorithm. Details regarding these cohorts are located in Table 1, on the consortium website (http://www.atacc-ra.com) and in previous publications [2, 3, 14–24]. The study was approved by ethical boards/committees at each center. These included the Mayo Clinic and Olmsted Medical Center Institutional Review Boards, CMO Arnhem Nijmegen, Dudley Local Research Ethics Committee, Office of the Oslo University Hospital's Privacy and Data Protection Officer, Ethics Committee at the University Hospital of Umeå, Harbor-UCLA Institutional Review Board, Laiko Hospital Institutional Review Board (Athens, Greece), Human Research Ethics Committee (Medical) from the University of the Witwatersrand in Johannesburg, South Africa, University of Manitoba Health Research Ethics Board and the Manitoba Health Information Privacy Committee, Ethics Committee of Cantabria for Hospital Universitario de Valdecilla in Santander (Spain), Comites de Ética e Investigación del Instituto Nacional de Ciencias Médicas y Nutrición Salvador Zubirán, and Arthritis Center Twente Institutional Review Board. The Mayo Clinic Institutional Review Board approved the cohort aggregation. Data were anonymized and aggregated for analyses.

thumbnail
Download:
Table 1. Characteristics of included cohorts.

https://doi.org/10.1371/journal.pone.0174656.t001

Variable definitions

Outcomes.

The primary outcomes were fatal/non-fatal CVD events including acute coronary syndrome (ST-elevation and non-ST elevation myocardial infarction and unstable angina pectoris), chronic ischemic heart disease (stable angina pectoris), coronary revascularization (e.g., percutaneous coronary intervention and coronary artery bypass grafting), coronary death, other cardiovascular death, cerebrovascular events (ischemic cerebrovascular accident and transient ischemic attack) and peripheral vascular events (with and without revascularization procedures, peripheral artery disease). Not included were cases of confirmed cerebral hemorrhage, non-coronary cardiac death, non-ischemic cardiovascular disease, heart failure and aortic aneurysm.

Cardiovascular risk factors.

Traditional CVD risk factors collected at baseline were: age, sex, smoking status (current, former, never), systolic and diastolic blood pressure, lipid levels (total cholesterol [TC], high density lipoprotein cholesterol [HDL-c], low density lipoprotein cholesterol [LDL-c], triglycerides, and TC:HDL-c ratio), body mass index (BMI), family history of CVD, diabetes mellitus and hypertension. The use of statins and antihypertensive medications at baseline were also registered.

RA disease characteristics.

Clinical characteristics were collected at baseline including rheumatoid factor (RF) positivity, anti-citrullinated protein antibodies (ACPA) positivity, erythrocyte sedimentation rate (ESR), C-reactive protein (CRP), swollen and tender joint counts based on 28 joints, patient and physician global visual analog scales (VAS), disease activity score including 28 joints and ESR (DAS28ESR), RA disease duration and health assessment questionnaire (HAQ) disability index. RF and ACPA were considered positive based on the tests performed at each center. ACPA testing was lacking in some patients. CRP was not available from 1 cohort, DAS28ESR was not available from 1 cohort, swollen and tender joint counts were not available from 2 cohorts, patient global VAS was not available from 4 cohorts, physician global VAS was not available from 8 cohorts, and HAQ was not available from 3 cohorts. Data were also collected on antirheumatic medication use at baseline including synthetic and biologic disease modifying anti-rheumatic drugs (DMARDs) and corticosteroids.

Statistical methods

Data from all cohorts were combined into one database. Patients with a history of CVD prior to baseline were excluded. Multiple imputation methods were used to impute missing values for the CVD risk factors using 10 repetitions. Two cohorts were missing a large amount (~75%) of lipid data, but the lipid data were mostly complete from the other cohorts, so only 20% of the lipid data was missing in the combined database. The missing lipid data were imputed, but analyses excluding these 2 cohorts were also performed to ensure the imputed data did not overly influence our results. Log-transformations were used when imputing lipid levels to avoid bias when computing lipid ratios from imputed data [25]. Descriptive statistics (means, percentages, etc.) were used to summarize the CVD risk factor and RA disease characteristic data and comparisons across cohorts were performed using chi-square and rank sum tests. Person-year methods were used to calculate the rate of CVD events for each cohort. Kaplan-Meier methods were used to plot the rates of development of CVD events during follow-up. Inverse probability weighting was used to adjust the CVD event rates for differences in CVD risk factors across the cohorts [26].

Cox proportional hazard regression analysis was used to develop the risk calculator model. Due to heterogeneity in CVD event rates between centers, the Cox models were stratified by center. To avoid issues with instability of estimates for small centers and to reduce the complexity of the model, centers were grouped into 2 strata (high and low CVD risk) based on CVD event rates. For patients with long follow-up, follow-up time was truncated at 12 years after baseline to facilitate accurate prediction of CVD risk at the 10 year time point, as truncating at ten years would necessitate predicting at the edge of the data and including data well beyond 10 years could influence the coefficients of the risk factors. Least Absolute Shrinkage and Selection Operator (LASSO)-penalized Cox regression was used for variable selection of CV risk factors [27]. Determination of the optimal model was based on cross-validation methods (using the minimal lambda minus 1 standard error rule to reduce the potential for overfitting). RA disease characteristics were then considered one-by-one for inclusion in the model following inclusion of the CVD adjustors because many of these measures were missing for 1 or more centers.

Smoothing splines were also used to examine potential non-linear effects of the continuous risk factors. No significant non-linear effects were noted for the CVD risk factors. Assumptions of proportional hazards were checked. Predicted probabilities for CVD were obtained from the final multivariable model.

Model performance was assessed with measures of discrimination and calibration using 10-fold cross validation. Discrimination was assessed using the concordance statistic (c-statistic) as adapted for the Cox model by Harrell [28]. The c-statistic is analogous to the area under the receiver operating characteristic (ROC) curve. Calibration was assessed by comparing observed versus expected events in deciles of predicted risk using goodness-of-fit tests similar to the Hosmer-Lemeshow test, as well as with standardized incidence ratios (SIR) (which compare the overall observed to predicted number of events) and with calibration plots [10]. Calculated predicted events were transformed to expected events [10]. An SIR>1 indicates that the observed events are higher than expected, meaning the risk calculator underestimates actual risk. Conversely, an SIR<1 indicates that the observed events are lower than expected, meaning that the risk calculator overestimates actual risk.

CVD risk was calculated for the purpose of comparison using risk algorithms designed for use in the general population, including Framingham risk score (FRS) for general CVD risk, American College of Cardiology/American Heart Association (ACC/AHA) Pooled Cohort Equation, QRISK2 and Systematic COronoary Risk Evaluation (SCORE) [5, 12, 29, 30]. Data on heart failure were not available in some cohorts, so the FRS predictions were reduced proportionately based on the fraction of non-heart failure events in the Framingham cohort to ensure a fair comparison between observed and expected events. For QRISK2, the Townsend deprivation score, atrial fibrillation and chronic kidney disease were not available in our data, so CVD risk was calculated using a modified QRISK2 algorithm excluding these variables. The outcome for SCORE includes only fatal events, which limited our analyses using SCORE due to the small number of fatal events in our cohort. Reclassification was examined, but the net reclassification index was not computed, as its use is not recommended when miscalibration exists, which is the case for centers in our low CVD risk strata [9, 31]. Analyses were performed using SAS version 9.4 (SAS Institute, Cary, NC, USA) and R 3.2.0 (R Foundation for Statistical Computing, Vienna, Austria).

Results

Baseline characteristics

A total 5638 RA patients without prior CVD were included (mean age: 55 [SD: 14] years, 76% female; Table 2). During a mean follow-up of 5.8 years (30139 person years), 389 patients developed a CVD event. Two cohorts consisted of Hispanics (US/UCLA and Mexico), the rest were almost exclusively Caucasians. RA disease duration varied by cohort: 4 with early RA (<1 year), 7 established RA (mean 9–13 years) and 2 with both. All demographics, RA disease characteristics and CVD risk factors differed significantly between cohorts (p<0.001 for all).

thumbnail
Download:
Table 2. Descriptive baseline characteristics of 5638 patients with rheumatoid arthritis without prior cardiovascular disease, overall along with lowest and highest mean/percentage by cohort.

https://doi.org/10.1371/journal.pone.0174656.t002

Primary outcome

CVD event rates varied across countries (range 0.1–1.8%/year) with the lowest observed in Canada, Mexico and UK and the highest in US, Netherlands, and Sweden (Table 3 and Fig 1A). Adjusting for CVD risk factors to account for the differences between cohorts modified the CVD event rates, but did not reduce the heterogeneity in CVD event rates across cohorts (Fig 1B). Due to heterogeneity in CVD event rates between centers, risk calculator models were stratified by high and low CVD risk center groups with Netherlands (Nijmegen and Enschede), US (Olmsted and Brigham and Women’s Rheumatoid Arthritis Sequential Study) and Sweden in the high CVD risk strata and all the other centers in the low CVD risk strata.

thumbnail
Download:
Fig 1. Cardiovascular event rates by cohort.

(A) unadjusted rates. (B) rates adjusted for cardiovascular risk factors.

https://doi.org/10.1371/journal.pone.0174656.g001

thumbnail
Download:
Table 3. Cardiovascular Disease (CVD) event rates by cohort (ordered by CVD event rate).

https://doi.org/10.1371/journal.pone.0174656.t003

Risk model development

The next step of the risk calculator development involved assessment of CVD risk factors. CVD risk factors considered for inclusion were: age, sex, current smoker, ever smoker, systolic and diastolic blood pressure measures (in mmHg and log-transformed), lipid measures (TC, HDL-c, LDL-c, triglycerides, and TC:HDL-c ratio), BMI, family history of CVD, diabetes mellitus, history of hypertension, use of anti-hypertensives and use of anti-lipemic medications. The optimal model based on cross-validation methods included 5 variables (age, sex, presence of hypertension, current smoker and TC:HDL-c ratio) consistently across all 10 imputation sets. Inclusion of the presence of hypertension instead of systolic blood pressure was undesirable because there is a loss of information in the dichotomous hypertension variable compared to the continuous blood pressure measure. However, inexplicably, systolic and diastolic blood pressure values were not significant predictors of CVD events in this RA cohort.

Adjusting for the CVD risk factors, each RA disease characteristics was assessed individually for potential inclusion in the risk calculator (Table 4). Several RA disease characteristics were significant predictors of CVD (i.e., DAS28ESR, swollen and tender joint counts, both patient and physician VAS and HAQ). Several other RA disease related characteristics only approached statistical significance. The multivariable modeling revealed 2 possible risk calculators including either DAS28ESR or HAQ along with age, sex, presence of hypertension, current smoker and TC:HDL-c ratio (Table 5). Additional RA disease characteristics did not add significantly to these models and non-linear effects were not identified.

thumbnail
Download:
Table 4. Assessment of rheumatoid arthritis disease characteristics in cardiovascular disease risk score model.

Each potential risk factor is added to a model including age, sex, current smoking, hypertension and lipid ratio (total cholesterol: high density lipoprotein cholesterol).

https://doi.org/10.1371/journal.pone.0174656.t004

thumbnail
Download:
Table 5. Coefficients and hazard ratios of risk calculator models.

https://doi.org/10.1371/journal.pone.0174656.t005

Model performance

The developed risk calculators demonstrated discrimination that was no better than the established risk calculators (c-statistic: 0.71 for FRS, 0.72 for ACC/AHA, 0.70 for SCORE and 0.72 for QRISK2). Ten-fold cross-validation was used to provide an internal validation, and it revealed c-statistics of 0.70 for Model A and 0.71 for Model B (Table 6). Overall calibration of the developed risk calculators assessed using ten-fold cross-validation yielded SIR = 0.83 for both models, indicating that the risk calculator estimates were significantly higher than observed events. Moderate calibration, assessed according to deciles of predicted risk, was better in the developed risk calculators than in the established risk calculators as none of the deciles demonstrated significant differences between observed and predicted risks in the 2 new risk calculators (Fig 2). In contrast, both the FRS and the ACC/AHA underestimated CVD risk in the highest decile of predicted risk. The ACC/AHA also overestimated CVD risk in several other deciles of predicted risk. SCORE was not depicted because the smaller number of fatal events in our cohort made it difficult to assess deciles of predicted risk. However, SCORE significantly overestimated the observed fatal CVD events in our cohort (SIR: 0.43) QRISK2 also substantially overestimated the observed risk of CVD in nearly all of the deciles (SIR: 0.60).

thumbnail
Download:
Fig 2. Calibration plots: Observed versus predicted 10-year risk for Cardiovascular Disease (CVD) according to deciles of predicted risk.

(A) Model A, (B) Model B, (C) the Framingham risk score (FRS) and (D) the American College of Cardiology/American Heart Association Pooled Cohort Equation (ACC/AHA). The vertical lines are 95% confidence intervals for the observed CVD risk estimates.

https://doi.org/10.1371/journal.pone.0174656.g002

thumbnail
Download:
Table 6. Performance of developed risk calculators compared with general CVD risk calculators.

https://doi.org/10.1371/journal.pone.0174656.t006

Reclassification of risk estimates

Scatter plot comparisons of the calculated risk of CVD by the developed risk calculators compared to the calculated risk by FRS demonstrated that many of the new risk calculator values were lower than the FRS values in the high risk centers and the majority of the new risk calculator values were lower than the FRS values in the low risk centers (Fig 3).

thumbnail
Download:
Fig 3. Comparison of the developed risk calculators (Model A and Model B) to the Framingham Risk Score (FRS).

(A) Comparing Model A and FRS for patients in the high cardiovascular risk strata. (B) Comparing Model A and FRS for patients in the low cardiovascular risk strata. (C) Comparing Model B and FRS for patients in the high cardiovascular risk strata. (D) Comparing Model B and FRS for patients in the low cardiovascular risk strata. In each panel, the dashed line represents identity and the solid line is a smoother line of trend.

https://doi.org/10.1371/journal.pone.0174656.g003

Reclassification was assessed separately for patients in the high and low CVD risk centers (Table 7). In the high CVD risk centers, 46% of patients remained in the same CVD risk category when classified by Model A as when classified by FRS, 36% of patients were classified into a higher risk category by Model A than by FRS and 8% of patients were classified into a lower risk category by Model A than by FRS. In the low CVD risk centers, 53% of patients remained in the same CVD risk category when classified by Model A as when classified by FRS, only 2% of patients were classified into a higher risk category by Model A than by FRS and 45% of patients were classified into a lower risk category by Model A than by FRS. In both the high and low risk centers, patients with ≥20% CVD risk calculated by FRS who were classified to 10-<20% risk by Model A did not demonstrate substantially lower observed CVD rates (21.6% for high risk centers and 19.9% for low risk centers), calling into question the accuracy of reclassification to lower risk by Model A. The observed event rates in the other re-categorized patients agreed with the new classification by Model A in most cases, but the confidence intervals were wide due to small numbers of patients with events in many categories. Reclassification was not examined for SCORE and QRISK2 due to their extremely poor calibration in our cohort.

thumbnail
Download:
Table 7. Reclassification for centers with high and low CVD risk for Model A compared with Framingham Risk Score (FRS).

https://doi.org/10.1371/journal.pone.0174656.t007

Discussion

Efforts to develop a specific CVD risk calculator for patients with RA yielded 2 potential models including RA disease characteristics, but neither demonstrated improved performance compared to CVD risk calculators designed for use in the general population. While it is statistically feasible to develop an RA specific CVD risk calculator by pooling resources from many centers, its ability to optimize prediction of future CVD in this patient population is questionable due to several challenges.

Population of interest

It is important that the data used to develop a risk calculator is representative of the patients who will be assessed by the risk calculator. For this purpose, population-based data may be preferred to ensure that the entire spectrum of CVD risk is represented in the data. Referral bias can have an unpredictable impact on CVD risk rates. For example, patients with RA have a high burden of disease that includes increased risks for multiple comorbidities, such as interstitial lung disease, osteoporosis, infections, hematologic cancers, as well as psychosocial impairments (e.g., fatigue, depression, and cognitive decline) [32]. Patients with RA referred to a cardiologist for CVD risk assessment may encompass a broad spectrum including both the high CVD risk patients with several traditional CVD risk factors and the patients whose RA and other comorbidities are controlled well enough that they will not be overwhelmed by information about their increased risk of CVD. Thus referral bias may lead to either over or under estimation of the true CVD risk in patients with RA. We aimed to develop a trans-Atlantic CVD risk calculator, which may have failed, among other factors, due to geographical differences. The countries included in our cohort have different healthcare systems, with differing access to health care, which may lead to differences in CVD risk [33]. In fact, countries with lower socioeconomic welfare tend to have delays in diagnosis of RA and stricter eligibility criteria for biologic therapies [34]. This inequity in access to treatment may influence disease activity levels, the ability to achieve remission and other long-term outcomes of patients with RA [35]. However, additional biases may also be at play, as the CVD risk estimated in our low CVD risk strata is substantially lower than the CVD event rates in the general population of the countries of origin. Perhaps optimal treatment of CVD risk factors and/or the RA disease activity with use of biologics and tight disease control resulting in remission has reduced the CVD event rate in these cohorts. Alternatively, there could be ascertainment bias leading to under-reporting of CVD events. Either way, it is unlikely that the low CVD event rates predicted by our risk calculator will provide accurate estimates of CVD risk for all patients with RA in the countries that had very low CVD risk rates in our cohort, since our calculator re-classified 45% of patient in the low risk centers to lower risk categories than the FRS calculator.

Purpose of risk prediction

Many CVD risk calculators were designed to help determine which patients should receive lipid-lowering therapies, such as statins [5]. Due to this goal, historic cohorts with data collection prior to the availability of statins are often used to develop the risk calculators. These cohorts have the advantages of long follow-up and do not have any biases related to statin use. However, the CVD event rates in these cohorts may not reflect current CVD event rates, as there have been many advances in addition to statins that have improved CVD outcomes in recent years [36]. Another option would be to use more contemporary data and exclude (i.e., censor) subjects at the time when statins were initiated, but this could equate to systematic exclusion of higher risk patients, which would result in unrealistic CVD event rates. Our pooled cohort includes data from after the introduction of statins, but we did not make any adjustments for use of statins during follow-up, as these data were not available in all cohorts.

Study design

Another important consideration is the study design. Longitudinal cohort studies and registries are most often used for developing risk calculators for long-term outcomes. These observational studies are a more practical and cost-effective way to obtain long-term follow-up than randomized clinical trials. However, observational studies suffer from confounding that can be difficult or impossible to address. For example, event rates from the past may not apply to contemporary patients due to changes in treatments, medical care or patient characteristics (e.g., obesity). Another source of confounding in observational studies is confounding by indication. In patients with RA, those with more severe disease are more likely to receive biologics. If biologics have an impact on CVD risk, this could bias the results of CVD risk assessment in patients with RA. The alternative to observational studies is randomized clinical trials. While randomized trials have the advantage of reducing confounding, they also have limitations. Importantly, patients with comorbidities are often excluded in randomized trials, so the trial population is unlikely to be representative of the population of interest for CVD risk evaluation in RA.

Risk factor selection

Risk calculators are predicated on the notion that multiple risk factors may interact to increase the risk of developing a disease. Patients with modest increases in several risk factors may be at an equivalent or higher risk of disease than patients with one highly elevated risk factor. Therefore, clinical management decisions should account for the constellation of risk factors in order to optimize preventive strategies. Traditional CVD risk factors include age, sex, hypertension, hyperlipidemia, smoking and diabetes mellitus. Considerations for adding other risk factors for calculation of total CVD risk include assessments of the incremental value with regard to improvement in risk predictions, as well as the cost of recording of the risk factor in terms of both money and patient and clinician time/effort. DAS28 and HAQ were both considered and included in the risk prediction models developed in our study. HAQ is obtained from a short patient questionnaire, but DAS28 requires a professional joint assessment as well as an inflammatory marker, which make it a bit more time-consuming and expensive to measure than many of the traditional CVD risk factors.

In addition, causality is important, as risk factors included in a risk calculator are assumed to result in lower risk when they are improved. However, causality is difficult to prove except in randomized clinical trials, which are currently lacking regarding preventive interventions and CVD outcome in patients with RA. Yet promising results from post hoc analyses from 2 large randomized placebo controlled trials with hard CVD endpoints (the IDEAL and TNT trials) showed that patients without and with inflammatory joint diseases, including patients with RA, had comparable lipid lowering effects and reduction in CVD events after intensive treatment with statins [37].

There is a lack of optimal biomarkers to track RA disease activity. RF and ACPA are measures of disease severity, not disease activity. ESR and CRP are markers of inflammation, which correlate with disease activity. However, CRP has been shown to be a mediator, not a cause, of CVD in Mendelian randomization studies [38]. Swollen and tender joint counts and patient and physician global VAS are measures of disease activity, which have been incorporated into composite disease activity measures, such as the DAS28 and Clinical Disease Activity Index, although these measures are somewhat subjective [39, 40]. In addition, disease activity fluctuates over time in patients with RA, so it may not be reasonable to expect a single disease activity measure to accurately predict 10-year CVD risk. Indeed, cumulative measures of disease activity have shown stronger associations with CVD risk than single measurements among patients with RA [41, 42]. However, cumulative measures are impractical to implement in a risk calculator due to the difficulties of ascertaining them in clinical practice. Therapies for RA are also tempting to consider as risk factors for CVD in patients with RA, but their use is confounded by indication in observational studies. High dose glucocorticoids are thought to increase CVD risk, but low dose glucocorticoids may reduce CVD risk by controlling inflammation [43]. Methotrexate and biologic response modifiers may also modulate the risk of CVD, but causal associations have not been proven [44, 45]. These issues may partially explain why our risk prediction models did not perform better than CVD risk calculators for the general population, which do not incorporate RA-specific factors.

Sample size considerations

Risk calculator development requires a large dataset with a high number of events. Obtaining a large sample size for a study of patients with RA can be difficult. Despite the fact that RA is one of the more common rheumatologic diseases, its prevalence is only 1%. Single centers typically have too few patients to develop risk calculators for patients with RA. However, pooling data from multiple centers may have challenges. Retrospectively collected data is often used to avoid waiting for follow-up to accrue in a prospective study. Despite predefined variables it may be difficult to ensure that study-specific factors (e.g., variable definitions, surveillance methods, identification of subjects, participation rates, etc.) are similar across centers. Registries often include larger numbers of patients than single center studies and are thought to be less heterogeneous due to standardized data collection procedures, but selection bias for enrolled subjects is an important issue that is often ignored. Heterogeneity can make it difficult or even impossible to develop a risk calculator depending on whether the sources of the heterogeneity can be adequately explained and counteracted or not. Our study found significant heterogeneity between centers, which was addressed by including low and high risk strata in our risk prediction models.

Performance assessment

The performance of a risk calculator requires assessment of both discrimination and calibration. Discrimination is correctly ranking patients from low to high risk. It is commonly assessed using the area under the receiver operating characteristic curve, which is also known as the c-statistic. The c-statistic has a small practical range from a minimum of 0.5 (i.e., no predictive ability) to a hypothetical maximum of about 0.88. This narrow range has led to criticism of the c-statistic as a useful measure of the incremental value of new risk factors [46]. The c-statistics of our models were not better than the c-statistics for the general population risk calculators.

Calibration is accurately predicting the absolute risk level. It is often assessed using the Hosmer-Lemeshow goodness-of-fit test for binary outcomes or analogous variations for time-to-event outcomes [10]. Van Calster et al define 4 levels of calibration including: 1) mean calibration (i.e., calibration-in-the-large; or simply that the average predicted risk is similar to the average observed risk), 2) weak calibration (i.e., a calibration intercept of zero and a calibration slope of one; or simply that there is no systematic over or under estimation of predicted risks), 3) moderate calibration (i.e., among patients with the same predicted risk, the observed risk equals the predicted risk, so among patients with a 10% predicted risk, 10% will have an observed event) and 4) strong calibration (i.e., predicted risks and observed event rates correspond for every covariate pattern) [8]. Strong correlation is desirable but unrealistic and may lead to overly complex models, so moderate calibration has been recommended. Graphical assessments of calibration are often more useful than statistical tests [6]. While our risk prediction models showed better calibration than for the general population risk calculators, this could reflect the fact that re-calibration is often needed when applying a risk calculator to a different population and calibration is expected to be better in the cohort used to develop the model than in other cohorts.

All risk calculators should be validated prior to adoption for clinical use. Validation can be done internally or externally. Internal validation can be performed by dividing the original cohort into derivation and test sets or by using cross-validation techniques. We used 10-fold cross-validation to assess the performance of our risk calculator models, in order to avoid the over-optimism associated with assessing a model in the same data that was used to develop it. Unfortunately, this internal validation did not show improved performance of our models compared to general population risk calculators.

The lack of improvement in CVD risk assessment demonstrated by our models is consistent with the recent findings of Alemao et al that adding CRP to the FRS or QRISK2 did not improve CVD risk assessment for patients with RA [47]. Similarly, the Reynolds risk score, which includes CRP, did not demonstrate improved performance for CVD risk assessment among patients with RA [2] Since the increased risk of CVD among patients with RA likely stems from the systemic inflammation that characterizes RA, adding an inflammatory marker to a CVD risk calculator seems reasonable, but it has not proven to be effective. As mentioned previously, this lack of improvement may stem from the inherent fluctuations in disease activity over time in patients with RA, so it may not be reasonable to expect a single disease activity measure to accurately predict 10-year CVD risk. Alternatively, RA therapies may be altering CVD risk. Since most patients with RA will change treatments several times during a 10 year period, it could be difficult to accurately estimate 10-year CVD risk without knowing which treatments the patient will receive in the future. Therefore, accurate assessment of CVD risk in patients with RA is complex.

Conclusion

Our risk calculators did not demonstrate improved performance among patients with RA compared to general population CVD risk calculators. There are many issues involved in derivation of a risk calculator, which make development of a CVD risk calculator for use in patients with RA particularly challenging. These challenges should be considered by others who may attempt to make a CVD risk calculator for patients with RA and by those who seek to develop risk calculators for other purposes in other patient populations. In addition to the challenges in development of a risk calculator, there may be certain obstacles in implementation of such a calculator in clinical practice. A risk calculator that includes both cardiovascular and rheumatologic factors might require coordinated care from both a rheumatologist and a cardiologist or general practitioners to obtain all the measures needed to assess it, particularly if joint counts were required. Even patient-reported measures like the HAQ require time for calculation [48].

Finally, the true worth of a risk calculator depends on its ability to improve health outcomes. However, few studies have been done in the general population to prove that the use of a CVD risk calculator results in improved outcomes compared to single factor based approaches, despite the fact that the utility of risk calculators is assumed by several treatment guidelines for CVD risk prevention [5, 49, 50]. Among patients with RA, traditional CVD risk factors cannot explain all of the increased risk for CVD. RA disease specific factors have also been shown to play a role in the increased CVD risk, but it is not known whether optimal control of inflammation and RA disease activity would reduce the risk of CVD to that of the general population [51, 52]. Further research is needed to determine the best approaches to mitigating the increased risk of CVD among patients with RA. Meanwhile, clinicians should follow guidelines for CVD risk assessment in the general population and make sure that traditional CVD risk factors are optimally treated in their patients with RA.

Acknowledgments

ATACC-RA members: Anne Grete Semb1 (email: a-semb@diakonsyk.no), Silvia Rollefstad1, Eirik Ikdahl1, Tore K. Kvien1, Sherine E. Gabriel6,16, Cynthia S. Crowson6, Eric L Matteson6, George D. Kitas3, Karen Douglas3, Aamer Sandoo3, Piet Van Riel5, Elke Arts5, Jaap Fransen5, Solbritt Rantapää-Dahlqvist2, Solveig Wållberg-Jonsson2, Lena Innala2, George Karpouzas7, Petros P. Sfikakis4, Evi Zampeli4, Patrick H. Dessein10, Linda Tsang10, Miguel A. Gonzalez-Gay9, Alfonso Corrales9, Hani El-Gabalawy11, Carol Hitchon11, Virginia Pascual Ramos12, Irazú Contreras Yáñez12, Daniel Solomon8, Katherine Liao8, Mart van de Laar13, Harald Vonkeman13, Inger Meek13, Elaine Husni14, Robert Overman14, Iris Colunga15, Dionicio Galarza15 José Ramón Azpiri-López15. 1Diakonhjemmet hospital, Oslo, Norway, 2University of Umeå, Umeå, Sweden, 3Dudley Group NHS Foundation Trust, West Midlands, United Kingdom, 4University of Athens, Athens, Greece, 5Radboud University, Nijmegen, Netherlands, 6Mayo Clinic, Rochester, MN, 7Harbor UCLA Medical Center RHU, Torrance, CA, 8Harvard Medical School Brigham and Women's Hospital, Boston, MA, United States, 9Hospital Universitario Marques de Valdecilla, Santander (Cantabria), Spain, 10 Rheumatology Division, Universitair Ziekenhuis and Vrije Universiteit Brussel, Belgium and Cardiovascular Pathophysiology and Genomics Research Unit, School of Physiology, Faculty of Heath Sciences, University of Witwatersrand, Johannesburg, South Africa and Rheumatology Division, Universitair Ziekenhuis and Vrije Universiteit, Brussels, Belgium, 11University of Manitoba, Winnipeg, Manitoba, Canada, 12Instituto Nacional de Ciencias Médicas y Nutrición Salvador Zubirán, Mexico City, Mexico, 13 Hospital Medisch Spectrum Twente, Enschede, Netherlands, 14Cleveland Clinic, Cleveland, Ohio, United States, 15Hospital Universitario "Dr. José E. González", Monterrey, Mexico, 16 Rutgers Robert Wood Johnson Medical School, New Brunswick, NJ, USA

Author Contributions

  1. Conceptualization: GDK PLCMR SEG AGS.
  2. Data curation: CSC.
  3. Formal analysis: CSC.
  4. Funding acquisition: GDK PLCMR SEG AGS.
  5. Investigation: CSC SR GDK PLCMR SEG AGS.
  6. Methodology: CSC.
  7. Project administration: CSC SR GDK PLCMR SEG AGS.
  8. Resources: CSC SR GDK PLCMR SEG AGS.
  9. Software: CSC.
  10. Supervision: CSC SR GDK PLCMR SEG AGS.
  11. Validation: CSC.
  12. Visualization: CSC.
  13. Writing – original draft: CSC.
  14. Writing – review & editing: CSC SR GDK PLCMR SEG AGS.

References

  1. 1. Peters MJ, Symmons DP, McCarey D, Dijkmans BA, Nicola P, Kvien TK, et al. EULAR evidence-based recommendations for cardiovascular risk management in patients with rheumatoid arthritis and other forms of inflammatory arthritis. Ann Rheum Dis. 2010;69(2):325–31. Epub 2009/09/24. pmid:19773290
  2. 2. Crowson CS, Matteson EL, Roger VL, Therneau TM, Gabriel SE. Usefulness of risk scores to estimate the risk of cardiovascular disease in patients with rheumatoid arthritis. Am J Cardiol. 2012;110(3):420–4. Epub 2012/04/24. pmid:22521305
  3. 3. Arts EE, Popa C, Den Broeder AA, Semb AG, Toms T, Kitas GD, et al. Performance of four current risk algorithms in predicting cardiovascular events in patients with early rheumatoid arthritis. Ann Rheum Dis. 2015;74(4):668–74. Epub 2014/01/07. pmid:24389293
  4. 4. Piepoli MF, Hoes AW, Agewall S, Albus C, Brotons C, Catapano AL, et al. 2016 European Guidelines on cardiovascular disease prevention in clinical practice: The Sixth Joint Task Force of the European Society of Cardiology and Other Societies on Cardiovascular Disease Prevention in Clinical Practice (constituted by representatives of 10 societies and by invited experts) Developed with the special contribution of the European Association for Cardiovascular Prevention & Rehabilitation (EACPR). Atherosclerosis. 2016;252:207–74. Epub 2016/09/25. pmid:27664503
  5. 5. Goff DC Jr., Lloyd-Jones DM, Bennett G, Coady S, D'Agostino RB Sr., Gibbons R, et al. 2013 ACC/AHA guideline on the assessment of cardiovascular risk: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines. J Am Coll Cardiol. 2014;63(25 Pt B):2935–59. Epub 2013/11/19. pmid:24239921
  6. 6. Steyerberg EW, Vickers AJ, Cook NR, Gerds T, Gonen M, Obuchowski N, et al. Assessing the performance of prediction models: a framework for traditional and novel measures. Epidemiology. 2010;21(1):128–38. Epub 2009/12/17. pmid:20010215
  7. 7. Hilden J, Gerds TA. A note on the evaluation of novel biomarkers: do not rely on integrated discrimination improvement and net reclassification index. Stat Med. 2014;33(19):3405–14. Epub 2013/04/05. pmid:23553436
  8. 8. Van Calster B, Nieboer D, Vergouwe Y, De Cock B, Pencina MJ, Steyerberg EW. A calibration hierarchy for risk models was defined: from utopia to empirical data. J Clin Epidemiol. 2016;74:167–76. Epub 2016/01/17. pmid:26772608
  9. 9. Leening MJ, Steyerberg EW, Van Calster B, D'Agostino RB Sr., Pencina MJ. Net reclassification improvement and integrated discrimination improvement require calibrated models: relevance from a marker and model perspective. Stat Med. 2014;33(19):3415–8. Epub 2014/07/22. pmid:25042215
  10. 10. Crowson CS, Atkinson EJ, Therneau TM. Assessing calibration of prognostic risk scores. Stat Methods Med Res. 2016;25(4):1692–706. Epub 2013/08/03. pmid:23907781
  11. 11. Moons KG, Altman DG, Reitsma JB, Ioannidis JP, Macaskill P, Steyerberg EW, et al. Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis (TRIPOD): explanation and elaboration. Ann Intern Med. 2015;162(1):W1–73. Epub 2015/01/07. pmid:25560730
  12. 12. Hippisley-Cox J, Coupland C, Vinogradova Y, Robson J, Minhas R, Sheikh A, et al. Predicting cardiovascular risk in England and Wales: prospective derivation and validation of QRISK2. Bmj. 2008;336(7659):1475–82. Epub 2008/06/25. pmid:18573856
  13. 13. Solomon DH, Greenberg J, Curtis JR, Liu M, Farkouh ME, Tsao P, et al. Derivation and internal validation of an expanded cardiovascular risk prediction score for rheumatoid arthritis: a Consortium of Rheumatology Researchers of North America Registry Study. Arthritis Rheumatol. 2015;67(8):1995–2003. Epub 2015/05/20. pmid:25989470
  14. 14. Panoulas VF, Douglas KM, Milionis HJ, Stavropoulos-Kalinglou A, Nightingale P, Kita MD, et al. Prevalence and associations of hypertension and its control in patients with rheumatoid arthritis. Rheumatology (Oxford). 2007;46(9):1477–82. Epub 2007/08/21.
  15. 15. Rollefstad S, Kvien TK, Holme I, Eirheim AS, Pedersen TR, Semb AG. Treatment to lipid targets in patients with inflammatory joint diseases in a preventive cardio-rheuma clinic. Ann Rheum Dis. 2013;72(12):1968–74. Epub 2012/12/25. pmid:23264359
  16. 16. Innala L, Moller B, Ljung L, Magnusson S, Smedby T, Sodergren A, et al. Cardiovascular events in early RA are a result of inflammatory burden and traditional risk factors: a five year prospective study. Arthritis Res Ther. 2011;13(4):R131. Epub 2011/08/17. pmid:21843325
  17. 17. Karpouzas GA, Malpeso J, Choi TY, Li D, Munoz S, Budoff MJ. Prevalence, extent and composition of coronary plaque in patients with rheumatoid arthritis without symptoms or prior diagnosis of coronary artery disease. Ann Rheum Dis. 2014;73(10):1797–804. Epub 2013/07/28. pmid:23887286
  18. 18. Protogerou AD, Panagiotakos DB, Zampeli E, Argyris AA, Arida K, Konstantonis GD, et al. Arterial hypertension assessed "out-of-office" in a contemporary cohort of rheumatoid arthritis patients free of cardiovascular disease is characterized by high prevalence, low awareness, poor control and increased vascular damage-associated "white coat" phenomenon. Arthritis Res Ther. 2013;15(5):R142. Epub 2013/11/30. pmid:24286134
  19. 19. Dessein PH, Tsang L, Woodiwiss AJ, Norton GR, Solomon A. Circulating concentrations of the novel adipokine chemerin are associated with cardiovascular disease risk in rheumatoid arthritis. J Rheumatol. 2014;41(9):1746–54. Epub 2014/07/17. pmid:25028378
  20. 20. Corrales A, Gonzalez-Juanatey C, Peiro ME, Blanco R, Llorca J, Gonzalez-Gay MA. Carotid ultrasound is useful for the cardiovascular risk stratification of patients with rheumatoid arthritis: results of a population-based study. Ann Rheum Dis. 2014;73(4):722–7. Epub 2013/03/19. pmid:23505241
  21. 21. Harris JA, Bykerk VP, Hitchon CA, Keystone EC, Thorne JC, Boire G, et al. Determining best practices in early rheumatoid arthritis by comparing differences in treatment at sites in the Canadian Early Arthritis Cohort. J Rheumatol. 2013;40(11):1823–30. Epub 2013/09/17. pmid:24037554
  22. 22. Contreras-Yanez I, Pascual-Ramos V. Window of opportunity to achieve major outcomes in early rheumatoid arthritis patients: how persistence with therapy matters. Arthritis Res Ther. 2015;17:177. Epub 2015/07/15. pmid:26162892
  23. 23. Mjaavatten MD, Radner H, Yoshida K, Shadick NA, Frits ML, Iannaccone CK, et al. Inconsistent treatment with disease-modifying antirheumatic drugs: a longitudinal data analysis. J Rheumatol. 2014;41(12):2370–8. Epub 2014/10/17. pmid:25320216
  24. 24. Meek IL, Picavet HS, Vonkeman HE, Verschuren WM, van de Laar MA. Increased cardiovascular risk factors in different rheumatic diseases compared with the general population. Rheumatology (Oxford). 2013;52(1):210–6. Epub 2012/08/01.
  25. 25. Morris TP, White IR, Royston P, Seaman SR, Wood AM. Multiple imputation for an incomplete covariate that is a ratio. Stat Med. 2014;33(1):88–104. Epub 2013/08/08. pmid:23922236
  26. 26. Cole SR, Hernan MA. Adjusted survival curves with inverse probability weights. Comput Methods Programs Biomed. 2004;75(1):45–9. Epub 2004/05/26. pmid:15158046
  27. 27. Tibshirani R. The lasso method for variable selection in the Cox model. Stat Med. 1997;16(4):385–95. pmid:9044528
  28. 28. Harrell FE Jr., Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. 1996;15(4):361–87. Epub 1996/02/28. pmid:8668867
  29. 29. D'Agostino RB Sr., Vasan RS, Pencina MJ, Wolf PA, Cobain M, Massaro JM, et al. General cardiovascular risk profile for use in primary care: the Framingham Heart Study. Circulation. 2008;117(6):743–53. Epub 2008/01/24. pmid:18212285
  30. 30. Conroy RM, Pyorala K, Fitzgerald AP, Sans S, Menotti A, De Backer G, et al. Estimation of ten-year risk of fatal cardiovascular disease in Europe: the SCORE project. Eur Heart J. 2003;24(11):987–1003. Epub 2003/06/06. pmid:12788299
  31. 31. Leening MJ, Vedder MM, Witteman JC, Pencina MJ, Steyerberg EW. Net reclassification improvement: computation, interpretation, and controversies: a literature review and clinician's guide. Ann Intern Med. 2014;160(2):122–31. Epub 2014/03/05. pmid:24592497
  32. 32. Cutolo M, Kitas GD, van Riel PL. Burden of disease in treated rheumatoid arthritis patients: going beyond the joint. Semin Arthritis Rheum. 2014;43(4):479–88. Epub 2013/10/02. pmid:24080116
  33. 33. Putrik P, Sokka T, Ramiro S, Boonen A. Impact of socioeconomic gradients within and between countries on health of patients with rheumatoid arthritis (RA): lessons from QUEST RA. Best Pract Res Clin Rheumatol. 2012;26(5):705–20. Epub 2012/12/12. pmid:23218433
  34. 34. Putrik P, Ramiro S, Hifinger M, Keszei AP, Hmamouchi I, Dougados M, et al. In wealthier countries, patients perceive worse impact of the disease although they have lower objectively assessed disease activity: results from the cross-sectional COMORA study. Ann Rheum Dis. 2016;75(4):715–20. Epub 2015/09/01. pmid:26314921
  35. 35. Putrik P, Ramiro S, Keszei AP, Hmamouchi I, Dougados M, Uhlig T, et al. Lower education and living in countries with lower wealth are associated with higher disease activity in rheumatoid arthritis: results from the multinational COMORA study. Ann Rheum Dis. 2016;75(3):540–6. Epub 2015/01/30. pmid:25630346
  36. 36. Yoon SS, Dillon CF, Illoh K, Carroll M. Trends in the Prevalence of Coronary Heart Disease in the U.S.: National Health and Nutrition Examination Survey, 2001–2012. Am J Prev Med. 2016. Epub 2016/04/27.
  37. 37. Semb AG, Kvien TK, DeMicco DA, Fayyad R, Wun CC, LaRosa JC, et al. Effect of intensive lipid-lowering therapy on cardiovascular outcome in patients with and those without inflammatory joint disease. Arthritis Rheum. 2012;64(9):2836–46. Epub 2012/05/12. pmid:22576673
  38. 38. Wensley F, Gao P, Burgess S, Kaptoge S, Di Angelantonio E, Shah T, et al. Association between C reactive protein and coronary heart disease: mendelian randomisation analysis based on individual participant data. Bmj. 2011;342:d548. Epub 2011/02/18. pmid:21325005
  39. 39. Prevoo ML, van 't Hof MA, Kuper HH, van Leeuwen MA, van de Putte LB, van Riel PL. Modified disease activity scores that include twenty-eight-joint counts. Development and validation in a prospective longitudinal study of patients with rheumatoid arthritis. Arthritis Rheum. 1995;38(1):44–8. Epub 1995/01/01. pmid:7818570
  40. 40. Aletaha D, Nell VP, Stamm T, Uffmann M, Pflugbeil S, Machold K, et al. Acute phase reactants add little to composite disease activity indices for rheumatoid arthritis: validation of a clinical activity score. Arthritis Res Ther. 2005;7(4):R796–806. Epub 2005/07/01. pmid:15987481
  41. 41. Myasoedova E, Chandran A, Ilhan B, Major BT, Michet CJ, Matteson EL, et al. The role of rheumatoid arthritis (RA) flare and cumulative burden of RA severity in the risk of cardiovascular disease. Ann Rheum Dis. 2016;75(3):560–5. Epub 2015/02/01. pmid:25637001
  42. 42. Myasoedova E, Crowson CS, Kremers HM, Roger VL, Fitz-Gibbon PD, Therneau TM, et al. Lipid paradox in rheumatoid arthritis: the impact of serum lipid measures and systemic inflammation on the risk of cardiovascular disease. Ann Rheum Dis. 2011;70(3):482–7. Epub 2011/01/11. pmid:21216812
  43. 43. Davis JM 3rd, Maradit Kremers H, Crowson CS, Nicola PJ, Ballman KV, Therneau TM, et al. Glucocorticoids and cardiovascular events in rheumatoid arthritis: a population-based cohort study. Arthritis Rheum. 2007;56(3):820–30. Epub 2007/03/03. pmid:17330254
  44. 44. Barnabe C, Martin BJ, Ghali WA. Systematic review and meta-analysis: anti-tumor necrosis factor alpha therapy and cardiovascular events in rheumatoid arthritis. Arthritis Care Res (Hoboken). 2011;63(4):522–9. Epub 2010/10/20.
  45. 45. Roubille C, Richer V, Starnino T, McCourt C, McFarlane A, Fleming P, et al. The effects of tumour necrosis factor inhibitors, methotrexate, non-steroidal anti-inflammatory drugs and corticosteroids on cardiovascular events in rheumatoid arthritis, psoriasis and psoriatic arthritis: a systematic review and meta-analysis. Ann Rheum Dis. 2015;74(3):480–9. Epub 2015/01/07. pmid:25561362
  46. 46. Sniderman AD, Pencina M, Thanassoulis G. Limitations in the conventional assessment of the incremental value of predictors of cardiovascular risk. Curr Opin Lipidol. 2015;26(3):210–4. Epub 2015/04/19. pmid:25887681
  47. 47. Alemao E, Cawston H, Bourhis F, Al M, Rutten-van Molken M, Liao KP, et al. Comparison of cardiovascular risk algorithms in patients with vs without rheumatoid arthritis and the role of C-reactive protein in predicting cardiovascular outcomes in rheumatoid arthritis. Rheumatology (Oxford). 2017. Epub 2017/01/15.
  48. 48. Bruce B, Fries JF. The Stanford Health Assessment Questionnaire: a review of its history, issues, progress, and documentation. J Rheumatol. 2003;30(1):167–78. Epub 2003/01/01. pmid:12508408
  49. 49. Cooney MT, Dudina AL, Graham IM. Value and limitations of existing scores for the assessment of cardiovascular risk: a review for clinicians. J Am Coll Cardiol. 2009;54(14):1209–27. Epub 2009/09/26. pmid:19778661
  50. 50. Reiner Z, Catapano AL, De Backer G, Graham I, Taskinen MR, Wiklund O, et al. ESC/EAS Guidelines for the management of dyslipidaemias: the Task Force for the management of dyslipidaemias of the European Society of Cardiology (ESC) and the European Atherosclerosis Society (EAS). Eur Heart J. 2011;32(14):1769–818. Epub 2011/06/30. pmid:21712404
  51. 51. Solomon DH, Kremer J, Curtis JR, Hochberg MC, Reed G, Tsao P, et al. Explaining the cardiovascular risk associated with rheumatoid arthritis: traditional risk factors versus markers of rheumatoid arthritis severity. Ann Rheum Dis. 2010;69(11):1920–5. Epub 2010/05/07. pmid:20444756
  52. 52. Symmons DP. Do we need a disease specific cardiovascular risk calculator for patients with rheumatoid arthritis? Arthritis Rheumatol. 2015. Epub 2015/05/20.