Search Strategy

This review has been undertaken with adherence to the PRISMA statement[2]. MEDLINE, Embase, Scopus and ISI Web of Science were searched using combinations of the following terms and mapped to Medical Subject Headings (MeSH): “dementia”, “Alzheimer disease”, “Alzheimer and disease”, “predict”, “develop”, “incident”, “sensitivity”, “specificity”, “ROC” and “area under the curve”. The search included all literature published between the 1 January 2009 to 17 March 2014 (see Table 1 for the search strategy). Only articles published in English were considered. Additional articles were identified from the reference lists of eligible studies and relevant reviews. All articles published from January 2009 –November 2009 were also removed as this was covered in the original review.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Search Strategy.

https://doi.org/10.1371/journal.pone.0136181.t001

Selection of Studies

Selection of articles followed the same protocol as the 2010 review[1]. Two authors (ET and SH) independently searched publications using the following inclusion criteria: the sample was population-based and the article examined risk of dementia in non-demented individuals and included measurements of sensitivity, specificity or discrimination (i.e., area under the curve: AUC or c-statistic). Cross-sectional, case-control and clinical-based studies were excluded, as were studies that restricted the baseline sample by cognitive criteria other than dementia status; for example studies were excluded that focused only on subjects with Mild Cognitive Impairment (MCI). Articles where the outcome was a combined cognitive group, for example dementia cases combined with individuals with MCI, were also excluded. Titles and abstracts were searched first, followed by the full text of any identified articles. Where duplicate studies were identified, details of all novel risk models and their performance indices (AUC/c-statistic, sensitivity or specificity) were extracted from each publication and reported separately. One set of duplicate publications was found[3, 4]. However, the first paper[3] only presented the model and did not undertake testing. This paper was therefore excluded. Disagreements were solved by consensus between the two authors (ET and SH), or by a third author (BCMS) if the disagreement could not be resolved.

Data Extraction

Two authors (ET and SH) independently extracted information from each study including: sample, country, length of follow-up, baseline age, sex distribution, outcome tested (e.g., all-cause dementia vs. dementia subtypes), the components of each prediction model, discriminative accuracy (AUC or c-statistic), and where available, sensitivity and specificity estimates of cut-off scores and positive/negative likelihood ratio (LR+ or LR-, respectively). Two authors (ET, SH) independently assessed the quality of the included studies using an adapted version of the Newcastle-Ottawa Scale (NOS) for non-randomized studies, specifically cohort studies[5], as endorsed by the Cochrane collaboration [6]. The NOS uses a star rating system to assess selection, comparability and outcome criteria. Items describing a non-intervention cohort were excluded and therefore the total ranking was out of 6 (rather than 9).

Included Studies

A total of 1,234 articles were identified from the electronic literature search (after removing duplicate publications) and 11 articles were identified from other sources. In total, 21 articles describing dementia risk prediction models were included in this review (Fig 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA (2009) flow diagram of article selection.

https://doi.org/10.1371/journal.pone.0136181.g001

Table 2 summarises the study characteristics and predictive models. 13 new study populations[7–20] have been used to create risk models compared to the first review (all literature till 2009)[1]. There was however some overlap including use of data from the Cardiovascular Health Cognition Study[20, 21], Leipzig Longitudinal Study of the Aged (LEILA 75+)[22], Einstein Aging Study[23], Vienna TransDanube Aging[24], Canadian Study of Health and Aging[25] and the Kungsholmen Project[4]. Sample sizes ranged from 194[11] to 29,961[13]. Follow-up duration ranged from 1.4[22] to 20 years[7]. Follow-up rate ranged from 66%[12] to 86%[19], with 10 studies[4, 7, 11, 14, 15, 17, 20, 21, 23, 25] not reporting attrition or dropout. In ten studies[7, 10, 12–14, 16, 20–22, 26] the outcome tested was all-cause dementia, in nine studies[8, 9, 11, 15, 17–19, 23, 24] the outcome was AD and two studies had separate models for both AD and all-cause dementia [4, 25]. The number of predictors ranged from one[22] to 19[25].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Characteristics of included studies and reported dementia risk prediction models.

https://doi.org/10.1371/journal.pone.0136181.t002

Quality Assessment

Articles were assessed on selection, comparability and outcome (out of a maximum of six stars). In total, six[8, 13, 16, 18, 19, 26] articles scored six stars (maximum), 13 scored five stars[4, 7, 9–12, 14, 15, 17, 20, 22–24] and two[21, 25] scored four stars. This indicates that most articles were of high or moderate quality. Star ratings for each article are shown in Table 2.

Model Development

Most models have been derived using Logistic Regression[7, 9, 10, 21, 24–26] or Cox Proportional Hazards Regression analysis[8, 11, 13, 15–20, 22, 23], usually using stepwise selection to identify candidate predictors (e.g., based on a p-value; forwards or backwards), with one model using the Bayesian Information Criterion[19]). One model, the Australian National University Alzheimer’s Disease Risk Score (ANU-ADRI), was developed using an Evidence-Based Medicine Approach rather than through a data analytical approach[4] and another, the Brief Dementia Screening Indicator (BDSI) was developed using data synthesis based on the best dementia predictors identified in four different cohort studies[20]. When computed, simple risk scores have been derived from the model’s Beta-[4, 13, 16, 19, 20] or logit-coefficients[21]. This is similar to the methodology employed for risk model development in other fields of medicine such as cardiovascular disease[27, 28]. No models have yet been developed using systems biology or neural network approaches. Neural networks simulate the functions of neurons of human brains which can interact for processing data and learning from experiences [29]. Given the potential complexity of the risk factor variables used in dementia risk prediction, neural networks may also hold promise although this has yet to be evaluated.

Risk Models

Models can be broadly divided into the following categories: (1) demographic only models; (2) cognitive based models (incorporating cognitive test scores, with or without subjective memory/cognitive complaint indicators or demographic data); (3) health variables and health risk indices (incorporating self-reported or objectively measured health status); (4) genetic risk scores including APOE, PICALM (Phosphatidylinositol binding clathrin assembly protein), CLU (Clusterin) and other genes associated with AD (i.e., BIN1, CR1, ABCA7, MS4A6A, MS4A4E, CD2AP, EPHA1 and CD33), either alone or in combination with non-genetic variables; and, (5) multi-variable models typically incorporating demographic, health and lifestyle measures. Table 3 shows comparisons of the model components between this and the 2010 review and illustrates that there is large variability in model components and differences across the two reviews.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Component Variables Used (Either Alone or in Combination) in the Different Risk Prediction Models (Previous and Current Review).

https://doi.org/10.1371/journal.pone.0136181.t003

Differences are mainly in the addition of novel (non-traditional) dementia risk variables[25], information on diet[4], depression symptomology[4, 13, 24], ethnicity[14, 20, 23], and extension of genetic analysis to include non-APOE genes[17]. In addition, fewer cognitive tests are used and there is a smaller pool of candidate risk factors, likely due to more evidence being available.

Model Diagnostics

Performance of models has been assessed using measures of discriminative accuracy (e.g., AUC/c-statistic), sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), internal calibration, LR+/LR-, the net reclassification index (NRI) and the integrated discrimination improvement (IDI) statistic. Discriminative accuracy was measured in all studies and ranged from low 0.49[4] to moderate 0.89[9]. Cut-off points with sensitivity and specificity estimates were only reported in five studies[7, 19, 22, 23, 26]. Where available, cut-off points were determined as follows: (1) maximisation of Youden’s index (Formula = Sensitivity + Specificity– 1)[22, 23, 26]; (2) computed by cross validation to correct for optimism as a result of validation on the learning data to correspond to a sensitivity value[7]; or, (3) defining the cut-off scores as those values with high specificity and increased PPV[19]. No model reported a cut-off score with both sensitivity and specificity over 80%. Three studies reported PPVs: (1) 9 to 41% (range across different length of follow-up interval and educational level)[7]; (2) 6.6 to 49.9% (range across different cut-off scores)[23]; and, (3) 14.7% with a cut-off that provided sensitivity of at least 80%[19]. Two studies reported NPVs: range across different cut-off scores: 86.0 to 99.0%[7] and 97.7%[19]. NPV should be higher than the proportion of subjects who did not have the outcome of dementia (i.e., the “stable” subjects), if the prediction is better than chance. Since the proportion of subjects not becoming demented was generally >85% (and always >70%)–as inferred from Table 2 –a high NPV is expected, because the number depends on the prevalence of the disease in the sample. The same logic can be applied to PPV. It should be noted that because the proportion of subjects becoming demented (or not) varies among the populations, variations in PPV and NPV do not necessarily reflect differences in performance of the models. Only one study[26] reported LR+/LR- and found that a neuropsychological prediction model provided a clinically important change in pre- to post-test probability of converting to dementia (all cause) over 5 and 10 years follow-up.

Model calibration was rarely reported, and where reported indicated good fit[13, 14, 20]. Reclassification indices including NRI and the IDI statistics that test the addition of variables to risk models, were used in two studies[14, 15]. The first study evaluated the influence of the APOE genotype on the accuracy of AD risk assessment using NRI and found a significant improvement (NRI 0.18, Z_NRI = 2.47, P = 0.01) compared to a non-APOE model; the IDI was estimated as 6.25 (Z_IDI = 3.75, P <0.001)[15]. The second used NRI and IDI to assess the improvements in performance of the Cardiovascular Risk Factors, Aging and Dementia (CAIDE) risk score by adding new risk factors[14]. Here, both the NRI and IDI showed no model improvements with the additional variables.

Models for Specific Subgroups of Individuals

One study developed a risk score, the Type-2 Diabetes Specific Dementia Risk Score (DSDRS), for predicting 10-year incident dementia, in a primary care setting, in a large cohort of individuals (N = 29,961) with type II diabetes[13]. The model incorporated age, education, microvascular disease, diabetic foot, cerebrovascular disease, cardiovascular disease, acute metabolic event and depression and was well calibrated and externally validated with moderate levels of predictive accuracy (AUC = 0.74 development cohort vs. AUC = 0.75 validation cohort). In another study, model development was undertaken in a sample stratified by education level (low: no elementary school diploma vs. high: secondary school or university) and follow-up time (3 vs. 10 years)[7]. This resulted in four different models that varied by education and length of follow-up as shown in Table 2.

Model Validation

Only four studies have undertaken validation[4, 13, 14, 20]. Differences between the AUCs in the development and validation cohorts for the different models tested are shown in Fig 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Comparison of AUC indices in development vs. validation cohorts across different dementia risk prediction models.

Key: BDSI, Brief Dementia Screening Index; CAIDE, Cardiovascualr Risk Factors, Aging and Dementia; CHS, Cardiovascular Health Study; CVHS, Cardiovascular Health Cognition Study; DSDRS, Type-2 Diabetes Specific Dementia Risk Score; FHS, Framingham Heart Study; KP, Kungsholmen Project; KPNC, Kaiser Permanente Medical Care Program of Northern California; MAP, Rush Memory and Aging Project; PS-W, Pathways study cohort from Washington; Pts, Points; SALSA Sacramento Area Latino Study on Aging. References [1] Anstey KJ, Cherbuin N, Herath PM, Qiu C, Kuller LH, Lopez OL, et al. A Self-Report Risk Index to Predict Occurrence of Dementia in Three Independent Cohorts of Older Adults: The ANU-ADRI. PLoS One. 2014;9(1):e86141; [2] Exalto LG QC, Barnes D, Kivipelto M, Biessels GJ, Whitmer RA. Midlife risk score for the prediction of dementia four decades later. Alzheimers Dementia. 2013; [3] Exalto LG, Biessels GJ, Karter AJ, Huang ES, Katon WJ, Minkoff JR, et al. Risk score for prediction of 10 year dementia risk in individuals with type 2 diabetes: a cohort study. The Lancet Diabetes and Endocrinology. 2013; [4] Barnes DE, Beiser AS, Lee A, Langa KM, Koyama A, Preis SR, et al. Development and validation of a brief dementia screening indicator for primary care. Alzheimers Dementia. 2014:S1552-5260. Notes * No development dataset. Rather, model tested in different cohorts.

https://doi.org/10.1371/journal.pone.0136181.g002

Cost Considerations

One study considered the impact on discriminative accuracy of modifying the calculation of a resource intensive risk score to incorporate less expensive measures. The original resource intensive model, the Late Life Dementia Risk Index, included demographic (age), lifestyle (alcohol consumption), neuropsychological (Modified Mini Mental State Examination (MMSE) score and Digit Symbol Substitution Score), medical (history of coronary bypass surgery and BMI), physical functioning (time to put on and button a shirt in seconds), genetic (APOE), cerebral magnetic resonance imaging (MRI) (white matter disease and enlarged ventricles), and carotid artery ultrasound (internal carotid artery thickness >2.2mm) measures, and had good discrimination for the prediction of 6-year incident dementia (c-statistic = 0.81, 95%CI: 0.79 to 0.83)[32]. The revised model, the Brief Dementia Risk Index, incorporated age, neuropsychological testing (3 word delayed recall, interlocking pentagon copying, verbal instructions (paper taking and folding), four legged animal naming task (30 seconds)), self-reported attention difficulties (3 or more days per week in the last month), medical history (stroke, peripheral artery disease, or coronary artery bypass surgery and BMI) and alcohol consumption and had a significantly lower discriminative accuracy (c-statistic = 0.77, p<0.001), but was able to categorize subjects as having low, moderate, or high risk of dementia with similar accuracy compared to the more resource intensive score[21].

Strengths and Limitations

The strengths of this review are its systematic approach and inclusivity. There are some limitations. It is difficult to synthesise the literature on dementia risk prediction due to the large variability across studies in follow-up length (range: 1.5 years[9] to 17 years[15]), sample age (range: 40 to 99 years), outcomes tested (e.g., AD vs. all-cause dementia vs. dementia subtypes, quality of the diagnosis), source of population (volunteer vs. population representative) and the different variables incorporated into the prediction models. As such a meta-analysis was not possible. Furthermore, any meaningful conclusions for population screening are limited by the lack of cost-effective analysis and limited assessment of model transportability.

Clinical Implications

There is currently a clinical drive towards timelier diagnosis particularly in developed countries such as the UK with the introduction of primary care direct enhanced services looking to identify those at risk of developing dementia e.g. stroke, diabetes, cardiovascular disease[33]. A risk prediction tool, particularly in at risk populations such as diabetes[13] could further enhance existing services. However, if model development in the field of dementia continues at its current pace and if dementia risk prediction is found to be useful and cost-effective, then researchers and possibly clinicians will face difficult choices regarding which model to apply, particularly as study comparison is difficult.

Stratified Analyses

Results from stratified analysis suggest that unique dementia risk prediction models may need to be developed depending on follow-up length (e.g., BMI and hypertension may be more important in mid-life compared to later life models)[7, 23, 26, 36–38], an individual’s education level (found in one study and requires replication) [7], health status (e.g., diabetes)[13], APOE status[23] and the outcome tested (e.g., most studies focus on all-cause dementia and different models maybe needed depending on dementia subtype)[19]. Further research is required to develop risk models in samples stratified by other confounding factors known to influence the timing and presentation of dementia symptoms (such as mid vs. later life) as well as investigate the interactions between different risk variables (e.g., such as AOPE status and age).

Model Transportability

Before a risk prediction tool can be used in clinical practice or for research, transportability of the model outside the cohort from which it was developed needs to be assessed. Only four studies have externally validated dementia risk prediction models and the results were mixed[4, 13, 14, 20]. The DSDRS model was developed and validated with moderate but similar levels of discrimination (AUC 0.74 development vs. 0.75 validation)[13]. This is in comparison to the CAIDE score (originally developed for midlife), which was poorly validated in three separate (older) cohorts (AUC 0.77 development vs. AUC range 0.49–0.57 validation)[4]. However, it is important to note that in this validation of the CAIDE score rather different populations were used, that varied by age (mid vs. later life). Indeed, factors like higher BMI, blood pressure and cholesterol levels are associated with lower incidence of dementia in the oldest old [31]. In contrast, the CAIDE score was found to transport well when the test population better resembled the original population (i.e., was a mid-life cohort)[14]. Generally, external validation is difficult largely due to the lack of available cohorts with which to test models in terms of follow-up times, data collected, age groups and risk variable measurement.

Most studies have been developed in datasets from North America (N = 12 studies), with others developed in the UK (N = 1), Japan (N = 1), Austria (N = 1), the Netherlands (N = 2), Pan-Europe (N = 1), Germany (N = 3) and France (N = 1). Whether the different models are applicable across countries that vary by health and wealth is not known. Further, no models have been developed for predicting other dementia sub-types, such as vascular dementia or dementia with Lewy Bodies or for predicting different disease severity (e.g., mild, moderate and severe dementia). This may have implications for treatment options.

Issues Around Cost

The ability to assess the relative cost of calculating the different risk models and compare this against the model’s accuracy will significantly influence recommendations about possible protocols for screening for dementia risk. Further, the incorporation of readily accessible primary care related factors would be most useful for application within clinical and population based settings. Three studies [4, 20, 21] have considered time and/or financial implications in risk score calculation. However, in one study it was found that reducing the cost of risk score computation by removing the need for MRI, ECG and detailed neuropsychological measures and replacing them with less resource intensive variables (e.g., more detailed self-reported health history and simple cognitive test items) resulted in a significant decline in discriminative accuracy[21, 32]. Thus raising the issue of what is the best information and minimum data set needed for accurate dementia risk prediction. It is important to note that current models have not yet utilized cerebrospinal fluid or positron emission tomography data as recommended for classifying AD and its preclinical stages in new (clinical and research) criteria for AD and its preclinical/prodromal states[39, 40]. Although these factors can be used to assist dementia diagnosis, the feasibility and acceptance of incorporating them into risk prediction models and, particularly in population-based settings, would likely be low.