Skip to main content
Advertisement
Browse Subject Areas
?

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here.

  • Loading metrics

Meta-Analysis of Longitudinal Cohort Studies of Suicide Risk Assessment among Psychiatric Patients: Heterogeneity in Results and Lack of Improvement over Time

  • Matthew Large ,

    mmbl@bigpond.com

    Affiliations School of Psychiatry, University of New South Wales, Randwick, Australia, Prince of Wales Hospital, Randwick, Australia

  • Muthusamy Kaneson,

    Affiliation Faculty of Medicine, University of New South Wales, Randwick, Australia

  • Nicholas Myles,

    Affiliation The Queen Elizabeth Hospital, Woodville South, Australia

  • Hannah Myles,

    Affiliations Discipline of Psychiatry, School of Medicine, The University of Adelaide, SA, Australia, Country Health SA Mental Health, SA, Australia

  • Pramudie Gunaratne,

    Affiliation Prince of Wales Hospital, Randwick, Australia

  • Christopher Ryan

    Affiliations Discipline of Psychiatry and Centre for Values Ethics and the Law in Medicine, University of Sydney, Sydney, Australia, Department of Psychiatry, Westmead Hospital, Westmead, NSW, Australia

Abstract

Objective

It is widely assumed that the clinical care of psychiatric patients can be guided by estimates of suicide risk and by using patient characteristics to define a group of high-risk patients. However, the statistical strength and reliability of suicide risk categorization is unknown. Our objective was to investigate the odds of suicide in high-risk compared to lower-risk categories and the suicide rates in high-risk and lower-risk groups.

Method

We located longitudinal cohort studies where psychiatric patients or people who had made suicide attempts were stratified into high-risk and lower-risk groups for suicide with suicide mortality as the outcome by searching for peer reviewed publications indexed in PubMed or PsychINFO. Electronic searches were supplemented by hand searching of included studies and relevant review articles. Two authors independently extracted data regarding effect size, study population and study design from 53 samples of risk-assessed patients reported in 37 studies.

Results

The pooled odds of suicide among high-risk patients compared to lower-risk patients calculated by random effects meta-analysis was of 4.84 (95% Confidence Interval (CI) 3.79–6.20). Between-study heterogeneity was very high (I2 = 93.3). There was no evidence that more recent studies had greater statistical strength than older studies. Over an average follow up period of 63 months the proportion of suicides among the high-risk patients was 5.5% and was 0.9% among lower-risk patients. The meta-analytically derived sensitivity and specificity of a high-risk categorization were 56% and 79% respectively. There was evidence of publication bias in favour of studies that inflated the pooled odds of suicide in high-risk patients.

Conclusions

The strength of suicide risk categorizations based on the presence of multiple risk factors does not greatly exceed the association between individual suicide risk factors and suicide. A statistically strong and reliable method to usefully distinguish patients with a high-risk of suicide remains elusive.

Introduction

It is widely assumed that patients presenting to psychiatric services should routinely undergo a suicide risk assessment in order to allow the identification of high-risk patients who warrant closer monitoring and who should be afforded more clinical resources [14]. However, some authors doubt that clinically meaningful suicide risk categories can be defined by either suicidal thoughts or behaviors [5,6] or a combination of multiple risk factors [710]. Complicating this debate is a lack of knowledge about the statistical strength of suicide risk categorization, the extent to which this statistical strength varies between studies or whether there has been genuine progress in this area of research over time.

Numerous longitudinal cohort studies published in the last four decades have defined suicide risk categories or strata by combining various clinical and socio-demographic risk factors. These studies are of two types. The first type, which we will term ‘exploratory’ studies, combine potential risk factors observed at baseline to develop a post-hoc risk model based on eventual suicide at follow-up [7,11,12]. Exploratory studies can consider large numbers of potential risk factors and employ statistical methods such as multiple logistic regression and survival analysis to determine variables that are independently associated with suicide. As a consequence of examining a large number of variables some statistical associations with suicide will arise purely by chance. If these chance associations are incorporated into multivariate high-risk models generated by these studies, the apparent strength of the models can be artificially inflated [8,13,14]. The second type of study, which we will term ‘validation’ studies, determine risk categories through the use of scales that are either previously published or that combine a defined set of variables identified in previous exploratory studies [1517]. Validation studies typically examine a more limited set of variables than exploratory studies but are less prone to chance findings.

Aims and hypotheses

We performed a meta-analysis incorporating both exploratory and validation longitudinal cohort studies. Our primary aim was to calculate overall strength of the effect size of suicide risk assessment using a pooled estimate of the odds of suicide in high-risk groups compared to lower-risk groups. We hypothesized that the effect size associated with suicide risk assessment would i) be reliable between studies and resulting in low between study heterogeneity and ii) have improved over time with stronger in results in more recent studies.

Secondary aims were to explore potential moderators of between-study heterogeneity in the primary research according to the methods employed, the type of patients included, the overall strength of reporting, the base rate of suicide, the year of publication, the duration of follow up and the number of independent variables that were examined. Finally we aimed to examine the performance of high-risk models by calculating the proportions of suicides in high-risk groups (positive predicative value) and lower-risk groups, and to calculate the aggregate sensitivity and specificity of risk categorization.

Methods

We meta-analyzed published longitudinal cohort studies that examined multiple patient factors in order to define a stratum of psychiatric patients at high risk of suicide. ‘Psychiatric patients’ here refers to persons, who received inpatient or outpatient psychiatric treatment, or persons who were assessed after a suicide attempt or an act of deliberate self-harm. Mental health professionals commonly assess both of these patient groups and the suicide rate in both groups is comparatively higher than in the general population [18,19].

We chose to examine longitudinal cohort studies. We did not consider case-control studies because of their potential for bias in data collection of some variables, because of the potential bias associated with retrospective variable selection following suicide outcomes, and because these studies do not allow direct calculation of suicide rates according to risk category. Both exploratory and validation studies were included to provide a complete analysis of suicide risk categorization and allow statistical comparison of study type. When a study reported both exploratory and validation approaches both were included in the meta-analysis.

Our methods conformed to the items in the quality checklist from the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) and Meta-analysis Of Observational Studies in Epidemiology (MOOSE) guidelines [20,21]. PRISMA checklist, see S1 PRISMA Checklist.

Search strategy

Extensive preliminary literature searches using broad subject headings proved insufficiently sensitive to identify several studies known to the authors. Consequently, we conducted less specific searches using the term ‘suicide’ or ‘suicides’ in the title. Two such searches were independently conducted of PubMed and PsycINFO. The searches were conducted in English. Studies were assessed by the inclusion and exclusion criteria below and winnowed first by examination of title, then abstract, then full text,see Fig 1. The reference lists of all the included studies were then hand searched first by examination of relevant titles, abstracts and full text publications without language selection. Finally, we searched the reference list of relevant clinical guidelines,[22,23] books,[24,25] and review articles [6,8,13,2632].

thumbnail
Fig 1. Flow chart of searches for cohort studies reporting multivariate models of later suicide.

https://doi.org/10.1371/journal.pone.0156322.g001

Study selection

Inclusion criteria.

We included studies that: i) reported on longitudinal cohorts of psychiatric patients; ii) measured patient factors at baseline assessment (either in a validation study of a suicide risk scale or in an exploratory study of multiple variables); iii) reported subsequent deaths by suicide as the dependent variable and; iv) used two or more variables (other than age and sex) to define a high suicide risk group.

Exclusion criteria.

We excluded studies that: were retrospective case controlled studies; reported on suicide attempts as the dependent variable; examined general populations rather than patient groups; did not describe a high-risk group or; described a high-risk group on the basis of a single characteristic or solely demographic characteristics. We also excluded studies that examined potential biological markers because of our focus on clinical practice.

Data extraction

Two authors (ML and HM) independently extracted the data. The preferred data format was the number of suicides within high-risk and lower-risk groups, and total number within each group. This was imputed from reported sensitivities and specificities in some circumstances. Most studies dichotomized the patients into high and lower-risk groups. Where multiple cut-off points were reported, the data with the highest proportion of suicides in the high-risk group was used in the meta-analysis. Where data reporting suicides and total numbers in risk groups was not available we extracted other effect size data including odds ratios or chi-square statistics. All effect size data were converted to odds ratios.

Moderator variables were collected for each sample comprising:

  1. Whether the study was a validation or an exploratory study, because of the possibility of chance findings inflating the statistical power with the latter methodology.
  2. Whether the cohort consisted exclusively of people who had made a suicide attempt or subjects recruited from general psychiatric settings.
  3. The year the study was published, because suicide risk categorization might have improved over time.
  4. The number of potential suicide risk variables initially examined because studies including a larger number of variables are more prone to chance associations.
  5. The number of variables in the high-risk model because more detailed risk categories might be more accurate.
  6. The mean length of follow up (in months), because studies with longer follow up are less likely to misclassify eventual suicide.
  7. The base rate of suicide to allow calculation of sensitivity, specificity and positive predictive value (PPV).
  8. Whether coronial or mortality database data were used to define the outcome of suicide, because this methodology has more accurate case ascertainment.
  9. Whether subjects were recruited from a geographically defined catchment area, because such studies are less prone to bias.
  10. Whether the studies examined suicides of current psychiatric inpatients, because, these studies have a short follow up period restricted to the length of stay in hospital, because inpatient psychiatric care might mitigate suicide risk and because these studies used the number of admissions rather than the number of patients as the denominator.

Data Synthesis

Random-effects meta-analysis was used to calculate pooled estimates of the odds ratio for suicide among those who were assessed as being at high-risk versus lower-risk using Comprehensive Meta-Analysis (CMA; Version 3, Biostat, Englewood NJ). A random-effects model was chosen a priori for all analyses because of the differences in study populations and definitions of high-risk strata. Odds ratios were used as the measure of effect size. Between-study heterogeneity in effect size was examined using the I2 and with Q-value statistics. Between-group heterogeneity (sensitivity analysis) was examined without assuming a common within-study variance and the significance of between-group heterogeneity was determined with Q-value statistics.

Random-effects meta-regression (method of moments) was used to examine whether the year of publication, the length of follow up, the base rate of suicide, the number of variables initially considered, and the number of variables in the high-risk model, were associated with between-study heterogeneity. The sensitivity and specificity of the risk categorization and the proportion of suicides in the high-risk and lower-risk groups were also calculated using random-effects meta-analysis.

Assessment of reporting strength

Six moderator variables derived from relevant items of the Newcastle-Ottawa scale for the assessment of reporting strength of non-randomized cohort studies in meta-analyses were collected to use as items in a strength of reporting scale [33]. These variables included whether: the study was a validation study; the study was drawn from a defined catchment area; the suicides were ascertained using mortality databases or coronial findings; the study did not exclusively report inpatient suicides; the study had a length of follow up that was greater than the median length of follow up; the study reported more suicides than the median number of suicides reported. Any study that featured one of these characteristics was awarded one point, allowing each study to be awarded a maximum of six points.

Validation studies were regarded as higher quality than exploratory studies because of the reduced possibility of variables being included in high-risk models by chance [8]. Studies that reported mortality data or coronial records were regarded as having stronger methodology because of more accurate case ascertainment [34]. Studies with a longer period of follow up were regarded as having stronger methodology due to the decreased likelihood of misclassifying survivors who may eventually suicide. Studies reporting on fewer suicides were regarded as being lower quality because of the increased possibility of chance findings. Studies that did not exclusively examine suicides by psychiatric inpatients were considered to have stronger methodology, because inpatient psychiatric care might mitigate suicide risk and because these studies used the number of admissions rather than the number of patients as the denominator.

Assessment of publication bias

Publication bias was assessed using an Eggers regression test and with Duval and Tweedie’s trim and fill method [35].

Sensitivity analyses

Three sensitivity analyses were performed to compare: i) validation studies to exploratory studies; ii) studies that examined patients who presented with deliberate self harm and/or suicide attempts to studies of other psychiatric patients; iii) studies with a higher versus lower total strength of reporting score dichotomized by the median score.

Results

Searches

The searches identified 37 relevant studies, see Table 1, reporting 53 samples of patients who were categorized by suicide risk assessment, see S1 File. There was one disagreement about the selection of one included study that was resolved by consensus. There were no disagreements with regard to independently extracted effect size data. Disagreements about 15 (2.2%) of 689 data points in relation to study methods or other moderator variables were resolved by further examination and consensus. In no case was it deemed necessary to contact the authors of the primary research for further clarification of their data.

The included papers examined 315,309 people (mean per study 8522, standard deviation (SD) = 22,812, median = 1052) of whom 3114 died by suicide (mean per study 84.2, standard deviation (SD) = 206, median = 27). Eighteen studies recruited patients in psychiatric treatment settings and 19 studies were of patients who had presented after suicide attempts and/or episodes of self-harm. Of the 53 tests of suicide risk categorization 24 were validation studies and 29 were exploratory studies. Three papers reported both validation and exploratory methods [7,18,36]. Four studies were of the suicide of current psychiatric inpatients [12,3739]. The mean length of study follow up was 64 months (SD = 50, median = 62). Exploratory risk assessments examined an average of 66.8 (SD = 86.9) variables of which 9.2 (SD = 13.6) were included in high-risk models. Validation studies examined 21.9 (SD = 12.7) variables and reported suicide risk scales with an average of 13.9 (SD = 6.4) items. The methods in the studies varied considerably. The Suicidal Intent Scale [40] was most frequently used in validation studies (ten samples). In the 29 exploratory models, a prior history of a suicide attempt was the single most commonly included variable (21 high-risk models), followed by more psychiatric treatment (15 models), a depressed mood or an affective disorder (15 models), and substance use (seven models).

Meta-analysis

The pooled odds of suicide in high risk groups compared to the lower-risk was 4.84 (Table 2 & Fig 2), equivalent to a standardized mean difference 0.87 and indicating a strong effect size) [41]. There was very high between-study heterogeneity (I2 = 93.3, Q-value 773, p<0.001). The lowest effect size was an odds ratio of 1.023, the first quartile was an odds ratio of 2.43, the median odds were 4.92, the third quartile was 12.90 and the highest odds ratio was 37.27.

thumbnail
Table 2. Meta-analysis of the odds of suicide in high-risk strata compared to other patients.

https://doi.org/10.1371/journal.pone.0156322.t002

thumbnail
Fig 2. Forrest plot of cohort studies of the odds of suicide in high-risk and lower-risk patients.

Studies listed in order of publication. Summary statistic and 95% confidence intervals represented by the diamond. Abbreviations: BHS = Beck Hopelessness Score, SSI = Scale of Suicidal Ideation, SUAS = Suicide assessment scale, BDI = Beck Depression Inventory, SIS = Suicide Intent Scale, SIS-W = Suicide Intent Scale at worst point, SIS-C = Suicide Intent Scale current, SIS-S = Suicide Intent Scale, Short, SIS-L = Suicide Intent Scale, long, SIS-M = Suicide Intent Scale, modified, CHS = Clinicians Hopelessness Scale, KIVS = Karolinska interpersonal violence scale, ReACT = ReACT self harm rule.

https://doi.org/10.1371/journal.pone.0156322.g002

The 29 samples from studies published before 2000 (odds ratio 4.9, 95% CI 3.7–6.6, I-square = 81) did not from the 24 studies published during or after 2000 (odds ratio 4.6, 95% CI 3.3–5.9, I-square = 93) in either the strength of the effect size or in the extent of between study heterogeneity). Meta regression found that publication date was not significantly associated with effect size of the 53 samples.

There was evidence of publication bias in favor of studies reporting a stronger association between high-risk strata and suicide using Egger’s test (intercept = 3.56, t-value 8.64, two tailed p = 0.001). Duval and Tweedie’s trim and fill method identified 15 hypothetically missing studies with a weaker association between high-risk status and suicide which, if included, would have returned a lower adjusted odds ratio of 3.18 (95% CI 2.55 to 3.99).

The pooled sensitivity of a high-risk categorization was 56% (N = 39 studies, 95% CI 48–64%, I2 = 87.5) indicating that just over half of the suicides occurred in the high-risk groups. The pooled specificity of a lower-risk categorization was 79% (95% CI 70–86%, I-Square = 99.9) indicating that four in five of the survivors were in the low risk group. The pooled estimate for the crude suicide rate among high-risk patients (or positive predictive value) was 5.5% (n = 39, 95% CI 3.5–8.5%, I2 = 97.4). The pooled rate of suicide in the lower-risk patients was 0.9% (N = 39 samples 95% CI 0.5–1.7%, I2 = 98.8).

Sensitivity analysis and Meta regression.

Validation studies and exploratory studies reported similar pooled effect sizes, both with very high between-study heterogeneity (exploratory, I2 = 95.5, Q-value 618, p<0.001; validations, I2 = 85.1, Q-value 154, p<0.001). Studies of general psychiatric patients and studies of patients who were recruited after a suicide attempt or an episode of deliberate self-harm had a similar effect size. Studies with a total strength of reporting score of four or more had a similar effect size to studies with a lower strength of reporting score, see Table 2.

The between-study heterogeneity in odds ratios was not explained by the year in which the study was published, the base rate of suicide, the length of follow up, or the number of variables in the high-risk model, see Table 3. A larger number of variables examined at baseline was associated with a stronger effect size but accounted for little of the observed between-study heterogeneity (unexplained variance, I2 = 93%).

thumbnail
Table 3. Meta-regression examining factors associated with between study heterogeneity in the odds of suicide in high-risk strata.

https://doi.org/10.1371/journal.pone.0156322.t003

Discussion

The pooled estimate from a large and representative body of research conducted over 40 years suggests a statistically strong association between high-risk strata and completed suicide. However the meta-analysis of the sensitivity of suicide risk categorization found that about half of all suicides are likely to occur in lower-risk groups and the meta-analysis of PPV suggests that 95% of high-risk patients will not suicide. Importantly, the pooled odds ratio (and the estimates of the sensitivity and PPV) and any assessment of the overall strength of risk assessment should be interpreted very cautiously in the context of several limitations documented below.

With respect to our first hypothesis, the statistical estimates of between study heterogeneity and the distribution of the outlying, quartile and median effect sizes values suggests that the statistical strength of suicide risk assessment cannot be considered to be consistent between studies, potentially limiting the generalizability of the pooled estimate.

With respect to our second hypothesis we found no evidence that the statistical strength of suicide risk assessment has improved over time.

Limitations to the generalizability of the pooled estimate

The most important limitation to our pooled estimate is the very high between-study heterogeneity of the effect size. This between-study heterogeneity was not well explained by our predetermined moderator variables or measures of reporting strength. This suggests that our results should not be considered to be generalizable. Moreover, the pooled estimate was potentially influenced by evidence of publication bias towards selective reporting of studies with a stronger effect size. While we cannot know for certain whether this bias is present, or to what extent it might occur, it may be that the results reported here are better than can be readily achieved.

A further limitation, potentially inflating the pooled estimate of the effect size, is that studies that initially examined a larger number of variables tended to have a stronger effect size than studies that examined fewer initial variables. This may be because a more detailed assessment of patient factors might have resulted in a model better able to categorize patient’s suicide risk, or because of the inclusion of more variables with chance associations. However, neither of these explanations seems likely because there was no evidence that studies that used more variables to define the high-risk model had a stronger effect size. In fact, the pooled effect size in this study of risk categories based on multiple variables is similar to the meta-analytically derived effect size of individual factors of self-harm, depressed mood and hopelessness among patient groups [13,29].

Limitations to the meta-analysis

A number of limitations inherent to both this meta-analysis and the primary literature also warrant discussion. A weakness of this meta-analysis is the inclusion of studies conducted over more than 40 years, during which the studies differed greatly in their sample sizes, methods and in their reporting strength.

Our focus on high-risk patients can also be considered a limitation. The included literature does not allow determination of whether a clinically meaningful low-risk group can be defined based on protective factors. It might be that the high prevalence of non-suicide could allow the identification of a group of patients with low suicide rates that is similar to the general community. Moreover, because our focus was on patients, we did not examine the potential strength of risk categorization for suicide among the general community.

A weakness of the primary literature is that all included studies were naturalistic. As a result, this meta-analysis was unable to consider the impact of any interventions that might have been provided to people who were perceived as being at high-risk of suicide. Successful interventions provided to high-risk patients in the primary studies may have the effect of reducing the odds of suicide in that group. The extent of this effect cannot be estimated without studies that directly investigate the effectiveness of providing increased resource allocation or enhanced clinical surveillance to high-risk patients.

Finally, the meta-analysis does not address the statistical power of imminent suicide risk assessment nor did it examine clinical risk assessment. We found no studies of suicide outcomes over periods of less than a month and no study examined the type of heuristic assessment of suicide risk that is common in clinical practice or the relative performance of this to codified risk assessment tools.

Disappointingly, there was no evidence that heterogeneity in effect size could be explained by the year of study publication. This suggests that generally there might have been little progress over time in the ability of published models to identify high-risk groups of patients. However, the lack of evidence for the development of more accurate risk assessment models over time does not mean that such developments are impossible. More sophisticated or effective methods of suicide risk categorization might be developed in the future. For example, one recent study examining post discharge suicide in the US military was able to define a high-risk group with an odds ratio of 22 when compared to lower-risk patients [11]. This study drew on an extensive data set using sophisticated methods of modeling derived from artificial intelligence research, methods that might be able to more strongly and more reliably define high-risk groups in the future.

Conclusion

Despite decades of research, the psychometric properties of optimal suicide risk categorization remains uncertain. The extent of this uncertainty is profound and our results are not reassuring. It remains to be seen if methods can be developed to consistently and clearly distinguish high-risk from lower-risk patients. However, it should not be forgotten that the ultimate utility of risk categorization depends on its potential for application. Even a strong statistical discrimination between high and lower-risk groups lacks meaning if there are no rational interventions that should be provided to high risk patients (the vast majority of whom will not suicide) yet should not be given to low risk patients, among whom about half of all suicides might occur. Moreover, ultimately the value of suicide risk categorization must be judged by whether it can actually contribute to a reduction in patient suicide mortality.

Author Contributions

Conceived and designed the experiments: ML CR. Performed the experiments: ML MK NM HM PG. Analyzed the data: ML MK. Contributed reagents/materials/analysis tools: ML. Wrote the paper: ML MK NM HM PG CR.

References

  1. 1. Chu C, Klein KM, Buchman-Schmitt JM, Hom MA, Hagan CR, et al. (2015) Routinized Assessment of Suicide Risk in Clinical Practice: An Empirically Informed Update. J Clin Psychol 71:1186–2000. pmid:26287362
  2. 2. Berman AL, Silverman MM (2014) Suicide risk assessment and risk formulation part II: Suicide risk formulation and the determination of levels of risk. Suicide Life Threat Behav 44: 432–443. pmid:24286521
  3. 3. Boudreaux ED, Horowitz LM (2014) Suicide risk screening and assessment: designing instruments with dissemination in mind. Am J Prev Med 47: S163–169. pmid:25145734
  4. 4. Motto JA, Bostrom A (1990) Empirical indicators of near-term suicide risk. Crisis 11: 52–59. pmid:2376147
  5. 5. Ribeiro JD, Franklin JC, Fox KR, Bentley KH, Kleiman EM, et al. (2015) Self-injurious thoughts and behaviors as risk factors for future suicide ideation, attempts, and death: a meta-analysis of longitudinal studies. Psychol Med: 1–12.
  6. 6. Chapman CL, Mullin K, Ryan CJ, Kuffel A, Nielssen O, et al. (2015) Meta-analysis of the association between suicidal ideation and later suicide among patients with either a schizophrenia spectrum psychosis or a mood disorder. Acta Psychiatr Scand 131: 162–173. pmid:25358861
  7. 7. Pokorny AD (1983) Prediction of suicide in psychiatric patients. Report of a prospective study. Arch Gen Psychiatry 40: 249–257. pmid:6830404
  8. 8. Cohen J (1986) Statistical approaches to suicidal risk factor analysis. Ann N Y Acad Sci 487: 34–41. pmid:3471164
  9. 9. Clark DC, Young MA, Scheftner WA, Fawcett J, Fogg L (1987) A field test of Motto's risk estimator for suicide. Am J Psychiatry 144: 923–926. pmid:3605405
  10. 10. Goldstein RB, Black DW, Nasrallah A, Winokur G (1991) The prediction of suicide. Sensitivity, specificity, and predictive value of a multivariate model applied to suicide among 1906 patients with affective disorders. Arch Gen Psychiatry 48: 418–422. pmid:2021294
  11. 11. Kessler RC, Warner CH, Ivany C, Petukhova MV, Rose S, et al. (2015) Predicting Suicides After Psychiatric Hospitalization in US Army Soldiers: The Army Study to Assess Risk and Resilience in Servicemembers (Army STARRS). JAMA Psychiatry 72: 49–57. pmid:25390793
  12. 12. Krupinski M, Fischer A, Grohmann R, Engel R, Hollweg M, et al. (1998) Risk factors for suicides of inpatients with depressive psychoses. Eur Arch Psychiatry Clin Neurosci 248: 141–147. pmid:9728733
  13. 13. Large M, Smith G, Sharma S, Nielssen O, Singh SP (2011) Systematic review and meta-analysis of the clinical factors associated with the suicide of psychiatric in-patients. Acta Psychiatr Scand 124: 18–29. pmid:21261599
  14. 14. McCarthy JF, Bossarte RM, Katz IR, Thompson C, Kemp J, et al. (2015) Predictive Modeling and Concentration of the Risk of Suicide: Implications for Preventive Interventions in the US Department of Veterans Affairs. Am J Public Health 105: 1935–1942. pmid:26066914
  15. 15. Beck AT, Brown G, Steer RA (1989) Prediction of eventual suicide in psychiatric inpatients by clinical ratings of hopelessness. J Consult Clin Psychol 57: 309–310. pmid:2708621
  16. 16. Brown GK, Beck AT, Steer RA, Grisham JR (2000) Risk factors for suicide in psychiatric outpatients: a 20-year prospective study. J Consult Clin Psychol 68: 371–377. pmid:10883553
  17. 17. Stefansson J, Nordstrom P, Jokinen J (2012) Suicide Intent Scale in the prediction of suicide. J Affect Disord 136: 167–171. pmid:21144592
  18. 18. Steeg S, Kapur N, Webb R, Applegate E, Stewart SL, et al. (2012) The development of a population-level clinical screening tool for self-harm repetition and suicide: the ReACT Self-Harm Rule. Psychol Med 42: 2383–2394. pmid:22394511
  19. 19. Hjorthoj CR, Madsen T, Agerbo E, Nordentoft M (2014) Risk of suicide according to level of psychiatric treatment: a nationwide nested case-control study. Soc Psychiatry Psychiatr Epidemiol 49: 1357–1365. pmid:24647741
  20. 20. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gotzsche PC, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health care interventions: explanation and elaboration. J Clin Epidemiol 62: e1–34.
  21. 21. Stroup DF, Berlin JA, Morton SC, Olkin I, Williamson GD, et al. (2000) Meta-analysis of observational studies in epidemiology: a proposal for reporting. Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group. JAMA 283: 2008–2012. pmid:10789670
  22. 22. National Institute for Health and Care Excellence (2012) Self-harm; longer term managemnt The British Psychological Society & The Royal College of Psychiatrists,.
  23. 23. Department of Health (2010) Suicide risk assessment and management; A systematic evidence review for the Clinicalpractice guidelines for emergency departments and mental health services project. http://www.health.vic.gov.au/mentalhealth/: Published by the Mental Health, Drugs and Regions Division, Victorian Government,
  24. 24. Haney EM, O'Neil ME, Carson S, Low A, Peterson K, et al. (2012). Suicide Risk Factors and Risk Assessment Tools: A Systematic Review. Washington (DC).
  25. 25. Beck AT, Resnick HLP, Lettieri DJ (1974) The prediction of suicide Bowie, Maryland: The Charles Press Publishers Inc.
  26. 26. Mann JJ, Apter A, Bertolote J, Beautrais A, Currier D, et al. (2005) Suicide prevention strategies: a systematic review. JAMA 294: 2064–2074. pmid:16249421
  27. 27. Gaynes BN, West SL, Ford CA, Frame P, Klein J, et al. (2004) Screening for suicide risk in adults: a summary of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med 140: 822–835. pmid:15148072
  28. 28. McMillan D, Gilbody S, Beresford E, Neilly L (2007) Can we predict suicide and non-fatal self-harm with the Beck Hopelessness Scale? A meta-analysis. Psychol Med 37: 769–778. pmid:17202001
  29. 29. Large M, Sharma S, Cannon E, Ryan C, Nielssen O (2011) Risk factors for suicide within a year of discharge from psychiatric hospital: a systematic meta-analysis. Aust N Z J Psychiatry 45: 619–628. pmid:21740345
  30. 30. Hawton K, Sutton L, Haw C, Sinclair J, Deeks JJ (2005) Schizophrenia and suicide: systematic review of risk factors. Br J Psychiatry 187: 9–20. pmid:15994566
  31. 31. Hawton K, Sutton L, Haw C, Sinclair J, Harriss L (2005) Suicide and attempted suicide in bipolar disorder: a systematic review of risk factors. J Clin Psychiatry 66: 693–704. pmid:15960561
  32. 32. Hawton K, Casanas ICC, Haw C, Saunders K (2013) Risk factors for suicide in individuals with depression: a systematic review. J Affect Disord 147: 17–28. pmid:23411024
  33. 33. Wells GA, Shea B, O'Connnell D, Peterson L, Welsh V, et al. (2013) The Newcastle-Ottawa Scale (NOS) for assessing the quality of nonrandomised studies in meta-analyses. Accessed from http://wwwohrica/programs/clinical_epidemiology/oxfordasp.
  34. 34. Walsh G, Sara G, Ryan CJ, Large M (2015) Meta-analysis of suicide rates among psychiatric in-patients. Acta Psychiatr Scand131:174–84 pmid:25559375
  35. 35. Duval S, Tweedie R (2000) Trim and fill: A simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56: 455–463. pmid:10877304
  36. 36. Buglass D, Horton J (1974) A scale for predicting subsequent suicidal behaviour. Br J Psychiatry 124: 573–578. pmid:4850288
  37. 37. Madsen T, Agerbo E, Mortensen PB, Nordentoft M (2012) Predictors of psychiatric inpatient suicide: a national prospective register-based study. J Clin Psychiatry 73: 144–151. pmid:21903026
  38. 38. Krupinski M, Fischer A, Grohmann R, Engel RR, Hollweg M, et al. (2000) [Schizophrenic psychoses and suicide in the clinic. Risk factors, psychopharmacologic treatment]. Nervenarzt 71: 906–911. pmid:11103366
  39. 39. Neuner T, Schmid R, Wolfersdorf M, Spiessl H (2008) Predicting inpatient suicides and suicide attempts by using clinical routine data? Gen Hosp Psychiatry 30: 324–330. pmid:18585535
  40. 40. Beck RW, Morris JB, Beck AT (1974) Cross-validation of the Suicidal Intent Scale. Psychol Rep 34: 445–446. pmid:4820501
  41. 41. Rosenthal J (1996) Qualitative descriptors of strength of association and effect size. J Soc Serv Res 21: 37–57.
  42. 42. Buglass D, McCulloch JW (1970) Further suicidal behaviour: the development and validation of predictive scales. Br J Psychiatry 116: 483–491. pmid:5460558
  43. 43. Rosen DH (1976) The serious suicide attempt. Five-year follow-up study of 886 patients. JAMA 235: 2105–2109. pmid:946536
  44. 44. Pokorny AD (1993) Suicide prediction revisited. Suicide Life Threat Behav 23: 1–10. pmid:8475527
  45. 45. Pallis DJ, Gibbons JS, Pierce DW (1984) Estimating suicide risk among attempted suicides. II. Efficiency of predictive scales after the attempt. Br J Psychiatry 144: 139–148. pmid:6704599
  46. 46. Beck AT, Steer RA, Kovacs M, Garrison B (1985) Hopelessness and eventual suicide: a 10-year prospective study of patients hospitalized with suicidal ideation. Am J Psychiatry 142: 559–563. pmid:3985195
  47. 47. Motto JA, Heilbron DC, Juster RP (1985) Development of a clinical instrument to estimate suicide risk. Am J Psychiatry 142: 680–686. pmid:4003585
  48. 48. Beck AT, Steer RA (1989) Clinical predictors of eventual suicide: a 5- to 10-year prospective study of suicide attempters. J Affect Disord 17: 203–209. pmid:2529288
  49. 49. Allgulander C, Fisher LD (1990) Clinical predictors of completed suicide and repeated self-poisoning in 8895 self-poisoning patients. Eur Arch Psychiatry Neurol Sci 239: 270–276. pmid:2138552
  50. 50. Nordentoft M, Breum L, Munck LK, Nordestgaard AG, Hunding A, et al. (1993) High mortality by natural and unnatural causes: a 10 year follow up study of patients admitted to a poisoning treatment centre after suicide attempts. BMJ 306: 1637–1641. pmid:8324430
  51. 51. Nordstrom P, Asberg M, Aberg-Wistedt A, Nordin C (1995) Attempted suicide predicts suicide risk in mood disorders. Acta Psychiatr Scand 92: 345–350. pmid:8619338
  52. 52. Nimeus A, Traskman-Bendz L, Alsen M (1997) Hopelessness and suicidal behavior. J Affect Disord 42: 137–144. pmid:9105955
  53. 53. Beck AT, Brown GK, Steer RA, Dahlsgaard KK, Grisham JR (1999) Suicide ideation at its worst point: a predictor of eventual suicide in psychiatric outpatients. Suicide Life Threat Behav 29: 1–9. pmid:10322616
  54. 54. Stephens JH, Richard P, McHugh PR (1999) Suicide in patients hospitalized for schizophrenia: 1913–1940. J Nerv Ment Dis 187: 10–14. pmid:9952248
  55. 55. Tejedor MC, Diaz A, Castillon JJ, Pericay JM (1999) Attempted suicide: repetition and survival—findings of a follow-up study. Acta Psychiatr Scand 100: 205–211. pmid:10493087
  56. 56. Nimeus A, Alsen M, Traskman-Bendz L (2000) The suicide assessment scale: an instrument assessing suicide risk of suicide attempters. Eur Psychiatry 15: 416–423. pmid:11112934
  57. 57. Schneider B, Philipp M, Muller MJ (2001) Psychopathological predictors of suicide in patients with major depression during a 5-year follow-up. Eur Psychiatry 16: 283–288. pmid:11514130
  58. 58. Niméus A, Alsen M, Träskman-Bendz L (2002) High Suicidal Intent Scores Indicate Future Suicide. Archives of Suicide Research 6: 211–219.
  59. 59. Skogman K, Alsen M, Ojehagen A (2004) Sex differences in risk factors for suicide after attempted suicide-a follow-up study of 1052 suicide attempters. Soc Psychiatry Psychiatr Epidemiol 39: 113–120. pmid:15052392
  60. 60. Suominen K, Isometsa E, Ostamo A, Lonnqvist J (2004) Level of suicidal intent predicts overall mortality and suicide after attempted suicide: a 12-year follow-up study. BMC Psychiatry 4: 11. pmid:15099401
  61. 61. Harriss L, Hawton K (2005) Suicidal intent in deliberate self-harm and the risk of suicide: the predictive power of the Suicide Intent Scale. J Affect Disord 86: 225–233. pmid:15935242
  62. 62. Loas G (2007) Anhedonia and suicide: a 6.5-yr. follow-up study of patients hospitalised for a suicide attempt. Psychol Rep 100: 183–190. pmid:17451023
  63. 63. Madsen T, Nordentoft M (2012) Risk factors for inpatient suicide do not translate into meaningful risk categories—all psychiatric inpatients are high-risk, author reply 1034–5. J Clin Psychiatry 73: 1034–1035. pmid:22901353
  64. 64. Rajalin M, Hirvikoski T, Jokinen J (2013) Family history of suicide and exposure to interpersonal violence in childhood predict suicide in male suicide attempters. J Affect Disord 148: 92–97. pmid:23273935
  65. 65. Runeson B, Haglund A, Lichtenstein P, Tidemalm D (2015) Suicide risk after nonfatal self-harm: a national cohort study, 2000–2008. J Clin Psychiatry.
  66. 66. Stefansson J, Nordstrom P, Runeson B, Asberg M, Jokinen J (2015) Combining the Suicide Intent Scale and the Karolinska Interpersonal Violence Scale in suicide risk assessments. BMC Psychiatry 15: 226. pmid:26400812