Review protocol

In order to ensure a transparent and comprehensive report of results, this study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [13]. Doing so ensured a thorough and accurate reporting of the methodology and results.

Systematic search strategy

To optimise the systematic literature search, 20 databases were searched across five different subject areas, namely law, medicine, psychology, sociology and politics, and included the following: Westlaw, Index to Legal Periodicals and Books, Index to Foreign Legal Periodicals, HeinOnline, Wiley Online Library, LexisNexis Library, MEDLINE, PubMed, PsychINFO, PsychARTICLES, PsychTESTS, Scopus, Applied Social Science Index and Abstract, International Bibliography of the Social Sciences, Web of Science, Race Relations Abstracts, Sociological Abstracts, International Political Science Abstracts, Worldwide Political Science Abstracts, and Declassified Documents Reference System. After having met with information specialists, the search strategy consisted of a combination of the following Boolean keywords: (extrem* OR radicali* OR terroris* OR salafi* OR islam* OR jihad* OR ideolog* OR “sleeper cell” OR “political violence” OR “suicide bombing”) AND (question* OR screen* OR trait* OR profil* OR assess* OR item* OR characteristic* OR “risk factor” OR scale OR criteria). Due to the extensive list of databases searched, and given the scope of the literature covered in these platforms, further individual searches were developed for each database as to strengthen the quality of the overall search. Comprehensive and detailed information was made available from the authors upon request. No language or time restrictions were applied, resulting mostly in English-language articles, but the search nevertheless included publications in French, Italian, Spanish and German. Articles in different languages were not analysed by the authors due to resource limitations. As an additional quality control standard, and when given this possibility by the database search engines, the search was restricted to peer-reviewed articles only. The databases were last assessed on March 13, 2016. However, a handful of articles have been further assessed following the peer review process.

After removal of the duplicates, the initial search revealed a total of 816 records, to which were added manually retrieved articles (n = 31), amounting to a total of 835 records (Fig 1). These additional articles were located through analyses of bibliographies and reference sections of the initial search results.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Flow diagram of the systematic search strategy for tools predicting and assessing psychological predictors, affinities, and attitudes towards terrorism, extremism, radicalisation and ideas conveyed by those concepts.

https://doi.org/10.1371/journal.pone.0166947.g001

Study selection

After having completed the computerised search of all 20 databases, two reviewers independently assessed the records one by one, and identified unrelated titles. Such a strategy enabled the exclusion of 553 records (see Fig 1 for detailed reasons of exclusion).

In the second stage of the selection, two reviewers worked independently and assessed the records based on their titles and abstracts. Applying the inclusion and exclusion criteria delineated below, the records deemed not relevant to the question were excluded, and corresponded to a total of 208 records (see Fig 1 for detailed reasons of exclusion).

The third and final stage consisted of a full-text screening of the records by both reviewers. Upon assessment, each reviewer independently decided which articles were eligible to be included in the systematic review. Any disagreement was resolved by a discussion between the two reviewers. When the two reviewers were unable to find consensus, a third reviewer was consulted to make the final decision. This final screen yielded the exclusion of 35 records (see Fig 1 for detailed reasons of exclusion).

A total of 37 articles were included in the systematic review and consisted of 4 instruments designed to be used operationally by professionals to give an opinion on individuals that might be at risk, 17 tools developed as research measures of a particular and relevant psychological construct, and 9 inventories that have not been generated from a study.

Inclusion and exclusion criteria

Study characteristics, data extraction and quality assessment

A formal threefold classification of the articles depending on the type of tool the studies were making use of was created: instruments to be used operationally by professionals, tools developed as research measures, and inventories not generated from a study.

Each study was thoroughly analysed, and the information was extracted into standardised coding sheets.

Information concerning the details of each study was consolidated in a first coding sheet (see Appendix A in S1 File), and included the following characteristics:

1. Country and setting of the study/tool
2. Domain(s) assessed by the tool
3. Population—number of respondents and gender distribution
4. Age of the respondents—age range, mean, and standard deviation
5. Description of items used in the study/tool

An additional coding sheet was developed to extract background information (see Table 3):

1. Year of publication
2. Author(s)
3. Area of expertise of the author(s)
4. Journal of publication

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Background summary of studies/tools and abbreviations used in the systematic review.

https://doi.org/10.1371/journal.pone.0166947.t003

A preliminary and methodological quality check was conducted using the 26-item checklist developed by Bennet and colleagues (2011), and constituted the third coding sheet of this systematic review (see Table 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Methodological quality markers in the different articles analysed.

https://doi.org/10.1371/journal.pone.0166947.t004

Finally, the authors developed a fourth and final coding sheet to explore the psychometric properties of each tool, and hence assess their validity and reliability (see Table 5). The extracted properties (see Table 2 for the definitions) included:

1. Readability, cultural translation, respondent burden
2. Validity (content, criterion, construct)
3. Internal consistency
4. Reliability (inter-rater, intra-rater, test-retest)
5. Positive and negative predicted value
6. Sensitivity and specificity
7. Floor and ceiling effect
8. Responsiveness

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Studies/tools psychometric properties.

https://doi.org/10.1371/journal.pone.0166947.t005

This process was conducted by two reviewers, with the second reviewer completing a proportion for quality control. Where necessary, any disagreement was resolved by discussion; if this failed, the third reviewer made a final decision.

Methods of review

Following the threefold division developed by the authors, this systematic review will discuss the background information of the selected tools. A primary assessment of the methodological quality markers of the studies will then be conducted, after which will ensue an in-depth analysis of their psychometric properties, and overall validity and reliability. Finally, preliminary conclusions will be drawn.

Data synthesis

As delineated above, the findings were synthesised in clear and concise tables in order to smooth comparisons, and were subsequently reported in a narrative overview.

Overall description of the studies and tools

A total of 37 articles were included and reviewed using the PRISMA tool (see Fig 1 and S1 File for details). This analysis resulted in a total of 6,636 respondents between the ages of 14 and 92 years, encompassing all geographical continents with Belarus, Italy, Serbia, Slovakia, Spain and the U.K. representing Europe; China, Korea, Kyrgyzstan, Malaysia, Pakistan and Sri Lanka representing Asia; Canada, Guatemala, Mexico and the U.S.A. representing North America; Israel and Lebanon representing the Middle East; Chile representing South America; Australia representing Oceania; and one study indicated sampling African respondents. In addition to geographical coverage, the respondents came from various backgrounds. Some sampled high school and college students and their parents, others were directed at Christian and Muslim beliefs, whereas other studies sampled volunteers in places of worship or members or religious seminaries. Certain studies included samples of sympathisers, activists and militants of radical movements as well, or even incarcerated individuals convicted of terrorist offenses.

The 37 articles were transformed into 30 tools due to the fact that certain articles developed more than one tool at a time, whereas other articles served to analyse the same tool from different perspectives. Of the 30 tools extracted, 12 evaluated attitudes and risk factors of extremism (38%), 8 evaluated terrorism (26%), 5 evaluated fundamentalism (16%), 3 evaluated radicalisation (10%), and finally 2 evaluated authoritarianism (8%).

The tools were separated according to a threefold division. Instruments to be used operationally by professionals included the VERA-2 with two distinct studies [14, 15], ERG 22+ [16], ERS [16] and IVPG with two distinct studies [17]. The tools developed as research measures of particular constructs were the 1992-RWA [6], RF-R [18], PHS [19], MMPI-2 [19], RWA-R [20], ITFS [21], ARIS [22], NBMASA [23], MEMS [24], MDFI [25], RF-I [26], SyfoR [27], IFS [28], SSS [29], ARIS-S [30], TCS [31], TRAP-18 [32] and a tool created by Schbley [33]. Finally, inventories not generated from a study comprised tools developed by Ross [34], Vaisman-Tzachor [35], Horgan [36], Saucier et al. [37], Kebbell and Porter [38], Monahan [39], USAID [40], Borum [41] and the EMI-20 [42].

It must be noted that the last category, inventories not generated from a study, has not been screened for markers of reporting quality, because the checklist developed by Bennett and colleagues is aimed at studies in which information on a specific topic is gathered from a population sample [43], and therefore would not be suitable for other articles. Their checklist was developed in an attempt to appraise the optimal reporting guidelines of survey research. As such, without an empirical study around which the article can revolves, it becomes impossible to successfully identify key reporting domains. Likewise, this review did not analyse the psychometric properties of these inventories, for the simple reason that it is inappropriate for articles that have not tested the tools. Their inclusion in this systematic review was solely aimed at giving a numerical representation of how many instruments are out there awaiting testing, and to stimulate further research in the field.

Preliminary quality reporting

Validity, reliability, and other psychometric properties

Implications

The non-existence of any uniform coding of the tools primarily hampered the comparison across studies, hence rendering the assessment of instruments somewhat less comprehensive, less transparent, and, as a result, less valid. However, having valid instruments is crucial in forensic contexts, and even more so in meta-analyses, as substantiated by studies by Hurducas and colleagues [44], White, Meares, and Batchelor [45] and Fusar-Poli and colleagues [46]. Hence, this systematic review argues that the relatively modest performance of all included studies can be attributed to the fact that none of them reported their results according to standardised reporting guidelines. To date, numerous guidelines and checklists have been developed across different fields in order to maximise the assessment of validity and reliability, and the authors strongly encourage researchers to accept such a ‘gold standard’ approach in their upcoming work, especially when those instruments are aimed at professionals to use proactively to identify individuals at risk.

Despite the obvious lack of transparency, and lack of information with regards to the validity and reliability of the tools, the studies varied in the number and type of potential risk factors tested. Most studies focused on extremism and terrorism, but also included an array of tools assessing fundamentalism, radicalisation and authoritarianism. They assessed various predictors ranging from inner beliefs, (socio-) psychological traits and processes acquired, attitudes and tendencies, willingness to engage in extreme means, hostility, mental wellbeing, and other predisposing vulnerabilities. However, on scrutiny of the background of studies, it became evident that certain predictors were undoubtedly study-specific, as the studies were based on peculiar events rather than being created for preventing a particular issue at hand. As a matter of fact, a large number of studies have been created following the 9/11 events. Studies of this nature, even when producing valuable results, are difficult to generalise. This level of precision renders it unclear as to whether or not health-care professionals across different legal jurisdictions could utilise these as a predictive indicator in their clinical practice.

Finally, one could ponder about the ethicality of implementing and proactively using certain instruments and policies without prior disclosure of the study results they were based on. Out of the four professional instruments, the ERG 22+ and ERS have restricted access to trained forensic psychologists or probation officers, on the hypothesis that disseminating their guidelines to the general public would be detrimental to their use [16]. As before mentioned, instruments need to be tested, validity and reliability need to be cross-verified, and studies need to be capable of being replicated and critiqued. The practice of withholding transparency of the results obtained prevents the (re-) evaluation of the methodology, data collection and further analyses by other professionals [47]. Hence, basing policies on such limited peer-reviewed information could be detrimental to those affected by it. Historically, it has been proven that empirical studies are at risk of being faulty, and failure to cross-check their validity has harmful consequences [48]. However, defective screening of individuals at risk of committing terrorist or extremist offenses would have different outcomes, as it would ‘merely’ result in the false incrimination of suspects. And similarly, having valid and widely recognised instruments would facilitate the taking into custody of individuals at risk, and hence avoid any potential false negatives situations, where high-potential individuals could mistakenly be released following an insufficient assessment.

To add to the controversy, if the authors of an instrument succeed in identifying all the relevant risk factors, it is their moral duty to share their findings with the wider community of researchers and authorities in order to try and prevent other crimes of this magnitude. However, the authors of the ERG 22+ claimed their tool to be unique and substantially different from the VERA-2. The former adopts a case-formulation approach, in which the presence of a factor combined with its role in the offense is assessed, whereas the latter is merely a conceptual formulation based on literature only [16]. As such, the ERG 22+ is purportedly aimed at guiding practice as well as stimulating further work in the field. The authors of this systematic review nonetheless argue that it is somewhat idealistic to expect results without having previously publicised the primary results.

Limitations

Whilst this systematic review is the most robust study to date, some limitations should be acknowledged. First of all, due to the vast scope of the literature covered, our study was restricted. The exclusion criteria were precise, and the authors took the liberty to eliminate articles related to ‘general’ violence, nationalism, and other far-reaching topics at the outset. Only studies that specifically assessed extremism, terrorism, fundamentalism, radicalisation, and/or authoritarianism were considered. Dissertations and other unpublished studies were not included, as the systematic review focused solely on peer-reviewed articles. This highlights acceptance of minor publication bias in the review. Another limitation, with respect to publication bias, concerns the range of languages of the articles assessed. The search strategy only covered articles written in English, French, Italian, Spanish, German and Dutch. The search resulted in only five articles written in a different language (two in Arabic, one in Japanese, one in Polish, and one in Lithuanian), but these were excluded due to a lack of resources. Finally, one must bear in mind that, even though all necessary measures were taken to ensure a comprehensive study, the possibility of missing eligible articles can never entirely be excluded.

Recommendations

Future research is needed to find a pragmatic way to progress efforts in the identification of risk factors of terrorism, extremism, radicalisation, authoritarianism, and fundamentalism, and to provide definitive conclusions about the relative value of the risk factors and attitudes identified by the different measures used.

First and foremost, the creation of joint working groups to review the findings of the articles might add substantial benefit in identifying tangible results in the field. As previously stated, practitioners from the fields of psychology or law alone are not best equipped to develop these instruments. Therefore, commissioning a multi-disciplinary committee to expand on specific research of this issue would be a step in the right direction. Ideally, the working group would include members of different faculties, in order to ensure the generalizability of the results. Reiterating what has been discussed before, this work would subsequently need to be field-tested to ensure valid and reliable results. However, given the recent course of events with an increase in terrorism, which is continuously spreading to new parts of the world, this would need to be fast-tracked in order to have a real and valuable input.

In the attempt to achieve this, researchers are encouraged to use standardised reporting guidelines to ensure a maximal level of transparency and comprehensiveness, and to ensure that the instruments proposed are valid and reliable. Not doing so would amount to research misconduct, and lead to refusal of being published by journal editors [49].

Finally, further research should aim at exploring the potential confounds of previous studies. As previously discussed, most studies have been created following specific acts of terrorism, and hence the possibility that confounds exist is a significant and non-negligible risk.

Conclusion

In summary, this systematic review highlights, in no uncertain terms, that the methodological reporting and overall quality, together with the psychometric soundness of the identified instruments, are weak and leave room for improvement. Numerous methodological flaws have been identified in all of the studies included, resulting in a limited interpretation and generalisation of the findings they presented. Even though assessments used by professionals are generally assumed to be the gold standard, there is, in this case, limited choice between the evaluated instruments, in that they are all relatively narrow and disclose very little information, if any at all. Based on the quality reporting and on the psychometric properties (or the lack thereof), there is no substantial evidence that would enable the authors to recommend one instrument over another.

The systematic review observed that significant policies and instruments have been, and continue to be developed based on limited information. This review encourage such studies to be published in their entirety and critiqued in order to ensure transparency.

Finally, it is recommended that a multi-disciplinary working committee is established to find a way to help identify individuals at risk of participating in terrorist and extremist acts of violence in a fully comprehensive and evidence-based manner.