http://orcid.org/0000-0002-1468-6029
Figures
Abstract
Background
The visual analogue scale (VAS) is widely used in clinical practice by occupational physicians to assess perceived stress in workers. However, a single cut-off (black-or-white decision) inadequately discriminates between workers with and without stress. We explored an innovative statistical approach to distinguish an at-risk population among stressed workers, and to establish a threshold over which an action is urgently required, via the use of two cut-offs.
Methods
Participants were recruited during annual work medical examinations by a random sample of workers from five occupational health centres. We previously proposed a single cut-off of VAS stress in comparison with the Perceived Stress Scale (PSS14). Similar methodology was used in the current study, along with a gray zone approach. The lower limit of the gray zone supports sensitivity (“at-risk” threshold; interpreted as requiring closer surveillance) and the upper limit supports specificity (i.e. “intervention” threshold–emergency action required).
Results
We included 500 workers (49.6% males), aged 40±11 years, with a PSS14 score of 3.8±1.4 and a VAS score of 4.0±2.4. Using a receiver operating characteristic curve and the PSS cut-off score of 7.2, the optimal VAS threshold was 6.8 (sensitivity = 0.89, specificity = 0.87). The lower and upper thresholds of the gray zone were 5 and 8.2, respectively.
Conclusions
We identified two clinically relevant cut-offs on the VAS of stress: a first cut-off of 5.0 for an at-risk population, and a second cut-off of 8.2 over which an action is urgently required. Future investigations into the relationships between this upper threshold and deleterious events are required.
Citation: Dutheil F, Pereira B, Moustafa F, Naughton G, Lesage F-X, Lambert C (2017) At-risk and intervention thresholds of occupational stress using a visual analogue scale. PLoS ONE 12(6): e0178948. https://doi.org/10.1371/journal.pone.0178948
Editor: Yu Ru Kou, National Yang-Ming University, TAIWAN
Received: January 21, 2015; Accepted: May 22, 2017; Published: June 6, 2017
Copyright: © 2017 Dutheil et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors received no specific funding for this work.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Workplace stress is a concern for both workers and employers because it can be the cause of absenteeism [1,2,3] and various pathologies such as anxiety [4], depression [4] and cardiovascular diseases [5,6,7]. To fight against work-related stress, and improve occupational health and safety concerns, the European directive on stress at work stated that occupational physicians needed to assess psychosocial risks [8].
Stress can be measured through questionnaires such as the Perceived Stress Scale (PSS) of Cohen [9] or Karasek’s scale [10]. However, their length and complexity render them impractical especially due to the usually limited time of occupational medicine visits [11]. The visual analogue scale (VAS) was originally developed to assess the perception of pain [12]. It has also been used to assess a variety of subjective feelings [13,14,15,16,17,18], including stress [13,14,17,18,19]. The use of VAS is attractive because it is simple to implement, easy to understand by the patient, and time-efficient in execution. Currently, the VAS is the common tool used by occupational physicians to assess stress among workers [13,14,19].
VAS was validated for stress assessment in clinical practice [11,19]. A single cut-off point was proposed for clinical discrimination between stressed and non-stressed workers [11]. However, a single cut-off point (black-or-white decision) inadequately discriminates between workers with and without stress [20]. The most stressed workers are known to commit suicide [21,22,23] and require urgent action. Identifying this group of workers is crucial. Moreover, occupational physicians have limited time to deal with considerable numbers of workers and worksites [24]. As such, physicians cannot thoroughly investigate the large proportion of stressed workers identified by a single threshold. Physicians don’t have time to go to workplaces of every stressed workers to investigate working conditions, to individually clarify the situation with managers, and to find solutions. Identifying workers under the highest pressure with improved accuracy would assist by reducing the numbers in this group to a manageable sample size. Conversely, as stress can vary over time, occupational physicians also need to consider more “potentially stressed” workers who might later need an intervention. “Potentially stressed” workers can be easily followed by the nursing staff sparing time for occupational physicians. Thus, it could be useful to distinguish among workers: (i) a low level of stress not requiring follow-up; (ii) an intermediate zone for further questioning by the clinician; and (iii) a high zone requiring immediate investigation and action. This approach can be modelled with the use of a gray zone [20].
Therefore, the aim of this study was to distinguish an at-risk population among stressed workers, and to establish a threshold over which action is urgently required, with the use of two cut-offs comprising the borders of the gray zone. The lower limit supports sensitivity (at-risk threshold–under closer surveillance) and the upper limit supports specificity (intervention threshold–emergency action required).
Methods
Participants
A previous study from our team proposed a cut-off of stress using VAS in comparison with PSS14 of Cohen [11]. We used the same methodology and we applied the gray zone approach. Workers were a random sample of workers from five occupational health centres in four different administrative regions in France. Those occupational health centres followed workers from private companies. Workers were recruited during annual work medical examinations. No consent was given because we used anonymous data from standard clinical records, and only access responses in questionnaires. The ethics committee from the university hospital of Clermont-Ferrand, France, assessed that there was no need for ethics approval.
Outcomes
Participants were asked to anonymously complete the PSS14 of Cohen [9], and to evaluate their level of stress by moving a cursor on a horizontal, non-calibrated line of 10 cm, ranging from very low (0) on the left to very high (10) on the right [13].
Statistics
It was difficult to propose sample size estimation according to literature in order to distinguish an at-risk population among stressed workers, and to establish a threshold over which action is urgently required, via the two cut-off points of the gray zone. Also, the sample size was limited to the available workers from several occupational health centres in different administrative regions in France. However, a posteriori statistical power calculation was proposed and is further clarified in the discussion.
Statistical analysis was performed using R software (http://cran.r-project.org/) and Stata software, version 13 (StataCorp, College Station, TX, US). All tests were two-sided, with a Type I error set at an alpha level of 0.05. Continuous data were expressed as means and standard deviations (SD) or as medians with interquartile ranges [IQR], and categorical parameters were reported as frequencies and relative percentages. The association between scores on the PSS14 and VAS was assessed with Spearman’s r correlation coefficient. First, based on previous data [11], workers were divided into two groups according to a PSS14 score cut-off set at 7.2. Comparisons between independent groups were analysed using the ‘N-1’ Chi-squared test [25] for the categorical variable of gender, and by the Mann-Whitney test for quantitative parameters (age and VAS score), as appropriate. The Gaussian distribution was verified by the Shapiro-Wilk test and homoscedasticity by the Fisher-Snedecor test. Then, a receiver operating characteristic (ROC) curve was used to illustrate the discriminating power of the VAS. A 95% confidence interval (CI) of the area under the ROC curve (AUC) was determined according to the method of DeLong et al. [26]. A “high stress” cut-off for the VAS was then determined by: maximising Youden’s index (J = sensitivity + specificity– 1); maximising the product of sensitivity and specificity [27], and choosing the nearest points to 0, and 1 on the curve. From these three methods, a gray zone was used to determine the values of the VAS for which workers were potentially stressed. This approach has been developed as a combination of two methods: bootstrap (sampling with replacement) and Two-Graph ROC (TG-ROC) [28]. The first bootstrap method generates 1,000 bootstrapped samples, determines the best threshold for each sample, and builds a 95% CI of them using their observed distribution (percentile confidence interval). All bootstrapped samples used the same size as the original sample. We used bootstrap in a conventional way (without any conditions) and also bootstrapped samples with the same prevalence of “stressed” workers (those for which PSS14 ≥7.2) as the original sample, in order to check that results were the same. The second method (TG-ROC) consisted of defining three classes of responses: negative, inconclusive, and positive. The inconclusive zone corresponded to the interval of cut-offs for which neither the sensitivity nor the specificity was greater than 0.90; illustrated by a two-curve representation (sensitivity, specificity). These curves were estimated using an artificial neural network. Bootstrap and TG-ROC are complementary, thus the largest interval was defined as the gray zone.
Results
Participants
We tested 500 workers (248 men, 252 women), aged 40±11 years, with a PSS14 score of 3.8±1.4 and a VAS score of 4.0±2.4. PSS14 and VAS were correlated (r = 0.65; p<0.001) (Fig 1). Nine (1.8%) of the 500 participants had a PSS14 score ≥7.2. Considering PSS14 as a binary variable (<7.2 vs. ≥7.2), there was a significant difference between gender (50.3% men with a PSS14 <7.2 vs. 11.1% men with a PSS14 ≥7.2, p<0.05) The VAS score differed between those with a PSS14 <7.2 (VAS = 3.8 [2.2–5.5] compared with individuals with a PSS14 ≥7.2, who had a VAS score of 8.2 [7.4–10.0] (p<0.001). In contrast, no significant differences were observed in age, education, professional roles, and occupational sectors (Table 1), as well as for occupations (Supplemental file S1 Database for complete list of occupations).
A single cut-off with ROC-curve approach
Using a ROC curve approach, the global discriminative power of the VAS was significant with an AUC of 0.93 (95% CI: 0.86–1.00) (Fig 2). The optimal VAS threshold was 6.8 (sensitivity = 0.89, 95% CI: 0.57–0.98; specificity = 0.87, 95% CI: 0.83–0.89) independent of the method used (maximising Youden’s index, maximising the product of sensitivity and specificity, and choosing the nearest point to 0, and 1).
The area under the curve (AUC) is presented with 95% confidence interval. The gray zone is represented by the gray square.
The gray zone approach: Determination of at-risk and intervention threshold
An optimal threshold was then determined for 1,000 bootstrapped samples having the same prevalence of ‘stressed’ workers as the original sample (i.e. 1.8%), and the 95% CI of their distribution was 5.0 to 8.2 for each of these three methods. Without requiring any condition to realize sampling with replacement, the 95% CI was exactly the same. Using a TG-ROC, an inconclusive zone ranging from 6.0 to 7.5 was retrieved (Fig 3). The results of bootstrap sampling and TG-ROC were merged into a single gray zone (the larger one); 5.0 to 8.2. This gray zone included 172 workers, i.e. 34.4% of the whole sample (at-risk). Seventeen workers of 500 (3.4%) were above the upper limit (intervention). Using the single threshold of 6.8, 74 workers (14.8%) would have been considered as stressed workers. Without the use of a gray zone approach, 115 (23.0%) workers potentially at risk, would not have been identified (those with a VAS score between 5 and 6.8).
The inconclusive zone (hatch rectangle) corresponds to the interval of cut-offs for which neither the sensitivity nor the specificity are greater than 0.90.
Discussion
The main finding of this study was the identification of two cut-offs for stress using the gray zone approach, a first cut-off of 5.0 to distinguish an at-risk population among stressed workers, and a second cut-off of 8.2 over which action is urgently required.
VAS of stress: A discriminant tool
We confirmed with a larger dataset, the previous finding of a relationship between the VAS and the PSS14 of Cohen (r = 0.65, p<0.001 in the current study and r = 0.68, p<0.001 in a previous sample [11]). Results support the discriminative power of the VAS score to predict a PSS score ≥7.2 [11]. Using three different methods to determine cut-off points, we retrieved an optimal VAS threshold of 6.8 to predict a PSS score ≥7.2, which is close to the threshold of 7.0 previously reported in a smaller sample size [11].
The gray zone approach: A three-zone analysis
Despite confirmation that the VAS is a useful and valid tool for the assessment of stress by occupational physicians, the use of a gray zone shows the limitations of using a single cut-off point to support clinical application [20]. Similar to photography, a black and white picture is less accurate than a grey scale one. A three zones analysis is postulated to provide a better compromise between a pragmatic and an accurate assessment. The distinction of three levels (low, moderate, high) is common both in psychological evaluation i.e. Maslach burn-out inventory [29,30,31], and in medical evaluation i.e. probability [32,33] or activity of a disease [34]. For assessing stress, and because occupational physicians may also experience burn-out due to the overload demand [24], they need to differentiate an at-risk population from workers requiring urgent action within the workers previously identified with the highest stress. Indeed, while a single cut-off point previously dichotomized the workers into stressed and unstressed, the gray zone provided two thresholds. First, workers with a VAS score lower than the lower limit of the gray zone can be confidently considered as unstressed workers. Then, workers with a VAS score in the gray zone can be seen as “potentially stressed”. For these individuals, an extended interview can be conducted; preventive action may be implemented and a closer follow-up should be proposed. A non-negligible proportion (23.0%) of workers potentially at risk would not have been identified with the use of a single threshold. This demonstrates another benefit of the use of a gray zone approach. Also, workers with a VAS score greater than the upper limit of the gray zone can be confidently considered as workers under pressure, for whom action is urgently required. Indeed, the most stressed workers are known to commit suicide [21,22,23].
Management of workers based on stress levels
As it is impossible for an occupational physician to thoroughly and urgently investigate a vast number of workers, the use of a gray zone permitted a reduction in the number of workers in need of urgent action, to a manageable sample, i.e. from 74 workers via the previous single cut-off to 17 via the use of a gray zone. Conversely, the number of “potentially stressed” workers became rather large. Nonetheless, the number of workers in the gray zone is relatively more easily manageable in occupational medicine. The more advanced interview can be assessed easily by simple questions during routine medical examination. Moreover, preventive action may be implemented by the occupational team. Intervention at the workplace can be conducted by occupational therapist. And the closer follow-up can be easily provided by the nursing staff. Indeed, occupational consultations can be conducted by an occupational physician or by a nurse with specific diploma, based on the decision of the occupational physician for each worker. Naturally, workers requiring urgent action may also benefit from the whole occupational team. Lowering the number of workers above the intervention threshold for whom emergency action is required is crucial because occupational physicians are over-worked and are themselves under pressure [24]. Indeed, occupational physicians have to manage workers requiring an emergency action themselves and cannot delegate this task. For workers above the intervention threshold, physicians must go to workplaces to investigate working conditions and to find solutions with managers for resolving conflicts [35]. Physicians can adapt working conditions [36] or promote a temporary sick leave. In extreme cases, workers can receive an unfit to work diagnosis [36]. Thus, physicians will benefit from access to a tool that is relatively simple to administer; with a potential to identify a smaller number of the most stressed workers. The increasing number of “potentially stressed” workers with the use of the gray zone approach is not problematic for occupational physicians because they can delegate the follow-up to the occupational team, especially to occupational nurses for a closer follow-up with dedicated consultations.
VAS of stress: A first approach strategy
The conceptual construct of stress is complex [37,38]. The PSS14 is composed of several items estimating the level of stress based on objective stressful events, as well as the ability to cope with challenges and personality. [9]. Both the VAS of stress and the PSS14 appear to measure a similar concept of stress as previously demonstrated [11]. However, because of the usually limited time of occupational medicine consultations [24] and because of the variety of problems that an occupational physician must assess [24], a less time-demanding tool may be more efficient; this is one of the advantage of the VAS for assessing stress. Then, further questionnaires can be used in case-specific situations to explore the cause of stress such as the Karasek’s scale [10,39] or the Siegrist questionnaire [40].
Medical consequences of stress
High levels of stress promote the onset of multiple chronic and acute diseases such as cardiovascular events [6,7]. Thus, the upper limit of stress that we established at 8.2 on the VAS needs to be considered as seriously in need of medical support. In addition to previously cited interventions (investigation at the workplace, discussion with managers, adapting working conditions, sick leave, unfit to work diagnosis), those workers need medical specialists such as psychiatrists, psychologists, or dedicated consultations in specialized occupational centers for bullying at work [35]. From our sample, we retrieved 17 workers (3.4%) needing urgent intervention. This upper limit is the threshold that favours the specificity. Thus, these 17 workers most certainly required medical support. However, some workers in the gray zone (n = 172, at-risk) may also require a medical support. We believe that within these at-risk workers, further questions during medical examination could distinguish the workers needing an intervention. In the absence of a cost effective alternative strategy for careful clinical assessment, the use of a gray zone with appropriate screening tools may help the clinician to instigate additional medical support. Considering that an occupational physician follows around 3000 workers per year, this upper intervention threshold seems manageable for the physician. We retrieved 38% of workers within both the at-risk and intervention groups, whereas previous studies reported a variable prevalence of job-related distress depending on the assessment tools [14,41,42,43]. However, there are no epidemiological data on severe stress prevalence in workers. Some studies assessed the impact of stress on stress-related diseases with assessment methods other than VAS [1,2,3,4,5,6,7]. To our knowledge, despite the VAS of stress being linked to physiological measures of stress such as inflammatory cytokines [13] or heart rate variability [44] that is linked to life expectancy [45,46,47,48], no studies have investigated levels of stress measured with VAS and disease. Thus, further cohort studies [49] are needed to investigate the relationships between deleterious events and at-risk and intervention thresholds. Additional VAS detected stress measures could further explore relationships between stress and absenteeism and or, work productivity as previously suggested [3,17,36,50].
Limitations
This study has some limitations. The replacement of a single cut-off point by a gray zone may improve the decision-making process of occupational health practitioners. However, clinical examination and occupational physicians’ observations remain essential for assessing repercussions of stress and to support the most appropriate decisions. Also, the low sample size of stressed workers (n = 9) did not allow separate gender analyses in order to investigate a potentially significant relationship between gender and the PSS14 using a binary analysis (<7.2 and ≥7.2).
As outlined in the statistical descriptions, due to difficulties in estimating sample size, a posteriori statistical power calculation was proposed according to results observed for the primary outcome. The AUC-ROC that determined the global discriminative power of the VAS was 0.93, with CI’s lower limit greater than 0.8 (95% CI: 0.86–1.00). Furthermore, with n = 311, n = 172 and n = 17 individuals, the statistical power was greater than 95% in order to show a difference between groups [0 to 5[, [5 to 8.2] and] 8.2 to 10] (respectively, 2.49 ± 1.38 vs. 6.23 ± 0.99 vs. 9.38 ± 0.70, using sampsi and power one-way commands of Stata software considering inflation of type I error due to multiple comparisons). This statistical power seemed satisfactory despite the imbalanced size of groups [51]. Moreover, as indicated by Feise [52], it is necessary to focus not only on statistical significance, but also on the quality of the research within the study and the magnitude of improvement.
Only a modest correlation was observed between the VAS and PSS14 scores (r = 0.65, p<0.001). However, this does not mean that the PSS is capturing something that the VAS is missing. The multi item scales of the PSS were built to provide a lower variance, but the VAS has demonstrated a greater discriminative capacity than PSS [19]. The VAS, and short screening tools in general, are probably more efficient for clinical use [53].
To conduct a gray zone analysis using the VAS of stress, we had to know the binary classification (stress–no stress) using the gold standard (PSS14 in our study). Thus, as PSS14 was already the gold standard, we cannot calculate a gray zone for PSS14.
Our 500 workers were similar in demographic characteristics to total population from which they were sampled. However, as workers included in this study were only from private companies, further studies should be conducted with a larger sample including non-private sectors to confirm the generalizability of our results to all workers, and to better understand potential associations between stress scores and demographics, employment sectors and current health status or occupational conditions [36,54], as well as with biomarkers of stress [35,48,55,56]. Investigating these relations in non-workers, such as for people living in institutional care, may also have a considerable interest and provide directions for future interventions.
Conclusion
The use of a VAS for stress assessment is a useful and suitable tool for occupational physicians to estimate stress during clinical examination. To overcome the imperfection of the discrimination caused by the choice of a single threshold, alternative approaches were considered. We identified, with the use of the gray zone approach, two clinically relevant cut-offs on the VAS of stress: a first cut-off of 5.0 for an at-risk population, and a second cut-off of 8.2 over which immediate action is strongly recommended. Future investigations into the relationships between this upper threshold and deleterious events are required.
Supporting information
S1 Database. Titles of columns are written without abbreviations.
https://doi.org/10.1371/journal.pone.0178948.s001
(XLSX)
Acknowledgments
We would like to express our gratitude to Dr. Maxime Cannesson and to Dr. Yannick Le Manach for their help regarding the gray zone methodology.
Author Contributions
- Conceptualization: FD.
- Data curation: CL BP.
- Formal analysis: CL BP FXL FD.
- Funding acquisition: FD BP FXL.
- Investigation: CL BP FM GN FXL FD.
- Methodology: FD BP FM GN FXL CL.
- Project administration: FD.
- Resources: FXL.
- Software: CL.
- Supervision: FD.
- Validation: BP.
- Visualization: FD CL.
- Writing – original draft: FD CL.
- Writing – review & editing: FD GN FXL.
References
- 1. Woo M, Yap AK, Oh TG, Long FY (1999) The relationship between stress and absenteeism. Singapore Medical Journal 40: 590–595. pmid:10628250
- 2. Nielsen ML, Rugulies R, Smith-Hansen L, Christensen KB, Kristensen TS (2006) Psychosocial work environment and registered absence from work: estimating the etiologic fraction. American Journal of Industrial Medicine 49: 187–196. pmid:16470544
- 3. Shelledy DC, Mikles SP, May DF, Youtsey JW (1992) Analysis of job satisfaction, burnout, and intent of respiratory care practitioners to leave the field or the job. Respiratory Care 37: 46–60. pmid:10145581
- 4. Rusli BN, Edimansyah BA, Naing L (2008) Working conditions, self-perceived stress, anxiety, depression and quality of life: a structural equation modelling approach. BMC Public Health 8: 48. pmid:18254966
- 5. Rosengren A, Hawken S, Ounpuu S, Sliwa K, Zubaid M, Almahmeed WA, et al. (2004) Association of psychosocial risk factors with risk of acute myocardial infarction in 11119 cases and 13648 controls from 52 countries (the INTERHEART study): case-control study. Lancet 364: 953–962. pmid:15364186
- 6. Richardson S, Shaffer JA, Falzon L, Krupka D, Davidson KW, Edmondson D (2012) Meta-analysis of perceived stress and its association with incident coronary heart disease. American Journal of Cardiology 110: 1711–1716. pmid:22975465
- 7. Chandola T, Britton A, Brunner E, Hemingway H, Malik M, Kumari M, et al. (2008) Work stress and coronary heart disease: what are the mechanisms? European Heart Journal 29: 640–648. pmid:18216031
- 8. Iavicoli S, Natali E, Deitinger P, Maria Rondinone B, Ertel M, Jain A, et al. (2011) Occupational health and safety policy and psychosocial risks in Europe: the role of stakeholders' perceptions. Health Policy 101: 87–94. pmid:20832135
- 9. Cohen S, Kamarck T, Mermelstein R (1983) A global measure of perceived stress. J Health Soc Behav 24: 385–396. pmid:6668417
- 10. Karasek R, Brisson C, Kawakami N, Houtman I, Bongers P, Amick B (1998) The Job Content Questionnaire (JCQ): an instrument for internationally comparative assessments of psychosocial job characteristics. J Occup Health Psychol 3: 322–355. pmid:9805280
- 11. Lesage FX, Berjot S (2011) Validity of occupational stress assessment using a visual analogue scale. Occup Med (Lond) 61: 434–436.
- 12. Huskisson EC (1974) Measurement of pain. Lancet 2: 1127–1131. pmid:4139420
- 13. Dutheil F, Trousselard M, Perrier C, Lac G, Chamoux A, Duclos M, et al. (2013) Urinary Interleukin-8 Is a Biomarker of Stress in Emergency Physicians, Especially with Advancing Age—The JOBSTRESS* Randomized Trial. PLoS One 8: e71658. pmid:23977105
- 14. Dutheil F, Boudet G, Perrier C, Lac G, Ouchchane L, Chamoux A, et al. (2012) JOBSTRESS study: comparison of heart rate variability in emergency physicians working a 24-hour shift or a 14-hour night shift—a randomized trial. Int J Cardiol 158: 322–325. pmid:22608270
- 15. Boshuizen RC, Vincent AD, van den Heuvel MM (2013) Comparison of modified Borg scale and visual analog scale dyspnea scores in predicting re-intervention after drainage of malignant pleural effusion. Supportive Care in Cancer 21: 3109–3116. pmid:23842597
- 16. Eaton TA, Comer SD, Revicki DA, Trudeau JJ, van Inwegen RG, Stauffer JW, et al. (2012) Determining the clinically important difference in visual analog scale scores in abuse liability studies evaluating novel opioid formulations. Quality of Life Research 21: 975–981. pmid:21964915
- 17. Dutheil F, Duclos M, Naughton G, Dewavrin S, Cornet T, Huguet P, et al. (2017) WittyFit-Live Your Work Differently: Study Protocol for a Workplace-Delivered Health Promotion. JMIR Res Protoc 6: e58. pmid:28408363
- 18. Dutheil F, Marhar F, Boudet G, Perrier C, Naughton G, Chamoux A, et al. (2017) Maximal tachycardia and high cardiac strain during night shifts of emergency physicians. Int Arch Occup Environ Health.
- 19. Lesage FX, Berjot S, Deschamps F (2012) Clinical stress assessment using a visual analogue scale. Occup Med (Lond) 62: 600–605.
- 20. Cannesson M, Le Manach Y, Hofer CK, Goarin JP, Lehot JJ, Vallet B, et al. (2011) Assessing the diagnostic accuracy of pulse pressure variations for the prediction of fluid responsiveness: a "gray zone" approach. Anesthesiology 115: 231–241. pmid:21705869
- 21. Brodsky CM (1977) Suicide attributed to work. Suicide and Life-Threatening Behavior 7: 216–229. pmid:148757
- 22. Gradus JL, Qin P, Lincoln AK, Miller M, Lawler E, Sorensen HT, et al. (2010) Acute stress reaction and completed suicide. International Journal of Epidemiology 39: 1478–1484. pmid:20624822
- 23. Feskanich D, Hastrup JL, Marshall JR, Colditz GA, Stampfer MJ, Willett WC, et al. (2002) Stress and suicide in the Nurses' Health Study. Journal of Epidemiology and Community Health 56: 95–98. pmid:11812806
- 24. Lesage FX, Berjot S, Altintas E, Paty B (2013) Burnout Among Occupational Physicians: A Threat to Occupational Health Systems?—A Nationwide Cross-sectional Survey. Annals of Occupational Hygiene 57: 913–919. pmid:23524634
- 25. Campbell I (2007) Chi-squared and Fisher-Irwin tests of two-by-two tables with small sample recommendations. Stat Med 26: 3661–3675. pmid:17315184
- 26. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44: 837–845. pmid:3203132
- 27. Liu X (2012) Classification accuracy and cut point selection. Statistics in Medicine 31: 2676–2686. pmid:22307964
- 28. Greiner M, Sohr D, Gobel P (1995) A modified ROC analysis for the selection of cut-off values and the definition of intermediate results of serodiagnostic tests. J Immunol Methods 185: 123–132. pmid:7665894
- 29. Maslach C, Schaufeli WB, Leiter MP (2001) Job burnout. Annual Review of Psychology 52: 397–422. pmid:11148311
- 30. Maslach C, Jackson SE (1981) The measurement of experienced burnout. Journal of Organizational Behavior 2: 99–113.
- 31. Choron G, Dutheil F, Lesage FX (2016) Are nurses burned out? Int J Nurs Stud 58: 80–81. pmid:27087300
- 32. Wells PS, Anderson DR, Rodger M, Ginsberg JS, Kearon C, Gent M, et al. (2000) Derivation of a simple clinical model to categorize patients probability of pulmonary embolism: increasing the models utility with the SimpliRED D-dimer. Thrombosis and Haemostasis 83: 416–420. pmid:10744147
- 33. Wicki J, Perneger TV, Junod AF, Bounameaux H, Perrier A (2001) Assessing clinical probability of pulmonary embolism in the emergency ward: a simple score. Archives of Internal Medicine 161: 92–97. pmid:11146703
- 34. Conaghan PG, Hensor EM, Keenan AM, Morgan AW, Emery P (2010) Persistently moderate DAS-28 is not benign: loss of function occurs in early RA despite step-up DMARD therapy. Rheumatology 49: 1894–1899. pmid:20542894
- 35. Lac G, Dutheil F, Brousse G, Triboulet-Kelly C, Chamoux A (2012) Saliva DHEAS changes in patients suffering from psychopathological disorders arising from bullying at work. Brain Cogn 80: 277–281. pmid:22940752
- 36. Dutheil F, Naughton G, Sindyga P, Lesage FX (2016) Ill Health-Related Job Loss: A One-Year Follow-Up of 54,026 Employees. J Occup Environ Med 58: 918–923. pmid:27454396
- 37. Koolhaas JM, Bartolomucci A, Buwalda B, de Boer SF, Flugge G, Korte SM, et al. (2011) Stress revisited: a critical evaluation of the stress concept. Neuroscience and Biobehavioral Reviews 35: 1291–1301. pmid:21316391
- 38. Woda A, Picard P, Dutheil F (2016) Dysfunctional stress responses in chronic pain. Psychoneuroendocrinology 71: 127–135. pmid:27262345
- 39. Trousselard M, Dutheil F, Naughton G, Cosserant S, Amadon S, Duale C, et al. (2015) Stress among nurses working in emergency, anesthesiology and intensive care units depends on qualification: a Job Demand-Control survey. International Archives of Occupational and Environmental Health.
- 40. Siegrist J (1996) Adverse health effects of high-effort/low-reward conditions. Journal of Occupational Health Psychology 1: 27–41. pmid:9547031
- 41. Kawakami N, Haratani T (1999) Epidemiology of job stress and health in Japan: review of current evidence and future direction. Industrial Health 37: 174–186. pmid:10319566
- 42. Olatona FA, Ezeobika EN, Okafor IP, Owoeye OB (2014) Work related stress and coping mechanisms among bankers in Lagos, Nigeria. African Journal of Medicine and Medical Sciences 43: 59–65.
- 43. Tsai YC, Liu CH (2012) Factors and symptoms associated with work stress and health-promoting lifestyles among hospital staff: a pilot study in Taiwan. BMC Health Serv Res 12: 199. pmid:22799278
- 44. Langelotz C, Scharfenberg M, Haase O, Schwenk W (2008) Stress and heart rate variability in surgeons during a 24-hour shift. Archives of Surgery 143: 751–755. pmid:18711034
- 45. Giese-Davis J, Wilhelm FH, Tamagawa R, Palesh O, Neri E, Taylor CB, et al. (2015) Higher Vagal Activity as Related to Survival in Patients With Advanced Breast Cancer: An Analysis of Autonomic Dysregulation. Psychosomatic Medicine.
- 46. Almoznino-Sarafian D, Sarafian G, Zyssman I, Shteinshnaider M, Tzur I, Kaplan BZ, et al. (2009) Application of HRV-CD for estimation of life expectancy in various clinical disorders. Eur J Intern Med 20: 779–783. pmid:19892308
- 47. Boudet G, Walther G, Courteix D, Obert P, Lesourd B, Pereira B, et al. (2017) Paradoxical dissociation between heart rate and heart rate variability following different modalities of exercise in individuals with metabolic syndrome: The RESOLVE study. Eur J Prev Cardiol 24: 281–296. pmid:27856807
- 48. Hufnagel C, Chambres P, Bertrand PR, Dutheil F (2017) The Need for Objective Measures of Stress in Autism. Front Psychol 8: 64. pmid:28191002
- 49. Saadaoui F, Bertrand P, Boudet G, Rouffiac K, Dutheil F, Chamoux A (2015) A Dimensionally Reduced Clustering Methodology for Heterogeneous Occupational Medicine Data Modelling. IEEE Trans Nanobioscience.
- 50. Dollard MF, Neser DY (2013) Worker health is good for the economy: union density and psychosocial safety climate as determinants of country differences in worker health and productivity in 31 European countries. Social Science and Medicine 92: 114–123. pmid:23849285
- 51. Noether GE (1987) Sample size determination for some common nonparametric statistics. Journal of the American Statistical Association 82: 645–647.
- 52. Feise RJ (2002) Do multiple outcome measures require p-value adjustment? BMC Med Res Methodol 2: 8. pmid:12069695
- 53. Mitchell AJ (2010) Short screening tools for cancer-related distress: a review and diagnostic validity meta-analysis. J Natl Compr Canc Netw 8: 487–494. pmid:20410338
- 54. Lopez V, Chamoux A, Tempier M, Thiel H, Ughetto S, Trousselard M, et al. (2013) The long-term effects of occupational exposure to vinyl chloride monomer on microcirculation: a cross-sectional study 15 years after retirement. BMJ Open 3.
- 55.
Azzaoui N, Guillin A, Dutheil F, Boudet G, Chamoux A, Perrier C, et al. (2014) Classifying heartrate by change detection and wavelet methods for emergency physicians. ESAIM Proceedings.
- 56. Dutheil F, Chambres P, Hufnagel C, Auxiette C, Chausse P, Ghozi R, et al. (2015) 'Do Well B.': Design Of WELL Being monitoring systems. A study protocol for the application in autism. BMJ Open 5: e007716. pmid:25710916