Data collection and filtering

We used data from a cohort of 4,117 tinnitus patients who had been treated at the Tinnitus Center of Charité Universitaetsmedizin Berlin between January 2011 and October 2015. All included patients had been suffering from tinnitus for 3 months or longer, were 18 years of age or older and had sufficient knowledge of the German language. Treatment comprised an intensive, multimodal 7-day program that included informational counseling, detailed ear-nose-throat (ENT) as well as psychological diagnostics, cognitive-behavior therapy interventions, relaxation exercises and physiotherapy. Ethical approval was granted by Charité Universitaetsmedizin Berlin ethics committee (reference number EA1/115/15) and informed written consent was received from all patients. All relevant guidelines and regulations were followed. Prior to the analyses, all data had been anonymized.

A total of 205 features were extracted at baseline (T0) and after treatment (T1), comprising answers to single questionnaire items, subscale scores and total scores from the following 10 questionnaires: (a) General Depression Scale—long form (Allgemeine Depressionsskala; ADSL) [15, 16], (b) Berlin Complaint Inventory (Berliner Beschwerdeinventar; BI) [17], (c) Berlin Mood Questionnaire (Berliner Stimmungsfragebogen; BSF) [18], (d) Perceived Stress Questionnaire (PSQ) [19], (e) Short Form-8 Health Survey (SF8) [20], (f) a sociodemographics questionnaire (SOZK) [21], (g) Self-Efficacy- Optimism-Pessimism Scale (Selbstwirksamkeits-Optimismus-Pessimismus Skala; SWOP) [22], (h) visual analogue scales measuring tinnitus loudness, frequency and distress (TINSKAL) and the (i) Tinnitus Localization and Quality questionnaire (TLQ) [23]. Tinnitus-related distress was assessed using the German version of the (j) Tinnitus Questionnaire (TQ) [24]. Furthermore, for each questionnaire, the average time needed to fill-in an item was recorded. Most questionnaire items comprised multiple-choice questions with answers on a Likert scale. The corresponding ordinal features were handled as continuous features in the analysis. Categorical features, e.g. sex, marital status and education level were encoded as dichotomous features. A brief overview of all features is provided in S1 Table. 1,416 patients (34.4%) who completed all 10 questionnaires before and after treatment were included in the analyses. Table 1 depicts baseline characteristics of all included patients. The TQ tinnitus-related distress score [24] was discretized using the cutoff value of 46 [24] distinguishing between “compensated” (0-46) and “decompensated” tinnitus (47-84). The associated feature TQ_distress measured after treatment (T1) was used as target feature variable for all analyses.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Baseline characteristics of patients (data at T0).

https://doi.org/10.1371/journal.pone.0228037.t001

Exploratory correlational analysis

Exploratory correlational analysis assessed the strength of bivariate relationships among the recorded features at T0 as well as between each feature and the TQ distress score. More specifically, the Spearman correlation coefficient was calculated (i) to identify groups of features with similar intra-group and inter-group correlations, (ii) to assess questionnaire median correlations with the TQ distress score measured in T0 and T1, (iii) to identify features with the highest correlational magnitude with respect to the TQ distress score in T0 and T1 and (iv) to identify features whose correlational effects with the TQ distress score differed between T1 and T0.

Classifier training

The potential of machine learning for the prediction of TQ_distress at T1 using questionnaire data from T0 only was investigated with the following 11 algorithms: LASSO [25], RIDGE [26], generalized partial least squares (GPLS) [27], support vector machine (SVM) [28], a feed-forward neural network with one single hidden layer (NNET) [29], weighted k-nearest neighbor classifier (WKNN) [30], Naïve Bayes classifier (NB), CART decision tree [31], C5.0 decision tree [32], random forest (RF) [33] and gradient boosted trees (GBT) [34].

• LASSO and RIDGE are extensions of ordinary least squares linear regression. Their objective function contains an additional penalty term, either to control the inclusion of predictors (LASSO) or to shrink the magnitudes of the regression coefficients (RIDGE). As a result, models tend to have better predictive performance with better interpretability due to their inbuilt feature subset selection in comparison with ordinary linear regression.
• Partial least squares is another extension of linear regression which first performs a dimension reduction by constructing a new set of features that are linear combinations of the original features, and then fit a linear regression to these new features. Often, the number of features of the projection is set to be much lower than the number of features of the original feature space.
• SVMs are capable of modeling non-linear relationships between the predictors and the target feature. They use a non-linear mapping to enlarge the feature space of the original training data into a higher dimension. Within this new dimension, the optimal linear separating hyperplane is identified. This hyperplane is the decision boundary separating the observations from different classes.
• NNET is an example of neural networks. Neural networks extract new features by linear combinations of the original features and to use them to model the target feature as non-linear function of these features.
• WKNN is a variant of the KNN classification algorithm which does not build a model at all, but rather identifies for a particular observation with unknown class label the K “closest” observations from the training data and uses their majority class as prediction. Closeness is defined by a distance measure such as the Euclidean distance. WKNN uses distance to weight the influence a training observation has on the prediction, such that training instances with low distance obtain a higher weight.
• NB uses Bayes’ theorem to calculate class membership probabilities. The “naive” property refers to the assumption of class-conditional independence among the features, which is employed to reduce computational complexity.
• CART, C5.0, RF and GBT are tree-based methods. Classifiers of this family partition the feature space into a set of non-overlapping rectangles based on combinations of feature-value conditions, such as “IF age < 52 & ADSL_depression > 19”. To make a prediction for a given observation, they use the majority class of training data assigned to the rectangle it belongs. Random forests and gradient boosted trees are ensembles of different simple decision trees, where each tree casts a vote towards the final prediction. Whereas trees in a random forest are built independently from each other, boosted trees iteratively add a tree to the composite model, aiming to reduce the classification error of the previously learned set of trees.

10-fold stratified cross-validation was used for classifier evaluation. In k-fold cross-validation, the data is split into k partitions. Each partition serves once as test set for the model which is trained on the remainder of the partitions. Finally, the k performance results are averaged. A grid search was employed for algorithm hyperparameter selection (cf. tuning grid in S2 Table). The area under the receiver operating characteristic curve (AUC) was used as performance measure. A receiver operating characteristic curve is a plot that juxtaposes sensitivity (true positive rate (TPR)) and false positive rate (FPR) for varying thresholds of a binary classifier. The area under the ROC curve (AUC) takes values from 0 (0% TPR, 100% FPR) to 1 (100% TPR, 0% FPR). The higher the AUC, the better is the classifier at distinguishing between patients with decompensated and compensated tinnitus.

Feature elimination

Although some of the utilized classification algorithms are insensitive to a high number of features, there are several reasons to remove superfluous predictors. For example, the selection of a feature subset contributes to the prevention of overfitting, the avoidance of multicollinearity and the identification of a model with good trade-off between high predictive performance and low complexity, i.e., a low number of features. Here, a feature elimination wrapper was developed that iteratively discarded a subset of features which were not contributing to predictive performance. This mechanism is an extension of the feature importance score for random forests [33] and its generalization to any model type [35], referred to as “model reliance”. The model reliance estimates the worth of an individual feature f by comparing the classification error on the original training set with the classification error on a modified version of the training set where the values of f are randomly permuted. A high model reliance score expresses high dependency of the model prediction on f, since the random permutation increased the classification error.

The model reliance MR of a model ζ on a feature f ∈ F is calculated as follows: First, the classification error on the original training data is calculated as e_orig = CE(y, ζ(X_orig)), where CE is the classification error function, y is the target feature and ζ(X_orig) is predicted target feature on the original training data. Then, the values of f are randomly permuted and the classification error on the slightly modified dataset X_perm is calculated as e_perm = CE(y, ζ(X_perm)). Finally, the model reliance MR(f) is the ratio of the two terms, i.e., . Since feature perturbation introduces a degree of randomness, MR was calculated as average over 10 runs as a more stable estimate. A MR value greater below 1 suggests that f is adversarial to model performance. Thus, our feature elimination wrapper successively removes these features to then train a new classifier on the subset of predictors with MR > 1. In the first iteration i = 1, an initial model ζ₁ is calculated on the full feature set F₁ = F. For each feature, the model reliance MR(f, ζ_i) is calculated. Features with MR(f, ζ_i) > 1 are retained for iteration i + 1 while the remaining features are dropped. This procedure continues until either none of the MR values exceed 1, i.e., ∀f ∈ F_i: MR(f, ζ_i) ≤ 1, or the feature set in iteration i is identical to the feature set in iteration i − 1, i.e., F_i = F_i−1.

Feature importance

Understanding the prediction of a classification model is a major challenge in order to obtain actionable insights that can ultimately contribute to improve prevention, diagnosis and treatment. Many state-of-the-art algorithms such as gradient boosted trees produce models with high accuracy. However, these so called black-box models are complex and not intrinsically understandable as they usually incorporate many multi-variate, non-linear relationships among groups of features, which are hard to present to the domain expert intuitively. A trade-off between predictive quality and understandability often means using less complex methods such as linear models and decision trees. Instead, in this study, both complex and understandable classifiers were investigated. To facilitate model interpretation, the model-agnostic post-hoc framework SHAP [36, 37] was used to assess feature importance. Briefly, the SHAP value ϕ_f(ζ, x) expresses the estimated importance of a feature f to the prediction of model ζ for an instance x as change in the expected value of the prediction if for f the feature vector of x is observed instead of being random. The SHAP framework composes the model prediction as sum of SHAP values of each feature, i.e., , where ϕ₀(ζ, x) is the expected value of the model (bias) and M is the number of features.

SHAP values were calculated for the best model ζ_opt according to AUC. A ranking of T0 feature attribution towards ζ_opt was determined by calculating the average SHAP value magnitude over all instances, i.e., , where A(j) is the attribution of the j-th feature. The N × M SHAP matrix was clustered with agglomerative hierarchical clustering to identify subgroups of patients with similar SHAP values.

Distribution of tinnitus-related distress at T0 and T1

Approximately a third (32.1%) of the 1,416 subjects reported decompensated tinnitus at T0 (Fig 2A). Almost half of these patients (14.5%) transitioned to compensated tinnitus (CT) with treatment. Overall, 283 out of 1,416 patients (20.0%) showed decompensated tinnitus at T1. A general positive effect of treatment is indicated by the slope of the linear regression line below 1.0 in Fig 2B with TQ_distress in T1 as dependent variable and TQ_distress in T0 as independent variable.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Tinnitus-related distress (TQ_distress) before and after treatment.

(A) 83% of patients did not move between categories of overall tinnitus-related distress status (i.e. compensated or decompensated) with treatment. However, significantly more patients transitioned from “decompensated” to “compensated” tinnitus (14.5%) than vice versa (2.5%). (B) The slope of the regression line indicates a general decrease in tinnitus-related distress as a result of treatment.

https://doi.org/10.1371/journal.pone.0228037.g002

Correlation analysis

The heatmap in Fig 3A shows all pairwise feature-feature correlations in T0. Two major feature groups were identified, indicated by red squares, with moderate to high intra-group and negative inter-group correlations. The upper group (cf. Fig 3A) comprised 114/205 features (55.6%) representing negatively polarized questionnaire items and scales, e.g. the ADSL depressive disorder score (ADSL_depression) and the BI complaint strain score (BI_complaint). Conversely, the lower group included 47/205 positively polarized features (22.9%), e.g. the SF8 mental health score (SF8_mental) and the BSF elevated mood score (BSF_mood). Fig 3B juxtaposes pairwise correlations of each feature and TQ tinnitus-related distress score before (x-axis) and after treatment (y-axis). No strong bivariate correlation was observed, as all values lay within the interval -0.6 and 0.6. The average change in absolute correlations between the beginning and end of treatment was 0.031. The change in absolute correlation was less than 0.067 for 95% of the features (cf. closeness of the points to the diagonal line in Fig 3B). For 137 out of 205 features (66.8%), the absolute value of correlation decreased from T0 to T1. Median correlations of the questionnaires ADSL, BSF and BI (SF8) were above (below) 0.3 (-0.3) at both moments, respectively, and thus higher than for the remaining questionnaires. Fig 3C and 3D reveal that features from 6 questionnaires were among the top-20 features ranked by absolute correlation with TQ_distress in T0 and T1. The general depression score ADSL_depression showed strongest correlation before (ρ = 0.630) and after treatment (ρ = 0.564). Fig 3E depicts the 10 features with the largest differences in correlation magnitudes after vs. before treatment. For each of these features, the correlation before treatment is higher.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Feature-feature correlation & feature correlation with respect to TQ tinnitus-related distress score in T0 and T1.

(A) Correlation heatmap for all pairs of features (T0). Features are ordered by agglomerative hierarchical clustering with complete linkage. (B) Correlation of each feature with TQ tinnitus-related distress score, in T0 (x-axis) and in T1 (y-axis). The diamond symbol represents a questionnaire’s median. (C) Top-20 features with highest correlation to TQ tinnitus-related distress score (T0). (D) Top-20 features with highest correlation to TQ tinnitus-related distress score (T1). (E) Top-10 features whose correlational effects with TQ tinnitus-related distress score differ in T0 vs. T1. Correlation values before and after treatment are shown as light blue and dark blue bars, respectively. Differences in correlation are represented as black bars centered in between.

https://doi.org/10.1371/journal.pone.0228037.g003

Predictive performance of classification models

The classification models predicted tinnitus-related distress compensation after treatment based on questionnaire answers and socio-demographic data acquired at baseline with high AUC. Table 2 depicts the performance of all 11 classification methods across 10 feature elimination rounds. The gradient boosted trees model (GBT) achieved highest AUC (iteration i = 7, AUC = 0.890±0.04 [0.887,0.893]; mean±SD [95% CI]), using only 26 features. The RIDGE classifier yielded second-best performance (i = 2, AUC: 0.876±0.05 [0.873,0.879]), relying on 127 features, followed by the random forest model (i = 3, AUC: 0.872±0.05 [0.869,0.875]) using 77 features. Classification using the best model (GBT, i = 7) based on a probability threshold of 0.5 resulted in an accuracy of 0.86, a true positive rate (sensitivity) of 0.72, a true negative rate (specificity) of 0.88, a precision of 0.48 and a negative predictive value of 0.95.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Average cross-validation AUC and number of features (p) for each classifier with optimal hyperparameter configuration and for each feature selection iteration (i).

For each classifier, the best AUC is highlighted in boldface. Classifiers are ordered by their maximum AUC. All methods induced at least one model with AUC of 0.790 or higher. Cells with a “/” indicate that the feature elimination wrapper had already been terminated after a previous iteration.

https://doi.org/10.1371/journal.pone.0228037.t002

Classifier performance on smaller feature spaces

When trained using a smaller feature space, each classifier produced at least one model with nearly equal or even improved performance compared to the respective model learned on the whole feature space. In fact, except for WKNN, all classification methods benefited from feature elimination as they produced the best model on a reduced feature space (cf. Table 2). For GBT, the increase in AUC from 185 features to 26 features (i = 11) was 0.01. This model achieves both maximum AUC and a well-balanced trade-off between high predictive performance and low model complexity, and we thus decided to further investigate this model.

Feature importance

For the best model, the attributions of the 26 selected features are shown in Fig 4A. The TINSKAL impairment score (TINSKAL_impairment) was identified as most important, with an average absolute SHAP value magnitude (change in log odds) of 0.448. The ADSL depression score (ADSL_depression) and a single question from the general depression score questionnaire (ADSL_adsl11: “During the past week my sleep was restless.”) emerged as second and third most relevant. Besides 6 aggregated (sub-)scores, 12 single questionnaire items, 4 socio-demographic features (number of visited doctors, university-level education, lower secondary education, duration of tinnitus) and 4 features indicating time spent filling the questionnaires were selected. Notably, at least 1 feature from each of the 9 questionnaires was chosen. Fig 4B depicts the patient-individual SHAP values for each feature as points where color represents the actual feature value. The high attribution of TINSKAL_impairment is emphasized by the wide spread in the value distribution. For this feature, high feature values correspond to an increased probability of tinnitus decompensation. However, this trend is not monotone, since small values (light orange) are associated with a SHAP value just slightly below 0. This is emphasized in the feature dependence plots in Fig 5 which juxtapose feature values and corresponding SHAP values for all patients and for each of the 26 selected features. In fact, the top-left feature dependence plot in Fig 5 reveals a curvilinear, J-shaped relationship between TINSKAL_impairment and SHAP values. More specifically, the predicted tinnitus-related distress decreases within the range [0,3] with increasing values, whereas it decreases within (3,10]. Similar patterns were observed for ADSL_depression, TINSKAL_loudness, BI_complaint, BSF_timestamp and SWOP_pessimism. Although some of the features are not highly ranked globally, their impact on specific subgroups is high. For example, the feature SOZK_lowersec (lower secondary education) considerably increases the risk of tinnitus-related distress for patients with lower secondary education, but only marginally reduces the risk for patients of other education levels. For the majority of features, the relationship with predicted tinnitus-related distress is monotone. For instance, the SF8 physical score (SF8_physical) shows decreasing predicted tinnitus-related distress with increasing physical health. Fig 4C shows stacked patient-individual SHAP values for the six features with highest average SHAP value magnitude and the combined rest. Five subgroups of patients with similar explanation similarity were identified. Cluster 1 comprises a large fraction of patients with secondary school education (43 out of 76) which is considerably higher than the overall average of 11%. This subgroup has the highest risk of being classified with decompensated tinnitus (blue line depicts cluster average). Cluster 2 is the largest subgroup containing 50.7% of all patients. This subgroup can be characterized with depression severity and an overall low risk of high tinnitus severity. Cluster 4 is described by a high tinnitus impairment (TINSKAL_impairment) whereas clusters 3 and 5 are more heterogeneous with their average prediction close to the prior, respectively. For each of the 26 features with the highest average SHAP value magnitude, Fig 5 shows patient-individual feature values and corresponding attribution towards the best model. TINSKAL_impairment (tinnitus impairment), ADSL_depression (depression severity), TINSKAL_loudness, BSF_engagement and the timestamp features appear to exhibit non-monotonic relationships.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. SHAP analysis results for the best model (GBT, i = 7).

(A) Global feature importance based on the mean absolute magnitude of the SHAP values over all training instances. Values represent absolute change in log odds where higher values indicate higher feature importance. (B) Instance-individual SHAP values. A point represents the SHAP value for the feature depicted on the y-axis with respect to a single patient. The further afar a point from the vertical line at 0.0, the larger the attribution of the corresponding feature value to the model prediction. Vertically offset points depict regions of high density. Points are colored according to the actual feature value of the respective patient. (C) Combined SHAP feature attribution for all patients. Patients are ordered according to hierarchical clustering with complete linkage and k = 5. Blue horizontal lines depict the average sum of SHAP values of the cluster members.

https://doi.org/10.1371/journal.pone.0228037.g004

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. SHAP feature dependence plots.

The relationship between actual feature values (x-axis) and corresponding SHAP values (y-axis) is shown as green points. Positive SHAP values indicate an increased risk of decompensated tinnitus relative to the training set average, and vice versa. A locally weighted scatterplot smoothing (LOWESS) is placed as black curve on top to indicate the global trend. Gray bottom histograms represent the distribution of the actual feature values.

https://doi.org/10.1371/journal.pone.0228037.g005

How many features are really necessary for a good tinnitus-related distress prediction?

Fig 6A illustrates the increase of predictive performance of GBT models when a further feature is successively added to the feature space. A model that uses only the feature TINSKAL_impairment yields an AUC of 0.79±0.06. While adding ADSL_depression leads to an increase in AUC of 0.06, none of the remaining 24 features from the best model makes for a total improvement of more than 0.01, respectively. Only 3 features are necessary to achieve a mean AUC of 0.85, 8 features for a mean AUC of 0.87 and 15 features for a mean AUC of 0.89 (cf. Fig 6A). One potential reason might be a considerable degree of multicollinearity among groups of features. Fig 6B shows a network of 3 feature groups among the 26 features of the best model. For example, the features TINSKAL_impairment and TINSKAL_loudness are moderately correlated (ρ = 0.69), which leads to the question whether one of the pair might be omitted without a considerable loss in predictive power. The largest subgroup spanning 14 features involves descriptors of depression, perceived stress and reported physical health. In future work, an investigation of possible interaction effects among these moderately to strongly correlated features could be investigated, to better understand why all of them were selected and to determine whether some of them could be removed to achieve a better trade-off between model accuracy and complexity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 6. Marginal feature importance.

(A) Average cross-validation AUC (± SD) of a row refers to the performance of a GBT model trained on the feature subset that consists of the feature depicted on the y-axis label and all features of the above rows. The ordering of features is according to mean absolute SHAP value magnitude (cf. Fig 4A). (B) Network visualization illustrating 3 groups among the 26 selected features of the best model with high intra-group correlation. 8 features (predominantly SOZK features) without pairwise correlation of magnitude 0.5 or higher were dropped.

https://doi.org/10.1371/journal.pone.0228037.g006

Strengths and weaknesses of the study

To our knowledge, this is the first study that investigates the potential of machine learning for the identification of most important predictors from a wide range of features acquired before treatment for tinnitus compensation after treatment. The data-driven approach ensures that any potential predictor is included in the analysis. Another strength of the study is the internal validation of the classification models using cross-validation and hyperparameter tuning. Further, due to the selection of a variety of classification algorithm families, both linear as well as non-linear relationships between a feature and the outcome could be identified. A consequent limitation of this hypothesis-free approach is that the learned models could contain features that quantify the same or similar patient characteristics. For example, the best model in this study incorporated the two strongly correlated features ADSL_depression and BSF_anx_depression (anxious depressiveness score). Whilst the inclusion of both features was contributing to model performance, from a medical perspective a prediction model with only distinct features might be more beneficial. A pre-selection of features to avoid multicollinearity could be a direction for future work.

Finally, the exclusion of 2,701 out of 4,117 patients (65.6%) who did not complete all 10 questionnaires might led to a selection bias. Many patients spent more than one hour filling the questionnaire on a dedicated minicomputer and thus were more likely to abort the completion process, partly due to a gradual loss of motivation to give answers to a large number of questions, technical unfamiliarity with a computer, or interruptions by the hospital staff who demanded to continue with other baseline examinations. Completers were slightly younger than non-completers (mean age 49.8±12.2 vs. 51.7±13.8), were more likely to have the highest German school degree “Abitur” (48.2% vs. 42.0%) and had been suffering from tinnitus longer (> 5 years: 33.3% vs. 25.1%). A detailed comparison between completers and non-completers can be found in S3 Table. To our knowledge, no study has as yet investigated differential treatment responses in completers vs. non-completers; this can be explained by the absence of adequate information on the latter. We intend to investigate to what extent insights from completers can be used on subsamples of non-completers. For this, we can use the DIVA framework of Hielscher et al. [43]. However, psychological treatment approaches are only likely to benefit those reporting psychological distress prior to or associated with the tinnitus percept.