Model

For this study on heterogenous treatment effects, we used the same logistic regression model used in the original group comparison analysis, with the same covariates and primary outcome. However, a Bayesian approach to coefficient inference was taken. This was done in order to: (1) calculate predictions using the joint posterior predictive distribution over coefficients, which may produce better predictions than approaches relying on point estimates [21]; (2) incorporate uncertainty regarding which coefficients to include in the model using shrinkage priors (see later description). Posterior distributions were calculated using Hamiltonian Monte Carlo (HMC) [22], a Markov chain Monte Carlo method particularly used in the probabilistic programming language STAN [23]. So called horseshoe priors were used for covariate coefficients, which favors shrinking coefficients towards zero, encoding an a-priori belief that single covariates do not dominate the odds of smoking cessation.

The full Bayesian model is presented in Eq 1, with priors as recommended by [24]. In the equation, the covariates are as follows: G = group allocation, S = sex, Y = the number of years smoked, W = the number of cigarettes smoked weekly, D = dependence according to Fagerström’s test for nicotine dependence, U = the amount of snuff used. Coefficients (except for the intercept) are given normal priors with variance defined by a global parameter τ and local parameters λ_1–6. Half-Cauchy priors for τ and λ_1–6 are set so that τ shrinks all coefficients towards zero and λ_1–6 allow for individual coefficients to avoid this shrinkage. The scalar τ₀ is defined by , where p₀ represents the number of covariates that a-priori are believed to be effectively non-zero, K the number of covariates, and N the number of records in the dataset. In our case, we replaced categorical variables (S, D and U in Eq 1) with dummy variables representing the levels of the original variables. This resulted in 11 columns in the dataset (ie. K = 11), and we decided to set p₀ = 6, which equals the number of columns in our dataset divided by two (rounded up). The reasoning for this is that we do not believe that all covariates should be included in the model (or else we would not have included shrinkage priors), nor do we a-priori believe that none of the covariates should be included (or we would not have included them at all). Instead our best guess, prior to seeing any data, is to assume that half of them should remain. The inference is quite insensitive to the choice of p₀, especially when the number of records available far outweighs the number of covariates, as it does in our case. A detailed discussion about this insensitivity can be found in [24].

We used precompiled STAN code supplied in the R package rstanarm. R version 3.5.3 and rstanarm 2.18.2 was used for all inference. HMC was used to collect 2000 samples from the joint posterior distribution for each iteration of the cross-validation procedure. The procedure outlined in this Methods section took approximately 20 minutes to run on a 2017 MacBook Pro.

(1)

Eq 1. Logistic regression model for smoking cessation using horseshoe priors.

Predictive performance and cross-validation

In order to validly explore heterogeneous treatment effects, we have to make predictions that are accurate. If a model tells us that 15% of the individuals with access to a certain treatment will report prolonged smoking abstinence, then we also expect 15% of these individuals to actually have been abstinent, not 5% or 25%. We refer to a model’s capacity to produce accurate probabilities as the model’s predictive performance.

A visualization of the procedure we use to assess predictive performance using a technique known as cross-validation is shown in Fig 1. Each circle in Step 1 represents a participant in the RCT. In Step 2, we randomly split the participants into two subsets, one which contains 90% of the participants, and one that contains the remaining 10%. We call these two subsets the learning data (the larger subset) and the testing data (the smaller subset).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Cross-validation procedure to assess the performance of a prediction model.

https://doi.org/10.1371/journal.pone.0229637.g001

Using the learning data, we run HMC to infer posterior distributions of the covariates in Eq 1, and thereby define a model M in Step 3. We do not involve the participants from the testing data when creating this model, but rather we treat them as if they were never part of the RCT at all. In Step 4, we input each participant withheld in the testing data into M, one at a time, and output a predicted probability that this individual will quit smoking. In Step 5, we place each participant from the testing data into a bin. The first bin is labelled 0%-10%, the next is labeled 10%-20%, and so on until we get the final bin labeled 90%-100%. We place each individual in the bin that corresponds to the prediction that we made for them, eg. if M predicted that an individual had a 15% probability of prolonged abstinence, then we place this individual in the bin labelled 10%-20%.

We then go back and repeat the procedure from Step 2 onwards in such a manner that each participant will be part of the testing data exactly once. This results in each participant being placed in one of the ten bins in Step 5.

Predictive performance of the model can be assessed by looking into the bins. The average prediction made within each bin should correspond to the actual ratio of occurrence of the outcome being predicted. For instance, assuming that we on average predicted a 15% probability of prolonged abstinence for the participants that we placed in the 10%-20% bin, then we expect 15% of these participants to actually have reported prolonged abstinence. We can make these calculations since we have follow-up data from the RCT, and a model’s predictive performance can be judged by how well the predictions are aligned with the empirical findings.

Note, when this procedure is operationalized it is common to use locally weighted scatterplot smoothing (lowess) in order to visually show a model’s predictive performance. Doing so allows us to examine the model’s performance across a range of predicted values rather than within a set of predefined bins [25,26]. The visualization using bins is however still useful to understand the concept behind what we refer to as predictive performance, and one may think about using lowess as using an infinite number of bins.

Individual treatment effects

Estimates of individual level treatment effects will be calculated during the cross-validation procedure described earlier. In Step 4 (see Fig 1), we predict the probability of prolonged abstinence (Russel standard [20], no more than 5 cigarettes the past week) for each individual given both the intervention and control setting. By subtracting these two probabilities from each other we get the individual level effect of the intervention in terms of absolute difference in percentage points. For instance, if an individual is predicted to have a 15% probability of prolonged abstinence if they are given access to the intervention, and a 5% without access, then the probability of prolonged abstinence increases with 10 percentage points if the individual has access to the intervention. In this context, access to the intervention is defined as being offered the intervention, as this reflects the intervention group in the NEXit trial [11].

Predictive performance

Fig 2(a) illustrates the predictive performance of the model in Eq 1 on the withheld testing data during cross-validation. The black line represents a perfect model which predicts probabilities that are accurate to what has been observed empirically, the closer we get to this line the better. The red line represents the model learnt using the procedure outlined in Fig 1. The line is drawn using lowess, and as described earlier, this can be understood as using an infinite number of bins rather than choosing a fixed number. The model’s performance is close to the black line; thus, it tells us that when the model predicts that an individual has a certain probability of prolonged abstinence, then this is also the probability we should expect to see empirically.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. The red line will overlap the black line when predictions are accurate.

(a) Predictive performance of model in Eq 1 on the withheld testing data during cross-validation using Bayesian inference. (b) Predictive performance of the same model using maximum likelihood estimates.

https://doi.org/10.1371/journal.pone.0229637.g002

For reference, we ran the procedure in Fig 1 using the model used in the original publication [11], ie. the same model as in Eq 1 but using maximum likelihood estimates rather than Bayesian inference. Fig 2(b) visualizes the predictive performance of this model. It is clear that this model produces erroneous predictions, as the red line indicates that optimistic predictions are being made, for instance some cases are predicted to have a 60% probability of prolonged abstinence despite this not being observed empirically. This should serve as a reminder that while it is common to report maximum likelihood estimates of coefficients when contrasting groups in trials, it is not necessarily the case that these models have good predictive performance.

Conditional average treatment effects

In Fig 3(a)–3(e) we have plotted the absolute difference in predicted prolonged abstinence (no more than 5 cigarettes the past 8 weeks) against each of the covariates used in the model in Eq 1. In Fig 3(a) we can see that the absolute difference in predicted prolonged abstinence among female students are similar to those among male students, on average 10 percentage point absolute difference between being offered the intervention and not offered the intervention. Thus, there does not seem to be any heterogeneity in treatment effects with respect to sex. Similarly, there does not seem to be much heterogeneity with respect to the use of snuff (Fig 3(b)) or the number of years smoked (Fig 3(c)).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Heterogeneous treatment effects of the NEXit smoking cessation intervention.

(a-c) No heterogeneity is apparent with respect to sex, the use of snuff, nor the number of years smoked. (d-e) The NEXit intervention is less effective among those with stronger nicotine dependence and those who smoke more per week.

https://doi.org/10.1371/journal.pone.0229637.g003

Heterogeneity does however seem to exist when taking into consideration students’ nicotine dependence (Fagerström’s test for nicotine dependence), as can be seen in Fig 3(d). The absolute difference in predicted probability of prolonged abstinence is on average approximately 11 percentage points for those with low dependence, but closer to 9 percentage points for those with high dependence. The ranges are quite wide for these estimates, yet there does seem to exist a trend of reduced effect of the intervention as the severity of dependence increases. Put differently, the model predicts that the NEXit intervention is more effective among university students with low dependence to nicotine than among those with high dependence. This should however not be interpreted as a causal connection, thus we do not suggest that changing the nicotine dependence for a specific student will change the effectiveness of the intervention for this individual. Instead, the findings are observational in nature: among those with higher nicotine dependence the NEXit intervention was predicted to be less effective. A similar narrative can be given when looking at the number of cigarettes smoked per week (Fig 3(e)). As the number of cigarettes smoked increases, the absolute difference of predicted prolonged abstinence between intervention and control decreases, suggesting that the intervention is less effective among students who smoke more.

Limitations

The results of the analyses included herein should be understood under several conditions. The population under consideration are university students, thus our findings may not be generalizable to other subpopulations or the general population as a whole. Also, readers should recall that in all RCTs we are formally testing the randomization component rather than one treatment against another, as individuals may decide to adhere and engage with an intervention at their own discretion. Thus, while being randomized may sometimes be a good proxy to treatment, the estimates in this analysis should be strictly understood as comparing access to the intervention rather than use of it.

Conclusions

Using methods common in the machine learning field, we were able to get reliable estimates of heterogeneous treatment effects of the NEXit intervention. The findings suggest that subpopulations among university students were predicted to react differently to the NEXit intervention, and this was not evident from the original analyses when the intervention and control groups were compared directly. Thus, the heterogenous treatment effects were masked in the original analysis [11].

This study is the first of its kind with respect to text messaging interventions for smoking cessation among university students. We recommend other researchers to pursue similar studies in order to strengthen the evidence and corroborate results. It should be noted that in trials where no average treatment effect was found, the exploration of heterogenous treatment effects may reveal if there are subgroups for which the effect is being masked by the overall treatment effect. We also need a better understanding of heterogeneous treatment effects of other available interventions, eg. face-to-face and group counselling, in order to decide which individuals should be targeted by future intervention developments.