Computational modelling results

To validate and enrich the logistic regression analysis, we fitted a variety of pure MF (S2 and S3 Tables; S1, S2 and S3 Figs) and pure MS RL models (S4 Table; S1, S2 and S3 Figs) to each subject’s trial-by-trial choices using both fixed-effects (individual fits for each session) and mixed-effects (taking parameters of each subject as random effects across sessions) fitting procedures. As in previous studies [8, 24], we also considered a Hybrid model in which the best MF and MS models operated in parallel, and with their decision values being combined to determine choice probabilities (S5 Table; S1, S2 and S3 Figs). This uses a parameter (ω ∈ [0, 1) which specifies the relative weight of MS (ω ≃ 1) and MF (ω ≃ 0) control [8, 25]. Note that this resulting hybrid model is just the same as in the human studies that inspired it [8, 25], with the contribution from the MS system here being equivalent to that from the MB system in those studies. However, we retain the MS designation, since we cannot distinguish between various ways that model sensitivity might be arranged.

A careful examination of the data revealed that this Hybrid model required further refinement, leading us to develop a novel Hybrid+ model (see below), which accurately reproduced the strong influence of the previous trial reward on current choice.

The complexity-adjusted likelihoods of the models were compared to determine which best fit the behavioural data (S6 Table). In both subjects, choice behaviour was best explained by a combined MF and MS strategy, corroborating our logistic analysis. The best Hybrid model fit had a lower BIC score and a higher protected exceedance probability (PEP) than the best pure MF and best pure MS models. This winning approach combined the SARSA MF model (a better pure MF approach than Q-learning) without an eligibility trace parameter with the Forward₁ MS model (the best pure MS approach) for which the state-transition probabilities are assumed known from the beginning of the task (see Materials and methods for explanation of differences between each of the MF and MS models). Note that when only a pure MF method was allowed (rather than a hybrid combination), the best possible was SARSA with the inclusion of an eligibility trace. However, in the Hybrid model, the effect that the eligibility trace (or other parameters) is capturing could have been more parsimoniously encompassed in other components. Therefore, in fitting the Hybrid model, we again assessed which parameters were required for the MF aspect.

With regard to the balance between MF and MS control (Table 1), the mean of the ω hyperparameter was close to 90% in both subjects (different from 0 and 100% with p < 0.001 on sign tests in all sessions), in line with the MS dominance found in our regression analysis. The best-fit learning rate, α, was the same for both decision stages and was relatively high (close to 0.8 in both subjects, Table 1), probably due to the non-stationary and occasionally switching second-stage reward structure. On the other hand, first-stage choice was more deterministic than second-stage choice (β₁ > β₂, Table 1). Finally, the modeling also captured the small but positive tendency to repeat recently chosen options (parameter κ, Table 1).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Best fitting mixed-effects hyperparameters from the best models of each reinforcement learning approach.

https://doi.org/10.1371/journal.pcbi.1007944.t001

Model validation and simulation results

An important test for the models concerns whether they can accurately replicate the observed choice behaviour. Therefore, we used the best RL models for each learning strategy to simulate choice data on the same task, and then analysed the resulting simulated behaviour in the same way (Fig 2D–2F, S1, S2 and S3 Figs). These generated data confirmed the previously described differences between MF and MS RL and confirmed the qualitative validity of the best Hybrid model. However, they also highlighted important quantitative limitations. One of the most striking differences between the Hybrid model simulations and the observed data was the excess weight given to the most recent trial and, consequently, the discrepancies in the exponential decays (compare Fig 2B–2C with S3C Fig; decay constants for the reward main effect observed -0.78/-1.62 versus simulated -0.37/-0.36 for C/J; and reward × transition effect observed -0.94/-1.50 versus simulated -0.22/-0.17 for C/J).

This overweighting of the previous trial is akin to a sophisticated, MS, form of perseveration—i.e., a one step credit assignment influence on choice depending on reward and transition information of the last trial. For pure perseveration, the Q_Hybrid value of the previous first-stage choice is boosted, independent of the transition or reward. For this new effect, the influence on the Q_Hybrid value of the previous first-stage choice could depend on both reward and transition, with a factor dependent on the outcome level of the previous trial (L₁, L₂ or L₃, for high, medium and low) being added or subtracted according to whether the transition on the previous trial was common or rare, respectively. This way, a positive value (L > 0) denotes the strength of the reinforcement by reward, whereas a negative value (L < 0) quantifies the aversion for that particular reward. We call this new model Hybrid+. Given the structure of the environment, with non-overlapping second-stage reward statistics, it would have behooved the subjects to have relied just on this sort of MS perseveration rather than the progressive learning of second-stage values inherent to both the MF and MS methods in the Hybrid model. However, this does not describe their behavior accurately.

Comparisons (S6 Table) showed that Hybrid+ outperformed all the other RL accounts, including pure MS reasoning and the previous Hybrid model (all PEP values > 0.99). The extra parameters were justified according to the BIC, BIC_int and the exceedance probability. An important question is whether the original RL parameters remained stable after refitting all parameters. Indeed, very few changes were observed (Table 1). Critically, Hybrid+ captured behavioural characteristics that eluded Hybrid; the simulated choice data generated by Hybrid+ successfully captured not only the observed pattern of repeat probability at first-stage choice (Fig 2A and 2D), but also the profiles of both reward main effect (Fig 2B and 2E) and reward × transition interaction (Fig 2C and 2F) shown in the logistic regressions. Moreover, the best-fitted values of the additional parameters (L₁, L₂, L₃) revealed that high reward had a high reinforcement strength, but both medium and low reward had an aversive impact (Table 1), as previously noted. Thus, both model comparison and simulation results supported the validity of the new Hybrid+ account.

To examine the relationship between both descriptive and computational results, we explicitly compared coefficients obtained from the regression with the best fit Hybrid+ parameters for each subject and session. In addition, we also simulated new data from Hybrid+ using those parameters, performed logistic regression on these new data, and compared the resulting coefficients with the generating parameters. We found that stronger (i.e., more negative) reward × transition interaction effects were associated with greater MS ω Hybrid+ parameters (S5A Fig). We also observed a significant negative correlation between the first-stage inverse temperature parameters (lower values reflect stochasticity in choice) and the residuals from the regression (S5B Fig), and a positive correlation between both logistic and computational first-stage choice perseverance measures (S5C Fig). Taken together, these results demonstrate the strong correspondence between the regression analysis and computational modelling approaches.

Finally, to test the tradeoff and stability of MF vs. MS influences over time, as well as whether habits were forming with repeated experience of the task [3], we assessed the correlations between model parameters and their respective session number (S6 Fig). We found no significant relationship between session number and the main effect of previous reward, nor a reduction in the ω parameter across sessions (S6A and S6B Fig). On the other hand, the previous reward × transition interaction effect reduced across sessions in both subjects (S6C Fig).

This effect is likely caused by enhanced stochasticity, associated with a reduction in the first-stage inverse temperature parameter (S7 Fig), which would be expected to reduce the logistic regression weights. Another is the one-step MS contributions in the additional parameters of Hybrid+: we found the L₁ parameter decreased across sessions (S8 Fig), implying the influence of the high reward is reduced. Taken together, these results highlight the potential advantage of the computational modelling approach (over the regression approach) to decouple structurally different contributors (i.e., perseveration and MF/MS weight) that may otherwise be captured in a single regression weight (reward × transition).

Reaction time analysis

Various (and potentially conflicting) considerations might affect reaction times (RTs), including the expectation that changing a choice might take longer than repeating one, and the observation that first-stage choices can be planned as soon as the reward is revealed on the previous trial, whereas second-stage choices cannot, since they depend on the transition (albeit not in the conditions employed by [26]). We therefore analyzed RTs from a number of perspectives.

First-stage RTs were fast, and significantly shorter than second-stage RTs (first-stage: M ± SD = 499±201/647±191ms; second-stage 514±210/663±194ms for C/J; p < 0.001 for both two-sample t-tests). In both subjects, we found consistent RT differences between common and rare trials as a function of high or low rewards received (Fig 3). First-stage RTs were slower following high rewards obtained through rare versus common transitions, and low rewards obtained through common versus rare transitions. When considered alongside choice data (Fig 2A), these slower RTs occurred when the likelihood of choice switching was highest, and were just the situations when model sensitivity is most acute.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. The impact of both reward and transition information on first-stage choice reaction time.

(A) The averaged across sessions z-scored first-stage reaction time (RT) difference between previous common and previous rare trials as a function of reward on the previous trial (high z-scores indicate responses faster if previous transition was rare). Error bars depict SEM. (B-C) Multiple linear regression results on first-stage reaction time with the contributions of the reward main effect (B) and the reward × transition interaction term (C) from the five previous trials. Dots represent the fixed-effects coefficients for each session (coloured red when p < 0.05 and grey otherwise). Bar and error bar values correspond, respectively, to the mixed-effect coefficients and their SE. Dashed lines illustrate the exponential best fit on the mean fixed-effects coefficients of each trial into the past. ** α = 0.01 and * α = 0.05 in two-tailed one sample t-test with null-hypothesis mean equal to zero.

https://doi.org/10.1371/journal.pcbi.1007944.g003

We also performed multiple linear regression on first-stage RT (Fig 3 and S7 Table). Despite some similarities with the approach used for choice behaviour, we note that in our predictive model for first-stage RT the effects of previous reward, transition and reward × transition do not include the interaction with previous first-stage choice information. In both subjects, the RT was modulated over multiple past trials by both the reward (Fig 3B) and the reward × transition (Fig 3C). Although the interaction term was similar between subjects, the effect of previous reward differed: a high reward (independent of transition) on trial t−1 led to faster/slower RTs in subject J/C, respectively (also seen in Fig 3A). These t − 1 RT differences might be explained by differential trial lengths between subjects (delays and task timings were shorter for C than J to maintain motivation) or by different speed/accuracy strategies following high rewards.

A different index of response vigour and task motivation is how quickly subjects acquire fixation (fixRT) based on previous reward. Both subjects exhibited a main effect of reward on fixRT (i.e., faster fixation following better rewards; S9A Fig). No effect of the reward × transition was found (S9B Fig), suggesting that fixRT reflected only a MF influence, whereas RT was influenced by both MF and MS systems.

Conclusion

In conclusion, we have been able to show clear evidence of combined MF and MS RL behaviour in non-human primates. Our computational analyses of choice suggested an enriched picture of the combination; the analyses of RTs showed that they are subject to different influences. Future studies focusing on the neural signals may uncover the biological substrates of these computational mechanisms.

Subjects and experimental apparatus

Two rhesus monkeys Macaca mullata were used as subjects: subject C weighing 8 Kg; and subject J weighing 11 Kg. Daily fluid intake was regulated to maintain motivation on the task. During the experiment, subjects were seated in a primate chair inside a darkened room with their heads fixed and facing a 19-inch computer screen (60Hz video refresh rate) positioned 62 cm from the subject’s eyes. Each subject’s eye position and pupil dilation was monitored with an infrared eye tracking system having a sampling rate of 240 Hz (ISCAN ETL-200). Both subjects indicated their choice by moving a joystick with a left arm movement towards one of three possible locations (C: left, right and down; J: left, right and up). The reward (C: cranberry juice diluted to one-fourth with water; J: apple juice diluted to one half with water) was provided by a spout positioned in front of the subject’s mouth and delivered at a constant flow-rate using a peristaltic pump (Ismatec IPC). We used Monkeylogic software (http://www.monkeylogic.net/): to control the presentation of stimuli and task contingencies; to generate timestamps of behaviourally-relevant events; and to acquire joystick as well as eye data (1000 Hz of analog data acquisition). All visual stimuli used were the same across sessions for both subjects, and were presented at pre-determined degrees of visual angle (see below). Six decision option pictures were chosen from a stimulus database, reduced in size and modified through a custom-made image processing algorithm to make the average luminance equivalent for all. Similarly, the background colours used (grey, violet and brown) were tested with a luminance meter and adjusted accordingly. Finally, three stimuli used as secondary reinforcers were generated as different spatial combinations of the same number of dark pixels in a white background, also to assure luminance equality. All experimental procedures were approved by the UCL Local Ethical Procedures Committee and the UK Home Office (PPL Number 70/8842), and carried out in accordance with the UK Animals (Scientific Procedures) Act.

Task: Design and timeline

Subjects performed a two-stage Markov decision task (see Fig 1), similar to the one used in a previous human study [8] that was designed to detect simultaneous signatures of MF and MS systems as they concurrently learn. In brief, two decisions had to be made before the subject received an outcome (see Fig 1A). The first-stage state was represented by a grey background and the choice was between two options presented as pictures (the same fixed set of pictures was used throughout the entire task). Each of these first-stage choices could lead to either a common (70% transition probability) or rare (30% transition probability) second-stage state, represented by different background colours (brown and violet). This state-transition structure was kept fixed throughout the experiment. In the second-stage, another two-option choice between stimuli was required and it was reinforced according to one of three different levels of outcome (see Fig 1B). Importantly, to encourage learning, each of the four second stage options had independent reward structures according to a form of random walk that was sampled afresh on each session (see below). In both decision stages, each choice option (or each presented stimuli) could randomly assume one of three possible locations (C: left, right and down; J: left, right and up). No significant preference for any first-stage stimulus across sessions was found (both one-sample t-tests with p > 0.05) but, given the three physically possible actions, small side biases were observed (both one-way ANOVAs with p < 0.01). Fifteen percent of the trials were forced, i.e., where only one stimulus was presented—these could be at either the first or second-stage. Unless stated otherwise, such forced trials were not included in the data analysis. The trial type sequence was randomly generated at the start of the session and was followed even after error trials. Error types included trials with no choice, no eye fixation, eye fixation break, early joystick response, joystick not centred before choice or movement towards a location not available. Error trials resulted in time-outs for the subjects. Unless otherwise specified, we excluded such trials from the data analysis (C: M = 5%; J: M = 8%).

The outcome (referred to as “Reward”) could assume one of three categorical levels, defined according to the amount of juice delivered (determined by the time the juice pump was on) and a specific delay (in addition to a fixed 500 ms delay common to all outcome levels) before juice delivery. Therefore, the reward could be: high (big reward and no delay), medium (small reward and small delay) or low (no reward and big delay). The precise reward amounts for big and small rewards were tailored for each subject to ensure that they received their daily fluid allotment over the course of the experimental sessions. Consequently, the duration for which the reward pump was active (and hence the magnitude of delivered rewards) differed slightly between the two subjects. Furthermore, instead of a fixed reward amount, big and small rewards corresponded to non-overlapping time intervals (C: high reward ranged on average from 682 to 962 ms and medium reward ranged on average from 117 to 390 ms; J: high reward ranged on average from 976 to 1257 ms and medium reward level ranged on average from 507 to 826 ms) of juice delivery where a small Gaussian drift (mean/standard deviation of 0/200 ms for high reward and 0/100 ms for medium reward) was added. This was used not only to promote constant valuation of the reward amount, but also to help the computational model fitting procedure. The additional specific delay periods were fixed throughout the experiment but varied across subjects (C: 750 ms for small delay and 2500 ms for big delay; J: 1500 ms for small delay and 4000 ms for big delay). Importantly, for each of the second-stage pictures the outcome level remained the same for a minimum number of trials (a uniformly distributed pseudorandom integer between 5 and 9) and then, either stayed in the same level (with one-third probability) or changed randomly to one of the other two possible outcome levels. Three different stimuli were used as secondary reinforcers, providing feedback for each of the three outcome levels. Both subjects had prior classical conditioning training with these stimuli (see Fig 1B), with the above mentioned reward magnitude ranges and delays for each outcome level used in the experiment being respected.

The sequence of events in the behavioural task is shown in Fig 1A. Each trial started with the presentation of a grey background (start epoch). A central square fixation cue 0.4° in width then appeared after a random interval of 200-500 ms. After this, subjects were required to keep the joystick in the centre position as well as to maintain eye fixation within 3.4° (C) or 2.8° (J) of the cue for a 500 ms (C) or 750 ms (J) period (fixation epoch). Then, the fixation cue was removed and two stimuli (5° in size) appeared at 7° away from fixation in the available locations (choice epoch). During the task, in the absence of a fixation cue, the animal was free to look around. The maximum time allowed for eye fixation as well as response with the joystick was 5000 ms for both choice stages. After a choice was made, the non-selected stimulus was removed and the background color changed according to the second-stage state to which the transition had occurred (transition epoch). After 500 ms, the stimulus selected in the first-stage was removed from the screen. Similar fixation and choice epochs were used for the second-stage. Once the choice had been made in the second-stage, the non selected stimulus was removed and the selected one remained for 750 ms before the secondary reinforcer stimulus (5° square) appeared at the center of the screen (pre-feedback epoch). Following its appearance, the feedback stimulus remained present for 750 ms. After the removal from the screen of the secondary reinforcer, a fixed 500 ms delay period occurred before either the reward delivery (for high reward) or both small and big additional delays started (for both medium and low rewards, respectively). Therefore, a total of 1250 ms was the minimum time from the secondary reinforcer presentation to the delivery of any juice (feedback epoch). The inter-trial period duration was 1500 ms (ITI epoch).

Behavioural analysis

All analyses were conducted using MATLAB R2014b (MathWorks). The data required to replicate all reported findings (together with a glossary for details) is provided as Supporting Information files (S1 and S2 Datasets). Statistical significance was assessed at α = 0.05, unless otherwise stated. Behavioural variables were defined as: C is first-stage choice (1 = car picture, 0 = watering can picture); R is outcome level (referred to as “Reward”; assumed as continuous, with low = 1, medium = 2, high = 3); and T is transition (rare = 1, common = 0). In regressions, these variables were mean centred, and continuous variables were also scaled by dividing them by twice their standard deviations so that the magnitudes of regression coefficients could be directly compared [51]. To quantify the factors predicting first-stage choice at trial t, C_t a multiple logistic regression was used in which the predictors included information from the last 5 trials, i ∈ {1, 2, 3, 4, 5}, and were: Const (constant term) captured any potential first-stage picture bias; C_t−i, modelling a potential independent tendency to stick with the same option; R_t−i, T_t−i, R_t−i × T_t−i, measuring any potential preference in first-stage picture choice given the previous reward, the previous transitions and the interaction effect of both, respectively; R_t−i × C_t−i, T_t−i × C_t−i, R_t−i × T_t−i × C_t−i, were the predictors of interest which quantified the main effects of reward, transition and the reward × transition interaction effect, respectively. Although unexpected, both subjects showed a small but significant main effect of transition (S4A Fig) but a similar effect was present in the simulations derived from our best RL model (S4B Fig) suggesting that correlations within the task design and reward structure may underlie this effect. Linear hypothesis testing on the vector of regression coefficients (performed for each individual session in the fixed-effects; and using the estimated mixed effects for each predictor) was performed to test either if more than one coefficient or a difference between coefficients was significantly different from zero. First-stage RT was defined as the time from first-stage stimuli presentation to joystick movement towards the specified location (all side locations with the same target radius). For each subject and session, first-stage RT were independently log transformed and z-scored for the three possible side responses (this was done as side RT differences were with both one-way ANOVAs with p < 0.001). Data points greater than three times the SDs from the individual means were removed. The first-stage eye fixation time (fixRT) was defined as the time from fixation cue presentation to the first time the x and y position eye position waswithin that subject’s required fixation radius. The raw data was then log transformed and z-scored. To determine the effect of behavioural variables on first-stage RT and fixRT, we performed a multiple linear regression analysis on the current trial t log transformed and z-scored first- stage RT/fixRT, using as predictors: F_t, used to model (linearly-increasing) fatigue by counting the trials in the session; R_t−i, T_t−i and R_t−i× T_t−i were the predictors of interest which quantified the main effect of reward, the main effect of transition and the reward × transition interaction effect, respectively.

Regression analysis fitting

Fixed-effects (fitting the regression models individually to each session) and mixed-effects (assuming regression coefficients to be random effects across sessions) analyses were performed for each subject. Fixed-effects fitting was performed using a generalized linear model regression package (glmfit in MATLAB with: a binomial distribution and the logit link function for logistic regressions, a normal distribution and the identity link function for linear regressions), and the statistical importance of each predictor’s estimates was assessed by both the p-values obtained from each session as well as their distribution across sessions (two-tailed one-sample t-test for a mean of 0 and unknown variance). Mixed-effects fitting was achieved with either a non-linear model with a stochastic approximation expectation-maximization method for logistic regression (nlmefitsa in MATLAB with importance sampling for approximating the loglikelihood) or a linear model method for the RTs (filme in MATLAB). The standard errors for the coefficient estimates as well as their 95% confidence intervals (CI) were reported.

Computational modelling

We fitted choice behaviour in the task in a similar manner to previous human studies [52], assessing three different reinforcement learning approaches: MF learning, MS learning and a hybrid strategy combining the decision values of both [8, 24]. The task consists of three states (first stage: A; second stage: B and C), each with two actions (x and y). Importantly, we assume that the subjects already know that the action corresponds to the choice of a picture belonging to the respective state (rather than the side, given their very modest side biases). The main goal is to learn to compute a state-action value function,Q(s,a), mapping each state-action pair to its expected future value. On trial t, the first-stage state (always s_A) is denoted by s_1,t, the second-stage state by s_2,t, the first and second-stage actions by a_1,t and a_2,t and the first and second-stage rewards as r_1,t (always zero) and r_2,t. For the model fitting r_2,t corresponded to the amount of juice delivered at trial t divided by the maximum amount of juice obtained by the subject within the entire respective session. Note that because the delay amounts were fixed for outcome level and hence contained no variability across trials, they were not explicitly included in the model-fitting.

In MF-RL the value for the visited state-action pair at each stage i and trial t, Q(s_i,t, a_i,t), is updated based on the temporal difference prediction error, δ_i,t, which sums the actual reward r_i,t and the difference between predictions at successive states s_i+1,t and s_i,t. For the first-stage choice, r_1,t = 0 and δ_1,t is driven by the second-stage value Q(s_2,t, a_2,t). On the other hand, at second-stage there is no further value apart from the immediate reward, r_2,t, and ultimately the start of a new trial. For convenience, we create a fictitious state, s_3,t, and action, a_3,t, for which Q(s_3,t, a_3,t) is always 0. Two different MF-RL models were used to fit behaviour: the SARSA variant of temporal difference learning [53], which has previously been observed in non-human primates [18]; and the Q-learning model, as described in rodents [19].

In SARSA, state s_i+1,t is evaluated according to the actual action a_i+1,t that the subject selects. This makes the prediction error: (1) By contrast, in Q-learning, the state is evaluated based on what the subject believes to be the best action available there, independent of the policy being followed. This makes the prediction error: (2) Either of these errors in the estimate drives learning by correcting the respective MF prediction through the following update rule: (3) where α_i is the learning rate at stage i, and was fit to the observed behaviour. In previous work, different learning rates were found for the stages [8]. Given the two-stage design of the task, the model also permits an additional stage-skipping update of first-stage values by having an eligibility trace parameter λ [1], which connects the two stages and allows the reward prediction error at the second-stage to influence first-stage values: (4) The parameter λ was also fit to the observed behaviour. Consistent with the episodic structure of the task (with an explicit inter-trial epoch), it is assumed that eligibility does not carry over from trial to trial.

In MS-RL, the agent not only maps state-action pairs to a probability distribution over the subsequent state but also learns the immediate reward values for each state. More specifically, it requires knowledge of the probabilities with which each first-stage action leads to each second-stage state, as well as learning the expected reward associated with each second-stage actions. The MS second-stage state-action values Q_MS(s_2,t, a_2,t) are just estimates of the immediate reward r_2,t, and so coincide with MF values there (since Q(s_3,t,a_3,t) = 0). We define Q_MS = Q_MF at those states. On the other hand, the first-stage action values Q_MS(s_1,t,a_1,t) differ and are computed by weighting the estimates on trial t of the rewards by the appropriate probabilities: (5) Different approaches to estimating the state-transition probabilities give rise to three different MS models, designated here as Forward₁, Forward₂ and Forward₃. In the first model, the agent had explicit knowledge of the correct state-transition probabilities, P = {0.3, 0.7}. The extensive training of both subjects prior to this experiment makes this plausible. In the second model, agents were assumed to map action-state pairs a₁, s₂ to transition probabilities, P = {0.3, 0.7}, by counting whether they had more often encountered transitions a₁ = 1, s_B and a₁ = 2, s_B or transitions a₁ = 1, s_C and a₁ = 2, s_B and concluding that the more frequent category corresponds to p = 0.7. This latter model corresponds to the one used in the modelling of the original two-step task study [8]. Finally, in the Forward₃ model the agent incrementally learn the transition structure by performing an hypothesis test between p = {0.3, 0.7} versus p = {0.5, 0.5} with an additional parameter (ζ) modelling the weight given to each of these models. In both Forward₂ and Forward₃ the data for the hypothesis test was reset at the start of every session.

Finally, a so-called Hybrid model assumes that first-stage choices are computed as a weighted sum of the state-action values from MF and MS learning systems: (6) where ω is a weighting parameter that determines the relative contribution of MS and MF values. When ω = 0 the model reflects pure MF control; when ω = 1, it reflects pure MS control. For convenience the hybrid model was constructed using the best fitting MF (SARSA model) and MS (Forward₁) models, given the computational burden of fitting all possible combinations simultaneously.

A careful examination of the data revealed that the original hybrid model required further refinement in order to reproduce more accurately the strong influence of the previous trial on the present one. In this new Hybrid+ model, the value of the chosen (a_1,t) or unchosen (a ≠ a_1,t) first-stage action was boosted or suppressed as a function of whether the state-transition (Trans) observed at trial t was common or rare and the level of the outcome achieved (Rew). Algorithmically, after the previously described Q_{HY B} calculation (Eq 6) an additional boost (or decrease) occurred according to: and where there are separate parameters L_j for each outcome level which can be positive or negative, expressing support or opposition for that particular outcome level. This extra factor can be seen as a MF implementation of a MS effect [15]—MF, since it depends on an effect of the past trial rather than an assessment of a future one; MS, since it includes a one-step version of the interaction to which MS reasoning leads.

For any of the above reinforcement learning strategies, actions were assumed to be stochastic and chosen for each stage according to action probabilities determined by the respective Q-action values: (7) where β_i is the inverse temperature parameter (distinct inverse temperatures are considered for each stage) controlling the determinism of the choices, and so capturing noise and exploration (for β_i = 0 choices are fully random and for β_i = ∞, choices are fully deterministic in the sense that higher-valued options are always preferred). rep(a) is an indicator variable coding whether the current choice is the same as the one chosen on the previous visit to the same state, with κ_i being a further parameter that captures choice perseveration (κ_i > 0) or switching (κ_i < 0) [23], again with the possibility of distinct values for first and second-stage choices.

In the most general form, the conventional Hybrid model involved a total of eight free parameters (θ = {α₁, α₂, β₁, β₂, κ₁, κ₂, λ, ω}), nesting pure MS (ω = 1, with arbitrary α₁ and λ) and MF (ω = 0) learning as special cases. The Hybrid+ model involved three additional parameters L₁, L₂, L₃. We also generated several simpler variants of these models by allowing α₁ = α₂, β₁ = β₂, κ₁ = κ₂, κ₁ = 0, κ₂ = 0 and λ = 0. All parameters were fixed within a session, but could vary across sessions.

Model fitting procedures

Two forms of log-likelihood maximisation were used to fit the computational models to each subject’s choice behaviour and estimate their free parameters. The first approach was a so-called fixed-effects analysis, maximizing the likelihood with respect to the parameters separately for each session. The second approach was a mixed-effects analysis, assuming, for each subject, parameters to be random effects across sessions. This implied maximizing the likelihood with respect to a characterization of empirical priors over the parameters (based on Gaussian distributions for the vector of parameters h; enforcing constraints on the parameters: 0 < α_i < 1; β_i > 0; 0 < λ < 1; and 0 < ω < 1 by transforming samples from the Gaussian distributions using log and sigmoid transforms). In this scheme, one calculates approximate posterior distributions over the parameters for each session by combining these priors with the likelihoods. The effect of the prior is to regularize and stabilize estimates, particularly when the parameters are not well constrained by the data in particular sessions. The mixed-effects procedures used are identical to those described by [52], but for completeness are detailed here.

The hyperparameters of the prior distribution θ, which consist of a prior mean μ and a prior standard deviation σ, were set to the maximum likelihood estimates (ML) for all N sessions, using empirical Bayes: (8) where comprised all the actions (including first-stage and second-stage) by all the N sessions. For first and second-stage actions taken in all the T trials of the session it was assumed that .

The above maximization was achieved by Expectation–Maximization (EM) [54]. At the k^th iteration of the E-step of the algorithm, a Laplacian approximation to the individual posterior distributions of model parameters was used and the maximum a posteriori estimate m_i of the parameters for each session i was found: (9) (10) where denotes a normal distribution over h with mean and standard deviation derived from the diagonals of the inverse Hessian matrix of the posterior at its maximum m^(k). In order to increase the chance of finding a good maximum a posteriori value, the largest value out of 101 separate optimizations was used, one starting from the best value on the previous iteration (or the output of the fixed effects analysis for the first iteration), and 100 more using random starting points. Optimization was performed using MATLAB’s parallel processing toolbox and fminunc function.

In the M-step, the hyperparameters θ were estimated by setting the prior distribution mean μ and standard deviation σ to: (11) (12) To help convergence, the algorithm was initialised with a prior mean and variance that corresponded to the 25% trimmed mean and variance of the parameters initially obtained with the maximum likelihood fixed-effects fit. The E and M steps were then repeated until the changes in the estimates between two E-steps were 0.005, signifying convergence. Once the subject’s maximum likelihood prior parameters had converged, a final E-step was performed to determine the maximum a posteriori parameters for each session. All model fitting procedures were verified on surrogate generated data.

Model comparison and validation procedures

We fit the two pure MF (ω = 0), the three pure MB (ω = 1), the Hybrid (with ω a free parameter) and the Hybrid+ models, and then sought to determine the model that was best supported by the behavioural data. To note that rather than fitting all possible algorithmic combinations, the Hybrid model variants tested just combined the best MF, the SARSA, and the best MB, the forward₁ algorithms. Regarding the Hybrid+ model fit, it used the already best fitting Hybrid model variant for each subject but with all free parameters, including the extra three parameters, estimated afresh.

For the fixed-effects analyses, the Bayesian information criterion, BIC, [55] based on the negative log-likelihood, was determined for each algorithm tested. Since the Hybrid algorithm nested MF and MB algorithms, likelihood-ratio tests (LRTs) were also used to compare it against the other learning approaches.

For the mixed-effects analyses, model comparison was achieved by computing, for each model and given all the observed first and second-stage choices , the posterior log likelihood . Because each of the models tested are equally likely a priori, the model log likelihood is the measure to examine. To approximate this quantity at the subject-level and at the individual session-level, we followed a similar approach as the one described in [52]. The approximation at the subject-level was obtained via [56]: (13) where is the total number of trials performed by the subject in all sessions, and is the number of prior parameters fitted (mean and variance for each parameter). The subscript “int” to the BIC was added because the log likelihood is not the sum of individual likelihoods, but the sum of integrals over the individual session’s parameters approximated via sampling: (14) where K = 1000 indicates the number of samples drawn from the empirical prior distribution . This ensures comparison of not how well a particular model fits the data when its parameters are optimised, but rather how well it fits on average under the random effects empirical prior over the parameters.

In addition to comparing the BIC_int, which is akin to a likelihood ratio test, the mixed-effects model comparison also included the protected exceedance probability [57] of each model being more likely than any of the other models tested. The computation of this latter measure involved the model likelihood obtained in the fitting of the maximum a posteriori estimates for each session of a given subject and the calculations were performed using the spm_BMS function contained in SPM12 (http://www.fil.ion.ucl.ac.uk/spm/). To assess how well each model performed on subject’s data, the overall predictive probability for all choices in all trials and sessions, = , was calculated and tested, according to a binomial test, whether this was greater than chance.

We further tested the best-fitting models and their respective distribution of parameter priors, , by using them to simulate choice data for each subject (100 simulation runs for each session) on the task respecting the exact same reward structures as present in the behavioural data. We then performed the same descriptive and logistic analysis as used for choice behaviour. Finally, Pearson’s linear correlation coefficients were used to assess the relation between the behavioural computational modelling estimates (observed or simulated) and other variables, such as the logistic regression coefficients.