Introduction

A basic assumption of many models of adaptive behavior, is that the value of primary rewards are modulated by their capacity to rectify future homeostatic deficits [1, 2]. Compatible with this notion, deprivation-induced hypoglycaemia increases willingness to work for food in rats and humans [3], as well as the subjectively reported pleasure [2]. Dopamine is a neurotransmitter that plays a key role in signalling reward [4] and is involved in behavioural reinforcement, learning, and motivation [5, 6]. Via meso-cortical and mesolimbic dopaminergic projections, synaptic dopamine release modulates the plasticity of cortico-striatal networks and thereby sculpts behavioural policies according to their reward contingencies [4, 7]. Patterns of phasic dopaminergic firing have been demonstrated to follow closely the principles of reinforcement learning, encoding the errors in the prediction of reward [6, 8–10]. Reward prediction error (RPE) signals appear commensurate with the economic construct of marginal utility, defined as the additional utility obtained through additional units of consumption, where utility is a subjective value inferred from choice [7, 11, 12].

Although animals are motivated by a homeostatic deficit of thirst or hunger, homeostatic states are rarely considered as relevant modulators of dopaminergic signalling of reward prediction errors. In typical paradigms involving cumulative consumption, the homeostatic deficit gradually diminishes as the animal plays for consumption of water or sugar-containing juice. Eventually, the animal rejects further play, presumably because the marginal utility of consumption diminished to a point of indifference or even aversion. Interestingly, a recent electrophysiology study in rats, demonstrated that oral consumption of sodium solution causes phasic dopaminergic signals in the nucleus accumbens, that are modulated by sodium depletion [13].

There is now growing evidence for a multifaceted interface between dopamine mediated reward-signalling and the systems underpinning energy homeostasis. Firstly, dopamine neurons in the ventral tegmental area (VTA) express a suite of receptors targeted by energy-reporting hormones ghrelin, insulin, amylin, leptin and Glucagon Like Peptide 1 (GLP-1) [14, 15]. This provides numerous degrees of freedom for flexibly interfacing between homeostatic state and reward signalling. Although hormonal modulations of phasic dopamine are yet to be fully scrutinised, there is emerging evidence that circulating factors do indeed modulate its magnitude. For instance, amylin, a hormone co-released with insulin, acts on the VTA to reduce phasic dopamine release in its mesolimbic projection sites [16]. In terms of neuronal input, there are many such opportunities for the appetitive control of dopamine mediated signalling.

Appetitive control can be delineated into three interacting systems [17]. The first system generates a negative valence signal which involves activity of the Agouti-related peptide (AgRP) neurons of the arcuate nucleus of the hypothalamus (ARC). Activity of ARC_AgRP neurons reports on energy deficits, inhibits energy expenditure, and regulates glucose metabolism [18–21]. ARC neurons that contain peptide products of pro-opiomelanocortin (POMC) form an opponent code compared with ARC_AgRP neurons. The balance between the two neuronal ARC sub-populations putatively encodes the value of near-term energetic states, becoming rapidly modulated just prior to food consumption [22]. The second system codes positive valence signals and consists of circuits involving the lateral hypothalamus (LH). It is linked to positively reinforcing consummatory behaviours via its GABAergic projections to VTA dopamine neurons [23] assumed to trigger positive feedback to keep consumption going during feeding bouts. The third valuation system involves calcitonin gene-related protein (CGRP)-expressing neurons in the parabrachial nuclei (PBN) that potently suppress eating when activated, but do not increase food intake when inhibited. PBN_CGRP neurons are activated by signals associated with food intake, and they provide a signal of satiety that has negative valence when strongly activated [24]. The PBN has been characterised as a hedonic hotspot, the modulation of which by either GABA or Benzodiazepines potently modulates experienced reward [25]; ARC_AgRP neurons GABA-ergically inhibit PBN neurons, thus stimuli predicting glucose consumption should inhibit ARC_AgRP, releasing the PBN from inhibition [26]. Further, hormones related to hunger and feeding (GLP-1 & leptin) modulate PBN activity and subsequent behaviour [27, 28]. Of note, these three valuation systems all project to and modulate the dopaminergic neurons in the ventral tegmental area (VTA_DA). The interface between these hypothalamic-brainstem networks and the VTA_DA, is arguably the most important interface for mediating the dialogue between energy homeostasis and value computation.

While most evidence for encoding of RPEs is obtained under homeostatic deprivation, the modulation of RPE signalling triggered by physiological fluctuations in glucose availability (glycaemic flux) remains yet to be characterised in the human brain. This begs the questions, how are RPE signals modulated by these subcortical circuits that integrate, evaluate, and predict energy-homeostatic states? We hypothesize that glucose fluctuations above and below average levels of serum glucose, will down and up modulate RPE responses in these hypothalamic-brainstem networks. To test these hypotheses, we acquired a large volume of fMRI data in five participants during a simple Pavlovian cue-conditioning task, while their serum glucose was systematically manipulated.

Methods

Experimental procedure

The experimental design constituted a single-blinded, randomised control trial, with repeated measures crossover-design. On four separate days, subjects fasted for a minimum of twelve hours before testing. Compliance with the fasting instruction was based both on trust, and on the understanding that we would be able to detect if participants had not fasted via blood tests. Any participant that was not in the hypoglycemic range (defined here as <6mmol/L) at the start of the experiment would be assumed to have not fasted, and the session would be aborted. This was not necessary for any participant. At the beginning of an experimental session, subjects ingested either a hi-glucose (75 g, 300 kcal) or lo-glucose preload (10 g, 40 kcal) diluted to 100 ml with a non-caloric lemon juice, used in order to mask the taste of the glucose. The lo-glucose preload resulted in glucose ascending during the fMRI acquisition period, due to the consumption of glucose, whereas during the same period after the hi-glucose preload, glucose levels descended (Fig 1B). The preload conditions are thus referred to as ascending and descending conditions. The temporal order of the conditions was randomised within subjects, with each condition being performed twice. Both preloads were anecdotally reported by independent samplers to be highly palatable. Each delivery of glucose reward was 0.4ml, corresponding to 0.3 g of glucose (1.2 kcal) per delivery.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Experimental design, glucose trajectories, and expected reward signals.

a, participants were presented with the Cue_onset (grey fixation cross) for 1-3s after which either Cue_high (blue cross) or Cue_low (brown cross) is presented with a probability of 0.5 each. Cue_high signalled a high probability (0.8) of glucose delivery and a low probability (0.2) of water delivery. Cue_low signalled a low probability of glucose (0.2) and a high probability of water (0.8). The 0.4ml of the liquid were delivered over 2.5 seconds, followed by 10-15s wait period and a Cue_swallow that cued the subject to swallow (here, purple) which lasted for 5s. All jitters are uniformly distributed within the ranges specified. b, serum glucose trajectories for the high and low glucose preload conditions. Grey shading indicates the period of fMRI acquisition for a single session. The different line plots indicate different sessions for all subjects. Glucose levels ascend during the fMRI acquisition period in the lo-glucose condition, and descend in the hi-glucose condition. c, graph depicts the objective reward expectations, expressed as the expected value in ml glucose, and the perturbation of these expectations under the onset of the experimental cues and outcomes. Note that reward expectations are updated three times per trial: at the onset of the Cue_onset; at the onset of Cue_high or Cue_low; at the onset of Outcome_glucose or Outcome_water. d, illustrates simulated BOLD responses to RPE signals resulting from the updated reward expectations shown in c, generated by convolving the canonical hemodynamic response function with the RPE stick functions evoked by changes to the reward expectations.

https://doi.org/10.1371/journal.pone.0243899.g001

Results

Cue_onset induced brain activity

An RPE signal should respond to the Cue_onset, with an error signal that signals the expected value of glucose reward for the whole trial [11]. Computing the main effect of this regressor, this was found to evoke an increase in activity in VTA bilaterally (Fig 2A). Thus cue-induced VTA activation is consistent with existing evidence of VTA signalling RPEs [35–37]. Cue_onset also led to the deactivation of the postcentral gyrus (primary somatosensory cortex), mediodorsal thalamus, and the striatum [whole brain, uncorrected p < 0.001] (not shown).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Statistical parametric maps of main effects of trial onset, positive and negative RPEs.

a, main effect of Cue_onset, which reflects an RPE following the mean reward expectation for the whole trial, revealed activity in VTA bilaterally (β = 2.77) (R: [4–17–20] and L: [–8–17–20], FWE SVC). Further this revealed deactivation of precentral gyrus (primary somatosensory cortex), mediodorsal thalamus, and striatum (FWE whole brain, not shown). Lower panel shows fitted response for the Cue_onset event within the same region. b, main effect of RPE_pos revealed activity in left lateral caudate [β = 1.21; coordinates -8 4 7; FWE SVC]. Lower panel shows fitted response for the RPE_pos contrast within the region. The colours of time courses follow the same meaning as shown in Fig 1C and 1D. c, main effect of RPE_neg revealed bilateral activity in caudate (L: -11–2 13; R: 10 7 1; β = 12.2) medial dorsal thalamic nucleus [7, –2, 22], and lateral insula [43, –2, –17] (all FWE). Lower panel shows fitted response for the RPE_neg contrast within the region. All fitted responses were generated by convolving the canonical hemodynamic response function with the RPE stick function multiplied by their respective beta-values extracted from the local maxima of the ROI in units of percent signal change. Again, the time courses follow the same meaning as shown in Fig 1C and 1D. Error bars show standard errors of the mean.

https://doi.org/10.1371/journal.pone.0243899.g002

Modulation of task-related brain activity by glycaemic state

We were interested to identify changes in RPE signalling over time as serum glucose either ascended or descended. A bilateral cluster, including the parabrachial nuclei (PBN), showed a modulation of the regional neural responses to RPEs by the glycaemic state dynamics (Fig 4A). Higher levels of serum glucose amplified the response to RPE_pos in the PBN region (Fig 3B). The main effect of RPE_neg*state, which models the interaction between RPE_neg and state, did not yield any significant results in any ROI, or in exploratory analyses using uncorrected thresholds, in positive or negative contrasts. When considering both ascending and descending serum glucose fluctuations together, there was no detectable region where the RPE_neg signal was either positively or negatively modulated by serum glucose. Brain responses to Cue_onset were also not altered by glycaemic state.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Statistical parametric maps of RPE_pos*state and fitted responses over varying glycaemic state.

a, main effect of RPE_pos*state revealed bilateral activity in the PBN [-2–29–26; FWE SVC]. The colour bar indicates the t-value on a scale of white to blue. b, fitted response (β = 1.66) of the local maxima of PBN cluster (7 voxels) to the possible trajectories that RPE_pos*state, yield (Fig 1D) modulated by serum glucose state. There are four possible trajectories of the positive RPE, according to the four different possible trial types depicted in Fig 1C. These trajectories are modulated by serum glucose and shown as different colours with 5 different equally spaced glycemic states. There are thus 4 sets of these trajectories (the four closest to the viewer), each showing their modulation by glucose in the different colours. In order of closeness to the viewer, the first set of trajectories is for the Cue_high- Outcome_gluc trial; the second set is for the Cue_high−Outcome_water trial; the third set is for Cuelow−Outcome_gluc trial; the fourth set is for the Cuelow−Outcome_water trial. The fifth set of trajectories (furthest from the viewer) superimpose together all possible trajectories depicting the complexity of how the trajectories are patterned according to their modulation by glucose state. For comparison, this presentation is analogous to the superimposed trajectories shown in Fig 1D. Note that the colour bar for the graphs in (b) is serum glucose and is distinct from the colour bar in (a) used to indicate t-values.

https://doi.org/10.1371/journal.pone.0243899.g003

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Statistical parametric maps of RPE_neg*state subtracted for increasing minus decreasing.

a, negative reward prediction error RPE_neg*state revealed glucose modulated activity in SN [±12, -22, -10] and VTA [0, –15, –9] when subtracting the effect of descending from the ascending glucose state [FWE SVC]. b, fitted response (β = 0.34) of the local maxima of cluster [7, -11, 8; 52 voxels] to the three possible trajectories that RPE_neg*state yield modulated by serum glucose state. Onsets are not at zero because the negative trajectories do not envelop the trial mean which has a positive expectation. In order of closeness to the viewer, the first set of trajectories is for the Cue_high- Outcome_gluc trial; the second set is for the Cue_high−Outcome_water trial; the third set is for Cuelow−Outcome_gluc trial; the fourth set is for the Cuelow−Outcome_water trial. The fifth set of trajectories (furthest from the viewer) superimpose together all possible trajectories depicting the complexity of how the trajectories are patterned according to their modulation by glucose state. The set of trajectories furthest away from the viewer superimposes the trajectories from all trial types into one plot.

https://doi.org/10.1371/journal.pone.0243899.g004

We also tested for state-dependent modulatory effects on RPE processing which depends on whether serum glucose was ascending (Fig 1B, left) or descending (Fig 1B, right) over time. This yields four different contrasts (ascending vs. descending and the converse, for RPE_pos*state and RPE_neg*state) that are directly relevant to glucose state. Subtracting descending trajectories from ascending and vice versa, revealed no significant activity changes for RPE_pos*state [whole brain, uncorrected]. The same comparisons for RPE_neg*state did reveal significant effects in VTA and substantia nigra for ascending trajectories relative to descending trajectories (Fig 4A). This result shows a relative amplification of the RPE_neg*state signal as glucose state increases. In instances where reward was lower-than-expected (thus yielding negative RPE), the glucose state modulated the RPE_neg signal in VTA and SN more so when glucose levels were ascending than descending.

Discussion

Participants performed a simple cue-conditioning task involving the probabilistic delivery of glucose or water, whilst their blood glucose fluctuated over the course of an hour. We had hypothesized that low levels of serum glucose will positively modulate the scale of positive RPE responses in hypothalamic-brainstem networks, reflecting the marginal utility of glucose as a function of homeostatic needs. Contrary to this hypothesis, we did not observe any positive RPE that increased its scaling with decreasing levels of serum glucose levels. In exploratory analyses there were however several observations of a dependency between serum glucose and RPE signals. Reward prediction error signalling in the parabrachial nuclei scaled positively with serum glucose levels, and this was true whether glucose was ascending or descending over time. We found that both the VTA and SN became more sensitive to negative RPEs for ascending compared to descending glycaemic trajectories. We begin by discussing the interpretation of these state modulated RPE effects, before considering other effects, and the limitations inherent under this paradigm.

In rodent models, the PBN acts as a 2^nd order relay of inputs from the nucleus tractus solitarius, and is critical in the control of energy homeostasis via its projections to amygdala [38, 39], VTA [40], hypothalamus [39, 41] and the nucleus accumbens [42]. Subnuclei of the PBN are targeted by descending projections from several nuclei implicated in energy homeostasis, including hypothalamus, amygdala, and the bed nucleus of the stria terminalis [39, 43]. The PBN is known to be a potent site of reward modulation and subsequent behaviour in rodents. Microinjection of benzodiazepines [44–46], endocannabinoids [47], opioids [48, 49] and melanocortin agonists [50] into the PBN, all evoke hyperphagias. To our knowledge, the involvement of PBN in context of hedonics and reward signalling in the human brain remains to be charted. Here we provide tentative evidence that PBN activity generates a positive RPE-like signal that is sensitive to blood glucose and is time-locked to both the sensory cues predicting glucose, as well as glucose consumption.

We found no state modulation of RPE_neg signalling (RPE_neg*state), expressed during both ascending and descending glycaemic trajectories. For the RPE_neg signal, the modulatory effect of the glycaemic trajectory depended on whether glucose trajectories were ascending or descending. Regional activity scaling with RPE_neg, the VTA and SN showed significantly higher state modulation effects during ascending compared to the descending glycaemic paths. In our experiment, the ascending glucose trajectory resulted from a low-glucose preload with the subsequent increase over time likely occurring by virtue of the continual ingestion of glucose throughout the paradigm (Fig 1B). In the ascending condition, the neural response to RPE_neg is attenuated at lower levels of serum glucose, while it becomes amplified by the transition to higher serum glucose. Given that there is some evidence that dopaminergic neurons of the VTA and SN are directly inhibited by insulin [15], it is possible that the insulin release following hi-glucose preload was highest at the start of the paradigm, decreasing over time, and thus resulting in a gradual decrease in inhibition. However, it should be noted insulin can have a stimulatory effect on dopaminergic firing rates [51]. The difference in RPE_neg in its state modulation (RPE_neg*state) between ascending and descending may therefore be attributed to differential dynamics of insulin secretion [52], though other hormones such as ghrelin [52–54] or leptin [55–58] may play a role. It is not presently known, why it would make sense that a behaviorally reinforcing signal, such as phasic RPE, is expressed less as glucose levels are decreasing in a situation where the body is moving towards a state of potential dyshomeostasis.

Our finding that the VTA and SN responses are linked to RPE_neg may appear counterintuitive, given that these midbrain regions are typically associated with BOLD responses signalling positive-going RPEs. This is assumed to be by virtue of the fact that a greater range of firing rates can be devoted to the better-than-expected range, signalled by above baseline firing. This is contrasted to the worse-than-expected range, which can only be signalled by a decrease from an already low baseline frequency. It is conceivable that what we are asserting as being RPE_neg is in fact a positive RPE resulting from the gradual avoidance of glucose, which increases in magnitude with increasing levels of serum glucose as reported in humans [2] and rats [59]. Thus, as the experimental paradigm continues, especially under the conditions of glucose preload, serum glucose increases, and this may change the valence of the outcome, switching the affective connotation of glucose from palatable to aversive.

As detailed in the introduction, little is known about how the interface between dopaminergic RPE signalling and energy homeostasis is implemented in the human brain. While there are many means by which circulating factors can modulate activity in the VTA and SN, the mechanisms by which this is mediated cannot be revealed without wider hormonal assays. Contemporaneous hormonal sampling, as well as continuous glucose monitoring in the scanner will prove an important step in revealing these latent factors.

There are several technical limitations that should be noted in discussing this experiment. Though relatively high volumes of functional data (150 minutes per subject) were acquired in each subject, the total number of subjects was small. The reason for this was a focusing on maximising experimental power within subjects, for finer scale inference on the longer timescale glucose dynamics. It was not known ahead of time how large the modulatory effects would be, and thus we deployed a conservative strategy of testing fewer subjects for longer. The long regressors that result from the concatenation of sessions may have meant that the high pass filtering would have reduced the effects of ascending and descending glucose levels. Inferring slow timescale dynamics is generally a problem for fMRI, however it is circumvented to some degree here, insofar as we are inferring the modulatory effect of glucose on faster RPE signalling dynamics which occur on a faster timescale than that which is filtered by the high-pass filter. Due to the small number of subjects, we deployed a fixed effects analysis over all subjects. It should be noted that this makes assumptions about the nature of the noise that might not be compatible with the repeated measures design, since it is difficult to correct for non-sphericity in this setting. Future work will expand this paradigm with a larger group of subjects to afford random effects modelling, and thus generalisation to the population sampled from. Contrary to our hypotheses, we found no modulatory effect of hypothalamic nuclei on RPE signalling. We stress that the current imaging protocols and field-strength (3T) were not optimal to dissociate neural activity in the hypothalamic nuclei. Due to the proximity of air sinuses adjacent to the hypothalamus and the effective resolution available, the present study most likely had insufficient sensitivity to capture activity in hypothalamic regions of interest. Another issue is whether the hemodynamic modelling was appropriate for detected evoked responses in subcortical regions which may deviate from the canonical hemodynamic response functions typically used. The regression model we used deployed temporal and dispersion derivatives for all regressors of interest in order to account for idiosyncratic variance in the timing and temporal spread of the hemodynamic response function. Finally, it should be noted that the cue-conditioning employed in this study was passive. Hence, subjects produced no overt choice behaviour against which to fit learning rate parameters for the RPE model, instead we relied on the asymptote values for the RPE signals. The problem of modelling RPEs in the absence of choice behaviour, motivates fitting learning rate parameters directly to brain data, a computational imaging approach that future work will exploit [60].

Recent literature on the computational neuroscience of reinforcement Learning (RL) has evidenced how decision-making in the mammalian brain is driven by optimizing the net value of both primary and non-primary rewards. Such reward computations have been shown to rest on a comparison between the expectation and outcome of external environmental cues, integrating both the physical and cognitive effort costs of the agent [61–63]. The work presented here tentatively expands this perspective this by showing that reward and RPE signals are dependent on internal homeostatic states, which may serve to modify the motivational values according to the personal and time varying homeostatic needs of the organism.

In conclusion, we exploited a simple paradigm, capable of eliciting RPEs under differential glycaemic trajectories, to identify brain stem structures that show a modulation of RPE signalling depending on the glycaemic state. We found that the PBN signals a positive-going reward prediction that is subject to systematic modulation by serum glucose. In the VTA and SN, negative-going RPEs were modulated by serum glucose trajectories, but in a way that was specific to an ascending glycaemic slope. Together the results show that RPE signals in key brainstem structures can be modulated by homeostatic trajectories inherent in naturally occurring glycaemic flux, revealing a potentially tight interplay between homeostatic state and the signalling of primary reward in the human brain.

Acknowledgments

We thank Mehdi Keramati and Boris Gutkin for several helpful discussions.