Visual stimuli

Stimuli were presented in free-viewing, and participants were free to move their eyes throughout the study. Participants were seated 128 cm from a canvas onto which two spatially aligned projectors projected a movie which contained an embedded 0.25 Hz modulation of photoreceptor contrast enabled by the method silent substitution (see below). The stimulus was a rectangle of dimensions 81 x 109 cm, yielding a total visual angle of approximately 35 x 46 deg². The targeted photoreceptors were the LMS cones at equal contrast (LMS), melanopsin (Mel), or all photoreceptors in unison (Light Flux). A fourth condition (Reference) contained no modulations. In our low-frequency modulations, we did not control for the effect of cones in the shadow of the retinal blood vessels, so-called penumbral cones, which are known to be stimulated in high-frequency (8–16 Hz) melanopsin-directed modulations [18].

Principle of silent substitution

The spectral tuning of the individual photoreceptors is relatively broadband, and therefore any given light will activate all photoreceptors (Fig 1B). Measurement of downstream responses, such as pupil size, using narrowband, or monochromatic light, will therefore only yield non-specific responses. For example, while in the living human eye, melanopsin will respond maximally to a monochromatic ~488 nm light (the shift from 480 nm of peak pigment absorbance to longer wavelengths due to pre-receptoral filtering by the lens and ocular media), such a light will also activate S cones at ~41%, M cones at ~43%, and L cones at ~26% of their maximal response, respectively. This non-specificity can be overcome by the method of silent substitution [19, 20]. In this method, pairs of spectra are presented to an observer such that the exchange between the spectra only stimulates one class of photoreceptors (the stimulated photoreceptors), while not changing the activation of the set of silenced photoreceptors. Silent substitution can be achieved using any light source which has at least as many independent lights as spectral photoreceptors under examination, such as sets of LEDs combined optically [7, 9–11, 13–16, 21–23], or combination of broadband light filtered through a diffraction grating and imaged on digital micromirror device [12, 18, 24, 25] (see [26] for technical details), or using modified projectors [17, 27] as in this study.

Projector system

The design of the projector system was first described in [17], but this concrete realization was slightly different. Two 8-bit Hitachi CP-X5022WN 3LCD projectors were stacked on top of each other and the boundaries of their respective projected outputs were aligned by the experimenters using lens adjustment in vertical and horizontal axes, which was further refined with digital cornerstone correction. Alignment was confirmed by visual inspection of corners and edges and in subsequent experiments (not reported here) using high-spatial frequency stimuli. The projectors were modified using optical filters. In Projector 1, the blue primary was modified using a 470 nm cut-off yellow longpass filter (PIXELTEQ, Largo, FL; part # LP470-r40x25x1), making a cyan primary; the green primary was modified using a 463–571 nm notch filter (MidOpt, Palatine, IL; part number #102340892), making yellow. In Projector 2, the blue primary was modified using a 463–571 nm magenta notch filter (MidOpt, Palatine, IL; part #102340892), making violet; and the green primary was modified using a 550 nm bandpass filter (PIXELTEQ, Largo, FL part # Bi550-r40x25x1), producing a narrower green primary. This yielded a dual projector system with five independent primaries (Fig 1C and 1D). At maximum output, the primaries had CIE 1931 xy chromaticities: (0.65, 0.34), (0.50, 0.49), (0.08, 0.22), (0.28, 0.70), and (0.16, 0.02), and photopic luminances 54.36, 131.64, 20.84, 185.66, and 8.18 cd/m², respectively. The projectors were calibrated and linearised using a CRS SpectroCAL spectrometer (Cambridge Research Systems, Rochester, UK).

Movie stimuli

The movie selected was the Tom & Jerry cartoon “Pent-House Mouse” (1963) [28] (Fig 1E, length 6m11s). The film was first turned into a greyscale movie and then processed in MATLAB to embed photoreceptor-directed modulations into the luminance variations (Fig 1F, 1G and 1H). The photoreceptor contrast of the modulation was 28% for all directions. The modulations were embedded as follows. The (assumed) linear intensity for pixel at position x, y at time t is given by I(x, y , t). The linear 0–1 value for primary n (where n ≤ 5) is then given by:

The resultant linear values per primary were then converted into 8-bit resolution.

Photoreceptor spectral sensitivities

For the LMS cones, we assumed the physiologically-relevant CIE 2006 photoreceptor sensitivities [29]. Melanopsin spectral sensitivity was derived using the Govardovskii nomogram with λ_max = 480 nm, assuming low photopigment density, and applying the same pre-receptoral filtering as for the cones. The receptor spectral sensitivities are available in the Silent Substitution Toolbox (http://github.com/spitschan/SilentSubstitutionToolbox).

Age adjustment

To account for age-dependent differences in pre-receptoral filtering (specifically age-dependent lens-density), stimuli were adjusted in age group for each observer, with discrete categories 20–24 years of age, (22 years reference age) 25–29 years of age (27 years reference age), and 30–34 years of age (32 years of age). Because of the relatively large variability of lens density even across individuals of the same age [30, 31], we reasoned that adjusting according to age group in this way is an acceptable strategy. The age adjustment of pre-receptoral filtering was performed following the calculations set by the CIE [29]. Within those age groups, lens density does not abruptly change.

Protocol and procedure

After giving informed consent, observers were tested for visual acuity and normal colour vision. Sitting down, they then viewed the four video conditions consecutively, with a two-minute break in between each video. The order of the conditions was randomized between participants. All testing took place during regular daytime working hours (9 to 5 pm).

Pupillometry

The participants’ pupil responses were measured using a light-weight binocular eye tracking headset (Pupil Labs GmbH, Berlin, Germany), recording pupil responses at 30 Hz, and also a front-facing view of the participant’s view. Using software provided by the manufacturer, 2D ellipses were fit to the resulting videos off-line, and time-locked to the onset of the stimuli using open-source software provided by the manufacturer (https://github.com/pupil-labs/pupil).

Participants

Eleven observers (mean age±SD 24.5±6.5 years, 7 female) were recruited from the wider University of Manchester community. Observers were entered into a raffle to win a hamper. Two participants were authors of this study (A.A. and M.G.). All subjects had 6/5 or better visual acuity (Snellen chart) and normal red-green colour vision as assessed by the Ishihara test [32].

Data analysis

Raw pupil data were processed using custom MATLAB software as follows. Missing data reported from the 2D fitting algorithm were interpreted as eye blinks; to account for transient mis-estimations of the pupil size before and after eye lid closure, we removed additional samples around these events (padding window ±5 samples at 30 Hz). These missing samples were then linearly interpolated. Data were them smoothed with a Savitzky-Golay filter (11^th order, frame length 21 samples) using MATLAB’s built-in sgolayfilt() function. Raw pupil data (expressed in pixels of the 2D ellipse) were mean-normalised across the entire time series and expressed as percent change around that mean. We then extracted the per-cycle average pupil responses, collapsing across all cycles during the entire cartoon. This per-cycle average was then fit, in a least-squares sense, with a sum of sine and cosine at the stimulation frequency (0.25 Hz); the weights of the sine and cosine were then transformed into amplitude and phase of the evoked pupil response.

Ethics

Ethical approval was obtained from the University of Manchester Ethics commission (approval number #2017-2958-43594359).

Code and data availability

All data and code to perform the data analyses reported here can be found at https://github.com/spitschan/Spitschan2019_PLOSONE.

Photoreceptor-selective modulations embedded in movies produce distinct pupil responses

The 0.25 Hz photoreceptor-specific modulation stimuli evoked distinct responses in all observers (Fig 2A–2D). The strongest response (in amplitude expressed as percent change in pupil diameter relative to the mean) was elicited by the Light flux modulations (mean±1SD = 5.0616±1.2554%, median = 5.3051%), followed by L+M+S (mean±1SD = 4.1340±0.9263%, median = 4.6839%) (Fig 3A). In many observers we find a small melanopsin-evoked pupil response and in aggregate, there is a significant difference between the Melanopsin (mean±1SD = 2.1182±1.3096%, median = 1.5605%) and the Reference condition (mean±1SD = 1.0792±0.7652, median = 0.7462%) (Wilcoxon rank sum test, Z = 2.3639, p = 0.0181). We note that the melanopsin-evoked pupil response is somewhat inconsistent across observers, raising the possibility of individual differences in the observers’ melanopsin sensitivity (Figs 2B, 3A and 3B). Though there is nominal difference in mean amplitude between the Light flux and the L+M+S modulation, this difference is not statistically significant (Wilcoxon rank sum test, Z = 1.7730, p = 0.0762).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Individual subject pupil data.

Data are collapsed (column-wise) across stimulus types. A) L+M+S, B) Melanopsin (Mel), C) Light flux (all photoreceptors are modulated), D) no modulation. Red lines indicate the least-squares fit of a sum and cosine at the modulation frequency to cycle-average response.

https://doi.org/10.1371/journal.pone.0216307.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Extracted response properties of the pupil responses to photoreceptor-directed modulations embedded in a cartoon movie.

(A—amplitude, B—phase). Data shown as red crosses are represent mean±SEM across participants. A) Amplitude results. Noise floor is given by the mean amplitude of the fitted pupil response to the Reference modulations, which contained no additional power at the stimulation frequency. B) Phase data, with negative angles corresponding to a phase delay (pupil response lags behind the stimulus), and positive angles corresponding to a phase advance (pupil response precedes the stimulus). Because of the nature of phase, a very long phase delay (longer than –180°) may manifest as an apparent phase advance as the phase wraps around. The amplitude and phase data shown here correspond directly to the red fit lines in Fig 2.

https://doi.org/10.1371/journal.pone.0216307.g003

Amplitude and phase of pupil responses

Expressed as amplitude and phase, we find that LMS and light flux pupil responses are well above the noise floor defined by the aggregate fit of the pupil response to the stimulus containing no stimulus at the modulation frequency (0.25 Hz). For all observers, the phase of the LMS and light flux response is delayed relative to the stimulus onset (Figs 2A, 2C and 3B), which is consistent with the “dead time” of the human pupil response (~200–300 msec), i.e. the minimum delay between a stimulus and a pupil response. For the melanopsin-evoked pupil response, we find variability in both amplitude and phase (Figs 2B, 3A and 3B), which warrants further investigation. For some observers and in the group mean data, it even appears as though there is a phase advance in the melanopsin stimulus (positive phase values, Fig 3B). We note that this might well be a very long phase delay consistent with the slow signaling properties of the melanopsin-encoded ipRGC response. We cannot address this in our data, as we focused only on one frequency for the embedded contrast modulations.

Summation of pupil responses

We examined to what extent the sum of the responses to the LMS and Mel stimuli resembles the response to the Light Flux stimulus, which is, when expressed in contrast, the sum of the LMS and Mel stimuli in the stimulus domain: The Light Flux stimulus modulates the L, M, and S cones in the same way that the LMS modulation does; and it modulates melanopsin in the same way that the Mel modulation does. We applied a simple linear model by adding the amplitude and phase, expressed as complex sum, of the LMS and Mel responses, and reading out the amplitude again (Fig 4). Conceptually, this is a vector sum model, where the pupil response to each modulation corresponds to a vector (defined by a given phase and amplitude), and we simply add vectors (Fig 4, inset), yielding a new vector. We find that the amplitude of the complex sum of the LMS and melanopsin pupil response can account for the Light Flux pupil response while the LMS response only does so in a limited way (Fig 4). We note that a better fit of the amplitude of the complex sum already requires that the phase of the two summands is aligned in such a way as to yield a better-fitting amplitude, we did not consider the goodness-of-fit of phase.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Linear summation model.

We compare the amplitude of the pupil response to the Light flux modulation to the amplitude of two candidate models: The response to the LMS modulation, and the complex sum of the response to the LMS modulation and the response to the Mel modulation. Inset: Visualization of the vector summation model.

https://doi.org/10.1371/journal.pone.0216307.g004

Summary of results

Pupillary responses to melanopsin-directed stimuli using the method of silent substitution have been documented in the literature [7–16, 24], including by the authors [12, 24]. These studies typically have employed very carefully controlled artificial stimuli. However, past work has not addressed the question of how these photoreceptors each define pupil size during natural viewing. In this study, we find that by embedding photoreceptor-selective modulations in cartoon movie stimuli viewed under real-world conditions, we can elicit pupil responses specific to the receptor type. Consistent with what we would predict from studies employing spatially homogenous stimuli, we find a subtle but significant response to the melanopsin-mediated stimulus at the stimulation frequency (0.25 Hz). The response to the Light flux modulation is in mean amplitude larger than the response to the LMS modulation (though not statistically significant), providing some evidence that our paradigm may detect melanopsin contributions in the Light flux modulation. Further, we find that we can model the pupil response to the Light flux modulation as the complex sum of the responses to the Melanopsin and the LMS modulation, which is corroborated by an earlier finding [12] (see their Fig S8). We note that using our paradigm, we cannot determine the site of this integration process.

Defining pupil size during free viewing of movie stimuli under real-world conditions

By embedding the modulations in an “entertainment” movie, we have placed the task in an ecologically-relevant context for using a video display unit (VDU). Allowing the participants to freely view the movie and not adhere to specific fixation procedures gives the paradigm an additional real-world aspect. While much emphasis has been placed on examining the photoreceptor inputs into pupillary control under well-controlled but relatively reduced stimulus conditions, our approach provides an understanding of these inputs that is complementary to more parametric and analytical approaches. Using this pragmatic approach to see if we could modulate the activity of photoreceptors in a task that human participants might engage in, in a normal day-to-day sense, we indeed confirm that all photoreceptors known to be active at photopic light levels (as established in previous studies, e.g. [12, 13]), can be manipulated using modulations embedded in movies which can be watched in real-world conditions.

Uncontrolled factors

We wish to highlight that this study has purposely not accounted for a set of factors known to influence pupil size, on the grounds that the goal of this study was to examine pupil responses in an approximation of real-world conditions. Because of the near triad, vergence eye movements, changes of accommodation, and pupillary response go hand in hand when shifting view from near to far distances [33]. Under the unconstrained viewing conditions employed in this study, observers were able to move both head and eyes freely, thus introducing pupil changes in pupil diameter unrelated to our experimental parameter under control, photoreceptor contrast. As the stimuli were presented using a projection system in an otherwise dark room, eye movements of any sort could displace high contrast edges between the illuminated part of the canvas and the rest of the (dark) wall, which could lead to a transient signal. Because the modulation of contrast was uncorrelated with changes in movie content (and indeed, agnostic to it) which might have elicited gaze shifts, we feel confident that any effects of accommodation and vergence on pupil size average out, as evidence by the flat pupil response in the reference condition. Changes in autonomic arousal, which leads to differences in pupil size [34], might have occurred during the study, which was addressed by counterbalancing the order stimulus conditions.

Data variability

We note that there appears to be large variability in the data. These could reflect individual differences, though this is not explicitly addressed in our data. The variability in the melanopsin response, both in amplitude and phase, warrants further investigation.

Challenges to silent substitution

The silent substitution nominally allows selective stimulation of a given photoreceptor class. Silent substitution requires, however, a good estimate of the human participants’ spectral sensitivities. In practice, these are assumed based on standard cone fundamentals such as the physiologically-relevant CIE cone fundamentals [29]; some investigators also allow for an individual-observer calibration routine (e.g. [9–11, 13, 23]). In this work, for practical and pragmatic reasons, we forwent calibration at the level of the individual observer, and simply accounted for age-dependent differences in pre-receptoral filtering, which largely affects the amount of short-wavelength filtering, within a given age bracket spanning 5 years.

In our modulations, we did not account for the potential intrusion of rods at daylight levels. It is generally thought that rods do not participate in visual function at photopic light levels [35, 36]. There are some suggestions that rod saturation might not be complete [37, 38]. In mice, there is also evidence that rods retain the capability to signal at daylight light levels [39, 40]. Generating melanopsin-isolating stimuli to test the hypothesis that rods may be at play at daylight levels requires a severe reduction in melanopsin-isolating contrast [41]. In our data, while we cannot rule out a small rod contribution at the nominally rod-saturating light levels, we think that it is unlikely that the contributions we measured are due to rods as their signals would be heavily reduced at the photopic light levels we used. We also note that if rods were considered to contribute to these types of modulations, they would do so in a similar way for any practical application involving VDUs.

Opportunities for translation

Given the broad impact of light on aspects of human physiology and behavior (such as melatonin suppression, circadian phase-shifting and modulation of alertness), the system and paradigm described here to modulate visual responses in a photoreceptor-selective fashion represents an opportunity to modulate these while people are engaging in everyday tasks such as movie-watching [17]. One powerful further application in the future could be as a method for low-impact “passive” assessment of photoreceptor health in clinical contexts, specifically in pediatric and geriatric populations.

Towards a revision of colorimetry?

In this study, we have employed a 5-primary projector-based system for silent substitution with spatial control at the pixel level, and controlling the activation of the photoreceptors active at daylight: the cones and the melanopsin-containing ipRGCs. Commercially available VDUs have three primaries, which reflects the fact that human colour vision in photopic light levels is trichromatic. However, this study and a recent one [17] employing a similar system find that non-visual responses can be manipulated with a modified VDU with more than three primaries. Additionally, there is more and more evidence that melanopsin contributes to “classical” visual functions such as the perception of brightness [22, 42], colour [9, 43], space (humans: [9]; mice: [44, 45]), contrast [46, 47], and other visual attributes (humans: [24], mice: [48–50]). It remains an interesting question to what extent the classical colour matching functions, which are the basis of our current cone fundamentals, reflect the activity of melanopsin (though the amount of rod contributions have been discussed extensively [51]). Given the numerous roles of melanopsin in visual perception and in regulating physiology and behavior, and the emergence of tetrachromatic or higher-order primary displays, we expect that colorimetry may need some revisions to account for these effects, visual or non-visual.