2.1.1 Video clips.

We considered emotionally-charged video clips of the public database developed by Schaefer and co-workers [35] (http://nemo.psp.ucl.ac.be/FilmStim/) (Table 1). The spoken language of these video clips was French or dubbed into French. Each video clip in the database is labeled in terms of emotional category (fear, anger, sadness, disgust, amusement, tenderness, neutral, further called “standard label”), which Schaefer and co-workers obtained by asking their participants to report what they personally felt in reaction to the video clips, not what they believed people would generally feel. The average duration of all video clips was approximately 3 minutes.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Names, labels and affect scores of the video clips used in our main and control experiments.

Name of videos between quotes and scene numbers between round brackets, standard labels between square brackets, positive or negative emotional affect scores (mean scores between round brackets), and our participants’ self-labels (underlined followed by the number of respondents between brackets). Video clips were taken from http://nemo.psp.ucl.ac.be/FilmStim/, standard labels and affect scores from Table 1 in [35] or provided by Alexandre Schaefer (personal communication).

https://doi.org/10.1371/journal.pone.0186916.t001

We performed two experiments, further called main and control. For our main experiment, we selected two sets of 4 video clips (out of 70 video clips), all with top ten scores in their respective emotional categories. For the first set, 3 out of 4 videos (disgust, amusement, tenderness) had, in addition, also top ten positive or negative affect scores (cf., Table 1); the 4^th video (sadness) did not have a top ten affect score. It was selected to verify the statement of Aftanas and co-workers [21] that higher complexity values can be observed for affective stimuli compared to more neutral ones. As these videos have different emotional labels and affect scores, we want to see whether this translates into a difference in EEG complexity values. For the second study, 4 video clips were chosen but not from the top ten affect score list: as these videos shared lower affect scores, we used them in our experiment as controls. All videos were presented in random order to our participants.

For the implanted patient, as the implant serves a medical purpose, the time window that we could dispose of to perform experiments was restricted. Hence, we used only two video clips: fragment 3 of ‘Schindler’s list’ as an emotional video (standard labeled as ‘anger’) and the weather forecast as an example of a more neutral video. The data recorded in this patient was used as a case study for assessing the differential activation of the amygdala in response to the two video clips and the putative relation between amygdala activation and scalp EEG.

After watching each video clip, all participants (including the implanted patient) were asked to categorize the evoked emotion (after providing them with the same list as used in Schaefer et al.’s: fear, anger, sadness, disgust, amusement, tenderness, neutral). We further call these the ‘self-labels’. Note that our participants were not informed about the video clips’ standard labels. Only in the case of ‘anger’ the self-labels were not univocal as 16 participants reported ‘disgust’ instead (‘Sleepers’, i.e., a video about child abuse). To summarize, we have 3 labels (Table 1):

• standard label: the emotional category of each video clip, taken from Schaefer et al.;
• self-label: the emotional category of each video clip selected by our participants from the list of standard labels used by Schaefer et al.;
• self-reported emotional affect scores of each video, also taken from Schaefer et al.

Unless noted otherwise, we will use the self-labels for labeling the entropy curves.

2.1.2 Participants.

The main experiment was performed with 30 healthy volunteers (20 female, 10 male, mean age = 32.48, SD = 15.77, age range 19–70) who master French language (i.e., French as mother tongue or French-Dutch bilinguals). We also recruited 2 non-French speaking volunteers (1 Flemish-Dutch speaking 24 yo male, 1 Armenian speaking 28 yo female). For the experiment with the control videos (with low affect scores), we tested 6 volunteers (2 female, 4 male, mean age = 27, SD = 1.6, age range 25–30). Volunteers were recruited via emails, social media posts, flyers, and billboard announcements. Some were university graduate students (KU Leuven, VUB), often being regular subjects in our EEG experiments, and were paid. No participant had any known neurological or psychiatric disorder. Ethical approval for this study was granted by an independent ethical committee (“Commissie voor Medische Ethiek” of UZ Leuven, our University Hospital). The study was conducted in accordance with the most recent version of the Declaration of Helsinki (2013). Before participating in the experiment, all recruited participants were informed about the goal of the study, what would be their task, and what would be done with the recorded data (privacy), after which they read and signed the informed consent form that was previously approved by the said ethical committee. All EEG recordings were performed between 10/12/2014 and 13/05/2015. The raw scalp EEG recordings are available from https://kuleuven.box.com/v/EntropyEmotion

2.1.3 EEG recording and preprocessing.

Participants were tested in a sound-attenuated, darkened room with a constant temperature of 20 degrees, sitting in front of an LCD screen. Each participant’s task was to watch the video clips and report the emotional categories. When viewing the video clips, EEG was recorded⁵ continuously using 32 active electrodes, evenly distributed over the entire scalp (positioning and naming convention following a subset of the extended 10–20 system) using a BioSemi ActiveTwo system (BioSemi, Amsterdam, the Netherlands) as well as an electro-oculogram (EOG) using the set-up of Croft and co-workers [50]. The EEG signals were re-referenced offline from the original common mode sense reference [51] (CMS, positioned next to electrode Pz) to the average of two additional electrodes that were placed on the subject’s mastoids. The duration of the experiment excluding electrode setup was 20 minutes. The EEG signals were filtered using a 4^th-order Butterworth bandpass filter with range 0.5–30 Hz. The original sampling rate of 2048 Hz was downsampled to 128 Hz (including anti-aliasing) to reduce computational costs. Finally, the EOG signal was utilized for removing eye artifacts following the Revised Artifact-Aligned Averaging (RAAA) procedure described in [50].

2.1.4 Intracranial EEG recording and preprocessing.

We also recorded intracranial EEG (iEEG), also termed Electrocorticography (ECoG), from a patient, scheduled for resective surgery, as part of her epileptic seizure treatment, when viewing an emotional and a neutral video clip. The patient was implanted with an intracranial electrode (depth electrode with 10 contacts, contact size 2.4/1.1 mm (overall length/diameter) and 4 mm inter-contact spacing) in the right hippocampus until the amygdala (2 contact points present) and with a subdural electrode grid (4x5 electrodes with 4 mm electrode diameter, 2.3 mm electrode exposure and 10 or 15 mm inter-contact spacing) over the right occipital cortex (S5 Appendix). Recordings were made with a Micromed digital video compatible EEG recording system (Micromed Spa, Mogliano Veneto, Italy). The sampling frequency of the recording was set to 1028 Hz. Offline pre-processing and downsampling were done using the same parameters as for the scalp-recorded EEG, except that a CMS reference per intracranial electrode (10 contacts) and grid (20 contacts) was used. Ethical approval for this study was granted by an independent ethical committee (“Commissie voor Medische Ethiek” of UZ Gent) and conducted in accordance with the most recent version of the Declaration of Helsinki (2013). Before participating in the experiment, the patient was informed about the goal of the study, what would be her task, and what would be done with the recorded data (privacy), after which she read and signed the informed consent form that was previously approved by the said ethical committee. The recordings were done on 29/02/2016.

2.2.1 Sample entropy.

We compare 3 entropy-based methods. The first one is Sample Entropy (SE) [52]. It corresponds to the conditional probability that two sequences that are similar to each other for m consecutive data points (samples), within tolerance level r, and remain similar when one more data point to each sequence is added. Formally, SE is expressed as follows: where B^m(r) is the probability that the similarity between two sequences of length m obeys the tolerance level r, A^m+1 (r) the probability that the similarity between the same two sequences but now extended with one data point, thus of length m + 1, also obeys r, and N the number of sequences. The tolerance level r is usually set to a percentage of the standard deviation of the normalized data; for our case we selected 15% [31].

In order to estimate sample entropy for a multivariate case (MSE), the sequences are formulated as follows. Recalling multivariate embedding theory [53], for d-variate time series , the multivariate embedded sequence (so-called a composite delay vector) can be constructed as: where , is the embedding vector and τ = [τ₁, τ₂, …, τ_d] the time lag vector. The above sample entropy definition is then applied to the composite delay vector.

In our case, we have d electrode channels, n the number of samples or data points in a sequence (further called ‘snippet’ as its length is small in practice), and N the number of sequences. The computational complexity of MSE for multichannel EEG recordings is qubic: ϑ(dNn + dn).

2.2.2 Rényi’s entropy.

Consider a discrete random variable x that adopts n values with probabilities p₁, p₂, …, p_n. When the k-th value delivers I_k bits of information, then the total amount of information becomes the so-called Shannon’s entropy: (1)

In (1), a linear averaging operator is assumed however, in general, for any monotonic function g(x) with an inverse g^-1(x), the general mean associated with g(x) for a set of real values {x_k, k = 1, …, n} with probabilities {p_k, k = 1, …, n} can be written as:

Hence, using the general mean, the total amount of information will be: (2) with g(x) a Kolmogorov-Nagumo invertible function. Rényi proved that, when the criterion of additivity of independent events applies to (2), then the range of usable functions g(x) is dramatically restricted. There are two possible classes:

• g(x) = c, where c is a constant, so the general mean becomes linear and Shannon entropy (1) is obtained;
• g(x) = c2^(1−α)x, which implies that the entropy definition becomes: (3) which is called Rényi’s information measure of order α.

We are particularly interested in the case where α = 2, further called Rényi’s quadratic entropy (RQE), as the term between brackets in (3) corresponds to the expected value of the probability density function (PDF). We wish to estimate RQE directly from the sampled signal using a Gaussian kernel for Parzen’s density estimate: with N the number of samples and σ the kernel size or bandwith parameter. Hence, we obtain: where IP is the information potential[54]. Note that, for the multivariate case, the average of the univariate kernels is taken. For ease of reference, we will call it multivariate RQE (MRQE). The bandwidth σ is a free parameter that can be chosen according to Silverman’s rule: where n is the number of data points in a snippet, d the data dimensionality (number of electrodes), and σ_X the standard deviation.

The computational complexity of MRQE is quadratic: ϑ(dn + n).

2.2.3 Kernel-based Shannon entropy.

As a third entropy-based method, we consider Shannon entropy (ShE): where p_k is a probability density function (PDF). As in the case of RQE, we estimate the PDF directly from the samples using a Gaussian kernel for Parzen’s density estimate: where nis the number of data points in a snippet and σ the kernel size or bandwidth parameter. Similarly to RQE, we develop the multivariate case with the Shannon entropy (MShE). The computational complexity of MShE is quadratic as well:ϑ(dn + n).

2.2.4 Multivariate Empirical Mode Decomposition (MEMD).

With Empirical Mode Decomposition (EMD) a signal of length M data points is split into a l = log₂M narrow-band, amplitude/frequency modulated components called Intrinsic Mode Functions (IMFs) [55]:

IMF₁ corresponds to the highest frequency component and the subsequent IMFs to lower, more narrow-banded frequency components. The last component, IMF_n, is the trend in the signal and is usually omitted from further analysis. The multivariate extension of EMD (MEMD) [31] aligns similar frequency bands of multiple channels, thus, providing an assessment of their possible interdependence (mode alignment property).

2.2.5 MEMD-enhanced multiscale, multivariate entropy.

There are at least two ways to compute a multiscale version of entropy when using MEMD: we can compute multivariate entropy (MSE, MRQE or MShE) for each scale individually (IMFs) or for their accumulated scales (cumulative IMFs, CIMFs). We explain the algorithm for the CIMF case:

1. obtain the IMFs for each subject’s entire recording length of each video clip (M data points, with M in our case between 30000 and 52000, depending on video clip length) by applying the MEMD method to a given number d of EEG electrodes;
2. accumulate the IMFs one by one, starting with the first one, CIMF₁ = IMF₁, and then add the second IMF to the first, CIMF₂ = IMF₁ + IMF₂, and so on, until all IMFs are added;
3. for each CIMF, calculate multivariate entropy (d electrodes) for each non-overlapping snippet of a prior defined EEG signal track and take the average over all those N snippets (MSE) or calculate the univariate entropy per electrode for those snippets and take the average over all d electrodes (MRQE, MShE);
4. plot the entropy estimates as a function of CIMF₁,CIMF₂, … to obtain the MEMD-enhanced multiscale, multivariate entropy curve of the targeted EEG signal track (i.e., MEMD-enhanced MMSE, MMRQE, MMShE)

Note that the total number of IMFs is log₂ M [55] but the used algorithm selects by itself the number of IMFs in a data-driven, subject-dependent way (in our case, between 15 and 17 IMFs), hence, for clarity’s sake, we decided to show 15 IMFs for all subjects. Both operations imply that the original signal is only approximated by CIMF_n, so their entropies could be different.

For the interested reader, the Matlab code for the multiscale MSE algorithm can we found at http://www.commsp.ee.ic.ac.uk/~mandic/research/Complexity_Stuff.htm, and the Matlab code for the multivariate EMD at http://www.commsp.ee.ic.ac.uk/~mandic/research/emd.htm

2.2.6 Scalp space projection of intracranial recordings.

In order to explore the relation between scalp- and intracranial recordings, in response to emotion-evoking and neutral video clips, we performed a source reconstruction analysis with the Brainstorm toolbox as it can work directly with intracranial electrodes. We started with the patient’s post-implantation MRI headscan to define the standard Montreal Neurological Institute (MNI) stereotactic coordinates of the depth electrode’s contact points in the amygdala-hippocampal complex and the electrode positions of the grid covering the occipital cortex. Then, we used the patient’s MRI scan before implantation to extract the cortex envelopes from the MRI scan (inner and outer skull, scalp surface and cortex) using the Brainsuite software[56]. Fiducial points were selected manually. For the forward model, Boundary Element Model (BEM) surfaces were created using 15000 dipoles on the entire MRI volume so as to include deep sources, using the OpenMEEG BEM model [57]. For the inverse modeling, as there was no resting state data available, the identity matrix was selected for the noise covariance, and the sLORETA algorithm used for distributed source modeling [58].

The depth electrode had 10 contact points but, based on anatomical grounds, the first 2 contact points were considered to be in the amygdala. However, as their recordings did not display a linear relation (no obvious Pearson correlation, see S4 Appendix), but instead the recordings of the second contact point seemed to correlate with those of contact points 3 to 6, we only used electrode 1 to simulate data from that contact point to see how its projects back on scalp space. In order to obtain a sizeable estimate of amygdala activation, we assumed that 1 cm³ of amygdala tissue, in proportion to the entire head volume, corresponds to about 27 amygdala dipoles out of 15000 in total. We put all values of the headmodel to zero, except for the said 27 dipoles, which we filled with contact point 1’s recordings. As the headmodel for the entire MRI volume is not constrained in orientation and orientation, but needs to be defined for our simulations, we chose four possible orientations for simulating our deep source—the X-, Y-, and Z-directions, and also the equally mixed version, with an equal weight in each direction—, and projected the result on the entire scalp.

Next, we wanted examine whether the sources that generated the intracranial recordings of the electrode grid covering the occipital lobe also contribute to the frontal scalp recordings. For this, we first solved the inverse problem, thus starting from the 20 electrode grid recordings, and then addressed the forward problem to see how these sources show up in the frontal part of the scalp.