Figures
Abstract
In patients with unilateral upper limb paralysis from strokes and other brain lesions, strategies for functional recovery may eventually include brain-machine interfaces (BMIs) using control signals from residual sensorimotor systems in the damaged hemisphere. When voluntary movements of the contralateral limb are not possible due to brain pathology, initial training of such a BMI may require use of the unaffected ipsilateral limb. We conducted an offline investigation of the feasibility of decoding ipsilateral upper limb movements from electrocorticographic (ECoG) recordings in three patients with different lesions of sensorimotor systems associated with upper limb control. We found that the first principal component (PC) of unconstrained, naturalistic reaching movements of the upper limb could be decoded from ipsilateral ECoG using a linear model. ECoG signal features yielding the best decoding accuracy were different across subjects. Performance saturated with very few input features. Decoding performances of 0.77, 0.73, and 0.66 (median Pearson's r between the predicted and actual first PC of movement using nine signal features) were achieved in the three subjects. The performance achieved here with small numbers of electrodes and computationally simple decoding algorithms suggests that it may be possible to control a BMI using ECoG recorded from damaged sensorimotor brain systems.
Citation: Hotson G, Fifer MS, Acharya S, Benz HL, Anderson WS, Thakor NV, et al. (2014) Coarse Electrocorticographic Decoding of Ipsilateral Reach in Patients with Brain Lesions. PLoS ONE 9(12): e115236. https://doi.org/10.1371/journal.pone.0115236
Editor: Donald A. Robin, University of Texas Health Science Center at San Antonio, Research Imaging Institute, United States of America
Received: March 20, 2014; Accepted: November 10, 2014; Published: December 29, 2014
Copyright: © 2014 Hotson et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: The authors confirm that all data underlying the findings are fully available without restriction. All relevant data are publicly available at http://headit-beta.ucsd.edu/studies/2f0551f6-70c8-11e4-9629-0050563f2612.
Funding: This work was supported in part by the National Institute of Neurological Disorders and Stroke under grant 3R01NS0405956-09S1 (MSF, HLB) and by the National Institute of Biomedical Imaging and Bioengineering under grant 5T32EB003383-08 (GH). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
The brain-machine interface (BMI) is a tool to replace areas of human functionality lost due to trauma or degenerative disease. To date, BMIs have been used with humans to control cursors [1]–[3], move robotic limbs [4]–[7], and allow patients with locked-in syndrome to communicate with the outside world [8]. Electrocorticography (ECoG) electrodes are implantable, non-penetrating, and occupy a unique space between the high fidelity of cortex-penetrating microelectrode arrays recordings and the low fidelity and bandwidth of noninvasive EEG. A long history of work with EEG and ECoG signals has highlighted mu (8–12 Hz), beta (14–30 Hz) [9], [10] and high gamma (>70 Hz) [11] band signals as indices of cortical motor processing. More recent work has shown that ECoG signals may also contain movement-related information in their smoothed amplitudes, or local motor potentials (LMPs) [12]–[14].
In most efforts to develop a brain-machine interface (BMI) to restore upper limb function, the design has assumed a normally functioning brain as the source of neural control signals, with the BMI serving to bypass lesions of neural pathways connecting the brain to its muscle effectors. However, a large patient population with hemiplegia from strokes and other cerebral lesions may also benefit from BMIs [15]. In these patients, a variety of approaches are already being used or developed to restore function. These include intensive neurorehabilitation [16]–[18], direct current stimulation of the motor cortex [19], [20], and neurobiological therapies, e.g. infusions of stem cells and growth factors, aimed at promoting neurogenesis in the damaged hemisphere [21]–[23]. It might also be feasible to restore function via neural signals from the undamaged hemisphere [24]–[26]. However, cerebral representations for upper limb control are primarily contralateral rather than ipsilateral, and the risks associated with surgical implantation of electrodes over healthy, functional cortex may not outweigh the potential benefits. To overcome these limitations, an alternative approach could be to utilize residual sensorimotor systems in the damaged hemisphere. This approach would improve the risk-benefit ratio because subdural electrodes for an ECoG-based BMI could be implanted along with other invasive experimental therapies. Furthermore, training of the BMI, along with intensive neurorehabilitation, could generate neural activity encouraging neurogenesis and plasticity in residual brain tissue, potentially working synergistically towards functional recovery. However, in patients with total paralysis of the contralateral upper limb, arguably the population to whom a BMI would be most attractive, it would be challenging to train the BMI's decoding algorithms. To overcome this limitation, subjects could be trained by imagining their own arm moving [27], viewing arm movements [28], [29], being passively moved [30], or even performing movements with the ipsilateral arm [31]. The neurophysiological correlates of reaching movements in residual motor systems of the ipsilateral cortex therefore warrant investigation.
The descending corticospinal tract is not solely comprised of axons that cross the midline. A minority of the fibers also arise from ipsilateral cortex [32]. Ipsilateral wrist movements may be impaired in hemiparetic subjects with unilateral brain lesions [33]. A substantial portion of PMd and a nontrivial number of M1 cells display tuning to ipsilateral arm movements in primates [34]. Indeed, the efficacy of rehabilitative therapies in many patients with upper limb paralysis from unilateral brain lesions [18] implies not only that functional reserve and potential for plasticity is present in the lesioned hemisphere, but also that ipsilateral descending motor pathways can be utilized for upper limb control. The degree to which arm movements can be decoded from the neural activity of a damaged ipsilateral cortex, however, is not well known and is of particular interest for training an upper limb BMI in patients with total upper limb paralysis from a stroke or other unilateral cerebral lesion.
Reach trajectories have been decoded extensively in nonhuman primates performing center-out reaches with microelectrode arrays [35]–[38], and more recently in humans and primates implanted with ECoG grids [39]–[42], using neural signals contralateral to the reaching arm. Previous work has also shown the potential for extracting neural correlates of overt and imagined hand movements from intact ipsilateral cortex in humans using ECoG [31], [43] and EEG [44]. However, it has not previously been shown if neural signals from damaged cortical areas can retain any information about ipsilateral arm movements. In this study, we investigated the neural correlates of ipsilateral reaching movements in human subjects with lesions in the hemisphere recorded with ECoG. In two subjects, these lesions disrupted motor pathways and were associated with motor impairment of the contralateral arm. In one subject, the lesion involved superior parietal lobule, likely including the human homolog of parietal reach region, but there was no associated motor impairment.
Materials and Methods
Data Acquisition
Three subjects, two male and one female aged 24–36 years, were implanted with ECoG arrays prior to surgical resection of their seizure foci. Additional subject information can be found in the supporting materials. Grid placement was determined solely on the basis of their clinical need without any influence from this study. Only subjects with motor cortex coverage were considered for this study. Table 1 summarizes the clinical and demographic information associated with each study participant. All subjects gave written informed consent and the study was conducted under a protocol approved by the Johns Hopkins Institutional Review Board (IRB). ECoG grids consisted of rectangular arrays of platinum electrodes (Adtech Medical Instrument Corp; Racine, WI) with 2.3 mm diameter and center-to-center spacing of 10 mm. Electrodes were embedded in a Silastic sheet and implanted on the brain in the subdural space. Arrays of nonpenetrating micro-ECoG leads (75-micron diameter, 0.66-mm spacing) were also implanted in each of the three subjects, but in subjects 1 and 2, these micro-ECoG leads were not over motor areas and therefore not included in our analysis. Of the implanted electrodes, 44–48 per subject were analyzed for this study. A 4×4 array of 16 micro-ECoG leads was included in the analysis for subject 3. These micro-ECoG leads were located in posterior middle frontal gyrus, anterior to electrodes where electrocortical stimulation mapping (ESM) affected motor function, and their signals did not produce notable decoding results. The tests described below were part of a larger battery of research testing. These tests were conducted between two and seven days after implantation, and removal of the grids occurred at the time of seizure focus resection, seven to eight days after implantation.
ECoG signals were referenced to an inactive intracranial electrode and recorded by a 64-channel Neuroscan Synamps2 (Compumedics; Charlotte, NC) for subject 1 and 2; ECoG signals from subject 3 were also referenced to an inactive intracranial electrode, but were recorded by a 128-channel NeuroPort system (BlackRock Microsystems; Salt Lake City, UT). Signals were digitized by the Neuroscan system at 1 kHz after being band-pass filtered between 0.15 and 200 Hz (subject 1 and 2). Signals recorded by the BlackRock system were band-pass filtered in analog with a low-pass cutoff of 7,500 Hz and a high-pass cutoff of 0.3 Hz, digitized at 30 kHz, then downsampled to 1 kHz. Anatomical reconstructions of each subject's electrode placement were generated by volumetrically co-registering post-implantation brain CT with high-resolution pre-implantation brain MRI images using BioImage (Yale, [45]). Additional information from surgical photos, post-implantation skull X-rays in the antero-posterior and lateral axes, and ESM were used to verify electrode locations. Reconstructions of electrode locations, overlaid with the color-coded results of ESM, are shown in Fig. 1. Fig. 2 displays the locations of the cortical lesions for each subject.
The grids shown are the subset of implanted electrodes that were recorded from during this study. The green highlighted areas correspond to regions of cortical lesions. The lesion in subject 3 could not be seen on the brain surface rendering because it was located beneath the surface of the brain. Colored rectangles joining electrodes imply that bipolar stimulation was applied to that pair of electrodes. In subject 1, several electrode pairs were further investigated by performing unipolar stimulation relative to a distant reference electrode. The results of this unipolar stimulation are shown with colored circles surrounding the electrodes.
Reconstructions are shown for subject 1 (first row) subject 2 (bottom left and middle) and subject 3 (bottom right). The previous resection margins anterior to the precentral gyrus in subject 1 are highlighted in green in the upper right. Superior oblique and top axial views of the reconstruction for subject 2 show the lesion from different viewpoints. Pre-surgical MRI (FLAIR) of subject 3 reveals a lesion of posterior left insula also involving left internal capsule.
The three-dimensional positions of the subject's shoulder and arm were tracked optically using the Northern Digital Optotrak system with a sampling frequency of 100 Hz. Neural data and movement data were synchronized using parallel port triggers sent simultaneously from the experimental computer over a split cable to the Neuroscan (subject 1 and 2) or NeuroPort (subject 3) amplifier's external trigger inputs and the Optotrak Data Acquisition Unit (ODAU).
Experimental Paradigm
Subjects were instructed to make reaches to the tip of a long wooden dowel being moved by the experimenter in three-dimensional space. Subjects used the arm ipsilateral to the ECoG electrode implantation. Each reach was either terminated by touching the dowel with the subject's pointing index finger (subject 1, session 1; subject 2; subject 3) or with an index-thumb pinch (subject 1, session 2). The subject returned his or her hand to a comfortable resting position following completion of each reach to the target. The subject rested his or her hand in the home position for a variable amount of time (0–16.7 seconds), after which the target was moved to another point in three-dimensional space. The position of the target in three-dimensional space was determined by the experimenter in an attempt to probe the natural workspace of the subject's upper limb as thoroughly as possible. The target tended to be placed in front of and above the subject's rest position, but varied in the lateral (i.e., left or right) directions. The durations of the reaches performed by subjects 1, 2, and 3 were 3.2–6.2, 2.0–5.0, and 1.2–5.3 seconds, respectively, with median durations of 4.7, 3.0, and 2.7 seconds. The volume encompassed by the subject's workspace is detailed in S1 Table. A visual depiction of the workspace and experimental setup is shown in Fig. 3, along with example subject kinematics. The subject was cued to begin each reach once the target had stopped moving to its new location. Our analysis included two blocks of trials each from subject 1 and 3 and three blocks of trials from subject 2; details are listed in S2 Table. The cuing dowel was tracked using the same Optotrak system that was used simultaneously to track the subject's shoulder and hand, with a sampling frequency of 100 Hz. Because the tracked target tip was occluded during manipulation by the subject, the position of the distal tip was estimated using proximal sensor locations and the known, fixed distance between the sensors and the end of the dowel.
(A) Shows the weights given to each of the three dimensions (height, depth, and lateral) when computing the first PC of the movement kinematics. Each point represents the PC weight calculated using a different cross-validation. (B–D) Show different views of the experimental paradigm and example kinematics from one of the subjects' sessions. The red dashed line represents the first PC plotted in Cartesian space for that same session.
Neural Data Feature Extraction
Recorded ECoG signals were imported into MATLAB (MathWorks, Natick, MA) and analyzed using the Signal Analysis Toolbox. These signals were re-referenced to a common average reference (CAR) to avoid spatial biases from varying distances between active and reference electrodes [46]. The CAR-filtered signals were then filtered forward and backward (to avoid phase distortions) using a Hamming window and a series of 400-order FIR filters with bandpass cutoff frequencies corresponding to delta (0–4 Hz), theta (4–8 Hz), mu (7–13 Hz), beta (14–30 Hz), low gamma (30–50 Hz), high gamma 1 (70–110 Hz) and high gamma 2 (130–200 Hz) bands. The frequency limits for these bands were chosen to ensure inclusion of all relevant frequencies and to avoid 60 Hz line noise and its harmonics. The filtered signals were then down-sampled to 100 Hz to match the sampling frequency of the kinematic data. These signals were squared to yield power in each band, then passed through a smoothing integrator (1 second moving average, applied forward and backward), and log-transformed to approximate normal distributions. Transients in the neural data induced by the filtering operations at the beginning and end of each block of trials were not analyzed in this study.
In addition to frequency domain features, previous work indicated that smoothed time domain features contained information related to movement [12], [14]. These features, known as local motor potentials (LMPs) were extracted in this study using a moving average filter of two seconds, applied forward and backward to remove phase distortions. All feature extraction was done in a non-causal manner (using data from both the past and the future) to avoid introducing a group delay as a confound in our results.
Decoding Model Construction and Evaluation
The kinematics were modeled as a linear combination of the input features with a constant offset and a Gaussian noise term: , where
denotes the predicted output of the kth dimension of the kinematic output,
is the vector of feature values at time
,
is the constant offset term, and
is the vector of linear weights applied to the features, and
is zero-mean Gaussian noise. The parameters of this model were estimated using the glmfit function in MATLAB.
All models were tested under fivefold cross-validation. Pearson's correlation (r) between observed and predicted kinematics was used to quantify model accuracy. Feature selection and model training were performed on 80% of each block of trials, and the remaining 20% was used only for evaluation. Each training session contained at least 18 movements, and each testing session contained at least 4 movements. Chance decoding performance was calculated for each session by randomly shuffling the extracted and smoothed neural features 1024 times, similar to the random shuffle procedure done in [47]. This shuffled data was then used for testing and training with fivefold cross-validation. The distributions of the decoding accuracies with actual and shuffled data were compared with a Bonferroni-corrected Wilcoxon test.
Decoding accuracies were transformed into z-scores using the Fisher transformation, , where
is the number of points used to calculate the correlation. These transformed accuracies were used as inputs to a two-way ANOVA with the number of inputs and feature type (e.g., theta features only, all features) as independent factors. A post-hoc t test was done using the Dunn-Sidak correction [48] for multiple comparisons to determine which numbers of inputs and/or which feature types significantly differed, if any.
Both the neural features (after all preprocessing) and the kinematic data were normalized by subtracting the means and dividing by the standard deviations. The means and standard deviations were recomputed during every fold of cross-validation using only the training data. This normalization was done to ensure the magnitude of the features did not impact how heavily they were weighted in the decoding models.
Principal Component Analysis
While Cartesian coordinates for arm position (i.e., vertical, lateral, and depth) are orthogonal, constraints on the positioning and capabilities of the subjects resulted in an inability to fully probe these axes of the workspace independently. The kinematics of the resulting movements in this space were correlated to the extent that presenting decoding results in Cartesian axes would be somewhat misleading. We therefore used principal component analysis (PCA) to decorrelate the kinematic dimensions by projecting the 3D arm position onto another set of orthogonal axes. These decorrelated kinematics, or principal components (PCs), accounted for decreasing proportions of the variance in the original signals. The first PC was therefore oriented in the direction of largest variance (i.e. the vector along which the subject moved the most), while the second PC was oriented in the direction of the most remaining variance, etc.
The coefficients used for the PCA transform were recalculated for each fold of cross-validation using only the normalized training data. In all three subjects, decoding accuracy of the first PC was greater than the second and third PCs (p<0.05, ANOVA with Dunn-Sidak post-hoc). All but one of the models built on the second and third PCs across all subjects were indistinguishable from chance. We therefore focus on the decoding accuracies of the first PC in this study.
Model Input Feature Selection
Models with increasing numbers of input features were trained; inputs were added in decreasing order of their correlation with the kinematic variables in the training set. Correlations were calculated for the kinematic signal lagged with respect to the neural features at time lags varying between ±1 second at 50 ms intervals. A single spectral domain or time domain signal was only included once at its best lag. Results from model orders of up to 40 are reported below. Additionally, restricted models were trained from single feature types (e.g., high gamma 1 features only). Performance of these restricted models is also reported below in comparison to the full model performance for model orders one of up to 40. An analysis of the performance saturation of these models with increasing numbers of inputs was performed with a Kruskal-Wallis non-parametric one-way analysis of variance. The least significant difference post-hoc test was used to determine the first number of inputs at which additional inputs did not statistically increase performance.
The large smoothing window applied to the extracted features meant it was possible that changes in the neural features could occur substantially before or after their associated movements but still yield a high correlation with the movements. To investigate whether the feature changes preceded or followed the associated movements, we calculated correlations every 50 ms between the extracted features and the first principal component of movement offset by at most 1000 ms relative to one another.
Results
Feature Selection
The extracted signal features had substantial correlations with the first PC of the ipsilateral hand position, and the maximally correlated feature types varied across subjects. The three-dimensional kinematics and their contributions to the first PC are shown in Fig. 3. Examples of how each of the PCs relate to the original kinematics is shown in Fig. 4. Fig. 5 illustrates the correlations between different ECoG signal features (LMP and bandpass power) and the first PC of hand position in each subject. The magnitudes of these correlations are color-mapped onto the ECoG recording sites from which the features were derived. The lagged correlations were calculated at 50 ms intervals between −1 and +1 seconds relative to the hand kinematics. The peak correlations (i.e., across all lags) were averaged across sessions for each electrode. Subject 3 contained micro-ECoG electrodes, but these are not depicted due to their lack of correlation.
A representative example of the first, second, and third principal components of the kinematics from each subject, overlaid with the original kinematics.
Correlations were calculated at each feature's best lag (within ±1 second) relative to the movement data. Color-coded circles corresponding to maximum correlations, averaged across experimental sessions, are shown for each subject.
The large smoothing window applied to the extracted features meant it was possible that changes in the neural features could occur substantially before or after their associated movements but still yield a high correlation with the movements. To investigate whether the feature changes preceded or followed the associated movements, we plotted the magnitudes of the correlations between the neural features and the hand kinematics at different time offsets relative to one another (Fig. 6). Subject 1 typically had maximum correlations corresponding to neural features changing after the hand kinematic changes, particularly in session 1. This may indicate the activations were primarily in response to sensory feedback. Subject 2 and 3 had peak correlations when the changes in the features occurred before the kinematics.
Each row corresponds to one of eight extracted features from one electrode, resulting in 384, 352, and 360 rows for subjects 1, 2, and 3 respectively. The x-axis corresponds to the lag between the neural feature and the reach position. A negative lag represents changes in the neural features occurring before the corresponding kinematics, and a positive lag represents feature changes after the kinematics. Rows are ordered by the magnitude of their peak correlation. Correlations between the hand kinematics and the features were calculated every 50 ms between leads or lags of one second.
Multiple Inputs
Fig. 7 displays the models' mean correlations with the first PC of movement as a function of the number of model inputs. Input features were selected in order of decreasing correlation with the first PC of movement in the training set. Fivefold cross-validation was performed for each session, resulting in 10 folds for subject 1 and 3 and 15 for subject 2. For subject 1 and 3, the addition of more inputs to the model did not significantly improve model performance. In subject 2, models using a single input performed significantly worse (p<0.05, ANOVA with Dunn-Sidak post-hoc) than models with the best performing number of inputs.
Mean decoding accuracies of the first PC of movement across cross-validations and sessions are displayed for features chosen in decreasing order of correlation. Shading corresponds to the 95th percentile confidence interval for the mean.
The input features in Fig. 7 were selected from all signal feature types. This analysis was repeated with inputs from different ECoG recording sites restricted to a single signal feature type (e.g. LMP, delta band power, etc.). Fig. 8 shows the decoding performance for models restricted to one signal feature type, using the number of inputs that saturated performance. Most of these restricted models saturated with only a single electrode, and all but one saturated with three or fewer electrodes. Decoding performance for each signal feature type was fairly unique to each subject. In all subjects, models trained with all features significantly outperformed all single feature models (p<0.05). Smaller ranges of time lags (e.g. −100 to 0, −200 to 0, −100 to +100, −200 to +200, and 0 to +200 ms) resulted only in small reductions in performance. The low frequency features (i.e., delta, theta, mu, and beta) and LMP performed significantly better than the high frequency features (low gamma through high gamma 2) for subject 1 (p<0.05). In subject 2, there was no significant difference between models trained with high gamma 1 or high gamma 2, and these sets of models outperformed all other single signal feature models (p<0.05). In subject 3, models trained with delta, low gamma, and high gamma 2 performed statistically worse than all other single signal feature models, which had no significant difference between each other (p<0.05). All p-values reported between feature types were obtained as a part of the two-way ANOVA with Dunn-Sidak post-hoc test described in the Methods.
Between 1 and 40 recording sites were selected for each signal feature type, and the means were calculated across cross-validations and sessions. The peak accuracies are displayed here with error bars corresponding to the standard error of the mean. Numbers over each bar indicate the minimal number of features required for statistical saturation.
The decoding performance of the first PC of movement for each session is shown in Fig. 9. Results are shown for the two sessions for subject 1 and 3 and the three sessions for subject 2. Five folds of cross-validation using one, two, and nine input features are plotted for each session. During post-hoc analysis, we found that the decoding accuracy did not significantly improve (p<0.05, one-way ANOVA with least significant difference correction) if more signal features and/or recording sites were added to the one best model input for subject 1 or 3, or to the two best model inputs for subject 2. Models trained with nine inputs were chosen a posteriori as a proxy for the peak decoding accuracy across sessions and subjects, as it gave consistently high results. Decoding accuracy was found to be significantly higher than chance (p<0.05, Bonferroni-corrected Wilcoxon). Table 2 displays results for both raw kinematics and all PC's.
Distributions are displayed for the five cross-validations of the performance for one, two, and nine input features. S1–1 stands for subject 1 session 1, S1–2 stands for subject 1 session 2, etc. The maximum of the 1024 chance decoding attempts for all five folds with shuffled neural data is shown with an asterisk.
S1, S2, and S3 Videos show a virtual depiction of the actual and decoded kinematics for representative folds of cross-validation for each subject. The actual and decoded kinematics were scaled and translated by the same amount to fit in the virtual workspace. The actual and decoded positions were then sent to a virtual modular prosthetic limb through VulcanX limb control software, as described in an online setting in [7], [49]. The videos do not account for any delays which would occur in an online control scenario because of the non-causal methods we employed.
Discussion
Our findings in three different subjects show that it is possible to decode a low-dimensional representation of natural reaches from ipsilateral ECoG electrodes in the presence of damaged cortical motor systems. This decoding attained a median Pearson's correlation (r) between actual and predicted reach kinematics of 0.77, 0.73, and 0.66 for subjects 1–3. This accuracy was accomplished using as little as 133 seconds, and no more than eight minutes, of training data in each session. Furthermore, two of the subjects in this study (subject 1 and 3) had severe upper limb weakness contralateral to ECoG recording grids, arising from lesions of brain structures critical for motor control. Although subject 2 did not have contralateral limb weakness, he had a lesion of parietal lobe structures that participate in visually guided reaching and grasping movements of the upper limb [50]–[52]. Our findings indicate that the first PC of movement is robustly represented in the ipsilateral hemisphere, even in the face of damaged sensorimotor systems.
We found that high frequency signal features such as high gamma power tended to have a positive correlation with the first PC of movement, while low frequency features such as delta generally had a negative correlation. This result agrees with previous studies [10], [11], which have found that functional activation of cortex, including sensorimotor cortex, is accompanied by an event-related increase in power in the gamma band (>30 Hz) and an event-related decrease in power in lower frequencies, especially alpha (8–13 Hz) and beta (15–25 Hz) bands. The presence of this phenomenon in ipsilateral cortex agrees with results found in [43]. While typically not as robust as theta or mu, the delta band displayed notable power suppression during movements. The filters we employed were of an exceedingly high order (400), providing a roll-off steep enough to discount the possibility of bleed over from other low frequency bands. The LMP, however, did not consistently have a positive or negative correlation with the reaching movements.
There was substantial variability as to which spectral components of the ECoG signals were most predictive of the movement trajectory. For example, in subject 1 the mu band models achieved statistically higher decoding accuracies for the first PC of movement than did those constructed from high gamma 1 features; the opposite was true in subject 2, and both model types performed equivalently in subject 3. This variability in the performance of different ECoG spectral features across subjects suggests that an agnostic approach may be necessary when evaluating and selecting signal features for modeling ipsilateral movement kinematics, particularly in the face of different lesion locations and types. Much of the variability we observed was likely due to substantial differences in lesion size and location across our three subjects. Although patients with motor impairments do not frequently undergo intracranial monitoring, future studies might accumulate enough subjects to determine the spectral features that yield the best performance across subjects. Better yet, this might make it possible to better characterize the relationship between lesion characteristics (e.g. extracted from MRI via voxel based morphometry) and the performance of different spectral features. For example, more intact motor and premotor cortices in subjects 2 and 3 may have allowed for better decoding with high gamma power.
It is also possible that compensatory movements with the contralateral hemibody presented a confound [31], [34]. Indeed, reaching movements impart forces which are often accompanied by postural changes throughout the body, presenting a potential confound in any study of this sort. Our results should therefore be interpreted as the result of a movement involving complex dynamics in systems throughout the body.
The results depicted in Fig. 6 show that neural feature changes in subject 1 occurred following the onset of movement. This delayed change suggests that these neural responses arose chiefly from processing sensory feedback, perhaps because ipsilateral motor pathways were lesioned. Specifically, cortical lesions from prior resections in subject 1 probably involved hand and arm motor areas of the primary motor cortex. In contrast, subject 2 did not have motor deficits and the motor deficits in subject 3 were associated with a lesion in the left internal capsule. Unlike neural feature changes in subject 1, neural feature changes in subjects 2 and 3 preceded movement onset.
We found that decoding performance saturated with a small number of model inputs. That is, a few signal features at a few recording sites yielded the best performance, and the addition of more signal features and/or recording sites did not significantly improve performance. This finding could reflect a very coarse neural representation for ipsilateral limb movement that is widely distributed across sensorimotor systems typically specialized for contralateral limb movement. Alternatively, it could reflect transcallosal activation of these systems by homologous systems in the contralateral hemisphere. In this way, activity in sensorimotor cortex contralateral to movement (e.g. left hemisphere controlling right hand in subject 1 and 2) could modulate the activity of sensorimotor cortex ipsilateral to movement (e.g. in right hemisphere in subject 1 and 2). The encoding of this modulation during reaching movements might also be coarse, serving primarily to coordinate the movements of the two arms during reaching. Such explanations are necessarily highly speculative. The spatial resolution and anatomical distribution of our ECoG recordings were not sufficient to test these hypotheses. Nevertheless, our findings indicate that only a few electrodes are needed to achieve reach decoding from ECoG features, and that these electrodes qualitatively correspond to sensorimotor cortex as identified by ESM (Fig. 1).
We found that decoding of the second and third PCs of the movement kinematics was significantly worse than decoding of the first PC (p<0.05, ANOVA with Dunn-Sidak post-hoc), and that all but one of the models built for the second and third PCs were indistinguishable from chance. One possible explanation is that the second PC captured bi-directional movements in the lateral dimension, i.e. to the left or to the right (Fig. 4). The third PC qualitatively appeared to contain the most noise, and also often contained some bidirectional movements. These PCs stand in contrast to the first PC, in which movements only resulted in unidirectional deflections. One possible explanation for the poor accuracy with the second PC is that it is not the direction of movement, but instead its magnitude that is being decoded. It is possible that the correlation between movement effort and neural activity captured by ECoG is highly robust, such that it can even be captured in ipsilateral sensorimotor systems, even when these brain systems have been damaged. The directionality of movement, however, may present a greater challenge for decoding, especially under the circumstances of the subjects in this study. Whether the directionality of naturalistic reaching movements can be decoded from ECoG under more optimal conditions remains to be seen. Previous efforts [31], [44] have shown it is possible to decode hand movements from intact ipsilateral cortex in a constrained one- or two-dimensional experimental environment. However, decoding the directionality of movements may be more challenging in the setting of natural 3D movements, especially when performed by subjects with lesions of sensorimotor systems. While a preliminary analysis with ANN's did not improve decoding accuracy, there may be non-linearities within the data that could allow for directionality to be better decoded.
The ability to decode motor commands from ipsilateral sensorimotor systems, even when lesioned, may facilitate the development of brain-machine interfaces (BMIs) for a wider population of patients with upper limb paralysis from brain lesions such as stroke, trauma, or surgical resection of a tumor. For example, upper limb function could be partially restored by low dimensional control of a neuroprosthetic device in which ECoG electrodes are implanted over healthy ipsilateral cortex. Alternatively, electrodes could be implanted in and around cortical sensorimotor areas that have been damaged, in order to control the affected, contralateral upper limb. An important advantage of this approach is that it avoids the risks of surgically implanting chronic indwelling electrodes in healthy cortical areas controlling the patient's only functioning upper limb. In addition, implantation of ECoG electrodes in the lesioned hemisphere can be done in conjunction with other invasive interventions, such as infusions of stem cells and growth factors encouraging neurogenesis and neuroplasticity, thereby sharing the risk with other experimental therapies.
Our results suggest that it may also be possible to bootstrap a BMI system controlling contralateral arm movements by first using it to control ipsilateral arm movements. For example, the movements of the ipsilateral limb under BMI control could be mirrored in the contralateral arm using robotic assistance or functional electrical stimulation. This could provide an opportunity for development of contralateral arm control through functional reorganization and even neurogenesis. The neural activity generated while training the BMI could potentially facilitate neurobiological interventions through activity-dependent neuroplasticity, allowing the patient to eventually wean off ipsilateral arm control as control of the contralateral arm is restored. This might expedite the training process for BMIs by exploiting the more intact movements of the patient's unaffected limb.
Supporting Information
S1 Table.
Extrema of hand position relative to the shoulder in each dimension. Units are in cm, e.g. S1's hand position moved between 21.6 cm below their shoulder and 15.2 cm above it.
https://doi.org/10.1371/journal.pone.0115236.s001
(DOCX)
S1 Video.
Actual (solid) and predicted (transparent) 3d hand position for subject 1. Any time delays that would be induced by our offline methods have not been included. The sphere denotes the position of the target.
https://doi.org/10.1371/journal.pone.0115236.s003
(MP4)
S2 Video.
Actual (solid) and predicted (transparent) 3d hand position for subject 2. Any time delays that would be induced by our offline methods have not been included.
https://doi.org/10.1371/journal.pone.0115236.s004
(MP4)
S3 Video.
Actual (solid) and predicted (transparent) 3d hand position for subject 3. Any time delays that would be induced by our offline methods have not been included.
https://doi.org/10.1371/journal.pone.0115236.s005
(MP4)
Acknowledgments
The authors are grateful to Zachary Huff, Vishwa Parek, Jakir Hussain, James Corines, and Nipun Mistry for their assistance with experimental preparations and brain image reconstructions. The authors are grateful to Brock Wester, Scott Swetz, Tim Gion, and Ken Fischer for the development of the Unity environment used for the visualizations provided in the supplementary videos.
Author Contributions
Conceived and designed the experiments: MSF SA HLB NVT NEC. Performed the experiments: GH MSF HLB NEC. Analyzed the data: GH MSF. Contributed reagents/materials/analysis tools: GH MSF SA WSA. Wrote the paper: GH MSF NEC.
References
- 1. Wolpaw JR, McFarland DJ, Neat GW, Forneris CA (1991) An EEG-based brain-computer interface for cursor control. Electroencephalogr Clin Neurophysiol 78:252–259.
- 2. McFarland DJ, Sarnacki WA, Wolpaw JR (2010) Electroencephalographic (EEG) control of three-dimensional movement. J Neural Eng 7:036007.
- 3. Schalk G, Miller KJ, Anderson NR, Wilson JA, Smyth MD, et al. (2008) Two-dimensional movement control using electrocorticographic signals in humans. J Neural Eng 5:75–84.
- 4. Yanagisawa T, Hirata M, Saitoh Y, Goto T, Kishima H, et al. (2011) Real-time control of a prosthetic hand using human electrocorticography signals. J Neurosurg 114:1715–1722.
- 5. Collinger JL, Wodlinger B, Downey JE, Wang W, Tyler-Kabara EC, et al. (2013) High-performance neuroprosthetic control by an individual with tetraplegia. Lancet 381:557–564.
- 6. Hochberg LR, Bacher D, Jarosiewicz B, Masse NY, Simeral JD, et al. (2012) Reach and grasp by people with tetraplegia using a neurally controlled robotic arm. Nature 485:372–U121.
- 7.
Fifer M, Hotson G, Wester B, McMullen D, Wang Y, et al. (2013) Simultaneous Neural Control of Simple Reaching and Grasping with the Modular Prosthetic Limb using Intracranial EEG. IEEE Trans Neural Syst Rehabil Eng.
- 8. Birbaumer N, Ghanayim N, Hinterberger T, Iversen I, Kotchoubey B, et al. (1999) A spelling device for the paralysed. Nature 398:297–298.
- 9.
Pfurtscheller G, Klimesch W (1991) Event-related desynchronization during motor behavior and visual information processing. Electroencephalogr Clin Neurophysiol Suppl 42 58–65.
- 10. Crone NE, Miglioretti DL, Gordon B, Sieracki JM, Wilson MT, et al. (1998) Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event-related desynchronization. Brain 121 (Pt 12):2271–2299.
- 11. Crone NE, Miglioretti DL, Gordon B, Lesser RP (1998) Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band. Brain 121 (Pt 12):2301–2315.
- 12. Schalk G, Kubanek J, Miller KJ, Anderson NR, Leuthardt EC, et al. (2007) Decoding two-dimensional movement trajectories using electrocorticographic signals in humans. J Neural Eng 4:264–275.
- 13. Kubanek J, Miller KJ, Ojemann JG, Wolpaw JR, Schalk G (2009) Decoding flexion of individual fingers using electrocorticographic signals in humans. J Neural Eng 6:66001.
- 14. Acharya S, Fifer MS, Benz HL, Crone NE, Thakor NV (2010) Electrocorticographic amplitude predicts finger positions during slow grasping motions of the hand. J Neural Eng 7:046002.
- 15. Venkatakrishnan A, Francisco GE, Contreras-Vidal JL (2014) Applications of Brain-Machine Interface Systems in Stroke Recovery and Rehabilitation. Curr Phys Med Rehabil Rep 2:93–105.
- 16. Lo AC, Guarino PD, Richards LG, Haselkorn JK, Wittenberg GF, et al. (2010) Robot-assisted therapy for long-term upper-limb impairment after stroke. N Engl J Med 362:1772–1783.
- 17. Hanlon RE (1996) Motor learning following unilateral stroke. Arch Phys Med Rehabil 77:811–815.
- 18. Schaechter JD (2004) Motor rehabilitation and brain plasticity after hemiparetic stroke. Prog Neurobiol 73:61–72.
- 19. Brown JA, Lutsep HL, Weinand M, Cramer SC (2006) Motor cortex stimulation for the enhancement of recovery from stroke: a prospective, multicenter safety study. Neurosurgery 58:464–473.
- 20. Kleim JA, Bruneau R, VandenBerg P, MacDonald E, Mulrooney R, et al. (2003) Motor cortex stimulation enhances motor recovery and reduces peri-infarct dysfunction following ischemic insult. Neurol Res 25:789–793.
- 21. Thored P, Arvidsson A, Cacci E, Ahlenius H, Kallur T, et al. (2006) Persistent production of neurons from adult brain stem cells during recovery after stroke. Stem Cells 24:739–747.
- 22. Oki K, Tatarishvili J, Wood J, Koch P, Wattananit S, et al. (2012) Human-induced pluripotent stem cells form functional neurons and improve recovery after grafting in stroke-damaged brain. Stem Cells 30:1120–1133.
- 23. Veizovic T, Beech JS, Stroemer RP, Watson WP, Hodges H (2001) Resolution of stroke deficits following contralateral grafts of conditionally immortal neuroepithelial stem cells. Stroke 32:1012–1019.
- 24. Wu W, Kulkarni JE, Hatsopoulos NG, Paninski L (2009) Neural decoding of hand motion using a linear state-space model with hidden states. IEEE Trans Neural Syst Rehabil Eng 17:370–378.
- 25. Pistohl T, Ball T, Schulze-Bonhage A, Aertsen A, Mehring C (2008) Prediction of arm movement trajectories from ECoG-recordings in humans. J Neurosci Methods 167:105–114.
- 26.
Gunduz A, Kwon JP, Sanchez JC, Principe JC (2009) Decoding Hand Trajectories from ECoG Recordings via Kernel Least-Mean-Square Algorithm. 2009 4th International Ieee/Embs Conference on Neural Engineering: 260–263.
- 27. Hochberg LR, Serruya MD, Friehs GM, Mukand JA, Saleh M, et al. (2006) Neuronal ensemble control of prosthetic devices by a human with tetraplegia. Nature 442:164–171.
- 28. Pineda JA, Allison BZ, Vankov A (2000) The effects of self-movement, observation, and imagination on mu rhythms and readiness potentials (RP's): toward a brain-computer interface (BCI). IEEE Trans Rehabil Eng 8:219–222.
- 29. Filimon F, Nelson JD, Hagler DJ, Sereno MI (2007) Human cortical representations for reaching: mirror neurons for execution, observation, and imagery. Neuroimage 37:1315–1328.
- 30. Suminski AJ, Tkach DC, Fagg AH, Hatsopoulos NG (2010) Incorporating feedback from multiple sensory modalities enhances brain-machine interface control. J Neurosci 30:16777–16787.
- 31. Ganguly K, Secundo L, Ranade G, Orsborn A, Chang EF, et al. (2009) Cortical representation of ipsilateral arm movements in monkey and man. J Neurosci 29:12948–12956.
- 32.
Barnes S (1901) Degenerations in hemiplegia: with special reference to a ventro-lateral pyramidal tract, the accessory fillet and Pick's bundle. Brain 24.
- 33. Yarosh CA, Hoffman DS, Strick PL (2004) Deficits in movements of the wrist ipsilateral to a stroke in hemiparetic subjects. J Neurophysiol 92:3276–3285.
- 34. Cisek P, Crammond DJ, Kalaska JF (2003) Neural activity in primary motor and dorsal premotor cortex in reaching tasks with the contralateral versus ipsilateral arm. J Neurophysiol 89:922–942.
- 35. Georgopoulos AP, Schwartz AB, Kettner RE (1986) Neuronal population coding of movement direction. Science 233:1416–1419.
- 36. Moran DW, Schwartz AB (1999) Motor cortical representation of speed and direction during reaching. J Neurophysiol 82:2676–2692.
- 37. Reina GA, Moran DW, Schwartz AB (2001) On the relationship between joint angular velocity and motor cortical discharge during reaching. J Neurophysiol 85:2576–2589.
- 38. Vargas-Irwin CE, Shakhnarovich G, Yadollahpour P, Mislow JM, Black MJ, et al. (2010) Decoding complete reach and grasp actions from local primary motor cortex populations. J Neurosci 30:9659–9669.
- 39. Anderson NR, Blakely T, Schalk G, Leuthardt EC, Moran DW (2012) Electrocorticographic (ECoG) correlates of human arm movements. Exp Brain Res 223:1–10.
- 40. Nakanishi Y, Yanagisawa T, Shin D, Fukuma R, Chen C, et al. (2013) Prediction of Three-Dimensional Arm Trajectories Based on ECoG Signals Recorded from Human Sensorimotor Cortex. PLoS One 8:e72085.
- 41. Chen C, Shin D, Watanabe H, Nakanishi Y, Kambara H, et al. (2013) Prediction of hand trajectory from electrocorticography signals in primary motor cortex. PLoS One 8:e83534.
- 42. Wang W, Collinger JL, Degenhart AD, Tyler-Kabara EC, Schwartz AB, et al. (2013) An electrocorticographic brain interface in an individual with tetraplegia. PLoS One 8:e55344.
- 43. Wisneski KJ, Anderson N, Schalk G, Smyth M, Moran D, et al. (2008) Unique cortical physiology associated with ipsilateral hand movements and neuroprosthetic implications. Stroke 39:3351–3359.
- 44. Bundy DT, Wronkiewicz M, Sharma M, Moran DW, Corbetta M, et al. (2012) Using ipsilateral motor signals in the unaffected cerebral hemisphere as a signal platform for brain-computer interfaces in hemiplegic stroke survivors. J Neural Eng 9:036011.
- 45. Duncan JS, Papademetris X, Yang J, Jackowski M, Zeng X, et al. (2004) Geometric strategies for neuroanatomic analysis from MRI. Neuroimage 23 Suppl 1S34–45.
- 46. McFarland DJ, McCane LM, David SV, Wolpaw JR (1997) Spatial filter selection for EEG-based communication. Electroencephalogr Clin Neurophysiol 103:386–394.
- 47. Ojakangas CL, Shaikhouni A, Friehs GM, Caplan AH, Serruya MD, et al. (2006) Decoding movement intent from human premotor cortex neurons for neural prosthetic applications. J Clin Neurophysiol 23:577–584.
- 48.
Sidak Z (1967) Rectangular confidence regions for the means of multivariate normal distributions. Journal of the American Statistical Association: pp.626–633.
- 49. McMullen DP, Hotson G, Katyal KD, Wester BA, Fifer MS, et al. (2014) Demonstration of a semi-autonomous hybrid brain-machine interface using human intracranial EEG, eye tracking, and computer vision to control a robotic upper limb prosthetic. IEEE Trans Neural Syst Rehabil Eng 22:784–796.
- 50. Levine DN, Kaufman KJ, Mohr JP (1978) Inaccurate reaching associated with a superior parietal lobe tumor. Neurology 28:555–561.
- 51. Galletti C, Fattori P, Kutz DF, Battaglini PP (1997) Arm movement-related neurons in the visual area V6A of the macaque superior parietal lobule. Eur J Neurosci 9:410–413.
- 52. Chapman H, Gavrilescu M, Wang H, Kean M, Egan G, et al. (2002) Posterior parietal cortex control of reach-to-grasp movements in humans. Eur J Neurosci 15:2037–2042.