RQ1. What are the key criteria to establish a taxonomy of hybrid BCIs?

• To provide a structural methodology for categorizing the current hybrid BCI studies
• To clarify the current research limitations for future research directions

To address RQ1, we summarized and clarified the criteria, such as brain signal measurement types, role and mode of operation, strategy, signature, and modality of BCI systems which were used in the previous studies to categorize the distinct characteristics of hybrid BCIs. Based on the literature review, we proposed a taxonomy of hybrid BCIs to provide a structural methodology to categorize the current hybrid BCI studies with respect to 1) diversity of input signal, 2) role of operation, 3) mode of operation, 4) mental strategy, 5) brain signal signature, and 6) stimulus modality. The proposed taxonomy, which includes the most important features in a BCI system can help BCI researchers 1) classify hybrid BCIs systematically, 2) understand hybrid BCI studies at a glance, and 3) choose effective hybrid BCI types according to application goals [43–46]. Afterwards, we presented the grounds for an argument which elaborates on which combinations would be appropriate for certain environments and conditions based on the advantages and disadvantages of each hybrid BCI system. The proposed taxonomy can also clarify the current research limitations for future research directions.

RQ2. What are the key evaluation dimensions of usability in BCI usability studies?

• To classify and summarize studies related to BCI usability according to task and measurement characteristics
• To drive suggestions to evaluate usability of hybrid BCIs

To address RQ2, we categorized BCI usability studies according to task and measurement characteristics. From the results of classification, we proposed usability dimensions for BCI systems. Then, we suggested further research related to usability evaluation of BCI and hybrid BCI systems considering ergonomic issues of BCIs and hybrid BCI systems. The proposed usability dimensions can help researchers and practitioners 1) understand BCI studies related to usability evaluation, 2) choose proper metrics for usability evaluation, and 3) evaluate usability of hybrid BCIs as well as general BCIs. Also, suggestions of future research directions in this study can be helpful in establishing research directions and gaining insights under the perspectives of ergonomics.

The remainder of this paper is organized as follows: Study 1 described an elaborated taxonomy based on the systematic literature review, followed by categorized current research studies according to the proposed taxonomy. Study 2 presented usability evaluation metrics for BCIs and hybrid BCI systems. Finally, the following sections explained opportunities for further research of hybrid BCIs, and presented the major conclusions.

Information sources.

Various online databases were searched in this study: (1) IEEE Xplore to provide an electrical/electronic engineering perspective, (2) PubMed to provide a medical perspective, (3) Engineering Village to provide an engineering perspective, (4) Web of Science to provide a cross-disciplinary perspective, and (5) Scopus to provide a broad-spectrum perspective.

Inclusion and prescreening criteria.

Inclusion criteria were journal articles written in English from 2007 to 20 December 2016, since the first journal article related to the hybrid BCI system was published in 2007 [31,32]. Other publication forms (e.g., proceeding papers, unpublished working papers, master’s and doctoral dissertations, newspapers, and books, etc.) were not included. Since journal articles indicate a high level of research, journal articles can help both practitioners and academicians to obtain knowledge and spread their study findings. Keywords used in search engines were 1) “Hybrid” and “Brain computer interface”, 2) “Hybrid” and “Brain machine interface”, 3) combinations of either “Brain computer interface” or “Brain machine interface” with “electromyography,” “heart rate”, gaze, “eye tracker”, and “speech recognition”. After conducting the keyword search, 50, 164, 302, 345, and 331 articles were found from each search engine respectively, and then addition records were identified through other sources. Fig 1 shows the combinations of keywords and the number of studies for each keyword combination from the online databases. The total number of papers in each search engine was smaller than the summation of each keywords as shown in Fig 1, because some papers were found more than one time with different keywords. For example, the paper titled “Quantitative evaluation of a low-cost noninvasive hybrid interface based on EEG and eye movement” [71] was searched with keywords “hybrid & brain computer interface”, as well as with keywords “brain computer interface & eye tracker” in the IEEExplore. After the keyword search, duplicates were removed, and 527 articles remained. Those articles were screened based on titles and abstracts which were related to hybrid BCI topics, and 163 research studies were remained.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. PRISMA flow diagram of hybrid BCIs.

https://doi.org/10.1371/journal.pone.0176674.g001

Eligibility criteria.

Prescreened articles were checked for the eligibility via full-text screening by following analyses of populations, interventions, comparisons, outcomes, and study design (PICOS) [69]:

• Populations: Studies conducted with human subjects for any age, gender, or clinical conditions met the inclusion criteria, but any studies with non-human subjects, such as primates, were excluded.
• Interventions: All non-invasive hybrid BCI systems including at least one BCI system satisfied the eligibility.
• Comparators: Any study conditions such as multiple groups, single group, or case studies, were considered, because the main objective of Study 1 was proposing a hybrid BCI taxonomy method.
• Outcomes: All studies including classification procedures to achieve the main goal of BCIs met the inclusion criteria. However, some neuroscience studies that did not include classification results, but only examined the characteristics of the brain signal were excluded.
• Study designs: Any study designs had applied multiple systems including at least one BCI system were selected for further review.

Taxonomic criteria for hybrid BCIs.

After eligibility screening, 75 studies remained for the review from the initial 527 candidates. The selected journal articles were utilized to investigate important BCI features for taxonomic criteria of hybrid BCIs, and to propose a novel hybrid BCI taxonomy method (displayed in the following sections). Afterwards, the characteristics of each type, such as advantages and disadvantages, experimental environment, and applications, were discussed. Fig 1 shows the flow diagram of PRISMA with the results of keyword searching.

From the full-text review of articles, the following BCI features were found:

• Diversity of input signal: Single brain signal, multiple brain signals, brain signal(s) with physiological signals, and brain signal(s) with other signals from external devices
• Mental strategy: Selective attention and operant conditioning
• Stimulus modality: Visual, auditory, tactile, and operant conditioning
• Brain signal signature: Transient, steady-state, and different cognitive efforts
• Role of operation: Simultaneous and sequential
• Mode of operation: Synchronous and asynchronous

Based on the selected BCI features, hybrid BCI systems can be classified in terms of 1) the source of the signals, 2) the characteristics of the signal, and 3) the characteristics of operation in each system. Thus, each feature will be used as a taxonomic criterion in the following sections, and be utilized as a basis for the proposed taxonomy of hybrid BCIs. Besides the aforementioned features, brain signal recording methods, also known as brain imaging methods, are also an important BCI feature. Brain imaging methods can be categorized as non-invasive methods including EEG, fNIRS, and functional magnetic resonance imaging, and invasive methods including electrocorticography and intracortical neuron recording. In this review, only non-invasive methods were discussed due to limited applications of invasive technologies requiring surgical interventions [11].

Diversity of input signal.

In the hybrid BCI paradigm, a brain signal can be combined with other brain signal(s), physiological signal(s), or external signal(s). For example, two or more brain imaging methods can be combined in a hybrid BCI such that brain signals from an EEG and fNIRS system in order to take advantage of each brain imaging technology [41,72–75]. Some research studies applied other physiological signals, such as electromyography (EMG) [32,76], electrooculogram (EOG) [77,78] and electrocardiography (ECG) [31,79] to brain signal(s) to address common limitations of brain signals, such as lower amplitude, non-stationarity, and vulnerability to muscle artifact. In addition, external signals can be added to support BCI systems including eye-tracking [71,80], a gyroscope [81], a position sensor [82], and a joystick [83,84]. Single BCI systems using one brain signal can also be classified into hybrid BCIs by combining two brain signal signatures such as Event-related Potential (ERP) and sensorimotor rhythm (SMR) induced by Motor Imagery (MI) [85,86], ERP and Steady State Evoked Potential (SSEP) [87,88], and SMR and SSEP [89,90]. The different brain signal signatures in the hybrid BCI will be discussed in the following section.

In this review, a hybrid system that combined a brain signal with other brain signal(s) was defined as a homogeneous hybrid BCI system, while one combined with other physiological signals (non-neurological), or external signals was defined as a heterogeneous hybrid BCI system in terms of the signal diversity. Fig 2(A) illustrates a flow diagram to categorize hybrid BCIs with respect to different signal types.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Flow diagram of taxonomy for hybrid BCIs.

https://doi.org/10.1371/journal.pone.0176674.g002

Characteristics of Different Input Signal: As defined in the previous section, hybrid BCIs can consist of multiple brain signals, brain signal(s) with other non-neurological signal(s), or even a single brain signal. Firstly, the advantages of a single-brain signal approach are simple and easy to measure with a single brain imaging device [38,40]. Meanwhile, a multi-brain signal acquisition approach can resolve the inherent limitations of individual brain imaging methods [41]. For example, EEG and fNIRS could be used complementary to one another to measure brain signal features, because EEG has high temporal resolution and low spatial resolution while fNIRS has high spatial resolution and low temporal resolution [41,72–75]. A multi-physiological acquisition method has the advantage of higher classification accuracy due to not only the application of the classification result with additional physiological signals, but also the high signal-to-noise ratio of EMG and EOG signal [32,76–78]. Contrary to the combinations of physiological signals, external inputs including joysticks, eye trackers, and gyroscopes are directly utilized as a controller by modulating hand or body movements for directional applications such as a navigation [59], robot control [91] and game control [83]. However, multiple physiological signals and external inputs have limitations on the usage because of the need for physical movement, which can lead to electrode drift noise and muscle artifact on brain signals [32]. Table 1 shows the combinations of input signals and the number of studies for different diversities of input signals, and as shown in the table, most of the previous hybrid BCI research has been studied using a single-brain signal; and multiple physiological signals, combined with external inputs, and multiple brain signals follow in the order.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. The number of studies with each combination of biosignal.

https://doi.org/10.1371/journal.pone.0176674.t001

Experiment Paradigm and Applications: The experimental environment varies with respect to a goal of research studies. The studies using the single brain signal and multiple brain signals cover general goals of BCI research. However, the research studies utilizing multiple physiological signal and external inputs usually involve physical movements including hand, head, and eyeball movements either to detect subject status or to improve the BCI performance. For example, Li and Chung [81] analyzed EEG signals for different attention levels and head-movement for yawning and rubbing face to detect a driver’s drowsiness in a driving simulation environment. Ma et al. [77] used eye-movements including blinking, frowning, winking, and gazing, to select a target action between different actions of robots by analyzing EOG signals, while EEG signals were utilized to control a robot according to the selected action. On the other hand, Park & colleagues [92] distinguished between navigational intentions (searching images to obtain the information) and informational intentions (finding a predefined target from images) by analyzing eye tracking data and EEG signal, and the authors reported that the classification accuracy of combining eye movement and EEG features showed higher accuracy than that of the eye movement feature and EEG features alone (90.9%, 85.8%, and 83.9%, respectively). From the literature review, we found that the role of each signal varies in experimental conditions, and the physiological signals were usually utilized as a selector or switch to support the neurological signal [76,93,94] while external inputs utilized direct control [83,84].

Mental strategy, signal signature, and stimulus modality.

Brain signal can be either evoked by a stimulus or modulated by operant conditioning with respect to mental strategy [32,82,83]. Stimulus evoked brain signal requires selective attention on stimulus such as visual, tactile, and auditory modalities. On the contrary, operant conditioning does not depend on an external stimulus, but can be modulated by operant modalities, such as movement related efforts, attention, mental speech, and memory tasks.

In selective attention, stimuli can be categorized into either steady-state, or transient signature. The former can evoke SSEPs [89,90], and the latter elicits either ERPs or motion-onset (visual) evoked potentials (mVEPs) with different sensory modalities [39,80,95,96]. The mVEP signature was first employed by Guo et al. [97], and the motion-onset VEP-based BCI has the advantages of less visual fatigue and discomfort compared to other visual-based BCI systems [98]. For operant conditioning, Slow Cortical Potential (SCP) signatures can be modulated via different operant modalities. However, many researchers have differentiated SCP modulated by mental tasks from either movement related efforts or different attention levels elicited brain signal patterns [99–101]. The former is classified as an SMR signature evoked by movement related efforts, while the latter is known as a μ-rhythm signature. The movement related efforts include motor execution, movement attempt, and MI. Motor execution indicates actual a physical movement [81,102,103], while MI is mental movement imagination [40,89,90]. Movement attempt is a special case of motor execution only occurring during a motor attempt that involves paralyzed body parts [82,104]. Also speech and music imageries can be decoded to different brain signal signatures [79,105], and these brain signal signatures were categorized as SCP in this review. Fig 2(C) illustrates the flow diagram used to categorize hybrid BCIs with respect to different mental strategies and brain signal signatures.

In the hybrid BCI paradigm, either one stimulus or multiple stimuli with respect to sensory pathway such as visual (e.g., Steady-State Visual Evoked Potential or SSVEP), tactile (e.g., Steady-State Somatosensory Evoked Potential or SSSEP), and auditory (e.g., steady-state auditory evoked potential) modalities can be utilized. Similarly, ERP, also known as P300, can include visual, tactile, and auditory stimuli. If the hybrid BCI includes SSVEP and visual P300, then this hybrid BCI system is categorized as single modality [38,106]. On the contrary, if the hybrid BCI consists of SSVEP and SSSEP, then this system has multi-modality. In this review, operant conditionings including cognitive efforts, MI, and μ-rhythm, also belong to stimulus modality, because these conditionings can be deal with internal stimulus elicited by mental tasks. Fig 2(B) shows the flow diagram for categorizing hybrid BCIs with respect to stimulus modalities.

Characteristics of Different Strategies, Signatures, and Modalities: There are two main mental strategies including the selective attention such as SSEP and ERP, and operant conditioning such as SMR, SCP, and μ-rhythm [107]. Since selective attention only requires either focusing on continuous stimuli or counting transient events, the advantages of selective attention are 1) it is easier to perform BCI tasks, 2) a shorter training time is required, and 3) a higher classification accuracy especially with visual stimulus than operant conditioning exists [87,106]. However, this approach relies on external stimuli such as visual, tactile, and auditory. On the contrary, operant conditioning does not require any stimulus to evoke brain signal, but longer training periods are usually required to achieve reasonable classification accuracy [101,108]. The other possible advantage of operant conditioning with SMR-based BCI tasks is the neuroplasticity by facilitating motor-related brain area [109].

As each BCI modality has different characteristics, their advantages and disadvantages are also distinct [110]. Among different brain signal modalities, the advantages of visual modality are 1) higher classification accuracy and 2) easy to apply stimuli in the experimental environment by using either an LCD or LED [37,111,112]. However, since participants attend visual stimuli to evoke brain signals, they might feel annoyed, experience eye fatigue, and get even disturbance [77]. This issue could be addressed by combining mVEP with less visual fatigue and SMR without visual stimulation in multiple modalities and signatures [39]. The advantage of tactile and auditory modalities is the dependence of visual sensory, but there are some limitations of this application, such as lower classification accuracies and difficulties increasing the number of stimuli due to the nature of tactile and auditory sensory [75,89]. Operant modality such as MI is independent from visual stimulus but shows lower classification accuracy and needs longer training. For example, Allison et al. [111] utilized two different signatures and modalities in one EEG signal, SMR and SSVEP, for simultaneous two dimensional cursor control, and Combaz and Van Hulle [38] combined multiple signatures in a single modality, SSVEP and visual P300, to improve ITR. Yin et al. [41] measured both EEG and fNIRS brain signals to improve the performance of decoding SMRs evoked by MIs, while Lim et al. [35] combined SSVEP with eye tracking data to prevent errors in a SSVEP-based BCI system.

Experiment Paradigm and Applications: Visual-based BCIs are some of the more commonly studied BCI research studies because of the reliable results and short training time, and most of them are for a BCI speller [87,113,114]. Furthermore, Li et al. [115] proposed a multi-signature hybrid BCI system with visual P300 and SSVEP to control a wheelchair, while Pan et al. [88] to detect awareness of patients with disorders of consciousness in a similar BCI system. In addition, visual-based BCIs can be applied to games and virtual reality environments because visual stimuli can easily be embedded in these systems. There are also many SMR-based BCI studies that set out to control external devices such as a neuro-prosthesis, wheelchair, and exoskeleton [116,117], and the target users of these BCI systems are usually severely disabled patients. However, due to the limitations of SMR-based BCI systems discussed in the previous section, some SMR-based BCI research applied other modalities such as visual [34,86,118] and tactile [89,119] to increase the accuracy and robustness of the classification algorithms. Table 2 summarizes applications of each modalities with external signals braced with parentheses from the literature review.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. Applications of each modality in hybrid BCIs.

https://doi.org/10.1371/journal.pone.0176674.t002

Role and mode of operation.

A role of each system in the hybrid BCI can be different in terms of usages [34]. For example, both systems can play the same role simultaneously to achieve a certain goal. In this case, multiple input signals from different systems can be fed into one classification algorithm, or each decision can be fused to make one final decision. Yin et al. [113] utilized both SSVEP and visual P300 simultaneously to increase classification accuracy and ITR of a BCI speller, while Jiang et al. [131] fused MI features from EEG signals and gaze directions from EOG signals to improve the BCI performance for a multi-class target selection. It is also possible that one system can initiate the other system as a switch by detecting a distinct signal. For instance, SSVEP-based BCI can be used to turn an MI-based BCI on, then the MI-based BCI controls a hand orthosis to complete a hand grasping task [126]. Furthermore, both BCI systems can play different roles simultaneously to achieve different goals such as two-dimensional control [39,40,120]. For example, Ma et al. [39] combined SMR and mVEP signatures simultaneously to realize a 2-dimensionla cursor control, while Li et al. [120] utilized visual P300 and SMR signatures for the similar task. Finally, one system can decide to choose a certain function as a selector, then the other system can control levels in a specified sequence [124]. Fig 2(D) shows the flow diagram for categorizing hybrid BCIs with respect to the role of operation, and Fig 3 represents three different roles of operation.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Schematic diagram for three different roles of operation.

https://doi.org/10.1371/journal.pone.0176674.g003

Any BCI experiment falls into two different modes of operation. One mode of operation is defined as the BCI experiment being conducted under a synchronous, cue-paced scenario, and the other is under an asynchronous, self-paced manner. Synchronous experiments rely on a certain cue with a fixed time per trial, and the BCI systems control an application or provide feedback by analyzing brain signals of the fixed time [72,116]. On the contrary, experiments under asynchronous manner do not depend upon cues, but participants conduct BCI tasks towards a certain goal at their own pace [99,126].

Characteristics of Each Operation: The advantages of simultaneous processing are that 1) one-time signal processing is required, 2) multiple decisions can be made at one time, such as 2-demensional cursor control [120], and 3) the classification accuracy can be increased by facilitating two classification results complementarily. However, this approach also has limitations in applications of some BCI systems which involve multiple tasks occurring in a sequence such as sending an email [123] and controlling different levels between multiple functions [126]. However, hybrid BCI systems with a sequential processing mode can address these issues but this approach requires multiple BCI tasks to complete multiple steps.

Experiments under the synchronous mode are usually utilized to build parameters of classification algorithms [117,132]. Once the classifier is ready, participants can perform asynchronous experiments with goal-oriented tasks without cues. Therefore, the brain signal should be continuously monitored, and the BCI system will act only if the distinct signal is detected from the classifier during a series of tasks [102]. These two modes can be applied to any BCI system, and both modes have their own advantages and disadvantages discussed in the following section. Fig 2(E) illustrates the flow diagram for categorizing hybrid BCIs with respect to the mode of operation.

Experiment Paradigm and Applications: Both operation modes can be applied to many BCI applications. Xu et al. [87] applied P300 and SSVEP simultaneously for a BCI speller, and achieved higher accuracy and ITR than each BCI systems by combining two BCI features. Simultaneous BCI can be utilized for 2-demensional space control, and Allison et al. [111] showed promising results by apply MI for vertical movement and SSVEP for horizontal movement at the same time. Malechka et al. [94] proposed a BCI system with graphical User Interface (UI) to control activities of daily living applications by analyzing eye-tracker, SSVEP, and MI signals. In this system, participants select a device via eye tracker, select submenus such as volume and channel of radio by either SSVEP or MI. Kim et al. [100] built a BCI system to fly a drone in a real environment by using eye tracking and EEG data. EEG data was used to select two different modes, such as horizontal movements and vertical movements including turning. Afterwards, the results of the eye-tracker were used to control the drone. From the literature review, the applications of simultaneous mode were usually applied either to control two functions at one time or to increase classification accuracies and ITR. BCI systems using sequential processing were used either to control multiple functions with steps or to apply a switch function in asynchronous mode.

The experiments under the synchronous manner are useful to validate a proposed BCI system and to find user-specific classification parameters using offline analysis [117,132]. Since a synchronous BCI relies on external cues and a fixed BCI task time is required to perform signal processing, this approach has a practical limitation to apply in real-life tasks utilizing continuous controls [116]. This issue can be addressed by applying asynchronous techniques in which participants conduct BCI tasks towards a certain goal at their own pace without cues [99,126]. In this paradigm, the decisions are made in (near) real-time by classifying brain signals with classification parameters defined in the offline analysis. However, due to the non-stationarity of EEG signals [18], the classification accuracies under an asynchronous based system were usually lower than that of a synchronous approach.

Table 3 shows the classified hybrid BCI types from the selected 75 journal articles according to the proposed taxonomy, and non-neurological signals used in each study are braced with parentheses similar to Table 2.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Classified hybrid BCI types according to the proposed taxonomy.

https://doi.org/10.1371/journal.pone.0176674.t003

Information sources and inclusion criteria.

The online databases searched in Study 2 were identical to Study 1. Original studies that conducted usability evaluation on BCI with subjective measures or performance measures were included. This study covers only journal articles published in English. Other publication forms (e.g., proceeding papers, unpublished working papers, master’s and doctoral dissertations, newspapers, and books, etc.) were excluded.

Full-text review of articles includes the following analyses of populations, interventions, comparisons, outcomes, and study design:

• Populations: All populations were considered, but non-human subjects were excluded.
• Interventions: We screened for studies that measured the usability of BCI using subjective measures or performance measures.
• Comparators: We did not screen for studies that included the results of some kind of functional control comparison. Because our research goal focused on the investigation of usability measure we did not consider studies that included experiment designs with a control group or treatments.
• Outcomes: Studies were required to include objective/subjective measures of efficiency, effectiveness, satisfaction. For studies that did not explicitly include such a component, we screened for those studies the outcomes of which could be logically linked with usability.
• Study designs: Given the nature of BCI research, there is a narrow range of study designs employed. Most frequently we encountered small-n, within-subject designs. Thus, we did not screen for design type, beyond the requirement already stated that the design involve human subjects.

Search strategy and limits.

As few studies related to usability evaluation of hybrid BCIs have been conducted, we focused on reviewing previous studies related to usability evaluation of BCI. Afterwards, the results of the review were utilized to suggest usability evaluation metrics for hybrid BCIs. Thus, the general search strategy included key terms such as “Brain computer interface”, and “Usability”.

Participants.

Among the 31 studies that we reviewed, the average number of participants was 12.71, with a maximum of 39 and a minimum of 1. Some studies engaged a very small group of subjects because they targeted a population of disabled persons [52,143–145]. The age of participants varied depending on the study. Twenty out of 31 articles explicitly disclosed the age information of their subjects; one article did not mention participants’ age [62,146]. Based on the known information, the oldest participant was 73 years old [143], while the youngest one was 16 [147]. Subjects with disabilities tend to be older and subjects in the healthy group tend to be younger. Six articles reported the gender distribution of their participants [65,66,76,148–150]. Among most of them, the percentage of male subjects was more than 50%. Sixteen studies recruited participants with disabilities [55,65,66,76,143–145,148–156]. The most common type of disability was sclerosis [52,55,65,66,76,143–145,149–151,153–155]. Among them, two articles recruited multiple sclerosis patients, and nine articles recruited amyotrophic lateral sclerosis patients. Seven articles included disabled participants that had suffered from strokes [66,76,144,149,152,153]. Thirteen studies reported all healthy subjects [58,63,64,146,147,157–164], and one study did not disclose this information [62].

Study design.

Most of the studies conducted were proof of concept design or within-subject design because of the limited number of patient-subjects. In the studies, researchers introduced their newly developed BCI system and evaluated its usability with or without existing systems. Only four studies used a between-subject design and they conducted experiments with only healthy subjects [147,157–159].

Study environment.

Most of the experiments took place in a laboratory environment. Only three studies utilized a “daily life” environment (e.g, in the subject’s home) [143,152,155].

Task characteristics.

Collected articles were categorized according to the type of task which is used for usability evaluation of BCI (See Table 4). The collected articles were categorized into seven tasks:

• Spelling: the task to type given or free words via BCI systems.
• Movement control: the task to control the movement of system.
• Selection control: the task to target and choose the icons or buttons.
• Brain painting: the task to utilize the painting program.
• Mental task: kind of thinking activity such as imagination, mental calculation and etc.
• Cognitive rehabilitation task: is kind of strengthening activity of intellectual capacity.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 4. Classification of articles by task characteristics.

https://doi.org/10.1371/journal.pone.0176674.t004

According to the characteristics of the task goal, we divided tasks into two types: open task and closed task. If the user defines the outcome of the task, it is considered an open task. In the case of a closed task, the experimenter gives pre-defined goals to users [165]. For example, freely using the program is an open task experiment but using the program according to pre-defined instruction is considered a closed task design. Again, we categorized closed task experiments by the characteristics of strategy. If the user can freely choose the strategy to achieve the pre-defined goal, it is a closed self-managing task. If users have no choice and just follow the pre-defined strategy, it is a closed copying task. For example, typing words from the users’ own thought is considered a closed self-managing task but typing given words is a closed copying task.

In the BCI usability studies, spelling tasks were the most frequently used tasks. Control tasks (e.g., movement and control) were often used too. In order of frequency of use, brain painting task, mental task, and cognitive rehabilitation task were used. With consideration of type of task, closed tasks were dominantly used and closed copying task was more used than self-managing task. Type of open task is given only in the few studies which used movement control, selection control, and brain painting as a task. In studies involving the spelling and movement control task, closed copying task was used most frequently.

Measurement characteristics.

Overall, 10 evaluation tools of subjective measures were used in 31 BCI usability studies (see Table 5). NASA Task Load Index (NASA-TLX), Visual Analogue Scale (VAS), Assistive Technology Device Predisposition Assessment (ATD-PA) device form, System Usability Scale (SUS) survey, Quebec User Evaluation of Satisfaction with assistive Technology 2.0 (QUEST 2.0), IBM’s computer usability satisfaction questionnaire and Usefulness, Satisfaction, and Ease of use (USE) questionnaire were utilized. Some of them, QUEST 2.0 and SUS survey, were also used as modified versions. IBM’s computer usability satisfaction questionnaires and USE questionnaires were only used as modified versions. Lastly, some studies proposed and conducted their own evaluation tools.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 5. Classification of articles by evaluation tool of subjective measures.

https://doi.org/10.1371/journal.pone.0176674.t005

NASA-TLX is a popular mental workload assessment technique which relies on a multidimensional construct. It derives overall workload based on 6 subscales: mental demand, physical demand, temporal demand, performance, effort and frustration [166]. VAS is one of the methods of assessing a “feeling” [167]. It is usually conducted to assess the satisfaction of a system in BCI usability studies. ATD-PA device form and QUEST 2.0 are specialized subjective assessment tools used to evaluate the assistive devices. ATD-PA is a set of questionnaires used to assess the match quality experienced between the person and the assistive technology [168]. In BCI usability studies, only a set of 12 items, called ATD-PA device form, is usually utilized to ask users’ opinions of 12 aspects of using the proposed BCI system as an assistive device. QUEST 2.0 is an instrument used to evaluate users’ satisfaction with assistive technology. It contains 12 items rated on a 5-point satisfaction scale with regards to the device and services [169]. SUS survey and USE questionnaires are simple, yet effective tools used for assessing the usability of various products. SUS survey contains 10-item scale giving a global view of usability [170], and USE questionnaires contains 14-item scale consisting of four domains: satisfaction, ease of use, ease of learning, and usefulness [171]. IBM computer usability satisfaction questionnaires also measure user satisfaction with usability, but it is specialized on a computer system [172]. Questionnaire for current motivation (QCM) is the subjective assessment tool designed to measure users’ motivation with respect four motivational factors: mastery confidence, incompetence fear, challenge, and interest [173].

To find out the frequently used subjective measures in each type of task, we categorized studies by type of task and counted the number of studies separately. Among 19 spelling tasks, NASA-TLX and VAS were the most frequently used subjective evaluation tools in both closed copy tasks and self-managed tasks. In closed copy tasks, four studies proposed new subjective measures. Proposed measures varied from study to study. For example, Deravi et al. [149] evaluated aesthetic, attractiveness, cognitive workload, comfort, ease of use, effectiveness, functionality, helpfulness, operability, safety, and usefulness of system through their own developed set of questionnaires. Hohne & Tangermann [52] evaluated controllability, effectiveness, efficiency, and exhaustion; and Nam, Li, & Johnson [161] evaluated preference. Won et al. [164] evaluated comfort, and Nijboer et al. [162] did aesthetic, comfort, operability, and preference.

In movement control tasks, NASA-TLX and VAS were also most frequently used in all types of task. Among the seven studies of movement control tasks, only three studies did not use any subjective measures for usability evaluation of BCI [62,66,155]. In addition, Modified QUEST 2.0, SUS survey, and modified SUS surveys were used in the closed copying task and ATD PA device form was only used in closed self-managing tasks and QUEST 2.0 was only used in open tasks. Among nine studies using selection control tasks, seven studies used subjective measures, and one study proposed new measures in the closed copying task [149]. Only two studies did not use subjective measures in the selection control task experiments [64,155]. In the cognitive rehabilitation and brain painting tasks, all studies used subjective measures. One study using a brain painting task proposed a new measure in open task: exhaustion [52]. Among two studies using a mental task, only one study used subjective measures [147]. Weyand et al. [147] proposed the new measure “helpfulness of feedback”.

Overall, task accuracy and ITR were most frequently used for performance measures. The rest of the measures were related to brain activity (e.g., amplitude, latency), time dependent variables (e.g., task speed, task time, time for selection), and the difficulties of task completion (e.g., error rate, feasibility of finishing the task, task completion rate). Also, there were some studies that proposed new metrics for performance measures (e.g., effectiveness [152], efficiency [63,144], WS score [147]).

To find out the frequently used performance measure in each type of task, we categorized studies by type of task and counted the number of studies separately. Among 19 spelling task studies, only one study did not use performance measures to evaluate usability of BCI [58]. In both closed self-managed tasks and closed copy tasks, task accuracy and ITR were most frequently used as performance measures. Because there were more studies using a closed copy task than a closed self-managed task, performance measures, which were used in the copy task, except task accuracy and ITR, varied depending on study. In movement control tasks, all studies used performance measures. Task accuracy was most frequently used, whereas ITR was used in only one study [52]. One study of the brain painting task did not use performance measures [143].

Since the objective of this study was to establish the usability dimensions measured in BCI usability studies, we reorganized them in terms of usability dimensions. Table 6 presents a summary of 40 measured subjective usability dimensions. A preliminary inspection of Table 6 shows that the constructs of satisfaction, cognitive workload, and ease of use are most commonly measured in BCI usability studies. All of these measures were defined in the work of Han et al. [174] on the classification of performance and image/impression dimensions with slight variations.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 6. Frequency of subjective measures used in the reviewed articles.

https://doi.org/10.1371/journal.pone.0176674.t006

We reorganized performance measures (objective measures). Table 7 presents a summary of 21 performance measures. From Table 7, task accuracy and ITR are most commonly measured in BCI usability studies. The remaining performance measures varied depending on study.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 7. Frequency of performance measures used in the reviewed articles.

https://doi.org/10.1371/journal.pone.0176674.t007

Upon review of the measures’ frequency in the collected articles the three core-constructs for the measurement of usability appear to be the following [175]:

• Efficiency: Degree to which the product is enabling the tasks to be performed in a quick, effective, and economical manner, or is hindering performance.
• Effectiveness: Accuracy and completeness with which specified users achieved specified goals in a particular environment.
• Satisfaction: The degree to which a product is giving contentment or making the user satisfied.

Most subjective measures take into account Satisfaction and Efficiency. The metrics for evaluating cognitive workload by NASA-TLX were used for assessment of efficiency of BCI systems. Although various subjective measures were used and the definitions of each measure were different, the purpose of determining the usage of the measures except cognitive workload is to assess user’s satisfaction on BCI systems. Performance measures varied depending on study. An accuracy measure was typically used to assess the effectiveness of BCI systems. Among the collected articles, ITR was typically used to assess efficiency of BCI systems. Because the measures are related to how the user performs the task in a quick and effective manner, task speed time, throughput time, the feasibility of finishing the task, and time for selection can be involved in efficiency measures. A summary of usability dimensions is illustrated in Fig 5.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 5. Usability dimensions for BCI systems.

https://doi.org/10.1371/journal.pone.0176674.g005