The Berlin Brain-Computer Interface: Progress Beyond Communication and Control - PubMed (original) (raw)
Review
The Berlin Brain-Computer Interface: Progress Beyond Communication and Control
Benjamin Blankertz et al. Front Neurosci. 2016.
Abstract
The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.
Keywords: Brain-Computer Interfacing (BCI); cognitive neuroscience; covert user states; electroencephalography (EEG); implicit information; machine learning; mental workload; video quality.
Figures
Figure 1
Grand-average stimulus-aligned EEG responses to forced emergency braking during real-world (upper panel) and laboratory driving (lower panel). Potentials are visualized as topographical maps of grand-average ERPs in five temporal intervals. The stimulus onset (t = 0 ms) is the time of the brake lightbrake light flashing of the lead vehicle. Figure taken from Haufe et al. (2014a) with permission.
Figure 2
Grand-average area under the curve (AUC) scores calculated from the outputs of linear classifiers that were optimized to distinguish normal driving intervals from stimulus-aligned target intervals representing different stages of emergency braking situations. STIM denotes the onset of braking (brake light flashing) of the lead vehicle. Thick lines represent the results of the real-world driving study (Haufe et al., 2014a), while thin lines represent results obtained in the driving simulator study of Haufe et al. (2011). The distribution of pooled braking response times in both datasets is indicated by box plots showing the 5th, 25th, 50th (median), 75th, and 95th percentiles. Classification was based on (spatio-) temporal features observed prior to the decision points. Performance of combinations of modalities. Blue: EEG+EMG+Gas+Brake (electrophysiological and behavioral channels). Red: Gas+Brake (only behavioral channels). The intervals, in which the inclusion of electrophysiological channels significantly improved classification accuracy are marked as square boxes (no filling for simulated driving, light gray filling for real-world driving). Figure taken from Haufe et al. (2014a) with permission.
Figure 3
SSVEP-based paradigm for video quality assessment. Each video comprised the six textures presented in all the levels of distortion (D1, …, D6) in random order. Each texture was displayed distorted for 333 ms, followed by the undistorted form for 333 ms (D0) and the same succession was repeated four times for each level. Figure adapted from Acqualagna et al. (2015) with permission.
Figure 4
Single-trial classification of VEPs based on spatio-temporal features. (A) Grand average brain activity over all participants at channel Oz, at maximum distortion level D6. The magenta line represents “Class 1” and the gray line “Class 2.” Scalp plots underneath refer to the shaded areas in the time plot and display the magnitude of the sign − _r_2 for each channel. (B) Classification performances using shrinkage LDA for all participants (colored lines) and mean (black thick line). Figure adapted from Acqualagna et al. (2015) with permission.
Figure 5
Experimental task and impact of the experimental paradigm on task performance and peripheral physiological measures (PPM). (A) Snapshot from one of the experiments showing a subject playing the game on the touch screen. (B) Block structure of the experiment. Participants performed four runs of 24 min, each consisting of 90-s blocks of alternating low (L) and high (H) workload conditions. (C) Error rate. (D) Respiratory frequency in breaths per minute. (E) Cardiac frequency in beats per minute. (F) Electrodermal response in Galvanic skin potential. Shown are the grand averages of the mean over all L–H block pairs. The light blue shadings indicate the standard error of the mean. Due to large inter-subject differences in the average of the PPMs, the grand average and standard error were computed after subtracting the mean in the indicated bar. Thus, the plotted values represent changes from this baseline. Figure taken from Schultze-Kraft et al. (2016b) with permission.
Figure 6
Results. (A) Mean classification accuracy and standard error of the six models across subjects. (B) Added value of peripheral physiological measures. Mean classification accuracy and standard error across subjects when using only PPM features (white) and comparison to the two unsupervised predictive models before (dotted) and after (solid) augmenting with PPM features. Figure taken from Schultze-Kraft et al. (2016b) with permission. **Indicates a significance level of p < 0.01.
Figure 7
Spatial activation patterns and power envelopes of components extracted by the three EEG spatial filtering methods. The shown activation patterns (scalp maps) and power envelopes correspond to the components with the highest value of the optimization criterion of the respective method. The left and middle column show the activation patterns of components obtained from the theta (blue) and alpha (red) bandpassed data, respectively. The color coding and sign of the activation patterns were adjusted to be consistent across methods, but are arbitrary otherwise. The power envelopes (right column) are color-coded accordingly (theta: blue; alpha: red), the x-axis shows time in seconds. Due to standardizing to _z_-scores, the amplitudes of the curves do not relate to discriminative power. Figure taken from Schultze-Kraft et al. (2016b) with permission.
Figure 8
(A) Left: A standard Landolt broken ring. Right: Eight modified Landolt rings that we used in our study. (B) Illustration of the stimuli presentation flow for different conditions. Top-left: In the PU condition, stimuli appear in one step. Top-right: Three intermediate steps of stimulus evolution in time are presented for the SA condition, followed by a completely revealed stimulus. Bottom-right The dashed line indicates the order of the appearance of stimuli in the PU and SA conditions. Bottom-left Several intermediate steps in the evolution of two successive stimuli are illustrated for the MA condition. The arrows indicate the direction of their continuous motion. This illustration is simplified, since multiple objects were present on the screen during the motion condition (MA). Stimuli are enlarged in comparison to the real screen dimensions. Figure taken from Ušćumlić and Blankertz (2016) with permission.
Figure 9
Upper plot: Intra-protocol classification with an HDCA was cross-validated on the three conditions. Lower plot: For the inter-protocol classifier transfer, the HDCA was trained on the training data (PU condition) on a fixed time interval. The intermediate results of the testing datasets (conditions SA and MA), obtained for different positions of the sliding window (small boxplots), are combined to give the results indicated by the broad bars. Figure adapted from Ušćumlić and Blankertz (2016).
Figure 10
(A) Sequential centered “BCI presentation.” A simplified “mental typewriter” serves as example for an ERP-based BCI (cf. Treder et al., 2011). Different items (square, triangle, disc, pentagon) are flashed one-by-one, on the same spot on the screen. Each item stands for a (group of) letter(s). The subject selects a letter and silently counts the flashes of the corresponding item in order to direct their attention toward it. The selected (group of) letter(s) can be decoded from the EEG data using the flashes as time points of reference. (B) Item scanning in free-viewing. In an HCI scenario, words or pictograms (symbolized in the illustration) are displayed in parallel on the screen and are not flashed one-by-one. Items of particular interest for the user shall be decoded from the EEG data. The saccades (white arrow) to the items, as measured with an eye tracker, can serve as time points of reference for the EEG analysis. (C) The stimulus saliency may vary in HCI settings. An item of little saliency (here represented by a blue disc with indistinct interior) can only be recognized after a saccade when the item is in foveal vision. In contrast, a salient item (red crown) can be recognized already in peripheral vision, before a saccade toward it. A variable timing of recognition can therefore be expected with respect to the saccades, which are used as time points of reference for the EEG analysis.
Figure 11
(A) Possible trial outcomes during the experiment. See text for details. (B) Percentage of trial outcomes across stages for the four trial categories (as in panel (A)). All trial categories in one stage (bars of same color) add up to 100%. Shown is the average across subjects (error bars = SEM). Figure taken from Schultze-Kraft et al. (2016a) with permission.
Figure 12
(A) Distribution of BCI predictions time-locked to EMG onset (vertical line). The three panels show the distribution of stop signal timings in predicted button press trials (top, red) and in aborted button press trials (middle, green). The bottom panel (red and green) shows their joint distribution. The gray distribution superimposed as outline in all three panels shows the stop signal distribution in silent trials, adjusted to account for the imbalanced probability of a trial being silent (40%) or not (60%). All bins comprised intervals of 100 ms and counts were pooled across stages II and III of all subjects. (B) Accuracies of a classifier trained to detect an impending movement based on event-related desynchronization (ERD) occurring before stop signals. Bars show the mean accuracies of subjects (error bars = SEM) for four different trial types. Figure taken from Schultze-Kraft et al. (2016a) with permission. Significance above chance level is indicated by **p < 0.01 and ***p < 0.0001.
Figure 13
Regression approach for extracting neural correlates of music perception. The perception of sound is reflected in the brain signals, the power slope of the sound waves being a crucial factor. Linear Ridge Regression is applied to the temporally embedded multichannel EEG signal using the audio power slope as a target function. This results in a spatio-temporal filter (regression weight matrix) that can be applied to new data. It reduces the multichannel EEG to a one-dimensional projection that can subsequently be examined with respect to Cortico-Acoustic correlation (CACor).
Figure 14
EEG projections reflecting cortical responses to note onsets. The two examples of keyboard (A) and bass (B) show segments of an extracted EEG projection (blue) for a single stimulus presentation and a single subject and the respective audio power slope (red). Note that in the optimization procedure a time lag between stimulus and brain response is integrated in the spatio-temporal filter, and that, consequently, the EEG projections shown here are not delayed with respect to the audio power slope. Figure adapted from Sturm et al. (2015b).
Figure 15
The CACor score profile for the set of nine stimuli can be interpreted as a measure of reliability of the occurrence of significant CACor that can be compared across stimuli. The distribution of CACor scores for the set of nine stimuli is significantly correlated (r = 0.9, p = 0.005) with the distribution of coordination scores (not shown here) that quantify how strongly coordinated behavioral responses a stimulus produces (for details on the calculation of CACor and coordination scores see Sturm et al. (2015a)). The description on the right suggests that stimuli from the same category have similar CACor measures. Figure adapted from Sturm et al. (2015a) with permission.
References
- Acqualagna L., Blankertz B. (2015). Neural correlates of relevant stimuli processing for brain computer interfaces, in Engineering in Medicine and Biology Society (EMBC), 2015 37th Annual International Conference of the IEEE (Milano: IEEE; ). -PubMed
Publication types
LinkOut - more resources
Full Text Sources
Other Literature Sources