Brain Products Press Release
March 2011, Volume 38
Did you know ...!?
Towards passive BCI (3) - Accessing the covert aspects of user state and
adding contextual awareness to human-machine systems
by Thorsten Zander
>> Introduction
In the previous two articles of this series we discussed the
initial definition of BCI systems and their intended application.
Accordingly, we can say that a BCI represents a new input
modality in connection with human-machine interaction (HMI)
that can substitute for or supplement other input modalities
such as manual input. The intended primar y field of application
is support systems for severely disabled individuals. But from
the perspective of human-machine system (HMS) research in
general, a particular system could also benefit from information
on the user’s current state; its capture represents an implicit
input modality which conveys information about the state of the
user not intentionally transmitted by the latter. It would also
supply a valuable information element regarding the ongoing
context of a given HMS. This will be discussed in more detail
below.
This approach can be viewed as a modification of the general
approach of BCI whereby passively conveyed implicit information
replaces commands that are usually voluntar y and directed. In
a later section of this article I will describe a framework that
relates this approach to the definition of BCI and embeds it in a
broader framework. The resulting passive BCI approach opens up
the field for applications based on BCI technology that could be
used in both general HMS’s and the neurosciences. The passive
BCI approach allows for real-time access to information about the
current user state, in particular those aspects that are usually
hidden. In what follows below, these aspects will be referred
to as the covert aspects of user state. This information can be
utilized for enhanced HMIs, including automated adaptation, a
topic which will be discussed in the fourth and final article of
this series about BCI.
More generally, however, the passive BCI approach enables
the accessible information space in a given human-machine
system to be augmented by valuable information regarding
the context of the system. This allows contextual awareness
to be implemented in a given HMS. Accessing this information
might be especially relevant in a specific HMS application,
specifically in the technical environment that comprises the
setting for neuroscientific experiments. It could allow for
adaptive set-ups that respond directly to the subject’s internal
cognitive changes, potentially leading to new insights in the
cognitive neurosciences concerning how the brain functions in
an interactive environment.
>> Utilizing User State for Human-Computer Interaction and the
Neurosciences
Relevant information in human-machine systems includes
www.brainproducts.com
information about the state of the
technical system and the system
environment, as well as the state
of the user. In particular, cognitive
processes such as the user’s internal
interpretation of a situation are of
Thorsten Zander
considerable interest (see Figure 1).
This can be illustrated by taking a look at another type of
interaction – the interaction that takes place between humans.
One element of social interaction is explicit, namely the
intentional transmission of a message to another actor. In
addition, there is an implicit information flow. Through observing
the aspects of the user state that accompany an explicit
interaction, such as gestures, mimicry or body posture, actors
gain access to information about each others’ inner states.
Reading the interpretations and intentions of other individuals is
an important ability that involves representing and embedding
their mental states in one’s own mind, as postulated in the theory
of mind. Such information might also be relevant for constructing
more intuitive HMIs and adaptive experimental set-ups in the
neurosciences. Consequently, integrating information on the
various aspects of user state into HMIs could lead to a more
natural way for humans and machines to interact, and thereby
generate new insights into human cognition. The cognitive
aspects are of particular interest, as they could correspond to
highly complex information on current user state – something
that is mainly encoded in the human brain. One can roughly
assign these aspects of user state to two distinct groups, each
of which can carry relevant (and implicit) information.
First, there are latent cognitive processes like arousal and
fatigue, as well as more complex examples; these will be
discussed in a subsequent article.
Second, there are time-bounded cognitive processes that
neuroscience terms ‘cognitive events’.
In a system which captures the user state connected with
implicit communication, this information flow can be viewed
as input from the user into the BCI system that is not transmitted
intentionally; in other words, the flow of information can be
viewed as consisting of implicit commands. Because such
implicit commands are generated automatically in the course
of the process of interaction, the flow of information increases
even though the mental effort expended by the user does not
increase. Hence making use of cognitive user state information
represents a highly efficient tool for the general enhancement of
BCIs or HMSs.
Unfortunately, these aspects of the cognitive user state are not
easily captured by technical systems, as we shall discover below.
page 6 of 14
Brain Products Press Release
March 2011, Volume 38
>> Accessing User State with Psycho-Physiological Measures
A user state includes covert elements that are hard to access
from outside. Examples of such elements are the physiological
processes taking place in the human body, or the previouslymentioned processes of cognition. There are approaches that
utilize overt measures to derive information about these aspects
of user state, such as user behavior, as well as approaches that
extract information corresponding to particular aspects of the
user state. In addition, physiological measures like haptic data
or eye gaze have been shown to provide useful information on
user state. Even so, the scope of application of these methods
is limited, as they can only generate information that is weakly
correlated to the current user state. This is the justification
for terming these user-state elements covert aspects of user
state (CAUS), by analogy with covert attention, which is a term
commonly used by the eye-tracking research community.
>> Covert Aspects of User State
A covert aspect of user state is an internal process in the user
that overt measures can only detect with weak reliability. As
the user’s cognition is inherently hard to access using overt
measures, a large proportion of cognitive processes are CAUS.
Hence we need a sophisticated and continuous measure for
accessing those aspects and turning them into HMI input, as was
proposed in the previous section. A potentially suitable measure
for this is the EEG, which has high temporal resolution. Using
brain-computer interface technology, EEGs could be used for
the real-time detection and interpretation of distinct cognitive
processes that are usually covert. When applied in a broader
context, such BCI-based systems provide a powerful tool for
the general enrichment of HMS research using information
about CAUS such as cognitive events and conditions. Also, BCI
input can be combined with other input modalities (in common
use or newly developed), pointing to the definition of hybrid
BCIs described by Pfurtscheller et al. in 2010. The following
sections give an over view of classical BCI technology from an
HMS perspective, plus an over view of broader definitions of the
term ‘brain-computer interface’ that encompasses passive and
hybrid BCIs and extends the concept from medical applications
to HMS’s in general.
>> Generalized Notions of BCIs
By shifting the perspective from the user to the application,
we can place the previously identified groups into a framework
that captures additional types of BCI. This shift allows for the
following definition of a BCI, which covers a broader spectrum of
human-machine systems:
A BCI-based system is a system for providing computer
applications with access to real-time information about the
www.brainproducts.com
Fig. 1: Different stages of available information spaces in human-machine
systems. Stage 1 (dotted line) depicts a commonplace interaction. The user
provides information manually, usually using input modes such as mice,
keyboards or voice control. Stage 2 (dashed line) adds information on the
associated environment to the system. For example, this could consist of
the addition of a light sensor to a notebook computer, allowing the screen
brightness to be optimally adjusted in response to the environmental lighting
conditions. Finally, in Stage 3 (solid line), information on the current user
state is added to the system, leading to an interaction that is based on a
holistic information space. Below, we will propose making particular use of
the passive BCI approach in order to access this information.
cognitive state that is derived from measured brain activity.
Additionally, it is beneficial in this context not to restrict
the information available to BCIs to brain activity alone.
Instead, BCIs may use contextual parameters to improve the
accuracy of their predictions, leading to hybrid BCIs and more
generally to context-aware BCIs. Specifically, in the transition
from controlled laboratory conditions to highly variable realworld situations, contextual parameters help to factor out the
variations in brain activity that could otherwise allow features
of interest to be drowned out by noise. These parameters may
include the state of the application, such as program events, the
state of the environment, or the state of the user as captured
by physiological measures such as body posture, tone of voice
or gaze direction. In the framework of the definition set out
above, the classic BCI has the function of providing information
that is actively transmitted or modulated by the user in order to
control an application. However, what the classic notion does
not cover is information that is not consciously transmitted by
the user. This encompasses a large proportion of the implicit
user state. BCIs which omit voluntary control are clearly subject
to restrictions, but they have several benefits that are critical
for their effective use in human-machine interfaces. These will
be outlined below.
>> BCI Categories
I propose the following threefold categorization of applications
derived from BCI technology. This both includes applications
already defined in classic BCI research, and prospective
applications for use in a more general context.
page 7 of 14
Brain Products Press Release
Sub-category of BCI technology
Definition
Active BCI
An active BCI is one which
derives its outputs from brain
activity that is under the direct
conscious control of the user
and independent of external
events, with the purpose of
controlling an application.
Reactive BCI
activity arising as a response
to external stimulation and
is
indirectly
modulated
by
the user, with the purpose of
controlling an application.
A passive BCI is one which
derives its outputs from arbitrary
brain activity that does not
have the purpose of voluntary
control, in order to enrich a
human-computer
interaction
with implicit information.
www.brainproducts.com
Active and reactive BCIs match the sub-categories of classic
BCIs as they relate to direct and indirect control respectively,
and passive BCIs comprise all the other BCIs. These categories
are a subset of the overall set of BCI concepts, since 1) conscious
control either depends on external influences, making it reactive,
or it operates independently of them, making it active; and 2),
passive BCIs are defined as having a complementary function in
relation to conscious control. However, the boundaries between
the categories are fluid.
A reactive BCI is one which
derives its outputs from brain
Passive BCI
March 2011, Volume 38
>> Outlook
This article outlines some of the potential developments that
current BCI technology makes possible. It shows that this new
form of human-machine interaction could lead to new, and
hopefully highly efficient, types of Interaction between human
beings and machines. The potential of BCI could also be utilized
in scientific experiments to gain further insights into human
cognition.
Since this approach has only come into being in the past two
years, we are at the cutting edge of a new kind of technology
whose possibilities still remain to be exploited. In the next and
final article in this series, we will examine the initial steps that
have been taken in this direction.
page 8 of 14