Journal of Neuroscience Methods 201 (2011) 239250

Contents lists available at ScienceDirect

Journal of Neuroscience Methods

journal homepage: www.elsevier.com/locate/jneumeth

Cognitive ability assessment by BrainComputer Interface

Validation of a new assessment method for cognitive abilities
P. Perego a, , A.C. Turconi b , G. Andreoni a , L. Maggi c , E. Beretta b , S. Parini c , C. Gagliardi b
a
b
c

Politecnico di Milano, Indaco Dept., Milan, Italy

IRCCS Eugenio Medea, Bosisio Parini (LC), Italy
SXT Telemed srl, Lecco, Italy

a r t i c l e

i n f o

Article history:
Received 17 February 2011
Received in revised form 24 June 2011
Accepted 25 June 2011
Keywords:
BCI
BrainComputer Interface
Psychometric assessment
Color Progressive Matrices Raven Test

a b s t r a c t
BrainComputer Interfaces (BCIs) are systems which can provide communication and environmental
control to people with severe neuromuscular diseases. The current study proposes a new BCI-based
method for psychometric assessment when traditional or computerized testing cannot be used owing
to the subjects output impairment. This administration protocol was based on, and validated against, a
widely used clinical test (Raven Colored Progressive Matrix) in order to verify whether BCI affects the brain
in terms of cognitive resource with a misstatement result. The operating protocol was structured into two
phases: phase 1 was aimed at conguring the BCI system on the subjects features and train him/her to
use it; during phase 2 the BCI system was recongured and the test performed. A step-by-step checking
procedure was adopted to verify progressive inclusion/exclusion criteria and the underpinning variables.
The protocol was validated on 19 healthy subjects and the BCI-based administration was compared
with a paper-based administration. The results obtained by both methods were correlated as known for
traditional assessment of a similarly culture free and reasoning based test. Although our ndings need
to be validated on pathological participants, in our healthy population the BCI-based administration did
not affect performance and added a further control of the response due to the several variables included
and analyzed by the computerized task.
2011 Elsevier B.V. All rights reserved.

1. Introduction
Computer-based technology supports communication and participation, even when interaction is severely limited by an
underlying pathology (e.g. ALS Amyotrophic Lateral Sclerosis, CP Cerebral Palsy). Among the most interesting solutions,
BrainComputer Interface (BCI) is a system providing a direct communication channel to the human brain and a computer due to
real-time detection and classication of neural electrophysiological signals activated during a specic mental activity (Wolpaw et al.,
2002). The BCI system detects changes in the patterns of electroencephalographic signals and transforms them into an output signal
which can be used by the computer to control different kinds of
devices (e.g. spelling device, home automation, etc.). Communication and interaction with the external world broaden participation
possibilities, with a signicant improvement in the quality of life.
Up to now, research in BCI has grown rapidly to demonstrate its

Corresponding author at: Via Ghislanzoni, 24, 23900 Lecco, Italy.

Tel.: +39 341 488897; fax: +39 341 488884.
E-mail address: paolo.perego@polimi.it (P. Perego).
0165-0270/$ see front matter 2011 Elsevier B.V. All rights reserved.
doi:10.1016/j.jneumeth.2011.06.025

reliability, accuracy, and bit-rate (Allison et al., 2008; Cheng et al.,

2002).
Only a few studies have explored the cognitive abilities needed
to control the BCI system or, on the other hand, have used BCI
technology as a tool to assess cognitive competencies in patients
with severe motor impairment. Iversen et al. (2008a,b) used a BCI
interfaced and fully computerized method for presenting a variety of experimental two-choice tasks to an ALS patient with the
aim of assessing his cognitive abilities. The evaluation of cognitive
competencies (when motor impairment is severe) is a challenging
issue. Up to now, most psychometric tests have needed at least a
minimal motor response (e.g. gaze control, nodding) (Ding et al.,
2006). Moreover, experimental cognitive tasks, such as those used
by Iversen et al. (2008a,b), describe the performance of a single case
clearly and adequately, but lack the possibility of comparison.
Neurological diseases produce modications in the brain structure (both at cortical and subcortical level) and related signals,
thus affecting the users ability to achieve control of the BCI systems. Furthermore, they may also signicantly affect the users
engagement in the task, both because of specic neuropsychological decits and general cognitive abilities. However, neurocognitive
competencies are very difcult to detail in the presence of a

240

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

neuromuscular impairment which hampers the traditional administration of assessment protocols. In fact, if we accept that
BCI performance depends on the interaction of two different
adaptive controllers (one being the human and the other the hardware/software system) (Wolpaw et al., 2002), in order to correctly
evaluate the global and specic system results, we have to be mindful not only of the BCIs technical characteristics (e.g. translation
algorithms and outputs), but we also have to explore the difculties
experienced by users due to their own characteristics. Interaction
may thus be affected both by signal alterations (due to alterations in
brain structure and stream) and by the subjects characteristics (e.g.
in pathological situations: neurocognitive impairment and specic
neuropsychological decits).
In EEG-based BCI the most used brain patterns are: P300, VEP
(Visual Evoked Potential), and rhythms (Motor Imagery) and
Slow Cortical Potential (SCP) (Wolpaw et al., 2002). These protocols
can be divided into two categories:
Dependent BCI system: it does not use the brains normal output
pathways to carry the message, but activity in these pathways is
needed to generate the brain activity (e.g. EEG) that carries it (e.g.
VEP, P300).
Independent BCI: it does not depend in any way on the brains
normal output pathways and offers the brain a completely new
output pathway. It is likely to be more useful for people with the
most severe neuromuscular disabilities.
Fig. 1. Example of paper based Ravens Colored Progressive Matrices.

BCIs can also be (Mason et al. 2006):

Synchronous: the system is externally paced by a cue stimulus and the EEG signals have to be analyzed and classied
synchronously to each cue stimulus in xed, predened time
windows.
Asynchronous (or self-paced): the system is internally paced
without using any cue stimulus and the user intends an operation at free will. In this case the EEG signals are continuously
analyzed and classied.
We aimed at developing a new, easy-to-used and step-by-step
veried administration tool for cognitive tests by a BCI system that
is freely controlled by the user (self-paced BCI) and requires as little
cognitive ability as possible to be controlled. The proposed method
composed of a software and a protocol is meant for people
with severe impairments (e.g. ALS). This paper presents the stepby-step protocol designed to check the variables underpinning the
single phases both on the computer and human sides and the results
obtained with a group of healthy participants. A comparison of BCIbased and paper-based administration evidenced similar results.

2. Materials and methods

Since the BCI could potentially interfere with the cognitive test
performance at different levels (mainly because of the relatively
unusual setting and the need to adopt new strategies in order to
obtain results and for the effort required for sustaining attention),
this study aimed to answer to the following questions:
Do healthy participants perform similarly in a cognitive task
administered by a BCI-based and a paper-based method?
Can BCI be used for the administration of a cognitive task such as
the Ravens Colored Progressive Matrices Test (RCPM)?
Does the BCI system interfere with a users performance during
the psychometric test?

2.1. The psychometric test

2.1.1. Ravens Colored Progressive Matrices (RCPM)
Ravens Colored Progressive Matrices (RCPM) is the shorter, simpler form of the Ravens Progressive Matrices Test (Lezak et al.,
2004). Spreen and Strauss (1998) recommended it for persons
suffering from physical disabilities, aphasias, CP or deafness, and
persons with intellectual disability. The RCPM has traditionally
been used to measure global cognitive performance in terms of
intelligence (Nakamura et al., 2000), mental age (Alevriadou et al.,
2004), intellectual performance (Zaini et al., 2005) and non-verbal
intelligence (Counter et al., 2005) or visuo-perceptual and visuospatial functions (Nagano-Saito et al., 2005).
Pueyo et al. (2008) have demonstrated, in a sample of participants with Cerebral Palsy, that the performance on Ravens Colored
Progressive Matrices was associated with visuo-perceptual, language, visual and verbal memory, but not with frontal functions.
The RCPM is a fast, easy-to-administer test; it consists of 36 nonrepresentational colored pictures incomplete in the bottom right
corner, organized in three sets (A, AB and B) of 12 diagrammatic
puzzles (Fig. 1); the subject is given 6 alternatives from which to
select the one that best completes the given pattern. The items are
presented in a bold, accurately drawn and pleasant-looking format to maintain concentration and minimize fatigue; the test does
not involve manipulative activities, and verbal instructions are kept
to a minimum. Although mainly used for children (up to 15 years
old) and elderly people (>75 years old), norms for adults have been
published, too (Basso et al., 1987; Measso et al., 1993).
Since we decided to develop a BCI-based protocol to assess cognitive abilities in neuromuscular disorders, we chose the RCPM
because it:
is standardized and validated;
has published normative data;
is a non-verbal test and thus language and limited language abilities do not impact on the results;

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

241

is widely used;
is easy-to-use and fast to administer in order to avoid boredom
or fatigue;
can easily be administered by a computer.
2.1.2. Cattells culture fair test
As far as we know, the RCPM has never been administered by a
BCI system and its normative data refer to a paper-based administration. We thus needed to compare results obtained by BCI with
a more traditional assessment, since we could not administer the
same test because of both implicit and explicit learning processes
and the need to use the test in its entirety to get a score. Along
with the RCPM, the Culture Fair Intelligence Test (CFIT) (Cattell,
1949) is the most widely used test to measure the so-called uid
intelligence (while other well-known tests measure mainly crystallized intelligence). Both have been shown to be good predictors
of uid intelligence and typically load high on a general factor in
psychometric studies (Carroll, 1993). The two tests are quite similar (the subject has to choose the target stimulus among a number
of displayed options), with a consistent, though moderate, correlation index ranging from 0.40 (Salmoni et al. 1984) to 0.57 (Conway
et al., 2002). Both tests are (relatively) culture-free (like everything
dealing with intelligence) and require neither verbal reasoning nor
verbal answers; the subject has to point to his/her choice.
We chose two tasks characterized by a similar construct since
we intended to design a protocol exploring what cognitive competencies are needed to control a BCI system: no need of verbal
answers, limited verbal instructions, displayed visual alternatives
and uent reasoning. Moreover, our protocol could be easily replicated in other countries since the two tests are culture-free.
2.2. BrainComputer Interface
2.2.1. The BCI paradigm
We applied our already validated SSVEP-based BCI system
(Parini et al., 2009), which is a self-paced dependent BCI allowing
subjects to send four different commands to the user interface.
The software is independent of the BCI system so that it can be
used with different BCI paradigms (e.g. SSVEP, P300, SCP and Motor
Imagery). In this experiment we used a SSVEP BCI, which at present
is considered the most efcient system (Vialatte et al., 2010).
2.2.2. ERP data recording
The EEG recording set-up consisted of O1, Oz, O2, P3, Pz, P4, T5
and T6 electrodes placed according to the international 1020 system (Fig. 2). Given the high reliability of the BCI system, in some
cases it was possible to reduce to 5 the number of electrodes by
removing T5, T6 and Pz. The forehead (Fpz) electrode served as
ground electrode. The reference electrode was placed on the right
mastoid. The EEG amplier used was gMobilab (g.tec GmbH, Graz
AU) with a sampling rate of 256 Hz. The signals were transmitted via Bluetooth from the EEG device to the operators PC where
they were recorded and processed by the BCI++ Hardware Interface
Module. We used an on-line band-pass lter (545 Hz, 4th-order
FIR) to remove artifacts due to eye movements or blinks. A notch
lter (elliptic lter with about 60 dB stop-band attenuation) on AC
frequency (50 Hz) was also activated during acquisitions.
2.2.3. The BCI system
The BCI system was based on the Open Source BCI++ Framework
(Perego et al., 2009) as a technological platform for data acquisition,
real-time algorithm management, protocol development, stimulation and user interface. Fig. 3 shows the structure of the entire
system. The BCI system consisted of two modules one for the
hardware and the other for the users interface running on two
different personal computers which can be located separately so

Fig. 2. The adopted electrodes conguration.

that the subject can be tested in a quiet and silent room. The rst
module the operators one supports the Hardware Interface
Software (HIM) (Perego et al., 2009) and is used for signal acquisition, recording and visualization. It enables communication with
the users interface (i.e. the second module) and carries out realtime processing through algorithms developed using both C/C++
and Matlab (MathWorksTM , USA). The second module the users
interface program executes the AEnima software which is a exible tool for the implementation of new operating protocols for
BCI-based user applications.
Our BCI system used already validated algorithms (for HIM)
(Parini et al., 2009; Perego et al., 2009) for SSVEP-based BCI, while
a new user interface (for AEnima) was developed for the RCPM.
Our SSVEP-based BCI system is a dependent BCI, since it relies on
the users vision and ability to control gaze direction. Moreover, a
SSVEP-based BCI uses a stimulus-related potential, needing therefore visual stimulators. Our visual stimulation system (Fig. 3, point
7) consisted of four 3 cm-side cubic boxes, built with matt material
and a thin semi-transparent lm on the front in order to diffuse
the beam and avoid direct light exposure which could cause eye
fatigue after few minutes. Each box included a green LED and stood
on each side of a 19 LCD monitor to associate each stimulator with
a 2-dimensional direction (up, down, left and right) corresponding
to the four BCI commands.
The previously validated algorithm for real-time processing and
the identication of SSVEP patterns into EEG signals (see for details
Parini et al., 2009) relied on the Matlab engine and is characterized
as follows:
Pre-processing and Data Windowing: the eight EEG channel data
were ltered using two different lters (50 Hz 60 dB stop-band
and a 545 Hz 4th order FIR pass-band) and then windowed using
an n-second length window (with n from 1 s to 4 s according to
the SNR).
Spatial ltering and channel combining: the 8-EEG channels preprocessing data were spatially ltered and combined using a
single linear combiner with channel-specic weights obtained
with an automatic method (CSPs). An in-depth explanation of
CSP lter can be found in Parini et al. (2009).
Features extraction: the algorithm used a stimulus-locked average to extract an information index from pre-processed data. This
index was used both for biofeedback and classication.

242

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

Fig. 3. Structure of the SSVEP BCI system: (1) The brain generates the pattern needed to control the BCI system. (2) An EEG cap with 8 electrodes (3) and an EEG amplier
record the signals and transmit them to the (4) operator PC that stores them in a database (5). The operator PC uses an online algorithm to process signals and extract a
command. It passes the commands via TCP/IP to the user PC on which an (6) user interface is running. The user pc also regulates the stimulator (7) placed on each side of a
LCD display. The user interface can show the user a biofeedback in order to help him during use of the BCI.

Classication: the features extracted were classied by a 5-class

(left, right, up, down and null) supervised classier based on the
Regularized Linear Discriminant Analysis (Wolpaw et al., 2002),
thereby giving a command which can be used by the computer
in different ways (displacement of a pointer, actions to control
external peripherals). The classier was trained using the features obtained from the acquisition during the training session
(described in the Operating Protocol paragraph). (An in-depth
explanation of our LDA classier can be found in Parini et al.,
2009.)

2.2.4. The RCPM graphics interface

To avoid language interference and prevent user boredom and
fatigue, we adapted a validated and widely used psychometric test
faithfully replicating the structure, colors, shapes and textures of
the paper-based version. In the PC-based version of the test the
subject found all the possible solution on two rows, among which
he/she could move with the BCI commands. His/her current position is highlighted through a 1.25 magnication of the selected
answer (Fig. 4). The PC-based interface showed the main picture

Fig. 4. Computer Graphics Interface of a Ravens CPM puzzle. In the current state,
the answer 1 is highlighted by zooming in.

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

243

Fig. 5. Ravens CPM test graphics interface during selection: (A) The puzzle is shown, the user has 5 s to identify the right answer. (B) Using left and right command of the
BCI system, the user selects the answer. (C) A black border highlights the selected answer. (D) The answer is denitively selected using the up command.

in the upper half of the screen and the six possible options in
the bottom half (the graphic interface was tested on the RCPM
but can also be used with Standard and Advanced Raven Matrices,
which provide up to eight solutions). The background was changed
from white (as in the classic paper-based test) to a light grey color
(RGB = 190, 190, 190) in order to avoid visual fatigue.
The selection method was based on a scanning modality and
can be summarized in the following four steps:

muscular output channels (Wolpaw et al., 2002), a BCI also depends

on feedback. Thus, a BCI system must provide feedback so that the
user can adapt to the BCI system and vice versa.
In a BCI system feedback is essential for skill development and
control over the EEG response. However, the feedback from the
cursor movement can also have other effects, some of which are
benecial and some harmful. Below are featured the most pros and
cons of feedback:

A diagrammatic puzzle and solutions were displayed. For the rst

5 s no command was accepted by the system. This phase was
highlighted by a small animated picture in the top right corner of
the screen (Fig. 5A).
Afterwards the software enabled the left and right commands of
the BCI input system, by which the user could select the desired
answer (Fig. 5B), also shifting to all targets back and forth. The user
could use the right command to move upwards (from answer 1
to 6) and the left one to move downwards. The selected answer
was slightly zoomed in. There was no time limit to identify the
correct answer (i.e. the system allowed moving after the rst 5 s
but the subject could take all the time needed to start moving).
Once the subject had selected his/her answer, he/she had to hold
it for 3 s until a shadow border surrounding the selection signaled
that the UP command was enabled (Fig. 5C).
The subject conrmed his/her selection by the UP command; if
the answer was not the designated one (e.g. because of a pause
due to fatigue or in case the subject changed his/her of mind),
he/she could change his/her answer again by moving the cursor
with the left/right commands (Fig. 5D).

Provides continual motivation (+).

Ensures attention to the task by maintaining the subjects interest
(+).
Improves performance by allowing rapid reaction to wrong classications (+).
The feedback stimulus might hamper concentration on other
tasks ().
The false classications can elicit frustration and thus affect the
EEG response ().

This selection method implements a double conrmation of the

response, thus avoiding an incorrect selection of the answer due to
a low bit-rate (with consequent high error rate) and consequently
minimizing false selections. This reinforced scan could in principle
affect the subjects exploration strategy, letting the subject check
the given answer easily in the traditional assessment (where the
selected answer is not evidenced). The administration time was
slightly increased since immediate responses were inhibited since
the system compelled the subject to wait 5 s before selecting the
answer. This time could (but not necessarily had to) be used to
explore the stimulus and identify the desired answer. The software
recorded all the answers, the time spent for each, the number of
moves performed to give an answer and other user information.
2.2.5. Biofeedback
The brains normal neuromuscular output channels depend on
feedback or biofeedback for their successful operation. Both standard output such as speaking or walking and more specialized
outputs such as singing or dancing require, for their initial learning and subsequent maintenance, continued adjustments based on
oversight of intermediate and nal outcomes (Krakauer et al., 2000;
Salmoni et al., 1984). When feedback is absent from the beginning,
motor skills do not develop properly. When feedback is lost later on,
skills deteriorate. As a replacement for the brains normal neuro-

The subject and the BCI are adapted to each other. This means
that the BCI depends on the interaction of two adaptive controllers:
the users brain, which produces the signal measured by the BCI,
and the BCI system itself, which translates these signals into specic commands (Wolpaw et al., 2002). The arrangement of the BCI
system to the user was described by Wolpaw et al. (2002) as rstlevel adaptation, while the adjustment of the user to the system is
referred to as third-level adaptation. This process can be dened
as a mutual learning process. The two systems (man and machine)
are strongly dependent but have to be adapted independently in
order to obtain a well-balanced mutual adaptation. Theoretically,
the user and the machine learn progressively and continuously
from each other (in a so-called second-level adaptation): the BCI
system learns from the user through the reconguration of classier parameters in order to adapt to short- and long-term variability
(time of day, hormonal levels, fatigue, illness); the user learns stepby-step how to use the BCI system and perform the mental tasks
in a more functional (i.e. better classied) way. In our BCI system
the adaptation was continuous for the user (through biofeedback)
but was discrete for the machine (system re-calibration). In order
to ensure a pseudo-mutual man-machine adaptation, frequent recalibration (especially in the rst BCI sessions) was carried out and,
consequently, a fast recalibration method was designed to increase
the efciency of the BCI system. In our system, the biofeedback
was composed of a yellow bar placed close to the visual stimulator (see Fig. 3(7)). The biofeedback signal was given only on the
currently selected BCI command (the one with the best feature).
The width of the bar corresponded to the power of the SSVEP
pattern, calculated as the ratio between the standard deviation of
the signal (windowed with a window whose length was equal to
the period of stimulation) and the standard deviation of the entire
signal. This kind of biofeedback is independent of the calibration
and can thus be used during the calibration session and in case of
scarcely reliable calibration. Indeed, the use of a parameter which
is independent of calibration such as biofeedback could help the

244

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

Fig. 6. Structure of the administration protocol.

user have direct control over the parameters, thus increasing class
distinguishability by enhancing the signal repeatability.
2.3. Participants
Since we aimed at developing a protocol to administer cognitive
tests by means of a BCI, we established the following graduated
exclusion criteria (with each step being a pass/no pass criterion
to check the eligibility for both cognitive competencies and the
possibility to use the BCI system):
The subject has at least average or normal intelligence (IQ 90).
The subject is collaborative and electrode impedance is acceptable (Z < 20 k).
The subject produces an SSVEP response.
The SSVEP signal is strong enough to allow classier calibration.
The subject can understand and use the BCI system and the selection commands.
Only healthy subjects were included in this study, specically 19
healthy subjects (11 males and 8 females) aged from 13 to 40 years,
without any neurologic decit (Table 2). They had no previous BCI
experiences. Every subject with refractive disorder performed the
whole experimental session wearing appropriate corrective lenses.
During acquisitions, the subject sat on a comfortable chair at a distance of about 80 cm from a LCD monitor in a noise-controlled and
slightly dim room. All the measurements were carried out at the
Sensibilab Campus Point in Lecco (Italy) and the E. Medea IRCCS in
Bosisio Parini (LC Italy). Written informed consent was obtained
from the subjects and their families according to the Declaration
of Helsinki. The research was approved by the Ethics Committee of
the E. Medea Institute.
2.4. Administration protocol
The diagnostic protocol/system was intended for patients who
are unable to undertake traditionally administered (paper- or PCbased) cognitive assessment. In this rst validation phase, we
assessed healthy subjects and compared their performances on a
BCI-based administration vs. a traditional administration, adding a
supplementary step (the CFIT test) only for this special population.
We divided the protocol into two phases (Fig. 6), separated by
at least a 2-h break (the rst part was administered in the morning,
the second part in the afternoon).
In the rst phase (and only for this sample), each subjects general cognitive competencies were assessed by Cattells Culture Fair
Intelligence Test (CFIT) (Cattell, 1949). This task enabled us to verify
the 1st exclusion criterion. After this rst assessment, the protocol
envisaged a rst checkup session to identify the BCI conguration

parameters and then congure, train and validate the BCI classier. Before this checkup session, the protocol provided 15 min for
placing the electrodes excluding uncooperative subjects or highimpedance electrode placement (>20 k) (2nd exclusion criterion).
The checkup session was based on previous experiences as reported
by Parini et al. (2009) and consisted of two parts (Table 1): screening
and calibration. During screening the four most effective stimulation frequencies for the user were identied by presenting only one
light source at different stimulation frequencies from 6 to 17 Hz
with 1 Hz step. Each frequency was shown for 8 s, followed by a
corresponding resting phase with no stimuli. The recorded data
were fed into a specic ofine processing tool performing a Joint
Time-Frequency Analysis (JTFA) (by means of a short-time Fourier
transform) for each electrode referenced to the linked mastoid. The
JTFA results were shown as a color map-based graph which allowed
for an easy identication of the best stimulation frequencies for the
user.
During calibration data useful to train the classier were
acquired. All the four light sources were activated, and their ashing
rates were set based on the four frequencies identied in the previous session. Calibration was performed ofine using data recorded
in the calibration phase through a specic tool described in a previous paper (Parini et al., 2009). From the calibration phase, the user
was helped by the activation of the biofeedback bars (optional). The
entire check-up session lasted about 10 min. It could be repeated
to adjust the classier parameters so as to enable the BCI system
(rst level of adaptation) to achieve high performance rate. If the
subject did not show or had a weak SSVEP pattern, he/she was
classied as being unable to use the BCI system and was therefore excluded from the test (3rd/4th exclusion criteria). The rst
session ends with the driven test phase in order to validate the
conguration parameters adopted with the checkup session. In the
drive test the user was instructed to give eight commands (two
in each directions) in the shortest possible time, the command to
be sent was selected randomly by the software and was indicated
by an arrow on the LCD screen. During the drive-test the software
recorded times and number of correct/wrong commands in order
to compute the speed of the system (as described below). The rst
session was therefore aimed at verifying the exclusion criteria with
regard to the participants characteristics, both in terms of general
cognitive competencies and interaction with the BCI. The traditionally administered cognitive test was necessary in this study
with healthy subjects and was used to validate the administration protocol in order to verify the assessment reliability. In the
future, only the checkup session will be performed with patients
with neuromuscular diseases, thus testing in a single step the cognitive capacity (to understand the situation and subsequently adapt
behavior) as well as the degree of interaction with the system.
The second phase was aimed at verifying the presence of specic visual perceptual competencies underpinning the task such
as scanning or matching abilities. This phase was, in some sense,
redundant with healthy subjects, but it will be crucial with patients
in order to obtain reliable results (exclusion criteria). The second
phase consisted of a second but shorter checkup session aimed
at verifying the data acquired during the rst one. This session
replicated the rst phase except for the screening session, which
was unnecessary because the four best stimulation frequencies had
already been identied. However, a second Training Session was
proposed in order to customize the BCI system for the subject after
his/her rst, short experience, thus allowing a consequent increase
in bit-rate (rst level of adaptation).
The protocol then envisaged a testing session consisting of some
game applications to let the subjects familiarize with the use of the
BCI system and the selection methods. Among these game applications, the matching game allowed us to verify the presence/absence
of the specic visual perceptual skills required to solve the task,

Table 1
Administration protocol structure with relative descriptions, purposes, durations and subjects.
Phase

Step

Checkup session

Cattells Culture Fair Intelligence Test

Subject preparation
Screening
Screening

Level of adaptation (Wolpaw et al.,

2002)

Checking BCI suitability

Calibration

Training

Setting up BCI classier

First level of adaptation: BCI

adaptation to the subjecta

Total timec
Testing

Driven test

Verify and validate the BCI system

performance (Kronegg et al., 2005)

Third level of adaptation: user

adaptation to the BCI systemb

AstroBrain ght game

Matching game

Train the subject using the BCI

system

Train the subject using the

selection method with BCI system

Third level of adaptation: user

adaptation to the BCI systemb

Third level of adaptation: user

adaptation to the BCI systemb

Mode

Time

Subjects
passed

Subjects
rejected

Form A Scale 2

15
15
184

19
18
16

0
1
2

160

15

Driven sequence completion task

10
56 4

15
15

3
0

Verbally guided
Visual feedback
Target enlargement
Move a cursor to achieve targets

3 10

15

Verbally guided
Multimodal feedback
Fig. 7c
Matching target

1 3

15

12 items PCM test

5 17
3 8

15
15

0
0

Choose answers through scan

selection
Visual feedback
Verbally guided Fig. 7e
12 items PCM test

3 8

15

Choose answers through scan

selection
Visual feedback
Verbally guided Fig. 7e
12 items PCM test

3 8

15

9 24
40 80

15
15

0
4

Focus on a blinking light source

for 8 s
Ofine check for steady state VEP
pattern presence and selection of
the best four frequencies for SSVEP
stimulator
Collecting data for calibration
focusing on four light sources
blinking at difference frequencies
Ofine processing

3 questions
Conguration similar to the
cognitive test
Verbally guided
Visual feedback Fig. 7d
Testing session

Total timed
Cognitive test

Raven 1947 A

Raven 1947 AB

Raven 1947 B

General non-verbal intelligence

test

General non-verbal intelligence

test

General non-verbal intelligence

test

Third level of adaptation: user

adaptation to the BCI systemb

Third level of adaptation: user

adaptation to the BCI systemb

Third level of adaptation: user

adaptation to the BCI systemb

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

Learning session

Aim

Choose answers through scan

selection
Visual feedback
Verbally guided Fig. 7e
Total timed
Total time for the entire protocol with pause
a
b
c

245

First level of adaptation: the BCI system was calibrated on the users features; this adaptation is not continuous and should be repeated to enhance BCI performance.
Second level of adaptation: this kind of adaptation is continuous, the subject learns to use and adapts him/herself to the BCI system for the entire duration of the test.
The duration of the screening and calibration phase was set by the software. The total time of the checkup session also depended on the ofine processing which usually incremented the whole session time by about 10 min.
The time for the learning and testing session depended directly on the BCI performance. This table shows the minimum and maximum values measured during the test.

246

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

Table 2
Data for each participant. MV, MR, RD, LM did not complete the test because they did not meet the second (MV), the third (MR and RD) and the fourth (LM) exclusion criteria.
ID

Age

Sex

Years of
schooling

Bit-rate
(Kronegg
et al., 2005)

Ravens
RCPM
result

Cattells
CFIT IQb

z-Points
Cattellc

Total
moves

Total time
(s)

MV

29

13

133

2.2

MR

36

17

122

1.47

RD

39

13

96

0.26

LM

30

17

118

1.2

SC

14

57.31

33

0.89

97

0.2

57

775

EF

14

64.80

36

1.38

122

1.47

60

584

GL

28

17

67.09

35

1.21

109

0.60

56

578

SA

30

17

61.94

34

1.05

127

1.80

56

747

CT

29

17

67.10

36

1.38

118

1.20

54

428

FC

40

17

62.53

35

1.21

118

1.20

56

609

LP

13

65.08

34

1.05

127

1.80

60

517

NF

23

14

67.37

34

1.05

102

0.13

62

440

GA

40

17

63.90

35

1.21

133

2.20

59

578

MP

22

14

60.95

36

1.38

122

1.47

64

567

CR

29

17

58.03

34

1.05

122

1.47

71

855

FD

25

17

55.11

36

1.38

127

1.80

92

844

MR

28

17

54.70

36

1.38

118

1.20

63

731

MM

26

16

62.52

35

1.21

139

2.60

58

463

PP

27

17

60.58

36

1.38

139

2.60

54

397

a
b
c

zPoints
Ravena

The RCPM z-points are calculated according to the statistics reported by Measso et al. (1993).
Cattell IQ is calculated from the row result of the test by means of standardized tables (Cattell, 1949).
Cattells z-points are calculated according to the statistics reported in the manual.

namely to verify the different processes underlying a task with six

options and with a row and column spatial conguration. This part
of the protocol displayed the same conguration as the RCPM as a
simple game; using the same selection method used for the RCPM,
the subject had to select, from six different options, the image
matching the image shown in the upper center of the screen. The
game was over when the user performed three correct selections
(fth exclusion criteria).
Afterwards the subject was introduced to the proper cognitive
task (RCPM). The three Ravens sets were presented after a pause of
5 min. No feedback was provided and no time limits were imposed
on the subject.
The entire protocol (rst and second phases) lasted on average less than an hour. Despite the break introduced by BCI, the
RCPM administration took about 20 min, the same time suggested in the RCPM administration manual. The detailed structure
of the protocol, with session and exclusion criteria, is shown
in Fig. 7.

3. Results and discussion

phase. All 19 subjects completed the CFIT test and passed the
rst exclusion criteria. The electrode impedance of a subject (MV)
was higher than 50 k and the signal acquisition could not be
performed correctly; he was therefore excluded according to the
second exclusion criteria. Two subjects (MR and RD) were excluded
because they belonged to the 1025% (Nijholt et al., 2008) of
the population unable to produce a steady-state response to a
visual intermittent continuous stimulus. Another subject (LM) was
excluded because he was able to produce SSVEP but not enough
to control a four-command BCI. The remaining 15 volunteers (83%)
were able to control the BCI and use the selection method; they correctly completed the entire protocol. None of the subjects perceived
the SSVEP system or the entire protocol as annoying or difcult to
use, nor did they report signicant fatigue; only two subjects (SC
and FD) experienced a mild eye discomfort.
Table 2 presents the subjects results, the BCI performance,
and the RCPM raw scores and corrected results. The BCI performance was calculated, as in our previous work (Parini et al., 2009),
from the data recorded during the Driven-Test phase with the
formula proposed by Wolpaw (Kronegg et al., 2005) for bit-rate
computation:

Table 1 and Fig. 7 show the whole procedure, the timing, successful/failing participants and a summary of the features of each

BR = V R

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

247

Fig. 7. Detailed structure of the administration protocol. The rst part is aimed at validating the protocol with healthy people: the protocol designed for patients begins with
the subjects preparation and can be performed in a single session.

248

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

Fig. 8. Occurrences of stimulation frequencies.

where
R = log2 N + P log2 P + (1 P) log2

1P
N1

and BR is the bit-rate in bit/min (reported in Table 2), V is the

classication speed in symbols/min drawn from the driven test
phase and R represents the information carried by one symbol in
number of bits per symbol. The carrier R was calculated considering N as the number of symbol states (N = 4, the four directions)
and P as the probability of correct selection (as one minus the
error rate measured during the calibration session). This denition has been widely used by other research groups (Kronegg et al.,
2005) in order to evaluate and quantify BCI communication performances. Each subject performed the whole experimental session
using the system congured on a 3-s analysis window (further
details in Parini et al., 2009). This causes a delay consisting of the
average time interval between the control task trigger and the
rst correct classication. Despite this lag, results show a high
transfer rate which matches the aim of an easy and fast administration of the test. For some participants (EF, GL, CT, NF, GA,
and MM), the analysis window could be reduced to 1.5 s with an
increase in BCI performance (bit-rate). The same window length
was selected in order to simplify the analysis of recorded data. In
fact, a decrease in the windows length requires great concentration from the subjects to avoid false classications and errors, thus
requiring an increased cognitive load potentially affecting the test
results.
The whole protocol lasted at most 1 h and 15 min, with most of
the time taken up by the subject preparation for the EEG (placement of electrodes 1520 min) and BCI system conguration; the
Ravens test required approximately 1025 min, which compares
favorably with the timing suggested in the Ravens test manual
(Cattell, 1949). Nonetheless, both versions of the RCPM (paperbased and BCI-based) have no time limit but only a suggestion in
order to avoid fatigue and stress.
Fig. 8 shows a histogram calculated on the occurrences of each
stimulation frequency in the explored range (617 Hz). A low predominance of low frequencies can be seen. When the screening
phase allowed the subject to choose from more than four stimula-

tion frequencies, we offered the possibility of selecting frequencies

which were considered less stressful for the subject. Thus, the predominance of low frequencies was related to the users choice in
selecting these frequencies that were considered less tiring. We also
compared the stimulation frequencies with the BCI performance
and test duration without nding a signicant difference or correlation (R2 = 0.058, F2,14 = 0,89, p = 0.198). In their study comparing the
effect of different mental tasks on SSVEP stimulation frequencies,
Zhenghua and Dezhong (2007) suggested that the same cognitive
task affects different frequencies in a similar way, and low frequencies are a better choice than high ones for some of the different tasks
(e.g. working memory study).
With regard to demographic data, our healthy participants had
an average age of 25.8 years (1340) and almost all of them had
a university degree; their general cognitive abilities, assessed both
by a traditionally administered test (CFIT) and by BCI (RCPM), were
within the normal range, although quite high, supposedly biased
by recruitment criteria. Most CFIT scores were above norm (see
Table 2) and most subjects obtained a full score on the RCPM. A
5-factor ANOVA was carried out on the bit-rate. Factors were sex,
age, years of schooling, corrected RCPM score and CFIT. No effect
was statistically signicant; the same result was obtained when the
factors were combined. Fig. 10a shows that the performance of the
BCI system measured as a bit-rate is age independent. Although the
participants were interacting with a BCI system for the rst time,
most of them were familiar with computers; this supports Allison
et al.s (2008, 2010) recent conclusion that younger male subjects
were typically more accustomed in the use of BCIs because of their
computer skills. They were less annoyed by the ickering light and
obtained higher bit-rates.
On the contrary, the bit-rate was related to the number of total
moves for the entire RCPM (R2 = 0.75); in fact, when the bit-rate is
low, the error rate increases and can cause wrong classications
(i.e. movement of the cursor in a wrong direction). Fig. 10f displays
the linear correlation (R2 = 0.73) between bit-rate and the total time
needed to complete the RCPM. Fig. 9 shows the distribution of the
test results in our sample and in studies by Measso et al. (1993) and
Cattell (1949). Despite being characterized by a shift towards the
right, probably due to our recruiting and enrolling mainly highly
educated participants, the distribution of the results of the BCIbased RCPM and the paper-based Cattel test are comparable, with
similar slopes.
Salmoni et al. (1984) reported a correlation of about 0.4 for the
Cattell test and other non-verbal tests, and Conway et al. (2002)
reported a correlation of 0.57 between the RCPM and the CFIT:
z-point data in Table 1 show a similar correlation (0.45). Therefore, at least in our sample of healthy subjects with relatively
high education, the results obtained by subjects during a BCIbased RCPM are similar to those obtained during a paper-based
assessment.

Fig. 9. Distribution of Raven CPM results (a) and IQ as by Cattell (b): Blue: control group from normative; Red: our sample. (For interpretation of the references to color in
this gure legend, the reader is referred to the web version of the article.)

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

249

Fig. 10. Bit-rate across age (a), gender (b), Ravens CPM results (c) and IQ Cattell (d). Time for the test across Ravens test result (e) and bit-rate (f).

4. Conclusion
Our protocol provides an operative ow chart to assess cognitive skills by a BCI system, checking out the different variables
involved in the interaction by means of a step-by step-procedure.
This protocol helped us verify whether BCIs could provide reliable
data regarding general cognitive abilities in comparison with a normal, paper-based test in a sample of healthy subjects. The aim of
this comparison was to evaluate whether the cognitive tasks underlying the use of a BCI system interfere with the nal result of the
psychometric test by a putative overload due to the use and control
of the BCI system.
As depicted in Fig. 10e and f, participants obtained similar
results with paper-based and BCI-based assessments; the RCPM
score was independent of time (R2 = 0.07) and bit-rate (R2 = 0.004).
Therefore, bearing in mind the cognitive burden of the BCI system
and given our samples characteristics (healthy subjects, relatively
high education, high cognitive competencies), our SSVEP-based
BCI together with the protocol described here was quite easy
to use, did not tire participants and, most of all, did not seem
to inuence the test results. In fact, the correlation between
the scores obtained by the traditionally and the BCI administered tests was similar. The duration of the session was also
comparable with the session length suggested in the RCPM manual, although accuracy was more relevant than time for the test
score.

Our protocol seems suitable for BCI-based RCPM administration: were RCPM routinely administered through a BCI, it would be
necessary to provide a new standardization for the BCI assessment
for different populations (or different contexts). The computerized
RCPM version we provided is in fact in some sense facilitated: single targets are evidenced for a while letting the subject more easily
compare and verify the correct/wrong answer. Nevertheless, scores
obtained by our healthy participants seemed to depict their competencies fairly well. Thanks to the computerized RCPM version,
an additional qualitative analysis of the cognitive task could be
performed, thus highlighting selection strategies and difculties
that could be helpful in the diagnosis and treatment of pathological
conditions.
Performance on the BCI system was independent of age (at least
within the 1040 years range) (see Fig. 10a).
The protocol consisted of two parts, including a redundant calibration session used to improve the adaptation of the system to the
user. The protocol length can be reduced by removing this double
calibration session and administering the entire protocol in a single step. As described above, our novel method seemed feasible and
reliable with healthy subject for the SSVEP-based BCI administration of cognitive tests (RCPM). A SCP-based BCI system has already
been used by Iversen et al. (2008a,b) to assess cognitive function
in ALS patients through a non-validated test. They assumed that,
if subjects can control a BCI system with a two-choice task, they
should be able to answer questions related to their cognitive skills.

250

P. Perego et al. / Journal of Neuroscience Methods 201 (2011) 239250

Seemingly, Iversen et al. did not consider the possibility that the BCI
system may need a proper cognitive skill, potentially affecting the
response. With this protocol we strove to detect variables involved
both in the interaction underpinning the BCI system and in the
subjects characteristics, thus leading to a step-by-step procedure
and an easier interpretation of data. Exclusion criteria were clearly
identied. The neurocognitive functions identied as ground competencies necessary to solve the task were checked in order to avoid
a difcult to interpret oor effect.
We tested our protocol on a dependent BCI in which the gaze
control is mandatory (although some studies use attention to modulate SSVEP) (Wang and Wade, 2011). This kind of BCI may be more
reliable than an eye-tracker, above all in cases when a neuromotor
disease affects gaze causing for example nystagmus which prevents eye-tracker use. On the contrary, in other cases, eye-trackers
are faster and simpler systems. Future work is needed to conrm
the applicability of the protocol with pathological participants; the
same protocol could be tested with different BCI paradigms to evaluate the cognitive effort needed to drive SCP, Motor Imagery or
P300-based BCI.
Acknowledgments
This work has been partially supported by the Italian Institute of
Technology (IIT). The authors would also like to thank the clinical
staff (Giulia Livetti and Luigi Piccinini) for their invaluable help and
support and all the subjects for their collaboration and patience.
References
Alevriadou A, Hatzinikolaou K, Tsakiridou H, Grouios G. Field dependenceindependence of normally developing and mentally retarded boys of low and
upper/middle socioeconomic status. Percept Mot Skills 2004;99:91323.
Allison B, Luth T, Valbuena D, Teymourian A, Volosyak I, Graeser A. BCI demographics: how many (and what kinds of) people can use an SSVEP BCI? IEEE Trans
Neural Syst Rehabil Eng 2010;18(2):10716.
Allison BZ, McFarland DJ, Schalk G, Zheng SD, Jackson MM, Wolpaw JR. Towards an
independent braincomputer interface using steady state visual evoked potentials. Clin Neurophysiol 2008;119(2):399408.
Basso A, Capitani E, Laiacona M. Raven coloured progressive matrices: normative
values on 305 adult normal control. Funct Neurol 1987;2:18994.
Carroll JB. Human cognitive abilities: a survey of factor analytic studies. New York:
Cambridge University Press; 1993.
Conway ARA, Cowan N, Bunting MF, Therriault DJ. Minkoff SRB a latent
variable analysis of working memory capacity, short-term memory capacity, processing speed, and general uid intelligence. Intelligence 2002;30:
16383.
Cattell R. Culture free intelligence test, scale 2, handbook. Champaign, IL: Institute
of Personality and Ability; 1949.
Cheng M, Gao X, Gao S. Design and implementation of a braincomputer interface
with high transfer rates. IEEE Trans Biomed Eng 2002;49(10):11816.

Counter SA, Buchanan LH, Ortega F. Neurocognitive impairment m lead-exposed

children of Andean leadglazing workers. J Occup Environ Med 2005;47:30612.
Ding J, Sperling G, Srinivasan R. Attentional modulation of SSVEP power depends
on the network tagged by the icker frequency. Cereb Cortex 2006;16(7):
101629.
Iversen IH, Ghanayim N, Kbler A, Neumann N, Birbaumer N, Kaiser J. Conditional
associative learning examined in a paralyzed patient with amyotrophic lateral sclerosis using braincomputer interface technology. Behav Brain Funct
2008a;14:114.
Iversen IH, Ghanayim N, Kbler A, Neumann N, Birbaumer N, Kaiser J. A
braincomputer interface tool to assess cognitive functions in completely
paralyzed patients with amyotrophic lateral sclerosis. Clin Neurophysiol
2008b;119:221423.
Krakauer JW, Pine ZM, Ghilardi MF, Ghez C. Learning of visuomotor transformations for vectorial planning of reaching trajectories. J Neurosci 2000;20(23):
891624.
Kronegg J, Voloshynovskiy S, Pun T. Analysis of bit-rate denitions for
braincomputer interfaces. In: Proceedings at the 2005 int. conf. on
humancomputer interaction (HCI05); 2005.
Lezak MD, Howieson DB, Loring DW. Neuropsychological assessment. 4th ed.; 2004.
Mason S, Kronegg J, Huggins J, Fatourechi M, Schloegl A. Evaluating the performance
of self-paced BCI technology. Technical Report, Neil Squire Society; 2006.
Measso G, Zappal G, Cavarzeran F, Crook TH, Romani L, Pirozzolo FJ, et al. Ravens
colored progressive matrices: a normative study of a random sample of healthy
adults. Acta Neurol Scand 1993;88(1):704.
Nagano-Saito A, Washimi Y, Arahata Y, Kachi T, Lerch JP, Evans AC. Cerebral atrophy and its relation to cognitive impairment in Parkinson disease. Neurology
2005;64:2249.
Nakamura H, Nakanishi M, Furukawa TA, Hamanaka T, Tokudome S. Validity of brief
intelligence tests for patients with Alzheimers diseases. Psychiatr Clin Neurosci
2000;54:4359.
Nijholt A, Tan D, Pfurtscheller G, Brunner C, Mill J, Allison B, et al. Braincomputer
interfacing for intelligent systems. IEEE Intell Syst 2008;23(3):729.
Parini S, Maggi L, Turconi AC, Andreoni G. A robust and self-paced BCI system based on a four class SSVEP paradigm: algorithms and protocols for a
high-transfer-rate direct brain communication. Comput Intell Neurosci 2009.,
doi:10.1155/2009/864564.
Perego P, Maggi L, Parini S, Andreoni G. BCI ++: a new framework for Brain Computer
Interface application. In: Proceedings at SEDE ISCA; 2009. p. 3741.
Pueyo R, Junqu C, Vendrell P, Narberhaus A, Segarra D. Coloured progressive matrices as a measure of cognitive functioning in cerebral palsy. J Intellect Disabil Res
2008;52:43745.
Salmoni AW, Schmidt RA, Walter CB. Knowledge of results and motor learning: a
review and critical reappraisal. Psychol Bull 1984;95(3):35586.
Spreen O, Strauss E. A compendium of neuropsycological tests, administration,
norms, and commentary. 2nd ed.; 1998.
Vialatte F, Maurice M, Dauwels J, Cichocki A. Steady-state visually evoked potentials: focus on essential paradigms and future perspectives. Prog Neurobiol
2010;90(4):41838.
Wang J, Wade AR. Differential attentional modulation of cortical responses to S-cone
and luminance stimuli. J Vis 2011;11(6).
Wolpaw JR, Birbaumer N, McFarland DJ, Pfurtscheller G, Vaughan TM.
Braincomputer interfaces for communication and control. Clin Neurophysiol
2002;113(6):76791.
Zaini MZ, Lim CT, Low WY, Harun F. Effects of nutritional status on academic
performance of Malaysian primary school children. Asia Pac J Public Health
2005;17:817.
Zhenghua W, Dezhong Y. The inuence of cognitive tasks on different frequencies steady-state visual evoked potentials. Brain Topogr 2007;20(2):
97104.