Anda di halaman 1dari 4

BrainNets: Human Emotion Recognition Using an

Internet of Brian Things Platform

Huimin Lu, Hyoungseop Kim Yujie Li
Dept. of Mech. And Control Engineering Faculty of Engineering
Kyushu Institute of Technology Fukuoka University
Kitakyushu, Japan Fukuoka, Japan

Yin Zhang
School of Information and Safety Engineering
Zhongnan University of Economics and Law
Wuhan, China

Abstract— Human wearable helmet is a useful tool for BCI technology is a type of direct communication between
monitoring the status of miners in the mining industry. However, the brain and the outside world. BCI is a technique to
there is little research regarding human emotion recognition in communicate with the outside world without the assistance of
an extreme environment. In this paper, an emotional state evoked
paradigm is designed to identify the brain area where the the brain’s central nervous system and muscle tissues. The
emotion feature is most evident. Next, the correct electrode research in areas of neuroscience, psychology and cognitive
position is determined for the collection of the negative emotion science has demonstrated that many mental activities and
by the electroencephalograph (EEG) based on the international cognitive behaviors can be reflected by brain waves; therefore,
10-20 system of electrode placement. And then, a fusion emotion recognition has been introduced with the BCI
algorithm of the anxiety level is proposed to evaluate the person’s technology.
mental state using the θ, α, and β rhythms of an EEG.
Experiments demonstrate that the position Fp2 is the best Using the central nervous system for emotion recognition is
electrode position for obtaining the anxiety level parameter. The an important method of physiological information recognition.
most visible EEG changes appear within the first two seconds For the emotion stimulation paradigm, the conventional
following stimulation. The amplitudes of the θ rhythm increase methods utilize picture stimulators, video stimulator, sound
most significantly in the negative emotional state. stimulator, facial expression stimulator, and practical task
stimulator. The emotion features include the time domain
feature [2-4], the frequency domain feature [5-6], and the
Index Terms—Emotion recognition, EEG, Cloud computing,
Internet of Things time-frequency feature [7]. The mean, the standard deviation,
the skewness, and the peak of the EEG are typically used for
emotion recognition using time domain features. Additionally,
the power spectrum, power spectrum density, and energy
spectrum of the EEG rhythm are used for emotion recognition
I. INTRODUCTION using the frequency domain features. These methods use the

I N brain computer interface (BCI) systems, emotion sub-band energy and the mean square root of the Wavelet
recognition is an important function. It is difficult to coefficient as the emotion features. Additionally, entropy and
perceive the emotion of certain confined people through their Lyapunov exponents are used as the emotion features.
facial expressions, such as deep-sea miners and coal miners. For the mental recognition, machine learning methods, such
However, an extreme environment is likely to lead to anxiety as support vector machines, neural networks, Bayes decision
in individuals. The Human smart helmet is a part of the trees, and the hidden Markov models are adopted. Unlike the
wearable safety device. Most smart helmets for humans have above methods, this paper focuses on anxiety representation
the functionality of monitoring the environment or making and quantification for negative emotions and studies the
phone calls. However, there are no studies on human emotion anxiety level change using BCI for a human smart helmet in
changes using a BCI with a human smart helmet in an extreme an extreme environment.
environment [1, 13-20].

978-1-5386-2070-0/18/$31.00 ©2018 IEEE 1313

II. RELATED WORK signal processing to draw the features. Finally the emotional

Fig. 1 Overall human anxiety state estimation process of the proposed.

The emotion state can be divided into two spaces: discrete states are detected by the processing.
space and continuous space. The discrete emotional states
contains seven to ten emotions such as fear, contempt, disgust, III. CLOUD COMPUTING PLATFORM
anger, sadness, surprise and happiness [11]. The dimensional The miner’s smart helmet is designed to include the functions
emotion model build the continuous state space, which of collecting the anxiety level parameter, the fatigue parameter,
represents the emotional state as a vector in multidimensional and the concentration parameter of the miner. These brain
space. The EEG features can be categorized into three parameters are produced by the mine environment. The
domains: time domain, frequency domain and time-frequency miner’s smart helmet also collects the environmental
domain. In the time domain, the components of event-related parameters. The brain parameters and the environmental
potentials (ERPs) reflected underlying emotional states [12]. parameters are used to estimate and provide an early warning
Theses ERP components consisted in the following order: P1 of safety issues in coal production. Fig. 1 shows the scientific
and N1 components generated in a short latency, N2 and P2 in scheme of the smart helmet.
a middle latency, and P3 and slow cortical potential in a long Human emotion is a perceptual variable, which changes
latency. In frequency domain, the spectral power in various
complexly and subtly. Thus, the emotional stimulus material
frequency bands has been implicated in the emotional state.
The α power varied with a valence state. needs to be standardized and quantified. The International
The computational methods for estimating the emotional Affective Picture System (IAPS) [8] is a stimulus picture
state have been developed based on EEG features. The most library which has the quantitative evaluation standard using
used methods is: Preprocessing, feature extraction, the software E-Prime2.0 [9]. According to the image
classification and emotional states recognition [21-22]. In quantization parameters, we selected the positive and the
preprocessing, the noise reduction methods, spatial filtering or negative group images, which have values in the ranges of 1~3
temporal filtering are adopted. The feature extraction methods and 7~9 respectively. The arousal levels of the images are in
usually build a computational model to find the emotion-like
the ranges of 5~9. There were 50 images in each group, and a
features based on neurophysiologic and neuropsychological
knowledge. Different from the feature extraction methods, the total of 100 images were used to stimulate the tester’s
classification methods use the machine learning and statistical emotions.

The EEG collector is the NuAmps of Neuroscan. The signal
analysis programs were Scan and Curry. Twenty students
20~25 years old were selected as the subjects. The subjects
were healthy and without brain injury or mental illness. The
subjects were required to have normal sleep patterns and no
mental fatigue [10]. Additionally, the subjects were required
to be free of disease. During the experiments, the subjects
were required to keep their sitting posture fixed with no head
movement, and limited blinking, eye movement, and
The experiments were divided into two groups. One group
was the negative emotion stimulation experiment, and the
other was the positive emotion stimulation experiment. There
was an experiment guide and four practice images such that
the participants would be familiar with the experiment
procedure before beginning the experiment. Next, 20 seconds
of resting data were collected for control.
The duration of each group experiment was 15 minutes, and
50 images were used. Each picture was displayed for four
seconds. To reduce the influence of each picture on the
following picture, there was four seconds between two
adjacent images. The pure black image was used for the first
two seconds, and the red dot image was used for the second
two seconds while preparing for the next image stimulation.
Fig. 6 Best collection electrode position Fp2 using the 2-
This work used the NuAmps system of the Neuroscan
software. The software Scan and Curry were used to collect Anxiety level tests and anxiety level experiments were
EEG readings and to identify the electrode locations conducted to test and verify the anxiety level parameter. Fig. 3
corresponding to the EEG. The software MS Visual Studio (a) shows the anxiety level parameters of the slightly negative
C++ and MATLAB were also used. emotion and the obviously negative emotion, respectively. Fig.
Paradigm experiments were conducted to identify the 3 (b) shows the subsequent function tests of the anxiety level
optimal electrode location to measure anxiety level and parameter. The upper one shows the negative emotion from
complete the stimulator BCI paradigm design. Fig. 2 is a 2- mild to acute, and the bottom one shows the negative emotion
dimensional map of the brain. The best electrode position was from acute to the mild. It can be seen that the proposed anxiety
determined to be electrode position Fp2. level parameter reflects the corresponding emotions accurately.
The system EEG stimulator begins recording at 0ms. It can
be seen that the EEG changes significantly from 100ms to
1500ms. As a result, the human EEG changes last
approximately 1400ms. After this transient phase, the EEG
returns to the normal state.


[1] H. Lu, Y. Li, M. Chen, H. Kim, S. Serikawa, “Brain Intelligence: Go

beyond artificial intelligence,” Mobile Networks and Application, pp.1-10,
[2] Z. Khalili, M. Moradi, “Emotion recognition system using brain and
peripheral signals: Using correlation dimension to improve the results of
EEG,” in Proc. of International Joint Conference on Neural Networks,
pp.1920-1924, 2009.
[3] J. Kumar, J. Kumar, “Affective modelling of users in HCI using EEG,”
Procedia Computer Science, vol.84, pp.107-116, 2016.
[4] Y. Lee, S. Hsieh, “Classifying different emotional states by means of
EEG-based functional connectivity patterns,” PLOS One, pp.1-16, 2014.
[5] M. Murugappan, M. Rizon, R. Nagarajan, S. Yaacob, D. Hazry, I. Zunaidi,
“Time-frequency analysis of EEG signals for human emotion detection,”
IFMBE Proceedings, vol.21, pp.262-265, 2008.
[6] C. Wang, S. Feng, D. Wang, Y. Zhang, “Fuzzy-rough set based multi-
labeled emotion intensity analysis for sentence, paragraph and document,”
Lecture Notes in Computer Science, vol.9362, pp.444-452, 2015.
[7] J. Hu, “Comparison of different features and classifiers for driver fatigue
detection based on a single EEG channel,” Computational and Mathematical
Methods in Medicine, vol.2017, pp.1-9, 2017.
Fig. 3. Anxiety level parameter (a) In the slightly negative [8] Langp J, Bradleym M, Cuthbertb H, “International Affective Picture
emotion and the obviously negative emotion (b) The System (IAPS): Technical Manual and Affective Ratings”, 1997.
subsequent function tests. [9] Andrey VB, Gennady GK, Alexander NS, “Depression and implicit
emotion processing: An EEG study. Neurophysiologie Clinique/Clinical
Neurophysiology,” vol.47, no.3, pp.225-230, 2017.
Compared with the existing methods, the results of this [10] M. Wang, L. Guo, W. Chen, “Blink detection using Adaboost and
contour circle for fatigue recognition,” Computers and Electrical Engineering,
paper are limited to the anxiety level detection of miners. vol.58, pp.502-512, 2017.
Therefore, the evoked paradigm of the BCI is designed [11] L. Barrett, “Discrete emotions or dimensions? The role of valence focus
specifically for miners. The electrode position Fp2 is observed and arousal focus,” Cognition and Emotion, vol.12, no.4, pp.579-599, 1998.
[12] J. Olofsson, S. Nordin, H. Sequeira, J. Polich, “Affective picture
to be the best pole position based on the international 10-20 processing: an integrative review of ERP findings,” Biological Psychology,
system. Thus, the anxiety level parameter is defined to vol.77, no.3, pp.247-265, 2008.
[13] M. Wang, C. Xu, H. Lu, “Fault Location Without Wave Velocity
represent and evaluate the anxiety level by combining the θ, β, Influence Using Wavelet and Clark Transform”, Artificial Intelligence and
and α rhythms of human EEG. Robotics, pp.321-326, 2017.
[14] S. Serikawa, H. Lu, “Underwater image dehazing using joint trilateral
Through the study, we found that in the brainwave rhythm, filter”, Computers & Electrical Engineering, 40(1), pp.41-50, 2014.
the θ wave is usually related to the emotional state, prone to [15] H Lu, B Li, J Zhu, Y Li, Y Li, X Xu, L He, X Li, J Li, S Serikawa,
low mood, such as there will be a brief θ wave when frustrated “Wound intensity correction and segmentation with convolutional neural
networks”, Concurrency and computation: practice and experience, 29(6),
by human frustration, depression, etc. α wavelength duration pp.1-10, 2017.
helps to reduce human tension and anxiety. β wave and human [16] H. Lu, Y. Li, L. Zhang, S. Serikawa, “Contrast enhancement for images
in turbid water, JOSAA, 32(5), 2015.
emotional activity are closely related. [17] Y Li, H Lu, J Li, X Li, Y Li, S Serikawa, “Underwater image de-
scattering and classification by deep neural network”, Computers & Electrical
V. CONCLUSION Engineering, 54, pp.68-77, 2016.
[18] H Lu, Y Li, S Nakashima, S Serikawa, “Single image dehazing through
To research the human anxiety level changes using BCI for a improved atmospheric light estimation”, Multimedia Tools and Applications,
human smart helmet in the EEG cloud computing platform, a 75(24), pp.17081-17096, 2016.
[19] H Lu, Y Li, S Mu, D Wang, H Kim, S Serikawa, “Motor Anomaly
fusion algorithm of the anxiety level parameter was proposed. Detection for Unmanned Aerial Vehicles Using Reinforcement Learning,”
Experiments demonstrate that the amplitudes of the θ rhythm IEEE Internet of Things Journal, pp.1-8, 2017.
increase most significantly in the negative emotional state. [20] H Lu, Y Li, Y Li, S Serikawa, H Kim, “Highly Accurate Energy-
Additionally, the fusion algorithm of the anxiety level has an Conserving Flexible Touch Sensors,” Sensors and Materials 29 (6), 611-617,
accurate subsequent function to the negative emotional change.
This method quantitatively represents the anxiety level of a
human. This approach can accordingly be used to improve
operating safety and avoid improper miner operation.