ICOT 2015

Human Action Recognition System for Elderly and

Children Care Using Three Stream ConvNet

Chang-Di Huang, Chien-Yao Wang, and Jia-Ching Wang

Department of Computer Science and Information Engineering
National Central University
Taoyuan, Taiwan, R.O.C.
fighting8139@gmail.com, x102432003@yahoo.com.tw, and jcw@csie.ncu.edu.tw

Abstract—Because of the change of family structure and

population ageing, elderly and children care is become a very
important issue in modern society. When adults are busy
working, they have no time to care elderly and children who
standalone in the home. This paper proposes an elderly and
children care system to solve this important problem. The
proposed intelligent surveillance system is based on action
recognition technique of image processing. In this paper, a three
stream convolution neural network is proposed for recognize
human actions such as fall floor and baby craw. If the system
detect abnormal activities are occurred, it will raise alarm and
notice family members. In the experiment, there are totally 21
categories activities are collected from HMDB-51 dataset, UCF-
101 dataset and Internet. The proposed system achieves 93.42%
recognition rate of selected actions.

Keywords—action recognition; three stream ConvNet;

convolutional neural network; deep learning; spatial; temporal;
moving
1. Introduction
In modern society, most of there is double-income families.
When they are busy working, elderly and children care
becomes very important. If hiring a caregiver, it is not
anywhere and anytime to protect the elderly and children, in
addition to the cost of a caregiver is quite expensive. If there is
an IP camera to monitor home security and detect motion
information, which not limited by temporal and spatial
components. Using captured video from IP camera, action
recognition technique can be applied to elderly and children
care by recognizing whether the elderly or children take
dangerous actions. Fig. 1 shows some examples of safe actions
which is acceptable and dangerous actions which need to raise
alarm.
Image Retrieval has been applied to video process for years.
There is a large amount of research gone into Image Retrieval.
One of which is text-based image retrieval (TBIR), and the
other is content-based image retrieval (CBIR). But its
performance is still much more to improve. It faced different
kinds of problems such as action recognition [3, 4], and
activity understanding [5]. However, there is still lots of need
for action recognition that helps in classifying large-scale
videos [6].
High-level image feature learning for action recognition has
been making great progress in the past few years.
Figure 1. Example of Actions.
Convolutional networks (ConvNets) [8] are available for
extracting image features in convolutional layers. These
features then transfer learning tasks in the last fully-connected
layers. [9, 10] Learning features in ConvNet are better image
retrieval representation than the technology used before, and
improve the performance on image retrieving.
There are four types of ConvNet architectures are used for
action recognition task:
(1) 2-dimention convolutional network as feature extractor,
Support Vector Machine (SVM) and Neural Network
(NN) as classifier [1]. The proposed system is mainly
based on this approach.
(2) 3-dimention convolutional network as feature extractor,
SVM and NN as classifier [6, 11]. This framework uses
3D convolutional kernel (time by height by width) to
solve spatial-temporal feature.
(3) 2-dimention convolutional network as feature extractor,
Recurrent Neural Network (RNN) and Long Short-Term
Memory (LSTM) as classifier [12]. The architecture
combined with spatial and temporal features could solve
more complex video data.
(4) 3-dimention convolutional network as feature extractor
and RNN LSTM as classifier [13].
Using camera or video to recognize human action is
computationally demanding. In [1], it developed two-stream
ConvNets, which combined with two different ConvNets. Still

978-1-4673-8237-3/15/$31.00 ©2015 IEEE 5

 Figure 2. Three-stream convolutional network architecture for action recognition.
one ConvNet was trained for image classification, while the
other ConvNet was trained on the dense video optical flow.
Two separated recognition stream, that is spatial stream and
temporal stream, will be combined by late fusion. Spatial and
temporal streams successfully decrease numbers of learning
parameters. Owing to this reason, two-stream ConvNets
conquer the computational demand.
In this work, we proposed a three-stream ConvNets to
video data. Beside spatial stream and temporal stream, the
centroid of human is used to make moving stream network.
Recognize what action does the character in the video do, for
example: walking, running, opening the window and hanging
clothes, etc. Once the result of the action recognition
determined that behavior is abnormal [2, 7], then generates a
corresponding solution such as sending warning SMS to the
householder. Eventually build a smart living for elderly and
children care with surveillance systems.
The rest of the paper is constructed as follows. In section 2
we give our three-stream ConvNets system an overview, and
clarify each size of the layer. In section 3, firstly we present
the function each layer work in ConvNets. Secondly we
introduce the three-stream architecture. Thirdly we specify the
Spatial and Temporal ConvNet. In section 4 we introduce the
two action recognition datasets, and show which class we
select to apply on home care and smart living. Experimental
results are given in section 5 which include the recognition
rate evaluated. And conclusion is in section 6.
2. System Overview
The system overview of the proposed system framework is
shown in Fig. 2. First, the deformable part model (DPM) [18]
is used for extract human regions from video. Extracted
human regions are resize to 96x48 RGB images. The input of
the spatial-stream convolutional network is 96x48@6 RGB
6
images, the input of the temporal-stream convolutional
network is 94x48@6 optical flow images, and the input of the
moving-stream network is motion vectors of detected human
region centroids. Proposed convolutional network has 2
convolutional layers, 1 pooling layer, 2 fully connected layers
and final followed by a softmax classifier output object layer.
All convolutional kernels are 3x3@64. The pooling layer is
2x2. All full-connected layers have 256 output units. And the
Softmax full connected layer is used for fusion results of
three-stream convolutional network.
3. Proposed Method
This section contains two parts. The first part is
introduction of convolutional network and the second part is
details of proposed three-stream convolutional network.
3.1 Convolutional Network
ConvNet [19] is essentially a kind of input-to-output
mapping. The purpose of the training is to learn the mapping
ability between input and output. The model of the ConvNet is
mainly composed of several numbers of convolutional layers
and pooling layers. Finally use high-level features, which
generated by fully connected layers, as the basis of identifying
the input image.
The convolutional layer make use of convolutional kernels
to process convolutional evaluation with the input image.
These image features then convey to the next layer as mapping
feature. Thus the convolutional layer is just an image
convolution of the previous layer.
After each convolutional layer, there may be a pooling
layer. The pooling layer uses local correlation principle to
subsample images; meanwhile it decreases the dimension of
the pixels, retaining useful characteristic information. In Fig. 3,
there are two ways to do subsampling. One is average pooling
layer, another is maximum. In this work, we always use the

Figure 3. Pooling operation.
max-pooling layer which take the maximum of the block that
they are pooling.
After several convolutional layers and max-pooling layers,
the high-level deep learning result is done by the fully-
connected layer. Fully-connected layer can be visualized as
one-dimension since they are not located anymore. Apparently,
there is no convolutional layers after fully-connected layers.
3.2 Three-Stream Convolutional Network
In three-stream ConvNets, which is inspired from two-
stream ConvNets [1], video can be separated into temporal
stream and spatial stream naturally. The spatial stream
includes the information about frames and object scenes in the
video. On the other hand, the temporal stream, formed by
motion, carries the movement of camera or other observers.
Frames of this two streams are cropped and realignment by
extracted human regions and fed into deep convolutional
neural network, so they can learnt and tested separately.
During the human region extraction, the moving stream can be
extracted from movement of interested human and be the input
of multi-layer perceptron. After deep learning, these three
streams are finally combined with the hinge loss classifier.
Movement Neural Network. There are some actions have
similar pose change but with different speed or direction, for
example walking versus running and push versus pull. The
moving-stream can be used in these kind of actions. In the
other hand, some actions are safe when the movement is slow
but dangerous when the movement is quick. The moving-
stream network can also detect these warning. The input of
moving-stream is the centroid of detected human regions.
Spatial Convolutional Network. The input of the
proposed spatial stream is a sequence of serial frames while it
is an individual frame in [1]. The selection of frames in the
spatial stream is a very important element, because most of the
videos are associated with particular motions. [14] using the
three dimensional convolutional neural network for human
action recognition. It shows that six frames is sufficient for
action recognize task. In our framework, the number of input
frames also set as six.
Temporal Convolutional Network. In this work, the
temporal ConvNet evaluates on optical flow extracted by
Lucas-Kanade method. Optical flow estimation quantifies and
evaluates the movement of objects in the field of computer
vision with multiple frames. It often work at multiple image
scales, evaluating in an energy framework. A common optical
flow method for solving a displacement field is to use eight
7
parameters’ models and approximate image neighborhoods
with polynomials [1, 15]. Another method sets the energy
based on constancy assumptions for intensity [16]. Unlike
spatial ConvNet, when processing through the temporal
stream, input video with consecutive frames is composed of
optical flow which contains hundreds of millions displacement
fields. Since we don’t need to estimate the motion directly,
this kind of the input depicts the movement among video
frames, making the action recognition more accurate.
4. Experimental Setup
In our experiment, video training is carried out on UCF-
101 and HMDB-51 datasets. We introduce these two datasets
and list the video categories respectively.
UCF-101. UCF-101 is an action videos data sets, collected
mainly from YouTube. Before UCF-101, UCF-50 is the
previous version of action recognition data set which contains
50 action categories. But now UCF-101 has 101 action
categories, extending training task for action recognition.
Videos in UCF-101 data set are gathered into 25 groups. Same
video group may share some similar feature, such like
analogous background or analogous visual angle. UCF-101
video categories are grouped into five types [17]:
(1) Body-Motion Only
(2) Human-Human Interaction
(3) Human-Object Interaction
(4) Playing Musical Instruments
(5) Sports
We select several kinds of videos about elderly and
children care for smart living from five types of categories in
the UCF-101 dataset, including BabyCrawling,
CuttingInKitchen, MoppingFloor, Swing, TrampolineJumping
and WalkingWithDog.
HMDB-51. HMDB mostly collected from movies, and a small
part coming from Internet databases, including YouTube, the
Prelinger archive and Google. HMDB, which holds six-
thousands of clips, divide into 51 action categories. Categories
can be grouped into five types [18]:
(1) Body movements with human interaction
(2) Body movements with object interaction
(3) Facial actions with object manipulation
(4) General body movements
(5) General facial action
We select nine categories of videos matched the research
topic of elderly and children care. The categories we select
include ClimbStairs, FallFloor, Hit, Pick, Push, Run, Smoke,
Walk and Wave.
We also collect some video of activities such as Sit, Stand,
Jogging, etc. from the Internet. There are totally 21 categories
activities are selected for experiments.
5. Experimental Results
In this work, our topic is to build a smart living for elderly
and children care. The action category we selected are
generalized into two types including indoor activities and
outdoor activities. Here the activity is set as outdoor activity if
an activity may occur in both indoor and outdoor. In
experiment results, two types of single stream ConvNets and
two stream ConvNet are selected as baseline systems.
Indoor Activities. For elderly and children, some behavior is
dangerous when they are at home. Crawling babies may be hit
by falling objects when they knock down the table; other
behavior like cutting with knife and mopping floor are still at a
high risk for the elderly. To sum up, BabyCrawling,
CuttingInKitchen, MoppingFloor are summarized as indoor
activities.
Outdoor Activities. Amusement facilities in the yard
sometimes cause the children to fall and hurt. So if we want to
build a smart living, we need to monitor the amusement
facilities. Therefore, two classes of action recognition videos,
Swing and TrampolineJumping, are put in the type of outdoor
activities. Some outdoor activities are static, such as Pick,
Smoke, WalkingWithDog, Push, Walk and Wave. The others
are relatively dynamic, including the classes of ClimbStair,
FallFloor, Hit and Run.
TABLE I
RESULTS COMPARISON
ConvNets Inputs
Singe Steam ConvNet Spatial
Single Steam ConvNet Temporal
Two Steam ConvNet Spatial, Temporal
Three Steam ConvNet Spatial, Temporal, Movement
Table I presents the recognition rates of the baseline systems
and proposed system. The recognition rate of single stream
ConvNets are 93.19% and 92.95% when using spatial stream
input and temporal stream input. After combine this two kind
of inputs, the two stream ConvNet perform better results than
uses only one kind of input. It gets 93.27% recognition rate of
selected 21 activities. The proposed system get better
recognition rates than all baseline systems. The recognition
rate of proposed system is 93.42%, It improves 0.23%, 0.47%
and 0.15% than spatial stream ConvNet, temporal stream
ConvNet and two stream ConvNet.
spatial
temporal
2 stream
3 stream
0.4
10
energy
Fig. 4 shows the convergence of the baseline systems and
proposed system. The green line is objective value of spatial
stream ConvNet. The cyan line is for temporal stream
ConvNet. The blue line is for two stream ConvNet. And the
red line is for proposed three stream ConvNet. The results
shows that proposed system can converge to the best value.
6. Conclusion
In this work, an elderly and children system which is based
on action recognition via deep convolutional neural network is
proposed. To recognize human action, a novel ConvNet
architecture called three stream ConvNet is proposed. Three
stream ConvNet considerate spatial information, temporal
information and movement of human for classify the human
activities. The proposed system can recognize more than 20
categories of human daily activities and detect the activities is
normal or unnormal. The recognition rate can achieve 93.42%.
REFERENCES
[1] K. Simonyan, A. Zisserman, “Two-Stream Convolutional Networks for
Results Action Recognition in Videos”, arXiv:1406.2199v2 [cs.CV], 2014, In
93.19% NIPS, 2014.
92.95% [2] O. Boiman and M. Irani. Detecting irregularities in images and in video.
93.27% IJCV, 2007.
93.42% [3] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri. Actions [1]
as space-time shapes. In ICCV, pages 1395–1402, 2005.
[4] I. Laptev and T. Lindeberg. Space-time interest points. In ICCV, 2003.
[5] D. B. Kris M. Kitani, Brian D. Ziebart and M. Hebert. Activity
forecasting. In ECCV, 2012.
[6] Andrej Karpathy, George Toderici, SankethShetty, Thomas Leung,
Rahul Sukthankar, Li Fei-Fei, “Large-scale Video Classification with
Convolutional Neural Networks”, In CVPR, 2014.
[7] P. Doll´ar, V. Rabaud, G. Cottrell, and S. Belongie. Behavior
recognition via sparse spatio-temporal features. In International
Workshop on Visual Surveillance and Performance Evaluation of
Tracking and Surveillance, 2005.
[8] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S.
Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast
feature embedding. arXiv preprint arXiv:1408.5093, 2014.
[9] N. Zhang, M. Paluri, M. Ranzato, T. Darrell, and L. Bourdev. Panda:
Pose aligned networks for deep attribute modeling. In CVPR, 2014.
[10] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva. Learning
deep features for scene recognition using places database. In NIPS, 2014.
[11] Du Tran, LubomirBourdev, Rob Fergus, Lorenzo Torresani, Manohar
Paluri, “Learning Spatiotemporal Features with 3D Convolutional
Networks”, arXiv:1412.0767v3 [cs.CV], 2015.
[12] Jeff Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus
Rohrbach, SubhashiniVenugopalan, Kate Saenko, Trevor Darrell,
“Long-term Recurrent Convolutional Networks for Visual Recognition
and Description”, arXiv:1411.4389 [cs.CV], 2015.

[13] M. Baccouche, F. Mamalet, C. Wolf, C. Garcia, And A. Baskurt.

Sequential Deep Learning For Human Action Recognition. In Human
Behavior Understanding. 2011.
[14] Shuiwang Ji; Wei Xu; Ming Yang; Kai Yu, "3D Convolutional Neural
Networks for Human Action Recognition," in Pattern Analysis and
Machine Intelligence, IEEE Transactions on , vol.35, no.1, pp.221-231,
Jan. 2013.
[15] G. Farneb¨ack. Two-frame motion estimation based on polynomial
expansion. In SCIA, pages 363–370, 2003.
[16] T. Brox, A. Bruhn, N. Papenberg, and J. Weickert. High accuracy
0.3
10 optical flow estimation based on a theory for warping. In Proc. ECCV,
pages 25–36, 2004.
0 10 20 30 40 50 60 70 80
training epoch
[17] K. Soomro, A. R. Zamir, and M. Shah. UCF101: A dataset of 101
human actions classes from videos in the wild. CoRR, abs/1212.0402,
Figure 4. Objective. 2012.

8
[18] P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan. Object
Detection with Discriminatively Trained Part Based Models. IEEE
Transactions on Pattern Analysis and Machine Intelligence, Vol. 32, No.
9, Sep. 2010
[19] A. Krizhevsky, I. Sutskever, and G. E Hinton,. ImageNet Classification
with Deep Convolutional Neural Networks NIPS 2012: Neural
Information Processing Systems, Lake Tahoe, Nevada