Automatic Object Recognition in a Light-Weight
Augmented Reality-based Vocabulary Learning
Application for Children with Autism
Tiffany Y. Tang Jiasheng Xu
Media Lab & Media Lab &
Innovative Technology for Autism Innovative Technology for Autism
Research Center Research Center
Wenzhou-Kean University Wenzhou-Kean University
yatang@kean.edu xujias@kean.edu
ABSTRACT
A number of previous controlled studies have underscored the
importance of early diagnosis and intervention in autism.
However, despite the technological advances, augmented reality-
based (AR) intervention for Chinese autistic children is still rare,
which motivates our study. In particular, in this paper, we present
a mobile vocabulary-learning application for Chinese autistic
children by creating authentic opportunities in outdoor and home
use. The core object recognition module is implemented in the
deep learning platform, TensorFlow, on one hundred training
models; unlike other sophisticated systems, the algorithm has to
run in an offline fashion. A pilot study aiming at investigating the
system’s feasibility and usability had been conducted with
typically developing children and their parents with very
promising and satisfying results. We also further tested
performance of the offline learning algorithm using seven animal
toys with very satisfying results. Since the current literature of
AR-technology on Chinese word-learning for children with
special needs is still in its infancy and arguably lacks rigor in
especially design and assessment, which thus offers limited
insights into its therapeutic efficacy, feasibility and applicability
of individualized intervention for autistic individuals, particularly
children. It is our hope that this preliminary study adds to our
understanding towards the usability and usefulness of such AR-
based mobile learning application.
CCS Concepts
• Information systems➝Human-centered
computing • Human computer interaction (HCI)
• Interaction paradigms ➝ Mixed / augmented reality.
Keywords
Object Recognition; TensorFlow; autism; lightweight; offline;
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that
copies bear this notice and the full citation on the first page. Copyrights
for components of this work owned by others than the author(s) must be
honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior
specific permission and/or a fee. Request permissions from
Permissions@acm.org.
ICIAI 2019, March 15–18, 2019, Suzhou, China
© 2019 Copyright is held by the owner/author(s). Publication rights
licensed to ACM.
ACM ISBN 978-1-4503-6128-6/19/03…$15.00
https://doi.org/10.1145/3319921.3319945
65
Pinata Winoto
Media Lab &
Innovative Technology for Autism
Research Center
Wenzhou-Kean University
pwinoto@kean.edu
vocabulary learning.
1. INTRODUCTION
Autism spectrum disorder (ASD) is a lifelong
neurodevelopmental disorder mainly characterized by repetitive
and restricted behaviors and deficits in social and communicative
skills interaction and emotion recognition [Charman and
Swettenham 2001; Zwaigenbaum et al 2015]. Compared to those
typically developing (TD) children, children with ASD have only
some limited channels to acquire new knowledge. Additionally,
although autistic children demonstrated delays in expressive and
receptive language, the extent of such delays varies across the
population [Szatmari et al 2003] and remains an active interest in
the literature. On the other hand, infant brains are very malleable,
so early intervention which largely capitalize on the great
potential of learning that an infant brain has could lead to positive
effects in limiting some developmental impairments [Dawson et al
2010; Su and Haga 2018; Zwaigenbaum et al 2015; French and
Kenney 2018], including early language and nonverbal skills
[Szatmari et al 2003].
For the past decade, thanks to the huge advancement of computer
technologies, including augmented and mixed reality (AR and
XR), a large number of AR-based interventions had been
proposed for home and classroom use (among many, [Escobedo et
al 2014; Chung and Chen 2016; Brandão et al 2015; Chen et al
2015; Cunha et al 2016; Bai et al 2015; Dragomir et al 2018; Liu
et al 2017; Lorenzo et al 2018; Daniels et al 2018]). While most of
the previous studies agree upon superior advantages AR-based
applications have offered, no significant learning outcomes had
been observed in almost all of the previous studies. To the best of
our knowledge, however, there are fewer published English
studies describing the use of AR-based applications in Chinese
special education classrooms, which motivates our study here. In
particular, we proposed a lightweight AR-based word-learning
application for young children with ASD based on the Google
TensorFlow platform which supports learning at anytime and
anywhere. To the best of our knowledge, it is one of the few in
China[Tang and Flatla 2016].
The organization of this paper is as follows. In Section two, we
will present previous works, followed by our system architecture
and design in section 3. A small pilot study with TD children will
be shown in Section 4. We conclude our paper with our
speculations on the popularity of such application in Chinese
special education classrooms.
2. PREVIOUS WORK 3. SYSTEM OVERVIEW
2.1 AR-based Application for Children with 3.1 The Core Module of Offline Object
Autism to Enhance Social and Communication Recognition Powered in the TensorFlow
Skills Platform
The majority of AR-based applications had been developed The core part of the imagine recognition module had implemented
to enhance autistic children’s social and communication skills within Google’s TensorFlow 1 machine learning framework.
with reported increased engagement and positive emotion. For Without modifying the sample codes, we took advantage of
example, the overlay of digital contents on top of physical ones in around one hundred already-trained models to cold start our
[Escobedo et al 2014] revealed that autistic children demonstrated system so that it could function in an offline fashion to facilitate
increased sustained and selective attention; meanwhile, due to the especially outdoor, home and non-classroom use. At present, our
inherent fun and attractiveness of the AR-based outputs, positive system can recognize one hundred common daily objects without
emotion had been elicited, which thus promoted engagement connecting to the internet, which thus offers tremendous
during therapies. Increased engagement has also been reported in advantage especially for rural users who have much less access to
in [Chung and Chen 2016] where AR technologies overlaying therapists [Tang and Flatla 2016]. We also offered further testing
with corresponding virtual images in tangible markers for children scenarios and results with seven animal toys to demonstrated the
with high-functioning autism. [Brandão et al 2015] also adopted usefulness of the offline learning algorithm.
AR technology in an interactive game book where children can
play with one 3D AR avatar with different facial expressions and 3.2 System User Interfaces
choosing the correct one to the right scenario, with an aim to train Our system went through several design iterations, which aims at
autistic children emotion and cognitive skills. AR-based facilitating the ease of use for autistic children. Figure 1 showed
applications were also developed to train autistic children’s the latest user interface.
emotion expression and social skills [Chen et al 2015], and
cognitive and social skills [Cunha et al 2016]. Enhancing pretend
play had been the focus in both [Bai et al 2015] and [Dragomir et
al 2018], while the latter sets the training in the classroom which
might not fully exploit the advantages of the AR technologies.
AR-based Google Glass has been adopted in two recent
studies to teach children social and communication skills. In
particular, Liu et al [2017] first systematically explored the
feasibility of autism-focused AR-based smart-glasses for training
social communication and pointed out that the AR-based can
significantly increase children’s engagement and fun, which thus
might in turn enhance the respective skills during intervention.
The AR-based Google Glass was studied in [Daniels et al 2018]
for home-based social affective learning in children with autism
with greater social acuity and increased eye-contact.
Despite the increasing popularity of AR-based technology among Fig. 1. The system can recognize four objects in one scene.
children, parents and therapists, findings in the clinical aspect is In Figure 1, the colored border corresponding the colored buttons;
not conclusive. For example, a recent study on AR for social skills when a user presses a colored button, the Chinese word for the
intervention failed to find significant improvement between item corresponding to the colored button will be shown and
groups [Lorenzo et al 2018]. Thus, further ecological studies are spoken, as shown in Figure 2. Notice in Figure 2, the user targeted
necessary. Despise it, due to the unique opportunity an AR-based a photo in a browser, which shows that our system is capable of
system can offer, it would continue to be present in future systems. supporting learning anywhere and anytime for these children.
2.2 AR-based Application to Target Other
Skills of Children with Autism
A few studies address other deficits of the children and train the
corresponding skills accordingly, including speech [da Silva et al
2014] which allows parents and therapists to join in during the
intervention section to increase engagement, science vocabulary
[McMahon et al 2016] to promote authentic learning opportunities,
science knowledge understanding [Hung et al 2016],
Among these studies, [Hung et al 2016] systematically compared
three different teaching materials including a typical picture book,
physical interactions and an AR graphic book, and revealed that
AR-based book offers a practical and hands ‐ on channel for Fig. 2. A sample example of the word recognition where both
children to explore and learn about corresponding knowledge. Chinese and English words appear.
However, although the new technology increased learning
motivation, the overall learning outcomes does not seem to
particularly favor this type of learning materials. 1
https://www.tensorflow.org/tutorials/images/image_recognition

66
3.3 System Testing with Animal Toys
In order to further examine the accuracy and feasibility of the
lightweight, offline deep-learning algorithm, we further tested the
algorithm with seven animal toys, with very satisfying results.
Figures 3 shows the two testing moments. Of note, our algorithm,
although cold-started with a small-scale of data, performs with
satisfying results; it can also recognize multiple items (see the
right photo in Figure 3).
Fig. 3. Two testing moments in our system with seven animal toys.
(left) recognizing a bear.(right) recognizing a bear and a giraffe at
the same time.
4. OBSERVATION AND DISCUSSION OF
A PILOT STUDY
We perform a small-scale usability testing with typical children
and adults in a public show on the university campus. Another
aim of testing is to examine the accuracy of the deep-learning
based object recognition module, especially during an offline
fashion. Figure 4 captured a moment the researcher was
demonstrating the application for a young girl whose mother was
seen by her side.
Fig. 4. A child is shown of the application recognizing an object
by the researcher.
Consistent with our speculations that the acceptability of the
application is surprisingly high, particularly for parents who
claimed that such AR-based applications are very rare in China;
67
and a few children who were demonstrated the application show
high interest in trying it after several rounds. Parents are
particularly satisfied with the audio module of the application.
However, due to the limited training sets, the accuracy of the
application is not overall satisfying. Despite it, we argued that
such lightweight applications are very desirable and crucial
especially for non-classroom use to integrate with the classroom-
based intervention for infants and younger children with autism
[Virnes et al 2015].
In summary, our feasibility and usability studies are aligned with
those in previous ones, particularly in [Liu et al 2017, Dragomir et
al 2018] that the application greatly attracts children’s attention,
which thus might promote learning at their own pace outside the
classroom.
5. CONCLUDING REMARKS AND
FUTURE WORK
It is highly essential to point out that although current deep-
learning based object-recognition research has achieved quite a lot
of impressive results, its feasibility and applicability in offline
fashion is largely unexplored, which is of particular importance
and critical in some areas, including artificial intelligence based
special education areas, where usability and portability is
preferred. As such, as one of the few attempts down this path, our
study offers a first and yet preliminary glimpse into the feasibility
of such technology.
On the other hand, since the current literature of AR-technology
on Chinese word-learning for children with special needs is still in
its infancy and arguably lacks rigor in especially design and
assessment, which thus offers limited insights into its therapeutic
efficacy, feasibility and applicability of individualized
intervention for autistic individuals, particularly children. It is our
hope that this preliminary study adds to our understanding
towards the usability and usefulness of such AR-based mobile
learning application.
6. ACKNOWLEDGMENTS
The authors gratefully acknowledge financial support from
Zhejiang Provincial Natural Science Foundation of China
(LGJ19F020001).
7. REFERENCES
[1] Almeida, C., Ramires, A., & Grohman, A. (2015). STAR:
Speech Theragy with augmented reality for children with
autism Spectrum disorders. In E. I. Systems (Ed.), J.
Cordeiro, S., Hammoudi, L. Maciaszek, O. Camp, J. Filipe
(pp. 379–396). Germany: Springer.
[2] Bai, Z., Blackwell, A.F. and Coulouris, G., (2015). Using
augmented reality to elicit pretend play for children with
autism. IEEE transactions on visualization and computer
graphics, 21(5), pp.598--610.
[3] Brandão, J., Cunha, P., Vasconcelos, J., Carvalho, V., Soares,
F. 2015. An Augmented Reality GameBook for Children
with Autism Spectrum Disorders. In Proc. of ICELW 2015.
[4] Cunha, P., Brandao, J., Vasconcelos, J., Soares, F., Carvalho,
V. (2016). Augmented reality for cognitive and social skills
improvement in children with ASD. In Remote Engineering
and Virtual Instrumentation (REV), 2016 13th International
Conference on (pp. 334-335). IEEE computer society:
Whashington: USA.
[5] Charman, T. and Swettenham, J. (2001). Repetitive
behaviors and social-communicative impairments in autism:
Implications for developmental theory and diagnosis. In J.A.
Burack, T. Charman, N. Yirmiya, & P.R. Zelazo (Eds.), The
development of autism: Perspectives from theory and
research. Mahwah, New Jersey. Lawerence Eribaum
Associates.
[6] Chen, C., Lee, I., Lin, L. (2015). Augmented reality-based
self-facial modeling to promote the emotional expression and
social skills of adolescents with autism spectrum disorders.
Res. Dev. Disabil. 36, 396–403.
[7] Chi-Hsuan Chung, Chien-Hsu Chen. (2016). Augmented
Reality Based Social Stories Training System for Promoting
the Social Skills of Children with Autism. In: Soares M.,
Falcão C., Ahram T. (eds) Advances in Ergonomics
Modeling, Usability & Special Populations. Advances in
Intelligent Systems and Computing, vol 486. Springer, Cham.
[8] Daniels, J., Schwartz, J. N., Voss, C., Haber, N., Fazel, A.,
Kline, A., Washington, P., Feinstein, C., Winograd, T. &
Wall, D. P. (2018). Exploratory study examining the at-home
feasibility of a wearable tool for social-affective learning in
children with autism. npj Digital Medicine, volume 1, Article
number: 32.
[9] Geraldine Dawson, Sally Rogers, Jeffrey Munson, Milani
Smith, Jamie Winter, Jessica Greenson, Amy Donaldson,
Jennifer Varley (2010). Randomized controlled trial of the
Early Start Denver Model: A developmental behavioral
intervention for toddlers with autism: Effects on IQ, adaptive
behavior, and autism diagnosis. Pediatrics. 2010. 125(1):
e17-23.
[10] Dragomir, M., Manches, A., Fletcher-Watson, S. and Helen
Pain, H. (2018). Facilitating Pretend Play in Autistic
Children: Results from an Augmented Reality App
Evaluation. In Proc. of the 20th International ACM
SIGACCESS Conference on Computers and Accessibility
(ASSETS '18), Pages 407-409.
[11] Escobedo, L., Tentori, M., Quintana, E., Favela, J., and
Garcia-Rosas, D. (2014). Using Augmented Reality to Help
Children with Autism Stay Focused. IEEE Pervasive
Computing, 13(1):38 - 46.
[12] French L., Kennedy E.M.M. (2018). Annual Research
Review: Early intervention for infants and young children
with, or at-risk of, autism spectrum disorder: a systematic
review. J Child Psychol Psychiatry.59(4):444-456.
[13] Gonzalo Lorenzo, Marcos Gómez-Puerta, Graciela Arráez-
Vera, Alejandro Lorenzo-Lledó. (2018). Preliminary study of
augmented reality as an instrument for improvement of social
skills in children with autism spectrum disorder. Education
and Information Technologies (2018).
68
[14] Hung, Y., Chen, C., & Huang, S. (2016). Applying
augmented reality to enhance learning: A study of different
teaching materials. Journal of Computer Assisted Learning,
[15] Liu, R., Salisbury, J. P., Vahabzadeh, A., and Sahin, N. T.
(2017). Feasibility of an Autism-Focused Augmented Reality
Smartglasses System for Social Communication and
Behavioral Coaching. Frontiers in Pediatrics. 5: 145.
[16] McMahon, D. D., Cihak, D. F., Wright, R. E., and Bell, S. M.
(2016). Augmented Reality for Teaching Science Vocabulary
to Postsecondary Education Students with Intellectual
Disabilities and Autism. Journal of Research on Technology
in Education, 48(1):38-56.
[17] G-Jen Hwang, Po-Han Wu, Chi-Chang Chen and Nien-Ting
Tu. (2016). Effects of an augmented reality-based
educational game on students' learning achievements and
attitudes in real-world observations. Interactive Learning
Environments, 24(8):1895-1906.
[18] Su, Maw S and Haga C. (2018). Effectiveness of cognitive,
developmental, and behavioural interventions for Autism
Spectrum Disorder in preschool-aged children: A systematic
review and meta-analysis. Heliyon. 4(9): e00763.
[19] Szatmari P., Bryson S.E, Boyle M.H, Streiner D.L, Duku E.
(2003). Predictors of outcome among high functioning
children with autism and Asperger syndrome. J Child
Psychol Psychiatry. 44(4):520-528.
[20] Tang, T. Y., and D. R, Flatla, D. R. (2016). Autism
Awareness and Technology- Based Intervention Research in
China: the Good, the Bad, and the Challenging. Proceedings
of Workshop on Autism and Technology - Beyond Assistance
& Intervention, in Conjunction with the CHI 2016.
[21] Virnes, M., Kärnä, E., Vellonen, V. (2015). Review of
Research on Children with Autism Spectrum Disorder and
the Use of Technology. J. of Special Education Technology,
30(1):13-27.
[22] Lonnie Zwaigenbaum, Margaret L. Bauman, Roula Choueiri,
Connie Kasari, Alice Carter, Doreen Granpeesheh, Zoe
Mailloux, Susanne Smith Roley, Sheldon Wagner, Deborah
Fein, Karen Pierce, Timothy Buie, Patricia A. Davis, Craig
Newschaffer, Diana Robins, Amy Wetherby, Wendy L.
Stone, Nurit Yirmiya, Annette Estes, Robin L. Hansen,
James C. McPartland, Marvin R. Natowicz. (2015). Early
intervention for children with autism spectrum disorder
under 3 years of age: recommendations for practice and
research. Pediatrics, 136, S60-81. doi:10.1542/peds.2014-
3667E.