IN T E R A C T I O N S J U LY A U G U S T 2 0 17 V O L U M E X X I V.

4 Association for
Computing Machinery

Intelligence on Tap:
Artificial Intelligence as a
New Design Material
Socially Just Design
On Low-Tech in
High-Tech Development
Teaching Design Online
Chatbots and the
New World of HCI
 INSPIRING MINDS
FOR 200 YEARS
Adas Legacy illustrates the depth
and diversity of writers, things, and
makers who have been inspired
by Ada Lovelace, the English
mathematician and writer.
The volume commemorates the
bicentennial of Adas birth in
December 1815, celebrating her
many achievements as well as
the impact of her work which
reverberated widely since the late
19th century. This is a unique
contribution to a resurgence in
Lovelace scholarship, thanks to the
expanding influence of women in
science, technology, engineering and
mathematics.

ACM Books is a new series of high quality books for the computer science community, published by
the Association for Computing Machinery with Morgan & Claypool Publishers.
 ACM Books M MORGAN & CLAYPOOL
&C P U B L I S H E R S
Publish your next book in the
ACM Digital Library
ACM Books is a new series of advanced level books for the computer science community,
published by ACM in collaboration with Morgan & Claypool Publishers.
Im pleased that ACM Books is directed by a volunteer organization headed by a
dynamic, informed, energetic, visionary Editor-in-Chief (Tamer zsu), working
closely with a forward-looking publisher (Morgan and Claypool).
Richard Snodgrass, University of Arizona

books.acm.org ACM Books

will include books from across the entire
spectrum of computer science subject
matter and will appeal to computing
practitioners, researchers, educators, and
students.
will publish graduate level texts; research
monographs/overviews of established
and emerging fields; practitioner-level
professional books; and books devoted to
the history and social impact of computing.
will be quickly and attractively published
as ebooks and print volumes at affordable
prices, and widely distributed in both print
and digital formats through booksellers
and to libraries and individual ACM
members via the ACM Digital Library
platform.
is led by EIC M. Tamer zsu, University of
Waterloo, and a distinguished editorial
board representing most areas of CS.

Proposals and inquiries welcome! Association for

Contact: M. Tamer zsu, Editor in Chief
Computing Machinery
booksubmissions@acm.org Advancing Computing as a Science & Profession
CONTENTS J U LYAU G U S T 2 017
INTER ACTIONS
V O L U M E X X I V. 4

16 24 56

ENTER COLUMNS F E AT U R E S

10 DEMO HOUR 22 CONFESSIONS 28 COVER STORY

InTouch Wearables The Man Who Had Them All Intelligence on Tap:
Jinsil Hwaryoung Seo, Annie Uri Kartoun Artificial Intelligence
Sungkajun, Meghan Cook as a New Design Material
Tea with Crows 24 THE DESIGNERS SPEAKEASY Designers will one day be able to
Young Suk Lee Teaching Design Online use AI services in their creations.
Brighter Than a Thousand Suns Uday Gajendar How should they approach this
Susanna Hertrich, Akitoshi Honda emerging design material?
Second Skin Lars Erik Holmquist
Rachel Freire, Paul Strohmeier,
Cedric Honnet 34 Avoiding Pitfalls When
Using Machine Learning in
14 WHAT ARE YOU READING? HCI Studies
Ann Light Vassilis Kostakos, Mirco Musolesi

16 HOW WAS IT MADE? 38 Chatbots and the New

cringeMACHINE World of HCI
Asbjrn Flstad, Petter Bae Brandtzg
18 DAY IN THE LAB
Information Science 44 So You Want to Be an AI Designer?
at CU Boulder Nina (Zhuxiaona) Wei

50 Inspiring Innovation:
On Low-Tech in
High-Tech Development
MIDDLE IMAGE BY TOBIA S TOF T / FLICKR

Zara Mirmalek

56 Making Your
Presentation Accessible
Richard E. Ladner, Kyle Rector

2 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG


60
FORUMS
60 INTERACTION AND ARCHITECTURE
Toward Intelligent Environments:
Supporting Reflection with
Smart Objects in the Home
Maliheh Ghajargar
63 COMMUNITY + CULTURE
Socially Just Design and
Engendering Social Change
Lynn Dombrowski
66 HCI EDUCATION
The Future of User Experience
Education
L E F T I M A G E C O U R T E S Y O F H O L I; M I D D L E I M A G E B Y G R A M VA A N I; R I G H T I M A G E B Y E L I B L E V I S
Andrea Peer
70 UNIVERSAL INTERACTIONS
The 3Cs for Preschool
Childrens Technology:
Create, Connect, Communicate
Juan Pablo Hourcade, Luiza Superti
Pantoja, Kyle Diederich, Liam
Crawford, Glenda Revelle
INTER ACTIONS. ACM.ORG
Machine learning is
no silver bullet
for HCI research.
P. 34
74 88
D E PA R T M E N T S
74 EVALUATION AND USABILITY 5 WELCOME
Usability, Tested? Change, Challenges, Constants,
Neha Kumar, Naveena Karusala, Continuity, and Clarity
Aaditeshwar Seth, Biswajit Patra Simone Barbosa, Gilbert Cockton
78 DESIGN AS INQUIRY 6 BLOG@IX
Exploring DIY Tutorials Toward Affective Social
as a Way to Disseminate Interaction in VR
Research Through Design Giulio Jacucci
Audrey Desjardins, Ron Wakkary,
Will Odom, Henry Lin, 85 COMMUNITY SQUARE
Markus Lorenz Schilling HCI out of Namibia
Anicia Peters,
Heike Winschiers-Theophilus,
Tuomo Kujala
87 COMMUNITY CALENDAR
88 VISUAL THINKING GALLERY
Woman (Seated) with
Tablet Computer
Eli Blevis
On the cover: Composition by Andrij Borys
Associates; photo by Oliver Hoffmann
J U LY A U G U S T 2 0 17 INTER ACTIONS 3
INTER ACTIONS A B I M O N T H LY P U B L I C AT I O N O F A C M
Editors-in-Chief Interactions (ISSN 1072-5520) is published
Simone Barbosa six times a year in January, March, May, July,
Gilbert Cockton September, and November, by the Association
for Computing Machinery, Inc., 2 Penn Plaza,
Director of Publications Suite 701, New York, NY 10121-0701. Periodicals
Scott Delman postage paid at New York, NY 10001 and at
additional mailing offices. POSTMASTER:
Executive Editor Diane Crawford Please send address changes to Interactions,
Managing Editor John Stanik ACM, 2 Penn Plaza, Suite 701, New York, NY
Art Director Andrij Borys, 10121-0701.
Andrij Borys Associates
Copy Editor Kate Crane Editorial Information
Assistant to the Editors-in-Chief Rachel Clarke To contact Interactions, email our editors at
Editorial Assistant Jade Morris eic@interactions.acm.org or
feedback@interactions.acm.org
Forum Editors The EiCs generally respond to inquiries,
>ENTER: Rachel Clarke including those concerning unsolicited
COMMUNIT Y + CULTURE: Christopher Le Dantec manuscripts, in 2-3 weeks.
DESIGN A S INQUIRY: Daniela K. Rosner
E VALUATION AND USABILIT Y: Advertising
David Siegel and Susan Dray ACM Advertising Department
HCI EDUCATION: Sukeshini Grandhi 2 Penn Plaza, Suite 701
HE ALTH MAT TERS: Yunan Chen New York, NY 10121-0701
INTER ACTION AND ARCHITECTURE: Mikael Wiberg (212) 869-7440 FAX: (212) 869-0481
INTER ACTION TECHNOLOGIES: Albrecht Schmidt
SUSTAINABILIT Y IN (INTER)ACTION: Lisa Nathan Advertising Sales Account Manager
THE BUSINESS OF UX: Daniel Rosenberg Ilia Rodriguez
THE NE X T BILLION: Nithya Sambasivan ilia.rodriguez@hq.acm.org
UNIVERSAL INTER ACTIONS: Juan Pablo Hourcade
VISUAL THINKING GALLERY: Eli Blevis (Curator) Publications Board
Co-Chairs: Jack Davidson and Joseph Konstan
Columnists Board Members: Ronald F. Boisvert,
MAKE IT WORK: Jonathan Bean Karin K. Breitman, Terry Coatta,
Ps AND Qs: Elizabeth F. Churchill Anne Condon, Nikil Dutt, Roch Guerrin,
THE DESIGNERS SPE AKE A SY: Uday Gajendar Carol Hutchins, Yannis Ioannidis,
M. Tamer Ozsu, Mary Lou Soffa,
Founding Editors Alex Wade, Keith Webster
John Rheinfrank and Bill Hefley
Publications Office
ACM, 2 Penn Plaza, Suite 701
New York, New York 10121-0701, USA
+1-212-869-7440
FAX: +1-212-869-0481
Subscriptions
Various subscription rates are available;
for more information please visit
https://campus.acm.org/interactions/subscribe
4 INTER ACTIONS J U LY A U G U S T 2 0 17
Single copies are $13.00 to members, $20.00
to nonmembers. Please send orders prepaid
plus $4.00 for shipping and handling to ACM
Member Services Dept., General Post Office,
P.O. Box 30777, New York, NY 10087-0777 or
call +1-212-626-0500. For credit card orders,
call +1-800-342-6626. Order personnel
available 8:30-4:30 EST. After hours, please
leave message and order personnel will
return your call.
Change of Address
acmcoa@acm.org
For other services, questions, or information:
acmhelp@acm.org
ACM Copyright Notice
Copyright 2017 by Association for Computing
Machinery, Inc. (ACM). Permission to make
digital or hard copies of part or all of this work
for personal or classroom use is granted
without fee provided that copies are not
made or distributed for profit or commercial
advantage and that copies bear this notice and
full citation on the first page. Copyright for
components of this work owned by others than
ACM must be honored. Abstracting with credit
is permitted.
To copy otherwise, to republish, to post on
servers, or to redistribute to lists, requires
prior specific permission and/or fee. Request
permission to publish from: Publications Dept.,
ACM, Inc., Fax +1-212-869-0481 or
email permissions@acm.org
For other copying of articles that carry a code
at the bottom of the first or last page or screen
display, copying is permitted provided that
the per-copy fee indicated in the code is
paid through the Copyright Clearance Center,
222 Rosewood Drive, Danvers, MA 01923,
+1-978-750-8400, +1-978-750-4470 (fax).
INTER ACTIONS. ACM.ORG
 WELCOME
Simone Barbosa,
PUC-Rio
Change, Challenges, Constants,
Continuity, and Clarity
A
s Interactions approaches its
first quarter-century, it has
grown with the academic
and professional disciplines
of human-computer
interaction (HCI) and interaction design
(IxD). Once the magazine enters its
second quarter-century, it will continue
to broaden as long as HCI and IxD
continue to evolve, adapt, and reform.
And yet even at the close of Interactions
first half-century, there will be
continuities that have held some things
constant irrespective of the changes and
challenges to come.
Much of the breadth of this issue is
familiar ground. We have features on
conference accessibility (Ladner and
Rector) and the enduring importance of
low-tech artifacts in an ever higher-tech
world (Marmalek). We chronicle design
innovations in Demo Hour and How
Was It Made?, and present concerns
about educating and enabling current
and future interaction design specialists
(Designers Speakeasy column, HCI
Education and Design as Inquiry forums).
We also tackle creative preschool
education (Universal Interactions forum).
For each educational focus, challenges
arise due to the diversity of those being
educated and/or enabled. At one level,
the need for education remains
a constant, but at another, the changing
nature of those being educated, and
the manner in which they are educated,
introduce new challenges.
Interactions needs to share more
early-career experiences as
a counterbalance to those of
the default well-established sages.
DOI: 10.1145/3107082 COPYRIGHT HELD BY AUTHORS
INTER ACTIONS. ACM.ORG
Gilbert Cockton,
Northumbria
University
Some of what we cover in this issue
is not new but nevertheless remains
relatively underrepresented in HCI
and IxD. The Evaluation and Usability
forum addresses issues in the Global
South, and Community Square
presents Namibian perspectives
on responses to these and other
challenges. The Community + Culture
forum discusses social change and
justice through IxD and HCI. None
of these topics is yet mainstream,
so there is still work to be done in
achieving balance here across each year
of Interactions content. In this sense,
change will have to be a constant for
sourcing articles for Interactions.
There are three new contributions
to breadth in this issue. The first is
Ann Lights What Are You Reading?
with a very broad range of books
read and unread. The second is Nina
Weis feature on her experiences as
a relatively young IxD specialist in
Baidus artificial intelligence (AI)
research center. Interactions needs to
share more early-career experiences
as a counterbalance to those of the
default well-established sages. The
third debut of sorts is Day in the Lab,
where Colorado Information Sciences
lab is an entire department, working
across computational social science,
digital humanities, crisis informatics,
digital identity, personal information
management, philanthropic
informatics, data ethics and policy,
network science, machine learning,
information visualization, and
diversity and inclusion in STEM. This
is neither HCI nor IxD as we have come
to know it, and thus it is no surprise to
see their research extending to human-
data interaction. Together, these new
directions set us up for this issues
main focus: AI and big data.
Our cover story, Confessions
column, and two features (Kostakos
and Musolesi, Flstad and Brandtzg)
consider the prospects for AI and big
data in research and practice, as well as
their potential impacts on the future
directions of HCI and IxD. Weis feature
provides a balancing perspective on AI
design from a young professional, while
the Interaction and Architecture forum
explores the positive reflective benefits
from smart objects.
What is clear across this issue are
the constants and continuities, and
some changes that will or must follow
in the face of emerging challenges.
What is not clear, unsurprisingly,
is where this is all headed, but as
professionals or researchers we
can shape this future as it emerges.
Sometimes, however, it will all be
one big blur. But as we close with this
issues Visual Thinking Gallery, we
can reflect on how often a big blur can
actually make some things really clear.
Simone Barbosa and Gilbert Cockton
eic@interactions.acm.org
J U LY A U G U S T 2 0 17 INTER ACTIONS 5
 BLOG@IX
Giulio Jacucci,
University of Helsinki
Toward Affective
Social Interaction in VR
I
first encountered VR in the late
1990s, as a researcher looking at
how it provided engineers and
designers an environment for
prototyping. After that I became
more interested in looking at
how to augment reality and our
surrounding environment. However,
although VR had been around for
decades by that point, there were
many aspects, in particular from an
interaction point of view, that I felt still
deserved investigation. VR is enjoying
renewed interest thanks to the recent
proliferation of consumer products,
content, and applications. This is
accompanied by unprecedented interest
from consumers and by the maturity
of VR, now considered less and less to
be hype and to be more market ready.
However, important challenges remain,
associated with dizziness and the limits
of current wearable displays, as well
as interaction techniques. Despite
these limitations, application fields are
flourishing in training, therapy, and
well-being beyond the more traditional
VR fields of games and military
applications.
One of the most ambitious research
goals for interactive systems is to
be able to recognize and influence
emotions. Affect plays an important
role in all we doit is in fact an
essential aspect of social interaction. So
the study of affective social interaction
in VR can be important to the above-
mentioned fields to support mediated
communication. For example, in
mental or psychological disorders,
VR can be used for interventions and
training to monitor patient engagement
and emotional responses to stimuli,
providing feedback and correction on
particular behaviors. Moreover, VR
is increasingly accepted as a research
platform in psychology, social science,
6 INTER ACTIONS J U LY A U G U S T 2 0 17
The Interactions website (interactions.acm.org) hosts a stable of bloggers who
share insights and observations on HCI, often challenging current practices. Each
issue well publish selected posts from some of the leading voices in the field.
and neuroscience as an environment
that helps introduce contextual and
dynamic factors in a controllable way.
In such disciplines, affect recognition
and synthesis are critical to many
investigated phenomena.
MULTIMODAL SYNTHETIC
AFFECT IN VR
In social interaction, emotions can
be expressed through gestures,
posture, facial expressions, speech
and its acoustic features, and touch.
Our sense of touch plays a large role
in establishing how we feel and act
toward another person, considering,
for example, hostility, nurturance,
dependence, and affiliation.
Having done work on physiological
and affective computing and haptics
separately, I saw a unique way to
combine these techniques to develop
synthetic affect in VR, combining
different modalities. For example,
the emotional interpretation of
touch can vary depending on cues
from other modalities that provide
a social context. Facial expressions
have been found to modulate the
touch-perception [1] and post-touch-
orienting response. Such multimodal
affect stimuli are perceived differently
according to individual differences
of gender and behavioral inhibition.
For example, behavioral inhibition
sensitivity in males was associated with
stronger affective touch perception [2].
Taking facial expressions and touch
as modalities for affective interaction,
we can uncover different issues in
their production. Currently, emotional
expression on avatars can be produced
using off-the-shelf software that
analyzes the facial movements of an
actor modeling basic expressions, head
orientation, and gaze. Subsequently, the
descriptions are used to animate virtual
characters. Emotional expressions in
avatars are often the result of a process
involving several steps ensuring that
these relate to the intended emotions.
The expressions are recorded first by
capturing the live presentation from
a professional actor, using a facial-
tracking software that also animates a
virtual character. Expressions can then
be manually adjusted to last for the same
amount of time and end with a neutral
expression. Different animations are
created for each distinct emotion type.
The expressions can then be validated
by measuring the recognition accuracy
of participants who watch and classify
the animations. This process works well
to customize the facial expression to the
intended use in replicable experiments.
But this is resource-intensive and does
not scale well for other uses where facial
expression might need to be generated
in greater variations (for expressing
nuances) or for generalizability, since
every expression is unique. While
mediated touch has been proven to
affect emotion [1,3], behavior research
into the deployability, resolution, and
fidelity of haptics is ongoing. However,
in our recent studies, we compared
several techniques to simulate a virtual
hand of a character touching the hand of
a participant [4].
Emotion tracking is more
challenging in the case of a wearable VR
headset, as facial expressions cannot be
easily tracked through recent computer-
vision software [5]. For recognizing
change in psychophysiological states
or to assess emotional responses to
particular stimuli, physiological sensors
can be used. These sensors are also
being integrated into more and more
off-the-shelf consumer products, as
in the case of electrodermal activity
(EDA) or electroencephalogram (EEG).
While EDA provides a way to track
INTER ACTIONS. ACM.ORG
Figure 1. Bringing it all together: hand tracking of the user through a glass. Wearable haptics, an EEG cap, and an HMD for VR allow the simulation
of a situation in which a person sitting in front of the user touches her hand with different facial expressions [1,2,4].

arousal among other states and is easy
to use unobtrusively, it is suited for
changes in the order of minutes, not for
time-sensitive events in seconds and
milliseconds. EEG, on the other hand,
increasingly provided in commercial
devices, is better suited for time-precise
measurements. For example, the study
of how emotional expressions modulate
the processing of touch can be done by
event-related potential (ERP) in EEG
resulting from touch. Studies show
that the use of EEG is compatible with
commonly available HMDs.
Eye tracking, which recently
appeared commercially to be used
inside HMDs, can be used both to
identify whether users attend to
a particular stimulus to track its
emotional response, and to track
psychophysiological phenomena such
as cognitive load and arousal. As an
example, the setup in Figure 1 includes
VR, haptics, and physiological sensors.

Figure 2. RelaWorld using VR and physiological sensors (EEG) for a neuroadaptive meditation environment [6].

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 7

 BLOG@IX
It can be used to simulate a social
interaction at a table where mediated
multimodal affect can be studied while
an avatar touches the users hand, at the
same time delivering a facial expression.
ACM
ACM Conference
Conference The user recognizes the virtual hand
in Figure 1A as her own hand, as it is
Proceedings
Proceedings synchronized in real time.
This setup can be used for a number
Now
Now Available via
Available via of training, entertainment, or therapy
purposes. For example, a recent product
Print-on-Demand!
Print-on-Demand! applies VR for treating anxiety patients.
Recent studies have evaluated the
impact in training autism spectrum
Did you know that you can disorder patients to apply this to dealing
now order many popular with anxiety. In our own recent study,
we used the same setup for an air
ACM conference proceedings
hockey game. The haptics simulated the
via print-on-demand? hitting of the puck, and the emotional
expression of the avatar allowed us to
study effects on players performance
Institutions, libraries and and experience of the game.
individuals can choose
from more than 100 titles FUTURE STEPS
VR devices, applications, and content
on a continually updated are emerging quickly. An important
list through Amazon, Barnes feature in the future will be the
affective capability of the environment,
& Noble, Baker & Taylor, including the recognition and synthesis
Ingram and NACSCORP: of emotions.
A variety of research challenges exist
CHI, KDD, Multimedia,
for affective interaction:
SIGIR, SIGCOMM, SIGCSE, Techniques in recognizing users
SIGMOD/PODS, emotions from easily deployable
sensors, including the fusion of signals.
and many more. Physiological computing is advancing
fast in research and in commercial
products. Vision-based solutions that
For available titles and track facial expression have recently
ordering info, visit: seen success; physiological-based sensors
librarians.acm.org/pod could soon follow suit.
Synthesis of affect utilizing
multiple modalities, as exemplified
here. For example, combining touch
and facial expression, but considering
also speech and its acoustic features
and other nonverbal cues. How
to ensure that these multimodal
expressions are generally valid and can
be generated uniquely.
While these are challenges,
the potential application fields are
numerous and replete with emerging
evidence of their relevance:
Training in, for example, emergency
or disaster situations, but in principle
in any setting where a learner needs to
simulate a task in an environment where
DOI: 10.1145/3097462 COPYRIGHT HELD BY AUTHOR
8 INTER ACTIONS J U LY A U G U S T 2 0 17
she needs to attend to numerous features
and social interaction.
In certain training situations,
affective capabilities are essential to
carrying out the task, such as in therapy,
which can be more physical, as in limb
injuries, or more mental, as in autism
disorder and social phobias, or both, as
in cases such as stroke rehabilitation.
In several of these situationsfor
example, mental disorders such as
autism, anxiety, and social phobias
the patient practices social interaction
while monitoring how they recognize or
respond to emotional situations.
Well-being examples such as
physical exercise and meditation
(Figure 2). Affective interaction here
can motivate physical exercise or
monitor psychophysiological states
such as engagement or relaxation.
Endnotes
1. Ravaja, N., Harjunen, V., Ahmed, I.,
Jacucci, G., and Spap, M.M. Feeling
touched: Emotional modulation of
somatosensory potentials to interpersonal
touch. Scientific Reports 7 (2017), 40504.
2. Harjunen, V.J., Spap, M., Ahmed, I.,
Jacucci, G., and Ravaja, N. Individual
differences in affective touch: Behavioral
inhibition and gender define how
an interpersonal touch is perceived.
Personality and Individual Differences 107
(2017), 8895.
3. Spap, M.M., Hoggan, E.E., Jacucci, G.,
and Ravaja, N. The meaning of the virtual
Midas touch: An ERP study in economic
decision making. Psychophysiology 52, 3
(2015), 378387.
4. Ahmed, I., Harjunen, V., Jacucci, G.,
Hoggan, E., Ravaja, N., and Spap, M.M.
Reach out and touch me: Effects of four
distinct haptic technologies on affective
touch in virtual reality. Proc. of the 18th
ACM International Conference on Multimodal
Interaction. ACM, 2016, 341348.
5. Affectiva; http://www.affectiva.com/
6. Kosunen, I., Salminen, M., Jrvel, S.,
Ruonala, A., Ravaja, N., and Jacucci, G.
RelaWorld: Neuroadaptive and immersive
virtual reality meditation system. Proc. of
the 21st International Conference on Intelligent
User Interfaces. ACM, 2016, 208217.
Giulio Jacucci is a professor in computer
science at the University of Helsinki and
founder of MultiTaction (www.multitaction.
com). His research interests include multimodal
interaction; physiological, tangible, and
ubiquitous computing; search and information
discovery; as well as behavioral change.
giulio.jacucci@helsinki.fi
INTER ACTIONS. ACM.ORG
The newest
ACM forum.
Contributions
that cover the vast
information-rich world
where computing
Captivating
is embedded topics.
everywhere. Net Neutrality
and the Regulated
ACMs Ubiquity is
Internet
the online magazine
oriented toward the The End of Life
future of computing As We Know It
and the people who A Shortage of
are creating it. Technicians
We invite you to The Fractal
participate: leave Software
comments, vote for Hypothesis
your favorites, or
submit your own Your Grandfathers
contributions.
OldsmobileNOT!
Superscalar
Smart Cities

Visit us at
http://ubiquity.acm.org/blog/
10 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG
 10 D E M O H O U R
14 W H AT A R E YO U R E A D I N G ?
16 H OW WA S I T M A D E ?
18 DAY I N T H E L A B

ENTER

DEMO
HOUR

1. InTouch
Wearables
InTouch Wearables is a set of wearables that consists
of dresses and wearable accessories that allow a
mother and child to share remote touches through
garments with ambient feedback. This was developed
to explore how remote touches can convey emotion and
help people stay connected between remote locations.
Using InTouch Wearables, a parent can increase the
vividness of her conversation with a child through
contextualized touch, and loved ones may enhance the
affective tone of their communication using a remote
touch technology. All the electronic components for
sensing human touches and actuating the color-changing
garment are integrated into the main fabrics.

http://softinteraction.com/portfolio/intouch-wearables
https://youtu.be/ztErIhRWIvs
Seo, J.H., Sungkajun, A., and Cook, M.
InTouch Wearables: Exploring ambient remote
touch in child-parent relationships.
Proc. of TEI17. ACM, New York, 2017, 671676.

Jinsil Hwaryoung Seo, Annie Sungkajun, and Meghan Cook,

Texas A&M University
hwaryoung@tamu.edu

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 11

 DEMO
HOUR

2
A playful kinetic
sculpture invites
people to tea.

3
The ornament on
the helmet lights up
when radiation is
encountered.

3
Scanning for
radiation outside
Fukushima.

2. Tea with dance. When the audience leaves,

the table transforms back into
Tea with Crows: Towards
socially engaging digital
imperceptible radiation. The
protagonist of a short film walks
Crows
brighter than a thousand suns images by susanna hertrich.

its original swan shape. Tea interaction. Proc. of CHI17 across the devastated areas near
with Crows aims to create an Extended Abstracts. ACM, the nuclear power stations in
Tea with Crows explores
engaging interaction through New York, 2017; http://dx.doi. Fukushima. He wears a uniform
experiential aesthetics by
shared experience, emotionally org/10.1145/3027063.3050429 that allows him to measure
embedding playful kinetic
connecting people together and display ambient radiation.
motion and shape-changing
through technology in a social Young Suk Lee, Indiana University His handheld device contains a
interfaces into interactive
setting. artdesignys@gmail.com Geiger counter kit. The readings
art. When people are nearby,
from the device are transformed
the sculptural table opens
http://www.youngsuklee.com/ into illuminations through
its wings in three directions,
3. Brighter
TeawithCrows.html ornamentation on his helmet.
like a blooming flower. Once
https://vimeo.com/184890191 His uniform is a reminder of
opened, the table reveals three
cups and saucers, inviting the
Tea with Crows: Experiencing
proactive ubiquitous
Than a traditional Samurai clothing, as
well as contemporary workers
audience to tea time. The upper
sculpture is then activated,
technology by interactive Thousand Suns outfits. As the poetic short film
art. Proc. of TEI17. ACM, New progresses, information on
and three black birds attached Brighter Than a Thousand Suns
York, 2017; http://dx.doi. radiation suggests symbolic
to a chandelier begin a playful is an art project that explores
org/10.1145/3024969.3025058 meaning revealed over time.

12 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG


s e c o n d s k i n p h o t o b y a l l a n a m a t o ; m o d e l : a l i c i a w i t t; v r g l o v e s i n c o l l a b o r a t i o n w i t h a r t y o n m a x i m . s e c o n d s k i n d e t a i l p h o t o b y r a c h e l f r e i r e .
http://www.susannahertrich.
com/art/1000suns.shtml
https://vimeo.com/205215267
Hertrich, S. and Honda, A.
Brighter Than a Thousand Suns:
A uniform to detect and display
radiation. Proc. of TEI17. ACM,
New York, 2017, 659663.
Susanna Hertrich, Institute of
Experimental Design and Media
Cultures, Academy of Art and
Design FHNW
mail@susannahertrich.com
Akitoshi Honda, Ahonda.org
info@ahonda.org
DOI: 10.1145/3096482 2017 ACM 1072-5520/17/07 $15.00
INTER ACTIONS. ACM.ORG
4
An open source
eTextile platform
showcases elastic
materials and
an otherworldly
aesthetic.
4
Close-up detail of Second Skin eTextile
with stretch fabric and electronics.
4. Second Skin
Second Skin is an eTextile concept
for rapid prototyping, using
elastic materials and techniques
from activewear and lingerie. It
explores wearable electronics
that can withstand the robust
requirements of human
interaction, drawing inspiration
and practical solutions from
traditional pattern cutting and
stretch-garment technology.
The project explores the Alice
in Wonderland aesthetic of
eTextiles, focusing on scale and
circuitry as a design feature. The
aim is to facilitate both maker
and manufacturable techniques
to create viable, provocative
designs. This open source textile
platform invites others to adapt,
add functionality, and experiment
to optimize techniques and
designs that can be shared among
designers and researchers.
http://www.rachelfreire.com/
second-skin-login/
https://www.youtube.com/
watch?v=vd2GedFZLIs
Freire, R., Honnet, C., and
J U LY A U G U S T 2 0 17
Strohmeier, P. Second Skin: An
exploration of eTextile stretch
circuits on the body. Proc. of
TEI17. ACM, New York, 2017,
653-658. DOI: https://doi.
org/10.1145/3024969.3025054
Rachel Freire, Rachel Freire
Studio
rachel@rachelfreire.com
Paul Strohmeier, University of
Copenhagen
Cedric Honnet, Datapaulette
INTER ACTIONS 13
 WHAT
ARE YOU
READING?
Ann Light
For an English graduate
who read avidly for
pleasure through school
and university, I am
disappointed in how little
I read these days. I am
tempted to claim I write
more words than I read,
especially if we include
university administrative
writing. I have reached
a point where I feel
that as long as I buy the
books I care about and
pay the royalties, I have
14 INTER ACTIONS J U LY A U G U S T 2 0 17
done enough to show my
appreciation of them.
Reading them is a whole
other matter.
Atypically, you find me
deep in sci-fi. The book
is Crashing Heaven by Al
Robertson, given to me
by the author at an event
I ran on counterfactual
worlds and the design of
innovation. Inspired by
Philip K. Dicks The Man
in the High Castle (another
novel I read recently for
work) and other fiction in
which the Axis Powers win
the Second World War, the
day was structured
to give people parallel
worlds to imagine, fill
with novel technologies,
and then consider as a
play of tools and values.
Al was a participant in
the workshop; by the end,
he had co-constructed
a universe in which the
British Reformation
never happened, sin-o-
meters abounded, and all
electricity was controlled by
the Pope. For my part, I was
about to plunge into a world
of artificial intelligences
and giant spaceships,
where dead people sing and
puppets can run your brain.
Specs
Focus: Design research,
social well-being,
sustainable futures,
politics of participation,
long-term social and
cultural impact of digital
networks in post-industrial
and developing regions
Base: University of Sussex,
Brighton, U.K.
Mostly, when I get books
(and I love to buy them),
they sit in a queue to wait
till I go on holiday. Because I
will see Al again quite soon,
his book jumped the other
non-academic hardbacks
waiting for me (Johnny
Marrs autobiography on
his incredible musical
career Set the Boy Free
and Robert MacFarlanes
lyrical walking reflection
The Old Ways). I am curious
I was about to
plunge into a
world of artificial
intelligences and
giant spaceships,
where dead
people sing
and puppets can
run your brain.
INTER ACTIONS. ACM.ORG

about his work. And I am
grateful for his gift. As
usual, obligation is driving
my reading choices. But it
is also reminding me how
much I love to read fiction;
the story has caught me and
I am really happy to be back
in a page-turning alternative
reality.
In the other corner is the
pile of academic books that
I keep meaning to open. Top
of this pile is the weighty
Oxford Handbook of Process
Philosophy and Organization
Studies. It is about time that
I learnt of ontologies that
are not object-oriented; I
write so much about process,
relations, and states that
change. Still, Im not quite
there yet with this one.
Joining it is a slim new
volume by Luce Irigaray,
DOI: 10.1145/3095759 COPYRIGHT HELD BY AUTHOR
INTER ACTIONS. ACM.ORG
ACM
LEARNING CENTER
RESOURCES
FOR LIFELONG LEARNING
another philosopher of
becoming, entitled To Be
Born. I bought it from the learning.acm.org
author last week, almost
as a memento, after
listening to her introduce
her phenomenology of
birth. She was visiting the
University of Sussex for a
conference in her honor, and
sharing a new conception
of what it is to come into
the world and how best to
nurture that arrival. Having
heard the gist of it, I will
probably only read it if I
need to quote her, much as
I would like to settle down
with it.
Instead, I will be
reading a colleagues
working paper on Deep
Transitions: Emergence,
Acceleration, Stabilization
and Directionality (Johann
Schott and Laur Kanger) Online Courses
because it will be useful in
framing how I talk about from Skillsoft
vulnerabilities in current
technology development Online Books
in an application for
research funding that I am from Safari, Books24x7,
preparing. It seems all my
academic reading at the
Morgan Kaufmann
moment is on change. and Syngress
Ann Light is professor of
design and creative technology Webinars
at the University of Sussex in
Brighton, U.K., where she leads
on todays hottest topics
the Creative Technology Research
Group.
in computing
ann.light@sussex.ac.uk
J U LY A U G U S T 2 0 17 INTER ACTIONS 15
 HOW
WAS IT
MADE?
cringeMACHINE
Describe what you made.
cringeMachine is a
temporary installation
composed of 20 truncated
tetrahedrons that respond
to the sounds around it. Our
initial idea was to create a
transformable sculpture
made of self-similar parts
that could be assembled into
a variety of configurations.
When you make sufficient
noise around it, the
cringeMachine sends a
pulse of lights through a
sequence of LEDs built
C omputer rendering of original
design concept.
A
 ssembling flexible hinge
joints.
16 INTER ACTIONS J U LY A U G U S T 2 0 17
into its component parts,
varying in color and
intensity.
What materials and tools
did you use?
We used thin twin-wall
polypropylene sheets for
solid LED component
cladding, laser-cut plywood
for the flexible hinge
connectors, pine wooden
dowels, and 3D-printed
pronged fittings for the
hollow structural frame.
The use of expensive-
looking (but in reality,
dirt-cheap) white plastic in
combination with natural
timber tones created an
interesting visual blend
of materials reflecting the
combination of precisely
machined and meticulously
hand-crafted construction
techniques.
Did anything go wrong?
Halfway through the
process, we realized that
using hinged friction
Detail of cell joint and
sound sensor.
joints was not going to be
structurally viable while
also maintaining the
modularity of the structure
at the desired scale. The
way we got around it was
by fixing hinged joints
into position and then
settling on the most stable
configuration of parts: a
truncated dodecahedron.
This is interesting since
some naturally occurring
tessellations (like crystals)
tend to create forms that are
similarly stable.
What was the biggest
surprise in making this?
It was interesting to discover
The interaction
felt natural and
less mechanical
than what we
were initially
expecting.
Assembling the ring module.
Specs
Materials: 3mm
plywood, 3mm twin-wall
polypropylene sheet,
timber dowels, ABS
plastic, 36mm LED pixels,
microphone, Arduino Uno
with Waveshield
Tools: Laser cutter, CNC
cutter, desktop FDM
3D printer, Rhino3D +
Grasshopper, Processing,
nuts and bolts, cable ties
that even making simple
interactive systems could
elicit great interest and
responsethe sculpture
became more and more like
a reflector of the performers
in the space surrounding
it rather than a singular
sculptural statement. Its
reaction to sound often
became obfuscated,
appearing more like a
flickering candle with a
mind and thoughts of its
own. The interaction felt
natural and less mechanical
than what we were initially
expecting.
How would you improve on
this if you were to make it
again?
In terms of form, we would
definitely redesign the
friction hinge connectors;
though their complexity
added to the overall
aesthetic effect, they
potentially could have
been simplified, making
them easier to fabricate.
Detail of hollow-cell and
solid-cell joint.
INTER ACTIONS. ACM.ORG
Interlocking them
mechanically would also
increase overall stability
and potentially fulfill our
initial aspiration of making
the structure modular. We
would also like to expand on
the ideas of the modularity
of responsiveness and
structural modularity
making components
independently addressable
as well as potentially
building awareness and
simple behaviors into
individual cells.

Andrei Smolik,
Curtin University

Robert Cameron,
The University of Western Australia
thedeadpixelproject@outlook.com

http://www.
thedeadpixelproject.
com/subpages/research/
cringeMACHINE.html

DOI: 10.1145/3095803
COPYRIGHT HELD BY AUTHORS

D
 etail of completed sculpture Lighting tests with partially constructed
responding to sound. sculpture.

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 17

 DAY IN
THE LAB

18 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

 W
Information Science
hat is a unique
feature
of your lab?

at CU Boulder The Department

As told by Casey Fiesler, William Aspray, Lecia Barker,
Jed Brubaker, Laura Devendorf, Brian Keegan, Leysia Palen,
Michael Paul, Danielle Szafir, Ricarose Roque, Rick Robinson,
Amy Voida, and Stephen Voida
of Information
Science at the
University
of Colorado Boulder (CU) is brand-
new and small, so we have had the
opportunity to make our space and
processes highly collaborative. As such,
our lab is an entire department! We
have faculty and students working
in areas that include computational
social science, digital humanities,
crisis informatics, digital identity,
personal information management,
philanthropic informatics, data ethics
and policy, network science, machine
learning, information visualization,
and diversity and inclusion in
STEM. One theme that cuts across
some of our research (and serves to
translate research into undergraduate
education) is human-data interaction,
as an expansion of human-computer
interaction. We combine areas of
expertise for the benefit of the
classroom, Ph.D. student advising, and
numerous research collaborations. In
addition, the entirety of the faculty
and Ph.D. student group is working
together on a project to understand and
promote the practice of human-centered
data science across several academic
disciplines at CU. Though we all have
research agendas of our own, having a
collective identity is important to our
mission and our operations. We really
do feel like a research group as much as a
department.

How many people are in the lab, and

what is the mix of backgrounds and
roles? Information Science currently
includes 13 faculty members, 13
additional courtesy faculty members,
professional research staff, a dozen
Ph.D. students, and a number of
other graduate and undergraduate
researchers. We are fortunate to
have close connections to other
departments and institutes on campus,
so our student researchers come
from all overnot only information
science but also computer science, the
Institute of Cognitive Science, the
ATLAS Institute, communication,
and journalism. The faculty also has
diverse backgrounds, with training
and degrees in communication,
computer science, history, human-
INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 19
 DAY IN
THE LAB

Our faculty and

personnel practicing our
mountain selfie skills.

L aura Devendorf
works with
prototypes from
Steve Voidas
research group.

As a department
and research group,
we are interested in
pushing the boundaries
of what information
science can be.

Leysia Palen
works with student
researchers.

20 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

 A research
showcase in the lab
garage.
centered computing, journalism, law,
math, and psychology. Since we value
well-rounded researchers, it is rare for
someone to have a single role in research
projects. Students who build systems
may also be doing empirical work, and
it is common for someone to have a lead
role on one project and a supporting role
on the next.
Briefly describe a day in the life of your
lab. As a new unit that is building its
research portfolio, business processes,
and even curriculum from scratch,
every day is a little different, though
there are patterns to the ways in which
we work and collaborate. We have an
open office plan with a number of huddle
rooms for small meetings, and a large
working and meeting space dubbed
the Garage because of the rolling glass
garage doors that can section off public
space. Our huddle rooms are all named
after Internet cats, and throughout the
day you will come across ad hoc group
meetings. Perhaps Steve Voida and
some students are building prototypes
in NyanCat while Danielle Szafir and
Michael Paul consider visualizations of
social-media health data in Keyboard
DOI: 10.1145/3097266 COPYRIGHT HELD BY AUTHORS
INTER ACTIONS. ACM.ORG
A Ph.D. student showcases the Hololens
he is using in visualization research.
Cat. Meanwhile, Jed Brubaker and
Casey Fiesler are holding their research
meeting in the Garage with students
who are working on social-computing
topics. It is common to find small groups
of Ph.D. students working together on
the expansive whiteboard in the grad
cafe areafinding that spontaneous
conversations often turn into new
collaborations!
What is one feature of your lab that you
could not do without? Our open space
has been helpful in the development of a
new unit where consultations between
faculty, staff, and students can happen
readily as issues are being decided,
often for the first time. Aesthetically,
the long sight lines across the space
to the large bank of windows brings
light throughout, while also allowing
for people to identify places where
they can work alone or together. The
size of our space works well for a new
department, but we are also looking
forward to expanding our footprint as
student and faculty enrollments grow
to provide working spaces for different
forms of information science education
and research.
How would you describe how people
interact in your lab? The environment
is congenial and collaborative. The
personnel and physical environment
mutually reinforce our goals of
establishing a new set of educational
and research programs on our campus.
We strive for a high degree of both
cooperation and individuation that
supports professional development
and creates an organization with
porous boundaries that welcomes many
disciplines to its projects.
What is the one thing you see as most
important about the work you do
there? As a department and research
group, we are interested in pushing
the boundaries of what information
science can be. By bringing many
people together and surfacing as much
as possible how different kinds of
research are done, we strive to conduct
projects that integrate multiple forms
of expertise, methods, and analytic and
design expression.
http://www.colorado.edu/cmci/
academics/information-science
J U LY A U G U S T 2 0 17 INTER ACTIONS 21
 COLUMN CONFESSIONS
Uri Kartoun,
IBM Research
The Man Who Had Them All
A
significant portion
of my time at
Massachusetts
General Hospital
(MGH) was spent
analyzing a cohort of
65,099 individuals
diagnosed with type 2 diabetes mellitus
(T2DM). During these training years
as a research fellow (2013 to 2016),
I, along with my colleagues at MGH
and Harvard, implemented a variety
of predictive modeling methods
as well as incorporated natural
language processing techniques
to better understand diseases and
their complications. We focused on
cardiovascular disease, liver disease,
and insomnia.
This T2DM cohort contained
complete clinical details as well
as demographics of patients who
received care at MGH or Brigham and
Womens Hospital (BWH) between
1992 and 2010. The cohort was large,
considering all clinical narrative notes
(office, medication management, and
operative notes) that accompanied
the traditional electronic medical
records elements (billing codes and
medication prescriptions).
I had the opportunity to present
the methods used to create the cohort
as well as its content in more detail at
the American Medical Informatics
Association 2015 Annual Symposium
[1]. We were interested in analyzing
chronic conditions, including the
15 common conditions reported by
the Centers for Disease Control and
Prevention (CDC) as contributing to
multiple chronic comorbidities [2]. We
implemented the same comorbidity
identification method proposed by the
CDC, counting a certain comorbidity
for a patient if the comorbidity was
documented as a 1 inpatient or 2
22 INTER ACTIONS J U LY A U G U S T 2 0 17
outpatient code. We found that more
than 87 percent (56,691 of the patients)
had two or more chronic conditions,
and, looking at a sicker portion of the
population, more than 42 percent
(27,806 of the patients) had five or more
chronic conditions.
I recall that of those 65,099
individuals, one man had the full
spectrum of the 15 chronic conditions.
He had current or historical diagnoses
of hypertension, asthma, osteoporosis,
congestive heart failure, atrial
fibrillation/atrial flutter, ischemic
heart disease, chronic kidney disease,
cerebrovascular disease, depression,
arthritis, diabetes, hyperlipidemia,
chronic obstructive pulmonary disease,
Alzheimers, and at least one type of
cancer. This patient had all conditions
yet, strikingly, he was alive.
I now conduct research that
focuses primarily on healthcare at
IBM Research, and I often remember
that patient and wonder whether he
has survived. If so, what spectacular
mixture of physical and mental
characteristics made him so sick and, at
the same time, so invincible? Currently,
when I dissect patient profiles of a
medical complexity similar to that
mans, I often ask myself whether those
situations could be reversed.
Many publications describe novel
findings of using nontraditional risk
factors to better identify individuals at
Many hardware-based
technologies that once
existed only in the
realm of science fiction
are gradually
becoming available.
high risk of developing certain diseases.
Recent advances in software, such as
code-sharing tools, freely available R
and Python machine-learning libraries,
rapid communication tools, and new
algorithms have accelerated healthcare
applications (such as decision support
systems in the ER and remote patient
monitoring), as well as research. Such
capabilities were not sufficiently
available to us a decade ago. Prediction
models have gradually become more
accurate because of the software
and hardware advances that have
accumulated over the past few decades.
Combining the ability to store and
rapidly process the records of hundreds
of millions of individuals by using
current or developing new analytic
techniques may bring prediction
accuracy to crystal-ball levels.
Many hardware-based technologies
that once existed only in the realm
of science fiction are gradually
becoming available, including robotic
nurse assistants, artificial retinas,
and light bulbs that kill bacteria at
patients bedsides. An even more exotic
technology is Caltechs experimental
prototype that demonstrates how a
paralyzed man can drink beer with
the help of a mind-reading robot.
Advances in prosthetics, anti-aging
drugs, tooth-regeneration techniques,
sensors that allow us to watch veins
under the skin in real time, gel that
stops bleeding, and cholesterol-
removing machines are only a few
examples of the disruptive technologies
that may reverse chronic conditions.
Most scientists and researchers
believe that big data will transform
medicine [3] despite current substantial
technological barriers [4]. However,
when algorithms are applied to clinical
data, they are deficient in their ability
to make accurate predictions. The
INTER ACTIONS. ACM.ORG
 algorithms rely on information that
occurred in the past, then attempt
to create links between covariates
and outcomes, and reflect the
reasoning about the future at the level
of the individual patient. Had the
computational capabilities available to
us at present existed in the 15th century,
would it have been possible for Leonardo
da Vinci to develop a machine-learning
algorithm to predict the emergence
of the 21st centurys new infectious
diseases, such as severe acute respiratory
syndrome (SARS)? Probably not. Like
humans who rely on their intelligence,
imagination, and observations of the
past to make predictions [5], even the
I M A G E B Y P O N G M O J I / S H U T T E R S T O C K .C O M
most advanced algorithms [6] share a
similar deficiency. Efficient machine-
learning algorithms of the future must
incorporate elements unrestricted to
computer science or statistics into their
functionality, and at this stage such
elements are not yet known to science.
DOI: 10.1145/3096966 COPYRIGHT HELD BY AUTHOR
INTER ACTIONS. ACM.ORG
Hats off to the man who had all
15 chronic conditions, and hats off
to his heroic struggle for survival.
His unbreakable soul was enhanced
by care from clinicians equipped with
not only extensive experience but also
state-of-the-art tools and algorithms.
For now, technology combined with
human care can merely delay the
deterioration of his condition, at
least temporarily. I believe, however,
that some chronic conditions will
disappear in coming years, and we
scientists, engineers, and clinicians
are here to make that happen.
Endnotes
1. Kartoun, U., Kumar, V., Cheng, S.C., Yu,
S., Liao, K., Karlson, E., Ananthakrishnan,
A., Xia, Z., Gainer, V., Cagan, A., Savova,
G., Chen, P., Murphy, S., Churchill, S.,
Kohane, I., Szolovits, P., Cai, T., and Shaw,
S.Y. Demonstrating the advantages of
applying data mining techniques on time-
dependent electronic medical records.
Proc. of American Medical Informatics
Association 2015 Annual Symposium. Nov.
2015, San Francisco, CA.
2. Ward, B.W., Schiller, J.S., and Goodman,
R.A. Multiple chronic conditions among
U.S. adults: A 2012 update. Prev Chronic
Dis 11 (2014), E62.
3. Obermeyer, Z. and Emanuel, E.J.
Predicting the future - big data, machine
learning, and clinical medicine. N Engl J
Med 375, 13 (2016), 12169.
4. Boll, S., Heuten, W., and Meyer, J.
From tracking to personal health. ACM
Interactions 23, 1 (2016), 7275.
5. Kartoun, U. A user, an interface, or none.
ACM Interactions 24, 1 (2017), 2021.
6. Beam, A.L. and Kohane, I.S. Translating
artificial intelligence into clinical care.
JAMA 316, 22 (2016), 23682369.
Uri Kartoun is a research staff member at
IBM Research in Cambridge, MA. Previously
he was a research fellow at Harvard Medical
School/Massachusetts General Hospital. His
Ph.D. from Ben-Gurion University of the Negev,
Israel, focused on human-robot collaboration.
uri.kartoun@ibm.com
J U LY A U G U S T 2 0 17 INTER ACTIONS 23
 C O L U M N T H E D E S I G N E R S S P E A K E A S Y
Uday Gajendar
Principal Designer
Teaching Design Online
H
ow would you create a
design class that must
be entirely online, with
no in-person, in-studio
interaction? Its quite
the design problem in
itself, isnt it?
The past few years have seen a
dramatic rise in the availability of
online-only classes. Distance-based
learning programs have thrived for
quite some time, as supplements or
extensions of some core curriculum.
But now thanks to Khan Academy,
Coursera, Udacity, and others (not to
mention a lifetimes worth of lectures
posted on YouTube or iTunes), there
has been a veritable explosion of
such offerings, shaping a popular
expectationcertainly among
millennial/GenY demographicsto
be able to learn pretty much anything,
totally online, from economics to
programming to human anatomy.
And naturally, offerings centered on
user experience or HCI or interaction
design are popping up too. But
what would it mean to translate
something that is typically done as
an in-person, studio-based model of
learning exchange into a purely online
asynchronous transaction? What are
the considerations and impacts upon
instructor roles, student expectations,
tangible outcomes, and general
pedagogical framework?
Ive recently been involved in
exactly this situation, having signed
up to create an online class on design
thinking 101 as an elective for Lesley
University in Boston. The course is
part of its pioneering undergraduate
degree program on UX, which offers
a fully online, accredited design
program. Ive been serving as the
instructional designer for this class,
so it could be taught by any qualified
24 INTER ACTIONS J U LY A U G U S T 2 0 17
faculty at Lesley, not just me. Its
certainly been an illuminating and
at times quite trying experience that
I am, as of this writing, finishing up,
all to be integrated into the schools
official learning management system
(LMS), Blackboard.
Id like to share my perspectives
on this process, in the hopes of
stimulating useful debate on this topic
of online teachinga great challenge
for HCI professionals!
First, lets acknowledge that there
is something uniquely challenging
when moving a design course online.
Why is that? Design (whether
administered as HCI or fine arts
or D-School) is fundamentally a
practical activity aimed at shaping
informed perspectives and tackling
embodied problem solving via
experimentation, testing out theories
in some materially manifested form
(i.e., concepts with tangible outcomes
for evaluation: a device, an app, a
system model, a video, and so forth).
Indeed, a studio or lab space with
collaborative, project-based inquiry
is what drives pretty much all design
programs of good repute. Direct,
real-time interaction between a
teacher and her students is a vital
means of wrestling with various
What would it mean
to translate something
typically done as
an in-person,
studio-based model
of learning exchange
into a purely
online transaction?
propositions and questions that
become embodied as problems for
exploration (as exercises, projects,
etc.). And that dynamic fosters a
depth of investigation with materials
such as type, color, motion, behavior,
ergonomics, and affordances, from
screens and pixels to circuits and
sensors and beyond, as we advance
into the emerging areas of robotics,
wearables, autonomous vehicles, and
so on.
The numerous benefits of a live
studio context with the combined
presence of students and instructor(s)
together include:
Immediacy of conversations
undisturbed by filters/technologies
Development of trust and rapport
via question/answer dialogues
The challenging of assumptions
with right here, right now
stimulation of ideas and questions
from other students (or re-addressing/
re-explaining in other ways, with
physical gestures and demonstrations)
The generation of eureka
moments of insight through students
witnessing (via whiteboards or screens
or being in the shop, etc.) how such
ideas manifest in making something,
or simply by pointing out to students
examples that illustrate abstract
points (literally, show and tell).
All in all, the studio context
enables a rather ultra-high-definition
resolutionway better than 4K
HDTV!of educational experiences for
developing design aptitude and skills.
And lets not forget the raw
physicality in the exploration of
ideas! Making a quick prototype
right there to validate a claim and
then literally picking it up, playing
with it, debating its qualitiesthis
all yields a memorable, grounded
experience of learning, echoing John
INTER ACTIONS. ACM.ORG
 Deweys doing and undergoing in
forging comprehension as a testament
to the vitality of a truly educational
interaction. It is an experience
enabled by the active environment
of a studio, supporting the journey
to inquire, interpret, understand,
manifest, and learn.
But what about a class that exists
solely via the interfaces of an online
learning system; how does design
learning even happen? Its all quite
different! There is only a Web
browser to access predefined course
materials housed within a perhaps
IMAGE BY TOBIA S TOF T / FLICKR
unnecessarily complex LMS software
suite, whose difficulty may rival
that of the enterprise business apps
that all teachers and students must
use daily. Plus, there are layers of
interpretation occurring at a distance
INTER ACTIONS. ACM.ORG
and asynchronously: outlines, folders,
files, annotations, updates. Are
students getting the updates and truly
checking out the docs?? Who knows!
But let me back up a bit. As part
of my deal with Lesley University,
I was required to participate in a
four-week seminar on instructional
designyes, an online class on how
to design an online class, so quaintly
meta! This class provided tools
In this era of glance,
scan, and react,
expect that literacy
model to be applied to
your course materials.
and frameworks to help me shape
a syllabus with identified learning
activities and specific outcomes per
a rigorous structure (i.e., Blooms
Taxonomy as the main reference).
But more important, this class
offered a chance to feel what its like
to take a class purely online for an
extended timet hats right, gain
some empathyparticularly as a busy
professional with a hectic schedule,
compelled to work on group projects
with other classmates who are in
different time zones.
Some big questions and issues
surfaced immediately:
How do you persist and balance
multiple modes of presence with
an online class, with distributed
asynchronously interacting
students and instructors? Theres
J U LY A U G U S T 2 0 17 INTER ACTIONS 25
 C O L U M N T H E D E S I G N E R S S P E A K E A S Y
the teaching presence (offering
guidance and feedback), a social
presence (classmates collaborating
and dialoguing), and the cognitive
presence (the students intellectual
engagement). All of this happens
quite f luidly IRL (in real life) but
must be actively sought or managed
when online.
Related, how does a class form
collegial bonds of interaction that may
naturally and easily develop in a
regular semester studio class? For
online-only, such class dialogues
are conducted asynchronously via
interface constructs like discussion
forums, comments on journal entries,
and time-shifted video posts. It
definitely raises questions around
developing social bonds and collegial
rapport, as well as the accuracy of
interpretation of textual exchanges,
emotive reactions, and so on. Identity
and authenticity (not to mention the
timeliness) of communications, as
with any social online medium, form
another hump to get over in enabling
productive class dynamics.
How do you evolve your approach
from an adaptive model of dynamic feed-
back with students to a more static, pre-
scriptive model of detailed, predefined
lesson plans? Indeed, a big part of the
value in teaching is the guidance from
students to see if they get it and how
much iteration on your part is needed
to evolve your methods to foster their
learning. For an online class, every-
thingand I do mean EVERY read-
ing, exercise, and assignmentmust
be verbosely detailed with generous
instructions, with nothing taken for
granted. Remember, you are not there
to demonstrate how to perform an ac-
tivity, as you would in a real class, and
answer inevitable questions or clarifica-
tions. Clean-slate thinking is required;
you must truly assume nothing!
What is your role as a teacher in
a completely online situation? In a
studio context, you most likely are the
central authority figure, a demanding
coach, and an empathetic guide
to help students along the path of
understanding, which depends upon
those modes of presence described
earlier. However, for online teaching,
your presence is more fragmented
DOI: 10.1145/3101104 COPYRIGHT HELD BY AUTHOR
26 INTER ACTIONS J U LY A U G U S T 2 0 17
and theres more emphasis on
peer learning via online chats and
discussion forums, with time-
delayed, back-and-forth feedback
on posted assignments. So youre
more like a facilitator leveraging the
LMS tools at your disposal to guide
communications along, which is also
amplified by office hours and email.
Its a bit more of a quiet and subtle
role, since students arent showing
up to a place and seeing you in person
gesturing with your hands and
expressing your professorial guidance
with your face.
Oh, and speaking of time, get
ready to experience severe time
dilation of class exercises, due
to the asynchronous nature of the
class. What I mean is, something
that takes just 10 to 15 minutes of
real-time, extemporaneous demo
and explanation (e.g., showing the
class how to do paper prototyping
or wireframing) could take WEEKS
due to allowance for reasonable time
periods for studying background
materials, digesting detailed
instructions, setting up milestones for
check-ins, reflecting back in a virtual
chat room, and responding to those
reflections. Not efficient at all, but it
is a desperately crucial expectation to
contend with for online classes, as a
general pattern.
And, lets face it, your students
are interacting, as their primary
conduit, with the interface of the
LMS system to access readings, write
notes, post assignments, and engage
with the instructors. The structure
and personality of that LMS are
vital to enabling a smooth, efficient
experience, to truly enable not just
learning but also the experience of
being a student (or a teacher). Not
to be forgottencontent strategy is
crucial! As a course designer, part of
your job is to make your lessons easy
to scan, digest, and act upon. In this
era of glance, scan, and react (thanks
Facebook and Twitter!), expect that
literacy model to be applied to your
course materials, for better or worse.
Clearly theres an array of issues
that make it quite challenging to
enable any kind of design learning
online, compared with whats
possible in a studio contextthe
disembodiment; asynchronous,
fragmented attention; distributed
tools; and interference from poorly
designed online tools that get in the
way. And do we really want education
simply distilled to an app thats highly
transactional and prescriptive? Would
John Dewey approve? I think not.
So what does all this mean for the
future of teaching design online?
And how do we preserve the stellar
qualities of a studio or lab model of
design learning while respecting the
constraints of the online medium? Is
it truly possible?
I am optimistic as technologies
improve (high-resolution video,
virtual reality, robust discussion/
threading systems, truly fluid UX
across channels and devices); and we
in the field of HCI must continue to
develop novel approaches to express
virtual presence and interactivity as
new kinds of digital literacy emerge.
Great potential remains as technology
accelerates to simulate truly high-
resolution communications, with
greater fidelity of nuance and
engagement so that the virtual feels
truly live and physicaleven visceral,
not simply a pale simulation. Imagine
an LMS system that fully takes
advantage of VR with AI services,
personalized chatbots, and highly
responsive conversational interfaces
as mechanisms to compensate for
todays deficiencies, making the
educational experience a dynamic
online marvel that truly activates
learning. And yet we must be realistic:
There are critical constraints as new
forms of educational development
emerge. Lets remember that in the
end, regardless of the medium or
interface, to teach is to illuminate,
guide, enable, and inspire others to
take a path of discovery and mastery,
whose fruits bear out in reality,
beyond any classroom or screen.
Uday Gajendar (www.ghostinthepixel.com) is
a design leader focused on next-gen innovation
and guiding startups on UX fundamentals. He
has more than 12 years of versatile experience
at Citrix, Peel, CloudPhysics, Netflix, Adobe,
and others. He also routinely speaks worldwide
on design topics.
udanium@gmail.com
INTER ACTIONS. ACM.ORG
The
The SIGs,
Handbookactiveofchapters, individual members,
Multimodal-Multisensor notable
Interfaces, leaders,
Volume social and
1 provides the political issues,
first authoritative
international issues, computing and community education...all are topics found within this first
The
The SIGs,
resource on whatactive
Handbook ofchapters,
has become individual members,
the dominant
Multimodal-Multisensor notable
paradigm
Interfaces, leaders,
for
Volume new social and
computer
1 provides the political issues, input
interfacesuser
firstFeaturing
authoritative
book-length history of the Association for Computing Machinery (ACM).
international issues, computing and community education...all are topics found within this insightful
first
profiles
involvingof people
resource on
new media
book-length who
what shaped
has become
(speech,
history ACM,
the such
dominant
multi-touch,
of the Association as Edmund
paradigm
forgestures,
Computing Berkeley,
for new
writing)
Machinery George
computer
embedded Forsythe,
interfacesuser
(ACM).inFeaturing Jean Sammet,
input
multimodal-multisensor
insightful
Peter profiles
Denning,
involvingofnewand
people Kelly
mediawho Gotlieb, and honest
shapedmulti-touch,
(speech, ACM, such as assessments
Edmund
gestures, of
Berkeley,
writing) controversial
Georgein
embedded Forsythe,episodes, this volume
Jean Sammet,
multimodal-multisensor
interfaces. These interfaces support smart phones, wearables, in-vehicle and robotic applications,
deals Peter
with Denning,
compelling andandKellycomplex
Gotlieb, issues involving
and honest ACMofand
assessments computing.
controversial episodes, this volume
interfaces. These interfaces support smart phones, wearables, in-vehicle and robotic applications,
deals other
and many with compelling
areas thatand arecomplex issuescompetitive
now highly involving ACM and computing.
commercially. This edited collection is
This isand
notmany
a narrow
other areas that are now highly competitive commercially. This about
organizational history. While much information the SIGs is
edited collection and
written by international
This
committees is not a narrow
are presented, experts and pioneers
organizational
this book is about in
history. thethe
While
how field.
muchACM It provides
information athe
textbook,
about
definedtextbook, the reference,
SIGs
discipline, and
broadenedand
written by international
committees are presented, experts and pioneers
this book is about in
how thethe
field.
ACM It provides
defined athe discipline,reference,
broadenedand
the profession,
technology and how
roadmap forhowit has expanded
professionals research
working frontiers.
in this and related It is areas.
a permanent contribution to
the profession,
technology and
roadmap for it has expanded
professionals research
working in frontiers.
this and It is areas.
related a permanent contribution to
documenting the history of ACM and understanding its central role in the history of computing.
documenting the history of ACM and understanding its central role in the history of computing.
 C O V E R S T O RY
Intelligence
on Tap:
Artificial Intelligence
as a New Design Material
Lars Erik Holmquist, Northumbria University
Insights
C O V E R C O M P O S I T I O N B Y A N D R I J B O R Y S A S S O C I AT E S; P H O T O B Y V I K T O R G L A D K O V
Through a combination
of factors, AI has recently
made significant progress
and is now integrated in
many successful products.
In the future, AI will become
available as a resource
to use by non-experts
intelligence on tap.
Interaction designers need
to consider AI as a new
design material, with its
own unique opportunities
and limitations.
INTER ACTIONS. ACM.ORG
There has been a revolution, but it
snuck up on us so gradually that youd
be forgiven if you missed it. Its called
artificial intelligence, and it will have
a profound impact on how we design
digital products in the near future.
This has been something of an
unexpected comeback. In the very early
days of computing, many expected
that machines would soon be able to
complement or even surpass humans
in tasks requiring intelligence. But
while well-defined undertakings, such
as playing chess, have proven to be
solvable by using strict rules, more
fuzzy problems, such as recognizing
a cat in a photo, have turned out to be
much more elusive. And so for decades,
the idea of artificial intelligence has
been considered mostly an unkept
promise. While applications of machine
learning have been increasingly
useful when it comes to processing
big-data collections at major Internet
companies, the consensus has been that
for most practical applications, human
intelligence simply cannot be replaced.
But recently, artificial intelligence,
or AI for short, has actually begun to
deliver. New or revitalized techniques
have started to equal or even surpass
humans in tasks previously thought
out of reach, from speech recognition
to playing complex games. The rate
J U LY A U G U S T 2 0 17 INTER ACTIONS 29
cover story
of this AI resurgence has taken aback
even leaders of the industries being
affected the most. Google co-founder
Sergey Brin said in a recent interview
that he has been surprised by the recent
surge in practical applications for
artificial intelligence [1]. Approaches
such as neural networks and deep
learning, coupled with access to massive
amounts of data and new computational
hardware, have led to significantly
better results than traditional methods
in areas such as image recognition,
machine translation, and speech
synthesis.
In one of the more spectacular
examples, Googles Deep Mind software
was able to beat a grandmaster of the
ancient Chinese game Go in March
2016 [2]. Computers had already proven
they could beat world masters in chess,
but Go was thought to be out of reach
because it contains exponentially
more possible move combinations
far beyond what can be stored in any
computer. In order to achieve this
feat, rather than working from a list of
possible moves, the software instead
taught itself to play the game. First, it
got a solid foundation by training on
millions of existing human Go games.
But that was not enough. To improve its
game, the network then played many
more matches against itself. In this way,
it ended up with a vocabulary of moves
that consisted of both human and
self-taught strategies. The result was a
game-playing software that played from
its own experience, not from any strict
set of rules.
If playing an old Chinese game
sounds too esoteric, consider this: A
neural network derived from the same
basic techniques was trained to control
the cooling processes in Googles data
centers, in a way similar to how it
learned Go. This time it was no game;
it had dramatic financial consequences.
The company claims that through
the smarter control provided by this
software, it has been able to save several
hundred million dollars in electricity per
yearthus by itself paying for Googles
The right skill for drawing realistic zombies
on a teenagers video-game screen
also turns out to be exactly what is needed
for running a neural network!
30 INTER ACTIONS J U LY A U G U S T 2 0 17
acquisition of the AI startup company
whose research laid the foundation for
the system. [3]
And that is just the beginning.
In the past year, the collection of AI
techniques called deep learning have
contributed to significant advances in a
whole range of areas, including speech
synthesis, speech recognition, machine
translation, image recognition, and
image compression [4]. And although
the results are still largely coming in
areas dominated by big data and big
Internet companies, it is clear that
AI will soon have implications for a
whole range of new products. It will
eventually make it possible to inject a
little bit of intelligence into even the
most mundane product, whether a
toaster or a car. By extension, this will
fundamentally affect HCI research and
the practice of interaction design.
But before we go on, lets try to
unpack the recent developments that
have surprised even people like Googles
co-founder.
AI COMES OF AGE
Algorithms inspired by how the
brain works, so-called artificial neural
networks, date back to the 1960s;
most computer science students still
encounter them in introductory AI
classes. These networks are formed
by connections of artificial neurons,
which are basically just weighted
links between nodes in a graph. The
actual network itself does not have any
inherent meaning or knowledge. But
by subjecting the network to stimuli
and reinforcing the links that are used
when it makes the correct choices, it is
possible to train the network to make
choices. For instance, by subjecting
a network to a sequence of pictures
with simple geometrical shapes but
reinforcing it only when the network
selects those that contain a circle, it
would be possible to teach it to pick only
images that depict a circle.
The main technology leading
the current AI resurgence is neural
networks. However, for a long time it
looked like neural networks would be
limited to simple problems with little
practical use. This is because, first of
all, to do anything beyond the most
trivial tasks, the number of nodes and
connections in such a network would
have to be very large. This means that
it would take a long time to train it,
and even when it was fully trained,
the time it would take to reply to a
query would be too long for any time-
critical applications such as automatic
translation. Second, in order to learn
anything meaningful, the network would
also need huge amounts of training data.
Such data would need to be in machine-
readable form. The data would also have
to be coded, meaning it would already
contain the answer to the question the
network was being designed to answer.
For instance, for our fledgling network
to learn to recognize circles in pictures,
we would have to subject it to a large
number of pictures that contained circles
and were correctly labeled as such, as
well as pictures that did not contain
circles, so that it would eventually learn
the difference.
But recently, these barriers have
all but disappeared. When it comes to
size and speed, Moores law has been
helpful, but not sufficient, in reducing
the cost of storage and processing time.
Instead, a much bigger breakthrough
came from an unexpected source:
computer games. In 2012, researchers
at the University of Toronto showed
that the specialized chips that are
used to generate fast high-resolution
graphics in PCs, so-called graphics
processing units or GPUs, just
happen to be perfectly provisioned for
processing neural networks [5]. This is
because they are designed to process
massively parallel tasks at a very high
speed. In other words, the right skill
for drawing realistic zombies on a
teenagers video-game screen also turns
out to be exactly what is needed for
running a neural network! Thus, almost
by accident, neural network researchers
were handed fast and inexpensive
hardware on which to run their
experiments, something that is now
revolutionizing the entire chip industry
[5]. This in turn allowed for new and
more effective techniques such as deep
neural networks (the layering of several
levels of networks) and unsupervised
learning (which does away with explicit
labels and presents the network with
only rough clusters of data). Together,
INTER ACTIONS. ACM.ORG
these advances contributed to results
like the Go game victory.
And when it comes to data itself,
theres a veritable mother lode.
Facebook, Google, Amazon, and the
other Internet giants have already
been patiently Hoovering up every
scrap of input generated by their users
for decades. They now have access to
billions upon billions of photos, emails,
videos, and chat messages, not to
mention mouse clicks and finger taps on
everything from inspirational articles
about yoga to diaper advertisements.
This manic data collection is also
reaching its tendrils out into the real
world, for instance through mobile
phones, taking in things like the users
geographical location (through GPS) or
their physical activity (through motion
sensors). And if you hadnt noticed,
neural networks are already listening
to what you are saying! Companies like
Apple and Microsoft are storing every
command given to their respective voice
assistants for future use, in order to
better train their recognition software.
In this case, Siri, Cortana, and of course
Amazons Alexa and their ilk, are
serving not just as helpful assistants but
also as Trojan horses to gather unheard
amounts of voice utterances and
associated behaviors to feed the neural
networks of the future. As if this wasnt
enough, emerging technologies such as
drones and self-driving cars will soon
add ever bigger piles to this data stash.
Of course, this data gold rush has
consequences that can be troubling.
Most obviously, consider the fact that
all this data is in the hand of private
companies. They now have literally
unlimited access to everything
generated by our private and public
digital lives but are not governed by any
of the rules for transparency or privacy
that pertain to public organizations.
This leads to another, less obvious,
consequence, which is that many of the
best minds in the field will no longer be
found at universities, where they can
freely share their knowledge. Instead,
they are being aggressively recruited by
well-funded companies, where they not
only get better salaries (and free food to
boot) but, more important, much more
challenging problems to work on. This
is because the big data that is necessary
to provide truly groundbreaking
research resides at these companies,
where it is also increasingly well
protected, since it constitutes the very
INTER ACTIONS. ACM.ORG
essence of the companies value on the
stock market. While once upon a time
Flickr set its user agreement to the
altruistic Creative Commons license by
default, meaning that images could be
freely used for noncommercial purposes
and released as large training sets
for the benefit of science [6], current
services guard their content much
more jealously. For instance, Instagram
pictures, while free to browse, are
bound by agreements that prohibit any
application of computer vision, making
them in effect inaccessible for any
machine-learning approaches.
On the other hand, there are
encouraging signs that the tools of this
new and efficient AI will become more
accessible, often when universities and
industry work in concert. Open source
software such as Tensorflow is already
letting users adapt and train neural
networks for new purposes [7]. These
services are still far from plug-and-play;
they require extensive handholding
from experts to achieve any useful
results. But they point to a future
where neural networks are packaged
in such a way that non-experts can use
them through well-defined interface
mechanisms. Most likely, due to size
and speed limitations, this will happen
not on individual devices but on remote
servers. Thus, just like other data- and
processing-intensive tasks such as
cloud storage and Web hosting, AI will
transform into a service.
And with commercially available AI
services bound to arise, it will gradually
become easier to obtain and train an
artificial intelligence to do your bidding.
This means that in the near future,
designers will no longer have to be
experts in neural networking to use AI,
just as they do not need to know the ins
and outs of TCP/IP or even HTML to
design Web pages. The same services
will be available when designing
physical artifacts, too, to complement
other elements such as sensors and
actuators. When this happens, AI
will be thought of not as an exotic and
complicated technology that can be used
only by gurus with Ph.Ds in machine
learning, but rather as a resource you
can plug into any new product when you
need it. Think of it as intelligence on tap.
A NEW DESIGN MATERIAL
So what exactly does this intelligence on
tap mean for interaction design? First
and foremost, it means that intelligence
is becoming a new design material. As we
know, the options of a designer are to
a large extent defined by the materials
they have to work with. For instance, a
graphic designer working in the medium
of print must be familiar with paper sizes
and coating types, as well as color blends,
printing presses, and other means of
achieving their desired results. A product
designer would need to be aware of the
physical characteristics of materials such
as plastic, wood, and metal, as well as
how these fit together mechanically, in
order to design an aesthetically as well as
functionally pleasing experience. As AI
becomes a more and more vital part of
everyday products, designers will have to
figure out how to work with intelligence
as a new material, with its own specific
quirks and opportunities. This will not
be easy, as intelligence on tap could mean
a radical departure from previous design
practices, as when going from paper to
screen in the early days of the Web.
For anyone developing products
that contain AI (including but not in
any way limited to designers), it will be
necessary to form a clear understanding
of what AI can and cannot do. Again,
this does not mean that everyone has
to become a neural networking guru,
but it is necessary to understand the
underpinning principles of AI. In
particular, this means that if someone
tries to design a product without a firm
understanding of the limitations of AI,
the result will most certainly be failure.
Here, the most important limitation
to consider is the fact that AI still
cannot form an actual understanding
of the world. While neural networks
can indeed work better than humans
on problems that involve large amounts
of data, and can seemingly reply in
intelligent ways to many queries, they
still cannot understand a basic sentence
in natural language. This has particular
relevance to some of the most hyped AI
applications, such as natural-language
dialogue systems, aka chatbots. As
overly enthusiastic product designers
have already discovered, it is currently
far beyond the reach of any neural
network to carry out an intelligent
conversation. For instance, Facebooks
recent experiments in chatbots ended in
something of a fiasco after it turned out
it could correctly fulfill only about 30
percent of the requests [8].
There is an important lesson to be
learned there. Replacing human-to-
human interaction in realistic situations
J U LY A U G U S T 2 0 17 INTER ACTIONS 31
cover story
is exactly something that AI cannot
do yet. This is the kind of problem
that requires a real understanding of
the world and the intentions of the
conversation partnersomething that
todays neural networks are simply
incapable of. Furthermore, it is well
known from research that dialogue
systems are more efficient when users
do not expect the bot to have full,
human-like intelligence [9]. Thus, by
trying to apply human standards to an
automatic system, the constructors of
the Facebook chatbot literally set it up
for failure and made users even more
disappointed and frustrated.
Instead, artificially intelligent
systems should concentrate on things
that humans cannot do but that AI
can do well. In large part, this involves
sifting through immense amounts of
data and finding patterns. One area
where AI is making great progress is
image search, in which large amounts
of data and new neural-network
techniques have produced remarkable
results, such as actually being able to
find pictures that contain cats. Other
areas where AI does well, as long as
there is enough data, is matching one
dataset against another, for instance
in machine translation. It can also be
used to extrapolate from existing data
and make decisions based on that, as
with Googles server-cooling system.
But this also means that AI systems
are highly dependent on the data they
have access to. If the data is lacking in
quality or quantity, this will greatly
increase the risk of the system making
poor decisions.
Thus, anyone constructing an AI-
based system needs to tread lightly,
manage expectations, and be careful
not to overreach when it comes to
AIs capabilities. But apart from
understanding the overall potential
of AI, I believe there are a number of
interdependent challenges that pertain
more specifically to interaction design.
These have to do with how designers
can take the behavior of systems that
rely on artificial intelligence and make
Anyone constructing an AI-based
system needs to tread lightly,
manage expectations, and be careful
not to overreach.
32 INTER ACTIONS J U LY A U G U S T 2 0 17
it understandable for the end user.
They include:
Designing for transparency
Designing for opacity
Designing for unpredictability
Designing for learning
Designing for evolution
Designing for shared control.
The first challenge means that it is
necessary to let the user understand
how artificial intelligence is actually
affecting the interaction. It must
be clear to the user that a system is
actually making its own decisions
based on incoming information, rather
than working from a fixed set of rules.
This might require the rethinking
of fundamental UI components. For
instance, there are interaction cases
when users might want to override
the intelligence, and others when
they might want to cede control. For
a device, this could mean that rather
than just an on/off button, a device
might need an it depends button
that lets the device decide whether
to turn on or off. Similarly, there will
also be a need for interface elements
that communicate when a system has
made a decision, what that decision
was based on, and even a mechanism
to revert or undo the decision if the
user does not agree with it. There could
also be a need to communicate more
complex concepts and plans to an AI,
which might require more flexible
interfaces such as natural language. In
summary, designing for AI might entail
a lot more fuzzy, open-ended user
interfaces than we are used to.
The second, somewhat contradictory,
challenge has to do with the fact that it
is no longer possible to explain exactly
why or how an AI does what it does
they are opaque. The way that neural
networks are constructed means that
their inner workings are hidden even
from the person who programmed and
trained them. For example, Googles
engineers recently made the discovery
that a neural network trained for
machine translation had created its own
intermediary format [10]. This made
it possible for it to translate between
language pairs on which it had not
been trained; for instance, if it had
done Japanese to English, and English
to Korean, it could also in principle
translate between Japanese and Korean.
The point here is that this capability was
not designed into the system, but rather
evolved by itself. How can designers
communicate to the user that there
are things inside the product whose
workings nobody can quite explain?
And how does this affect qualities like
trust and confidence in the system?
This leads to the third challenge:
unpredictability. No matter how
well trained a neural network is, it is
still to some extent drawing its own
conclusions from given data. This is not
necessarily a bad thing. For instance,
the Go-playing network we mentioned
in the beginning had honed its game
not just on humans but also in matches
against itself, where it devised its own
strategies. This led it to make some
surprising moves that no human player
would make. While some of the choices
it made were inexplicable, they were
also part of a winning strategy, and
despite deviating from the human
playbook, in the end the system was able
to beat the human opponent. Designers
thus must be prepared for and design for
systems that behave in unanticipated
ways, which can be jarring even when
it leads to them solving the problem
better than a human would. How
can interaction design minimize the
damage and maximize the benefits that
arise from this unpredictability?
The fourth challenge has to do with
improving the AI through constant
learning. Ideally, a neural network
should never stop learning; it should
use all available new input to improve
its basic algorithms and make the
system even better. However, this
cannot be a chore for the user. If
the user has to explicitly train the
system, that will most likely become
a hindrance to efficient use. There are
already clever ways of having humans
solve problems to aid AI learning,
such as the captchas that separate
humans from bots on the Internet
by having them do simple image-
recognition tasks. Another example
is recommender systems on sites
such as Netflix that encourage users
to rate the content they have viewed,
thereby improving recommendations.
But ultimately, the learning has to be
INTER ACTIONS. ACM.ORG
built into the interaction itself and
completely unobtrusive, so it does not
feel like the user is doubling as the AIs
training wheels.
The fifth challenge has to do with
how these systems will continue to
evolve over time. As AI products solve
problems in collaboration with their
users, they should keep improving.
But this could be jarring if the systems
behavior starts to get better than it
was originally. In fact, we often build
behaviors around flaws like squeaky
doors or loose tiles in a staircase.
If these flaws suddenly disappear
without warning, it might be even
more disorienting than when they
first appeared. Say you have bought
an intelligent coffee brewer that is
supposed to prepare coffee at the right
time and temperature to help you
get up in the morning. You set it for a
certain time, but you have a hard time
getting up, so the coffee is always a
little cold. And thats OK; you need
your sleep. But imagine then that the
brewer observes how you are always
late getting up in the morning, and one
day it proactively decides to delay the
brewing of your coffee by 10 minutes
to better fit your schedule. The result
is that you scald your mouthand
probably throw the coffee maker
out the window! As systems evolve
and make new decisions, it will be
necessary to communicate this to
the user so that they know what to
expect, and can benefit while avoiding
unpleasant surprises.
The final challenge is one that
springs from all the others. It involves
how artificially intelligent systems
can be designed to allow the sharing
of control with the user. This will not
be an either/or situation, where one or
the other has full control. In systems
built on proactive intelligence, there
will have to be provisions for a truly
mutual responsibility. The interface
must give the user access to clear
controls, as well as indications as to
how the power is distributed in any
given moment. This includes how
much autonomy a system receives to
make its own decisions and how much
it is under the control of the user. It
also includes how much it is allowed to
evolve new functionality, how it collects
and evaluates data, how it is to handle
unexpected situations, and so on. Again,
DOI: 10.1145/3085571 2017 ACM 1072-5520/17/07 $15.00
INTER ACTIONS. ACM.ORG
some of this may be too complex to be
fully negotiated by a visual or tangible
interface, which may lead to the need for
speech or other more nuanced modes
of communication. But designing the
interaction of an AI system so that it can
work truly in concert with the user will
be one of the key measures of success.
There will be many other challenges
as wellwhat Ive discussed here
has just scratched the surface. We
did not even get into ethics, which
will have a huge impact. Who is
responsible if an AI system causes
damage or even the loss of life? This
could happen if the system made an
error or was inaccurately controlled
by the user, perhaps due to some flaw
in the interface design. This is not a
science fiction question; it is already
pressingly important for companies
developing self-driving vehicles.
And who gets sued for libel if an AI
runs amok because it is absorbing
data without questioning it, like the
Microsoft chatbot that became racist
by reading Twitter comments [11]?
Another issue is who owns and takes
responsibility for material that an AI
produces? Ownership was much easier
before autonomous systems, because
the creation of content was the result
of a conscious creative act. Now if an
autonomous security robot, or perhaps
an outdoor drone, manages to take
compromising photographs, who gets
to control the resultsthe subject, the
owner of the device, or (most likely)
the company that stores the images on
its servers?
Full-fledged intelligence on tap
might take a long time to arrive, but I
have no doubt that it will. And while
enthusiasm for AI in its many forms
is very high right now (Gartners hype
cycle for 2016 has machine learning
at the very top [12]) and is sure to hit
many snags along the way, there is no
doubt that the technology is going to
fundamentally change interaction
design. The sooner designers start to
think about intelligence as a design
material, the better prepared they will
be for the coming shift in how digital
systems will work, and in particular
how AI can function in concert with
their users. Hopefully, this article
has provided some first steps toward
understanding the future of AI as a new
design material.
Endnotes
1. Kharpal, A. Google co-founder Sergey
Brin says hes surprised by pace of
A.I. and uses a story of a cat to explain
it. CNBC.com. Jan. 19, 2017; http://
www.cnbc.com/2017/01/19/google-co-
founder-sergey-brin-said-he-is-surprised-
by-pace-of-ai.html
2. Metz, C. Googles AI wins fifth and
final game against Go genius Lee Sedol.
Wired. Mar. 3, 2016; https://www.wired.
com/2016/03/googles-ai-wins-fifth-final-
game-go-genius-lee-sedol/
3. Clark, J. Google cuts its giant electricity
bill with deep mind-powered AI.
Bloomberg Technology. Jul. 19, 2016;
https://www.bloomberg.com/news/
articles/2016-07-19/google-cuts-its-giant-
electricity-bill-with-deepmind-powered-ai
4. Metz, C. 2016: The year that deep
learning took over the Internet. Wired.
Dec. 26, 2016; https://www.wired.
com/2016/12/2016-year-deep-learning-
took-internet/
5. Greengard, S. GPUs reshape computing.
Communications of the ACM 59, 9 (2016),
1416.
6. Thomee, B., Shamma, D.A., Friedland,
G., Elizalde, B., Ni, K., Poland, D., Borth,
D., and Li, L. YFCC100M: The new data
in multimedia research. Communications
of the ACM 59, 2 (2016), 6473.
7. Abadi, M. et al. Tensorflow: Large-scale
machine learning on heterogeneous
distributed systems. arXiv preprint
arXiv:1603.04467 (2016).
8. Facebook scales back AI flagship after
chatbots hit 70% f-AI-lure rate. The
Register. Mar. 22, 2017; https://www.
theregister.co.uk/2017/02/22/facebook_
ai_fail/
9. Cassell, J. Embodied Conversational
Agents. MIT Press, 2000.
10. Wong, S. Google Translate AI invents
its own language to translate with. New
Scientist. Nov. 30, 2016; https://www.
newscientist.com/article/2114748-
google-translate-ai-invents-its-own-
language-to-translate-with/
11. Vincent, J. Twitter taught Microsofts AI
chatbot to be a racist asshole in less than a
day. The Verge. Mar. 24, 2016; http://www.
theverge.com/2016/3/24/11297050/tay-
microsoft-chatbot-racist
12. Gartner 2016 Hype Cycle. Aug. 16, 2016;
http://www.gartner.com/newsroom/
id/3412017
Lars Erik Holmquist is professor of
innovation at Northumbria University, U.K.
Previously, he did research in interaction
design and ubiquitous computing in Sweden,
Silicon Valley, and Japan. His first book,
Grounded Innovation: Strategies for Creating
Digital Products, was published in 2012. He just
finished his second, a science fiction novel set
in Silicon Valley.
lars.holmquist@northumbria.ac.uk
J U LY A U G U S T 2 0 17 INTER ACTIONS 33
 Vassilis Kostakos, University of Melbourne
Mirco Musolesi, University College London and the Alan Turing Institute

Avoiding Pitfalls
When Using
Machine Learning
in HCI Studies

Insights
We highlight some of
the pitfalls that HCI
researchers should avoid while
using ML techniques.
Prediction accuracy cannot
be used as a substitute for
classic hypothesis testing and
correlation/causation analysis.
In addition to accuracy,
researchers should also
report baseline performance.
M
Machine learning (ML) has come of age
and has revolutionized several fields
in computing and beyond, including
human-computer interaction (HCI).
Human-subject studies have been
adopting ML techniques for more
than a decade, for example for activity
recognition and wearable computing.
There now also exists a plethora of
application domains in which ML
approaches are enriching interactive
computing research. Here we wish to
highlight some of the pitfalls that HCI
researchers should avoid while using
ML techniques in their research.
A popular use of ML techniques in
HCI is to model human behavior. One
potential risk is that ML techniques
are sometimes used inappropriately
to draw conclusions, possibly strong,
about human behavior, instead of using
more classical statistical methods. It
is worth noting here that some ML
techniques are actually based on
classical statistical methods such as
regression or curve fitting. However,
some classification methods, such as
neural-networks-based approaches, are
much more difficult to interpret given
the complexity and dimensionality of
the underlying mathematical models
inferred from the data.
Another popular use of ML in
HCI is to develop novel user-interface

34 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

 techniques, such as to react to user
input (e.g., gesture recognition),
optimize system resources (e.g.,
smartphone battery conservation
[1]), or provide intelligent mobile
notifications [2]. The prediction of
future users activities and interactions
is another emerging area of interest:
The aim is to develop full-fledged
IMAGE BY PAN J J / SHUTTERSTOCK .COM
anticipatory computing systems
[3]. Indeed, a rigorous performance
evaluation of these systems is
fundamental in order to evaluate their
effectiveness and efficiency.
Specifically, the definition of the
training set needs to be considered
in detail when ML techniques are
INTER ACTIONS. ACM.ORG
used in HCI. Interactive systems are
usually evaluated with a training set
obtained from a certain population of
users. When evaluating the system,
authors should report both: results
using training data only from the
same individual (personalized model)
and results using data from the entire
population (generic model). This is
necessary for systems where no data
exists for first-time users and, therefore,
classifiers have to be bootstrapped
with data from other users. It might
also be helpful to show variations in the
performance for the entire population
in order to understand if, for example,
there are classes of users that are easier
to model and predict. Sometimes the
application of clustering techniques
might be necessary to identify users who
share the same characteristics.
C
ML IS NO SILVER BULLET
FOR HCI RESEARCH
Classification accuracy is not hypothesis
testing. It is important to underscore
that ML prediction accuracy cannot
be used as a substitute for classical
hypothesis testing and correlation/
causation analysis, especially
when deriving conclusions about
characteristics of human behavior. Let
us consider, for example, an application
for classifying the mood level of
J U LY A U G U S T 2 0 17 INTER ACTIONS 35
a person from certain behavioral
characteristics. In analyzing their
results, researchers have to be very
careful in interpreting how these
behavioral characteristics are linked to
the actual emotional states of users.
Some ML methods provide
insights about the interpretation of
the phenomena under observation.
For example, in the case of
descriptive methods (such as the
classic association-rule algorithm
[4]), it is possible to derive potential
interpretations of the observed data.
However, this is not the case for other
state-of-the-art algorithms, such as
deep-learning techniques [5]. Although
the interpretation of deep-learning-
algorithm output is an area of intense
research, the currently available tools
provide limited information about
the inner workings of the models. At
the same time, it is interesting that
the analysis of the output from the
intermediate steps of these multi-layer
architectures might provide some
suggestions for isolating interesting
behavioral patterns in the data.
We argue that researchers should
consider using hypothesis-testing
approaches in these cases to generate
new knowledge about the world. These
approaches may seem outdated, and in
fact may be less accurate at describing
the observed phenomena. However,
they do offer researchers complete
control over their inner workings, and
therefore provide a form of language
that researchers can use to construct
and test hypotheses, and therefore
interpret phenomena. We believe that
these are essential as preliminary tests
before adopting ML techniques for
estimation and prediction.
So far, we have implicitly assumed
that the ML algorithms taken into
consideration were ones that involved
supervised learning, meaning that
the scientist can provide labeled
data for training. We should be even
Although the interpretation of
deep-learning-algorithm output is
an area of intense research,
the currently available tools provide
limited information about
the inner workings of the models.
36 INTER ACTIONS J U LY A U G U S T 2 0 17
more careful in the interpretation
of the results from unsupervised
techniques, where scientists do not
have labeled data to begin with, and
therefore the interpretation of the
results cannot be directly guided
by existing examples. One should
consider, for example, the stability of
the results with different parameters
(e.g., in the case of topic models).
We would also like to stress
the importance of visualization
in interpreting behavioral data.
Visualization techniques can be
extremely important not only for
understanding raw data, but also for
interpreting (fitted) models derived
from the application of ML techniques,
for example through projections of
highly dimensional models.
Causality versus correlation.
Another important aspect to consider
is the problem of correlation versus
causation. Most of the results of ML
algorithms provide insights into
association relationships and not
causality relationships. Consequently,
researchers should be extremely
careful in extrapolating conclusions
from results that might be the effect of
correlation and not causation. This is
not a new problem, but it is exacerbated
by the fact that nowadays many studies
are based on data collected through
crowdsourcing, third-party APIs (such
as the Twitter API), and mobile apps
distributed in Web stores and open to
the public. It is also worth noting that
causality is a very active area in the ML
community at the moment. We expect
that many tools will be made available
to practitioners in the years to come.
Controlled versus non-controlled
experiments. Different techniques
should be used in controlled versus
non-controlled experiments. Indeed,
it is important to be very careful in
drawing conclusions from experiments
that rely on non-controlled designs, for
example systems for positive behavioral
intervention. Having said that, there
are well-established methods proposed
by the ML and statistics communities
for dealing with unbalanced
populations. In other words, it is
possible to analyze non-controlled
experiments, but researchers have to
be very careful in the analysis of their
results and in drawing appropriate
conclusions. In non-controlled
experiments, causality analysis is
very difficult but not impossible,
for example, if quasi-experimental
approaches are applied [7]. Indeed,
it is interesting to note that in many
application scenarios, quite often it
is simply impossible to build control
groups when data is crowdsourced or
collected through mobile applications
distributed on Google Play or the
Apple App Store. This is an area of
great interest not only for the ML/
statistics community but also in other
disciplines, for example health studies,
epidemiology, and geo-demographics.
HOW GOOD IS GOOD ENOUGH?
T
AND WHAT DO WE MEAN
BY GOOD?
There seems to exist an unwritten
convention that classifiers with
accuracy above 80 percent are good
enough and therefore publishable.
Yet there is little consistency in how
HCI researchers interpret classifier
accuracy, and in fact how they report
classifier accuracy. We argue that in
addition to accuracy, researchers should
also report baseline performance.
Consider a system that attempts to
infer the gender of a user by analyzing
their mobility habits. In this case, there
are two possible outcomes (male and
female), and therefore we can assume
that a baseline performance is 50
percent (e.g., reflecting the toss of a
random coin). Classifier performance
is judged against this baseline, and
therefore a classifier that performs
at 85 percent accuracy improves the
baseline by a factor of 0.7. Alternatively,
a gesture-recognition system that
differentiates between 15 different
gestures has a baseline performance of
1/15, or 6.6 percent. If such a system
achieves accuracy of 85 percent, then it
is improving the baseline by a factor of
11.9. Hence, interpreting the accuracy
of a classifier needs to be set against
a (random) baseline. And, actually,
we argue that often accuracy results
around 30 to 40 percent might already
INTER ACTIONS. ACM.ORG
be considered excellent in the case of
the difficult classification problems
described earlier. For this reason, it is
fundamentally important to discuss
performance always in relation to
the complexity of the ML task under
consideration (and, indeed, of the state
of the art in the field!).
Furthermore, especially in behavioral
studies, it is important to note that the
baseline is a function of both the possible
outcomes and the relative likelihood of
each. For instance, consider a system
that monitors all the sensors on the
smartphone and attempts to predict
whether a user is going to answer their
phone if someone calls. Even if we
assume only two possible outcomes
(answer, no answer), the baseline is not
necessarily 50 percent. This is because
we may observe that, overall, users
almost always answer their phone
when it rings. If, say, we observe that
90 percent of the time the user answers
the phone, then this also acts as our
baseline: If we construct a classifier
that constantly predicts that the phone
will be answered, its accuracy will
be 90 percent. In this case, if a study
reports their classifier performing at 85
percent, it is actually performing worse
than the baseline. The actual baseline
should then be not a purely random case,
but rather a frequency-based classifier.
Finally, it is worth noting that
accuracy is not sufficient to evaluate
ML classification algorithms. For
example, the existence of false positives
is another very important aspect that
is often not sufficiently considered in
the evaluation of studies that rely on
ML techniques. A false positive is the
result of a test that indicates a certain
finding or condition exists when it
actually does not. An example is the
case of a classifier that reports that a
user can be interrupted at a certain
point in time, when in fact the ground-
truth data demonstrates this is not the
case. A true positive instead is a result
of a test that indicates the condition is
actually verified. Indeed, it is necessary
to report indicators expressing the
sensitivity (i.e., the proportion of
positives that are classified as positives)
and specificity (i.e., the proportion of
negatives that are classified as negatives)
of the results. In the case of binary
classifiers, for example, standard
evaluation techniques include the use of
DOI: 10.1145/3085556 COPYRIGHT HELD BY AUTHORS. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
INTER ACTIONS. ACM.ORG
the Receiving Operating Characteristic
curve (ROC curve) and the Area Under
the (ROC) Curve (usually abbreviated as
AUC). ROC curves are used to evaluate
the specificity and sensitivity of a
classifier considering different threshold
settings of the classifiers. The discussion
of these techniques is beyond the scope
of this article; for an excellent step-
by-step discussion of these and other
evaluation strategies for ML techniques,
we refer the reader to [6].
W
CONCLUSION AND OUTLOOK
We believe that ML offers immense
opportunities to HCI researchers.
However, just as in performing
statistical modeling, we should
constantly remind ourselves of caveats
in the analysis (correlation does
not mean causality). Today too we
must embrace ML approaches while
having a keen understanding of their
current limitations and prospects for
improvement in the near future.
It is also worth noting that nowadays
a large number of tools and libraries
for ML are available as stand-alone
tools (e.g., Weka), R libraries (e.g.,
randomforest), or Python libraries (e.g.,
scikit-learn). We believe that, even if it
is not important for HCI researchers
to understand how the tools work, it is
essential to have a general knowledge
of the underlying algorithms and
key parametersthe knobs of the
algorithmsboth for improving their
performance and for understanding the
data. For these reasons, we argue that a
solid background in the basics of ML is
necessary before adopting these tools in
our research work and practice. Related
to this, it is interesting to note that
various universities have introduced
(or will introduce) an introduction to
ML concepts and techniques as part
of advanced courses in HCI and/or
ubiquitous computing.
Finally, we also believe that
qualitative methods must play a
fundamental role in interpreting
quantitative data obtained by means
of quantitative methods such as the
application of ML techniques. A mixed-
methods approach is usually the most
promising when interpreting human
behavioral data, which is inherently
complex, noisy, and incomplete.
Moreover, often ML techniques are
applied to subsets of the data and,
therefore, the resulting models capture
only a limited part of the phenomena
under observation.
In this article we have attempted to
highlight some issues that are becoming
increasingly important within HCI
research and offer some material as a
basis for starting a discussion in the
community around these themes.
We have emphasized the importance
of understanding the subtleties in
using these techniques and tools,
while keeping in mind the exceptional
opportunities deriving from their
adoption in our research work.
Endnotes
1. Kostakos, V., Ferreira, D., Goncalves,
J., and Hosio, S. Modelling smartphone
usage: A Markov state transition model.
Proc. of the 2016 International Joint
Conference on Pervasive and Ubiquitous
Computing. ACM, New York, 2016.
2. Mehrotra, A., Musolesi, M., Hendley, R.,
and Pejovic, V. Designing content-driven
intelligent notification mechanism for
mobile applications. Proc. of the 2016
International Joint Conference on Pervasive
and Ubiquitous Computing. ACM, New
York, 2015.
3. Pejovic, V. and Musolesi, M. Anticipatory
mobile computing: A survey of the state
of the art and research challenges. ACM
Computing Surveys 47, 3 (Apr. 2015).
4. Agrawal, R., Imielinski, T., and Swami,
A. Mining association rules between
sets of items in large databases. Proc. of
the 1993 ACM SIGMOD International
Conference on Management of Data. ACM.
New York, 1993.
5. Goodfellow, I., Bengio, Y., and Courville,
A. Deep Learning. MIT Press, 2017.
6. Flach, P. Machine Learning. The Art and
Science of Algorithms that Make Sense of
Data. Cambridge Univ. Press, 2012.
7. Tsapeli, F. and Musolesi, M. Investigating
causality in human behaviour from
smartphone sensor data: A quasi-
experimental approach. EPJ Data Science
4, 1 (2015).
Vassilis Kostakos is a professor of computer
engineering at the University of Melbourne.
His research interests include ubiquitous
computing, human-computer interaction, and
social computing. He has a Ph.D. in computer
science from the University of Bath.
vassilis.kostakos@unimelb.edu.au
Mirco Musolesi is a reader in data science
at UCL and Faculty Fellow at the Alan Turing
Institute. His research interests lie at the
intersection of ubiquitous computing, mobile
sensing, large-scale data mining, and network
science. He has a Ph.D. in computer science
from University College London.
m.musolesi@ucl.ac.uk
J U LY A U G U S T 2 0 17 INTER ACTIONS 37
 Asbjrn Flstad and Petter Bae Brandtzg, SINTEF

Chatbots
and the
New World
of HCI

A
Insights A potential revolution is happening
Major technology in front of our eyes. For decades,
companies see chatbots researchers and practitioners in
and natural language human-computer interaction (HCI)
user interfaces as have been improving their skills in
the next big thing. designing for graphical user interfaces.
Natural language as Now things may take an unexpected
a preferred interface turntoward natural language
for interacting with user interfaces, in which interaction
digital services has with digital systems happens not
many implications through scrolling, swiping, or
service providers [1], typically in the
context of messaging applications.
Need a reminder to pick up some
flowers for your husband on the way
home? Ask Jarvis the chatbot to
remind you. Wonder if you should
bring an umbrella to that meeting
in Stockholm? Send Poncho the
artificial weather cat a message and
ask. If technology giants like Google,
Facebook, and Microsoft are right, we
IMAGE BY AMBER CA SE / FLICKR

and opportunities button clicks, but rather through
for the field of HCI. strings of text in natural language.
This is particularly visible in recent
developments in chatbots, that is,
machine agents serving as natural
language user interfaces to data and
will be moving our digital interaction
from websites and apps with graphical
user interfaces to messaging platforms
such as Messenger and Allo. If
this happens, huge challenges and
opportunities await in the field of HCI.

38 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

credit tk

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 39

 Lets have a look at what is
happening in the still relatively
exotic world of chatbots. In early
spring 2016, Microsoft presented its
vision of conversations as a platform,
where artificial intelligence (AI) and
natural language interaction allow
new ways to experience interactive
systems. As a step toward this vision,
the company provided a framework
for bot developers and launched the
first chatbots on Skype. Microsoft,
however, is far from alone in
holding visions of natural language
interaction. At about the same time,
Facebook launched facilities for
building chatbots for its messaging
application, Messenger. Already,
tens of thousands of chatbots have
appeared on Messenger. Though
the vast majority of these are useful
or entertaining for only a tiny
group of peoplemaybe only their
developerssome pleasant surprises
point to the opportunities ahead. For
example, Smokey warns you when
the air quality in your city drops
below acceptable levels; Instalocate
helps you stay updated on your
upcoming flights; and Jessie Humani
takes you for an entertaining
ride through the everyday life of
young adults. Then there is Google
Assistant, a chatbot integrated into
the Allo messaging application
and recent versions of the Android
operating system that outperforms
all the others, though it too can
be pretty dumb at times. Google
Assistant reliably helps you out with
questions in natural language, such
as when the sun sets or where to
find the nearest coffee shop, even
when asked follow-up questions
for directions or opening hours.
Yet conversations break down fast
enough for this to be an interface for
only the most enthusiastic of techies.
It is no mystery why the
major technology companies see
In the not-too-distant future, chatbots
may be the preferred user interface for
many of the activities to which we have
grown accustomed to performing through
a webpage or a dedicated application.
40 INTER ACTIONS J U LY A U G U S T 2 0 17
chatbots as the next big thing.
Mobile messaging applications
now have more than 1.5 billion
users worldwide [2], and they are
competing with social networks
as the main user interface to the
Internet. For many users, natural
language is already the default
mode of interaction online, only
there, the interaction is typically
between human users through a
machine interface. At the same
time, advances in AI mean that
natural language interaction may
be a feasible option for connecting
machine agents and human users
[3]. This already is visible in Twitter,
a service that is welcoming to
machine agents to the extent that
this has become a social network
arena for humans and bots alike.
During the 2016 U.S. presidential
election, it was estimated that
more than one-fifth of the tweets
using the main election hashtags
were generated by machine agents,
potentially affecting the outcome of
the election [4]. There has also been
concern regarding the potential
of automated Twitter accounts for
promoting political positions in
the 2017 French presidential and
German federal elections.
For commercial and nonprofit
service providers alike, natural
language user interfaces are on the
verge of becoming an attractive
interface through which to engage
with customers. Companies like
Dominos Pizza and Taco Bell are
trying out chatbots as booking
agents in messaging applications.
Medical chatbots, such as Cardea, are
providing health advice and doctor
listings. Governments are exploring
chatbots as a means for facilitating
voting in elections. The authors of
this paper are currently exploring
how chatbots may provide youth
with support on mental health issues
[5]. In the not-too-distant future,
chatbots may be the preferred user
interface for many of the activities to
which we have grown accustomed to
performing through a webpage or a
dedicated application. This popularity
leads to an important question:
What implications will the rapid
developments and uptake of chatbots
have for how we approach the design
of interactive systems in HCI?
M
IMPLICATIONS FOR HCI
Microsoft CEO Satya Nadella
likened the foreseen transition to
chatbots and natural language user
interfaces to previous revolutions
such as the introduction of the
graphical user interface, the Web,
and mobile Internet. Facebook
CEO Mark Zuckerberg proclaimed
chatbots to be a solution to the
challenge of app overload. If the
vision of conversational interfaces
holds true, major changes are in
store for the field of HCI.
I can already hear you protesting:
Natural language user interfaces are
nothing new to the field of HCI. In fact,
HCI researchers have studied these
before, for example, in the context
of multimodal systems, interactive
voice-response systems, voice control
in the context of accessibility,
and conversational systems [6].
Nevertheless, the bulk of usability
research and practice arguably
concerns graphical user interfaces
and, to some extent, hardware
design. As a field, we have spent the
past two decades refining how to
design for interaction with webpages
or apps, drawing on ever richer
interaction mechanisms to support
usability and user experience.
A transition to chatbots and
natural language user interfaces has
many implications. The following are
particularly noteworthy.
Conversations as the object
of design. Design for chatbots
represents a transition from
the design of visual layout and
interaction mechanisms to the
design of conversation. In the
current era of graphical user
interfaces, designers benefit from
substantial control of visual design
and interaction mechanisms,
allowing for detailed presentation
of the features and content of an
interactive system. Here, design
INTER ACTIONS. ACM.ORG
for usability concerns navigation
through menus and links, the
browsing of content, and interaction
with graphical elements. In the
future era of chatbots and natural
language user interfaces, the
designer repertoire of graphical
and interaction mechanisms will
be greatly reduced. Indeed, current
messaging dialogues allow for the
inclusion of images, video, sound,
and textual elements. However,
the user interface is to a much
greater degree a blank canvas
where the content and features of
the underlying service are mostly
hidden from the user, and where
the interaction is more dependent
on the users input. Here, design
for usability involves suggesting
to the user what she may expect
in the service and the adequate
interpretation of her response.
Seeing conversations as the
object of design clearly represents
a challenge to the field of HCI. We
need to move from seeing design as
an explanatory taskthat is, a task of
explaining to the user which content
and features are available and which
steps to take to reach the desired
goalto an interpretational task
that is, a task of understanding
what the user needs and how she
may best be served. At the same
time, this challenge may strengthen
our understanding of how to
dynamically adapt the user interface
as the dialogue with the user evolves.
The need to move from user
interface design to service design.
HCI research and practice have, to
an overwhelming degree, addressed
the design of specific user interfaces.
That is, our attention as a field has
been directed toward the object
of design (the interactive system)
rather than the users goals. This is
reasonable, given our grounding in
software engineering and systems
design. However, the transition to
natural language user interfaces may
imply a need to rethink this focus.
Currently, chatbots are embedded
in messaging services. In the
future, they may be in purely voice-
based dialogue systems, like their
precursor, Amazons Echo. The
same natural language interface
will be used whether the user is
chatting with a friend, arguing with
her mother, or asking a chatbot
INTER ACTIONS. ACM.ORG
for help with making New Years
Eve reservations. Interactions,
services, and content previously
demarcated by different webpages
or apps will blur into the same
conversational threads. In the
future era of chatbots and natural
language user interfaces, content and
services do not differentiate by their
user interfaces but rather by their
convenience in accessing the context
of conversational threads. For
researchers and practitioners, this
implies designing for entire service
processes across conversational
touchpoints with the user, rather
than specific user interfaces. Here,
we will have something to learn from
the emerging field of service design.
The need to design for interaction
in networks of human and
intelligent machine actors. Design
in HCI often concerns one user, one
device. This is particularly true with
regard to interaction design. Here,
the object of design and evaluation
is typically the user interface, as it
will be perceived and acted on by
a single user. This is not to say that
other approaches to design do not
exist. For example, the bordering
fields of game design, computer-
supported collaborative work, and
design of sociotechnical systems
have explored design for multi-agent
systems.
In the future era of chatbots and
natural language user interfaces, the
multi-agent aspect of interaction
design must be accentuated. This
need to consider design in the
context of networked intelligent
machine actors is already seen in
social networks. In March 2016,
Microsoft launched Tay, a machine-
learning Twitter chatbot simulating
a teenage girl. Tay was set up to learn
and improve from interactions with
other Twitter users. Within hours of
deployment, the chatbot was turned
into a monster of derogatory and
insulting remarks learned from the
users approaching it. Chatbots can
also work against other chatbots.
For example, research on bots
supporting the maintenance of
Wikipedia articles suggests that
different bots often end up reworking
one anothers editing work [7].
When conversation threads are
populated by multiple actors
human users and chatbots alike
designing for interaction in networks
becomes a prominent challenge.
I
NEW OPPORTUNITIES
In the future, HCI may need to
consider conversations as the main
object of design, focus on services
rather than user interfaces, and
design for interaction in networks of
human and machine actors. All these
changes reflect an underlying need to
reorient HCI research and practice
to meet the challenges of the future
era of chatbot and natural language
user interfaces.
As a result, it may be beneficial
to take a step back and recap
not only the challenges but also
the opportunities that present
themselves in this new landscape,
especially opportunities that
present themselves to researchers
and practitioners of HCI, given the
competencies and capabilities we
have developed within this field.
We are confident that the
opportunities outnumber the
challenges. In the following, we
highlight four areas of research.
Combatting digital divides.
First, chatbots are supposed
to communicate with people
across gender, age, language, and
preferences. However, research such
as our own [8] suggests that new
technologies often create new digital
divides and biases across gender,
age, and societal status. Chatbots
hold great potential as an inclusive
technology. A well-designed natural
language interface should support
uptake of digital technologies and
services across groups that are less
tech-savvy.
Nevertheless, chatbots are
typically set up following a one-size-
fits-all approach, in which all users,
regardless of needs, preferences, and
degrees of digital literacy receive
responses in the same language
by way of the same underlying
set of data and services. Hence,
an important question is whether
the language and perspective
reflected in such one-size-fits-
all setups introduce undesirable
biases. A male-dominated tech
culture has resulted in the slang
term brogramming, referring to
computer code produced by bros
(male friends). In this context,
the knowledge and experience of
J U LY A U G U S T 2 0 17 INTER ACTIONS 41
HCI researchers and practitioners
from user research and analysis
will be essential to identifying
and combating digital divides as
they appear for chatbots. Possibly,
somewhere down the road, chatbots
powered by AI may support
personalization to a degree that such
biases and divides are mitigated.
Understanding conversational
processes. A key success factor
for chatbots and natural language
user interfaces is how well they can
support conversational processes
while providing useful output. The
current state of the art is arguably
Google Assistant and its ability
to hold a conversational thread
across several steps in a dialogue.
However, even with Assistant, the
conversation ultimately breaks
down and the input from the chatbot
becomes irrelevant.
In part, an adequate
conversational process depends
on massive developments within
AI, drawing on deep learning from
large volumes of interaction data.
However, as misinterpretation is
always a possibility in dialogue,
chatbots as conversational agents
need to be designed for both guiding
the user toward attainable goals
and providing acceptable responses
in the case of conversational
breakdown. As a field of studying
interaction processes and error
recovery, HCI will no doubt
have much to contribute toward
well-functioning conversational
processes between chatbots and
human users.
Benefiting from massive
volumes of user data. HCI has
been preoccupied by user-feedback
evaluation since its beginning.
Design and redesign are, by default,
driven by trials involving users
or usability experts, allowing for
iterative improvements in designs.
In particular, qualitative data has
been key to such feedback practices,
where the observations of users
have alerted designers to usability
problems and opportunities for
redesign. However, sufficient access
to users and data for evaluation has
nearly always been an issue in the
field. Not so for chatbot interaction.
In the future era of chatbots
DOI: 10.1145/3085558 2017 ACM 1072-5520/17/07 $15.00
42 INTER ACTIONS J U LY A U G U S T 2 0 17
and conversational interfaces,
HCI researchers and practitioners
may benefit from access to massive
volumes of user data. As the default
mode of interaction is natural
language rather than the clicking of
links and buttons, users intentions
and levels of understanding will
be significantly more accessible
than what is the case in current
interactive systems log data.
Benefiting from this data source
will require HCI researchers and
practitioners to navigate uncharted
territory in terms of data analysis,
as well as ethics and privacy.
However, the potential reward will
be substantial.
Safeguard ethics and privacy.
A final opportunity for HCI
researchers and practitioners within
the new world of chatbots and
natural language user interfaces is to
provide needed guidance on ethics
and privacy. Novel technologies
entail novel ethics and privacy
implications, and chatbots are no
exception. Interaction in natural
language with multi-actor threaded
conversations is a context where
ethical and privacy challenges
will flourish. Furthermore, the
intelligent use of chatbots for
persuasive purposes may entail
important societal implications. As
such, machine agents may be used
to sway individuals opinions in
undesirable ways.
HCI researchers and practitioners
have traditionally been forerunners
in issues of ethics and privacy, as
seen, for example, in the concern
for these in the context of social
networks. With the emergence of
chatbots, even stronger attention on
ethics and privacy is needed.
T
CONCLUSION
The field of HCI has already seen
several waves and has proven to be
a discipline that enthusiastically
takes on novel perspectives and
technological opportunities as they
emerge. Chatbots and AI-powered
conversational interfaces represent
a new world to be conquered. For
that to happen, HCI researchers and
practitioners should consider taking
on human-chatbot interaction design
as an area of research and practice.
We have provided an initial overview
of the challenges and opportunities
of this new world. Now it is up to the
HCI community to take these on.
ACKNOWLEDGMENTS
This work was conducted as part
of the research projects Human-
Chatbot Interaction Design and
Social Health Bots, both supported
by the Research Council of Norway.
Endnotes
1. Dale, R. The return of the chatbots.
Natural Language Engineering 22, 5 (2016),
811817.
2. Statista. Number of mobile phone
messaging app users worldwide from
2014 to 2019; https://www.statista.com/
statistics/483255/number-of-mobile-
messaging-users-worldwide/
3. Hill, J., Ford, W.R., and Farreras, I.G. Real
conversations with artificial intelligence:
A comparison between humanhuman
online conversations and humanchatbot
conversations. Computers in Human
Behavior 49 (2015), 245250.
4. Bessi, A. and Ferrara, E. Social bots distort
the 2016 US Presidential election online
discussion. First Monday 21, 11 (Nov. 2016).
5. Social Health Bots project page: https://
www.sintef.no/socialhealthbots/
6. Allen, J.F., Byron, D.K., Dzikovska, M.,
Ferguson, G., Galescu, L., and Stent, A.
Toward conversational human-computer
interaction. AI Magazine 22, 4 (2001),
2737.
7. Tsvetkova, M., Garca-Gavilanes, R.,
Floridi, L., and Yasseri, T. (2016). Even
good bots fight: The case of Wikipedia.
PLoS ONE 12, 2 (2017), e0171774.
8. Brandtzaeg, P.B., Heim, J., and
Karahasanovic, A. Understanding the new
digital divideA typology of Internet
users in Europe. International Journal
of Human Computer Studies 69, 3 (2011),
123138.
Asbjrn Flstad is a senior researcher
at SINTEF. His main research interests are
human-computer interaction and human-
centered design. He coordinates the EU
H2020 project HUMANE on human-machine
networks. He has a Ph.D. in psychology from
the University of Oslo.
asf@sintef.no
Petter Bae Brandtzaeg coordinates the
research projects Human-Chatbot Interaction
Design and Social Health Bots, supported by
the Research Council of Norway. His main
research interests are patterns of use and
the implications of new information and
communication technologies. He has a Ph.D. in
media and communications from the University
of Oslo.
pbb@sintef.no
INTER ACTIONS. ACM.ORG
IsIsInternet softwaresosodifferent
Internet software different from
from ordinary
ordinary software?
software? This practically
This book book practically
answersanswers this questio
this question
through
through thethe presentation
presentationofof a software
a software design
design method
method basedbased
on theon theChart
State State XML
ChartW3C
XML W3C standard
standard
along
along with Java.Web
with Java. Webenterprise,
enterprise, Internet-of-Things,
Internet-of-Things, and Android
and Android applications,
applications, in particular,
in particular, are are
seamlessly specifiedand
seamlessly specified andimplemented
implemented from
from executable
executable models.
models.
Internet software
Internet softwareputs putsforward
forward thethe
idea of event-driven
idea or reactive
of event-driven programming,
or reactive as pointed
programming, out in out in
as pointed
Bonr et
Bonr et al.
al.ssReactive
ReactiveManifesto.
Manifesto.It tells us that
It tells reactiveness
us that is a must.
reactiveness However,
is a must. beyondbeyond
However, concepts,concepts,
software engineers
software engineersrequire
requireeffective
effectivemeans
meanswithwith
which to puttoreactive
which programming
put reactive into practice.
programming into practice.
Reactive Internet Programming outlines and explains
Reactive Internet Programming outlines and explains such means.such means.
The lack of professional examples in the literature that illustrate how reactive software should
The lack of professional examples in the literature that illustrate how reactive software should
be shaped can be quite frustrating. Therefore, this book helps to fill in that gap by providing in-
be shaped
depth can be quite
professional frustrating.
case studies Therefore,
that contain this bookdetails
comprehensive helps and
to fill in that gap
meaningful by providing in-
alternatives.
depth professional
Furthermore, casestudies
these case studiescan
that
be contain comprehensive
downloaded details and meaningful alternatives.
for further investigation.
Furthermore, these case studies can be downloaded for further investigation.
Internet software requires higher adaptation, at run time in particular. After reading Reactive Internet
Internet software
Programming, you requires higher
will be ready adaptation,
to enter at run time
the forthcoming in particular.
Internet era. After reading Reactive Interne
Programming, you will be ready to enter the forthcoming Internet era.

So You Want
To Be An
AI Designer?
O
Insights On March 14, 2016, at 6:40 a.m., I
AI is an enabler. Products woke up in the cold darkness and got
that use AI should be called washed and dressed. I took an Uber to
AI-enabled products. the San Francisco Caltrain station and
The creator of AI-enabled hopped on the 7:56 train, one minute
products needs to have before it left. The train arrived in
great product thinking Mountain View at 8:49. The first rays
about the problem, user, of sunshine were flying into my eyes,
and use case. dancing aroundWelcome, Nina,
said the sun, smiling. Its like summer
here! I replied, smiling back. Another
IMAGE BY DMY TRO ZINKE V YCH
Uber drove me to the office. At 9:30 I
stood up from the sofa in the lobby and
stepped into the office
Welcome, Nina, to the AI Lab!
It was one of the most memorable
moments in my lifeIll never forget
INTER ACTIONS. ACM.ORG
Nina (Zhuxiaona) Wei
it. From that moment on, my journey
of designing for AI products began. I
felt lucky and grateful for being a so-
called AI designer.
DESIGNER? AI? WOW!
AI, artificial intelligence. One of the
hottest and sexiest terms of 2016,
2017, and the years beyond. But this
term makes me nervousit makes
me so nervous that I dont want
to say it aloud when I introduce myself
to people:
So, what do you do?
I am a designer.
Oh really? Where do you work?
Hmm, I am working at an AI lab
J U LY A U G U S T 2 0 17 INTER ACTIONS 45
 I say this word AI in a low voice, but
you still hear it clearly.
COMMUNICATIONSAPPS AI?! Do you mean artificial
A
intelligence? Cool!
This is what makes me nervous.
I
THE SO-CALLED AI DESIGNER
I am cool in a lot of ways, but not in the
ways youre thinking. I do not consider
myself to be an AI designer. What
exactly is an AI designer? A designer
who designs AI products, you answer.
What is an AI product? Isnt it an
artificial intelligence product? What is
artificial intelligence?
The truth is that I am a normal
designer. I design products you are
familiar with, such as mobile and
Web apps. My daily life is the same
as yoursbrainstorming, sketching,
pushing pixels, or talking all day
Access the nonstop for a user study. Do you still
think I am cool?
latest issue, Even if my job is cool, I do not enjoy
any privileges. Everyone is born to
past issues, serve, either themselves or others. We
designers are here to solve problems
BLOG@CACM, and make peoples lives better. The
News, and only difference between you and me
is the problems we are solving. Most
more. of the time, the problem is the same:
There arent many truly new problems.
We focus on one aspect of the problem
and use different techniques to solve it,
or a small piece of it.
W
MYSTERIOUS AI PRODUCTS
What is AI? In my view, AI enables the
augmentation of human intelligence.
Products that utilize AI technology
Available for iPad,
can be only AI-enabled products. A
iPhone, and Android
product is a solution for solving human
problems. AI is not the solution; AI
itself cannot be a product.
If AI were the core, we would
become too attached to it. We would
ignore the missionsolving human
problems. Every technology is a tool
but not the problem. As creators, we
need to focus on the problem and
Available for iOS,
Android, and Windows
http://cacm.acm.org/
There is a key difference between
about-communications/
mobile-apps
product thinking and design thinking:
The former focuses more on the design
of the problem while the latter focuses
more on the design of the solution.
46 INTER ACTIONS J U LY A U G U S T 2 0 17
detach ourselves from this fantastic
tool. The tool cannot bring value until
we find the right problems.
THE NEW ELECTRICITY
AI is the new electricity, says
Andrew Ng, former chief scientist
of Baidu. Electricity transformed
countless industries. AI will now do
the same.
I first fell in love with AI several
years ago when I was studying
psychology in college. I was attracted
by humans ability to develop artificial
intelligence. But how can we bring
the value of technologies such as AI to
everyone? If normal people like you
and me cannot enjoy the value of a
technology in our daily lives, what is
the technology for?
Products are one of the most
important carriers of technology, but
transforming technology into products
is as hard as developing the technology
itself. That is why I am hereto find
the right problem and design an AI-
enabled product in the right way.
O
LESSONS LEARNED
ON THE JOURNEY
Our product teams mission is to
innovate and incubate AI-enabled
products. As the solo designer, I am
responsible for all the design work as
well as the product strategy. On March
14, 2016, the journey began. Four
months later, on July 15, we released
our first product, the TalkType voice
keyboard (Figure 1). At the end of
February 2017, TalkType downloads
exceeded 100k. In the middle of March
2017, we released SwiftScribe, an AI-
enabled transcription service (http://
swiftscribe.ai/). Meanwhile, we are
exploring and testing more product
ideas. We tried and failed, again and
again. The great feedback from users
makes everything worthwhile. Lessons,
tons of lessons, are being learned on this
journey. Here are some of them.
INTER ACTIONS. ACM.ORG
 Product thinking. Not everyone
should be a product manager, but
everyone needs product thinking.
What is product thinking? Think in
products, not in features, says Nikkel
Blaase, product designer at XING [1].
Understanding the problem and
user needs is the first step in the
design process. User-centered design
is user-centered: Users, not features,
come first. Then how come designers
think in features? I was confused. And
what is the difference between product
thinking and design thinking? I asked
this question in Quora. Here is part of
someones response:
Product thinking is a more holistic
view than UX/design thinkingand its
also much more complicated.
Joseph Dickerson, UX Lead at
Microsoft [2].
I have increased my understanding
along the journey. For one thing,
Blaase comes from a traditional design
background. Everyone is talking about
UX, but much of the industry still
operates in traditional ways: Product
managers define products and write
product requirement definitions while
designers deliver designs, mostly the
UI. Designers with UX Designer titles
are actually UI designers.
Figure 1. Baidu Researchs TalkType voice keyboard.
There is a key difference between
product thinking and design thinking:
The former focuses more on the
design of the problem while the latter
focuses more on the design of the
solution. When building a product, 80
percent of the time we are defining
the problem. And we spend much less
time designing the solution, as shown
in Stanford DSchools design-process
model (Figure 2).
Problem, user, and use case. We
should not add technology to an
existing product without clarifying
the problem that this would better
address. Take chatbots. Chat as a
form of interaction is intuitive. MIT Figure 2. Stanford d.schools design-process model (https://dschool.stanford.edu/).
Technology Review acknowledged
conversational interfaces as one of for multiple and nested items to be problem. Bots are also not new.
the top 10 technology breakthroughs built in. It is a much simpler experience Joseph Weizenbaum from MIT built
[3]. But that does not mean chatbots without conversational capabilities. the first chatbot, Eliza, a computer
can be applied to every problem. E-commerce sites are starting to therapist (https://en.wikipedia.
Opera, Magic, Assist, Lukaso many pull their bots from the shelves, org/wiki/ELIZA). Lacking natural
bot startups were born in 2016 [4]. acknowledging that they didnt do language understanding/processing
Facebook, Telegram, Slackeveryone what they were supposed to [6]. They (NLU/NLP), Eliza is not considered
is building their own bot platform. are hard to use and do not provide to be a real conversational interface.
Facebook has since made changes to personalized service. However, with continual technological
its bot platform, with more emphasis Speech recognition can be the breakthroughs, I believe we will
on actions [5]. It added a menu of breakthrough: Bots cannot be soon be able to have a real chat with a
features offered by bots, which allows disruptive unless we find the right machine [3].
INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 47

Figure 3. AppStore pages for Chatbaka, Cassette, and Recorder.
User. Who is our target user?
Those cool young kids who love
technology and must be hungry for
new things? It turns out thats wrong.
From our data (user feedback and
reviews), we noticed that there are
a lot of not so young and even senior
users. Got this for my 92-year-old
great grandfather and he loves it.
This review made my entire week.
Users appreciate that TalkType solves
their problems. For example, they
might have shaky hands or fat fingers,
or they are nearsighted.
Speech to text is the most
searched keyword among all voice-
keyboard-related searches. People who
are looking for a solution must have a
problem. They are the real users.
I can type very fast with my eyes
closed. Cuz I remembered every
key! said a young girl, holding her
Samsung and showing us the default
Samsung keyboard. Those cool young
kids do not have a problem and they
do not need another input tool. They
are looking for cool new things. They
are eager to express their unique
personalities via the cool new things.
Use case. A use case defines the
There is a balance between scoping
the problem and scaling the solution.
We can build a target solution for
a specific use case, but since there are
similarities among different use cases,
we may lose the opportunity to scale up.
48 INTER ACTIONS J U LY A U G U S T 2 0 17
context for a problem as well as the
relationship between the user and the
problem. For example, note-taking
has various use cases, such as class
lectures, interviews, and jotting down
random ideas. There is a balance
between scoping the problem and
scaling the solution. We can build a
target solution for a specific use case,
but since there are similarities among
different use cases, we may lose the
opportunity to scale up.
The following products (Figure 3)
are solving a similar problem across
different scenarios. They all use
automatic speech recognition (ASR):
Chatbaka Voice Messenger the
safer way to send messages while
driving (http://chatbaka.com/)
Cassette transcribe, record,
and share conversations (http://www.
cassette.design/)
Recorder voice journaling
made easy (https://itunes.apple.com/
gb/app/recorder-voice-journaling/
id1140324040?mt=8).
Test the value of AI. There are two
broad contrasting approaches to build
AI-enabled products. 1) Technology
first: We have this technology. What
kind of product can we apply it to? 2)
Product first: We have this product
idea. What kind of technology do
we need? It costs more time and
resources to develop new technology.
We may already have technology
that has not yet shown its value. So,
we will start with the first approach
to build a product enabled by the
current technology. Then we will
shift to the second approach to work
with the research and engineering
teams to build new technology to serve
user needs.
But applying technology is not as
simple as an add-on or a plug-in. The
example products above all use ASR. Is
the value of ASR the same across every
product? Does ASR even bring value?
Adding AI to existing products can
have a range of outcomes, including:
Fails to solve the problem and
hurts the user experience
Fails to solve the problem
Solves the problem
Solves the problem and improves
the user experience
Solves the problem, improves the
user experience, and brings extra value
or creates new experiences.
We should avoid one and two. Three
is not recommended. Four already can
bring real value. Five is the goalthen
AI is truly the new electricity!
Business value is as important as
user value. But if we pay too much
attention to business value, we may
impose unwanted constraints. If our
product can reach the level of bullets
four and five, the business value will
be clearer.
The two sides of AI. Microsoft
Tay became a racist in less than a day;
Google Photos appallingly identified
two black people as gorillas. The more
value AI brings, the more risks it has.
Even a 1 percent error rate for an
autonomous car can lead to deaths. At
the other extreme, a smart light bulb
may fail to turn on when someone
arrives home; that person might then
become a bit mad and yell at the bulb.
But no big deal, right? The value it
brings is small. Life continues.
The creator of Microsoft Tay
ignored human nature, especially its
bad side. There are two approaches to
create Tay-like chatbots. The machine
gets answers from rules based on user
input. Or the machine will self-learn
from the data. Tay uses the second
approach. It is so smart that it learned
INTER ACTIONS. ACM.ORG
racist language and became obnoxious
in under 24 hours.
How about Google Photo? The
creator may not have used a diverse
enough dataset to train the model.
Or they have not considered all
predictable use cases and have not
tested enough to uncover likely
problems.
What can we learn from such
accidents?
Embrace human nature and design
with all of humanity in mind.
Testing, testingalways testing.
The simpler, the more complex.
What is the best part of AI? It makes
things simpler. Voice makes typing
faster; search-engine algorithms
help you to find whatever you want
in seconds. Smart e-commerce
recommendations know what you like.
How come it is so simple? Because
it is complex. The simpler it is to a
human, the more complex it is to the
machine. Take voice typing. It looks
simplevoice input and transcription
output. It looks like the user does not
have any interactions with the system
except the input. In fact, the user and
the system are interacting with each
other all the time:
User turns on the mic System
starts listening User starts speaking
System is gathering users input data
System starts transcribing System
finishes transcribing System shows
the transcription User pauses for a
moment System is still listening
System may stop listening if there is
no input for a while User resumes
speaking System is listening User
corrects the transcription
Send. The whole experience
ends with this action. There are
other use cases of course: What if
the user speaks too fast? What if the
environment is too noisy?
Think deeply and do not be
fooled by the surface. Look into the
experience of the user as well as that of
the system. Since we are designing the
interaction, wed better pay attention
to both sides to reveal the pain points
and improve the whole experience.
Behavior change is hard. We can
all be lazy. We tend to choose the
default to avoid costs and risks (aka
the default effect). Most users use
the default keyboard that comes with
their phones. It is good enough.
DOI: 10.1145/3106743 COPYRIGHT HELD BY AUTHOR. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
INTER ACTIONS. ACM.ORG
I never thought about looking for
another keyboard.
Behavior change is hard but it is
not impossible. It should better serve
users most urgent needs and guide
them to a new and better experience.
Traditional keyboards provide an
entrance for users to access the voice
input. Unusually, voice input is the
default on TalkType (Figure 1). We
put the mic button up front. Users
become more familiar with voice
input thanks to its strong exposure.
And there is a greater chance that
users will tap the mic button and
speak! Users also have ready access
to the regular keyboard and shortcut
keys such as numbers and emojis.
I
MORE THAN UI DESIGN
If I am alone, I may use it. But if it is in
public, it is too weird.
What do you mean by weird?
Isnt it weird to speak to your phone?
And the phone will not speak back to you?
Whats odd about artificial
intelligence is that it is artificial. It
cannot be more natural than speaking
with an actual person. But it is weird to
speak to something artificial, though it
should involve the same behavior. How
can we make this odd experience more
natural and user friendly? We need to
find out the why. We need to know
the user as well as the machine. We
need to study not only the technology
and the design, but also the sociology
and the humanity.
Two of many researchers who have
studied this relationship between
humans and technology are Clifford
Nass (https://en.wikipedia.org/wiki/
Clifford_Nass) and Sherry Turkle
(http://www.mit.edu/~sturkle/). I
admire them immensely.
This relationship involves more
than design, and far more than UI
design.
SO, YOU WANT TO BE
AN AI DESIGNER?
You enrolled in several machine-
learning courses on Coursera. You read
every news article about AI. You bought
an Amazon Echo and a Google Home.
You want to be an AI designer. You ask
me how.
First, ask a lot of whys: Figure out
your inner desire. Why do you want
to work on AI? Just because it sounds
cool? Or do you believe it can change
the world?
Just try: Try to use products that
make good or bad use of AI; try to
think about problems that AI can
help to solve. And talk to peopleA I
experts, creators of AI-enabled
products, users who like and are using
those products in daily life. Then ask
yourself a lot of whys again until you
find the root cause of your want.
Learn smart, not hard: Knowing
how it works is more important than
learning how to do it. Our goal is not
to be an AI researcher or engineer,
but rather to design better AI-
enabled products.
Collaboration is the key. Its
the beginning of the relationship
between the human and the machine.
There are a lot of challenges as well
as opportunities. It requires AI
researchers, software engineers, and
product designers to collaborate and
build AI-enabled products that can
solve existing problems and create new
experiences.
Endnotes
1. Blaase, N. Why product thinking is the
next big thing in UX design. 2015; https://
medium.com/@jaf_designer/why-product-
thinking-is-the-next-big-thing-in-ux-
design-ee7de959f3fe#.lg558qy4g
2. See full response at https://www.quora.
com/What-is-the-difference-in-product-
thinking-and-design-thinking/answer/
Joseph-Dickerson?srid=dPPX
3. Knight, W. Powerful speech technology
from Chinas leading Internet company
makes it much easier to use a smartphone.
2016; https://www.technologyreview.
com/s/600766/10-breakthrough-
technologies-2016-conversational-
interfaces/
4. CB Insights.The rise of bots: A timeline
of major VC-backed bot startups; https://
www.cbinsights.com/blog/bot-startups-
timeline/
5. Cohen, D. Here are the new features
available to Facebook Messenger chat bot
developers. 2017; http://www.adweek.
com/digital/facebook-messenger-
platform-1-4/
6. Pathak, S. Drop it like its bot: Brands
have cooled on chatbots. 2017;
https://digiday.com/marketing/
brand-bot-backlash-begun/
Nina (Zhuxiaona) Wei is a product designer
at an AI lab who has a great passion for
products, psychology, and the relationship
between humans and technology.
weizhuxiaona@gmail.com
J U LY A U G U S T 2 0 17 INTER ACTIONS 49

Insights
Low-tech plays an
essential role in the
development of high-tech
work environments.
Organizations supporting
I M A G E C O U R T E S Y N A S A / B A S A LT R E S E A R C H P R O J E C T
remote planetary science
can serve as valuable
examples on workflow
design and best practices for
organizations that also rely
on distributed workgroups,
telecommunication, and
analog environments
for product testing,
work-practice development,
or job training.
INTER ACTIONS. ACM.ORG
Inspiring
Innovation:
On Low-Tech
in High-Tech
Development
I
In June 2016, in the small town of
Arco, Idaho, several dozen visitors
set down temporary stakes to be near
Marsactually, a version of Mars,
located 18 miles southwest of Arco
on Route 24 in the Craters of the
Moon (COTM) National Monument
and Preserve. Named in the 1920s
when the moons visible craters were
still thought to be of volcanic origin,
COTMs terrain is geologically similar
to that of Mars: craters formed by
volcanic eruptions; swaths of basaltic
rock; no standing surface water. Its
physical environment provides an
analog for Mars, one of many around
the world. The untamed natural
Zara Mirmalek, Harvard University
environment is exactly the kind of
place with which people who build
systems for Mars exploration need to
grapple. People like Darlene Lim, who
leads this team of visitors on a project
called BASALT: Biologic Analog
Science Associated with Lava Terrains
(https://spacescience.arc.nasa.gov/
basalt). BASALT is a NASA-funded
project that is developing technical
and social systems for supporting a
future work environment in which
humans and robots on Mars conduct
scientific exploration while sharing
visual, audio, and other instrument
data with Mission Support experts on
Earth.
J U LY A U G U S T 2 0 17 INTER ACTIONS 51
 COTM is one of the sites where
principal investigator Lim and the
BASALT team (BASALTers)
are conducting simulations of
interplanetary science exploration.
While COTM is a stand-in for Mars,
the science is real. Their project
involves collecting actual field data
of scientific interest for analysis and
public dissemination (e.g., academic
conferences, news media, and
publications), but under simulated
Mars mission conditions. The
BASALT project is already yielding
new scientific insight into the volcanic
geology and biology of COTM (and
will later at another Mars analog
in Mauna Ulu at Kilauea, Hawaii),
as well as recommendations for
conducting human-robotic science
exploration on Mars.
I joined them in Idaho for
BASALTs first two-week simulation.
As an ethnographer who studies
human-technology relationships,
particularly in remote science work
and extreme environments, I have
conducted ethnographic research
among scientists and engineers
operating robots on Mars (NASAs
Mars Exploration Rovers mission)
and among scientists using robots in
the deep ocean to collect data. While
these domains are fantastical, the
work itself is very much of this world.
Like in almost any organization, there
is a need for developing workflows and
technologies that respond to the ways
in which work is actually conducted.
This means taking into consideration
contextual factors that affect how
people work in situ. Indeed, this
approach is a hallmark of workplace
ethnography, a method that has been
used effectively across a variety of
work domains, from transportation
and space exploration to machine
technicians and AI scientists
[1,2,3,4,5].
BASALT centralizes operations
While some high-tech items have life
spans that can be cut short the moment
a new, better, more adoptable version
comes along, many low-tech items have
hung around beyond expectation.
52 INTER ACTIONS J U LY A U G U S T 2 0 17
research in conjunction with the
goals of bio-, geo-, and space sciences,
which is one reason I know the project
is of interest to audiences outside
the traditional space exploration
community. BASALTers are situating
the development of their tools,
processes, and people in arduous,
unwieldy conditions in which a
future work system will operate. It
is one thing for a tool to function
properly in a controlled lab setting
and another for it to work under
stressful field conditions, where
instrument visibility is affected. Also,
rock formations are not naturally
even-surfaced specimens. Likewise,
a plan that looks foolproof on paper
easily encounters disruptions
when enacted in an environment
where the availability of sunlight
determines work hours. Their
project also includes the temporal
challenge of coordinating across
workgroups in different time zones.
Using software designed by a NASA
Ames team to support space-mission
operations, BASALT simulations
of interplanetary communications
between workgroups in Arco and
COTM include latency (time lag) of
five- and 15-minute delays between
sending and receiving information
from Mars to Earth.
BASALTs work-system research
questions are of universal interest,
such as: What are the optimal
configurations of workloads for
teams distributed across planets, and
why? How will this workflow adjust
in relation to changing planetary
conditions, like rotations that affect
communication-transmission time?
Which kinds of new technologies, or
modifications to existing technologies,
are needed for scientists on Earth to
be able to interact with collaborators
on Mars? BASALTs research will
offer answers to these questions
that benefit not only the space-work
domain but also intraplanetary work
systems. Their research findings
inform audiences of space scientists,
engineers, administrators, and the
public. And they are informative for
any audience interested in distributed
work, interdisciplinary workgroups,
technology development in simulated
environments, and participatory
design.
COTM is, like Mars, not a place
where BASALTers could set up their
entire work site, which includes
two stations for housing Mission
Operations (one on Mars, and one
on Earth). These stations were set
up in a long trailer formerly used by
NASA for public events, and parked
in a recreational vehicle (RV) park
on the edge of town. It was a popular
and quiet location. RV residents
changed every few days. Some would
walk over to ask about the BASALT
trailerthe large space-exploration
murals on its sides naturally attracted
attention. For many BASALTers, the
cultural norms of RV parks were new
but understood as important features
of the project. As for their own
temporary homes, the team resided in
two of the towns motels, walking or
carpooling the short distance to the
RV park.
Workdays were long. Each
day began at 7 a.m. with an all-
hands breakfast meeting in the
park restaurant. The day formally
ended between 9 p.m. and 10 p.m.,
following a meeting to review the
days operational and scientific
accomplishments and challenges.
For many, the workday continued
with after-the-meeting meetings,
affectionately called after-parties,
for further decisions and to knock out
tasks needing completion before the
next days simulations.
The BASALTers setup included
a worksite feature Id seen only on
research ships and, to a lesser degree,
at Google: fixed meal times with
food and space for the entire team
to eat together. It is a longstanding
practice often used when the work
community is isolated, like on a ship
at sea, or too large for local food
services to accommodate. BASALTs
arrangement with the RV park
included three meals a day, prepared
by management and sisters Beth
and Sherri, whod traveled to town
to help with hosting BASALT. The
INTER ACTIONS. ACM.ORG
meal setup (breakfast and dinner
in the RV park and bagged lunches
carried into the trailer and field) was
an important attribute supporting
the work of innovation, which rarely
follows a traditional work schedule.
It contributed stability to a process
that must be flexible and tolerant of
schedule changes, which can often
mean skipping activities that dont
immediately appear pertinent to the
task at hand. The fixed meal setup,
however, provides more than one
kind of nourishment. In a switch from
workrooms prescribed seating, the
open seating at mealtime allowed for
new seatmate configurations, planned
and random, and conversations
fostering a cross-fertilization of ideas.
BASALTs simulation of remote
planetary science uses two locations
and three workgroups, two on Mars
and one on Earth. On Mars, one
team is in the field (walking through
volcanic terrain) and another team
is in a worksite (a space habitat)
from which they can communicate
with the Mars field team, with no
latency, and with the Earth-based
Mission Operations worksite, with
latency. Between the Mars field team
and Mission Operations there is no
two-way communication, but the
Mission Operations teams can hear
the communication between the Mars
teams, with latency. On some days
they test streaming video with latency
from the Mars field team to Mission
Operations.
From Arco, the Mars field team
drove to COTM and (with permits)
hiked out across sharp and shifty old
lava flow. They moved through terrain
surveying and collecting samples,
relying on the other two teams for Figure 1. Two field team members conducting science exploration simulation on Mars.
direction and support. Part of the The image shows the actual COTM geology.
field teams task set is to provide
terrain descriptions using human members were also working on tablets, backpacks, antennas, desks,
senses and several spectroscopic field software support, maintaining the chairs, paper, monitors, refrigerators,
instruments to the Mars station team communications infrastructure batteries, dry-erase boards, binders,
and Mission Operations scientists (which includes people on the field markers, Velcro, duct tape, extension
(Figure 1). The science team confers team carrying antennas), science cords, Post-it notes, printers, and
over the information, responding decision-making, activity metrics, and fans. Some of these technologies are to
with questions and sample-collection time-schedule monitoring. computers what early computers are
requests. In the BASALT workspace, to 21st-century robots. While some
In the RV park, the Mars amid discussions about satellite high-tech items have life spans that
station workroom and the Mission communications, bandwidth, and the can be cut short the moment a new,
Operations workroom (Figure 2) were transmission of spectral readouts, better, more adoptable version comes
in the same trailer but separated by a there were low-tech work-support along, many low-tech items have
small hallway and a commitment to technologies attracting little to no hung around beyond expectation.
not crossing this imagined planetary attention. Items in various states of The dream (for some) of the paperless
divide during simulations. Team use included headsets, microphones, office never actualized; low-tech is not
INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 53
 Figure 2. Inside mission control where scientists are working with two teams on Mars,
including a team in the field that can be seen on display in the video feed on the back wall.
obsolete in high-tech work systems
[6,7]. Low-tech often holds steady
some of the many moving temporal
and spatial features of high-tech work.
They may be placeholders for pending
developments, manual shortcuts
for things that will be replaced by
automation, or irreplaceable items for
supporting work features that high-
tech continues to puzzle over, such as
intercultural and multidisciplinary
communication.
As high-tech tools are put
through the paces of building,
testing, dissection, rebuilding, and
examination for heterogeneous
workplace needs, low-tech provides
In some settings, a workgroups
power hierarchy can be indicated
by who does and does not have
access to the team laser pointer,
computer monitors, or printer.
54 INTER ACTIONS J U LY A U G U S T 2 0 17
a stable, familiar mechanism on
which to rely. It also captures the
transmutation process. Paper lineages
develop among notebook pages,
Post-it notes, and cocktail napkins,
carrying design ideas, workarounds,
complaints, and drawings that may
never get transferred to shared
organizational repositories. Post-
it notes hang on screens, walls,
windows, and laptops, brightly
flagging important information
about updates or issues, plus the
names of digital archives that are
in use but hard to locate in the sea
of email threads and image folders.
Dry-erase boards visible to every
INTER ACTIONS. ACM.ORG
person in the room display up-to-
the-minute information on work
changes faster than any other media.
In some settings, a workgroups
power hierarchy can be indicated by
who does and does not have access
to the team laser pointer, computer
monitors, or printer. Printer access
is dissemination power. It allows
people to produce mobile versions
of their text to be easily shared as
handouts in a meeting, taped to walls,
or carried into the field for use when
the high-tech information sources are
interrupted or break down.
For all the current and future
deserved attention the BASALT
project receives for its high-tech
innovations and complex systems
development, there is one aspect
that could be easily overlooked:
BASALT gives a compelling example
of the importance of low-tech in the
development of high-tech innovations.
That example is needed, given how
easy it is to brush past low-tech items,
particularly in work environments,
where the cultural value placed on
high-tech is absolute. The categories
of high-tech and low-tech help us to
look at such a work environment
and identify opportunities for new
tech and better work support. For
example, a pattern of reliance on a
low-tech item among user groups
within the context of an automated
system may signal an issue at the
infrastructure level (rather than user
error or deficit). As with most things,
defining what fits in the categories
should be considered in context rather
than universally prescribed.
In the various high-tech work
environments in which I have
worked, Ive found that the appeal
and importance of shiny objects (e.g.,
robots, new displays, lightweight
mobile tech, flying objects) often
overshadow the less attention-
grabbing but no less important
low-tech features that are essential
in the development of new social
processes and technologies. This is not
from intentional disregard (though it
occasionally can be). A low-tech object
can be so familiar that it blends in
with the rest of the taken-for-granted
infrastructure. In conversation with
some BASALTers on this topic, they
raised points about how research
DOI: 10.1145/3085562 COPYRIGHT HELD BY AUTHOR. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
INTER ACTIONS. ACM.ORG
funding and public interests can shape
which high-tech innovations are more
often studied and developed. We even
had our own event for discussing this
topic, which took place the day before
the start of simulations.
BASALTs summer 2016 research
in COTM coincided with the National
Park Services (NPS) 100-year
anniversary. A NASA Community
Day featuring BBQ , BASALT, another
NASA project team, FINESSE (Field
Investigations to Enable Solar System
Science and Exploration), and NPS
rangers was held in Bottolfsen Park in
Arco and open to all. By the count of
stickers handed out by NASA Ames
Public Affairs Officer K. Williams,
more than 200 people came through,
including NPS rangers, journalists,
an NPS artist-in-residence, college
students (from the nearby college
towns of Boise and Pocatello), adults,
and kids of all ages. Talking with
some of the visitors that day, I learned
that many had come to see what
NASA was up to and to learn about
Mars and COTM.
BASALTers had four stations
set up in Bottolfsen. The first, set
in the parking lot, was a recurring
demonstration of volcanic eruptions
using a large plastic barrel, water,
liquid nitrogen, and a plastic soda
bottle (Trashcano). On the park
greens, three canopied stations
with folding tables were set up,
each giving a different look into the
work of remote planetary science
explorationcommunication and
latency, geology, and unmanned aerial
vehicles (UAV). Through the lens of
high- and low-tech representations,
the displays as a whole also offered an
interesting subtext.
A review of four stations artifacts
highlights the use of high- and low-
tech to conduct remote planetary
science. BASALTers were using
these very tools to develop and test
a new interplanetary work system.
At the geology station, two scientists
stood at a table on which a paper map
was held in place by rock samples.
At the UAV station, set with large
flat-screen monitors, visitors could
fly a simulated UAV to scout terrain.
At the communication and latency
station, two scientists called for sets
of public volunteers who were then
separated on each side of the walled
BBQ picnic area so they could not see
or hear one another without the use of
digital media. They were guided to ask
questions back and forth via tablets
set to delay receiving communication.
A steady stream of visitors came to
each station staffed with enthusiastic
BASALTers; the most constant lively
discussions between the public and
scientists was at the geology station.
Shiny, high-tech objects attract
more attention, but the basic low-tech
technologies that are consistently
carrying part of the workload always
warrant a close looknot just for
descriptive colorful detail, or as a
juxtaposition for pointing out age
or speed, but as items that may
need to be modified or reinvented
in relation to work-support needs.
Among the research BASALT offers
is an opportunity to consider what
we can learn from developing work
environments for remote planetary
science missions that is applicable to
work environments that never leave
this planet.
Endnotes
1. Star, S.L. The Cultures of Computing.
Blackwell Publishers, Cambridge, MA, 1995.
2. Orr, J. Talking About Machines. Cornell
Univ. Press, Ithaca, NY, 1990.
3. Forsythe, D. Studying Those Who Study
Us: An Anthropologist in the World of
Artificial Intelligence. Stanford Univ. Press,
Stanford, CA, 2002.
4. Suchman, L. Plans and Situated Actions.
Cambridge Univ. Press, New York, 1987.
5. Hughes, J., King, V., Rodden, T., and
Andersen, H. The role of ethnography in
interactive systems design. Interactions 2,
2 (1995), 5665.
6. Sellen, A.J. and Harper, R.H.R. The
Myth of the Paperless Office. MIT Press,
Cambridge, MA, 2001.
7. Nomura, S., Hutchins, E., and Holder,
B.E. The uses of paper in commercial
airline flight operations. Proc. of the 2006
20th Anniversary Conference on Computer
Supported Cooperative Work. ACM, New
York, 2006, 249258.
Zara Mirmalek studies work culture,
professional identity, and human-machine
relationships in work environments where
people work with remote presence tools (e.g.,
robots and digital media). She received a Ph.D.
in communication and science studies from the
University of California, San Diego. Currently
she is a Fellow in the Program for Science,
Technology & Society, Harvard University.
zara_mirmalek@hks.harvard.edu
J U LY A U G U S T 2 0 17 INTER ACTIONS 55
 Richard E. Ladner, University of Washington
Kyle Rector, University of Iowa

Making
Your
Presentation
Accessible

Insights
You may have audience a talk at a conference such as CHI,
members in your
conference talk with
a disability, attending
remotely, or who learned
English as a second
language, so it is important
I M A G E B Y A N D R I J B O R Y S A S S O C I AT E S / S H U T T E R S T O C K .C O M
I
It is extremely common to accompany and resources for giving accessible
conference talks.
CSCW, IUI, DIS, UIST, and others Before discussing the guidelines,
with presentation visuals. But there we would like to give some examples
may be people in the audience who are of practices that limit accessibility at
blind, have low vision, or who cannot conference talks. In each example, we
see the visuals clearly. There may also state what the speaker might say and
be deaf or hard-of-hearing people in the then why the talk is not accessible.

to make your talk audience, or those who cannot hear your
accessible to all. talk clearly. Some audience members
The focus of the audience may be attending the conference by
should be on the speaker, telepresence robot. Some may have
not the slides. learned English as a second language or
Following guidelines to be unfamiliar with the topic. Since the
make your talk accessible talk is for everyone, it is important to
can improve the quality make the talk accessible. Indeed, even
of your talk. if there are no people with disabilities
in the audience, the talk should be
broadly inclusive. The purpose of this
short article is to provide guidelines
Images. These are examples of X.
Several images on the slide represent
examples of the topic at hand. If an
audience member is blind, that person is
at a loss to follow the examples.
Video 1. Let me show you a video
demonstrating process Y. The video
is narrated but has no captions. If an
audience member is deaf, he or she
cannot follow the narration.
Video 2. Let me show you a video
showing Z in action. The video has no

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 57

CACM_TACCESS_one-third_page_vertical:Layout 1
ACM
Transactions on
Accessible
Computing
This quarterly publication is a
quarterly journal that publishes
refereed articles addressing issues
of computing as it impacts the
lives of people with disabilities.
The journal will be of particular
interest to SIGACCESS members
and delegates to its aliated
conference (i.e., ASSETS), as well
as other international accessibility
conferences.
a teaching momentthe great teachers
www.acm.org/taccess
www.acm.org/subscribe
58 INTER ACTIONS J U LY A U G U S T 2 0 17
6/9/09 1:04 PM Page 1
narration, just some background music.
If an audience member is blind or has
low vision, then Zs action is a mystery.
Graphs. This graph shows some
growth of W over time. A blind or low-
vision audience member will not have
any idea of the magnitude of the growth
or the time period.
Pointing. Look at this equation that
shows the relationship between X and
Y. The speaker is using a laser pointer.
A blind or low-vision audience member
will not know what the equation is or
what the relationship is. A deaf audience
member who is watching the interpreter
might not be able to also see where the
laser is pointing.
Allusion. You never know who you
might meet. The slide shows a picture
of the speaker and the President of
the United States shaking hands. The
reference might be lost on a blind or low-
vision audience member.
Animation. Watch how our
algorithm manipulates the data. There
is no narration to help blind or low-
vision audience members understand
the algorithm.
These examples are not meant to
imply that images, videos, graphs,
pointing, allusion, and animation should
never be used. Instead, they illustrate
the risks of using these devices in talks.
W
BASIC GUIDELINES
What is the purpose of giving a talk
at a conference? Having attended
hundreds of conferences and perhaps
thousands of talks in our careers, we
have found that a great talk is one that
inspires us to read the paper and want
to talk to the speaker about the work.
A great talk is not about the slides; its
about the speaker. The focus of the
audience should be on the speaker, not
the slides. The slides only amplify what
the speaker is saying. How the speaker
connects with the audience makes
people want to listen, so it is paramount
that the speaker know who the listeners
are. Think of speaking at a conference as
visuals on the slides.
How the speaker connects with the
audience makes people want to listen,
so it is paramount that the speaker know
who the listeners are.
teach to the level of the students, not
over their heads. Naturally, it is always
good to face the audience and progress
slowly and deliberately through a talk so
that everyone can keep up.
I
ACCOMMODATIONS
It helps to understand some of
the accommodations, and their
implications, that audience members
may use.
Personal assistive listening devices for
people who are deaf or hard of hearing
include hearing aids and cochlear
implants. Since these devices generally
do not completely restore hearing,
those wearing them often want to be
able to read the lips of a speaker while
listening. This may mean that the
speaker should be as close as possible
to the audience and face the audience
as much as possible. Some assistive
listening devices have FM capability. In
this case, the speaker may be asked to
wear an additional microphone so that
the speakers voice arrives more clearly
at the listening device.
Sign-language interpreters are often
requested by deaf audience members.
It is important to recognize that a deaf
audience member using an interpreter
can focus on only one thing at a time:
the interpreter, the speaker, or the
slides. Furthermore, interpreters are
really language translators, so there is
a slight delay from when the speaker
says something to when the deaf person
gets the same information. This means
that when referring to information on
a slide, it is good to pause for a moment
to allow time for the translation and the
attention shift to the slide.
Tactile sign-language interpreters are
often requested by deaf-blind audience
members. It is even more important to
follow the guidelines above for sign-
language interpreters, but to go even
more slowly and have less cluttered
slides. The tactile sign-language
interpreter is not only interpreting what
you are saying, but also describing the
INTER ACTIONS. ACM.ORG
 Minimize the number of slides. No
one wants to be shot with a fire hose
ADDITIONAL RESOURCES while trying to understand your talk.
Keep graphics simple. No one wants
Here is a short list of resources that may be valuable in preparing your talk and your paper:
Cavender, A., Trewin, S., and Hanson, V. SIGACCESS Accessible Writing Guide; http:// to read a complicated graphic when
www.sigaccess.org/welcome-to-sigaccess/resources/accessible-writing-guide/ there are only a few important facts
Trewin, T. SIGACCESS Accessible Conference Guide; http://www.sigaccess.org/ about it. Save the complicated graphic
welcome-to-sigaccess/resources/accessible-conference-guide/ for the paper.
Foster, S., Long, G., and Snell, K. Inclusive instruction and learning for deaf students in Use high contrast and take care
postsecondary education. Journal of Deaf Studies and Deaf Education 4, 3 (1999), 225235.
with colors. Audience members with
Foster, S., Long, G., Ferrari, J., and Snell, K. Providing access for deaf students in a
technical university in the United States: Perspectives of teachers and instructors. In
low vision or color blindness will
Educating Deaf Students: Global Perspectives. D. Power, G. Leigh, eds. Gallaudet Univ. appreciate it.
Press, Washington, D.C., 2004, 185195. Avoid or control the speed of
Foster, S. and Holcomb, T. Hearing-impaired students: A student-teacher-class animations so they can be described fully.
partnership. In Special Educational Needs Review: Volume 3. N. Jones, ed. Falmer Press, This will help people who cannot see
London, 1990, 5782. the animation clearly. All audience
Holcomb, T. and Foster, S. Communication in mainstream classrooms: A matter of
members will appreciate an animation
courtesy. Perspectives in Education and Deafness 11, 2 (1992), 1011.
Burgstahler, S. Universal design: Implications for computing education. Trans. Comput. that moves slowly and is explained.
Educ. 11, 3 (Oct. 2011), Article 19, Page 7. Make sure that videos are captioned
Supalo, C. Techniques to enhance instructors' teaching effectiveness with chemistry and audio described. Sometimes it is
students who are blind or visually impaired. Journal of Chemical Education 82, 10 (2005), 1513. good to give a brief description of what
is in the video before it is played. This
will help blind audience members to
Real-time captioning is another provide an advance copy of the talk establish context for what they will
request that can be made by deaf in an accessible format. Fortunately, hear.
audience members. Such requests are PowerPoint and some other systems Make sure the Q&A period is
typically satisfied by using a professional support accessibility such as alternative accessible. This is helpful if the speaker
captionist, who in real time creates text for images. The notes section of or audience members cannot easily see
a written transcript of what is said. each slide can be used to provide textual who is asking a question, or if audience
The transcript could be broadcast descriptions as well. In the future, it members are sitting toward the back of
on a screen for everyone to see, or on may be possible for the slides to be the room, or if the room is large. If there
an individuals laptop display. In the automatically described rather than is a microphone for questioners, make
not-too-distant future, automated having the author add annotations sure they use it. Otherwise, repeat the
or crowdsourced speech-to-text may in the notes section. Sign-language questions so everyone can hear them.
replace professional captionists. interpreters and captionists can If a member of the audience is using a
Regardless of how real-time captioning also benefit by having copies of the telepresence robot, make sure they have
is done, it is important to note that a presentation slides ahead of time. the opportunity to ask a question.

H
deaf audience member can focus on
only one thing at a time: the captions,
the speaker, or the slides. Again, there
is a slight delay in transforming speech
to text. This means that the same
principles applicable to sign-language
translation also apply to real-time
captioning.
Telepresence robots may be requested
by members who wish to remotely
attend a conference. Most often, a Beam
telepresence robot is used; the attendee
can personalize the robot to wear a
badge and other accessories. With
the robot, the attendee can navigate
the conference and participate in the
audience. The person may be following
along with slides on their own computer.
Speak clearly and somewhat slowly so
this kind of attendee can follow along.
Advance materials may be requested
by blind or low-vision audience
members. A speaker may be asked to
PRACTICAL SUGGESTIONS
Here is a short list of practical
suggestions for giving an accessible talk:
Minimize the amount of text on slides.
This should help keep the focus of the
audience on what you are saying. As
soon as the slide appears, announce
it, then pause for a few seconds to let
people read it before saying anything.
This will allow deaf people and
everyone else in the audience to read the
slide before you start talking. Repeat
the text on the slide to make sure blind
people in the audience know what is on
the slide.
Minimize the number of visuals on
slides. Again, this should help keep the
focus of the audience on what you are
saying. Each image should be described
so that blind people in the audience will
know what is there. Graphs and charts
should be described and summarized.
ACKNOWLEDGMENTS
The preparation of this article was
supported in part by National Science
Foundation grant number CNS-1539179.
Richard E. Ladner is professor emeritus
in computer science and engineering at
the University of Washington. His research
interests are in HCI with an emphasis on
accessibility for people with disabilities. He
also leads AccessComputing, an NSF-funded
alliance to increase the participation of people
with disabilities in computing fields.
ladner@cs.washington.edu
Kyle Rector is an assistant professor of
computer science at the University of Iowa. She
has research interests in human-computer
interaction and accessibility. She is specifically
interested in developing eyes-free technologies
that enhance quality of life, including exercise
and art technologies for people who are blind
or low vision.
kyle-rector@uiowa.edu

DOI: 10.1145/3085564 2017 ACM 1072-5520/17/07 $15.00

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 59

 F O R U M I N T E R A C T I O N A N D A R C H I T EC T U R E
Interaction design is increasingly about embedding interactive technologies in our built environment; architecture is
increasingly about the use of interactive technologies to reimagine and dynamically repurpose our built environment.
This forum focuses on this intersection of interaction and architecture. Mikael Wiberg, Editor
Toward Intelligent Environments:
Supporting Reflection with
Smart Objects in the Home
T
Maliheh Ghajargar, Politecnico di Torino
he growing interest in the
Internet of Things and in
technological, connected,
and computing-enhanced
spaces such as smart
homes (Figure 1),
intelligent environments,
and responsive environments connects
interaction design more and more with
architecture. Everyday spaces such as
home environments are increasingly
filled with computing and smart objects.
This trend of ubiquitous computing,
as envisioned and pioneered by Mark
Weiser at Xerox PARC, has since
worked as a basis for designing smart
environmentsacross people, objects,
and spaces. Many researchers have
further investigated how the flows
and patterns of activities in a space
can guide the design of interactions
with smart objects [1]. In particular,
the notion of place is used when spaces
frame interactions through cultural
values and behavioral expectations [2].
In designing smart homes,
attention is often paid to the user
experience and to the ways in which
those environments can support
the inhabitants daily activities.
Furthermore, many smart objects aim
to not only support but also capture
awareness of and evoke reflections
about user activities in domestic
environments. For example, reflecting
about activities that require reductions
in energy or water consumption, or
reflecting about food consumption
in order to foster behavior changes
toward healthier food choices. Although
the purposes in those examples may
differ, they share the same principle:
Reflection can help people make better
60 INTER ACTIONS J U LY A U G U S T 2 0 17
choices and change behaviors.
In this regard, Lars Hallns and
Johan Redstrms slow technology
is an approach that emphasizes the
role of technology to foster moments
of reflectioninstead of efficiency
in performancein domestic
environments [3]. More recently, the
HCI community has provided many
experimental and speculative examples
of smart devices that can support
reflection in the home.
In his seminal work How We Think,
John Dewey describes reflection as
a deep consideration of experiences
and actions in order to discover
connections, that is, relations between
things [4]. Reflection demands time
and continuity; it helps guide people to
understand a situation deeply, allowing
them to take careful and informed
courses of action for change. Reflection
as an activity is not only an individual
and internal process; it also requires
external stimuli: objects, other people,
activities, and the environment (see,
for example, [5]). Furthermore, in HCI,
reflection refers to the action of thinking
Insights
In designing a smart space, the key
element is the relation between
people, activities, and smart
objects. This is also relevant for
designing smart objects that
support reflection about an activity
in such spaces.
Analysis of the three main relations
that a smart space has with other
elements is essential for designing
a smart object that supports
reflection about an activity.
about the information provided by
smart objects in order to capture
awareness about an action and its
consequences. Accordingly, reflection
can become a valuable concept in
the design of everyday smart objects
embedded in place [6].
People not only think with objects,
but they also often engage in an
activity with an object, so reflection
about an activity is connected to the
smart object, the activity, and the
place [7,8]. Consequently, designing
for reflection about an activity
requires consideration of the relations
among those factors, which deal with
the social, aesthetic, and technical
interactions in a given environment [2].
SUPPORTING REFLECTION
ABOUT AN ACTIVITY
To design effective smart objects that
support reflection, we can learn from
the relations among existing objects,
activities, and environments. However,
smart objects intended to evoke
reflection about an activity are not
always designed in relation to the spaces
where people perform that activity.
They may also not be the objects with
which users interact during that activity
in order to achieve a functional goal.
For example, the Energy Orb is a smart
object in the form of a glass ball that
provides real-time data about energy
consumption and energy price, enabling
users to modify their energy usage. It
communicates by glowing in different
colorsgreen when the consumption
and pricing are low, and red when the
consumption and pricing are high. It is a
calm ambient technology that requires
no cognitive effort from users. At this
INTER ACTIONS. ACM.ORG
 time, the Energy Orb is not an object
that is used during an activity; it does
not demand any relationship with the
activity and its place. Consequently,
users may soon not pay much attention
to it. There are many other smart
objects on the market similar to the
Energy Orbfor instance, Home
Joule and Energy Joulethat are
designed with the purpose of helping
users save energy and money and
also to capture awareness about their
energy consumption (e.g., http://www.
ambientdevices.com/).
While a few prototypes of smart
objects (mostly from eco-feedback
technologies) experiment with such
relations in mind, many others do
not. For example, we can consider
two examples, both related to water
consumption: an eco-feedback display
[9] and Waterbot [10]. While Froehlich
et al.s eco-feedback display is not used
in relation to any activity in which activity, and then design smart objects actually design the relations among those
water is used, Arroyo et al.s Waterbot is that evoke reflection about that activity. components. However, in order to have
actually installed and connected to the Thus, for designing such smart objects, a systematic approach and a clear idea
faucet, which is the object with which the key factor is the existing relations about the outcome of design, we may
the user interacts during an activity among the components of a space. focus on one component at time and
such as hand washing (Figure 2). then consider the relations around it.
Therefore, we need to consider that SPACES, SMART OBJECTS, For this article, I will start with
evoking reflection in users about an AND USER ACTIVITIES: the smart space, since it physically
activity is appropriate when the user is: RELATIONS contains the other components, and
actually doing that activity, The components of a space include then illustrate the relations within it.
in the place where the activity is objects (smart, digitally enabled, I will explore the three main relations,
usually being done/has been done, and or not), people, user activities, and namely: 1) smart space-activity 2)
interacting with the objects and/or the architectural structure of the smart space-objects and 3) smart
people involved in that activity. space itself. Considering that these space-people relations.
In addition, smart objects in components are actually interconnected Smart space-activity relations. In
a space that aim to support reflection and related to one another, what is the this relation, a smart space becomes a
need to provide guidance and design outcome? Is it the architectural place as it holds particular activities,
be persistent, instead of merely space, the object for use in a particular cultural expectations, and definitions
representing information. environment, or the object in use for a [2]. For example, home is a private
This analysis may have some particular activity? Or is it the relations architectural space where people
advantage for guiding the design of among objects, people, and activities live their private lives, have personal
meaningful forms of interaction for within an environment? relationships, and perform activities
reflection about an activity [4]. It advises Having reflection about an activity that are often distinct from those in
designers to first consider the activities which is in relation with objects, activity, their public or professional lives. Thus,
that naturally occur in a given space and people, and space [3]as a design the architectural space definition
the objects and people involved in that concept, it seems clear that we should and meaning are closely related to
TOP IMAGE COURTESY OF HOLI

Figure 1. The smart home, from left to right: Amazon Alexa, Insitu Smart Lock, Philips Hue Smart Lighting.

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 61

 F O R U M I N T E R A C T I O N A N D A R C H I T EC T U R E
Figure 2. left: Waterbot [10] right: eco-feedback display [9].
the activities the user usually does in
that space. For instance, considering
the home as a smart space, a kitchen is
defined as a place where people make
food. Accordingly, there are tasks related
to that place, such as cooking, boiling
water, cutting vegetables, and so on.
Those are activities that by definition
occur in that specific place. Other
examples are, for instance, sleeping
and waking up in a bedroom or taking a
shower in a bathroom. Thus, for example,
in designing for reflection about the
activity of taking a shower, the bathroom
is the right place for evoking reflections
about that activity.
Smart space-objects relations.
According to the definition and
meaning of the space, people engage
in tasks in relation to objects, which
are generally presented in that specific
space. Considering a smart home as an
example, in a kitchen we find pans and
an oven; in a bedroom, we find a bed;
and so on. So, for designing a smart
object for reflection about the activity
of sleeping, it seems appropriate to pick
a preexisting object that supports that
activity and other actions related to it.
For example, Bonjour is a smart alarm
clock that was designed to support the
same user activity for which the original
was invented: waking up on time. And
for that reason, it is usually placed
next to a bed in a bedroom (Bonjour
startup: https://www.indiegogo.com/
projects/bonjour-i-smart-alarm-clock-
with-a-i-sleep--2#/). Bonjour is an AI
conversational agent. It is connected
to the weather forecast, iCal, Google
calendar, Google maps, and traffic
monitoring so it can adjust the wake-up
time for a user if certain conditions are
met. This alarm clock also supports
good sleep, which is another activity
naturally related to waking up on time!
DOI: 10.1145/3095712 2017 ACM 1072-5520/17/07 $15.00
62 INTER ACTIONS J U LY A U G U S T 2 0 17
Thus, an alarm clock could become a
smart object not only to support waking
up on time and sleeping well, but also to
evoke reflections on those activities for
the user with the goal of improvement.
Smart space-people relations. Some
places are for a specific person. For
instance, when we call a specific place
in our home my room, this actually
means that place has been configured
accordingly to my taste, my daily
activities, and my things. When other
people interact with that place, they may
not fully recognize its whole structure
and configuration. Alternatively, there
are also spaces that are designed for
social interactions, for example the
dining area in a home environment,
which structures configurations that are
not specific to one person.
In our spaces, which are increasingly
computational and intelligent, we use
objects in our daily activities. Through
this article, I sought to build upon
existing bodies of knowledge that are
well grounded in architecture and HCI.
They suggest that we first observe the
pattern of peoples activities and the
objects of use in a space in order to design
better and supportive architectural
spaces, as well as to design better
computing artifacts that can support
user activities [1,8]. In this way, an
architectural space becomes smart by
supporting natural existing relations
within it, such as relations among people,
objects, activities, and the space itself.
Further, considering these relations
when designing smart objects to support
reflection about an activityinstead of
creating new objects and consequently
new usage and interactionsis a
valuable way of structuring the analysis
of complex spaces [8]. This is well
grounded in theories (e.g., distributed
cognition) that describe how people
think with objects, and that reflection
is distributed across people, objects,
and spaces [7,8]. There are three main
relations between the architectural space
and other components in it: people,
activities, and objects. Analyzing those
relations becomes even more relevant
as we increasingly consider reflection as
a goal for design outcomes, especially
for the design of smart and interactive
artifacts [6].
Endnotes
1. Venkatesh, V. Computers and other
interactive technologies for home.
Communications of the ACM 39, 12 (1996).
2. Harrison, S. and Dourish, P. Re-Placing
space: The Roles of place and space in
collaborative systems. Proc. of the 1996
ACM conference on Computer Supported
Cooperative Work. 1996, 6776.
3. Hallns, L. and Redstrm, J. Slow
technology: Designing for reflection.
Personal and Ubiquitous Computing 5, 3
(2001), 201212.
4. Dewey, J. How We Think. D.C. Heath and
Co., Boston, 1933.
5. Rogers, Y. A brief introduction to
distributed cognition. 1997; http://mcs.
open.ac.uk/yr258/papers/dcog/dcog-
brief-intro.pdf
6. Ghajargar, M. and Wiberg, M. Thinking
with interactive artifacts: Reflection as a
concept in design outcomes. Design Issues.
Forthcoming 2017.
7. Turkle, S. Evocative Objects: Things We
Think With. The MIT Press, Cambridge,
MA, 2011.
8. Whittaker, S., Terveen, L., and Nardi, B.
Let's stop pushing the envelope and start
addressing it: A reference task agenda
for HCI. Human-Computer Interaction 15
(2000), 75106.
9. Froehlich, J. et al. The design and
evaluation of prototype eco-feedback
displays for fixture-level water usage data.
Proc. of the SIGCHI Conference on Human
Factors in Computing Systems. ACM, New
York, 2012, 23672376.
10. Arroyo, E., Bonanni, L., and Selker,
T. Waterbot: Exploring feedback and
persuasive techniques at the sink. Proc. of
the SIGCHI Conference on Human Factors
in Computing Systems. ACM, New York,
2005, 631639.
Maliheh Ghajargar is a Ph.D. candidate in
the Department of Architecture and Design
at Politecnico di Torino, Italy. She is currently
a Ph.D. visiting student in the Department
of Informatics at Ume University, Sweden.
The main area of her research is the design
of reflective interaction between users and
computer-enhanced artifacts.
maliheh.ghajargar@polito.it
INTER ACTIONS. ACM.ORG
 F O R U M C O M M U N I T Y + C U LT U R E
Community + Culture features practitioner perspectives on designing technologies for and with communities. We highlight
compelling projects and provocative points of view that speak to both community technology practice and the interaction
design field as a whole. Christopher A. Le Dantec, Editor
Socially Just Design and
Engendering Social Change
H
Lynn Dombrowski, Indiana University Purdue University Indianapolis
CI as a discipline cares
about issues of equity,
fairness, and social
justice, as denoted by the
attention its scholarly
works and evolving
dialogue give to such
topics. For example, within work from
HCI and HCI-related disciplines such
as ICT4D, community informatics, and
digital civics, we can find celebratory
projects meant to help people improve
their diets, tools that help mistreated
workers act collectively to improve
their situations, and applications that
help those with disabilities gain more
equitable access to information, among
many others. By social justice within
design, I refer to how designers attend
to the ways in which people experience
oppression and marginalization,
including how burdens, obligations,
power, benefits, and privileges have
been unevenly distributed within
society. When related to HCI, these
concerns often include how technology
is designed, developed, and used, and
how public policy impacts information
and communication practices. Often
this means that concerns of social justice
focus on how oppression, such as racism,
sexism, ableism, ageism, classism, and
so on, impact peoples experiences with
technology, information, and design.
Taking on a social justice perspective
changes how designers engage in design
situations, including who they partner
with, methods of determining agreeable
outcomes, and how designers might
interrogate the design situations at
hand [1]. For example, within the space
of food insecurity, instead of limiting
their design interrogations to current
INTER ACTIONS. ACM.ORG
situations by asking How might I help
people who are food insecure have
enough to eat? designers may also
inquire about social inequalities related
to food access, which can help them
understand why a person may not have
enough to eat in the first place.
Working toward socially just
design is neither a straight path nor a
clear process; instead, it is a constant
struggle, as the ideals of social justice
continually shift to become more
inclusive. This work is often fraught
with good-faith efforts, allies acting
poorly, people struggling to get by,
moving targets, evolving tactics, and
many design failures. In the Design
Justice Lab at Indiana University
Purdue University Indianapolis
(IUPUI), we approach designing for
social justice as a multi-pronged effort
because we believe that diverse avenues
of design, scholarship, and resistance
are necessary to work toward the goals
of social justice. However, picking up
a multi-pronged effort often requires
contending with conflicts that arise
while interrogating multiple, and
sometimes competing, perspectives.
Insights
HCI, interaction design, and related Similarly, within the domain of food
disciplines already engage in social
justiceoriented design work, and
there are open questions about how
to design for social justice issues.
There are many different ways
to work toward social change.
Engaging in and attending to
resistance and conflict in
the design process may yield
productive insight!
Given that, I want to talk briefly about
the conversations my students and I
have while designing for social justice
among a variety of topics, including
equity issues related to food, workers
rights, immigration, human rights,
and policing. Specifically, I want to
focus on the different ways in which we
discuss how scholars might conceive of
the relationship between design, social
justice, and social change.
THERE ARE MANY WAYS
TO SUPPORT
SOCIAL CHANGE EFFORTS
Often, thoughts about social justice
and social change go hand in hand
as we identify an injustice but may
struggle to figure out what we can do
about it. A primary concern for my
lab is understanding and assessing
alternative pathways for social change
within the resource-constrained and
socially marginalized communities
with whom we work. While there may
be an allure of attending to the most
idealized versions of emancipation
and empowerment, as designers, we
cannot ignore the very real needs
that people face today, or ignore
the effects of longstanding trauma
within marginalized communities.
justicewhich focuses on social justice
concerns that pervade food systems
from production to consumption
and includes issues such as hunger,
long-term pesticide exposure, and
farm laborer concernsthere are
different debates about what kinds of
change might be most useful. Some
perspectives state that we ought to help
people with their immediate health and
J U LY A U G U S T 2 0 17 INTER ACTIONS 63
 F O R U M C O M M U N I T Y + C U LT U R E
safety concerns and other needs, such as
a hot meal for today. Other scholarship
focuses on fostering systemic social
change (e.g., food deserts; lack of equity
in transportation, jobs, and education)
that helps shift the food system so that
future generations will not know food
insecurity. For social change efforts to
be workable and useful, we need to both
support their urgent and immediate
needs and try to foster longer-term
social change. Ideally, we would focus
on both the changes we can make today
as well as longer-term social change
so that our efforts are unnecessary in
the future. Otherwise those needs will
persist. In my lab, we have discussions
focused on trying to balance current
needs with trying to work toward new
futures. The design projects range
from fostering better, more timely food
donations to persuasive, interactive data
visualizations and applications that help
nonprofits motivate donors, politicians,
and communities.
You might also be wondering, given
that social change is an ambitious
endeavor, why we think interaction
design can help facilitate real change.
We struggle with this question too.
Intellectually, we believe that better
access to specific forms of social,
collaborative information may help
people collectively act to contend
with wicked social problems. Thus, in
my lab, we try to better understand
what sociotechnical forms that data
and design might take to foster
social change. For example, how do
we design an application that helps
identify which soup kitchen might
be able to use hundreds of pounds
64 INTER ACTIONS J U LY A U G U S T 2 0 17
of excess hot prepared food; how do
we design a mobile application that
helps an undocumented worker track
their hours to help them get paid;
and how do we help governmental
aid recipients understand whether
they are getting the right benefits?
However, pragmatically we know
there are many facets of wicked
social problems for which traditional
design approaches may not be the
best avenue. Instead, we might need
to focus on efforts such as education,
community building, active citizen
engagement, voting, or donations as
more impactful alternatives. Thus,
as part of our approach, we try to
understand when and under what
conditions taking a computation
and interaction design approach is
appropriate and when it is not.
ENGAGING COMMUNITY
PARTNERS IN RESISTANCE
We know that partnering with
communities is useful in the design
process because it helps designers better
understand the practices, desires, and
goals of the people theyre designing
for. Working with communities fosters
necessary mutual trust and helps
designers check their assumptions about
people and a design space. Partnering
with communities may also yield other
benefits, as they can help designers
better understand possible preferable
futures and how to collectively work
toward those futures. Across our
fieldwork and partnerships, we see
people who are actively trying to resist
oppressive practices, policies, data, and
people. We see urban farmers trying
to understand how data, social media,
and marketing might help them reclaim
agriculture from big agribusiness, and
community social workers trying to
develop alternatives to state-controlled
websites to help their people better
understand benefits to which they
are legally entitled. In each of these
examples, we see people creating and
enacting narratives about how things
could be different in ways that oppose
or resist destructive narratives and acts
that hinder communities.
Fostering resistance creates the
possibility of positively changing the
relationship between the oppressed and
the oppressor. However, because the
people in most need of social justice are
typically the most vulnerable and most
resource constrained, my lab has been
having conversations about the merits
of different forms of resistance for
engendering social change. Many types
of direct resistance existprotest and
civil disobedience, for example. There
are also more subversive, less obvious
practices like peasant resistance [2],
where people do not directly contest a
powerful oppressor, but instead engage
in noncompliance practices. Design
has great potential to help in these
scenarios as designers attend to and
I M A G E B Y D U R A N T E L A L L E R A / S H U T T E R S T O C K .C O M
build toward their partners preferred
form of resistance.
CONFLICT, WHILE
UNCOMFORTABLE,
IS NECESSARY
FOR SOCIAL JUSTICE
When engaging in a thorny issue with
various stakeholders, a designer has
to work across and between multiple
INTER ACTIONS. ACM.ORG
sets of institutional and organizational
policies, expectations, and relations.
Conflict, in various forms, is likely to
occur. Though the presence of conflict
in socially just design may seem like
an impediment, it is actually a healthy
sign that a project is tackling topics
worthy of debate. This is especially
true for marginalized people, who have
interests in asking questions [about
power, oppression, and inequality],
and dominant groups have interests
in not hearing them [3]. Given that
marginalized groups frequently lack
protection during disagreements, we
often try to understand how to scaffold
conflicts in a way that best protects
these groups. A designer is thus likely
to experience multiple places where
conflict is possible, as they are managing
those different expectations from their
community partners and institutions.
In my lab, we see these conflicts arise in
several pragmatic ways. For example,
we will discuss the boundaries,
opportunities, and tensions of
being a good ally for a social justice
cause when you do not necessarily
participate in the struggle directly
because you are not a part of the
marginalized class. Another source of
conflict is managing both institutional
responsibilities and community goals
(i.e., whats useful for tenure may not
be useful to the community). Thus,
conflict is to be expected in the design
process but will be difficult given the
variety of areas in which dissonance
can occur (e.g., responsibilities as
a good partner to a community
organization versus the institutional
obligations as an academic).
Conflicts also happen when
people might share the same goal
but vehemently disagree on how to
reach it. Hunger-focused nonprofits
all have a shared goal of wanting to
serve their community. However,
these organizations have fundamental
disagreements about how to
demonstrate that they actually help
their communities become more food
secure. Typically, many such nonprofits
tell their funders and donors about
how much food they have delivered to
demonstrate impact. However, some
of these organizations have articulated
that pounds of food delivered is not a
DOI: 10.1145/3085560 COPYRIGHT HELD BY AUTHOR. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
INTER ACTIONS. ACM.ORG
good proxy for actual impact on the
local community. Thus, they wanted
to demonstrate impact by showing
how their programs, rather than just
giving food, actually help make people
less food insecure. This definitional
work matters for the kinds of data they
collect and the programs they develop
as they develop more resilient food
communities. Thus, this conflict was
an opportunity to rethink how they
were understanding and evaluating
their own impact.
BEING AN ACCOMPLICE
REQUIRES PRAGMATICS
AND SELF-CARE
Lastly, an under-discussed facet of
working toward social change is the
affective or emotional labor this work
often requires. For scholars, it tends to
be both deeply personal and at times
emotionally difficult. Within social
justice work, there is a distinction often
made between being an ally and being
an accomplice. Typically, an ally might
be able to better advocate for an issue
to external stakeholders and those who
are not yet enrolled in the problem,
whereas an accomplice works directly
with the affected community and
actively takes on risks for the benefit of
the group. There are several tensions
regarding the politics of being an ally
versus an accomplice.
First, from my personal experience,
scholars and designers who take up
this style of work tend to do so because
they feel a deep personal connection
with a vulnerable group. Frequently,
the people who do this style of work are
often themselves part of a vulnerable
group, but also feel in some way that
they are privileged because they are
scholars or have in some way made it.
Thus, in various aspects of their lives,
they are simultaneously marked with
particular privileges while contending
with feeling marginalized. Second, a
single person cannot be an accomplice
for all social issues that they might
care about at all times. To be a good
accomplice (and ally) requires resources,
time, and energy, and thus people must
choose which issues to actively tackle.
Often people sway between being an
accomplice and ally for different issues
throughout their social networks.
While I do not have perfect answers
for these tensions, in my lab I advocate
that we identify our own limits and
that of our work. That we ought to
remind ourselves that the social
issues we deal with have long legacies
and deep roots, and while we do not
know what effect our small design
interventions might have, the work
we do is prudent and timely. Finally,
social problems are big problems.
We may not be able to address all of
the underlying causes perpetuating
an issue. However, through analysis,
we can target specific tangible areas
where we might be able to gain
productive design traction. In my lab,
this often results in partnering with
nonprofits and activist groups to help
strengthen the work they already do.
CONCLUSION
While concerns of equity, fairness,
and social justice are longstanding,
in some ways the interaction design
and HCI professions are still in their
infancy, as the methods, theories, and
tools we employ continue to be shaped,
developed, and evaluated. These are
all productive challenges regarding
a socially just design practice. We
stand to learn a lot about design as
we come to better understand the
strengths, weaknesses, and trade-offs
of applying design practices toward
wicked social problems.
Endnotes
1. Dombrowski, D., Harmon, E., and Fox,
S. Social justice-oriented interaction
design: Outlining key design strategies
and commitments. Proc. of the 2016 ACM
Conference on Designing Interactive Systems.
ACM, New York, NY, 656671.
2. Scott, J.C. Weapons of the Weak: Everyday
Forms of Peasant Resistance. Yale Univ.
Press, 2008.
3. Harding, S. Is Science Multicultural?
Postcolonialisms, Feminisms, and
Epistemologies. Indiana Univ. Press,
Bloomington, IN, 1998.
Lynn Dombrowski is an assistant professor
in the human-centered computing department
in the School of Informatics and Computing
at Indiana University Purdue University
Indianapolis. She runs the Design Justice lab,
where she studies, designs, and evaluates
sociotechnical systems aimed at issues of
social inequity.
lsdombro@iupui.edu
J U LY A U G U S T 2 0 17 INTER ACTIONS 65
 F O R U M H C I E D U C AT I O N
HCI education reflects the continual evolution of HCI, embracing the changing landscapes of technology, infrastructure,
and technology use. This forum aims to provide a platform for HCI educators, practitioners, researchers, and students
to share their perspectives, reflections, and experiences related to HCI education. Sukeshini Grandhi, Editor
The Future of User
Experience Education
T
Andrea Peer, UserZoom
he HCI Living Curricu-
lum Committee was born
from the idea that the
field is evolving too fast
and is too varied to have
a static, one-size-fits-all
curriculum. Scholars
in the field have proposed that HCI
curriculum recommendations need to
be current, relevant, and reflective of
the multitude of HCI philosophies and
practices that exist in the ever expand-
ing global community [1]. Additional-
ly, previous work recognizes the need
to prepare professionals academically
in a more robust and structured way
to grow user experience (UX) capacity
in organizations [1]. One response to
the evolution of HCI [2] is to consider
educational programs that will provide
both structure and mechanisms to
navigate the complexity of the field.
In this article, I propose such
a framework in support of the HCI
Living Curriculum initiative.
Recently, I completed a two-year
project, co-creating the HCI curric-
ulum at Iowa State University (ISU)
[3]. The HCI program at ISU is part of
the Virtual Reality Application Center
(VRAC), which provides HCI educa-
tion for students from ISUs eight col-
leges and two schools. In this role, I saw
the challenges of developing a rigorous
research program for a STEM-oriented
multidisciplinary graduate degree that
embraced students from all under-
graduate programs, none of which
were HCI. There was the additional
challenge of balancing the I need these
skills now demand from those stu-
dents in industry with the I have this
really cool idea for technology excite-
66 INTER ACTIONS J U LY A U G U S T 2 0 17
ment from the full-time engineering
and computer science students. These
demands appear to be conflicting
motivations, yet all of the students are
seeking some level of HCI education.
I believe this experience is represen-
tative of the larger challenge facing
the HCI Living Curriculum. At our
disciplines core are humanist values,
namely values that place importance on
understanding and accounting for phe-
nomena that impact humans. General-
ly, this core serves the field quite well in
research and practice, providing a wide
variety of perspectives pulled together
by a shared set of drivers. But at this
time in HCIs maturation, the vastly
varied phenomena associated with the
human impact of this field cannot pro-
vide the structure and guidance needed
to navigate the education demands.
John Long and John Dowell stated
that a disciplinary field is the use of
knowledge to support practices seeking
solutions to a general problem having a
particular scope [4]. To build on this
concept and put the problem of our
ever-expanding discipline into busi-
Insights
The growing complexity
of the HCI discipline demands
a different approach to how
we design HCI curricula.
A new purposed framework
supports both HCI-sensible
and HCI-centric approaches
to HCI education.
Key components of the
framework include the
foundation, transparency layers,
and curricular structures.
ness language, HCI education calls for
a healthy dose of scope management.
I propose that the HCI Living Curric-
ulum effort may achieve structure and
guidance as well as growth by embrac-
ing action to move beyond the philo-
sophical debate about HCI as a field or a
philosophy and provide practical sets of
curricular structures that our practi-
tioners and scholars can share.
In order to really understand this
recommendation, it is helpful to first
briefly explain the key issues in the state
of HCI as a discipline.
HCI AS A COMPLEX FIELD
The depiction of the field as seen in two
classic figures (Figures 1 and 2) illus-
trates the shift from simple to complex.
Figure 1 depicts HCI in the 1992 cur-
riculum from ACM SIGCHI. What was
an excellent depiction at the time now
has evolved into a much more complex
depiction of HCI with John Carrolls
use of Dan Saffers image, as shown in
Figure 2.
As another example of the growing
complexity, a more nuanced under-
standing of use and context has emerged
since 1992 (Figure 1). One need only
look at the CHI 2016 program to find
a breadth of uses, ranging from death
to interacting in the wild, and contexts
that range from the kitchen to the
crowd. Each use and context demands a
curriculum in its own right.
In an attempt to understand the
ramifications of complex systems in
traditional academic fiefdoms, Wil-
liam Newell and Julie Thompson Klein
[7] discussed the effects of interdisci-
plinary studies in the 21st century on
higher education institutions. They
INTER ACTIONS. ACM.ORG
 stressed the shift from simple systems
to complex ones. Simple systems are
structured with linear connections and
rely on hierarchy. Complex systems,
on the other hand, may contain simple
systems but are not based on linear
connections. Further, [t]he interaction
of genuine or perceived incompatibili-
ties gives complex systems their unique
unstable behaviors [7]. Clearly, HCI,
once a simple system, is now complex;
the challenges found in recent SIGCHI
education research [1] are indications of
this shift.
In the case of a model with tightly
coupled structural relationships in
complex systems, no one variable in a
model that depicts the HCI discipline
whether you consider the five categories
included in the 1992 curriculum or the
40-plus focus areas in Saffers model
can be treated in isolation. The variables
impact each other and overlap. As a
result, we need a different approach to
how we design HCI curricula. In every
syllabus shared on the SIGCHI edu-
cation webpages (http://HCIbib.org/
education), mastery of either methods
or theories is the primary learning goal.
Methods reflect a truth that our com-
munity has adopted, as Carroll argued,
conceptions of how underlying science
informs and is informed by the worlds of
practice and activity [that] have evolved
continually in HCI since its inception
[8]. Methods, as described in the syllabi
and considering Carrolls comments,
figure 1 acm. used with permission. figure 2 courtesy john m. carroll. used with permission.
could be the HCI sensibility that
Churchill et al. suggest [1], while the
theory parts of the syllabi are the
Figure 1. Depiction of HCI as a discipline from the 1992 Curriculum initiative [5].
INTER ACTIONS. ACM.ORG
HCI-centric elements that support our
field. My purposed framework for the
Living Curriculum allows for a both/
and rather than an either/or approach to
HCI curricula. The purposed frame-
work supports both the HCI-sensible
and the HCI-centric. It also suggests a
blending of disciplinary boundaries for
portions of the curriculum for a specific
educational need or purpose.
A PURPOSED FRAMEWORK
The purposed framework consists of
three major concepts: (1) foundation (2)
transparency layers and (3) curricular
structures.
Purposed frameworkfoundation.
The foundation of the purposed frame-
work is based on the two major parts of
the field that HCI curricula tend to in-
corporate: HCI-sensible courses, where
the primary focus is on application and
practicemethods, procedures, and so
onand HCI-centric courses, where the
primary focus is on HCI theory, models,
research, and so on. In Figure 3, the
gray boxes represent the time spent or
the number of credit hours dedicated to
courses that are HCI-sensible as com-
pared with courses that are HCI-cen-
tric. This diagram offers one suggestion
of how the breakdown of time spent in
HCI-sensible and HCI-centric might
look for each degree type.
The undergraduate students seek-
ing HCI courses are usually coming
from other domains (as only five un-
dergraduate HCI programs exist in the
world; http://www.sigchi.org/resourc-
es/education/2011-education-project-1/
Figure 2. Carrolls conception of the HCI discipline [6].
syllabi). Undergraduates mostly seek a
way to test software or assess the impact
of their technology on humans, hence
the larger portion of the gray box in the
HCI-sensible area. However, they need
an introduction to the fundamentals of
the HCI field, hence the small part of the
gray box in the HCI-centric box.
Practitioners are usually in need
of immediately applicable knowledge.
In my experience in industry, practi-
tioners have little tolerance for theory
without a clear path to application. The
practitioner HCI courses are therefore
predominantly in the HCI-sensible area.
Full-time HCI masters students
may be fully immersed in an HCI
program and require both HCI theory
(HCI-centric) as well as methods and
practices (HCI-sensible). Hence the pro-
portion in HCI-sensible and HCI-cen-
tric is almost 50/50.
Finally, the Ph.D. students may
have the strongest potential eventually
to grow the field. Thus, they require an
HCI-centric curriculum rich in theory.
Although the Ph.D. gray box is predom-
inantly within the HCI-centric area,
HCI-sensible courses are still needed to
connect theory to practice.
Purposed frameworktranspar-
ency layers. In considering co-major
curricular models, we can imagine
handling various disciplines somewhat
like those old-fashioned projectors on
which teachers placed layers of trans-
parencies. Each discipline would have
its own transparency mapping the
core knowledge hubs of that discipline.
The beginning of the HCI discipline
J U LY A U G U S T 2 0 17 INTER ACTIONS 67
 F O R U M H C I E D U C AT I O N
map might look like Figure 4, where
each box is a knowledge hub for the
HCI-centric and HCI-sensible parts.
For the HCI-centric focus, Carroll
provides a wonderful set of possible
knowledge hubs in [9]. For an HCI-sen-
sible focus, courses on sites such as
Interaction-design.org (https://www.
interaction-design.org/) and UX
Matters (http://www.uxmatters.com/)
provide possible knowledge hubs.
Purposed frameworkcurric-
ular structures. The both/and ap-
proach also allows for variability of
the curricular structure for students
seeking greater customization. Lets
take, for example, a masters student
who is co-majoring in HCI. These
students vary in the amount of credits
that they can allocate toward HCI,
as it might simply be augmenting
their primary field. I propose that the
HCI Living Curriculum could offer
curricular structures for each possi-
ble co-discipline: Instead of courses
focusing either solely on HCI or solely
on another discipline, we make courses
that combine HCI topics and the other
disciplines topics for a portion of the
students total curriculum. Figure 5
shows a few examples of the common
use cases for masters students seeking
some degree of HCI education and
illustrates how the curriculum struc-
tures could be bound. The combined
courses that I propose are the mixed
applied courses, which focus on the
application aspects of both HCI and
the other discipline, and the advanced
co-major courses, which focus on re-
lationships between the theories of the
two disciplines.
Curricular structure 1. This may con-
sist of only the HCI discipline transpar-
ency layer (Figure 4). The HCI Living
Curriculum Committee could have an
agreed upon set of HCI knowledge hubs
on both the HCI-sensible and HCI-cen-
tric side, and recommend a 50/50 cur-
riculum for curricular structure 1.
Curricular structure 2. In consider-
ing this structure, imagine a student
co-majoring in an HCI curriculum who
wants to dedicate his research focus to
developing virtual spaces that promote
collaboration among experts in the
field of biology. From the HCI trans-
parency layer, the relevant HCI-centric
knowledge hubs for this student might
be introduction to HCI, collaborative
68 INTER ACTIONS J U LY A U G U S T 2 0 17
HCI-Sensible
HCI sensibility. HCI is an add-on.
Primary focus on
application and practice in industry.
Trajectory of these
students is industry.
Masters
Practitioner
Undergraduate
Figure 3. Purposed frameworkfoundation.
technologies, and computer-supported
cooperative work (CSCW), along with
the HCI-sensible hubs of user research,
visual design, and interaction design.
Looking at curricular structure 2
(Figure 5), these courses would be the
gray box. Now, overlay a transparency
layer for the field of biology. Mapping
the biology field transparency on top of
the HCI transparency, the HCI Living
Curriculum could look for similar rel-
evant theories as well as opportunities
for blending between the two domains.
Then, through the crowdsourcing
techniques discussed in the original
committee proposal, create/recom-
mend options could be offered for the
mixed applied courses and the advanced
co-major courses (yellow boxes) so that
the student may better design for and
ultimately serve the online community
of biologists. Given the wide variety of
possible co-major scenarios, I imagine
there will be multiple curricular struc-
ture 2 options.
Curricular structure 3. Last, imagine
an engineering student focused on
wearable technologies. If she were
seeking a masters degree in a comput-
er science program, the HCI Living
Curriculum effort would offer cur-
ricular structure 3 to that computer
science program. Part of the recom-
mended curricular structure would
contain how many classes and which
classes that student would need to have
an HCI-sensible and basic HCI-cen-
tric understanding. In addition, the
committee could collaborate with that
HCI-Centric
HCI theory, models, and framework.
Primarily focused on theory creation
and advancing the field.
Trajectory of these students
is academia and strategists.
Ph.D.
computer science program to create the
mixed applied courses by overlaying
the computer science transparency lay-
er on top of the HCI transparency layer
and looking for relevant and connected
theories/practices between the fields.
Tying the curricular-structures
concept together, the more domain
expertise that is required, the more
complicated it will be to create courses
that achieve that expertise while con-
sidering the traditional credit limits in
each academic program. If a student
needs more HCI-sensible courses and
is not co-majoring (curricular struc-
ture 1), the recommendation may be
to cover more of the HCI knowledge
hubs on the HCI transparency layer. If
the student needs more HCI-sensible
courses and is co-majoring (curricular
structure 2 or 3), the HCI Living Cur-
riculum may recommend traversing
less of the HCI transparency layer and
making strong connections to other
discipline layers. The key here is that,
in both cases, where a student either
covers more of the HCI transparency
layer or makes more connections to
other fields, the contribution to the
HCI field will potentially be more
profound and lasting compared with
that of students who traverse less of
the HCI transparency layer and make
fewer connections.
Limitations of the framework. While
the framework covers many multidis-
ciplinary situations, it is worth noting
that my background and bias is toward
STEM education in the U.S. Therefore,
INTER ACTIONS. ACM.ORG
 HCI-Sensible
HCI sensibility. HCI is an add-on.
Primary focus on
application and practice in industry.
Trajectory of these
students is industry.
Project
Management
Presenting Facilitating
Design Interaction
Sketching
Critiques Design
PDLC and Visual
SDLC Design
Information User
Architecture Research UX Tools
Content Communicating Storytelling
Strategy UX
Growing UX in Front End
Organizations Development
HCI Knowledge Hubs
Figure 4. Purposed frameworktransparency layers. A possible HCI
discipline knowledge map.
the framework may not adequately
account for non-STEM programs and
programs outside the U.S.
CONCLUSION
To summarize, in order for the HCI
Living Curriculum initiative to be
successful, it must first embrace the fact
that HCI is a complex field. With that in
mind, the curriculum committee could
collaborate with the HCI community
(scholars and practitioners alike) to
make curricular structure options for
the various scenarios at each institution.
Creating the disciplinary transparency
layers is key and will require a crowd-
sourced effort by members of the HCI
community and other disciplines. The
charge of the HCI Living Curriculum
Committee is to create the online
communities by which members may
seamlessly contribute to the crafting
of the layers. Once the layers exist, we
can go about connecting the knowledge
hubs of each layer with the goal of cre-
ating possible co-discipline courses that
can fit into each curricular structure.
The result of such an initiative will be
a healthy set of curricular options that
may serve educators and students while
providing structure that is critical to
fortify the foundation and connectivity
of our field.
I do not take lightly the difficulty in
creating co-discipline courses in institu-
tions that are currently still dominated
by disciplinary fiefdoms and all of the
DOI: 10.1145/3095710 2017 ACM 1072-5520/17/07 $15.00
INTER ACTIONS. ACM.ORG
HCI-Centric
HCI theory, models, and framework. HCI-Sensible HCI-Centric
Primarily focused on theory creation HCI sensibility. HCI is an add-on. HCI theory, models, and framework.
Primary focus on application and practice in industry. Primarily focused on theory creation and
and advancing the field. Trajectory of these students is industry. advancing the field. Trajectory of these students
Trajectory of these students is academia and strategists.
is academia and strategists.
Users Mental
Design as Predictive vs.
HCI Masters Program
Applied Descriptive All credits could be part of the HCI curriculum.
Models
Perception Models
Curricular Structure 1
Info GOMS,
Intro to
Foraging KLM,
HCI
Co-Masters Program
Theory Advanced
Mixed Advanced
Non-HCI Non-HCI
applied co-major
Mobile Courses Only some credits are part of the HCI curriculum. Courses
Design Clarks Cognitive courses courses
Common Activity Work
Ground Theory Theory Analysis Curricular Structure 2
Using
Collaborative CSCW Computational HCI Add on
with Masters
Technologies Research Formalisms Mixed
Non-HCI Courses applied Non-HCI Courses
courses A few credits dedicated to
Ethnography, Design UX gain an HCI sensibility.
Situated Action, Rationale as Strategy in
Ethnomethodology a Theory Organizations
Curricular Structure 3
Set number of credits for the Masters degree.
Figure 5. Purposed frameworkcurricular structures.
associated funding challenges. Nor am Computer Society. A. Sutcliffe and L.
I naive about the challenges of infusing Macauley, eds. Cambridge Univ. Press,
Cambridge, England, 1989, 932.
traditional academic courses with more
5. Hewett, T., Baecker, R., Card, S., Carey,
applied-industry-focus objectives. The
T., Gasen, J., Mantei, M., Perlman, G.,
creation of such courses is critical to Strong, G., and Verplank, W. ACM
the success of my purposed framework SIGCHI Curricula for Human-Computer
and will be addressed in subsequent Interaction. ACM, 1992, 26; http://
articles. As its name suggests, the Living dl.acm.org/citation.cfm?id=2594128.
Curriculum Committee effort will be an 6. Carroll, J.M. Conceptualizing a possible
discipline of human-computer interaction.
ever-evolving project, and the approach
Interacting with Computers 22, 1 (2010),
will need to be reevaluated periodically. 312; http://doi.org/10.1016/j.
I look forward to contributing to this intcom.2009.11.008
noteworthy endeavor. 7. Newell, W.H. and Klein, J.T.
Interdisciplinary studies into the 21st
century. Journal of General Education 45
Endnotes
(1996), 152169; http://www.jstor.org/
1. Churchill, E., Preece, J., and Bowser, A.
journal/jgeneeduc
Developing a living HCI curriculum to
8. Carroll, J.M. Human-computer
support a global community. Proc. of CHI
interaction: Brief intro. In The
14 Extended Abstracts on Human Factors in
Encyclopedia of Human-Computer
Computing Systems. ACM, New York, 2014.
Interaction (2nd. ed.) M. Soegaard and
DOI:10.1145/2559206.2559236
2. While the field referred to in this paper
R.F. Dam, eds., 2013; https://www.
interaction-design.org/literature/book/
is called human-computer interaction
the-encyclopedia-of-human-computer-
(HCI), there are other names by which it
interaction-2nd-ed
is known to institutions and practitioners,
9. Carroll, J.M., ed. HCI Models, Theories and
such as human-centered interaction,
Frameworks: Toward a Multidisciplinary
human-centered design, and interaction
Science. Morgan Kaufmann Publishers,
design (IxD). Additionally, the research
San Francisco, 2003.
and concepts extend to the overarching
umbrella field and/or practice of user
experience (UX). Andrea Peer is an applied social scientist,
3. I acknowledge my co-creators in the user experience researcher, and interactive
ISU HCI curriculum redesign effort: technology designer with a background
Dr. Stephen Gilbert, Dr. Les Miller, Dr. in engineering, software development,
Ana-Paula Correia, Pam Shil, and Hannah psychology, and organizational management.
Deering. Her specific area of research is dedicated to
4. Long, J. and Dowell, J. Conceptions of the examining how organizations can grow their UX
discipline of HCI: Craft, applied science, capacity. Currently she leads the Onboarding
and engineering. In People and Computers experience at UserZoom.
V: Proc. of the Fifth Conference of the British andrea.peer@gmail.com
J U LY A U G U S T 2 0 17 INTER ACTIONS 69
 FORUM UNIVERSAL INTER ACTIONS
In this forum we celebrate research that helps to successfully bring the benefits of computing technologies to children,
older adults, people with disabilities, and other populations that are often ignored in the design of mass-marketed products.
Juan Pablo Hourcade, Editor
The 3Cs for Preschool
Childrens Technology:
Create, Connect, Communicate
A
Juan Pablo Hourcade, Luiza Superti Pantoja, Kyle Diederich, and Liam Crawford, University of Iowa
Glenda Revelle, Family Experience Design
toddler using a
smartphone or a tablet
is not an unusual
sight in high-income
regions. In fact,
todays preschoolers
are the first generation
growing up with interactive computing
devices that are accessible to their
cognitive and motor abilities, with
some starting to use them as babies
[1]. Prior to the wide availability of
handheld touchscreen devices, the
mouse and keyboard posed motor
barriers to a significant portion of
children under the age of four when
using computers. In addition, the
size and lesser mobility of desktops
and laptops limited the practical
use of computers for this age group.
This barrier has now been lifted,
but the research community has had
difficulty keeping up with the rapid
expansion of technology available to
toddlers and preschoolers. Instead,
there are thousands of commercial
apps of varying quality available,
and preschool children are using
them without parents having a clear
understanding of the benefits or
potential risks.
It is too early to tell whether we
should be concerned about this
development. However, some of
the uses of mobile devices by young
children are not a particularly good fit
with the type of activities associated
with healthy development. They often
involve socially isolating activities
structured by apps, with a primary
focus on the device, the experience
of media, and engagement through
instant gratification, with little in the
70 INTER ACTIONS J U LY A U G U S T 2 0 17
way of personalization. The challenge
is that such activities can get in the
way of children experiencing healthy
activities that involve being creative,
connecting with the physical world,
and communicating and engaging with
others face-to-face.
This challenge comes with an
opportunity for our community
to make positive contributions to
childrens healthy development. If
preschool children are going to use
mobile apps, we might as well provide
options that lead to healthy activities.
To take advantage of this opportunity,
inspired by evidence-based curricula
and child-development theories, we
propose a developmentally appropriate
approach to technologies for this age
group focused on the 3Cs: creating,
connecting, and communicating.
More specifically, technologies that
follow the 3Cs approach are supportive
of creative activities that connect
children with their social and physical
environment while emphasizing
communication. Overall, the goal is
to leverage childrens interests to help
them arrive at powerful ideas.
Insights
Technologies following the 3Cs
approach support creative activities
that connect children with their
social and physical environment,
emphasizing communication.
StoryCarnival uses the 3Cs
approach to promote
self-regulation by scaffolding
make-believe play activities.
Consider using the 3Cs approach
to support learning processes.
INSPIRATION FOR
THE 3CS APPROACH
The idea of the 3Cs comes in part
from sociocultural approaches to
child development, in particular,
Lev Vygotskys ideas on the social
nature of learning. Vygotskys views
on development emphasize that
childrens development of skills and
concepts occurs first socially (with
help from others) and then individually
[2]. One area he highlighted was the
importance of make-believe play.
Vygotskys observation was that it
leads to children regulating their
behavior, in particular inhibiting
actions that do not fit the make-believe
context. This is due to children taking
on roles with specific characteristics
that bring about natural constraints
within a motivating activity [2]. This
self-regulation starts with physical
behaviors and is followed by social
behaviors, and then by cognitive
processes such as attention [3].
Additionally, Vygotsky thought
such play can help with the
development of abstract thinking if
children use generic props to represent
different objects symbolically.
According to the demands of role
play, children can use and reuse the
same props for different purposes.
In other words, the props stand for
something they are not, acting as
symbols that represent something
else. These concepts of play were
incorporated into the successful Tools
of the Mind curriculum. Tools of the
Minds make-believe activities also
include play planning by children
and the participation of adults as
facilitators. Tools of the Mind has
INTER ACTIONS. ACM.ORG
 strong evidence backing its approach,
including large studies documenting
childrens improved executive function
skills and academic performance when
compared with those of children in
standard preschool curricula [4].
Tools of the Minds activities have
a clear orientation toward children
creating in a social context, connecting
with their social and physical
environment, and communicating
with each other and adults through
the stories they act out. These
activities empower children, involve
the participation of adults in a
guiding role, and require planning
and the delay of gratification. In
addition, they link childrens strong
interests in stories and characters
with powerful ideas in self-regulation,
such as inhibitory control and task
switching. We have the same vision
for technology-supported learning
ecologies for preschool children.
IMAGE BY T YLER OL SON
Seymour Paperts ideas on learning
also influenced our approach. Papert,
one of the parents of child-computer
interaction, wanted children to be
active, empowered learners who create
INTER ACTIONS. ACM.ORG
items of interest in social settings. He
thought such creative projects could
leverage childrens strong interests to
allow them to arrive at powerful ideas
[5]. In his support for these social,
creative activities, Papert set the tone
for our conceptualization of the 3Cs.
Vygotskys and Paperts views
are also compatible with systems
approaches to child development.
While researchers using systems
approaches tend to focus on low-level
changes in development, the theories
also apply to high-level changes
in skills, such as self-regulation.
In particular, they emphasize the
embodied nature of development,
occurring through bidirectional
interactions between the brain,
the body, and the environment.
As children develop
and learn, the brain,
the body, and
the environment
change together.
As children develop and learn, the
brain, the body, and the environment
change together [6]. The 3Cs approach
takes into account this wider view
of development through activities
that involve social interaction
and interactions with the physical
environment, with the purpose of
developing self-regulation.
To be consistent with these
approaches to child development,
we need to identify and design the
most beneficial social and physical
contexts in which children can
develop. Technology-rich learning
environments will be limited in their
effectiveness for young children if
they do not include activities that
combine the use of technology with
social interactions, communication,
and interactions with the physical
environment. Hence, in the 3Cs
approach we use technology to
scaffold, facilitate, and encourage
creative activities that connect
children with their physical and
social environment, with an emphasis
on communication. This approach
increases the likelihood of beneficial
J U LY A U G U S T 2 0 17 INTER ACTIONS 71
 FORUM UNIVERSAL INTER ACTIONS
Non-3Cs vs. 3Cs technology activity characteristics.
On the left, a non-3Cs technology activity at a preschool. On the right, a 3Cs technology activity
at a preschool.
Character-selection screen in StoryCarnival for a space-themed story.
developmental outcomes.
Then again, we may ask whether
we need the digital realm at all. How
exactly can technology help? We
expect that computer technologies can
provide value in several ways. First,
computers can help bridge abstract
and concrete thinking [5] and thus
be a more accessible link to powerful
ideas (e.g., helping with planning
activities). Second, as very flexible
tools, computers can more easily link
children to their strong interests (e.g.,
through media) and enable them to
72 INTER ACTIONS J U LY A U G U S T 2 0 17
create a wider array of entities that
could be easily shared. Third, they
can provide additional channels of
communication. Fourth, they can
provide scaffolds to beneficial activities,
making it more likely that they will
be adopted and enhancing them by
providing appropriate supports.
CHALLENGES IN DEVELOPING
3CS TECHNOLOGIES
There are several challenges involved
in designing and implementing 3Cs
technologies for preschool children.
These include developing child-
centered design methods for this age
group, integrating technology with
the socio-physical environment,
promoting creativity and
communication, providing engaging
support for adults, and maintaining
childrens engagement in delayed-
gratification activities.
Identifying age-appropriate child-
centered design methods is likely to
be a challenge due to how quickly
childrens abilities change from age 3
to 4. Methods should adapt to a childs
ability to communicate and to their
social and creative skills, and should
also incorporate caregivers. While
it is tempting to rely on parents,
teachers, and child-development
experts to develop designs, we believe
it is important to also include children
in the design process as early and
often as possible.
Integrating technology with
the socio-physical environment
is perhaps the greatest challenge
in designing 3Cs technologies. At
some point during a facilitated
activity, the technology needs to
yield focus to childrens physical
surroundings and to the people there.
This may work in the form of having
technology scaffold activities that do
not involve technology. In fact, 3Cs
apps could be designed specifically
to get children to play without apps.
Another possibility would be for 3Cs
technologies to support connections
with the socio-physical environment
through sensors and actuators
distributed in a physical space.
3Cs technologies should also
promote creative activities and
communication. The challenge is
encouraging children to be authors
and communicators instead of media
consumers. Providing scaffolds
to activities with creative and
communicative aspects can help set the
stage for more independence in both.
The engagement of teachers,
parents, and other caregivers in
3Cs activities is crucial to their
success, as these are not babysitter
technologies. We need to inspire
and motivate these caregivers and
give them easily accessible guidance.
Therefore, we must study how to
design and deliver engaging advice and
support for adults.
INTER ACTIONS. ACM.ORG
 The 3Cs approach also involves
delayed gratification that is gained
through the creative, social activities.
The challenge is to get children engaged
in such activities even though they
may be used to instant gratification
when they use technologies. We expect
that one useful strategy is to design
activities that immerse children in
their strong interests.
STORYCARNIVAL:
A 3CS TECHNOLOGY
For the past few months, we have
been iteratively developing a 3Cs
technology for 3- to 4-year-old
children called StoryCarnival, with
the goal of helping children develop
self-regulation skills. StoryCarnival
takes inspiration from the make-
believe play activities in Tools of the
Mind, and from Paperts example
of Brazils escolas de samba (samba
schools) as matching his vision
of learning in society [5]. Brazils
escolas de samba traditionally
participated in a 3Cs process
involving storytelling, planning,
and performance that engaged the
strong interests of people at various
skill levels to arrive at powerful
ideas, which in turn led to spectacular
Carnival parades.
StoryCarnival helps children and
their caregivers get inspiration for and
plan make-believe play through a Web-
based app. The components we are
developing include:
interactive stories to introduce
themes and set the stage for
collaborative play;
a play planner that helps children
and caregivers select a theme and
roles, and set up the use of the physical
environment (including generic
physical props);
guidance for caregiver
participation; and
support for contextualization and
personalization through the ability to
edit and create new interactive stories.
A typical use of StoryCarnival
would involve a group of children
experiencing an interactive story that
introduces a theme and characters.
This would be followed by use of
the play planner to guide children
through negotiating which character
DOI: 10.1145/3096461 COPYRIGHT HELD BY AUTHORS. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
INTER ACTIONS. ACM.ORG
each of them will play, and how
they will make use of the physical
space and props. Once the make-
believe play starts, the app moves to
the background and children focus
on playing with each other, in the
physical space, using generic props
with no electronics, with guidance
from a caregiver.
In our deployment of prototypes
at a local preschool, we have
observed that StoryCarnival
prompts the type of behaviors
targeted by Tools of the Mind,
including the use of generic props to
represent different objects, children
staying in role, children explicitly
switching roles, and continuing
stories from previous play sessions.
Our hope is that a successful
development of StoryCarnival can
deliver many of the benef its of
curricula such as Tools of the Mind
in a more accessible manner. Similar
approaches could make other
evidence-based activities easier to
implement.
3CS ELSEWHERE?
As we developed the 3Cs concept,
we realized that our participatory
design activities in the design of
StoryCarnival, as well as those used
in many other projects, were 3Cs
activities too. Most user-centered
design activities involve creating
(designs, prototypes); connecting
with users, stakeholders, and their
context; and communicating with
all involved parties. It makes sense
that design processes benef it from
a 3Cs approach, as they are also
learning processes.
With that in mind, should we
apply the 3Cs approach to other
situations? Are there other learning
processes that could benefit from
a 3Cs approach? How about other
technology-driven activities? It is
certainly not only children who look
like they are isolated from their social
and physical environment when
using technologies. While there is
nothing wrong with individually
experiencing a reasonable amount of
very engaging, gratifying media, we
could do better in providing a more
balanced set of experiences with
technologies. Do you have a balance
of non-3Cs and 3Cs experiences when
using technology? How about when
you are not using technology?
As you consider these questions, we
will continue pursuing 3Cs approaches
in the design of technologies for young
children, with the hope that they
grow up to be creative adults with
strong connections to their social
and physical environment and fluid
communication skills.
Endnotes
1. Hourcade, J.P., Mascher, S., Wu, D., and
Pantoja, L. Look, my baby is using an iPad!
An analysis of YouTube videos of infants
and toddlers using tablets. Proc. of the 33rd
Annual ACM Conference on Human Factors
in Computing Systems. ACM, New York,
2015, 19151924.
2. Vygotsky, L.S. Play and its role in the
mental development of the child. Soviet
Psychology 5, 3 (1967), 618.
3. Bodrova, E. and Leong, D.J. Tools of the
Mind. Pearson, 2007.
4. Blair, C. and Raver, C.C. Closing the
achievement gap through modification
of neurocognitive and neuroendocrine
function: Results from a cluster
randomized controlled trial of an
innovative approach to the education of
children in kndergarten. PLoS ONE 9, 11
(2014), e112393; https://doi.org/10.1371/
journal.pone.0112393
5. Papert, S. Mindstorms: Children, Computers,
and Powerful Ideas. Basic Books, 1980.
6. Spencer, J.P., Perone, S., and Buss,
A.T. Twenty years and going strong: A
dynamic systems revolution in motor
and cognitive development. Child
Development Perspectives 5, 4 (2011),
260266; https://doi.org/10.1111/j.1750-
8606.2011.00194.x
Juan Pablo Hourcade is an associate
professor in the University of Iowas
Department of Computer Science.
juanpablo-hourcade@uiowa.edu
Luiza Superti Pantoja is an informatics
Ph.D. student at the University of Iowa.
luiza-supertipantoja@uiowa.edu
Kyle Diederich is a computer science Ph.D.
student at the University of Iowa.
kyle-diederich@uiowa.edu
Liam Crawford is a computer science
undergraduate student at the University of
Iowa.
liam-crawford@uiowa.edu
Glenda Revelle is the president of Family
Experience Design.
revelleg@gmail.com
J U LY A U G U S T 2 0 17 INTER ACTIONS 73
 F O R U M E VA L UAT I O N A N D U S A B I L I T Y
This forum addresses conceptual, methodological, and professional issues that arise
in the UX fields continuing effort to contribute robust information about users to product
planning and design. David Siegel and Susan Dray, Editors

Usability, Tested?

A
Neha Kumar, Georgia Institute of Technology, Naveena Karusala, Georgia Institute of Technology,
Aaditeshwar Seth, Gram Vaani and Indian Institute of Technology, Delhi, Biswajit Patra, Gram Vaani

s a team of HCI
researchers testing
the usability of a
media-sharing mobile
application with rural
Indian users, we
found our notions of
usability to be challenged in unexpected
ways. Our experience highlighted the
need to bring a deep understanding of
sociocultural contexts into conventional
usability evaluations. In this article,
we touch upon the importance of
considering prior technological
encounters, the contribution of peers
to the process of learning how to use
new technologies, and the role of
adaptability in the short term. Taken
together, these reflections offer us an
enhanced understanding of how to
conduct usability evaluations with new
technology adopters in cross-cultural
settings, particularly regions of the so-
called Global South that are still fairly
under-studied in human-computer
interaction (HCI).
Penetration of mobile coverage and
personal computing devices across the
world has grown dramatically, including
in relatively resource-constrained
regions such as parts of the Global
South. With this growth, HCI has also
witnessed the emergence of the domain
of HCI for development (HCI4D).
HCI4D research draws richly on the
diversity of mobile uses and users that
has grown in leaps and bounds, now
that we can design mobile apps for those
using low-cost devices, or even the
slightly more expensive devices in which
many economically disadvantaged
individuals invest. Yet, as these users
begin to engage more extensively with
various mobile devices, the challenges
that arise in the uptake of these
technologies also take on new flavors.
Much of the early and influential
work in the field of HCI4D focused
on text-free user interface (UI) design
to address the limitations of literacy
among these populations. Indrani
Medhi's seminal efforts toward defining
how to create, test, and implement
text-free user interfaces have led to
more concrete design principles as well
as a comprehensive understanding
of the various factors impacting the
use of interfaces by illiterate or low-
literate users. Over time, the HCI4D
community has developed a great
sensitivity to the relevance of using local
languages and dialects when designing
and testing new products and services,
emphasizing images as opposed to text,
and trying to simplify user experiences
as much as possible. Indeed, one might
argue that these practices would better
suit the more literate among us as well.
Other early work has further
characterized technology use in the
HCI4D field, laying out future research
directions. Medhis continuation of
her work has shown that while there
Insights
Usability evaluations conducted in
the Global South must increasingly
turn their attention to:
Users prior technological
encounters
The role of peers in the process
of learning how to use new
technologies
The importance of adaptability
in the short term.
are more barriers to usability than
illiteracy, including lack of confidence
with the technology and inexperience
with procedural instructions, group
interaction with the technology
can overcome complexity. Our first
author Neha Kumars prior work on
Facebook use among Indian youth
has also highlighted the ability of
the users to overcome social and
technological limitations to achieve
their aspirations [1]. Moreover, Nimmi
Rangaswamy [2] and Tom Smyth [3],
among others [4,5], have extensively
studied mobile media consumption
and sharing, pointing to the agency
of users in creating and maintaining
their role in complex systems of media
consumption despite serious barriers.
This body of work sets the stage for a
reflection on the realities of usability
and how it is shaped by the context of
expanding mobile adoption and use.
MOBILE VAANI IVR PLATFORM
Though we thought we were well
grounded in understanding the
kinds of considerations that Medhi
discusses, we encountered new lessons
during the research we conducted
in collaboration with Gram Vaani, a
social enterprise headquartered in
New Delhi (www.gramvaani.org).
Gram Vaani, established in 2009 with
a focus on using simple technologies
and social context to design tools, is
an interactive voice response (IVR)
social media platform targeting social
development. Its efforts have impacted
communities including more than 2.5
million users in several regions, across
15 Indian states. To mention a few
outcomes of their work, 30 rural radio

74 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

 stations across India are now equipped
to manage and share content over
mobile devices and the Web, corrupt
officials in Jharkhand were arrested
due to citizen complaints, and women
leaders in Uttar Pradesh (UP) were able
to share lessons and opinions.
One of Gram Vaanis most impactful
projects has been the Mobile Vaani
(MV) IVR system. Used throughout
multiple states in India, including
Uttar Pradesh, Orissa, Bihar, and
Jharkhand, it offers residents of small
towns and villages the ability to place
a missed call to the MV server, so that
the system can then call users back and
allow them to record and listen to news
categorized by local channels. A team
of moderators checks the recordings
for quality before posting them on
the IVR system for other listeners.
Topics range from job openings and
government accountability to social
issues and folk songs and poems.
Partner organizations of MV can
also post content, often pushing
informational campaigns, addressing
grievances, and collecting feedback
on government schemes. Thus, MVs
growth as an IVR platform was driven
I M A G E B Y G R A M VA A N I
by local volunteers and early adopters
from the community who demonstrated
that IVR was helpful to other users,
assisting with their onboarding (i.e.,
INTER ACTIONS. ACM.ORG
sign-up and registration). Not only did
this entail feature-level training, but
it also explained to prospective users
conceptually how such a platform could
help the community and what kind of
content and services could be useful.
LESSONS FROM THE FIELD
For our study, we designed and tested
the app version of the MV IVR system.
The app has limited functionality at
present, allowing users to listen to each
local channel in the form of a playlist.
Over the course of the summer 2016,
we developed an offline sharing feature
for this app. This feature allows users
to share multiple recordings from a
channel at once with one other MV
app user without using cellular data.
Inputs for the design of this feature
were taken from target users of the
app in the East Singhbum district of
Jharkhand. Our team consisted of
members both relatively new to HCI4D
(Naveena) as well as those with several
years experience in the field (Biswajit,
Aaditeshwar) and HCI4D expertise
in general (Aaditeshwar, Neha). This
combination of backgrounds also drove
a singular focus on situating our design
appropriately in the cultural context, to
the best of our ability.
As we made plans to head to the
field, we were curious also to examine
opportunities for assessing learning
on UIs for low-literate users over a
longer period of time, identified by
Medhi Thies as requiring further
exploration in her recent review of
literature on UI design for low-literate
users [6]. This direction brought up
questions including:
How might we account for
existing familiarity with other, similar
technologies as we test for usability?
What role might peer relations play
in facilitating usability?
How might we rely on users ability
to adapt so that technologies become
and stay usable over time?
We present here our initial
observations that sowed the seeds for
a longer, more in-depth exploration of
these questions.
First, increasing availability of
technologies that are usable and
relevant for a growing rural user base
means that these users no longer come
from a place of not knowing how mobile
apps work. Prior familiarization with
mobile apps is something that has
generally been overlooked in HCI4D
work, and for good reason, since
projects in this field tended also to
represent the users first interactions
with mobile technologies. Not too
surprisingly, we discovered that a
large portion of our participants had
J U LY A U G U S T 2 0 17 INTER ACTIONS 75
CACM_JOCCH_one-third_page_vertical:Layout 1 7/30/09 5:50 PM Page 1

F O R U M E VA LUATION A ND USA BILIT Y

ACM
Journal on
Computing and
Cultural
Heritage

Naveena and Biswajit conduct their usability study.

already adapted to social-networking such a thing as too much localization,

apps like Facebook and used them on a we were forced to ask, as we proceeded
daily basis. Not being fluent in English to test the app. At one point in our
appeared not to be a deterrent. In fact, usability evaluation, we realized that
these apps had successfully familiarized we had left English terms in the UI
our participants with English terms by mistake (while others had been
and phrases. This finding challenged translated to Hindi). This serendipitous
our approach to localization. Was there accident led us to observe that
English terms appeared to facilitate
JOCCH publishes papers of
completion of the task in a way that
significant and lasting value in reflected a level of comfort we werent
all areas relating to the use of ICT expecting. Words like Cancel and Share
in support of Cultural Heritage, are common in apps that the users
already had experience with and were
seeking to combine the best of easier to understand than unfamiliar
computing science with real and imperfect Hindi translations.
attention to any aspect of the Perhaps in the case of localizing
language, concepts like sharing a file
cultural heritage sector.
with another user become difficult
to translate, especially when those
concepts were first understood in
another language. When we asked
participants whether they would
prefer more of these English keywords
www.acm.org/jocch used throughout the app, many of
our participants who frequently used
www.acm.org/subscribe smartphones and mobile apps said
yes. Though localization is essential
to working with different language
fluencies and literacies, language
itself is ever changing. Long-term
exposure to other languages, especially
via technology use, is an important
factor to consider in usability. At the
One of the screens on which we accidentally same time, language and technology
left the English words Cancel and Delete. use can vary among age groups and

76 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

genders. Perhaps then localization is not
completely understood by a single slice
of data within time or a demographic,
but rather by a timeline of language
transformation and exposure among
different demographics. This is a
significant area for research to explore
in order to understand how these
elements impact usability evaluations
and subsequent product uptake
in rural areas in the Global South,
although they are not necessarily
unique to those settings.
Second, when we think of usability
and the user, we frequently view the
user as an individual, engaging with
technology on his/her own. The field of
HCI has increasingly drawn attention
to the importance of the users context,
above even the characteristics of the
individual alone. Indeed, what goes
on around users, we find, can strongly
influence them. We conducted parts of
our usability study in groups, realizing
along the way that the usability of the
app was often determined by the user
who was quickest to learn. Once the
first user learned how to use the app,
he or she gave other users pointers and
suggestions that helped them complete
the task faster. Even the first users
within a group-testing scenario had the
collective suggestions of onlookers to
aid them, although it was everyones
first attempt at using the app. More
than an isolated incident, this is often
how users get the hang of mobile apps in
generalthey are taught by a friend or
relative who introduces them to the app.
Particularly in scenarios that involved
the transfer or exchange of mobile
media, users utilized peer-to-peer
sharing apps that required the sender
and receiver to be located physically
next to each other, pointing to the
social nature of app use and learning.
Of course, usability for a single user is
also needed, considering phones are
personal devices and that there may
not always be someone around to help.
However, when phones are shared
among members of a family, or mobile
app use is a social activity, it is worth
exploring how usability and adaptability
might leverage group dynamics.
The third observation we noted was
that of the learnability of the app or, in
other words, usability over time. What
DOI: 10.1145/3098571 2017 ACM 1072-5520/17/07 $15.00
INTER ACTIONS. ACM.ORG
was not apparent to us at first glance was
that even when participants appeared to
struggle with features in an app that was
novel to them, there was a likelihood
of their learning how to use them over
time. When we asked them how long
it took to learn how to use these apps,
participants said that over the course
of a few days they would become
quite comfortable with the functions
they wanted to use, such as sending
messages or sharing media content.
This learning curve may be indicative
of users agency to adapt, particularly
when there is perceived value to be
derived from using the application,
such as how WhatsApp and Facebook
provided ways to communicate
and share both entertainment- and
business-related content with family
and friends, including those who lived
far away. These observations point to
the importance of the questions that
must be asked in addition to Can users
use this? such as Why would users
want to learn how to use this? and
Will users persist through the learning
process? and Will users continue to
use this??
If we view usability as an attempt to
close the gap between the potential user
and the technology in question, then
we can start to think about not only the
factors that bring the technology closer
to the user, but also those that bring
the user closer to the technology. The
three factors contributing to the latter
point to the importance of both taking
time and place into consideration and
viewing these users as learners with
agency, regardless of whether they are
low on print and digital literacy. We
stress that these challenges around
usability are relevant much more
broadly, beyond the rural Global South.
In fact, our lessons from the field serve
to highlight, perhaps in somewhat
exaggerated form, concerns that might
be relevant for us to consider in other
sociocultural contexts, in other parts of
the world as well.
Endnotes
1. Kumar, N. Facebook for self-empowerment?
A study of Facebook adoption in India. New
Media & Society 16, 7 (2014).
2. Rangaswamy, N., and Cutrell, E.
Anthropology, development and
ICTs: Slums, youth and the mobile
internet in urban India. Proc. of the Fifth
International Conference on Information
and Communication Technologies and
Development. ACM, 2012.
3. Smyth, T.N. et al. Where there's a will
there's a way: Mobile media sharing in
urban India. Proc. of the SIGCHI conference
on Human Factors in Computing Systems.
ACM, 2010.
4. Kumar, N. and Rangaswamy, N. The
mobile media actor-network in urban
India." Proc. of the SIGCHI Conference
on Human Factors in Computing Systems.
ACM, 2013.
5. O'Neill, J. et al. The increasing
sophistication of mobile media sharing
in lower-middle-class Bangalore. Proc.
of the Eighth International Conference
on Information and Communication
Technologies and Development. ACM, 2016.
6. Medhi Thies, I. User interface design
for low-literate and novice users: Past,
present and future. Foundations and
Trends HumanComputer Interaction 8,
1 (2015), 172.
Neha Kumar is an assistant professor at
Georgia Tech, jointly appointed at the schools
of International Affairs and Interactive
Computing. She focuses on human-centered
computing for global development. She
graduated from UC Berkeleys School of
Information and was a postdoc at the University
of Washingtons computer science and
engineering department and the Annenberg
School of Communication at the University of
Southern California.
neha.kumar@cc.gatech.edu
Naveena Karusala is a graduate of
Georgia Techs College of Computing. Her
research on human-centered computing and
development involves work on womens safety
and applications of feminist HCI. Her future
research interests lie at the intersection
of gender and the domains of learning and
healthcare.
nkarusala3@gatech.edu
Aaditeshwar Seth teaches at the Indian
Institute of Technology Delhi, where he
runs the ACT4D (Appropriate Computing
Technologies for Development) research
group. He won the Knight News Challenge
award in 2008 to start Gram Vaani, a social
enterprise that builds low-cost, voice-based
systems for community media in rural areas.
aseth@cse.iitd.ernet.in
Biswajit Patra is an employee at Gram
Vaani, a social technology company based in
Delhi whose objective is to reverse the flow
of information, that is, to make it bottom-up
instead of top-down, and to empower poorly
literate and low-income communities to create
and share their own media.
contact@gramvaani.org
J U LY A U G U S T 2 0 17 INTER ACTIONS 77
 F O R U M D E S I G N A S I N Q U I RY
This forum highlights conversations at the intersection of design methods and social studies of technology. By highlighting
a diversity of perspectives on design interventions and programs, we aim to forge new connections between HCI design and
communication, science and technology studies, and media studies scholarship. Daniela Rosner, Editor

Exploring DIY Tutorials as a Way to

Disseminate Research Through Design

R
Audrey Desjardins, University of Washington, Ron Wakkary, Simon Fraser University and
Eindhoven University of Technology, Will Odom, Simon Fraser University, Henry Lin, Simon Fraser University,
Markus Lorenz Schilling, Emily Carr University of Art + Design

esearch through
design (RtD) often
centers on the making
of thingsartifacts,
systems, services, or
other formsas a
means to construct new
knowledge in the interaction design
and human-computer interaction
(HCI) research communities. While
designing things plays a prominent
role in conducting research, there
is an ongoing discussion around
how insights, knowledge, and, more
broadly, theories emerging from RtD
should be documented, articulated,
and communicated in the HCI and
interaction design communities.
In recent years, annotated
portfolios [1], design workbooks [2],
and pictorials [3] have emerged as
approaches used to communicate
RtD insights through the lingua
franca of design [1]. There is a need
for a multiplicity of ways to better
disseminate and communicate
research insights, and these recent
developments are exciting. In this
article, we report on our explorations
of do-it-yourself (DIY) tutorials
as another approach to document,
communicate, and disseminate
the design details, processes, and
materials of RtD artifacts. An
inherent challenge in this approach
is straddling the ways of knowing
of two communities: DIY and RtD.
Foregrounding this challenge, we
examine two contrasting cases:
the conversion of a cargo van into a
camper van that relates strongly to
DIY communities, and the making
of a counterfactual artifact for
material speculation known as the
table-non-table that relates strongly
to RtD [4]. We highlight our lessons
learned in using DIY tutorials for RtD
dissemination that by the nature of
our approach straddles different ways
of knowing.
DIY TUTORIALS
DIY tutorials are pedagogical in
nature: They aim at concisely and
clearly communicating how to make a
wide range of things. Early print DIY
tutorials were oriented toward helping
everyday people conduct home
improvements and car repairs (with
Popular Mechanics and the Whole
Earth Catalog). DIY tutorials are now
broadly distributed through online
platforms such as Instructables, which
share instructions for topics including
Arduino projects, bread recipes,
handmade wooden toys, laser-cut
decorations, and more. DIY tutorials
traditionally offer information
in sequential steps through a
combination of images (photos and/
or diagrams) and text. Their level of
detail is high enough that someone
reading the tutorial should be able to
Insights
DIY tutorials can document,
communicate, and disseminate
details on the crafting processes,
form, and materials of RtD
artifacts.
Using DIY tutorials to document
and disseminate RtD artifacts
poses interesting challenges
because it straddles two
communities: DIY and RtD.
replicate the project on their own. DIY
tutorials have a long and important
history within DIY communities of
supporting the sharing and exchange of
design knowledge about processes, tools,
materials, and the artifacts themselves.
Considering factors in DIY
tutorials such as a strong focus on
the artifact, attention to detail, and
an aim for clarity of communication,
we wanted to explore this format as
a way to document, communicate,
and disseminate the crafting of
RtD artifacts. We do so with two
contrasting cases, as described below.
CASE 1. AUTOBIOGRAPHICAL
DESIGN PROJECT OF
A DIY VAN CONVERSION
The first case is the conversion of a
Mercedes Sprinter cargo van into
a winterized camper van [5]. This
project is an autobiographical design
research project. By definition, this
means that the project was created
to respond to the genuine needs of
its designers and makers [6]. The
project was created by and for Audrey
Desjardins and her partner Landre
Brub-LeBrun to have a cozy cabin
on wheels for biking and skiing trips.
Since they relied heavily on online
forums and tutorials to convert the
van, they created tutorials for each
of the major steps in the making of
the van as a way to give back to the
DIY community. While this project
was mostly DIY-oriented, Desjardins
and Brub-LeBruns experience
with the van resonated strongly with
current research on smart homes and
the Internet of Things. This prompted
us to use the van as a case to critically

78 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

 reflect on the current trajectory of the
design of technology for the home.
As we started to investigate this
autobiographical design project,
the information that was documented
in the DIY tutorials emerged as
detailed and comprehensive reports on
design process and design rationales,
which became a productive set of data
for our research-oriented analysis of
the project.
Currently, five tutorials about
the van conversion project have been
published (Figure 1). They include
tutorials for insulating the walls,
constructing a storage platform,
finishing the walls with cedar panels,
constructing benches and a table
TOP IMAGE BY KE VIN SIMMONS
Figure 1. Five tutorials for the van conversion project.
INTER ACTIONS. ACM.ORG
that convert into a bed, and making
cushions for the benches and bed [7].
SUPPORTING REFLECTION
IN AUTOBIOGRAPHICAL
DESIGN RESEARCH
THROUGH A DIY COMMUNITY
The van conversion tutorials are
closely related to the DIY community,
and they address a topic that is
often discussed on platforms such
as Instructables. The five tutorials
received significant interest from the
DIY community: Editors featured
them on the homepage of Instructables;
one of them won an Outdoors Project
contest; and they have generated
between 61,000 and 322,000 views to
date. Together, the five tutorials hold
more than 250 questions and answers
representing exchanges between the
authors of the tutorial and people in the
DIY community.
While many of the comments are
words of encouragement, some are
questions specific to why we chose
certain materials or how we used a
certain tool. Many questions also
emerged regarding how our builds
evolved over time, how they held
up to use in practice, and how they
reacted to various weather conditions
during travels. Our responses to
these comments offer a record of
our reflections on the design process
and design decisions. Hence, this
J U LY A U G U S T 2 0 17 INTER ACTIONS 79
 F O R U M D E S I G N A S I N Q U I RY
COMMUNICATIONSAPPS repository of reflections enabled us, as
researchers, to gain an understanding
of not only the making process but also
the van conversion process in its use.
For example, one reader asked us
if there is anything we would have
done differently and what advice we
would give her. Part of our reply was
stating how we currently store our
food and sports equipment under the
platform, but how we might eventually
upgrade to large drawers:
Also, this is something we can still
add and we might: in the back, under the
platform, we could also add really long
heavy duty drawers that slide in and out.
This would make it pretty easy to search
for things and store things away. Just an
idea. Right now, we are using milk crates
(which fit perfect, out of sheer luck!) to
store things and classify them.
In addition to the exchange with
the DIY community, the simple fact of
assembling the tutorials with the DIY

Access the
latest issue,
past issues,
BLOG@CACM,
News, and
more.

Figure 2. Introduction to the table-non-table tutorial.

Available for iPad,

iPhone, and Android

Available for iOS,

Android, and Windows

http://cacm.acm.org/
about-communications/
mobile-apps

Figure 3. Behind the scenes for the table-non-table tutorial.

80 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

community in mind directed the way
we documented our process. The DIY
tutorials were written and assembled
on the Instructables website one to two
weeks after each building stage was
completed. Throughout the process,
we wrote the tutorials by continually
clarifying the underlying reasons
grounding our design decisions.
We hoped that design rationales
would allow readers to go beyond the
detailed step-by-step instructions
and support their own appropriation
of our design process. This strategy
ultimately created valuable insights
when revisiting the tutorials with
researcher eyes. For instance, we
were able to describe with precision the
intimate and reciprocal relationship
between the makers and the van
throughout the building cycles.
The ease with which our tutorials
were received by the DIY community
proved to be highly relevant to
support reflections about the design
decisions we made throughout the
project, which led to a rich and
nuanced perspective on the design
of interactive spaces. However, this
same closeness to DIY practices
became a challenge when writing
for the RtD community with the
intention of publishing in academic,
peer-reviewed venues. In our writings,
we had to make a distinction between
the success of the DIY tutorials for the
DIY community and the success of the
project for the research community. In
fact, the tutorials were presented as a
way to collect data to serve our academic
reflections, rather than as a way to
disseminate the project itself. In the next
case, we explore how a DIY tutorial can
be much closer to the RtD community.
CASE 2. DOCUMENTING
AND DISSEMINATING
AN RTD ARTIFACT:
THE TABLE-NON-TABLE
Our second case is a DIY tutorial to
make the table-non-table [8] (Figure
2). This project contrasts with the van
conversion project since it is much
closer to the RtD community. The
table-non-table comprises a thousand
sheets of white paper stacked on an
aluminum chassis that moves in very
short durations (520 seconds) a
few times a day. The table-non-table
is a counterfactual artifact designed
INTER ACTIONS. ACM.ORG
counter to the logic of normative
design, yet it fits with everyday
contexts as a means to inquire through
design in an RtD approach known as
material speculation [4]. The inquiry
was aimed at the human-technology
relations that form through the
resourcefulness and acceptance of
everyday objects and everyday living.
We have previously written about
the ways in which people live with
and engage with the table-non-table
[9]; however, we offered only very
brief descriptions of how it was
designed and built. To deepen our
documentation of this collaborative
RtD project, we created a DIY tutorial
describing how to make a table-non-
table on the Instructables platform.
DOCUMENTATION,
DISSEMINATION,
AND PERPLEXITY
While the van conversion tutorials
were community-oriented and
eventually served as a tool for
reflection, the DIY tutorial for the
table-non-table was upfront a tool for
documentation and dissemination that
caused a certain level of perplexity in
the DIY community.
The table-non-table is a long-
term project involving a large team
with various skills and expertise;
the tutorial proved useful in helping
organize all the data concerning the
crafting of the research artifact. One
person was in charge of the 3D CAD
modeling for the chassis, another
worked on electronics design and
assembly, and yet another developed
the software for the table-non-table.
The files were distributed across
various computers and in different
locations. While we did have an online
file repository, the folder was not as
up-to-date, detailed, and organized as
we would have liked. It was also not
clear what files should be used if we
planned to make another table-non- Figure 4. Uniformity and quality of image
throughout the tutorial.
table. Putting the tutorial together
allowed our team to clearly identify
the final files for each aspect of the As an RtD studio, we also saw the
build, and furthermore to give clear opportunity to explore a clear and refined
instructions for how to use them. visual language in our photos. To create
Even before the publication of the DIY the tutorial, we disassembled the table-
tutorial, this documentation process non-table, set up a photo booth with
was the most in-depth and detailed appropriate lighting, and took a series
our design studio had produced for an of photos to demonstrate the assembly
RtD artifact. of the table-non-table (Figure 3).
J U LY A U G U S T 2 0 17 INTER ACTIONS 81
 F O R U M D E S I G N A S I N Q U I RY
The result is a highly polished photo set
(Figure 4). These photos stand out on
the Instructables platform, where the
majority of photos are taken on the go,
while projects are being built.
This distinction is interesting to reflect
on as it highlights the contrast
between the two communities
with respect to disseminating and
communicating information.
The table-non-table tutorial
generated a strikingly different
set of comments from the DIY
community when compared with
the van conversion tutorials. Since
it was published in January 2016,
the tutorial received comments that
often expressed confusion, surprise,
and misunderstanding. For example,
comments included: Interesting,
but I cant seem to think of anything
to use this for. Its way too short to
be used as any sort of table, Im
not sure what to ask. Cool though
lol, and Wow, Im seriously
confused. These comments reveal
a real challenge for disseminating
the crafting of an RtD artifact
especially a counterfactual artifact
on a platform meant for DIY
enthusiasts. We realize that the table-
non-table as an object on its own does
not showcase a clear enough purpose
and becomes difficult to make sense
of when presented in a context other
than research through design.
LOOKING FORWARD:
FINDING BALANCE BETWEEN
WAYS OF KNOWING
The contrasting examples of the
van conversion tutorials and the
table-non-table tutorial revealed the
challenge of sharing RtD artifacts to
a multiplicity of audiences: the DIY
community and the RtD community.
This sparks the first question for us:
What would be the qualities needed
for a sharing platform for the crafting
of RtD artifacts among a design or
research community separate from the
DIY community?
However, keeping the RtD DIY
tutorials on the same platform for both
DIY enthusiasts and RtD researchers
provides a valuable opportunity for
new kinds of discussions and dialogues
that led to research insights in the
DOI: 10.1145/3098319 COPYRIGHT HELD BY AUTHORS. PUBLICATION RIGHTS LICENSED TO ACM. $15.00
82 INTER ACTIONS J U LY A U G U S T 2 0 17
van conversion case. For instance,
both researchers and amateurs
could use the tutorials to re-create
or remix RtD artifacts. This raises
several interesting future research
questions. What kinds of experiences
and insights could be catalyzed
through a DIY enthusiast making and
living with a design artifact like the
table-non-table? What other kinds
of projects could the table-non-table
inspire DIY enthusiasts to make?
To what extent would they differ from
projects made by RtD researchers?
What kind of insights can we, as RtD
researchers, gain from this crossover
to a DIY community?
These questions represent intriguing
new ways in which DIY tutorials could
be mobilized as we continue to seek
out new ways of disseminating
RtD artifacts to audiences in and
beyond the HCI and interaction
design communities.
Endnotes
1. Bowers, J. The logic of annotated
portfolios: Communicating the value of
research through design. Proc. of the
Designing Interactive Systems Conference.
ACM, New York, 2012, 6877; https://doi.
org/10.1145/2317956.2317968
2. Gaver, W. Making spaces: How
design workbooks work. Proc. of the
SIGCHI Conference on Human Factors
in Computing Systems. ACM, New
York, 2011, 15511560; https://doi.
org/10.1145/1978942.1979169
3. Blevis, E., Hauser, S., and Odom, W.
Sharing the hidden treasure in pictorials.
Interactions 22, 3 (2015), 3243;
https://doi.org/10.1145/2755534
4. Wakkary, R., Odom, W., Hauser, S., Hertz,
G., and Lin, H. Material speculation:
Actual artifacts for critical inquiry. Proc.
of the 5th Decennial Aarhus Conference on
Critical Alternatives. Aarhus Univ. Press,
2015, 97108; http://dx.doi.org/10.7146/
aahcc.v1i1.21299
5. Desjardins, A. and Wakkary, R. Living
in a prototype: A reconfigured space.
Proc. of the SIGCHI Conference on Human
Factors in Computing Systems. ACM, New
York, 2016, 52745285; https://doi.
org/10.1145/2858036.2858261
6. Neustaedter, C. and Sengers, P.
Autobiographical design in HCI research:
Designing and learning through
use-it-yourself. Proc. of the Designing
Interactive Systems Conference. ACM,
New York, 2012, 514523; https://doi.
org/10.1145/2317956.2318034
7. https://www.instructables.com/id/How-
to-insulate-a-camper-van/
https://www.instructables.com/id/
Storage-platform-for-the-back-of-your-
Sprinter-van/
https://www.instructables.com/id/Cedar-
paneling-for-van-interior/
https://www.instructables.com/id/Bed-
Table-and-Benches-for-camper-van-All-
in-one/
https://www.instructables.com/id/How-
to-sew-cushions-for-a-camper-van/
8. https://www.instructables.com/id/How-
to-Make-a-Table-non-table/
9. Wakkary, R., Desjardins, A., and
Hauser, S. Unselfconscious interaction:
A conceptual construct. Interacting with
Computers 28, 4 (2015), 501520; https://
doi.org/10.1093/iwc/iwv018
Audrey Desjardins is an assistant
professor in the School of Art + Art History +
Design at the University of Washington. As a
design researcher, her work focuses on the
making of home, the design of technologies
for the home, do-it-yourself practices, and
methods in research through design.
adesjard@uw.edu
Ron Wakkary is a professor in the School
of Interactive Arts and Technology (SIAT) at
Simon Fraser University and professor and
chair of the Impact of Interaction Design on
Everyday Life in the Department of Industrial
Design at Eindhoven University of Technology.
His research investigates the changing nature
of human-technology relations through design
research in everyday living.
rwakkary@sfu.ca
William Odom is an assistant professor in
the School of Interactive Arts and Technology
(SIAT) at Simon Fraser University in Vancouver,
Canada. He leads projects themed within slow
interaction design, the growing presence
of digital data in everyday life, and methods
aimed at developing the practice of research
through design.
wodom@sfu.ca
Henry Lin is a master of arts student in
the Everyday Design Studio at Simon Fraser
University. His research explores and develops
electronics and fabrication techniques for
both the Internet of Things and the notion
of a research product. He also has an
undergraduate degree in interaction design
from Simon Fraser University.
hwlin@sfu.ca
Markus Lorenz Schilling recently finished
his M.A. in interaction design research at
Simon Fraser University and is currently
working as a sessional instructor at Emily Carr
University in Vancouver, BC. He is interested in
projects that positively shape our relationship
with technology or that involve the creative use
of code and making.
markusschilling@ecuad.ca
INTER ACTIONS. ACM.ORG
Agda is an advanced programming language based on Type Theory. Agdas type system is expressive
enough Agda
to support full functional verification of programs, in two styles. In external verification, we write
is an advanced programming language based on Type Theory. Agdas type system is expressive
pure functional
enough toprograms
support fulland then write
functional proofsofofprograms,
verification properties about
in two them.
styles. The proofs
In external are separate
verification, we writeexternal
artifacts,pure
typically using
functional structural
programs induction.
and then In internal
write proofs verification,
of properties weThe
about them. specify
proofsproperties
are separateofexternal
programs
throughartifacts,
rich types for the
typically programs
using structuralthemselves.
induction. InThis often
internal necessitates
verification, including
we specify proofs
properties inside code, to show
of programs
through rich types for the programs themselves. This often necessitates including
the type checker that the specified properties hold. The power to prove properties of programs proofs inside code, in
to show
these two
the type checker that the specified properties hold. The power to prove properties of
styles is a profound addition to the practice of programming, giving programmers the power to guarantee programs in these two the
styles is a profound addition to the practice of programming, giving programmers the power to guarantee the
absence of bugs, and thus improve the quality of software more than previously possible.
absence of bugs, and thus improve the quality of software more than previously possible.

Verified Verified
Functional Programming
Functional in Agda
Programming in Agdais isthe
thefirst
firstbook toprovide
book to providea systematic
a systematic exposition
exposition of external
of external and and
internalinternal
verification in Agda,
verification suitable
in Agda, forfor
suitable undergraduate studentsofofComputer
undergraduate students Computer Science.
Science. No familiarity
No familiarity with with
functional programming or computer-checked proofs
functional programming or computer-checked proofs is presupposed. is presupposed.

The book begins with an introduction to functional programming through familiar examples like booleans,
The book begins with an introduction to functional programming through familiar examples like booleans,
natural numbers, and lists, and techniques for external verification. Internal verification is considered
natural numbers,
through theand lists, of
examples and techniques
vectors, for external
binary search verification.
trees, and Braun trees.Internal verification
More advanced materialisonconsidered
type-level
throughcomputation,
the examples of vectors,
explicit binary
reasoning aboutsearch trees,and
termination, andnormalization
Braun trees.byMore advanced
evaluation is alsomaterial
included. on
Thetype-level
computation, explicit reasoning about termination, and normalization by
book also includes a medium-sized case study on Huffman encoding and decoding. evaluation is also included. The
book also includes a medium-sized case study on Huffman encoding and decoding.
Text Data Management and Analysis covers the major concepts, techniques, and ideas in
Text Data Management and Analysis covers the major concepts, techniques, and ideas in
information retrieval and text data mining. It focuses on the practical viewpoint and includes
information retrieval and text data mining. It focuses on the practical viewpoint and includes
many many
hands-on exercises
hands-on designed
exercises designed with
witha acompanion softwaretoolkit
companion software toolkit (i.e.,
(i.e., MeTA)MeTA) to help
to help readers
readers
learn how
learnto
howapply techniques
to apply of information
techniques of informationretrieval andtext
retrieval and textmining
mining to to real-world
real-world text data.
text data. It It
also shows readers
also shows howhow
readers to experiment
to experimentwithwithandandimprove someofofthethe
improve some algorithms
algorithms for interesting
for interesting
application tasks.tasks.
application The The
book cancan
book be be usedasasa atext
used text for
for computer science
computer science undergraduates
undergraduates and graduates,
and graduates,
library and information scientists, or as a reference for practitioners working on
library and information scientists, or as a reference for practitioners working on relevant problemsrelevant problems in in
managing and analyzing
managing and analyzing text data. text data.
 C O M M U N I T Y S Q UA R E
Anicia Peters and
Heike Winschiers-Theophilus,
Namibia University of Science and Technology
HCI out of Namibia
I
n the beginning of 2016, we formed
the Windhoek ACM SIGCHI
chapter in Namibia. The chapter
comprises academics, students,
and professionals from across
Namibia. Forming a local chapter
was a natural step for us, especially
at the Namibia University of Science
and Technology (NUST, formerly
known as Polytechnic of Namibia),
but also for other researchers and
students, for instance, at the University
of Namibia. HCI had already played a
central role in almost every computer
science students life in past years, and
it was embedded in most of our research
clusters. Students were exposed to HCI
courses, and research projects have a
strong HCI component.
HCI is embedded in major research-
focus areas of the Computing and
Informatics Faculty at NUST, such
as the digitization of indigenous
knowledge. In collaboration with
indigenous rural communities, HCI
principles and methods have been re-
appropriated to local epistemologies.
Another strong area of HCI application
has been our community and research
initiatives with marginalized
communities, such as the San
(Bushmen) people or unemployed
youth in informal settlements. Further
interesting work in embedding
HCI includes the cyber security,
e-participation, big data, and culture
research clusters.
In October 2016, one of our research
clusters held an International Culture
and Computer Science Conference in
DOI: 10.1145/3099120 COPYRIGHT HELD BY AUTHORS
INTER ACTIONS. ACM.ORG
Tuomo Kujala (Ed.),
ACM SIGCHI
cooperation with ACM SIGCHI. The
conference not only focused on virtual/
augmented reality and multimedia
technologies for creative industries
and cultural establishments like
museums, but it also showcased work in
cross-cultural and indigenous cultural
designs, such as interactive musical
boxes. The conference attracted 108
participants from 13 different countries,
including more than 30 local and
international students.
The chapter members have also
been actively involved in integrating
select HCI methods into their
hackathons and mobile-application-
development training. Students have
been exposed to personas, evaluations,
user testing, informant sessions, paper
prototyping, and so on. Students
learned principles of the user-centered
design approach, even though they
were not explicitly explained. An
upcoming stream of research relying
heavily on HCI is on the issue of
gender-based violence (GBV), where
very few tech solutions currently exist.
This is a sensitive and stigmatized
area of research in which traditional
methods of data collection fail; thus
the methods have to be dynamic. For
example, a traditional African hut
is used to house an interactive role-
playing application combined with
physical props to gather data about
GBV. This data in turn informs and
provides new directions for a social
media campaign against GBV.
AfriCHI 16 was another initiative
in which the chapter members were
involved in organizing and attending
the conference in Nairobi in November
2016. AfriCHI, hosted in cooperation
with ACM SIGCHI, was the inaugural
Africa Human-Computer Interaction
conference that sought to bring
HCI researchers and professionals
together across Africa. More than 200
participants attended from 21 different
countries, including 13 in Africa. The
conference offered workshops, courses,
papers, panels, posters, a Design
Challenge and an Africa Women in
Computing lunch. The conference also
hosted the first Summit on Community
Networks in Africa by the Internet
Society. Remote participants could
tune into the conference virtually.
The conference was organized by 43
committee members, either based in,
from, or interested in Africa under the
theme of Kujenga madaraja, kubomoa
vizuizi, which means building bridges,
breaking barriers. The contributions
were published in the ACM Digital
Library.
Anicia Peters is dean of the Faculty
of Computing and Informatics at Namibia
University of Science and Technology. She
chairs the Windhoek SIGCHI chapter.
apeters@nust.na
Heike Winschiers-Theophilus is a
professor in computer science in the Faculty
of Computing and Informatics at Namibia
University of Science and Technology. She is
the vice chair of the chapter.
hwinschiers@nust.na
Tuomo Kujala is ACM SIGCHI vice president
for local chapters.
sigchi-vp-chapters@acm.org
J U LY A U G U S T 2 0 17 INTER ACTIONS 85
This book is the first full-length biography of Edmund Berkeley, a computer pioneer and social activist who
This
has been book the
called is theconscience
first full-length biography
of the computerof Edmund Berkeley,
industry. a computer
Through pioneer
his work withand social
other activist
early who
computer
has been called the conscience of the computer industry. Through his work with other early computer
developers, he became aware of the potential dangers of these machines to society at large. He believed that
developers, he became aware of the potential dangers of these machines to society at large. He believed that
computer developers
computer had had
developers an obligation totoaddress
an obligation addressthe
the complex socialproblems
complex social problems facing
facing a Cold
a Cold War world;
War world; the the
threat of suicidal
threat nuclear
of suicidal warwar
nuclear andand
thethe
ethics
ethicsofofcomputer professionalsusing
computer professionals using their
their expertise
expertise to build
to build self- self-
guidedguided
weapons systems.
weapons systems.
This is This is an historical
an historical narrative
narrative of aofman
a man ultimatelyin
ultimately infavor
favor of
ofengineering
engineering peace, instead
peace, of war,
instead and how
of war, and how
his career was ultimately damaged by politicians determined to portray him as a Communist
his career was ultimately damaged by politicians determined to portray him as a Communist sympathizer. sympathizer.
Berkeleys
Berkeleys life work
life work provides
provides a lens
a lens to to understandsocial
understand social and
andpolitical
politicalissues surrounding
issues surroundingthe early
the early
development of electronic computers which ties directly to current debates about the use of autonomous
development of electronic computers which ties directly to current debates about the use of autonomous
intelligent systems.
intelligent systems.
This entertaining biography offers a humanistic approach to understanding technology and society via the
This entertaining
successes andbiography offers aBerkeley,
trials of Edmund humanistic approach
a founding memberto understanding technology
of the Association for Computingand Machinery
society via the
successes and Telling
(ACM). trials ofBerkeleys
Edmund Berkeley,
story providesa afounding member
more nuanced of the Association
and dimensional picture offor Computing
how the computersMachinery
we
use today came into being and why we ask the questions we do about our relationships
(ACM). Telling Berkeleys story provides a more nuanced and dimensional picture of how the computers wewith them. It explores
wayscame
use today in which
intoBerkeleys
being and lifewhy
illuminates
we ask issues we still deal
the questions wewith regarding
do about ourthe social responsibilities
relationships with them. of It explores
computer developers and human-computer relationships.
ways in which Berkeleys life illuminates issues we still deal with regarding the social responsibilities of
computer developers and human-computer relationships.
 CALENDAR

July October 2017

July RecSys 17 11th ACM Conference on

HCI 2017 31st British Human
Computer Interaction Conference
(Sunderland, UK)
Conference Dates: July 36, 2017
http://hci2017.bcs.org
UMAP 17 User Modeling, Adaptation
and Personalization Conference
(Bratislava, Slovakia)
Conference Dates: July 912, 2017
http://www.um.org/umap2017
SIGGRAPH 17 44th SIGGRAPH
Conference on Computer Graphics
and Interactive Techniques
(Los Angeles, California, USA)
Conference Dates: July 30August 3, 2017
http://s2017.siggraph.org
August
FDG 17 International Conference on
the Foundations of Digital Games 2017
(Cape Cod, MA, USA)
Conference Dates: August 1417, 2017
http://fdg2017.org
Recommender Systems (Como, Italy)
Conference Dates: August 2731, 2017
http://recsys.acm.org/recsys17
September
MobileHCI 17 19th International
Conference on Human-Computer
Interaction with Mobile Devices and
Services (Vienna, Austria)
Conference Dates: September 47, 2017
http://mobilehci.acm.org/2017/
Ubicomp 17 2017 ACM International
Joint Conference on Pervasive
and Ubiquitous Computing
(Maui, Hawaii, USA)
Conference Dates:
September 1115, 2017
http://ubicomp.org/ubicomp2017
AutomotiveUI 17 9th International
Conference on Automotive User
Interfaces and Interactive Vehicular
Applications (Oldenberg, Germany)
Conference Dates:
September 2427, 2017
http://www.auto-ui.org/17/
October
CHIPLAY 17 Annual Symposium on
Computer-Human Interaction in Play
(Amsterdam, Netherlands)
Conference Dates: October 1518, 2017
http://chiplay.acm.org
SUI 17 Symposium on Spatial User
Interaction (Brighton, UK)
Conference Dates: October 1617, 2017
http://www.sui2017.org
ISS 17 Interactive Surfaces and
Spaces (fomerly ITS) (Brighton, UK)
Conference Dates: October 1720, 2017
http://iss.acm.org/2017
UIST 17 30th Annual ACM
Symposium on User Interface
Software and Technology
(Quebec, Canada)
Conference Dates: October 2225, 2017
http://uist.acm.org

INTERACT 17 16th IFIP TC 13

International Conference
on Human-Computer Interaction
(Mumbai, India)
Conference Dates:
September 2529, 2017
http://interact2017.org

DOI: 10.1145/3099563 2017 ACM 1072-5520/17/07 $15.00

INTER ACTIONS. ACM.ORG J U LY A U G U S T 2 0 17 INTER ACTIONS 87

 VISUAL
THINKING
GALLERY

Woman (Seated)
with Tablet Computer
C
 ontributor: Eli Blevis
Curator/Editor: Eli Blevis
Genre: Photographic minimalism, truth in (digital) photography
Publication: Blevis, E. Pictorial: Qualities of focus.
Proc. of Creativity & Cognition 2017. ACM Press (In press).

Is this deliberately out-of-focus image, recorded in the moment using a manual focus digital camera,
more truthful than a sharply focused one edited using a postproduction tool such as Photoshop?
Or does it tell a different truth that would otherwise remain hiddenthat in our digital lives our
screens can appear to loom larger than our bodies and dominate our physical presence?
DOI: 10.1145/3098887 2017 ACM 1072-5520/17/07 $15.00

88 INTER ACTIONS J U LY A U G U S T 2 0 17 INTER ACTIONS. ACM.ORG

ACM Books.
In-depth.
Innovative.
Insightful.
ACM and Morgan & Claypool Publishers present ACM Books: an all-new series
of educational , research and reference works for the computing
community. Inspired by the need for high-quality computer
science publishing at the graduate, faculty and
professional levels, ACM Books is affordable,
current, and comprehensive in scope.
ACM Books collections are available
under an ownership model with
archival rights included. We
invite you to learn more about
this exciting new program.
For more info please visit
http://books.acm.org
or contact ACM at
ACMbooks-Info@acm.org

M
Association for
Computing Machinery

&C
2 Penn Plaza, Suite 701
New York, NY 10121-0701, USA
Phone: +1-212-626-0658
Email: acmbooks-info@acm.org

M
Morgan & Claypool
Publishers

&C
1210 Fifth Avenue, Suite 250
San Rafael, CA 94901, USA
Phone: +1-415-462-0004
Email: info@morganclaypool.com
 Connect with our
Community of Reviewers

I like CR because it covers the full

spectrum of computing research, beyond the
comfort zone of ones specialty. I always
look forward to the next Editors Pick to get
a new perspective.
- Alessandro Berni
ThinkLoud

www.computingreviews.com