V

a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
A novel interface system for
seamlessly integrating human-robot
cooperative activities in space
Nilanjan Sarkar
Mechanical Engineering
Electrical Engineering and Computer Science
Craig A. Smith
Psychology and Human Development
Vanderbilt University
Nashville, TN
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Inspiration
Exploration of space and solar system
will be most effective if human
capabilities are synergistically combined
with those of robots. Such a human-robot
system, developed correctly, will reduce
exploration risks, improve efficiency, and
achieve overall mission goals faster and
in a better manner.
[Objective of the ICASE/USRA/LaRC Workshop on Revolutionary
Aerospace Systems Concepts for Human/Robotic Exploration of the Solar
System, Nov. 2001]
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Major Obstacle
Natural interaction between human and robot
Is implicit communication possible?
- e.g., robot brings the right tool to the astronaut
without being explicitly commanded
Can the robot understand the psychological states of
the human?
- e.g., robot rushes to help the human if the robot senses
panic
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Novel Approach
affect detection and recognition
brainwave monitoring and characterization
design of control architecture for implicit
communication
integrates research in signal processing,
wearable computing, experimental
psychology and control theory
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Hypotheses
A robot will:
implicitly understand a task command
from an astronaut
sense the psychological state of the
astronaut and take necessary actions
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Rationale
Initially, the research will be:
person specific
context specific
Afterwards, with enough understanding and
data, an affect recognizer for a class of
people will be attempted
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Phase I Tasks
The following tasks were proposed:
A. Develop Affect Recognizer
1. Design human subject experiments to elicit target
affective states (e.g., engagement, anxiety, fatigue,
etc.)
2. Assess physiological indices that are important for the
target affective states
3. Conduct pilot studies
4. Data analysis and signal processing for affect detection
B. Investigate the currently available brainwave
monitoring technologies
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Phase I Tasks
The following additional tasks were
performed:
Developed a functional human-robot system
that is affect-sensitive, and
Conducted preliminary brainwave monitoring
experiments with EEG
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Development of
Affect Recognizer
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Experimental Tasks
Three Problem-Solving Tasks
(Each lasting ~1 hour)
Anagrams
Math Word Problems
Sound Discrimination
Two Sequences for Each Task
1. Easy: Fairly Easy --> Trivially Easy
2. Difficult: Fairly Easy --> Virtually Impossible
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Sample Anagrams
Easy Condition Difficult Condition
AWADR IYTED
DEITY
AWARD
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Easy Math Problem
An astronaut requires 2 pounds of oxygen per
day while in space. How many pounds of
oxygen are needed for a team of 3 astronauts
who are going to spend 5 days in space?
30 Answer:
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Difficult Math Problem
Tammy has $9.70 in nickels, dimes, and
quarters. The number of nickels is 4 more than
3 times the number of dimes, and the number
of quarters is 5 fewer than 2 times the number
of nickels. How many nickels does Tammy
have?
19 Answer:
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Sound Task
Sequence of three tones
Judge whether first and third tones are the
same or different
Difficulty manipulated by varying tone length
and frequency difference between tones
Easy Trials Difficult Trials
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Physiological Data Collection
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Physiological Measures
EKG
interbeat interval (IBI)
mean
variability
sympathetic power *
parasympathetic power *
Finger Pulse Amplitude
mean
variability
Pulse Transit Time (PTT)
Digit Skin Temperature
mean
slope of change
Skin Conductance
Tonic
mean
slope of change
Phasic
response rate *
average amplitude *
maximum amplitude
Facial Muscle Activity (EMG)
Brow (Corrugator)
mean
variability *
Jaw (Masseter)
mean *
variability
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Self Report Measures
Sampled:
Two minutes into each
task
Every seven minutes
thereafter
Assessed:
Task Difficulty
Perceived Ability
Affective States
Key Affective
Parameters:
Anxiety Index
Anxiety
Overload
Engagement Index
Task Importance
Hope
Challenge
Interest
(R) Resignation
(R) Apathy
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Correlations of Anxiety
with Physiology
-0.5
-0.3
-0.1
0.1
0.3
0.5
Symp Paras SCR-
Rate
SCR-
Amp
Corr-
Var
Mass
Physiological Parameter
Subject 2
Subject 4
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Correlations with Physiology:
Anxiety vs. Engagement (Subject 2)
-0.5
-0.3
-0.1
0.1
0.3
0.5
0.7
Symp Paras SCR-
Rate
SCR-
Amp
Corr-
Var
Mass
Physiological Parameter
Anxiety
Engagement
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Schematic of Fuzzy Logic
Analyzer
Sympathetic
Power
Parasympathetic
Power
Skin Conductance
Response Rate
Skin Conductance
Response Amplitude
Corrugator
Variability
Masseter
Average
Fuzzy Logic
Analysis
Affect
Diagnostic
Output
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Development of a
Functional Human-Robot
Interactive System
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Our Robot
Operates in three modes:
1. Explore -- Wanders
about environment
2. Survival -- Uses sonar to
detect and avoid obstacles
in environment
3. Affective -- Responds to
affective signals from
human operator
If robot senses high
anxiety, it uses light
detection algorithm to go
to operator
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Robot Control Architecture
A hybrid subsumption control paradigm
Emergency Stop
Reverse
Affect Layer
Obstacle avoidance
Wall Follow
Wandering
S
S
S
S
S
Sensor
Data
Survival Mode
Deliberative Mode
Reactive Mode
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Human-Robot Interaction Scenario
Human operator works at computer task while the
robot explores the environment and monitors
operators affective state
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
The Functioning System
QuickTime and a DV/DVCPRO - NTSC decompressor are needed to see this picture.
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Physiological Signals & Fuzzy Logic
Output for Moderate Anxiety Trigger
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Timing Diagram
from Robot Experiment
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Investigation of Brain Wave
Monitoring Technologies
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Technologies to Assess Brain Activity
EEG - Measures electric currents generated
by the brain at the scalp
MEG - Measures magnetic fields generated
by the brain
PET - Measures emissions from
radioactively labeled chemicals, monitors
metabolic rates and blood flow
fMRI - Measures oxygen concentration of
blood, correlates to blood flow
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Modality Comparison

EEG MEG fMRI PET
Spatial
Resolution
7-16 mm 3-11 mm 1 mm 5 mm
Temporal
Resolution
Milliseconds Milliseconds
1 second to
minutes
45 seconds to
minutes
Cost $15-120K
$500k (37
channel)
$300-400k
Shielding
Millions Millions
Wearability Yes No No No

V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Example EEG and MEG Systems
http://www.xltek.com/xlwebv2/products/eeg/xlamb.htm
http://www.vsmmedtech.com/MEG_Technical/MEG_Images_275_151.asp
http://www.biomag.helsinki.fi/meg.html
http://www.elixa.com/mental/MindSet.htm
http://www.emedicine.com/neuro/topic445.htm
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
EEG Experiment: Cognitive Load
0
500
1000
1500
2000
2500
3000
3500
4000
4500
A
l
p
h
a

P
o
w
e
r
Eyes Closed Eyes Open
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
EEG Experiment:
Math Problem-Solving
0
100
200
300
400
500
600
700
800
900
A
l
p
h
a

P
o
w
e
r
Eyes Open Easy Moderate Difficult
Problem Difficulty
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Summary
performed every proposed task for Phase I
performed two additional tasks beyond what was
proposed
were successful in eliciting and detecting the target
affective states under controlled situations
developed a functional human-robot system to
demonstrate the feasibility of the central concept
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Conclusions
Phase I work demonstrates that:
implicit human-robot communication is
feasible
can detect human affect on-line and in the
context of realistic tasks
control system can be made affect-sensitive
brainwave monitoring will likely supplement
peripheral physiology in affect detection and
implicit communication
V
a
n
d
e
r
b
i
l
t

U
n
i
v
e
r
s
i
t
y
Robotics and Autonomous
Systems Lab (RASL)
Future Work
Expand range of tasks and contexts to which
framework can be applied
Increase the reliability and sophistication of affect
recognition
Increase range of affects detected and discriminated
beyond anxiety and engagement to include frustration,
fatigue, boredom, etc.
Advance analytical tools for extracting relevant information
from physiological signals
Increase degree to which physiological recording is
ambulatory
Further explore utility of EEG recording to improve
affect recognition and implicit communication
Formalization of affect-sensitive control system