Dissertation Proposal

Combining Tactile and Kinesthetic Information for

Improvements in Human and Machine Haptic Systems

Joseph Romano
Advisor: Katherine Kuckenbecker

1/
 Presentation Overview
Introduction
● Tactile and kinesthetic information

Contributions to haptic display technology

● Completed research

● Planned research

Contributions to autonomous robotics

● Completed research

● Planned research

Conclusion
● Accomplishments and future roadmap

2/
 Proposed Thesis Contributions

Haptic Systems: Autonomous Systems:

Tested and proven electrical and
● ●Human-inspired algorithms for distilling
mechanical designs capable of useful information from raw sensory
producing high-fidelity haptic feedback data

●Algorithms for generating tactile ●A tactile-event-driven framework for

feedback signals, particularly texture directing robust robot task-completion
effects, based on kinesthetic actions
of the user

●An open-source framework to allow ●Open-source tools to enable tactile-event-

users to easily integrate these advances driven robot task-completion, and a set of
in haptic rendering with traditional tools to allow rapid development of new
approaches tasks.

3/
 Human Tactile Senses
Johansson and Flanagan
Nature Reviews Neuroscience

4/
 Tactile and Kinesthetic Integration

●Sensory Discrete Event

Control (SDEC).
Johansson, 1989.

●Inverse Intermal Model.

Wolpert & Ghahramani,
2000

Wolpert & Ghahramani

Computational Principles of Movement in Neuroscience

5/
Improving Haptic Devices

6/
 Improving Haptic Displays

Human Perception During

Real-World Interaction

Human Perception During

Virtual Interaction

7/
 Completed Research in
Haptic Display Technology

8/
Objective: Create virtual textures that feel the same as their
real counterparts.

Real Material Virtual Material

Why?
● Textures dramatically increase realism of interaction [McMahan, et al. '10]
● Could program virtual environments where:

● You feel the fine surface details of the virtual paper you are writing on

● Doctor feels the texture of skin as he cuts with a virtual scalpel

● Dentist feels the grit of a virtual tool scratching on tooth enamel

● You send a friend a virtual sample of a material you like

● A whole market of touch-enabled devices exists (Smartphones, Tablet PC,iPad)

9/
 Background
[Lederman '74] defines texture as a microscopic surface feature:

Vinyl Denim

Psychophysicists believe that we understand texture as roughness, stickiness, hardness.

Roughness information is sensed by the spectral power [Bensmaia et al. '05] [Yoshioka et al.
'07].
A variety of representations for virtual textures have been suggested; a few examples include:

Minksy, Lederman '96 Costa, Cutkosky '00 Fritz, Barner '96

… but none seem suitable for reproducing the complexities of the real thing
10/
 Texture Acceleration Data

Signals:

Vinyl @ 0.9 N, 150 mm/s

11/
How can we recreate the acceleration signal in real-time in a
virtual environment?
Paper scanned @ 0.6 N, 80 mm/s

Signal Replay
● Discontinuities

● Looping artifacts

● Unclear how to handle output

conditions with no recording

12/
How can we recreate the acceleration signal in real-time in a
virtual environment?
Paper scanned @ 0.6 N, 80 mm/s

Manually Modeling
● Difficult to get right

● Time consuming

13/
How can we recreate the acceleration signal in real-time in a
virtual environment?
Paper scanned @ 0.6 N, 80 mm/s

We need a method that:

Manually Modelng
● Can generate unique, infinitely
● Difficult
long signals
to get right

● Non-repetitive in the time-domain,

● Time consuming
but maintains the
proper energy in the frequency-domain

● Moves smoothly between various output signals

14/
Linear Predictive Filter Design

All-Pole Filter

15/
Signal Synthesis

16/
 Synthesis Results
Rough Plastic @ 0.6 N, 100 mm/s

17/
 Synthesis Results
Rough Plastic @ 0.6 N, 100 mm/s

18/
 Real-World Interactions
● Texture accelerations vary based on normal force and scanning speed

● The real world is messy. Users vary their input.

O(n) query-time 19/

Rendering Hardware

20/
Rendering Hardware

21/
Video Demonstration

22/
Realism Study

16 subjects 23/
Realism Study

Avg. = 0.084

Avg. = 5.14

24/
Future Research

25/
Combining With Impedance-Based
Haptic Systems

26/
CHAI3D

27/
 Texture Mapping

● Material Description

● Tool Desctiption

● Digital Sampling Rate

● Texture Model Entries

● Normal Force

● Scanning Speed
cs.unc.edu/~zhangjd
● LPC Texture Coefficients

● LPC Model Variance

●.... more (tapping, etc.)

28/
 Modeling Improvements

Higher-Order, non-regular
Directional Texture database wikipedia.org

arthursclipart.com

29/
 Improving Autonomous Robotics

●Traditional approaches for autonomous robotics in

unknown environments typically proceed as:

Sense Plan Act

30/
Improving Autonomous Robotics
Human Perception During
Real-World Interaction

Robot Perception During

Real-World Interaction

31/
Completed Research in
Autonomous Robotics

32/
 In addition to unwittingly failing at task
execuction, traditional approaches can cause
dangerous damage to the robot and environment

33

33/
 Goals
• Delicate grasping and manipulation
• Contact event detection (even through hand-held
objects and tools)

34

34/
Johansson and Flanagan
Nature Reviews Neuroscience
Tactile Sensors
Human Robot

35

35/
How Humans Grasp

Johansson and Flanagan 36

36/
 How The Robot Grasps

SA-I

FA-I

FA-II

37

37/
How The Robot Grasps

38

38/
Initial Grip Force Estimation

39

39/
 Additional Results
Reduced Motor Effort
Slip Performance

Reduced Grip Force

40

40/
Gentle Manipulation

41

41/
Grasp Pipeline

42

42/
Pick and Place

43

43/
 Event Detection

• Table contact
• High-fives
• Hand-offs
• Arm impact
• Tool contacts

44

44/
Event Detection

45

45/
Event Detection

46

46/
Future Research

47/
A Framework for Categorizing
Haptic Events

Supervised Learning: SVM, decision trees, etc.

Parameterization: fourier/wave t-form, etc.

48/
Conferences:
Publications
● Joseph M. Romano, Takashi Yoshioka, and Katherine J. Kuchenbecker, Automatic lter design for synthesis
of haptic textures from recorded acceleration data, Proceedings of the IEEE International Conference on
Robotics and Automation (ICRA), 2010.

● William McMahan, Joseph M. Romano, Amal M. Abdul Rahuman, and Katherine J. Kuchenbecker,
High frequency acceleration feedback significantly increases the realism of haptically rendered textured
surfaces, Proceedings of the IEEE Haptics Symposium, March 2010, pp. 141-148.

● Nils Landin, Joseph M. Romano, William McMahan, and Katherine J. Kuchenbecker, Haptics: Generating
and perceiving tangible sensations, Proceedings of the Eurohaptics Conference, Vol. 6192, 2010, pp. 79-86.

● Katherine J. Kuchenbecker, Joseph M. Romano, and William McMahan, Haptography: Capturing and
recreating the rich feel of real surfaces, Proceedings of the 14th International Symposium of Robotics
Research (ISRR) (Lucerne,Switzerland), 2009.

● IROS/Humanoids 2011. A framework for categorizing haptic events during autonomous automation.

● World Haptics 2012. A CHAI3D plug-in for realistic haptic textures.

Journal:
● Joseph M. Romano, Kaijen Hsiao, Gunter Niemeyer, Sachin Chitta, and Katherine J. Kuchenbecker,
Human-inspired robotic grasp control with tactile sensing, IEEE Transaction on Robotics (2011), Accepted.

● Joseph M. Romano, CraigMcDonald, and Katherine J. Kuchenbecker, Creating realistic virtual textures from
contact acceleration data, Transaction on Haptics (2011), Under Review.

● An additional publication extending the conference papers above. (October RAM).

49/
 Additional Successes

● Two official live conference demos. ● Official ROS released stack

● Actively maintained
● H.S. 2010 best short oral
● Base “cturtle-pr2” code (manip. stack)
● presentation award
● Used by many of pr2-beta-test sites

50/
Thank you for your time.

51/