Fusing Virtual Game

Environment and Reality World

A Cyber-Physical Game Using
Interactive Robotic Marionettes

I-Ming Chen Ke Yeng Tee

School of Mechanical and Aerospace Engineering
Nanyang Technological University
1st IFToMM Asian MMS Conference
Taipei, Taiwan, Oct 21-25, 2010
 Introduction

CYBER-PHYSCAL
GAME

Interactive game
system
with
Physical game agents
Intelligent algorithms
Real players

Cyber games Robot games

 Cyber-Physical Game

Puppeteer

Marionette
Staging
Scoreboard Manipulator

Gamepad

• Realistic gaming experience with physical agents

• Governed by physical laws that may not be perfectly simulated
• AI helps user to control complex gaming scenario to increase fun factors
• Complex and multi-modal human-machine interface
 Purpose of Study
• Explore new method for human-machine
interaction
• Explore novel robotic mechanism design for fast
paced CPG
• Develop system architecture for CPG game
control
– Game controller for user actions
– Autonomous and intelligent actions for smooth game
– Reward system: sensors and scoring app
 System Architecture
Physical System Cyber System

Servos Controller
Marionettes and
Puppeteers
Sensory Interface Cyber-
Physical
Game
Processor
Staging Manipulators Manipulators Motion Planner

User Interface

User Interface Driver

User Interface and
Interaction
Interaction Renderer

User

Users
 System Architecture
Physical System Cyber System

Servos Controller
Marionettes and
Puppeteers
Sensory Interface Cyber-
Physical
Game
Processor
Staging Manipulators Manipulators Motion Planner

User Interface

User Interface Driver

User Interface and
Interaction
Interaction Renderer

User

Users
 Marionette Design
• Marionette figure
– 12 inches wood figure with
movable joints (18 DOFs)
– types of actions:
• Kicks and Punches (Left & Right)
• Fall back
– Installed with Reed sensors and
magnets for hit detection
 Robotic Puppeteer Mechanism

Conventional marionette Inline stringing for

stringing fast action

• Inline stringing to hide strings in tube under limbs

• Type of limb motion limited to fighting actions but at
higher speed
Kicking Action
String

Leg Elastic Rubber Band

Joint
Tubes

Hinder

Direction of string pulling

After elongation

Before elongation

Shank is then lifted because

thigh has been lifted to the
limit, greater force exerted
from the servo motor to
overcome force of rubber
band and weight of shank.
Thigh is being lifted first
because greater force
needed to overcome the
weight of shank and force
of rubber band.
Punching Action

String attached to
shoulder and
elbow.

Rubber band String

Joint

Rubber band
By pulling the string, the
String
arm is bent by string which
attached at shoulder and
forearm. Both rubber bands
Without pulling the string, at both joints are stretched.
the arm is straightened by
the pulling force of rubber
bands at both joints.
 Hit Detection
• Reed Switch
– Activated when a magnet comes closely
– No interface circuitry and power supply required
– Installed on fist and foot
– Magnet installed on body
Reed Switch
(source: en.wikimedia.org)

Installation of reed switch

Installation of magnet
 Puppeteer Controller
• Manipulate strings to control the action of marionette
• Mounted on staging manipulators
• 4 Servo motors and four 50mm pulleys for each
puppeteer
• Interface with Phidget Servo Control

Phidget Servo Control

 System Architecture
Physical System Cyber System

Servos Controller
Marionettes and
Puppeteers
Sensory Interface Cyber-
Physical
Game
Processor
Staging Manipulators Manipulators Motion Planner

User Interface

User Interface Driver

User Interface and
Interaction
Interaction Renderer

User

Users
 Staging Manipulators Subsystem
• 6 PowerCube motors
• Two 3-degree of freedom planar manipulators
• Smooth trajectory movement in any direction with
specified facing angle for both marionettes
• Interface with CAN-bus to USB converter

Fixed link

Revolute Joint

PowerCube
(source: technikerprojekt.de)

Revolute Joint

Position of Marionette
Face-to-face Movements
Coordinate of Joint 1, (x1, y1)

Angle of Joint 1, θ1

Length of Link 1, L1

Coordinate of Joint 2, (x2, y2)

Angle of Joint 2, θ2

Length of Link 2, L2

Coordinate of Joint 3, (x3, y3)

Virtual slider Coordinate of B, (xB, yB)

Angle of Joint 3, θ3

Coordinate of A, (xA, yA) Length of Link 3, L3

• Define positions for Marionette A & B

• Calculate facing angles for both marionettes
• With length of Link 3, obtain position of Joint 3
• Solve IK with coord. of Joint 1 & 3, length of Link 1 & 2
• Obtain coordinates for Joint 2
• Attain all angular positions for all revolute joints
 System Architecture
Physical System Cyber System

Servos Controller
Marionettes and
Puppeteers
Sensory Interface Cyber-
Physical
Game
Processor
Staging Manipulators Manipulators Motion Planner

User Interface

User Interface Driver

User Interface and
Interaction
Interaction Renderer

User

Users
 Multi-modal User Interface and
Interaction System
• Provides interface and interaction features
for users to communicate with Cyber-
Physical Game Processor

• Multi-modal interface
– Reads user’s instruction through gamepads
– Displays score for both users Gamepad

– Plays sound effects from speakers

Scoreboard
 System Architecture
Physical System Cyber System

Servos Controller
Marionettes and
Puppeteers
Sensory Interface Cyber-
Physical
Game
Processor
Staging Manipulators Manipulators Motion Planner

User Interface

User Interface Driver

User Interface and
Interaction
Interaction Renderer

User

Users
 Implementation with Microsoft Robotics
Developer Studio
Software
System Architecture Implement as Purposes of services
Service
Processes for all inputs and outputs
Cyber-Physical Game
Main Makes major decisions
Processor Controls other services
Calculates trajectory of staging manipulators
Provides simulation of staging manipulators
Manipulators Motion Calculation Checks positions of all PowerCubes
Planner Prevents collision and out-of-boundary movement
Drives PowerCube motors
PowerCube Monitors PowerCube motors’ condition
Servo Controller Servo Drives servo motors
Game
Monitors and processes inputs from first gamepad
Controller
User Interface Driver
Game Monitors and processes inputs from second
Controller 2 gamepad
Pools status of sensors
Sensory Interface Processes raw values from sensors
Phidget Provides digital outputs for score board
Interaction Renderer Produces sound effect for different actions
 Software Architecture
Message Purpose Content
walking directions that
Walk marionette A to walk
marionette A to follow. Game Game
Controller Controller2

walking directions that UpdateAxes & UpdateAxes &

Walk2 marionette B to walk UpdateButtons UpdateButtons
marionette B to follow.

marionette A to types of action that

Action perform particular marionette A to Phidget ScoreHit Main
Walk & Walk2
Servo
Action & Action2
action perform.
marionette B to types of action that Walk & Walk2

Action2 perform particular marionette B to

action. perform. Calculation

there is change in the axis pressed by

UpdateAxes NewPosition
axis pressed. user.
there is change in buttons pressed by
UpdateButtons
button pressed. user. PowerCube

there is hit on body part of either

ScoreHit
of either marionette. marionette A or B is hit
Services Relationship Diagram
there is change in
coordinates of all the
NewPosition position of
PowerCube motors
PowerCube
Video of Cyber Physical Game
 Conclusion
• A CPG using fighting puppets is developed
– Novel cable-driven puppet with springs and strings for
fast action
– Staging manipulator design and implementation
– “Virtual slider” concept for face-to-face action for
smooth game operation
– CPG system architecture design and implementation
in Microsoft Robotics Studio
• Multi-modal user interface developed and tested
• Further development into new form of
entertainment
 Thank You

Screenshot of fighting game Screenshot of soccer game

Question and Answer