Musical Virtual Reality Applications

Michael Kriegel
Contents
 My 2005 final MSc project at the Centre for
Virtual Environments (University of Salford)
 A current Heriot Watt student project: Wii
Piano by Thomas Methven
Motivation
 Personal interest in electronic music technology
(sampling, synthesizers, MIDI)
 Use VR as a new interface to electronic musical
instruments
 Original idea: create a Virtual Reality drum kit
interface with optical tracking input
 Why do this?
 Might provide interesting musical performance
 good method of testing/measuring optical Vicon tracking
system
 Virtual interface doesn’t take up space (many people
don’t have space for a real drum kit)
Research Questions

 Can the feeling of playing the drums be

replicated with a VR interface?
 Is optical tracking suited (fast and accurate
enough) for this task ?
Literature Review 1
 First initial literature review strived to find similar
comparable projects
 Not much on the academic sector:
 University of Twente in Netherlands had a similar project,
but used different technology – also no papers on that
project
 Axel Mulder, Phd thesis, Simon Fraser University,
Canada – data glove / hand gestures to create sound
 Interdisciplinary Centre for Scientific Research in Music
(ICSRiM) at University of Leeds have a research
framework “Music Via Motion” that tries similar things,
however again with different technology
Literature Review 1

 Related commercial and art projects:

 Electronic Drum Kits
 Theremin, Roland‘s D-Beam controller, Alesis Air
FX
 Eye Toy
 Soundbeam
 Myron W. Krueger‘s installations
Technology - MIDI

 Musical Instrument Digital Interface

 A protocol that allows electronic musical
instruments to communicate with each other
(or with a computer)
 A MIDI instrument (or computer) usually has
a MIDI-in port to receive messages and a
MIDI-out port to send messages
 MIDI protocol defines messages: e.g. Note-
On, Note-Off
Technology - Creating drum
sounds electronically
 Can be easily done via Samplers, Drum
Machines, Synthesizers, etc. (both hard- and
software)
 For my project: Using a free sampling
software and a pre-defined drum kit
(collection of drum sound samples) that can
communicate with the application via MIDI
Technology – optical tracking

 Vicon tracking / motion capture system

consists of
 5 infrared cameras
 Dedicated computer for pattern recognition
 Pattern recognition software
Technology – optical tracking

 Objects that need to be tracked are equipped

with reflective markers
 Objects need distinctive marker patterns
 By observing the marker pattern from the 5
different perspectives, the objects position
and orientation can be calculated
 Software has training mode to learn to
recognize new objects
Tracked objects

 Shutter
glasses

 Drum
sticks

 Feet
Technology – Display System

 Barco Trace Large Screen Display

 Back Projection Screen with 120 Hz refresh
rate for 1280 x 1024 resolution
 Active Stereo with
stereo shutter
glasses
System Design
Implementation Steps
 Create 3D Models of Drum Kit with 3DS Max
 Import them scene graph
 Create interface with tracking system
(glasses, hand, feet)
 Create MIDI connection between application
and Software Sampler
 Implement collision detection between virtual
drum stick positions and drum skins (collision
triggers Midi out note, velocity of impact
determines the volume of the sound)
Main Program Loop
 Get current position of sticks
 Calculate if a collision has occurred (also compare with
previous position to avoid continuous sounds)
 If collision has occurred
 Calculate speed (again by comparing with the previous positions and
their timestamps)
 Send a midi message (Note On) with Note number relating to the
object that was hit and volume relating to the speed
 Update Scene (set new position of sticks)
 Render Scene
 Could be improved by seperating the collision
detection/sound output and grpahics in different threads
Screenshot
Problems discovered
 Missing haptic feedback seemed to be a major
drawback
 Some more research into haptic devices that could
be used in this context, Spidar seems to be the only
well suited device for drum sticks, but not available
at our department
Change of Research Direction
 Since the haptic path was a dead end due to
availability of hardware …
 and I was ahead of schedule with the
implementation..
 The implementation of two other virtual instruments
was decided:
 Virtual Piano – to compare both tracking performance
and haptics with drums
 Virtual Synth Control (Theremin like experimental
instrument) – to compare replications of traditional
instruments with experimental instrument
Virtual Piano

 Changes:
 New controller: gloves
 Different MIDI sound
module:
free software piano
 Piano model generated by the application and
not by modelling package
 Different collision detection algorithm (need note-
on and note-off events)
Screenshot
Virtual Synth Control (VSC)

 Design objectives:
 Easy to use
 No haptics required
 Psychedelic look and sound

 Input mode: drum sticks

 Sound module used: Free Software
synthesizer JX 220
Screenshot
Evaluation
 Main Purpose was to compare the user’s reception
of the 3 instruments.
 12 subjects
 Multiple choice questions:
 Which instrument did you like most?
 Which of the instruments, if any, could be best used for
making real music?
 Majority chose VSC for both questions (10/12 and
11/12)
 No-one chose “none” so everyone could see some
potential in the technology
Evaluation

 Another set of questions was aimed at

evaluating all instruments individually on a
scale of 1 to 7
 Some findings:
 No depth perception problems with any of the
instruments (question 1)
 Lack of haptics is severe, even more so for the
piano (question 2)
 People found it hard to drum to a rhythm / easy
to play VSC along with backing track (question 3)
3 ½ years later
Wii Piano

 Basic idea: use the WiiMote’s infrared

camera for very cheap finger tracking
 2D
 Compare with other input methods (e.g.
mouse, touch screen)
 Applications: Drawing, Piano
WiiMote

 Besides the usual buttons the WiiMote has 2

novel forms of input:
 Accelerometers
 Infrared Camera
WiiMote Finger tracking

 Normally the infrared camera is used to

evaluate the WiiMote’s position relative to the
Wii’s sensor bar

 For WiiMote Finger tracking, the WiiMote’s

camera is used other way round:
 WiiMote doesn’t move
 Infrared Light source moves
WiiMote Finger tracking
 No active light source on finger is used but
reflection (see passive markers in mocap)
WiiMote Finger tracking
User Interface: Button
Activation