Using Hand Gestures to Control Devices and Games "Using LabVIEW software and the Vision Development Module,

we developed two games and a robot control that does not need a keyboard or mouse, it just relies on hand gestures and an Xbox 360 Kinect sensor."
Chin Hock Yong , Nanyang Polytechnic

Figure 1: Xbox Kinect Sensor

The Challenge:
Using hand gestures to control devices traditionally controlled by a computer mouse and keyboard.

The Solution:
Using the Microsoft Kinect sensor and image pattern matching tools from the NI Vision Development Module to track our hand gestures to control devices and play BrickBreaker and Wac-A-Mole games without using a mouse or keyboard. Author(s): Chin Hock Yong - Nanyang Polytechnic Koh Chang Rong - Nanyang Polytechnic Dr. Edwin Foo - Nanyang Polytechnic The Kinect for Xbox 360 (Figure 1) brings games and entertainment to life in extraordinary new ways without using a controller. The Xbox 360 Kinect sensor can now run on the PC. Using NI LabVIEW software, we developed games that dont need a keyboard or a mouse; the games are fully controlled by the human hand. Figure 2 shows a system overview. The whole system starts with the Kinect sensor, which captures the image. We process and characterize the image using the NI Vision Development Module. We use pattern matching techniques in LabVIEW such as IMAQ Setup Learn Pattern 2 VI, IMAQ Learn Pattern 2 VI, and IMAQ Match Pattern 2 VI. We use these techniques to detect one position of the hand movement. This is then used in the three subprograms including Wac-A-Mole, BrickBreaker, and mobile platform control. Figure 3 shows one flow of the program. The programs two main functions are pattern matching and motion tracking. Pattern Matching First, we capture an image to use as a template for pattern matching. We save the template image in .png format for easier reference. We create the template image by selecting the region or object the players are interested in to use as a tracking reference. This reference is also used as a controller input. Motion Tracking After the template image is created, the program uses it as a reference to search inside the continuously updated image being captured by the Kinect sensor. By comparing the captured image to the saved template, the program can continuously track the moving object. Using these two functions, we can control cursor movement inside the front panel instead of using a mouse to move the cursor. This is how we developed the human hand tracking system. Program Flow and Selecting the Region of Interest Before selecting a game, the system must capture your palm. We set a control range for the mean value and standard deviation inside the region of interest (ROI) that is bound by a green box and recognizes only the human palm. After the mean value and standard deviation are in the range we set, the program captures your palm and saves it to a template image file before you choose your game. The Game Menu The cursor, which is a red bird, is fully controlled by human hand movements. We could have used any object to control movement, but we chose the human palm because it is intuitive to use. By using your hand to move the cursor into the highlighted yellow region, you can choose either BrickBreaker or Wac-A-Mole (Figure 4). If you want to exit the game menu, just move the cursor onto the EXIT button. The whole program stops running if the EXIT button is activated. BrickBreaker Figure 5 shows the BrickBreaker game front panel. The movement of the yellow bar is again controlled by the image captured as the template. You can control the BrickBreaker game with any object that matches one of the template images you set in the motion tracking device. The cursor, which is a yellow bar, can only move in one dimensionthe x-axis. A green LED determines if the hand is within the detecting range and region. You hit the ball with one bar to break as many bricks as possible. Whenever the ball breaks a brick, the score increases by 100 points and the highest score is saved and displayed. If you break all the bricks, you complete the game. The game can also end if the player misses one ball. The bounce sound effect is activated when the bar hits the ball and the break sound effect is activated when the ball breaks the brick. The game speed can be adjusted from 0 to 10, where 0 is the lowest speed and 10 is the highest speed. To start the game, you move the cursor to the center, and to end the game, you move the controller down until the cursor touches the quit button. Wac-A-Mole

1/5

www.ni.com

Figure 6 shows the Wac-A-Mole game front panel. The cursor, which is a hammer, is also controlled by hand movements. The hammer can move in two dimensionsthe x- and y-axis. The player uses the hammer to hit moles that jump up. You can hit the play button to start the game and the exit button to quit and return to the games menu. Players get 40 seconds to score as much as they can and the highest score is saved. When the hammer enters the region of the mole that jumps up, the hammer is activated to force the mole down. When a mole is hit, a BAM! graphic appears on the screen and a BAM sound effect is heard simultaneously. When the remaining time is less than 10 seconds, a countdown sound effect is activated and a whistle sound effect is activated when the remaining time equals zero. Figure 7 shows one block diagram implementation of the human hand recognition, motion tracking, Wac-A-Mole, and BrickBreaker. The algorithm for this application is divided into the following four main parts: a) b) c) d) Select the object that represents the players controller and scan the players palm. Initialize a value in the main program. Perform motion tracking to track the selected objects and control the cursor. The Mobile Platform

Besides using hand gestures to control the movement of games using the Kinect sensor, we also developed a subVI to Decide what to do: control a mobile platform, the SRV-1. The SRV-1 platform is a remote-controlled Play BrickBreaker webcam, or a self-navigating autonomous Play Wac-A-Mole robot. The original program developed by Exit this program another group of students only ran in LabVIEW and was controlled by a keyboard or joystick. Now, the SRV-1 can be controlled by a Kinect sensor. By using the Kinect sensor, the SRV-1 can follow human hand motion. By controlling the tracking position in the image as shown in Figure 8, which is obtained by the Kinect sensor, we can move the SRV-1 robot. Figure 9 shows the SRV-1 program front panel.

Conclusion Using LabVIEW and the Vision Development Module, we developed two games and a robot control system that did not need either a keyboard or a mouse. These systems just rely on hand gestures and an Xbox 360 Kinect. Therefore, we achieved our objective of controlling games and devices without using a mouse or keyboard. With the Kinect sensor and LabVIEW software, we realized our dream of controlling these devices using only hand gesture movements. This helps us move closer to the ideal presented in the movie The Minority Report. Author Information: Chin Hock Yong Nanyang Polytechnic 180 Ang Mo Kio Ave 8 569830 Singapore Tel: 65500792 Fax: 64549871

Figure 1: Xbox Kinect Sensor

Figure 2: System Overview

2/5

www.ni.com

Figure 3: Main Program Flow Chart

Figure 4: Game Menu Front Panel

Figure 5: BrickBreaker Front Panel

3/5

www.ni.com

Figure 6: Wac-A-Mole Front Panel

Figure 7: Block Diagram of the Main Program

Figure 8: Controlling the Image from the Kinect Sensor

Figure 9: SRV-1 Control Panel

4/5

www.ni.com

Next Steps
Learn More About LabVIEW Download LabVIEW

Legal This tutorial (this "tutorial") was developed by National Instruments ("NI"). Although technical support of this tutorial may be made available by National Instruments, the content in this tutorial may not be completely tested and verified, and NI does not guarantee its quality in any way or that NI will continue to support this content with each new revision of related products and drivers. THIS TUTORIAL IS PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND AND SUBJECT TO CERTAIN RESTRICTIONS AS MORE SPECIFICALLY SET FORTH IN NI.COM'S TERMS OF USE ( http://ni.com/legal/termsofuse/unitedstates/us/).

5/5

www.ni.com