ECE_PAPER SIXTH SENSE TECHNOLOGY

NAME OF THE PARTICIPANT 1: VASA YESHWANTH BRANCH:ECE NAME OF THE INSTITUTION:TEEGALA KRISHNA REDDY ENGINEEREING COLLEGE CONTACT NUMBER:9985776771 EMAIL ID:yeshwanthvasa@gmail.com

NAME OF THE PARTICIPANT 2:KEDAR PATHAK BRANCH:ECE NAME OF THE INSTITUTION: TEEGALA KRISHNA REDDY ENGINEEREING COLLEGE CONTACT NUMBER:9494243056 EMAIL ID:kedarrocks.tkr@gmail.com

ECE_PAPER-SIXTH SENSE TECHNOLOGY

We've evolved over millions of years to sense the world around us. . Information is confined traditionally on paper or digitally on a screen. Sixth Sense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. Sixth Sense frees information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer. Sixth Sense is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural and gestures to interact with that information. By using a camera and a tiny projector mounted in a pendant like wearable device. Sixth Sense sees what you sees and visually augments any surfaces or objects we are interacting with. It projects information on to surface, walls, and physical objects around us ,and lets us interact with the projected information through natural hand gestures, arm, movements, or our interaction with the object itself. Sixth Sense attempts to free information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer. A low-cost intelligent mobile phone-based wireless video surveillance solution using moving object recognition technology is proposed in this paper. The proposed solution can be applied not only to various security systems, but also to environmental surveillance. Basically it is a device which is a mini projector and which can be projected on any surface, it carries the information stored in it and also collects information from the web. It is the one which obey hand gestures of yours and gives you what you want to see ands know. It is the combined technology of computer along with cell phone. It works when a person hang it on his neck and start projecting through the micro-projector attached to it. Your fingers works like the keyboards as well as mouse.

Introduction:
'Sixth Sense' is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information.

Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world. Information is confined traditionally on paper or digitally on a screen. SixthSense bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. Imagine being able to take a digital photograph by making the shape of a picture frame in the air with your hands, or dialing a phone number by simply tapping your palm with a finger or even navigating your way around a city by projecting a map onto the nearest wall, then using it as a touch screen to find a convenient subway station or bus stop. This is the promise of Sixth-Sense, a prototype wearable device invented by MIT engineer Pranav Mistry. Mysterys device consists of a video camera and LED projector combined into a pendant worn around the neck. This is linked to a smart phone worn on the hip, which houses the devices software and links it wirelessly to the internet. The camera not only takes pictures, but also records the users hand gestures. The software processes these gestures and interprets them into actions.

The increasing need for intelligent video surveillance in public, commercial and family applications makes automated video surveillance systems one of the main current application domains in computer vision. Intelligent video surveillance systems deal with the real-time monitoring of persistent and transient objects within a specific environment. Intelligent surveillance system has been evolution to third generation, known as automated wide-area video surveillance system. Combined computer vision technology, the distributed system is
3

autonomic, which can also be controlled by remote terminals. A low-cost intelligent wireless security and monitoring solution using moving object recognition technology is presented in this paper. The system has good mobility, which makes it a useful supplement of traditional monitoring system. It can also perform independent surveillance mission and can be extended to a distributed surveillance system. Limited by the memory consuming and computing capacity in a mobile phone, background subtraction algorithm is presented to be adopted in mobile phones. In order to be adapted to the slow and slight changes of the natural environment, a self-adaptive background model updated automatically and timely is detailed. When the subtraction of the current captured image and the background reaches a certain threshold, a moving object is thought to be in the current view, and the mobile phone will automatically notify the central control unit or the user through phone call, SMS, or other means. Thesproposedsalgorithmscansbes implemented in an embedded system with little memory consumption and storage space, so its feasible for mobile phones and other embedded platforms, and the proposed solution can be used in constructing mobile security monitoring system with low-cost hardware and equipments.

What is sixth sense?

All of us are aware of the five basic senses seeing, feeling, smelling, tasting and hearing. but there is also another sense called the sixth sesnse.It is a basically a connection to something greater than what their physical sense are able to perceive. To a layman, it would be something super natural. Some might just consider it to be a superstition or something physiological. Grasping the inner nature of things intuitively. Extrasensory, the ability to sense something by means other than the five main sense of touch, taste, smell, sight, or hearing. The sixth sense is thus all about predicting about something or making correct decisions based on the information which already exists. The technology is mainly based on hand gesture recognition, image capturing, processing, and manipulation, etc. The camera is used to recognize and track users hand gestures and physical objects using computer-vision based techniques, while the projector is used to project visual information on walls or on any physical thing around us. Firstly, the basic
4

principle of moving object detecting is given. Limited by the memory consuming and computing capacity of a mobile phone, a background subtraction algorithm is presented for a adaptation. Then, a self-adaptive background Central control unit or the user through phone call, SMS (Short Message System) or other means. Other hardware includes mirror and colored caps to be used for fingers. The software of the technology uses the video stream, which is captured by the camera, and also tracks the location of the tips of the fingers to recognize the gestures. This process is done using some techniques of computer vision.

Module:
The sixth sense prototype is comprised of a pocket projector, a mirror and a camera. the hard ware components are coupled in a pendant like mobile wearable device. both the projector and the camera are connected to the mobile computing in the users pocket. the projector projects visual information enabling the surfaces, walls and physical objects around us to be used as interfaces; while the camera recognizes and tracks users hand gestures and physical objects using computer vision based techniques. the software program processes the video stream data captured by the camera end tracks the location of the colored marker (visual tracking fiducially) at the tip of the users fingers using simple computer vision techniques .the movements and arrangements of these fiducially are interpreted into gestures that acts as an interaction instructions for the projected applications interfaces. The maximum number tracked fingers is only constrained by the number of unique fiducials,thus sixth sense -user interaction

Interfacing:
You can carry the digital world with you wherever you carry it. you can start using any surface or any wall around you as an interface. This module as already told consists of three major parts .the projector, the camera, the mirror. If you lack a surface around you, then its easy as making your palm the Interface with your module. This module has the ability to set its interface making it convenient to the user handling it.

Background Subtraction Technology:

Background subtraction is a commonly used class of techniques for segmenting out moving objects of interest in a scene for applications such as surveillance. It involves comparing an observed image with an estimate of the image if it contained no objects of interest. The areas of the image plane where there is a significant difference between the observed and estimated images indicate the location of the objects of interest. The term background subtraction" comes from the simple technique of subtracting the timely updated background template from the observed image.

Background template counstruction & update moving objects

Real time frames

moving object identification

Background Subtraction Algorithm

6

Background Template Construction:

Before the moving objects can be identified, a background template must be built. Generally, background and foreground (moving objects) are mixed together such as waving leaves in the garden and running automobiles on high way. The foreground can not be removed so the ideal background image can not be retrieved. But the moving objects do not exist in the same location in each frame of a real-time video sequence. An average frame of the video sequence can be retrieved to approach the ideal background image. The gray values of pixels which have the same location in each frame of the Video sequence is averaged to represent the gray value of the pixel which located in the same place in the approximate background. An average value of pixels in the same location of each frame in a video sequence is calculated. To simplify, the approximate background is also called background template, background or template in the following contents.

In our prototype, the first 10 frames are captured to calculate the background template (i=10). Moving objects can not be identified in these frames. If the moving objects move too slowly, I should be increased to reduce the tolerance.

Moving Object Recognition:

After the background template has been constructed, the background image can be subtracted from the observed image. The result is foreground (moving objects). Actually, the background is timely updated. The update algorithm is detailed in the next section.

In case of some random disturbances, each pixel will fluctuate in a small range even there is no expected moving objects in the scene. So there must be a strategy to judge it. A threshold is defined in the system. If the difference of one pixel between real time frame and template is

more than 10, then add 1 to the threshold. When differences of all pixels in the frame are all calculated, moving objects is thought to appear if the threshold is more then 3 percent of the total number of pixels in the frame.

Background Template Update:

Due to the sun light changing very slowly, the background template must be updated timely. Otherwise the foreground can not be correctly identified anymore. the background template if Pixel k is a pixel in frame j, and Pixel background k is the corresponding pixel in background template. These two pixels have the same location in their frames. With such method, the background template can adjust automatically according to environment change.

J2ME Technology:
In this article, we have implemented a prototype on mobile telephones based on J2ME technology. Java Platform, Micro Edition (Java ME) is the most ubiquitous application platform for mobile devices across the globe. It provides a robust, flexible environment for applications running on a broad range of other embedded devices, such as mobile phones, PDAs, TV set-top boxes, and printers. Applications based on Java ME software are portable across a wide range of devices, yet leveraging each device's native capabilities.

Showing the Camera Video

The video coming from the camera can be displayed on the screen either as an Item in a Form or as part of a Canvas. A VideoControl makes this possible. To get a VideoControl, just ask the Player for it: VideoControl mVideoControl = (VideoControl)mPlayer.getControl("VideoControl");

Capturing an Image
Once the camera video is shown on the device, capturing an image is easy. All you need to do is call VideoControl's getSnapshot() method. The getSnapshot() method returns an array of bytes,

which is the image data in the format you requested. The default image format is PNG (Portable Network Graphic). byte[]raw=mVideoControl.getSnapshot(null); Image image = Image.createImage(raw, 0, raw.length);

Creating a Message Connection

To create a client MessageConnection just call Connector.open(), passing a URL that specifies a valid WMA messaging protocol. MessageConnection mc = (MessageConnection)Connector.open(addr);

Creating and Sending a TextMessage

Theconnection is a client, the destination address will already be set by the implementation (the address is taken from the URL that was passed when the client connection was created). Before sending the text message, the method populates the outgoing message by calling setPayloadText(). TextMessage tmsg= (TextMessage)mc.newMessage(MessageConnection.TEXT_MESSAGE); tmsg.setPayloadText(msg); mc.send(tmsg); Capitalize the first letter of nouns, pronouns, verbs, adjectives, and adverbs; do not capitalize articles, coordinate conjunctions, or preposition

Prototype
The system architecture is shown in Figure 2. In the prototype system, if the difference between real-time frame and template reaches a predefined threshold, moving objects are considered to

appear. Then the handset will send out an alert SMS. Since the device has good mobility, it can be put anywhere including those area not covered by other surveillance system. And it can be deployed rapidly in emergency.

real time frame capture

Background subtraction algorithm

sms alert/ ring alarm

Figure 2: System Architecture

Implementation:
The prototype has been implemented on Nokia N series of latest version. The first picture in the form is real time frame, which is got from the camera originally. The second image is the template image. If there are some moving objects being detected, the third picture will be displayed on the form. And some real time information is displayed below the pictures.

10

Since the first several frames must be stored to calculate the template, a big memory heap size is needed. The JVM heap cant provide so much memory. There are two methods to solve this problem. First, the frames can be stored to the EFS (Embedded File System). Second, the image size can be reduced. The first method can provide high resolution image data, which contains more detailed image features. But storing to EFS will take a much longer time than storing to memory, and much longer time is need in the following calculating processes. The second method will lost some detailed image features, but it can fully operated in memory and reduce much processing time. Considered the requirements of the real time ability, the second method is adopted. The image size is reduced to 160*120, which is still enough to be used to identify the moving objects. The performance for 100 frames is detailed in Table 2. The term Snapshot time is the time length to get the image though J2ME MMA API. It mainly depends on the capability of Hardware, Operating System and Java VM. The DIP (digital image process) time is the time length to perform the background subtraction algorithm, including images compare and template update. The Frame time is the total time to process a frame which equals to the sum of Snapshot time and DIP time. The Frame time Average is an averageof several frame time. The Template time is the total time to construct the background template. In this instance, the Snapshot time, DIP time and Frame time of the 100th frame are presented. And the Frame time Average of the 100 frames and Template time are also given. The background template can be built in less than half a minute. And the total time to calculate a frame is around one and a half seconds. It meets the requirements to be a family security monitoring system and an anti-theft system. The experiment demonstrates the feasibility of the proposed system. Some frames are magnified to be seen more clearly (See Figures 4 - 6). Figure 4 is the self-adaptive background template. As shown in Figure 5 and 6, a person run into the scene was identified immediately.

11

Applications:
The device has a huge number of Applications firstly, it is portable and easy to carry as you wear it in your neck. It is the combined technology of computer along with cell phone. It works when a person hang it on his neck and start projecting through the micro-projector attached to it. Your fingers works like the keyboard as well as the mouse.

Easy to Draw: A person can draw anything on paint with the help of his fingers. So, why to
use mouse for painting. The drawing application lets the user draw on any surface by observing the movement of index finger.

12

Capture photos with fingers: why to take camera on your holiday and no tension for the
photo space as this Sixth Sense computer will work like your camera. It caputes the photo, when you make a square with your fingers, highlighting which one you want to frame.

Phone Call: You can call to your friend by typing the numbers on your hand. It display the
keypad of the phone over your palm and the keys appears on the four finger... Use your another hand's finger to press the keys.

13

Check the brand of the Product: It helps you to choose the best brand product from the
super market.

Read Books easily: Check out the ratings of the Book you are going to buy, it checks the
ratings from the internet. And another amazing thing is that it reads the book for you

Newspapers: Did you saw the moving pictures of the Newspaper in the movie Harry Patter, it
is quite similar to it. It Searches the most appropriate video from the web by seeing the headlines or the caption of the News report.

Check your Flight Status: You can check the status of the flight while you are on Taxi.
Just place the ticket in front of the projector and it checks its status from the internet.

14

Google Maps
The sixth sense also implements map which lets the user display the map on any physical surface and find his destination by just using his gestures.

Future Scope and Conclusion:

This technology would help us not being machines sitting in front of other machines. There are a lot of companies like the mobile companies sponsoring for this module integration so as to use them in different ways. This can also be linked through the satellite to have communication where it is difficult to have a proper signals, this satellite communication can be used for the defence. The moving object recognition technology led to the development of autonomous systems, which also minimize the network traffic. With good mobile ability, the system can be deployed rapidly in emergency. And can be a useful supplement of traditional monitoring system. With the help of J2ME technology, the differences of various hardware platforms are minimized. All embedded platforms with camera equipped can install this system without making any changes to the application. Also, the system can be extended to a distributed wireless network system. Many terminals work together, reporting to a control center and receiving Commands from the center. Thus, a low-cost wide-area intelligent video surveillance system can be built. Further more, with the development of embedded hardware, more complex digital image process algorithms can be used to give more kinds of applications in the future.
15

This technology can be used as a fifth sense rather than a sixth sense for the dumb. Using the gestures of the people this instrument can be used to generate voice and make the dumb speak. This would be a greatest achievement for the mankind.

16