VIRTUAL KEYBOARD

Submitted in partial fulfillment of the requirements

of the degree of

T. E. Electronics and Telecommunications

By

VISHAL JANGALE 26
WAQAAR KHATRI 36
RAKESH HANDE 25
TEJAS KAMBLE 30

Supervisor:

Prof.Shailaja Udtewar

Department of Electronics and Telecommunications

Xavier Institute Of Engineering

University of Mumbai
2017-2018
 CERTIFICATE

This is to certify that the project entitled Virtual Keyboard is a bonafide work of Vishal

Jangale(26), Waqaar Khatri(36), Rakesh Hande(25) and Tejas Kamble(30) submitted to the

University of Mumbai in partial fulfillment of the requirement for the award of the degree of

B.E. in Electronics and Telecommunication Engineering.

Dr. (Internal Guide Name)

Supervisor/Guide

Dr.Suprava Patnaik Dr. Y.D.Venkatesh

Head of Department Principal

Table of Contents
Abstract
1. INTRODUCTION
1.1 General Introduction and Problem Description
1.2 Research background
1.3 Aims
1.4 Dissertational Organization
2. LITERATURE REVIEW
2.1 Technical Literature Review
2.2 System Architecture
3. SYSTEM DESIGN
3.1 Propoesd design
3.2 proposed System
4. FUNCTIONAL SPECIFICATION AND TECHNOLOGIES
4.1 Models Used
4.2 Working of Virtual Keyboard
4.3 Software Technologies
5. IMPLEMENTATION
6. TESTING
7. CONCLUSION AND FURTHER WORK
7.1 Future Scope
7.2 Results
7.3 Conclusion
7.4 Evaluation
References
Appendix
ABSTRACT
Traditional QWERTY keyboards are large in size and offer little in terms of their
ability to modify. A Virtual Keyboard, with slightest physical form may provide
an answer to the point of time, where the size of desktops and laptops is becoming
smaller, the traditional keyboard acts as a hindrance to further miniaturization. In
todays world many different Virtual Keyboard exist which are using 3D cameras.

This project involves the creation of Virtual Keyboard using the IR Sensors. The
project aimed to create such keyboard which requires no external hardware, easy
to use with reconfigurable option in it. This project uses a Logitech standard web
camera. The main focus of this project is to know the problems with current VK
and how to solve these problems using IR Sensors. Finally this project aimed to
discuss and evaluate the working of Virtual Keyboard.

The outcome of the Virtual keyboard project was a cost-effective, user friendly,
accurate and portable virtual keyboard, which sends typed characters to any
handheld device, mobile device or PC. This virtual keyboard gives the quick
touch-typing feedback to the user and allows the users to rest their hands on it,
which are some of the major drawbacks in most of the existing projection
keyboards.
1. INTRODUCTION
This section of the dissertation will give a general introduction and problem
description. This section gives the information about the project, also the aims and
the motivation to be accomplished. Then, initially it discuss about the
methodologies which is used in the system.[1] Moving on from next section is a
literature review, which combined with the introduction outlined in the coming
chapter and gives a foundation on which to base the technical and functional
specifications.

1.1 General Introduction and Problem Description

A virtual keyboard is basically a keyboard that has no physical support and
operated by the user via typing i.e. by moving fingers on it rather than typing on
depressing physical keys. A VK is simply a projection keyboard projected and
touched on any surface.
A virtual keyboard is one of the examples of todays fast trend. Computing
is not just limited to laptops and desktops; it found the ways to enter into many
devices for illustration mobile devices like cell phones . The only thing which
remains constant is old QWERTY keyboard and virtual keyboard (VK)
technology is the latest development[1].
In project, simple virtual keyboard has been created using IR Sensors with
the help of simple 2D camera and without using any external hardware.
The first biggest problem in normal QWERTY keyboards is its size, they
are large in size and it is really very uncomfortable for a person to carry it from
one place to the other. The chances of breaking up may also get increased.

Project allows artificial intelligence to those people who want to work on

the simple surface thinking as if it is a keyboard. Virtual keyboard lets people to
have multilingual writing content or text content on existing platform. The idea
behind creating this virtual keyboard is simple and understandable. A Virtual
keyboard that have been created is small, well designed, handy and really very
easy to operate also, which results in the perfect solution for cross multilingual
text input.

Some of the Main features in the project: Simple and user friendly design
(interface), size of file really small, test whether every part working properly.
1.2 Research background
Turk, M and Klsch described the Virtual Keyboard in different forms out
of which, mostly virtual keyboards depends on CCD cameras and 3D optical
ranging and significantly depends on primarily on image processing, M. and Turk,
M. did a research which was focussed on various forms like rings, gloves,
projection and devices based on hand gestures. A special type of 3D camera or
two 2D cameras is used.

In this project novel technique has been used where it will use finger
shadow to detect the whether the finger is touching the virtual keyboard or not. To
implement devices which are highly specialized is not that practical idea. But the
idea bestowed for finger, hand and fingertip detection are convincing in this
changeable field or activity done under virtual human interface devices (i.e.
VHID).

1.3 Aims
The aims and objectives of this project holding two themes both focus on
technical side, first one deal with creation of the project and other with
reconfiguration of the project.

The primary aim of this dissertation project is to work on the technologies

which are going to be use in project, the designing and development of the
application. Then the secondary aim of this project is to go through the literature
reviews and do the research and find the best way to satisfy the user.

Firstly, this project aims to create a Virtual keyboard using IR Sensors using
simple camera and no additional hardware. Novel technologies have adopted
during the development of the application. Finally, the project aims to reconfigure
the keyboard, through which user can design their own keyboard layout and use it
for different purposes like for gamming.
1.4 Dissertational Organization
This project report structured in seven chapters, where the first chapter
discussed about the introduction of the Virtual keyboard followed by the problem
description, research background, and aims.

Second chapter discussed about the literature review in terms of technical,

and reconfigurable virtual keyboard.

In Third chapter, report discussed about the system design. First it discussed
about the
Proposed Design then it discussed about the original design which was used in
final product.

In next chapter, report discussed about the implementation of the project. In

chapter Five, it discussed about the testing of the project.

In chapter six, it discussed about the different models used in Virtual

keyboard, working of Virtual keyboard, and software required in implementation
of Virtual Keyboard.

In chapter seven, it discussed about the conclusion and further work, which
includes result, conclusion, future scope and evaluation of the project like what
went well, what challenges faced and what are the improvements from future
perspectives, also the future work on Virtual Keyboard.
2. LITERATURE REVIEW
Initially, the literature review concentrates on technical literature. So that, it
is easy for the user to understand the design opportunities and the number of
possible pitfalls while implementing the virtual keyboard.
Secondly literature review concentrates on the System Architecture.

2.1 Technical Literature Review

This technical literature review will first look into the architecture of a
Virtual Keyboard, through the information provided on the surroundings. Next,
will discuss about how virtual keyboard reconfigured using the architecture. After
studying all this will move to the languages, methodologies that have been used.
At last, it discussed about the implementation of a application with some screen
shots.
2.2 System Architecture
Table 1. Literature survey for techniques used for key detection

Author Techniques used University/Publication

Eckert, M. Lopez, M. et al. Middleware for mapping Tech. Univ. of Madrid,

gestures Madrid, Spain

Motion sensing camera

device

Su, Xiaolin, Zhang et al. Every keystroke can be Cyber Technology in

determine accurately by Automation, Control, and
image processing including Intelligent Systems (CYBER),
morphology principle and 2015 IEEE International
ellipse fitting Conference on

Jun Hu, Guolin Li, Xiang Xie, Bare-finger touch IEEE Transactions on
Zhong Lv, and Zhihua Wang interaction on regular Circuits and Systems for
planar Video Technology Year:
2014
3. SYSTEM DESIGN
This part of the report deals with the initial Proposed design through which
it is easy to understand the designs and user flow for the application

3.1 Proposed Design

Fig 1: Flow chart of the proposed design

3.2 Proposed System

1. The keyboard will be drawn on blank paper.

2. A camera will be there to capture live feed of fingers typing on blank paper with
keyboard drawn on it.

3. Hence, with the Image processing, in real time typed words on keyboard will
be detected.

4. Those words will be screened on desktop.

4. FUNCTIONAL SPECIFICATION
AND TECHNOLOGIES
Functional Specification and technologies can properly refer to the
theoretical analysis of the methods which are as follows:

4.1 Models Used

In the project, Arduino uno and IR Sensors Model has been used.

4.1.1 Arduino uno board

Arduino is a single board. This means that everything you need for basic
arduino usage comes soldered to one little circuit board. A single board
microcontroller is different than a single board computer, like the raspberry pi.
Arduino is open source. As open source hardware, the schematics for Arduino are
available to anyone for free, so if you wanted you could buy the electronics
components and a circuit board, and build your own Arduino. Many people build
and sell third party Arduino hardware
 Finally, Arduino is cheap. You can get the standard Arduino uno board for
just about $30. The most common Arduino board is the uno, ehich features an
ATmega328 microcontroller, mounted on a board with 14 digital I/O pins and 6
analog input pins, along with a power connector and a USB port that allows you to
program it with your PC. The downside of the Uno is that the ATmega328 is
pretty puny, its an 8-bit processor, with only 32K of flash memory and 2K of
system memory.

4.1.2 IR Sensor

An infrared sensor is an electronic device, that emits in order to sense some

aspects of the surroundings. AnIR sensor can measure the heat of an object as well
as detects the motion.These types of sensorsmeasures only infrared radiation,
rather than emitting it that is called as a passive IR sensor.

An IR sensor is a device which detects IR radiation falling on it. There are

numerous types of IR sensors that are built and can be built depending on the
application. Proximity sensors (Used in Touch Screen phones and Edge Avoiding
Robots), contrast sensors (Used in Line Following Robots) and obstruction
counters/sensors (Used for counting goods and in Burglar Alarms) are some
examples, which use IR sensors.
4.2 Working of virtual keyboard
A projection keyboard is a form of computer input device whereby the
image of a virtual keyboard is projected onto a surface: when a user touches the
surface covered by an image of a key, the device records the corresponding
keystroke. Some connect to Bluetooth devices, including many of the latest
smartphone, tablet, and mini-PC devices with Android, iOS or Windows platform.
Without your computer keyboard, whether it's part of a desktop or a laptop, it
might've been fairly difficult to get to this article. Sure, maybe you've bookmarked
the Web site and you used your mouse to click around between several links, but
most of the time, you probably type the Web site address into your browser to get
here. Most of us with personal computers use a keyboard almost every day to
compose e-mails, type out word processing documents, visit Web sites and
perform searches. And as easy as it may seem, typing is a skill we have to learn
and practice in order to perform properly. Most students take typing classes as
early as elementary school, and word processing is one of the most common skills
listed on a job-seeker's resume.

Without keyboards, communicating between family, friends and businesses

would be a much less streamlined affair. Today, even smartphones come equipped
with smaller keyboards. Phones that are more focused on e-mail and text
messaging, like the BlackBerry or several of the slider phones, offer these as a
more straightforward way of typing out text. On a typical cell phone with a three-
by-four numeric keypad, for instance, where at least three letters are assigned to a
number, you often have to punch through each key several times to get one word
on the screen. Some people are remarkably adept at this -- just watch some people
on a train or at the airport rifle through several text messages in a matter of no time
-- but for some, the act is cumbersome and takes far too long. Miniature keyboards,
however, attempt to solve that problem by giving users a complete QWERTY
setup to type e-mails, text messages and reminders.

A laser or beamer projects visible virtual keyboard onto level surface.

A sensor or camera in the projector picks up finger movements Software converts
the coordinates to identify actions or characters.

Some devices project a second (invisible infrared) beam above the virtual
keyboard. The user's finger makes a keystroke on the virtual keyboard. This breaks
the infrared beam and reflects light back to the projector. The reflected beam
passes through an infrared filter to the camera. The camera photographs the angle
of incoming infrared light. The sensor chip determines where infrared beam was
broken. Software determines the action or character to be generated.
4.3 SOFTWARE TECHNOLOGIES
Image Processing with MATLAB Images are everywhere, from everyday
devices like cameras and smart phones to specialized devices for medical imaging,
automotive safety, industrial automation, and more. Each of these uses for image
processing has unique challenges. MATLAB and Image Processing Toolbox
provide a flexible environment to explore 684 | P a g e design ideas and create
unique solutions for imaging systems. Matlab toolbox used in our project are as
follows
Image acquisition Toolbox.
Image processing toolbox
GUI build

4.4 ADVANTAGES
Virtual input devices are designed to offer the functionality of their
hardware counterparts without the actual use of hardware. The concept applies to
any device that requires tactile interaction for the input of data or other
information, such as a computer keyboard or piano. Virtual keyboards may be
difficult to become accustomed to for anyone used to conventional hardware, but
they do offer advantages unique to their design.

It can be projected on any surface or you can type in the plain air.

It can be useful in places like operation theaters where low noise is

essential.

The typing does not require a lot of force. So easing the strain on
wrists and digits.

The Virtual Keyboard is not restricted to the QWERTY touch-typing;

adjustments
can be done to the software to fit other touch-typing paradigms as
well.

No driver software necessary, It can be used as a plug and play

device.

High battery life.

4.5 MAINTENANCE
Unlike conventional hardware keyboards, virtual keyboards have no
moving parts, and the bulk of their functionality requires no actual contact with the
device. Whereas conventional hardware is prone to damage by spills, drops and
other malfunctions, virtual laser keyboards are easy to maintain. Manufacturers
often market these devices as "damage free," and this is an important advantage in
any piece of portable hardware that undergoes extensive use.

4.6 SPECIFICATIONS
The primary aim of this concept was to produce a low-cost alternative to the
commonly used keyboard. To maintain the low cost and due to unavailability of
some components, the components and their specifications had to be compromised.

1. Camera: suitable camera available and is being used for development.

We used C170 webcam by
Logitech.

2. Processor: we are using processor of the computer. We have used HP

pavilion 8GB RAM, 1TB graphics laptop for testing and implementation.

3. Software: MATLAB R2015a has been installed on the computer on

which we tested the implementation.
5. IMPLEMENTATION
In MATLAB, we have created a video object acquired by the webcam. This
video is used for frame extraction.

Our MATLAB code constantly extracts 2 frames from video and with the
help of Frame Differencing algorithm it monitors movement of the finger. If
any movement is detected then next frame to that is extracted.

The extracted frame is firstly thresholded to get pixels of references and tag.
Various data redundancy algorithms like addition of coordinates and
elimination are run to extract 1 pixel per reference.

With the help of references, gridding is done. As of now we have

successfully implemented keyboard with any combination up to 4X4. Our
code has flexibility to allow a keyboard of any size with a mere change of 2
numbers.

After gridding, positioning of the tag is done. For positioning we use left
upper and right lower points in the grid to detect corresponding key.

Next module is GUI. GUI is used to display the key which is detected. It is
created using the GUIDE toolbox of MATLAB.
6. TESTING
System testing is the stage of implementation. It aims at testing and ensuring
that the system works accurately and efficiently before live operation commences.
The logical design and physical design should be thoroughly and continually
examined on paper to ensure that they will work when implemented. Thus the
system in implementation should be a confirmation that all system works. The
testing phase includes entering the sample data to verify whether the system is
suitably working to the requirements mentioned. This phase is important in the
way that it actually deals with the real data.

The rules that serve for testing are,

Testing is a process of executing a program of executing a program with the

intent of finding an error.
A good testing is the one that has the high probability of finding
undiscovered errors.
A successful test is the one that uncovers a discovered error.
If testing is conducted successfully according to the above objectives, it will
uncover the errors in the software.
The various types of testing are:
Unit testing
Integration testing
Validation testing
7.1 FUTURE SCOPE
The virtual keyboard that has been implemented is like a small application
coming from the larger idea where it detects the finger edge with the help of
simple 2D camera. User doesnt need to require any external hardware or
additional space for this. The main idea would be developing the virtual keyboard
application for the mobile devices which helps in playing the game on mobile
phones, 3D modeling and lot more. Countries like India where there are different
languages user can easily change the language of the keyboard by internal
mapping of characters and use it anytime anywhere without the requirement of
any change in hardware. This results in creating multilingual keyboards feasible
economically. Last but not the least such keyboards will play a great role for blind
people without the need of extra expenditure.

7.3 CONCLUSION
We developed a system to get an input of keyboard drawn on a blank paper
using a real - time camera and ir sensors providing the interrupts. However, it is
difficult to get stable results because of the variety of lighting and skin colors of
human races. Most vision algorithms have illumination issues. From the results, we
can expect that if the vision algorithms can work in all environments then our
system will work more efficiently. This system could be useful in presentations
and to reduce work space.
REFERENCES
[1]Eckert, M. Lopez, M. ; Lazaro, C. ; Meneses, J. ; Martinez Ortega, J.F., 2015
Mokey - A motion based keyboard interpreter .Tech. Univ. of Madrid, Madrid,
Spain

[2]Su, Xiaolin, Zhang, Yunzhou ; Zhao, Qingyang ; Gao, Liang, 2015 Virtual
keyboard: A human-computer interaction device based on laser and image
processing, Virtual keyboard: A human-computer interaction device based on laser
and image processing, College of Information Science and Engineering, North-
eastern University, Shenyang, China

[3]Jun Hu, Guolin Li, Xiang Xie, Zhong Lv, and Zhihua Wang, Senior Member,
IEEE:Bare-fingers Touch Detection by the Buttons Distortion in a Projector
Camera System