OBJECT TRACKING OF MOBILE ROBOT USING IMAGE PROCESSING

LIM TEIK YEE

This thesis is submitted in fulfillment for the Requirement for the award of the degree of Bachelor of Engineering (Electrical - Mechatronics)

FACULTY OF ELECTRICAL ENGINEERING UNIVERSITY TECHNOLOGY MALAYSIA

MAY 2011

I acknowledge that I have studied this piece of work and in my opinion it is in accordance with the scope requirement and quality for the purpose of awarding the Bachelor Degree in Electrical Engineering (Mechatronic)

Signature Date

: .. : 1-6-2011

Name of Supervisor : MR. JOHARI HALIM SHAH BIN OSMAN

DECLARATION

I declare that this thesis entitled Object Tracking of Mobile Robot with Image Processing is the result of my own research except as cited in the references. The thesis has not been accepted for any degree and is not concurrently submitted in candidature of any degree.

Signature

: ...................................................................... Name Date : : LIM TEIK YEE MAY 2011

ii

DEDICATION

Specially dedicated to My beloved parents, siblings and friends.

iii

ACKNOLEDGEMENT

First of all, I would like to send my heartily appreciation to my project supervisor, Prof. Dr. Johari Halim Shah bin Osman for his guidance throughout this two semester. With his support and guidance, this project is able to finish in time.

At the other side, I would also like to thanks my course mates who are also a Robocon team member for his technical knowledge of software and hardware. With their expertise, I solved hardware technical problem and learn more programming skills. .

Lastly, I would like to thank my family whose is always morally support me. Thank for their motivation, I manage to go through every difficulties I had faced.

iv

ABSTRACT

This project is to develop a mobile robot with computer vision. The robot is a flat base robot which can mount a laptop and the camera. It consists of two brush motors which responsible for the robot movement on the ground. The image is captured by using a low cost webcam. The system is expected to track a single object based on objects color characteristic and keep it in the center of view. This technique can be archived by calculating the target X coordinate. Such value can use to control motor power output and direction.

ABSTRACK

Projeck ini bertujuan untuk membina satu mobile robot dengan computer vision. Robot ini adalah robot dengan tapak mendatar dan boleh membawa laptop. Gambar ditangkap dengan menggunakan webcam kos rendah. Sistem ini dijangka boleh mengikut satu objek berdasarkan sifat color objek tersebut. Ia sentiasa mengekalkan object di tengah penglihatan. Teknik ini boleh dicapai dengan mengira koordinat X target. Nilai ini kemudiannya menentukan kuasa keluaran dan arah motor.

vi

TABLE OF CONTENT

CHAPTER

TITLE

PAGE

DECLARATION DEDICATION ACKNOWLEDGEMENTS ABSTRACT ABSTRAK TABLE OF CONTENTS LIST OF FIGURES LIST OF TABLE LIST OF APPENDICES

i ii iii iv v vi x xi xiv

INTRODUCTION 1.1 Background 1.2 Problem Statement 1.3 Objective 1.4 Scope 1.5 Thesis Organization

1 1 1 2 2 2

vii

LITERATURE REVIEW 2.1 2.2 2.3 2.4 Image Processing Object Tracking of Mobile Robot Kalman Filter Background Subtraction Methods 3 4 5 5 7 8 10 13

2.5 Smoothing 2.6 2.7 2.8 RGB (Red Green Blue) Lego Pan Tilt Camera and Objects Tracking Conclusion of Literature Review

METHODOLOFY AND APPROACH 3.1 3.2 Mobile Robot System Methodology and Approach

14 14 16 17 18 18 20 21 22 23 23 25 26 26 27 28 29

3.3 Hardware Design 3.4 Hardware Components 3.4.1.1 DC Motor with Encoder MO-SPG-30E-200K 3.4.1.2 3.4.1.3 3.4.2 Camera 3.4.3 Microcontroller 3.4.4 Sensor 3.4.5 Power Supply 3.4.6 Motor driver 3.4.7 UART 3.5 Circuit Diagram 3.5.1 3.5.2 Microcontroller Connection Motor Driver Connection State Diagrams and Waveform Pin Description

viii

3.5.3 Infra Red Sensor Connection 3.5.4 Circuit Board

31 31 33 33 33 33 34 34

3.6 Software Design 3.6.1 MPLab 3.6.2 Tiny Bootloader 3.6.3 Microsoft Visual Studio Basic 2010

3.6.4 Image Processing 3.6.5 Graphic User Interface (GUI) Development 3.6.6 3.6.7 3.6.8 USB Camera Detection Mode Selection Color Filter Mode Selection

35 36 36 37 38 39 39

3.6.10 Coordinate of Detected Object Center 3.6.11 Video Source Player 3.6.12 Picture Box 3.7 Flow Diagram

RESULT 4.1 Hardware Result 4.2 Software Result 4.2.1 Color Tracking 4.2.2 Edge Filter

41 41 43 43 44 46 46 46 47 47

FUTURE WORK AND CONCLUSION 5.1 Future Work 5.1.1 Hardware Improvement 5.1.2 Software Improvement 5.2 Conclusion

ix

REFERENCES APPENDIX

49 51

LIST OF FIGURE

FIGURE

TITLE

PAGE

2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 3.1 3.2

Flow Diagram of Image Processing Step Model underlying the Kalman Filter. Flow Diagram Motion Tracking Flow Diagram Background Subtraction Techniques Smoothing 1 Smoothing 2 Smoothing 3 Representative of Addictive Color Mixing Flow Diagram of Color Tracking Example of Color Based Tracking Stereo Vision Robot Top View Stereo Vision Robot Side View Stereo Vision Robot with Webcam Installed Graphic User Interface Color Filtering System Overview Flow Diagram of System Overview 1

4 5 6 6 7 7 8 8 9 10 10 11 11 12 12 15 15

xi

3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22 3.23 3.24 3.25 3.26 3.27 3.28 3.29 3.30 3.31

Flow Diagram of System Overview 2 Project Flow Top View of the Robot Base Side View of the Robot Base Front View of the Robot Base DC Geared Motor with Encoder Square Quadrature Waveform Connector Pin Descriptions. Logitech Webcam C120 Microcontroller 18F452 LM324 Sensors ATX Power Supply Unit Modified circuit for Output Voltage 5V and 12V Motor Driver L298 Main components to Build a RS 232 Microcontroller Connection Motor Driver Connection Infra red Sensors Connection Main Circuit Board Sensors Circuit Board Graphical User Interface Com Port Selection Panel Mode Selection Panel Color Filter Mode Selection Panel Webcam Device Detection Panel Coordinate Display Panel Source Code for Image Acquisition Video Source Player

16 17 17 18 18 29 21 21 22 24 24 24 25 25 26 27 28 30 31 32 32 35 35 36 36

36 37 38

xii

3.32 3.33 4.1 4.2 4.3 4.4 4.5

Picture Box Overall Flow Diagram Robot Front View Robot Side View Robot Rear View Color Tracking Edge Filter

38 39 41 42 42 44 45

xiii

LIST OF TABLE

TABLE

TITLE

PAGE

2.1 3.1 3.2 3.3 3.4 3.5 3.6

RGB Notation for Color Red State Diagram of DC Geared Motor Connector Pin Description Logic Level of RS 232 Pins Connection Description of the 18F452 Microcontroller Pins Connection Description of the L298 Motor Driver. Pins Connection Description of the LM324 OP AMP

9 18 20 25 29 30 31

xiv

LIST OF APPENDIX

APPENDIX

TITLE

PAGE

A B C D

Gantt Chart Full Circuit Schematic Microcontroller Programming Image Processing and GUI

51 52 53 59

CHAPTER 1

INTRODUCTION

1.1

Background

Object tracking has variety of use, such as security and surveillance, traffic control, video communication and compression and etc. If the amount of data is huge, video object tracking can be a time consuming process.

Object tracking colligates targets object in many consecutive video frame. If the object is moving, the colligation will become difficult especially if the speed is fast relative to the frame rate. Another difficulty is if object orientation keep changing over time.

Tracking object based on color properties is one of the quickest methods from one image frame to another. The speed of this technique makes it very attractive for near-realtime applications but due to its simplicity many issues exist that can cause the tracking to fail.

1.2

Problem Statement

Most of the cameras available in market have limited monitoring view, this is due to the cameras are stationary. If a single target needs to be monitored in all corner, it require too many cameras and is not cost effective.

Most of the cameras also need to be manually operated or else the cameras will only focus on one point.

1.3

Objective

To design a mobile robot that can follow an object base on its object color and to create a GUI that can monitor the process.

1.4

Scope

The scope of this Build a two wheels mobile robot, install camera (might be IP camera or CMOS) and infra red sensor and apply infra red sensor. While programming involves Microchip programming using MPLAB C++, Graphic User Interface is built with Visual Basic, and image processing using Aforgenet library.

1.5 Thesis Organization

In Chapter 2, there will be literature review for this project. Chapter 3 is about project overview. Chapter 4 discussed about project methodology. Hardware and software implementation are reviewed in Chapter 5 and Chapter 6. The last chapter, Chapter 7 discuss about the result and some recommendation for future work.

CHAPTER 2

LITERATURE REVIEW

This chapter will include information which had been studied related to this project. It discussed about similar project or previous research of this project. The previous works provided recommendation and suggestion to this project. This reference is referred carefully as a useful source. Most of the source is obtained from journal, article, thesis, book and internet forum.

2.1

Image Processing

Image processing is a physical process used to convert an image signal (either digital or analog) into physical image. The actual output itself can be an actual physical image or the characteristic of an image.For example, the most common type of image processing is photography.

In digital photography, the image is stored as a computer file. The file is translated using photographic software to generate actual image. The color, shading, and nuances are all captured at the time the photograph is taken the software translates this information into an image. Figure 2.1 shows the general 3 step of image processing.

By using optical scanner or digital photography, import the image.

Manipulate and analyze the image in some way. For example image enhancement and data compression

Output, the output might be altered image.

Figure 2.1 Flow Diagram of Image Processing Step

2.2

Object Tracking of Mobile Robot

Object tracking is a process tracing an object based on object properties such as color, shape, brightness or motion. It usually performed in higher application that requires the location/shape/color of the object in every frame.

For this project, object tracking using mobile robot implemented with image processing tracking technique.

By using mobile robot, the camera can change it position to track the target. Useful in surveillance. Object tracking of mobile robot consist of two main features, which is motion tracking and color tracking with several stage of algorithm such as object detection, object identification and object tracking.[7]

2.3

Kalman Filter

. Kalman Filter is introduced by Rudolf E.Kalman in 1960. It is a set of mathematic equation that provides computational means to estimate the state of process and minimize the square error.

Kalman filter estimate 3 state: past, present and future state. The filter is powerful, this three states can be estimated even when the precise nature of the modeled system is unknown.

Figure 2.2 Model Underlying the Kalman filter.

The equation of Kalman filter evolving for time k-1 to k is given by xk = Fxk-1 + Buk-1 + Wk

Where F is the state transition model which is applied to previous state x B is the control input which is applied to u W is the noise produced in the process. [12]

2.4

Background Subtraction Methods

. The moving object is identified by comparing current image frame to background model. Figure 2.3 explain the process of this method. First the

background image is set as reference image, after that the current image is capture and compare to the reference image. If any differences are found, the differences are set to white spot. Figure 2.4 visualize how this method is performed. [6]

Figure 2.3 Flow Diagram Motion Tracking Flow Diagram

Figure 2.4 Background Subtraction Techniques

However there are three limitations to these methods:

1st, it must be sturdy against changes in illumination. 2nd, it must avoid detecting insignificant moving background object such as shadow (casted by target), whether (such as rain), 3rd, the internal background model should be able to react quickly to the changes in background.

2.5

Smoothing

Smoothing is needed to improve the detection of objects, figures below show how the smoothing is made. In figure 2.5, there is snow flake in the left video frame and the right frame show that the flakes was removed. [6]

Figure 2.5 Smoothing 1

In figure 2.6, the moving tree leaves was removed using morphological processing as shown in figure 2.7.

Figure 2.6 Smoothing 2

In figure 2.7, the right video frame is more robust against illumination change compare with left video frame.

Figure 2.7 Smoothing 3

2.6

RGB (Red Green Blue)

In color tracking, standard RGB is used to determine the color detected. Every RGB color model is formed by different combination of red, green blue color as shown in figure 2.8. It is based on Young-Helmoholtz theory of trichromatic color vision which is developed by Thomas Young and Hermann Helmholtz in the early to mid 19th century. [3]

Figure 2.8 Representative of Addictive Color Mixing

During digital image processing the RGB can be represented as binary value. It can be represented in differences notation as shown in table 2.1.

Table 2.1. RGB Notation for Color Red

Figure 2.9 shows how the process of color filtering. First, when the image is captured, it compare to the RGB value wanted (normally in color tracking the value will not be set to only one value, but rather in a range, for example (1.0, 0.0, 0.0) ~ (0.8, 0.1, 0.1). After the color wanted is detected, we can set the wanted color or the unwanted colors (depending on the user) to black color (or other colors). Figure 2.10 shows example on how the color is filtered.

10

Figure 2.9 Flow Diagram of Color Tracking

Figure 2.10 Example of Color Based Tracking

2.7

Lego Pan Tilt Camera and Objects Tracking

Pan tilt camera shown in figure 2.11, 2.12 and 2.13 is a quite popular camera for people who like to build tracking camera on their own. It requires only simple electronics, stepper motor, and pan tilt equipment. This homemade pan tilt camera makes use of regular USB webcam and Lego robotic kits. The pan tilt camera doesnt need to move around, it just stay on the same spot and moving its camera to certain degree. As shown in figure 2.11 and figure 2.12, the pan module can be easily built by setting rotating platform piece. The tilt module is a block based one with a thread manipulated platform.

11

Figure 2.11 Stereo Vision Robot Top View

Figure 2.12 Stereo Vision Robot Side View

In figure 2.13, a Logitech camera is attached, giving it two degree of freedom with an interesting structure.

Figure 2.13 Stereo Vision Robot with Webcam Installed

12

This is a robot built on Lego Pan Tilt. It has quite interesting structure with 2 degree of freedom camera.

Figure 2.14 shows the GUI of the robot. This GUI provides the robot two special controls which allow controlling the camera - one controls the pan device and the second controls the tilt device.

Figure 2.14 Graphic User Interface

This robot task is to track simple object with solid color. The object detection is done quite easily utilizing image processing routines provided by the Aforge.NET framework. The result is shown in figure 2.15. [14]

Figure 2.15 Color Filtering

13

2.7

Conclusion of Literature Review

For this project, color filtering and robot with graphic user interface is useful. For color filtering, standard RGB is implemented.

14

CHAPTER 3

METHODOLOGY AND APPROACH

This chapter introduces how the project is carried at. After that, this chapter will discuss main component, mechanism and software used.

3.1

Mobile Robot System

The object tracking mobile robot consists of three main components, the image acquisition unit, computer and the mobile robot. The image acquisition unit act as eye of the robot, it receive information (capture image) from environment. The mobile robot act as body and muscle of the system, it receive command from the computer and move accordingly. The mobile robot consists of three main part, microcontroller, motor driver and motor. The computer act as brain, it receives information from eye and process the information and after that it tells the muscle what to do. So the overall process is:

Mobile robot receives various information from the sensors such as infra red and camera, sending the information to the computer. The computer will then detect the object shape and color and then calculate the distance between the target and mobile robot. Finally the computer will send a signal that determines the robot next move. Figure 3.1 shows the communication between the hardware.

15

Figure 3.1 System Overview

In this project, the computer and the mobile robot will be connected by cable to prevent data lost. The speed of motor will be controlled using PWM (Pulse Modulation Width).

Figure 3.2 shows communication of the main parts in this project.The camera is not directly connected to the microcontroller but needed to be processed by the computer first.

Figure 3.2 Flow Diagram of System Overview 1

16

Figure 3.3 show that if extra sensors are needed, such system can be applied. But taking consideration of the microcontroller memory, system in figure 3.3 can be improved into figure 3.3 systems. The uses of the sensors are to maintain the target distance.

Figure 3.3 Flow Diagram of System Overview 2

3.2

Approach

This project is started by designing the mechanical hardware. After that proceed with circuit design followed by software development. The next step is hardware and circuit construction. The last phase is testing, tunning and improvement as shown in figure 3.4.

17

Figure 3.4 Project Flow

3.3

Hardware Design

The robot base is built with tow hollow bar and two L bar as shown in figure 3.5. Each with 40 cm long, and a 34cm x 38cm Perspex(with 0.5cm thickness). In figure 3.6, two servowheel (3.5cm Radius) is used. While figure 3.7 shows two 2.5cm castor is used. The robot is 40cm x 40 cm x 2.5 cm (length x width x height).

Figure 3.5 Top View of the Robot Base.

18

Figure 3.6 Side View of the Robot Base.

Figure 3.7 Front view of the robot Base.

3.4

Hardware Components

This part discussed main component used.

3.4.1.1 DC Motor with Encoder MO-SPG-30E-200K

It is decided to use DC geared motor with encoder (17 revolutions per minute with 0.784 Nm torque) as shown in figure 3.8. This DC geared motor is typically

19

used in wide electrical appliance such as label printer, auto shutter welding machine, grill, oven etc. It runs on 12 volt, giving out 1.1 Watt output power, producing 17RPM speed and with rated 0.41 Ampere current. It is equipped with 5V Quadrature Hall Effect encoder that monitoring the position and direction of the encoder. The resolution of the encoder output is 12 counts per rear shaft revolution or 2400 counts per main shaft revolution. The motor is purchased in Cytron, please visit following webpage for further information http://alturl.com/u2juz (this is a shorten URL).

Figure 3.8 DC Geared Motor with Encoder MO-SPG-30E-200K

The features of quadrature hall effect encoder is it can operate from 4.5V to 5.5V, it has two digital outputs (quadrature waveform), it is small in size and light in weight. It has high resolutions with 12 counts per rear shaft revolution where:

240 counts per main shaft revolution for 1:20 geared motor 360 counts per main shaft revolution for 1:30 geared motor 720 counts per main shaft revolution for 1:60 geared motor 1800 count per main shaft revolution for 1:150 geared motor 2400 count per main shaft revolution for 1:200 geared motor 3600 count per main shaft revolution for 1:300 geared motor

Two DC motors will be used, to control the motor direction. For further

20

information please refer to the user manual pdf which can be downloaded in http://alturl.com/u2juz.

3.4.1.2

State Diagrams and Waveform

Table 3.1 show the signal produced by channel A and B when the robot is moving forward or backward. Depend on how the motor is install, clockwise rotation can be either moving forward or moving backward. The phases of clockwise rotation are reverse of the counter clockwise rotation phases (phase 1,2,3,4 of clockwise rotation is equal to phase 4,3,2,1 of counter clockwise rotation. So by reading the signal from channel A and B phase by phase, the distances traveled and also the direction of motor rotation can be determined. Figure 3.9 shows how the phases in Table 3.1(a) displayed in waveform.

(a) (b) Table 3.1 State Diagram of DC Geared Motor (a) Clockwise (b) Counter Clockwise

21

Figure 3.9 Square Quadrature Waveform for Channel A and B (Clockwise).

3.2.1.3

Pin Description

Figure 3.10 shows the configuration of the pins. Starting from left is Motor (motor voltage input), Motor + (motor voltage input), VCC (voltage supply for encoder), GND (encoder ground), A (channel A), B (channel B). Depending on the voltage input (from motor driver), Motor and Motor + will determine the direction of motor rotation. While VCC and GND enable the encoder inside the motor to operate. Channel A and Channel B is a signal output that will be received by microcontroller. Table 3.2 lists down the description of the pin (taken from the manual).

Figure 3.10 Connector Pin Descriptions.

22

Table 3.2 Connector Pin Descriptions

3.4.2

Camera

For this project, Logitech Webcam C120 is used. It is a CMOS camera with USB 2.0 UVC driverless interface. The camera has 1.3 megapixels when capturing image, 640*480 pixels when capturing video. Lastly it has frame rate up to 30 frames per seconds. The focus of the camera can be adjusted by turning the ring located at the outer part of the lens. Figure 3.11 shows the camera used.

Figure 3.11 Logitech Webcam C120

23

3.4.3

Microcontroller

For this project, microcontroller is chosen. This is because microcontroller is smaller in size, cost effective and lower power consumption. In microcontroller family, 18F452 is chosen to control the movement of the robot. 30F4011 is a high speed 16-bit digital signal controller, operating within 2.5V to 5V. Microcontroller receive signal from sensors and computer. From the data it received, it controls the motor. Figure 3.12 show the pin configuration of 18F452.

Figure 3.12 Microcontroller 18F452 Pin Diagram

Another important task of this microcontroller is speed control. The microcontroller comes with internal clock which can generate a PWM signal. The PWM signal is fast on-off signal, it act like a switch that rapidly turn on and off. By controlling the speed of the PWM, the microcontroller decides how long and how fast the motor will be turned on and directly controls the speed.

3.4.4

Sensor

The sensors used in project are to maintain the distance between target and the robot. The sensor part consists of two main components, the operational amplifier

24

and infra red sensor. The operational used in this project is LM324, a commonly used IC that consists of 4 operational amplifier. Figure 3.13 show the pin configuration of LM 324.

Figure 3.13

LM324

The about 4 infra red sensors will be used in this project. Each infra red sensor comes in pair, a transmitter and a receiver. Basically the transmitter will emit infra red and the receiver detect it by changing own resistance value. The receiver of the IR sensor is LDR (light dependent resistor), also known as photo resistor. Normally, this kind of receiver will have it resistance value dropped when exposed to light. Figure 3.14 shows the infra red transmitter (the blue ones) and the receiver (the black ones)

Figure 3.14

Infra Red Sensor

25

3.4.5

Power Supply

For this project, power supply to microcontroller and motor will come from ATX power supply unit (PSU) which is commonly used in old and discarded computer (single core processor).

Due to its built in current protecting feature, this power supply need to be modified. This modified PSU is chosen because it has high current output, short circuit protection and very tight voltage regulation. Figure 3.15 shows the PSU used and figure 3.16 shows the modified circuit for output voltage 5V and 12V which is used together with the PSU.

Figure 3.15 ATX Power Supply Unit

Figure 3.16 Modified circuit for Output Voltage 5V and 12V.

26

3.4.6

Motor Driver

The L298 is an integrated monolithic circuit in a 15-lead Multiwatt and PowerSO20 packages. It is a high voltage, high current dual full-bridge driver designed to accepts standard TTL logic levels and drive inductive loads such as relays, solenoids, DC and stepping motors. Two enable inputs are provided to enable or disable the device independently of the input signals. The emitters of the lower transistors of each bridge are connected together rand the corresponding external terminal can be used for the connection of an external sensing resistor. An additional supply input is provided so that the logic works at a lower voltage.-Figure 3.17 shows the pin configuration of motor driver L298.

Figure 3.17 Motor Driver L298

The specification of this full bridge motor driver is it has operating voltage up to 46V, total DC current up to 4A, low saturation voltage with over temperature protection, logical 0 input voltage up to 1.5V.

3.4.7 UART (Universal asynchronous receiver/transmitter)

The purpose of UART is to act as communicator between computer and mobile robot. The tricky part in here is the way to avoid data loss. For this project, a RS232 receive protocol will be built using MAX232. MAX 232 is an IC that convert

27

signal from RS232 serial port to signal suitable for use in TTL (transistor to transistor logic) compatible digital logic circuit. Table 3.3 shows the logic level of RS232 while figure 3.18 show the components needed to build the UART.

Table 3.3 Logic Level of RS 232

Figure 3.18 Main components to Build a RS 232: MAX232, Capacitor 104uF x 5, PC D89 Female

3.5

Circuit Diagram

This part will show the circuit connection. The circuit is manually soldered on donut board using solder gun, solder paste and solder lead. No PCB board involved. Wrapping wire are used as jumper to connect electronics components and pins..

28

3.5.1

Microcontroller Connection

Figure 3.19 below shows the connection of microcontroller. The microcontroller is connected to the 10 GHz crystal. It receives signal from IR sensor and the computer (through UART). All LEDs is act as indicator (to see whether there is output signal or input signal successfully received or transmitted). The microcontroller receives 5 V of voltage supply from a regulator. The four output of microcontroller is sent to the L298 motor driver. Table 3.4 shows the pins connection description of the microcontroller.

Figure 3.19 Microcontroller Connection

Pin 1 2

Description Master Clear Switch

29

3 4 5 6 7 13 14 15 16 25 26

Send signal to L298 Input 1 Send signal to L298 Input 2 Send signal to L298 Input 3 Send signal to L298 Input 4 Receive signal from IR sensor Connected to crystal which supply the frequency needed. Connected to crystal which supply the frequency needed. Supply the pulse width modulation to L298. Supply the pulse width modulation to L298. Transfer signal to UART. Receive signal from UART.

Table 3.4 Pins Connection Description of the 18F452 Microcontroller.

3.5.2 Motor Driver Connection

Figure 3.20 show the shows the connection of the motor driver. The motor driver receives four signals from microcontroller. These four signals is used to determine the direction of the two motors rotation. The motor driver also receive PWM signal from microcontroller, this PWM signal is act as switch of the motor driver output. Table3.5 shows the pins connection description of the L298 motor driver.

30

Figure 3.20 L298 Motor Driver Connections

Pin 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

Description Ground Motor 1 + Motor 1 Receive 12 V voltage supply (for motor). Receive signal from microcontroller. Receive PWM from microcontroller. Receive signal from microcontroller. Ground. Receive 5 V voltage supply (for motor driver) Receive signal from microcontroller. Receive PWM from microcontroller. Receive signal from microcontroller. Motor 2 + Motor 2 Ground

Table 3.5 Pins Connection Description of the L298 Motor Driver.

31

3.5.3

Infra Red Sensor Connection

Figure 3.21 shows the connection of infra red sensor. This is an active low configuration (the resistance of receiver will drop when exposed to infra red, low resistance will have low voltage output, the receiver symbol is the circled diode symbol). The LED D9 act as indicator while D8 is the infra red transmitter.

Figure 3.21 Infra Red Sensor Connection

Pin 1 2

Description Output signal send to microcontroller Motor 1 +

Table 3.6 Pins Connection Description of the LM324 OP AMP.

3.5.4

Crcuit Board

Figure 3.22 shows the main board. The main board is mainly consist of voltage regulator, the microcontroller and motor driver. Label A is connector to

32

power supply (fPSU unit), Label B is connector to UART, the two motor and Part D is connector to infra red sensor.

Part C is connector to

A B C

Figure 3.22 Main Circuit Board

Figure 3.24 shows circuit board of infra red sensor. Label A is connected to power supply (from main board) and Label B send the output signal to microcontroller.

+ Figure 3.23 Sensors Circuit Board

33

3.6

Software Design

This part will discussed about software used to development the GUI and microcontroller programming.

3.6.1 MP Lab

For this project, to compile and write the C++ language (which wrote into microcontroller), MP Lab is used. Mp Lab is a free software which can obtained from the internet.

In this project, for microcontroller, there is two main parts. One is the communication between the computer and UART, and another one is to act accordingly to the signal receive (main program).

3.6.2

Tiny Bootloader

Tiny Bootloader is a soft ware that load hex file and burn it into microcontroller. Normally, it is used together with MP Lab.

3.6.3

Microsoft Visual Studio Basic 20103 Microsoft Visual Studio Basic (VB) is a integrated development environment (IDE) from Microsoft. It is commercial software available with seven languages: English, French, German, Italian, Japanese, Korean, and Spanish. Visual Basic is a popular IDE due to it interface and also widely used Window OS platform.

34

Visual Basic is derived from Basic language. Microsoft provide VB express edition for no cost. VB come with intellisense function which is very handy for beginner.

3.6.4

Image Processing To detect camera from USB, the Microsoft DirectShow library is used. For this library, COM object programming interface such as graph filter is used.

In image processing, there is many libraries, for example, Open CV, Aforge.net, EMGU CV and etc. For this project, Aforge.Net framework is used. Aforge.Net is an opens source and free library which can be downloaded from the internet. This framework is developed by Andrew Kirillov. Aforge.Net is a artificial intelligence and computer vision library.

The framework includes support for computer vision, artificial intelligence, neural networks, genetic programming, fuzzy logic, machine learning, image processing.

For this project, only imaging library is used.

3.6.5

Graphic User Interface (GUI) Development

This part discussed about graphical user interface developed by using Visual Basic. Figure 3.24 shows the appearance of the GUI.

35

Figure 3.24 Graphical User Interface

3.6.6

USB Camera Detection

In the panel shown in figure 3.25, the GUI will detect all available and connected USB port. The detection will start at two condition, 1st is upon the GUI is loaded, second is upon the Detect Port button is pressed. If any of the USB port(s) is (are) connected to the UART. After that, the connected port(s) will be listed down on the combo box list. Choosing it and press the connect button to start connect. Only 1 port is allowed to be connected at one moment. Information of the port will be displayed on the rich text box. Click the connect/disconnect button to connect or disconnect. It also displays the baud rate.

Figure 3.25 Com Port Selection Panel

36

3.6.7 Mode Selection

The panel shown in figure 3.26 allows the user to select mode. There are two modes for this project, color filtering and edge filter. Color Filter enable user to track single color object. Upon this mode, the tracked object will be displayed in the picture box (7) and highlighted with a green box, while other color will be converted to black color. Under edge filter mode, the system will filter all color, leaving only edge. The edge is white in color while other will be converted to black color. Only one mode can be choose at one moment.

Figure 3.26 Mode Selection Panel

3.6.8 Color Filter Mode Selection

In the panel shown in figure 3.27, user can choose filter inside or filter outside mode and select color range for color filter. Any kind of color is made up by this basic three colors, red, green, blue. By setting the values inside the text boxes in different combination, the user can choose color wanted. Filter inside is the default mode, in this mode, the filter retain the color set in the text boxes. In filter outside mode, the filter will retain color which is not in the range set in the text boxes.

Figure 3.27 Color Filter Mode Selection Panel

37

3.6.9

Webcam Device Detection

The panel shown in figure 3.28 displays information of USB webcam which is connected to the computer. It displays the connection status and the device name of the webcam. The GUI automatically detects the webcam when the GUI is loaded. Only one webcam can be detected at one moment.

Figure 3.28 Webcam Device Detection Panel

`3.6.10

Coordinate of Detected Object Center

The panel shown in figure 3.29 displays the target object center coordinate. This information is useful to determine which direction the robot is heading to. The robot will try to keep the object in the middle of the sight.

Figure 3.29 Coordinate Display Panel

3.6.11

Video Source Player

38

This video source player user interface (UI) is imported from Aforge.Net library to Visual Basic. It is capable of detect the video source from the webcam (image acquisition) and displaying image. Figure 3.30 show the complete source code for displaying data file.

Figure 3.30 Source Code for Image Acquisition.

Figure 3.30 shows the video source player user interface. Pressing the Load Image button to start the video source player while pressing the Stop Loading button to stop the video source player.

Figure 3.31 Video Source Player

39

3.6.12

Picture Box

Picture box shown in figure 3.31 will displays any filtered image or result. This picture is modified picture box by Aforge.Net. It is similar to picture box available in Visual Basic but this picture box is much more compatible with video image.

Figure 3.32 Picture Box

3.7

Flow Diagram

The flow diagrams in figure 3.33 show how the robot will react. This is programmed in the microcontroller. There is two factors influencing the reaction of the robot, first is the horizontal coordinate of the target, second is the signal condition of the infra red. After the robot is started, the robot will detect the wanted object. After the target is founded, it will calculate the position of the target and send a string to microcontroller. (For example, if the target on the right side, it will send the right string). To determines whether the robot moving in front or backward, it depends on the signal from infra red.

40

Figure 3.32 Overall Flow Diagram

41

CHAPTER 4

RESULT

This chapter discusses the result obtained, the hardware implementation and color tracking features.

4.1

Hardware Result

Figures 4.1 shows that the robot base install with a laptop and the circuit. In order to mount a laptop, the base is made in large size and flat.

Figure 4.1 Robot Front View

42

In Figure 4.2 and Figure 4.3, label A is the sensor circuit which place in the robot front. Label B is the main circuit. Label C is the webcam. The hardware is successfully installed. But it seem like there is a problem with the communication problem between the UART and microcontroller. As for result, the robot failed to move correctly.

Figure 4.2 Robot Side View

Figure 4.3 Robot Rear View.

43

The problem in hardware failure is most probably in the main circuit board and the microcontroller. The signal output send from the computer has been checked with LCD monitor and the result is desired output. It is found that the problem reside in the microcontroller and poor circuitry.

4.2

Software Result

4.2.1

Color Tracking

The software implementation is successful and the output result is the desired result. When the desired color object is detected (foe this example, red), it will be highlighted in a green rectangular. The center coordinate of the object is obtained by halving the length and height of the green rectangular and add with the upper left coordinate of the green rectangular. The direction of the robot is display at the left of object center coordinate panel. This direction is determined by comparing the center horizontal axis coordinate of the webcam with X value of the object center coordinate. As been mention is previous chapter, this color tracking is not limited to red color, it can be adjusted to any color by changing the value in the filter setting panel. Figure 4.4 shows the result of color tracking.

44

Figure 4.4 Color Tracking

4.2.2

Edge Filter

Edge filter in Figure 4.5 below is an optional feature for this project. The filter implements convolution operator, which calculates each pixel of the result image as weighted sum of the correspond pixel and its neighbors in the source image. The weights are set by convolution kernel. The weighted sum is divided by Divisor before putting it into result image and also may be thresholded using Threshold value.

45

Figure 4.5 Edge Filter

Convolution is a simple mathematical operation which is fundamental to many common image processing filters. Depending on the type of provided kernel, the filter may produce different results, like blur image, sharpen it, find edges, etc.

46

CHAPTER 5

FUTURE WORK AND CONCLUSION

5.1

Future Work

5.1.1

Hardware Improvement

For more advance hardware, IP camera can be installed to the system, replacing current camera. IP camera is a wireless system, enable lesser circuitry and wiring.

The robot size is also a problem, too big is not suitable for tracking and will greatly reduce the movement speed and direction changing speed. The robot movement is also not flexible and limited for certain space only. It can be improve by applying wireless communication between the microcontroller and the computer, by doing so, the computer is not required to be placed on the robot and the robot size can be greatly reduce. This will sharply enhance the robot mobility and quickness of the robot, making it a much better land based tracking robot. Capable of moving into narrow space, faster direction changing and higher movement speed, and the user doesnt need to follow the robot in order to monitor the process. With this enhancement, the robot will be more suitable for military use, smaller size make it harder to be detected.

47

5.1.2

Software Improvement

For further improvement, color can be equipped with image recognition function. It can be used for simple tracking such as human faces, vehicle, geometric objects, hand written or printed character. The user interface can be further improved by enable the user to select target by clicking the object on the video source player. This requires shape recognition technique and better function user interface (require another library). Current GUI has limited target range, especially on target size (it automatically choose bigger target, the ambiguity increase when there is object with similar size), so by enable on screen object selection, user can avoid this problem and able to choose the target with less limitation and more accurate. The GUI should also equipped with video recording function, the image capture is able to save in video or picture format file (such as .avi and .bmp), this information is useful for surveillance purpose.

5.2

Conclusion

This project contains three parts, Image acquisition, inference unit and positioning unit. Each part is responsible Image acquisition unit can is involvement of webcam in grabbing video frame. Inference unit handle the image processing and graphical user interface with Visual Basic programming. Creation of GUI enable the device is usable by everyone. Lastly the positioning unit which consist of microcontroller (programmed in C language), by substitute powerful servo motor over the current dc motor, the robot can be more lightweight.

It is important that the robot small in size, so that it will cost less and occupy lesser space and faster to assemble. The image processing and GUI should be multifunction and user friendly. A camera that can perform multi task, such as video capturing, photographing, face/finger print recognition is better than three cameras with single task, especially in the aspect of cost and convenient.

48

Overall this project is partially successful, due to the failure in communication between microcontroller and computer. But in view of camera capability and image processing, part of objective scope is fulfilled. Further research in image processing field will help human better in process the information in image.

49

REFERENCES

[1]

Chee Pei Song (2010). Object Tracking Camera. Degree of Bachelor. Universiti Teknologi Malaysia, Skudai. Pages 70-80.

[2]

Lin Rui, Duzhijiang, He Fujun, Kong MIngxiu and Sun Lining(2008). Tracking a Moving Object with Mobile Robot Based On Vision. Pages 23.

[3]

Sanghoon Kim , Sangmu Lee, Seungjong Kim(2008). Object Tracking of Mobile Robot using Moving Color and Shape Information for the aged walking. A. Pages 56-67

[4]

Kai-Tai Song and Wen-Jun Chen(2007). Face Recognition and Tracking for Human-Robot. Interaction Department of Electrical and Control Engineering National Chiao Tung University Hsinchu, Taiwan, R.O.C.

[5]

Keita Itoh, Takashi Kikuchi, Hiroshi Takemura and Hiroshi Mizoguchi(2008). Development of a Person Following Mobile Robot in Complicated Background by Using Distance and Color Information. Tokyo University of Science 2641 Yamazaki Noda-shi Chiba 278-8510 Japan

[6]

Mohammed Asief Brey, The Segmentation and Tracking of individuals in an indoor video surveillance environment,2007 Christian Schlegel, Jorg Illmann, Heiko Jaberg,Matthias Schuster, Robert Worz(2003). Vision Based Person Tracking with a Mobile Robot. Research

[7]

50

Institute for Applied Knowledge Processing (FAW)PO-Box 2060, D 89010 Ulm, Germany.

[8]

Greg Welch and Gary Bishop(2006). An Introduction to the Kalman Filter. TR 95-041 Department of Computer Science University of North Carolina at Chapel Hill Chapel Hill, NC 27599-3175.

[9]

J. Canny(1983). Finding edges and lines in images. Technical Report AI-TR-720, MIT Artificial Intelligence Lab. M. Sullivan, C. Richards, C. Smith, O. Masoud, and N. Papanikolopoulos(1995.). Pedestrian tracking from a stationary camera using active deformablemodels. In IEEE Industrial Electronics Society, editor, Proc. of Intelligent Vehicles

[10]

[11]

S.A. Brock-Gunn, G.R. Dowling, and T.J Ellis(1994). Tracking using colour information.In 3rd ICCARV

[12]

http://en.wikipedia.org/wiki/Kalman_filter.

[13]

http://www.edaboard.com/

[14]

http://www.aforgenet.com/forum/

[15]

http://social.msdn.microsoft.com/Forums/en-US/Vsexpressvb/threads

51

APPENDIX A

Table 1 PSM 1 Gantt Chart

52

APPENDIX B

FULL CIRCUIT SCHEMATIC

53

APPENDIX C

MICROCONTROLLER PROGRAMMING

//*********************************************/ //* Include Header */ //*********************************************/ #include <p18f452.h> #include <delays.h> #include <usart.h> #pragma config OSC=HSPLL #pragma config OSCS=OFF #pragma config PWRT=OFF #pragma config BOR=OFF #pragma config WDT=OFF #pragma config CCP2MUX=ON #pragma config STVR=OFF #pragma config LVP=OFF #pragma config DEBUG=OFF //*********************************************/ //*********************************************/ //* Define */ //*********************************************/ #define ENB LATCbits.LATC1 #define ENA LATCbits.LATC2 #define IN1 LATAbits.LATA1 #define IN2 LATAbits.LATA2 #define IN3 LATAbits.LATA3 #define IN4 LATEbits.LATE1 #define Startled LATEbits.LATE0 #define IR PORTAbits.RA5 #define ChAR PORTBbits.RB1 #define ChBR PORTBbits.RB0 #define Start PORTAbits.RA0

54

#define PWM1 #define PWM2

CCPR1L CCPR2L

//*********************************************/ //*********************************************/ //* Function Prototype */ //*********************************************/ void Init(void); void Delay(unsigned long uldelay); void Goright(void); void Goleft(void); void Gomid(void); void Stop(void); //*********************************************/ //*********************************************/ //* Variable */ //*********************************************/ char temp[]; char usart=0; void rx_handler (void); //*********************************************/ //*********************************************/ //setting interrupt vector //*********************************************/ #pragma code rx_interrupt = 0x8 void rx_int (void) { _asm goto rx_handler _endasm } //interrupt subroutine //========================================================= #pragma code #pragma interrupt rx_handler void rx_handler (void) { while (!DataRdyUSART()); temp[0]=RCREG;

55

switch(temp[0]) { case 'R': Goright(); break; case 'L': Goleft(); break; case 'M': Gomid(); break; case 'S': Stop(); break; } usart=1; //clear the flag bit PIR1bits.RCIF = 0; }

//*********************************************/ //* Main Function */ //*********************************************/ void Init(void) { TRISA = 0b00100001; TRISB = 0b00000011; TRISC = 0b00000000; TRISD = 0b00110000; TRISE = 0b00000000;

//

PWM T2CON = 0b00000101; PR2 = 0xFF; CCP1CON = 0b00001100; CCP2CON = 0b00001100;

//timer2 used for pwm //set up PWM //PWM //PWM

// // // // // // //

UART setting through library OpenUSART( USART_TX_INT_OFF & USART_RX_INT_OFF & USART_ASYNCH_MODE & USART_EIGHT_BIT & USART_CONT_RX & USART_BRGH_LOW, 1);

56

// // // // // }

Interrupt RCONbits.IPEN = 1; IPR1bits.RCIP = 1; INTCONbits.GIEH = 1;

void main(void) { Init(); Startled=1; while(1) { if (Start !=1) { while(1) { while (!DataRdyUSART()); temp[0]=RCREG; switch(temp[0]) { case 'R': Goright(); break; case 'L': Goleft(); break; case 'M': Gomid(); break; case 'S': Stop(); break; } usart=1; if (Start ==0) Stop(); } } } } void Delay(unsigned long uldelay) { for( ; uldelay > 0; uldelay--); }

57

void Goleft() { PWM1=255; PWM2=150; if (IR =! 1) { IN1 = 0; IN2 = 1; IN3 = 0; IN4 = 1; } if (IR ==1) { IN1 = 1; IN2 = 0; IN3 = 1; IN4 = 0; } } void Goright() { PWM1=150; PWM2=255; if (IR =! 1) { IN1 = 0; IN2 = 1; IN3 = 0; IN4 = 1; } if (IR ==1) { IN1 = 1; IN2 = 0; IN3 = 1; IN4 = 0; } } void Gomid() {

58

PWM1=255; PWM2=255; if (IR =! 1) { IN1 = 0; IN2 = 1; IN3 = 0; IN4 = 1; } if (IR ==1) { IN1 = 1; IN2 = 0; IN3 = 1; IN4 = 0; } } void Stop() { IN1 = 0; IN2 = 0; IN3 = 0; IN4 = 0; PWM1=0; PWM2=0; }

59

APPENDIX D

GUI AND IMAGE PROCESSING

GRAPHIC USER INTERFACE Imports AForge.Video.DirectShow Imports AForge.Imaging.Filters Public Class Form1 ' create filter Dim colorFilter As New ColorFiltering() Dim WithEvents serialPort As New IO.Ports.SerialPort Dim image As Bitmap Private Sub Button1_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button1.Click load_device() GroupBox1.Enabled = False Label16.Visible = True Button3.Enabled = False GroupBox5.Enabled = False

End Sub Private Sub Form1_HandleDestroyed(ByVal sender As Object, ByVal e As System.EventArgs) Handles Me.HandleDestroyed VideoSourcePlayer1.Stop() load_device2() VideoSourcePlayer1.Dispose() SerialPort1.Close() End Sub Private Sub Form1_KeyDown(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyEventArgs) Handles Me.KeyDown If SerialPort1.IsOpen = True Then

60

Select Case e.KeyCode Case Keys.Right ' Transmit data serialPort.Write(CChar("R")) serialPort.Write(CChar("R")) serialPort.Write(CChar("R")) e.Handled = True Exit Select Case Keys.Left ' Transmit data serialPort.Write(CChar("L")) serialPort.Write(CChar("L")) serialPort.Write(CChar("L")) e.Handled = True Exit Select Case Keys.Up ' Transmit data serialPort.Write(CChar("M")) serialPort.Write(CChar("M")) serialPort.Write(CChar("M")) e.Handled = True Exit Select Case Keys.Down ' Transmit data serialPort.Write(CChar("S")) serialPort.Write(CChar("S")) serialPort.Write(CChar("S")) e.Handled = True Exit Select End Select End If End Sub Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load Label16.Visible = False load_device2()

61

Button1.Enabled = False Button2.Enabled = False For i As Integer = 0 To _ My.Computer.Ports.SerialPortNames.Count - 1 ComboBox1.Items.Add( _ My.Computer.Ports.SerialPortNames(i)) Next End Sub Private Sub Button1_Click_1(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button2.Click VideoSourcePlayer1.Stop() load_device2() PictureBox1.Image = Nothing GroupBox1.Enabled = True Label16.Visible = False Button3.Enabled = True GroupBox5.Enabled = True End Sub Private Sub VideoSourcePlayer1_NewFrame(ByVal sender As Object, ByRef image As System.Drawing.Bitmap) Handles VideoSourcePlayer1.NewFrame image = VideoSourcePlayer1.GetCurrentVideoFrame If RadioButton3.Checked = True Then If Me.InvokeRequired() Then Me.BeginInvoke(New MethodInvoker(AddressOf load_color_filtered_image)) Else load_color_filtered_image() End If ElseIf RadioButton4.Checked = True Then If Me.InvokeRequired() Then Me.BeginInvoke(New MethodInvoker(AddressOf load_threshold_filtered_image)) Else load_threshold_filtered_image() End If End If End Sub

62

Private Sub TextBox4_KeyPress(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyPressEventArgs) Handles TextBox4.KeyPress Dim num As Byte If Not Char.IsDigit(e.KeyChar) Then e.Handled = True If TextBox4.Text.Length = 3 Then e.Handled = True If e.KeyChar = Chr(8) Then e.Handled = False 'allow Backspace If Byte.TryParse(TextBox4.Text, num) = False Then TextBox4.Clear() TextBox4.Focus() Else If e.KeyChar = Chr(13) Then TextBox5.Focus() 'Enter key moves to specified control End If End Sub Private Sub TextBox6_KeyPress(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyPressEventArgs) Handles TextBox6.KeyPress Dim num As Byte If Not Char.IsDigit(e.KeyChar) Then e.Handled = True If TextBox6.Text.Length = 3 Then e.Handled = True If e.KeyChar = Chr(8) Then e.Handled = False 'allow Backspace If Byte.TryParse(TextBox6.Text, num) = False Then TextBox6.Clear() TextBox6.Focus() Else If e.KeyChar = Chr(13) Then TextBox1.Focus() 'Enter key moves to specified control End If End Sub Private Sub TextBox5_KeyPress(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyPressEventArgs) Handles TextBox5.KeyPress Dim num As Byte If Not Char.IsDigit(e.KeyChar) Then e.Handled = True If TextBox5.Text.Length = 3 Then e.Handled = True If e.KeyChar = Chr(8) Then e.Handled = False 'allow Backspace If Byte.TryParse(TextBox5.Text, num) = False Then TextBox5.Clear() TextBox5.Focus() Else If e.KeyChar = Chr(13) Then TextBox6.Focus() 'Enter key moves to specified control End If End Sub Private Sub TextBox3_KeyPress(ByVal sender As Object, ByVal e As

63

System.Windows.Forms.KeyPressEventArgs) Handles TextBox3.KeyPress Dim num As Byte If Not Char.IsDigit(e.KeyChar) Then e.Handled = True If TextBox3.Text.Length = 3 Then e.Handled = True If e.KeyChar = Chr(8) Then e.Handled = False 'allow Backspace If Byte.TryParse(TextBox3.Text, num) = False Then TextBox3.Clear() TextBox3.Focus() Else If e.KeyChar = Chr(13) Then TextBox4.Focus() 'Enter key moves to specified control End If End Sub Private Sub TextBox1_KeyPress(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyPressEventArgs) Handles TextBox1.KeyPress Dim num As Byte If Not Char.IsDigit(e.KeyChar) Then e.Handled = True If TextBox1.Text.Length = 3 Then e.Handled = True If e.KeyChar = Chr(8) Then e.Handled = False 'allow Backspace If Byte.TryParse(TextBox1.Text, num) = False Then TextBox1.Clear() TextBox1.Focus() Else If e.KeyChar = Chr(13) Then TextBox2.Focus() 'Enter key moves to specified control End If End Sub Private Sub TextBox2_KeyPress(ByVal sender As Object, ByVal e As System.Windows.Forms.KeyPressEventArgs) Handles TextBox2.KeyPress Dim num As Byte If Not Char.IsDigit(e.KeyChar) Then e.Handled = True If TextBox2.Text.Length = 3 Then e.Handled = True If e.KeyChar = Chr(8) Then e.Handled = False 'allow Backspace If Byte.TryParse(TextBox2.Text, num) = False Then TextBox2.Clear() TextBox2.Focus() Else If e.KeyChar = Chr(13) Then TextBox3.Focus() 'Enter key moves to specified control End If End Sub Private Sub TextBox1_Leave(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox1.Leave

64

Dim num As Byte If Byte.TryParse(TextBox1.Text, num) = False Then TextBox1.Clear() TextBox1.Focus() End If End Sub Private Sub TextBox2_Leave(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox2.Leave Dim num As Byte If Byte.TryParse(TextBox2.Text, num) = False Then TextBox2.Clear() TextBox2.Focus() End If End Sub Private Sub TextBox3_Leave(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox3.Leave Dim num As Byte If Byte.TryParse(TextBox3.Text, num) = False Then TextBox3.Clear() TextBox3.Focus() End If End Sub Private Sub TextBox4_Leave(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox4.Leave Dim num As Byte If Byte.TryParse(TextBox4.Text, num) = False Then TextBox4.Clear() TextBox4.Focus() End If End Sub Private Sub TextBox5_Leave(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox5.Leave Dim num As Byte If Byte.TryParse(TextBox5.Text, num) = False Then TextBox5.Clear() TextBox5.Focus() End If End Sub Private Sub TextBox6_Leave(ByVal sender As Object, ByVal e As System.EventArgs) Handles TextBox6.Leave Dim num As Byte If Byte.TryParse(TextBox6.Text, num) = False Then TextBox6.Clear() TextBox6.Focus()

65

End If End Sub Private Sub RadioButton1_CheckedChanged(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles RadioButton1.CheckedChanged colorFilter.FillOutsideRange = True End Sub Private Sub RadioButton2_CheckedChanged(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles RadioButton2.CheckedChanged colorFilter.FillOutsideRange = False End Sub Private Sub Button3_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button3.Click Button4.Enabled = False Button1.Enabled = True Application.DoEvents() If Button3.Text = "Connect" Then 'Check whether serial port is initially open or not If SerialPort1.IsOpen Then SerialPort1.Close() End If If ComboBox1.Text = Nothing Then Button4.Enabled = True MsgBox("Please Choose your Comm Port", MsgBoxStyle.Critical) Else Try With SerialPort1 .PortName = ComboBox1.SelectedItem .BaudRate = 115200 .Parity = IO.Ports.Parity.None .DataBits = 8 .StopBits = IO.Ports.StopBits.One End With ' Set the read/write timeouts SerialPort1.ReadTimeout = 1000 SerialPort1.WriteTimeout = 1000 SerialPort1.Open() Button3.Text = "Connected /" & Environment.NewLine &

66

"Disconnect" Label15.Visible = True Label15.Text = "Baud Rate:115200" RichTextBox1.AppendText(Environment.NewLine) RichTextBox1.AppendText("Port Connected :" & ComboBox1.SelectedItem) Catch ex As Exception MsgBox(ex.ToString) End Try End If

ElseIf Button3.Text = "Connected /" & Environment.NewLine & "Disconnect" Then Button1.Enabled = False Button4.Enabled = True Button3.Enabled = True Button3.Text = "Connect" SerialPort1.Close() Label15.Visible = False RichTextBox1.AppendText(Environment.NewLine) RichTextBox1.AppendText("Port " & ComboBox1.SelectedItem & " disconnect ") End If End Sub

Private Sub Button4_Click(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles Button4.Click ComboBox1.ResetText() ComboBox1.Items.Clear() For i As Integer = 0 To _ My.Computer.Ports.SerialPortNames.Count - 1 ComboBox1.Items.Add( _ My.Computer.Ports.SerialPortNames(i)) Next

End Sub Private Sub Button5_Click(ByVal sender As System.Object, ByVal e As System.EventArgs)

67

SerialPort1.Close() SerialPort1.Open() Try SerialPort1.Write(CChar("L")) SerialPort1.Write(CChar("L")) SerialPort1.Write(CChar("L")) Label16.Text = "Moving to Left" Catch ex As Exception MsgBox(ex.ToString) End Try End Sub End Class

******************************************************************** DESIGNER FORM ********************************************************************

Public Sub load_device() Try Dim video_device = New FilterInfoCollection(FilterCategory.VideoInputDevice) If video_device.Count = 0 Then Throw New ApplicationException() Else For Each device As FilterInfo In video_device Label1.Text = "Device Connected" Label2.Text = "Device:" & device.Name Label3.Text = "Displaying Image" Button1.Enabled = False Button2.Enabled = True Next End If ' create video source Dim videoSource As New VideoCaptureDevice(video_device(0).MonikerString)

68

VideoSourcePlayer1.VideoSource = videoSource videoSource.DesiredFrameSize = New Size(320, 240) VideoSourcePlayer1.AutoSizeControl = False 'New Size(160, 120) ' start the video source VideoSourcePlayer1.Start() Catch e1 As ApplicationException Label1.Text = "No Local Device Detected" Label2.Text = "Device:(Not Available)" Label3.Text = "Please connect a Webcam." Button2.Enabled = False End Try End Sub Public Sub load_device2() Try Dim video_device = New FilterInfoCollection(FilterCategory.VideoInputDevice) If video_device.Count = 0 Then Throw New ApplicationException() Else For Each device As FilterInfo In video_device Label1.Text = "Device Connected" Label2.Text = "Device:" & device.Name Label3.Text = "Select Com Port" Button2.Enabled = False Button1.Enabled = True Next End If Catch e1 As ApplicationException Label1.Text = "No Local Device Detected" Label2.Text = "Device:(Not Available)" Label3.Text = "Please connect to a Webcam" Button2.Enabled = False End Try End Sub

69

Public Sub load_color_filtered_image()

' configure the filter colorFilter.Red = New IntRange(CInt(TextBox1.Text), CInt(TextBox2.Text)) colorFilter.Green = New IntRange(CInt(TextBox3.Text), CInt(TextBox4.Text)) colorFilter.Blue = New IntRange(CInt(TextBox5.Text), CInt(TextBox6.Text)) ' apply the filter Dim img1 As Bitmap = Me.Invoke(Function() VideoSourcePlayer1.GetCurrentVideoFrame) Dim objectImage As Bitmap = Me.Invoke(Function() colorFilter.Apply(img1)) ' create blob counter and configure it Dim blobCounter As New BlobCounter() blobCounter.MinWidth = 25 ' set minimum size of blobCounter.MinHeight = 25 ' objects we look for blobCounter.FilterBlobs = True ' filter blobs by size blobCounter.ObjectsOrder = ObjectsOrder.Size ' order found object by size ' grayscaling Dim grayFilter As New Grayscale(0.2125, 0.7154, 0.0721) Dim grayImage As Bitmap = grayFilter.Apply(objectImage) ' locate blobs blobCounter.ProcessImage(objectImage) Dim rects() As Rectangle = blobCounter.GetObjectsRectangles() ' draw rectangle around the biggest blob If rects.Length > 0 Then Dim objectRect As Rectangle = rects(0) Dim g As Graphics = Graphics.FromImage(objectImage) Using pen As New Pen(Color.FromArgb(160, 255, 160), 3) g.DrawRectangle(pen, objectRect) End Using g.Dispose() End If

PictureBox1.Image = Me.Invoke(Function() objectImage) If (rects.Length <> 0) Then

70

Dim objectrect As Rectangle = rects(0) Dim object_x As Integer = objectrect.X + objectrect.Width / 2 Dim object_y As Integer = objectrect.Y + objectrect.Height / 2 Me.Invoke(Function() colorFilter.Apply(img1)) Label7.Text = "X:" & Me.Invoke(Function() object_x) Label8.Text = "Y:" & Me.Invoke(Function() object_y) If (190 < object_x) Then Try SerialPort1.Write(CChar("R")) SerialPort1.Write(CChar("R")) SerialPort1.Write(CChar("R")) Label16.Text = "Moving to Right" Catch ex As Exception MsgBox(ex.ToString) End Try ElseIf (160 > object_x) Then Try SerialPort1.Write(CChar("L")) SerialPort1.Write(CChar("L")) SerialPort1.Write(CChar("L")) Label16.Text = "Moving to Left" Catch ex As Exception MsgBox(ex.ToString) End Try ElseIf (160 < object_x < 190) Then Try SerialPort1.Write(CChar("M")) SerialPort1.Write(CChar("M")) SerialPort1.Write(CChar("M")) Label16.Text = "Moving Forward" Catch ex As Exception MsgBox(ex.ToString) End Try End If Else Try SerialPort1.Write(CChar("S")) SerialPort1.Write(CChar("S")) SerialPort1.Write(CChar("S"))

71

Label16.Text = "Stop" Catch ex As Exception MsgBox(ex.ToString) End Try End If End Sub Public Sub load_threshold_filtered_image() 'create filter Dim thresholdfilter As Edges = New Edges() 'apply the filter Dim img2 As Bitmap = VideoSourcePlayer1.GetCurrentVideoFrame thresholdfilter.ApplyInPlace(img2) PictureBox1.Image = img2 End Sub Public Sub Shape_Tracking() End Sub Public Sub Motion_Tracker() ' create motion detector Dim detector As New MotionDetector(New SimpleBackgroundModelingDetector(), New MotionAreaHighlighting()) ' continuously feed video frames to motion detector ' process new video frame and check motion level If detector.ProcessFrame(VideoSourcePlayer1.GetCurrentVideoFrame) > 0.02 Then ' ring alarm or do somethng else Label16.Text = "Moving to Right" End If PictureBox1.Image = image

End Sub