Anda di halaman 1dari 89

Technical report, IDE1144 , September 13, 2011

Method for Autonomous Picking of Paper Reels

Master’s Thesis in Intelligent Systems

By:- Hasan, Meqdad Hamdan and Kali, Rahul Raj.

Systems By:- Hasan, Meqdad Hamdan and Kali, Rahul Raj. School of Information Science, Computer and Electrical

School of Information Science, Computer and Electrical Engineering Halmstad University

Method for Autonomous Picking of Paper Reels

By:- Hasan, Meqdad Hamdan and Kali, Rahul Raj.

Halmstad University Project Report IDE1144

Master’s thesis in Intelligent Systems, 15 ECTS credits

˚

Supervisor: Ph.D. Bj¨orn Astrand Examiner: Prof. Antanas Verikas

September 13, 2011

Department of Intelligent systems School of Information Science, Computer and Electrical Engineering Halmstad University

Preface

Here in a short words were words never could describe the feeling of thankful for peoples who encourage us in this work and keep us on the track when our energy drops to zero. To the people who illuminate the road for us when our ideas interned a situation of chaos. Thanks to our parents, grand parents and our brothers who were with us over seas by their hearts and their praying.

˚

Thanks to Dr. Bj¨orn Astrand who guide us during this thesis and track us on every turn of it. Thanks to the help that is provided by Dr. Walid Taha from Information and computer department and by Nadya Karginova from Intelligent systems department. Finally thanks a lot for the people who help us with little words and they are in other lands nowadays.

i

Abstract

Autonomous forklift handling systems is one of the most interesting research in the last decades. While research fields such as path planning and map building are taking the most significant work for other type of autonomous vehicles, detecting objects that need to move and picking it up becomes one of the most important research fields in autonomous forklifts field. We in this research had provided an algorithm for detecting paper reels accurate position in paper reels warehouses giving a map of the ware- house itself. Another algorithm is provided for giving the priority of papers that want to be picked up. Finally two algorithms for choos- ing the most appropriate direction for picking the target reel and for choosing the safest path to reach the target reel without damage it are provided. While working on the last two algorithms shows very nice results, build- ing map for unknown stake of papers by accumulating maps over time still tricky. In the following pages we will go in detail by the steps that we followed to provide these algorithms started from giving an over view to the problem background and moving through the method that we used or we developed and ending by result and the conclusion that we got from this work.

iii

Contents

1 Introduction

 

1

1.1 Malta

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

2

1.2 Problem

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

2

1.3 Goals .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

3

1.4 Approach

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

4

1.5 Outlines

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

5

2 Background

 

7

2.1 Problem Description

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

7

2.2 State of the Art .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

10

2.3 Closely Related Works

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

11

2.3.1

Method of autonomous material handling

.

.

.

.

.

.

.

12

2.3.2

Perception and detection

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

12

2.3.3

Map building

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

13

3 Methods

15

3.1 Path Planning .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

15

3.1.1 Kinematic model

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

15

3.1.2 Path selection .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

17

3.2 Perception .

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

19

3.2.1 LRF simulation

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

20

3.2.2 Error simulation

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

20

3.2.3 Segmentation and Feature Extraction

.

.

.

.

.

.

.

.

21

3.3 Map Building

 

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

26

3.3.1 Matching algorithm

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

26

3.3.2 Map updating and map generation

.

.

.

.

.

.

.

.

.

.

.

30

3.4 Paper Reel Selection

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

31

3.4.1 Paper availability and reel priority

.

.

.

.

.

.

.

.

.

.

.

31

3.4.2 What is the grasping direction?

.

.

.

.

.

.

.

.

.

.

.

.

.

36

v

4 Results

37

4.1 Segmentation

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

40

4.2 Extracting Map

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

42

4.3 Sum Maps into one

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

46

4.4 Map analysis

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

54

5 Discussion

63

6 Conclusions

65

Appendix

69

vi

List of Figures

1.1

Oil drum picking is one of the application examples

.

.

.

.

.

.

.

.

1

1.2

Paper reels are a second application example

.

.

.

.

.

.

.

.

.

.

1

1.3

Truck Used in MALTA Project

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

3

1.4

Wooden pallets have specific direction for picking even some times

 

its a unique.

The picking direction is perpendicular to the top

wooden plates, the opposite direction of the red row.

.

.

.

.

.

.

.

3

1.5

Loading Unknown Position Paper

4

2.1

Illustration to the problem, (Prescribed environment contain stack of paper reel needs to transport. 1- Letters show the truck in several locations (A,B and C). 2- Numbers show different paper reel in different location. 3- Arrow refer to several direction to pick. 4- Bold arrow show the best praxis exist now. 5- Black dots refer to available directions to pick. 6- Cross sign refer to the reflected laser beams.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

8

2.2

Estimation of paper reel locations from uncertain laser range finder reading. Black circles are estimated by using circle fitting algo- rithms and red ones are the true paper reels location. See the dif- ference between estimated and true ones. Picture is taken from

 

[1]

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

9

3.1

Simplification of MALTA robot kinematic model used to compute the position of the vehicle from wheel encoder information (bicycle model).

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

16

3.2

The arrangement of points that chosen to generate the path. Six points are chosen. P 1 is the original point of the robot frame. P 2 is a point in the middle of the forks. P 3 is a point in a distance of 1.5d on the robot X-axis. P 6 is the center of the paper reel. P 5 is a point on the surface of the paper reel at the same direction of picking direction. P 4 is a point on the same direction of P 5 but with 1.5d distance of the paper reel surface. ”d” refers to the robot total length. The path is generated by using B-spline cubic function. 17

vii

3.3

The path selection to transport reels from unloading area to the loading area. The path is described by 9 points. P 1 , P 2 and P 3 as points (P 4 ,P 5 and P 6 ) in Figure 3.2. 3 points (P 4 ,P 5 and P 6 ) to guide the vehicle through the container entrance. Other 3 points (P 7 ,P 8 and P 9 ) to guide the vehicle to the farthest unoccupied cell in the container.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

18

3.4

Line fitting problem.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

24

3.5

Circle fitting results. σ ρ (LRF beam error variance) here is

25

3.6

Weighted linear regression. Note there is high error in calculating start and end points using equation 3.30. σ ρ (LRF beam error variance) here is 0.05.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

25

3.7

example of matching created map with predefined one

30

3.8

Each reel has 360 accessibility angles. Instead, It is considered as interval. Thus each reel has 72 accessibility angels.

.

.

.

.

.

.

.

.

32

3.9

Each object is considered as an obstacle if the distance between that object and the reel more than 1.7 of the length of the truck until the end of the gripers

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

32

3.10

If the obstacle is some thing like wall, i.e the reel location is close to wall.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

33

3.11

Description of use of built problem to calculate the accessibility angle. The dots refer to the possible picking direction. The X’s refer to inaccessible directions.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

35

3.12

Change the presentation of accessibility angles to linear presenta- tion then remove the inaccessible angles and flag them.

.

.

.

.

.

.

36

4.1

Testing environment

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

37

4.2

The truck shape and the arrangement that is chosen to the LRF fitting on the truck. Both are fitted on the same

.

.

.

.

.

37

4.3

Scanning image from simulated LRF. The image shows that the image is relative to the scanner position

.

.

.

.

.

.

.

.

.

.

.

.

.

.

39

4.4

Simulated errors in scans

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

40

4.5

Scanning points after preprocessing.

All point are rotated and

 

translated to the global map.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

40

4.6

Extracting features and building local map. σ ρ here is 0.2. Good estimation for line segment in the bottom.

.

.

.

.

.

.

.

.

.

.

.

.

.

41

4.7

Extracting features and building local map. σ ρ here is 0.2. Note the bad estimation of line in the left

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

42

4.8

The result of building local map from circles and

.

.

.

.

.

43

4.9

Extracting features and building local map. The best estimation for close paper reels.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

44

viii

4.10 The summation of two maps into bigger one. The biggest circles represent the extraction of paper reels in first map.The smallest ones represent the extraction of them in the second map. The mid- dle one show the summation of both by using Kalman

46

4.11 The result of summation two line segments together.The algorithm sums the first two left segments but it does not add to them the third

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

47

4.12 The adding of two segments when they are verticals. Note that the result not as expected, where one can expect the result should lay somewhere between the two smallest lines but what happen, that is the new longer line segment lay out of both original

48

4.13 Other example of summation of two circle columns or paper reels from two different map using Kalman filter. The black one is the result of the summation.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

48

4.14 Summation of two local maps into one again. The bottom line segments are summed correctly together while one segment appeared in the left. This segment simply transferred to the new map without

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

49

4.15 In this figure two local map are summed together the map from

Figure 4.16 and the map from Figure 4.17.

Note the difference

between them.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

50

4.16 The local map that is extracted from first LRF at scan image num- ber 71.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

52

4.17 The local map that is extracted from second LRF at scan image number 71.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

53

4.18 The result of building the global map of combining 112 local map, 56 local map from each LRF are built from 56 scan image over 56 position of the truck.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

53

4.19 The simulation environment and the initial position of the truck.

Light circles are concrete pillars.

Dark circles are paper reels.

Lines are walls and container.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

54

4.20 The paper reels arrangements.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

55

4.21 All papers are processed then the access directions for each paper reel are determined (the magenta points).

.

.

.

.

.

.

.

.

.

.

.

.

.

55

4.22 The paper reel #13 is removed first then the rest paper reel are processed again.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

56

4.23 Paper reel number 12 is removed this time, then the map processed

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

56

4.24 Now paper reel number 26 is removed. Note two different and sep- arate intervals of accessibility direction on paper reel # 11 are ap- peared

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

57

ix

4.25

The same for paper reel number 11.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

57

4.26

And so on for all other reels until all papers are processed.

.

.

.

.

57

4.27

The picture shows picking intervals and access direction.

.

.

.

.

.

58

4.28

The path is generated by using b-spline function and by giving 10 different points to the function as input. The path calculated to each paper real from each possible picking direction then the short- est one is

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

58

4.29

All possible paths to the paper

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

59

4.30

The chosen path is the shortest one and the easiest from paths in figure 4.29.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

59

4.31

Selected paths and picking directions for each

.

.

.

.

.

.

.

59

4.32

The chosen picking direction provide safe moving for truck to not be in conflict with any other obstacle

61

4.33

Also the selected path provide safe reaching to the paper reel.

.

.

.

61

4.34

However, we could not simulate the return back movement where the rotation of truck direction happen instantaneously not exactly as in real

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

61

6.1

Ackerman Steering Principle.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

69

6.2

The paper reel in the middle is located between a wall a column.

.

71

6.3

Adding two availability together after converting the angles to linear representation to get the total available directions.

.

.

.

.

.

.

.

.

.

73

x

List of Tables

4.1 A comparison between segmentation results; ”number of effective segments and length of segments” when three values of noise re- duction constant are used.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

41

4.2 Features extraction from one LRF scan, and building local map as in fig.4.6. L i refers to line feature. C i refers to circle feature. Empty place refers to no reading available for this measure because its not available for that feature. error refers to bad

42

4.3 The uncertainty and location of features in fig.4.7. L i refers to line feature. C i refers to circle feature. Empty place refers to no reading available for this measure because its not available for that feature.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

43

4.4 The uncertainty and location of features in fig.4.8

44

4.5 The uncertainty and location of features in fig.4.9

.

.

.

.

.

.

.

.

.

45

4.6 Summation of two lines into global map,fig.4.11 a case spot light .

47

4.7 Summation of two Vertical lines. Objects in figure 4.12. A case spot light

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

.

48

4.8 result of Summation two pillars as in fig.4.13

49

4.9 The result of fuse two local map together using Kalman filter.M ap 1 represents some features from Figure 4.16.M ap 2 represents the same of them from Figure 4.17. The last part of the table show the result of summing them together as appears in Figure

51

4.10 The position, uncertainty and error of the same paper reels and column from table 4.9 after accumulating several local map. These feature are represented in a right place as appears in Figure

51

4.11 The shape of the table that generated from the algorithms described in this research from analysing a given map for paper locations. This is only a sample from the simulation environment

60

xi

Chapter 1

Introduction

In industry and other fields like mining, material handling and material trans- portation is one of the most costly activity. The cost of this activity increases if the material have to pick and transport in some kind of order or arrange- ment. The cost of order picking is estimated by 55% from warehouse op- eration expense [7]. However this cost increases in case of damages due to errors of picking. To avoid such high cost and to avoid other type of prob- lems, demands for efficient material handling with efficient picking regime solutions are growing rapidly. Examples for such a kind are, steel wire reels manufactures, Oil drums handling Figure 1.1, paper mills Figure 1.2 and other industries. The autonomous mobile robotic solution has a great poten- tial to offer improvements in terms of costs, safety, efficiency, flexibility and availability [1].

safety, efficiency, flexibility and availability [1]. Figure 1.1: Oil drum picking is one of the application

Figure 1.1: Oil drum picking is one of the application examples

1.1: Oil drum picking is one of the application examples Figure 1.2: Paper reels are a

Figure 1.2: Paper reels are a second application example

1

2

1.1 Malta

MALTA project (Multiple Autonomous forklift for Loading and Transp-

ortation Applications) [1] is a project developed in

Halmstad University in Sweden, together with Danaher Motion S¨ar¨o, Linde Material Handling, and Stora Enso Logistics. The research in this project is to develop a fully autonomous AGV (Automated Guided Vehicle) (forklift) for loading, unloading and transporting materials in highly dynamic envi- ronments. The forklift in this project was modified by replacing the two forks with clamps Figure 1.3. The system was equipped with two laser range finders for detecting obstacles and paper reels. The system also equipped with AGV control system comprises a set of hardware and software compo- nents (PC, IO modules, field bus controller, rotating laser ranger, etc.). The control system interfaces the actuators and sensors of the truck through the already built-in local CAN network [1]. The task of the AGV controller is to provide tradition functionality (navigate the truck between two location (initial and goal locations)). A system of spinning laser range finder fitted on the top of the truck canopy and reflective markers fitted around the ware- house used to determine the exact location of the truck. The path planning is done by external PC to determine the path between the current location and the target location. In the case of picking paper reels the target location is the paper reel itself. The paper reel can be detected by two laser range finder fitted in front of the truck. The truck has been operated and tested for several times. As a result of testing the truck able to pick paper reels which arranges in special way, see Figure 1.5. One can observe from the picture that truck can pick paper from only one direction. One other thing one can observe from this paper that truck can not avoid picking paper if there is two paper reels closed to each other. This cause damage in paper reels. Although the percentage of damage is small its still huge if it is calculated by money.

Orebro University and

¨

1.2 Problem

The problem of material handling is studied and researched in different ar- ticles. [1], [6], [5] are some examples where this topic have been studied. Any way non of them answer the question; ” The material have to be picked in what order?” Detection the target (in this research and in [1] is a paper reel while in both [6] and [5] it is a wooden pallet) is one of the problems. To detect the target its required to estimate its position very accurate to avoid hitting it instead of picking it. For wooden pallets which studies in [5] the order of picking them was according to the top one of them how-

Method for Autonomous Picking of Paper Reels

3

Method for Autonomous Picking of Paper Reels 3 Figure 1.3: Truck Used in MALTA Project Figure

Figure 1.3: Truck Used in MALTA Project

of Paper Reels 3 Figure 1.3: Truck Used in MALTA Project Figure 1.4: Wooden pallets have

Figure 1.4: Wooden pallets have spe- cific direction for picking even some times its a unique. The picking di- rection is perpendicular to the top wooden plates, the opposite direction of the red row.

ever his work does not provide the operator by any kind of tables about ”In what order should pallet be picked?” the same in [6]. Both of them did not meet any problem in defining the direction of picking where wooden pallets have a special and sometimes a unique direction of picking (Figure 1.4). [1] who provide MALTA truck to us studies the case of detecting paper reels (cylindrical object) in 2D environment. However, they did not answer the previous question. The main problem in cylindrical object handling is that cylindrical object (the reel) has 360 picking direction. So ”what is the best direction of piking”. These two questions: For a stack of paper reels that are needed to be moved from unloading to loading area (say from warehouse to a container) by using autonomous guided vehi- cle, what is the order of moving paper reels? What is the grasping direction for each one of them? were our research interests.

1.3

Goals

The goal of the thesis is to provide MALTA project and such other applica- tions by algorithm to help the autonomous trucks which handle cylindrical materials like paper reels and oil drums to decide what is the best picking direction. Also this thesis aims to provide these applications by algorithm to decide - for a kind of material arrangements - which piece should pick first. However, these two objectives not only what we expect from these thesis.

4

4 Figure 1.5: Loading Unknown Position Paper Reel. While we looking forward to solve the problems

Figure 1.5: Loading Unknown Position Paper Reel.

While we looking forward to solve the problems discussed above, we also looking forward to provide good algorithm to build and update the truck map from the sensor data. We aim in this thesis to reduce number of damages in paper reels by provide a safe picking algorithm. Also we aim to reduce storage area by give forklift a good filling algorithm. We also aim to save fuel consumption and increase truck maintenance period by using good ordering algorithm for piking case and container loading scenario. For these aims the expected results are lists of paper reels locations and picking order. These lists should also contain the grasping direction of each paper real and the optimal location for each (where it should put?). These results are expected to generate from predefined map, fused data from sensory data and predefined map.

1.4

Approach

For these objectives, and to study problem case, a simulation environment on matlab application is to be built. Firstly, the two objectives are to be studied, developed and to be tested on a predefined map in a static environment. Then a simulation of laser range finder and environment are to be built. Then the case of building map from sensor data are to be studied. Actually to make the problem easy to be studied, the case of dynamic environment where there are a lot of variables (such as other moving truck in the environment or the

Method for Autonomous Picking of Paper Reels

5

change in number of material pieces (paper reels in this case) by increasing or decreasing due to other truck activities) is not considered. Hopefully, this thesis can also provide a good background to other researchers in future to develop picking function to cover dynamic environment. To build and test this simulation environment different algorithms have to be used such as segmentation algorithm, line and circle detection and fitting algorithms. Kalman filter is to use to fuse data together, other necessary method are descried in detail later on in this thesis.

1.5

Outlines

This research is coming in six different chapters. Chapter 1 is the introduc- tion part. In chapter 2 we give a deeper view to the problem studied in this thesis, the best implementation for it so far and previous work related to this problem. Then in chapter 3, we describe the method that we used here. We start with describing kinematic model of our truck. Then we move to describe the mathematical models and equations that are used to build the simulator. Then we describe in some details the algorithms that used in segmentation and detection problems especially form laser scanner data.Then we describe how we build a global map using data from different laser scanners and how we update it by time. Then we move to discuss the algorithm that we develop to analyse the constructed map to find the available paper reel in the map and the best picking angle for it. After that we discuss the filling problem and the algorithm used in loading (filling) scenario. In chapter 4 we discuss the results. Then in chapter 5 we return back to our work and try to high light its break points and analyse it. Then finally in chapter 6 we give our conclusion of what we did and what we did not in this thesis.

6

Chapter 2 Background

In reality, working inside warehouses or outside, to transport cylindrical ma- terials like drums and paper reels usually done by forklifts that driven by humans. In the case of autonomous forklift, transporting materials of any kind is a very challenging work. Machines (forklifts) are not as cleaver as hu- man. In such case problems start from machine localization, dead reckoning, edge detecting, map building, path planning and decision making including control systems, sensor data understanding and etc. Our work here will con- sider one of these problems in detail and some of other problems where we cannot finish our work without considering them. In this chapter we will start by discussing our problem in more detail then we will move to mention the best available praxis and we will end by mention the previous related work to our work in this thesis.

2.1 Problem Description

Let us return back to Figure 1.2 and think a while how could an autonomous forklift handle this huge number of paper reels. How it can decide which one of them it has to pick up first? Also at what elevation? To give deeper view to the problem and understand it from its all sides let us take a look to the Figure 2.1. The figure illustrates paper reels stack in some storage area and an autonomous forklift wants to transport them one by one to inside the container. As in MALTA forklift the forklift in the figure has an LRF (Laser Range Finder) to detect paper reels and to determine their locations. The first problem appears from the figure, that the truck can detect different papers in different locations when it moves. Note how the truck can detect papers (1,2,3,4 and 6) when it is in location (A). While it can detect papers (1,2,3,4,5 and 6) when it is in location (B).

7

8

Chapter 2.

Background

8 Chapter 2. Background Figure 2.1: Illustration to the problem, (Prescribed environment contain stack of paper

Figure 2.1: Illustration to the problem, (Prescribed environment contain stack of paper reel needs to transport. 1- Letters show the truck in several locations (A,B and C). 2- Numbers show different paper reel in different location. 3- Arrow refer to several direction to pick. 4- Bold arrow show the best praxis exist now. 5- Black dots refer to available directions to pick. 6- Cross sign refer to the reflected laser beams.

Note, that paper 5 is a plus. If the truck in location (C), it can detect only papers (8,7,6,4 and 3). Note, that papers 7 and 8 appeared for the first time here while papers 1 and 2 disappeared in this situation. Note, in all locations papers in the back cannot be detected. As the truck can not figure whole stack using only the LRF sensor. So how it can decide which paper reel should be picked up in the first? let us assume that truck does not move in several free paths. Let us assume, like in Figure 1.5, that truck move in predefined and fix paths. Even in this situation the question still arises , what paper reel should pick in the first? let us assume the worst situation where the forklift in location B and it will move directly to pick reel number 5. This will cause damage in the two other reels (4 and 5). Let us also describe other dangerous situation. Assume that there is no space between reels, and truck moved to choose one of reels number 3 or 4. In this case truck will damage 2 and 4 or 3 respectively by its gripers. Of course if the driver is a human, then reel number 2 will be chosen because off removing it is easier and will make grasping other easier, also because it is the nearest

Method for Autonomous Picking of Paper Reels

9

one to the container. Again, assume we have the same dangerous situation as above, where all reels close to each other, and the decision was to pick reel number 2 for the same reasons mentioned before. If the driver for this truck is a human he will come to this reel from a safe direction then he will grasp it. In autonomous forklift if there is no algorithm to give the forklift a decision from which direction it will grasp paper reel, exactly like what happen in MALTA forklift, see Figure 1.5, the truck will move toward reel number 2 in Figure 2.1 with the direction of the bold arrow, and again that is a really dangerous situation which should be avoid. Note, such situation will damage reel number 3. The question is if there are a lot of available direction, truck can come to reel and grasp it from, (Note, the black dots on reels and the thin arrows.) which one of them is the safer? If they all are safe as each other, which one the truck should come and grasp reel from?

other, which one the truck should come and grasp reel from? Figure 2.2: Estimation of paper

Figure 2.2: Estimation of paper reel locations from uncertain laser range finder reading. Black circles are estimated by using circle fitting algorithms and red ones are the true paper reels location. See the difference between estimated and true

ones. Picture is taken from [1]

Other problem appears when using sensory data. That is sometimes errors in sensory data, see Figure 2.2, cause the same problem above even if there is a space between reels. So how could truck know the exact location of the paper reels? Finally, because off using data from LRF to detect and figure reels, and that give limited information as we mentioned above, could

10

Chapter 2.

Background

it possible there is other paper reel which is more suitable to pick-up first? In short, this is a typical example for what could happen and the expected problem in situation of loading reels. Other kind of problems appear when we move to think in filling container scenario. The main question appear here where forklift should put the second paper reel inside the container? Should it put it behind the first one or next to it? Other question also comes to the mind in this scenario when forklift start fill the second line in the container (where the first reel of this line should be? Is it exactly next to the first one of the first line or between two reels form the first line to save the area of container? How the situation will be when paper reels from different diameters? These are the problems what we try to make a solution for them in this research.

2.2 State of the Art

The most recent and the best praxis can found in some short details in [1]. As mentioned above, A. Bouguerra et. al. (2007) created a forklift used in special case; which is to transport cylindrical materials with several base diameter from storage area to load in a train wagons, containers or any other transportation means. This forklift developed to work inside warehouse in a high dynamic area. A. Bouguerra et. al. used a modified environment by pre-installed infras- tructure to solve the problem of localization. As mentioned in chapter one they used spinning LRF with reflectors that is fitted around the environment to determine the exact location of the truck. Due to the high stack of paper reel in the environment as in 1.2 they found this solution is not so sufficient. Later they suggested two solutions. The first one was by fitting reflectors under bottom of the ceiling and pointing the LRF or a camera to ceiling. The second solution, they suggested, was by using paper reels themselves as landmarks in a SLAM process. Also they suggest to address each paper reel by a unique bar -code. For the navigation function they used a predefined map uploaded to the truck by operator. This map contains a defined possible drivable paths as a collection of lines and B-splines. It also contains location of different things like dangerous places (floor drop-down elevation), storage area, loading and unloading areas and doors of the warehouse. They use an external PC to generate run time trajectory using cubic B-spline. This function allows them to provide machine with safe motion and obstacle avoidance functions. How- ever using the predefined map made some limitation (as they said). These

Method for Autonomous Picking of Paper Reels

11

limitations were, the truck not allowed to change its path at runtime. That means the truck have to stop before generate its new path. This appears in loading scenario as in Figure 1.5 where the truck has to move until point A before generate its new path to inside container or toward paper reels. For perception functions, as said before, they used two LRF to detect

paper reels and other obstacles. Later, after they publish their works, they add a third LRF in back of truck to detect and track obstacles and reels in backward motion. However the main function of perception process is to detect and track paper reels, which means, in 2D, finding circles center and diameters. They segments LRF data to several segments. They use distance between points to segment the scan image. Then the circle fitting algorithm

is

applied and the position and radius are estimated. If the diameter of circle

is

outside predefined interval the circle is rejected. Others are considered as

paper reels. To estimate location of reels in global coordinate the relative position provided by previous step is combined with global position of the truck provided in a previous paragraph. Finally tracker keep a global map of detected paper reels. So the global position of each paper reel is updated using Kalman filter.

Since the truck work in a dynamic environment, its so important to keep

a generated map in the previous step updated all the time. So they used

Euclidean distance to associate closest reel in the map to the sensing data. If there is no reel in the map, a new one is added to that location in the global map and vise versa.

In loading scenario they used the radial distance to pick the nearest paper reel to predefined point (point of generating path), point A in Figure 1.5. But they did not consider of picking direction of paper reel. Finally, we have to say, in there published papers [1] and [2], which we consider them as our first references the researchers did not give a deeper view to their works. These views can achieve by understanding the works in their references.

2.3 Closely Related Works

Autonomous forklift handling systems is to our opinion one of the most inter- esting research in the last decades. The robots’ intelligence is demonstrated by navigation and exploring area. A little number of researches demonstrate on the decision of picking and grasping areas. However for the purpose of this research we divide the previous works to three fields; whole system, perception and detection and map building and picking problem researches.

12

Chapter 2.

Background

2.3.1 Method of autonomous material handling

As we described in section 2.2 above researchers in MALTA project were the first who developed a solution to handle cylindrical objects by autonomous vehicle. However, they are not the first who deal with forklifts that handling materials. Garibotto in [5]was one of the earlier researchers in this filed. In his research he, and his partners developed a forklift to handle pallets autonomously with availability to operate forklift manually. Forklift task, drivable paths and location of loading and unloading areas was feed to the robot manually through a portable keypad. Robot localization is solved by fusion sensory data from three dimension vision with odometry data. They used H shape landmark fixed on the ground to be a suitable reference to make error correction for robot location. Same solution is used to determine beginning and ending of pallets loading area. Predefined map for working area or warehouse also is used. For detecting pallets and picking it up they used 3D vision provided by a video camera but not the same one used to navigation task.

2.3.2 Perception and detection

Most previous work used mainly two type of perceptions, the wide spread- ing one is the vision using video camera [5] or omni-directional one. Last decade and due to the decreasing in cost and computational demands almost researches and researchers head to use laser range finder (LRF) as the main environment preceptor for robots [1]- [6] and [14]. However there are several preceptors used instead of the previous two but the used of them decrease by the time due to technical problems such as the high noise (Sonar and infra red sensor) or needing to use more sensors with it to give direction and distance to the target (infrared sensors). The advance of using the LRF not only lower cost and low demands in computational power but also because its dependence on the environment is lower than others, for example the brightness, humidity and temperature not affect its reading. Beside any one of these or some time collection of them like in [5]they used other sensors to measure other robot variables such as speed (encoder) internal temperature, direction (Odometer) and other variables. While almost of researchers used data from real laser range finder (LRF) or they do not mention how they simulate this device. The data collected from sensors (LRF) in [1],[6], [14] and [16] are clustered to groups according to the distance between two consecutive points. If the distance between these two points is more than a predefined threshold the line fitting algorithm is applied [6] and [14], the circle fitting algorithm applies

Method for Autonomous Picking of Paper Reels

13

is in [1] or one of them is applied in [15], according to the result of comparing angles between four consecutive points as will be explained in next paragraph.

2.3.3 Map building

Map building, localization and object detection are wide research area and they studied a lot last decades. The process of map building start from object detection and object fitting. For indoor environment as our system here, objects usually are described by line segments, arc and circles. Jodo Xavier [15] one of the researchers who study map building technique for the indoor environment. In his research he try to cluster data collected from LRF to groups according to distance between two consecutive points. If the distance is more than the threshold a group is created. Then he ignored any group contain less than four point. After that he calculate the angle between two limits point and any point in the middle. If the angle was between the value of (90 and 135 ) then there are some circle inside these points. He said that he tuned these values to detect the maximum number of circles. When the circle detected he apply his algorithm to fit circle. To fit circle he drew a line between each two consecutive points, from the middle of this line he construct a perpendicular line. The intersection point of the two perpendicular lines is the center of the circle and the distance between this point and any of the two points is the radius of the circle. If there is more than one intersection point he consider the mean point as the center then he calculate the radius. More details about the application return to [15]. However, Xavier used his method to detect moving persons and peoples. So he said he tuned an inscribed angle between 90 and 135 . From a theoretical point of view there is a week point in this algorithm. That is the algorithm is sensitive to the error deviation. For example for large error in data detection the algorithm will fail. However he presented a method to avoid this weakness by ignoring from calculation points that lay out of threshold tolerance. The wide and most common method which is used to fit circles from laser data is the least square method, well known by curve regression. P. Nuez [10] and his team worked on this algorithm. They proposed a circle fitting algorithm in the Cartesian coordinate space. However, his algorithm deal with the case when the error in both coordinates X and Y are not independent. Also he provide a variance matrix associated to the estimated circle parameter. Indeed, a lot of algorithms used Cartesian coordinate to fit circles using least square method. However, Julian Ryde [12] used least square method to fit data directly from laser scanner reading based on polar coordinate system because of using that is faster and more accurate. For more details about his method return back to [12].

14

Chapter 2.

Background

For purpose of map buildings Zezhong [16] developed an algorithm depend on three steps to build a complete map. At the first step he took the data from LRF then he segment it according to algorithm named it (adaptive clustering algorithm). Then he compute line segment correspondence to each cluster by calculate center point, length and orientation from stochastic variables. In the second step he built a local map from the segments by arranging lines counter clockwise. Finally, he built a global map from the different local maps. To update the global map at each laser scan he compared local map and global map, if some line segments appears in the local map but not in the global one he insert them to the global map. The final global map is the merge of all local maps together. To localize the robot he used some features in the room as land marks and referred to them as a complete line segments in the environment map. Then he try to find the possible complete line segments in each local map by comparing the length between landmarks and each line of local map. The maximum likelihood was chose as the best possible matching. Then the current position is computed according to this match. After several matching steps in relative position he get exact relative position for the robot. The heading of the robot was calculated as the difference between two vectors. First vector is a vector between center points of two landmarks in the environment map. The second vector is the vector between the center points of the correspondence line segments in the local map. Then the absolute global coordination are computed by transfer relative position for any of the correspondence line segments to the global coordination knowing the calculated heading and the coordinates of the center point of the landmark.

Chapter 3 Methods

In this section we will describe in some details methods that are used to solve problems in this thesis and hypothesis supposed to deal with it. This section is divided into several parts as we move in the problem solving. First we will describe the approach used in path planning and trajectory generator. Then we will move to explain perception algorithm. After that we will describe algorithm used for generating a map from the perception data. The final two parts will be about the core of this research. They are, firstly, about reel selection then the problem of loading scenario.

3.1 Path Planning

We start from only one hypothesis. That is, we have predefined map for all unmovable facilities and object in the warehouse. This hypothesis is true and it is satisfactory to start from. As we mention in section 2.3.1, almost researches in the field of indoor autonomous vehicles are assumed that they have such kind of map. So we said this hypothesis are true. The reason makes this hypothesis satisfactory is the simplicity of creating this map. Usually architecture plans for warehouses are found in archive. So it is easy to create an accurate map from them contains location of main facilities such as walls, doors, columns, drop-down floors and drivable paths. In this case not so necessary to provide exact location for each paper reel.

3.1.1 Kinematic model

As we described in chapter 1 and as in Figure 1.5 the MALTA robot is an ordinary industrial forklift with two modified forks. The wheels (the main part of the driving system according to this topic) are a stander wheels with

15

16

Chapter 3.

Methods

Figure 3.1:

Simplification of MALTA robot kinematic model used to compute the po- sition of the vehicle from wheel encoder information (bicycle model).

the vehicle from wheel encoder information (bicycle model). two degree of freedom, one rotation around the

two degree of freedom, one rotation around the wheel axis and the other one around an offset steering point. The two front wheels are only with one degree of freedom without any ability to steering but they have the power to traction. The two rear wheels have the ability to steering. They are follow Ackermann steering principle 1 . For simplicity and with assuming that there is no slip in motion we can simplify the robot model to be as a robot with only two wheels one steering wheel in the back and one in the front (bicycle model as in figure 3.1). Let φ is the steering angle of the rear wheel in the simplification model 3.1. Let v is the linear velocity of the robot and let L is is the distance between the rear and front wheels. Then let X and Y be the x and y position of the robot in the global coordinate and let θ be the heading of the robot in the same coordinate and let dT is the sampling period. Then the position of the global coordinate is

P =

X

Y

θ

(3.1)

Then the change in position as a function of v and φ (kinematic model of our robot) can be described as follows:

1 For more information about Ackermann steering principle and the method of cal- culating steering angle for each wheel look to Desmond King-Hele, F.R.S., Erasmus Darwin’s Improved Design For Steering Carriages - and Cars. Notes and records of the Royal Society of London, Vol. 56, No.1 (Jan., 2002) pp (41- 61)

Method for Autonomous Picking of Paper Reels

17

Method for Autonomous Picking of Paper Reels 17 Figure 3.2: The arrangement of points that chosen

Figure 3.2: The arrangement of points that chosen to gener- ate the path. Six points are cho- sen. P 1 is the original point of the robot frame. P 2 is a point in the middle of the forks. P 3 is a point in a distance of 1.5d on the robot X-axis. P 6 is the center of the paper reel. P 5 is a point on the surface of the pa- per reel at the same direction of picking direction. P 4 is a point on the same direction of P 5 but with 1.5d distance of the paper reel surface. ”d” refers to the robot total length. The path is generated by using B-spline cu- bic function.

δX = v dT

2

δY

= v dT

2

δθ =

v

2L dT sin(2φ)