Book Detection for Service Robotics

Faisal Saeed, Institute of Automation University Bremen,Germany

Abstract
Book detection is always considered a challenging task in the field of service robotics. Service Robots are used to help elderly people in daily routine. The rehabilitation robot FRIEND at the IAT institute is intended to help and support disables people in real library environment. The task of the robot is to grasp and to detect autonomously books of unknown size. One challenging task in this project is to exact the real boundaries of the desired books in a clustered scene including the shelf plane and the book holder. The algorithm proposed in this work will integrate planar Segmentation with color segmentation to produce better results. Two different approaches were Investigated in this work for book detection: the first is based on Hough line detection and parallelogram extraction and the second is based on probability based contour extraction of the color segmented regions. The work presented here implements and evaluates both approaches. Although each approach has its own drawbacks, the experimental results in this work show a Tendency to adopt the contour position approach for reasons based on the processing speed. .

Keywords: 1 Introduction
detect autonomously book of unknown size and color. This is accomplish in our work by improving the result of previously computed segmented planar regions resulted from depth region as an input where graph based color segmentation has been integrated together to produce better result Two different approaches for segmenting the boundaries of the book have been investigated first one is based upon parallelogram detection using confidence score. and the second one is based upon probability contour extraction of different color segmented book regions if the segmentation result in book corner than the exact boundaries of the book can be detected which could further help the robot for grasping purpose.

Service robots are used to help handicapped or elderly people in their daily life activities. One of those tasks is to integrate the handicapped people in their working environment again. The rehabilitation robot "FRIEND III" which has been developed at the Institute of Automation (IAT) is intended to support and integrate the handicapped people in a real library environment. The user will enter the book into the library database by a speech recognition algorithm. The task of the robot is to visually detect and grasp right most books from the upper shelf in the book cartand then delivers it to the book holder. The book holder is a device to flip the pages of the book. The robot than grasp the book from the book holder and return it back to the lower shelf in the book cart. The main aim of the implementation is to design a robust image processing algorithm for detecting books in the library scenario specially for books on shelf .One of the challenging task for the successful grasping of the books is to extract the frontal face of the book including the book corner four points and to detect the thickness of the book in a clustered scene. Those grasping feature if extracted can help us to build a 3D model in the robot virtual environment There exists several methods from basic image processing[1,2] for detecting boundaries of the books based upon contour extraction and line detection but Hough lines ,Canny edge detection fails to segment the book in clustered scene. Also there are other methods which use simple bar codes or optical character recognition. For identification of each book in a library scenario. But in comparison to other approaches the proposed algorithm implemented in our work does not take any prior information about the books in the scene and can

State of Art approach In [2] an intelligent books managing system towards lending and borrowing of book has been proposed. While the algorithm will detect the object based upon object frontal face detection and background suppression. However the method requires that the posterior plane of the books is facing the cameras and new book can only be added to the shelf if the posterior information of the previous book is known in advance In [3] Hough transform have been used for detecting objects of different shapes and size .While the rectangle will only be detected if the sides of the rectangle are parallel and all the vertices are at right angle.

Book Detection

The autonomous book detection is a sophisticated process hence it has been further sub-divided into

several sub-processes. Each sub-process has several functionality and operations as in figure

The objective of this method is to extract the color information of region lying under each and every planar mask one by one and not to include those regions which are partially overlapping..Later on we want to make sure that a planar region really belong to the color region by seeing the percentage count of that region in the color segmented image .If the pixels in the color segmented regions are above a specific threshold we take them as book regions other its noise and we remove it.

The algorithm will take a pair of stereo left and stereo right images as an input and will find the disparity map by using the block matching algorithm. The planar segmentation in [9] will segment the disparity map into planar surface by using DGoD Directional gradient of disparity and Magnitude gradient of disparity (MGoD) which segments frontal face of the book as in figure (2.2(b)) the result of planar segmentation will used further for defining the region of interest of the book

After calculating the percentage between the planar surface and the segmented color region (i) suitable threshold has been proposed which likely detect the book regions respectively.

It was found experimentally that the algorithm gives good results at 0.6 but still includes noise regions from the surrounding as in figure (2.3(c)). because of over segmentation during color segmentation .In order to overcome this problem of over segmentation in color regions and to reduce the complexity of region and to further make the algorithm robust a new technique is proposed which will merge likely color region using color similarity measure in YCbCr color space.

2.1 Region of Interest Definition(ROI)

The result of planar segmentation represents the rough region of interest of the book. Our aim is improve the result of planar region of interest by incorporating the depth information of the planar region along with color information and to merge shadow and non shadow regions which have same color similarity in chrominance red and chrominance blue and finally detect them as book regions.

2.1.1Region merging using Color Similarity Measure

As said before during graph based color segmentation some regions are over segmented in to two or more than two regions which belong to one single region. As a result the segmentation result will not remain optimal anymore which can practically happen to any color segmentation algorithm .In order to solve this problem of over segmentation, minimum Mahalanobis distance between shadow and non shadow region under varying luminance condition is

computed and will further merge those regions which have color similarity in chrominance red (Cr) and chrominance blue (Cr).YCbCr color space is used during merging process because it was found experimentally that chrominance blue (Cb) and chrominance red(Cr) value will remain same for shadow regions as compared to the non shadow ones. Instead of employing equilidean distance, minimum Mahalanobis distance is computed between shadow regions in YCbCr color space. As said in [12] that the true color difference between adjacent color regions in any color space can be found by using Mahalanobis distance. During merging step it was found heuristically that the minimum distance in order to merge two region into one region having similar color is approximately 5 .But still noise is present as shown in figure (2.4(d)) but still this process will reduce the number of regions and to further overcome the complexity of the region which as a result will improve the results of line detection and edge segmentation respectively

the parallelogram which extract the exact boundaries of book.

Book detection with parallelogram

The result show that the algorithm is robust in extracting the corner of book of unknown size even if noise is present in book regions as shown in the figure (3.1(c, d)).But line detection fails to detect lines around thin books we tried to detect the boundaries of thin books but it ends up splitting the line into too small segments which as a result will increased the computational time of the algorithm. We come to another conclusion.

A new technique for detecting the real boundaries of the book based upon parallelogram detection with line and confidence score has been proposed i.e. intersecting corners of the book-.Those corners if extracted can help us to create a 3-D model which can help the robot for grasping purpose. Here parallelogram detection has been done instead of detecting rectangles because we want to detect books of unknown size where the opposite sides of the books are parallel but not necessarily equal. During parallelogram detection, multiple parallelogram of different sizes have been detected we need to select the parallelogram that perfectly fits the exact boundaries of the books. In order to do this confidence score calculation has been done. 3.1 Confidence Score Calculation We find the length of vertical sides of book L1 and L3 Merging score will take the summation of difference in lengths between opposite sides of the parallelogram i.e. horizontal and vertical sides. Normalize the merging score and one minus normalized score and will take the highest confidence score and will draw

4. Book Detection with Contour matching

During book detection with confidence score it was found that the line detection algorithm failed to detect too think books .We tried to fine tune the parameter of line detection algorithm but it ends up splitting the line into large number of too small segments of line which as a result will increase the computation time of the algorithm a lot. In order to overcome the processing time of the algorithm and to further improve the robustness of the algorithm a new experiment was performed detecting book frontal face based upon contour matching of color regions.

The objective of the work is to generate different combination of book regions resulted from segmented color region during the merging process and to further take the minimum area rectangle across each combination of book region and to detect those regions as book which have the minimum non intersecting region between the book region and its minimum area rectangle. But this criteria does not result in exactly detecting the whole book but rather it can also detect only a small part of the book .In order to make sure that the detected region really belongs to the whole book we need to see the percentage area of each region in the planar segmented (ROI).If the percentage ratio between book region and planar region of interest are above a specific threshold we consider it a book regions otherwise we remove it .The algorithm will find percentage ratio by counting the number of non zero pixels in the corresponding regions . As shown in figure we have the book region and its minimum area rectangle but non intersecting region is very high .In an ideal case we have the book region and its minimum area rectangle but it has low non intersecting region which is most likely a book region.

It was found that the algorithm gives good results and is more robust than book with parallelogram and can detect books of any size even if too thin books are present Also computational speed of algorithm is reduced and is better than book detection with parallelogram

Conclusion and Future work

In this work two different approaches have been presented for detecting books based upon book contour matching and book detection with parallelogram and confidence score. The goal of the work is to detect the exact boundaries of the book in the library scenario for reliable grasping .The evaluation experiments show that both the algorithms are robust for detecting book of unknown size and positions but the second method , book detection with contour matching is fast and takes less processing speed . But the limitation of parallelogram detection is that it fails to detect too thin books also during region of interest definition noise region is added to the book regions. So first future work would be to take color region from color segmentation that lies under the planar surface on the basis of their size instead of counting the number of pixels in a region. Second future work would be to improve the line detection algorithm by merging small segments of line which as

a result will decrease the number of line segments and will make the algorithm more robust and also improves the processing time of the algorithm.

[10]

4
[1]

References
Prats M., Ramos-Garijo R., Sanz P.J., del Pobil A.P. "Autonomous Localization and Extraction of Books in a Library". Intelligent Autonomous Systems, edited by F. Groen et al., IOS Press. pp.1138-1145 (ISBN1-58603-414-6) Amsterdam, 2004. Danny Crasto, Christopher O. Jaynes. The Smart Bookshelf: A Study of Camera ProjectorScene Augmentation of an Everyday Environment. In 7th IEEE Workshop on Applications of Computer Vision / IEEE Workshop on Motion and Video Computing (WACV/MOTION 2005),5-7 January 2005, Breckenridge, CO, USA.pages 218-225, IEEE Computer Society, 2005. Claudio Rosito Jung and Rodrigo Schramm UNISINOS - Universidade do Vale do Rio dosSinos Ciencias Exatas e Tecnologicas Av. UNISINOS, 950. Sao Leopoldo , RS, Brasil, 93022-00. "Rectangle detection based on a windowed Hough transform" .11/2004; ISBN: 07695-2227-0.In proceeding of: Computer Graphics and Image Processing, 2004. Proceedings. 17th Brazilian Symposium on. Tetsuo Tomizawat,Akihisa Ohyati and Shinichi Yuta ,Intelligent Robot Laboratory, University of Tsukuba, Japan "Remote Book Browsing System using a Mobile Manipulator ". Proceedings the 2003 IEEE international Conference on Robotics and Automation Taipei,Taiwan, September 1419,2003 . Prats M,Ramos-Garijo R,Sanz PJ,Del Pobil AP ,Department of Computer Science and Engineering University of Jaume-I Castellon, Spain ,"Recent Progress in the UJI Librarian Robot". Pascale, Danny. "A Review of RGB color spaces...from xyY to R'G'B'" (PDF). Retrieved 2008-01-21. Max K Agoston (2005) Computer Graphics and Geometric Modelling, Implementation and Algorithms London: Springer . page 300-306 Agoston book contains description of HSV and HSL and algorithms Charles Poynton, Digital Video and HDTV,Chapter 24 paper 291 to 292, Morgan Kaufmann,2003. Stefan Heyer, Bashar Enjarini, Christos Fragkopoulos and Axel Graser ,"Book Detection [11]

[12]

[2]

[13] [14]

and Grasping in Library Scenario ",Institute of Automation (IAT), University of Bremen. Pedro F Felzenszwalb ,Daniel P Huttenlocher" Effcient Graph Based Image Segmentation"Artificial Intelligence Lab, Massachusetts Institute of Technology and Computer Science department Cornell University". Sandeep Chalasani "Graph based Image Segmentation Department of Electrical and Computer Engineering Clemson University. ALAIN TREMEAU and NATHALIE BOREL ,A Region Growing and Merging Algorithm to Color Segmentation Rafael C. Gonzalez and Richard E. Woods, Digital Image processing. D. H. Ballard . Generalizing the hough transform to detect arbitrary shapes . pages 714725,1987.

[3]

[4]

[5]

[6]

[7]

[8]

[9]