A Robsut Neural Network Multi-Lane Recognition System

Amer K. Dawoud', Salah G. Foda* and Ahmad S. Tolba**

* Department of Electrical and Computer Engineering,

** Department of Physics,
Kuwait University, P.O. Box 5969, Safat 13060, Kuwait
E-mail: sfoda@eng.kuniv.edu.kw

AbsZruct: In this paper, the design and neural imp- vehicle. It should not show any tendency to
lementation of a vision based multi-lane highway break down when the environmental
lane recognition system are presented. The design variables such as road width, road conditions
objective of the system is to recognize the lane which and light conditions changes, and in the
a test vehicle is currently driving through by presence of external noise such as nearby
determining its left and right lane boundaries. When
passing vehicle.
the proposed lane recognition system was tested it
showed very high percentage of correct results in very
difficult circumstances which suggests that it
provides the basis for a reliable road following
system.
I. Introduction
For the past decade many institutes and research
centers have been conducting research on vision
based guidance systems for the purpose of
autonomously navigating land vehicles [1-91. The
building of such intelligent systems is a complex
and challenging task but with huge potentials in System low level medium high level
both civil and military application domains. level
SCARF r1-31 probability *** match filter
Autonomow land vehicle navigation system is a ALVINN [MI *** *** back prop.
system that can carry out tasks carried out by a ***
RBFN U] t**
radial basis
human driver. These tasks include road following,
speed control and tactical driving tasks such as Bristol [8,9] pixel voting *** match filter
obstacle avoidance, changing lane and negotiating
intersections. There are practical difficulties Table 1: Existing navigation systems:
related to the implementation of autonomous
navigation of an unmanned vehicle in the normal 11. Design of Lane Recognition System
roads. However, road following systems seem the
suitable and practical implementation where the The highway has been chosen as our
road following system can be used to monitor the environment in this work because it is less
vehicle's lateral position inside the road and to complicated than other road environments.
detect any unintended road departure. The components of road environment include
only asphalt road, lane markings, vehicles
We believe that any real progress towards and non-road surrounding areas like green
the implementation of vehicle navigation or plants or sand areas. It does not include
road following systems in normal roads intersections nor stop signals
should start with developing the recognition
system that could robustly produce correct The expected result from the proposed
judgments regarding the position of the recognition system is to identify the left and
right boundary lines of the lane that the
vehicle is currently driving through.
In the proposed lane recognition system, we
have chosen to follow the same steps of lane
recognition by human driver using neural
networks. It relies on image processing at
all the three levels. The lowest level of
recognition process involves classifying the
input image pixels into road and non-road
using Learning Vector Quantization
techniques. The medium level of recognition
process involves scanning the blocks of the
binary image resulting from the classification
process searching for the lane markings.
Classical Back Propagation network was
trained for this purpose. Fig.1 shows the
modules of the proposed lane recognition
system.
+-I
input color image
P=+---l
re rocessin
[road and non-road ]
lane marking
detection
v available and from these recordings we have
lane boundary
determination
Fig.]: Block diagram for the proposed lane
recognition system
A SONY CCD video camera recorder is used
to capture video recordings that are saved in
the form of successive frames. The
preprocessing module involves two
operations: Light variation compensation and
Image size reduction. The i w g e reduction
replaces a block of pixels in the original
image with one pixel in the reduced image.
It was empiricalJy round that the best results
for image reddction is obtained by using sub-
sampling and averaging.
111. Road and Non-road Classification Network
This part of the highway lane detection system
represents the low level of processing where each
pixel of the preprocessed color is classified into
either road or non-road. The input space of the
preprocessed color image is of dimension 3, and
each pixel is represented by a vector in a hyper
cube of red green and blue components.
Building a robust vision system capable of
recognizing highway lane in many different
conditions requires the diversity of the
highway image scenes used to derive the road
and non-road color models required for the
classificatior.. The diversity of these highway
scenes is assured by taking intc
consideration the main factors that would
differentiate an image from another in a
highway environment. These main factors
are:
(a) Color of the highway asphalt,
(b) Presence or absence of external noise of
nearby passing vehicle,
(c) Shape of the white lane markings which
can be solid long line or dashed line,
(d) Nature of non-road area surrounding the
highway which can widely vary.
Many highway video recordings were made
selected 250 images that represent a wide variety
of highway scenes. Then each image is
preprocessed by light variation compensation,
then sub-samptng and averaging 2x2 block of
pixels into one pixel in the reduced image as
shown in the previous chapter. From each of these
images 4 pixels were selected to represent road
input vectors. The total number of road input
vectors is 1000. The same was repeated for non-
road areas. Rozd input vectors included asphalt
road areas only, whiie the non-road input vectors
included all elements of highway environment
except the asphalt road such as:
(1) Highway surrounding areas like green plants
areas and desert areas and the blue sky,
(2) White highway lane markings,
(c) Barriers between the two sides of the highway,
(d) Obstacles like nearby passing vehicles of
different colors.

%wmLer 14 -16 1998
Using Matlab Neural Network Toolbox, learning
Vector Quantization GVQ) [lo-121 w a s utilized
for the purpose of optimizing the location of the
weight vectors. After training is completed, the
resulting LVQ network can be used to classify the
highway road images.
Each pixel of the preprocessed image is classified
into either road or non-road by presenting it to the
LVQ network of Fig. 2 and the output will be
either [0 13 for road or [l 01 for non-road. The
size of binary output image resulting from this
stage is 80x60 pixels.
Input Competitivelayer Linear layer Output
n l
Fig. 2: LVQ Classifying Network
Fig. 3: Input image and the classification binary
output.
Fig. 4: Road classifcation for highway image
which includes a dark vehicle
180
IV. Highway Lane Detection Network
This part of lane recognition system represents
the medium level of processing. The binary image
resulting from classification stage is scanned
searching for the highway lane markings. Medim
level means that “blocks” of pixel areas are being
processed not single pixel as in the previous low
level classification stage.
To locate the boundaries of a highway lane we
need first to find the positions of the white lane
markings which were classified as non-road areas
inside road areas. These lane markings could be
defmed as:- white non-road areas surrounded by
road areas from both sides and have certain
shapes. The shape of the lane marking could be
also used as a feature that distinguishes it from
other non-road areas such as nearby passing
vehicles.
The factors that contribute in variation of the
shape and size of the white lane marking are:
(1) The type of the lane marking whether solid
white or dashed line,
(2) The size of the lane marking changes with
change of the position of the marking with respect
to input camera. The closer the lane markings to
the camera the larger they will appear in the
image,
(3) The orientation of the lane rrxking with
respect to the input camera. Lane markings
located on the left of input camera will have
different orientation compared to lane markings
located on the right on input camera.
Using a supervised neural network algorithm, the
proposed system was trained to detect various
types of lane markings. The inputs to the lane
marking detection system are blocks of the binary
pixel image resulting from classlfylng the
preprocessed image into road and non-road. The
output is a binary number 0 or 1 indicating the
absence or presence of lane markmgs in that
block. With this direct and defrnite input\output
relationship the best choice was to use back-
propagation Deural network algorithm with
Levenberg-Marquardt approximation technique
for updating the weights and biases.
The first stage of the lane detection system is to
build models for the shapes of lane markings that
need to be recognized.
C
Fig. 5(a): Original input image
(b): Binary classification
(c): 5x2 lane marking model
Fig. 5(a) shows an original color image, while
Fig. 5(b) shows the classification into road and
non-road. Fig. 5(c) shows a lane marking model,
a box of white non-road area surrounded by road
area from both sides. Since close lane marking to
the input camera have a width of 3 pixels, while
far ones had a width of only one pixel.( 1 pixel =
2x2 pixels of original image), the width of the
block model was chosen to be 5 pixels so that it
can fit all types of lane marking in the midde of
the model plus road pixels from left and right
sides.
The architecture of the used back-propagation
network [13,14] is chosen as follows:
a) The number of inputs to the network = 10,
which represents the size of the 2x5 models in
Fig. 5(c). b) The number of neurons in the output
layer is one, indicating the presence or absence
lane marking.
c) The number of neurons in the hidden layer was
chosen to be 8. This number of hidden layer
neuron was enough to reach the desired learning
accuracy.
d) Log sigmoidal activation functions in hidden
and output layers neurons were chosen.
V. Design of Lane Marking Detection Network
Lane markings could be defined as white non-road
areas surrounded by road areas from both sides.
Assuming that the trained feed foreword network
could detect lane markings shapes of non-road
areas surrounded by road areas, then to satisfy the
definition of lane markings we must make sure
that non-road areas are white colored.
In the previou:; section, the non-road color space
was divided into 6 clusters, three of them
represented light colors and the other three
represented dark colors. The white color of lane
marking of course belongs to the light colored
clusters. Which means that all non-road pixels in
5x2 window should belong to light colored
clusters. And if any of non-road pixels belonged
to dark colored ciuster then that window will be
given output of 0 indicating the absence on lane
marking in that window.
To detect white lane marking, a 5x2 window will
scan the binary image resulting from the road and
non-road classification. This window will be the
input to the back-propagation network that will
highlight the positions in the image which match
any of the 42 models of lane marking. This
window will move with one pixel increment at a
time.
The white color detection neural ndwork checks
classification clusters of 5x2 window pixels and if
any non-road pixel does not bejong to light
colored clusters, then the output of the trained
feed-foreword will be forced to be 0 indicating the
absence of liane marking in that window
regardless the output of trained feed-foreword
network.
binary
classification
i nnge
Road classifisd pixe!
Non-rozd classified pixel
Detected lane marking
Fig. 6: Lane marking detection system.

-
Fig. 7: Examples of original - and lane
images
marking detection output.
VI. Conclusions and Future Work
The developed lane recognition system
performs a task that originally is carried out
by human driver. It can provide a skeleton
for future work in two main directions
related to active areas of Artificial
Intelligence as follows:
(a) real time implementation of lane
recognition system which involves the
interface with input color camera and
building dedicated hardware that satisfies
the real time requirements of processing each
frame.
(b) The development of road following or lane
keeping system that utilizes the proposed fane
recognition system in finding the boundaries of
the current lane and then guide the vehicle to the
middle of the lane. The development of lane
keeping system involves the generation of steering
angle decision, steering control mechanism and
the feedback from the vehicle steering mechanism
The highest level of recognition process
involves selecting the markings which
belongs to the current lane and isolating
them from the total detected lane marking
set. This can be done using Fuzzy logic
which is the subject of current research.
References:
[ 11 V. Graefe and K.D. Kuhnert “Vision-based
autonomous road vehicles”, In Vision-based
vehicle guidance, Masaki I.(eds), Springer-Verlag,
NY,1992. .L
[2] J.D.Crisman and C.E. Thorpe, “SCARF: A
color vision system that tracks roads and
intersection”, IEEE Tram. on robotics and
automation, 1993.
[3] M. Meng and A.C. Kak, “Fast vision-guided
mobile robot navigation using neural networks”,
IEEE Proc. of system, Man and Cybernetic,
1992.
[4] D.A. Pemerleau, “Rapidly adapting artificial
neural networks for autonomous navigation”,
Advances in neural information processing,
Morgan Kaufrnann Publiihers, 1991.
[5] T. Jochem and D. Pomerleau, “Life in the Fast
Lane: The Evolution of an Adaptive Vehicle
Control System”, AI wgazine, 1996.
[6] M. Rosenblum and L. Davis, “An Improved
Radial Basis Functiori Network for Visual
Autonomous Road Following”, IEEE Trans. on
Neural Networks, 1996.
[7] K. Dowling et al., “Navlab: An autonomous
navigation testbed” in Vision and Navigation:
The Carnegie Mellon Navlab, C . Thorpe (Ed.),
Norwell, MA: Kluwer, 1990.
181 F.W.J. Gibbs and B.T.Thomas, “The fusion
of multiple image analysis algorithms for robot
road following”, IEE Fij3h Intemtional
Conference on Image Processing and its
Applications, 1995.
[93 S. Spiegle, “Autonomous road vehicle
navigation”, Proc. Electronic Technology to the
year 2000, 1995.
[lo] Y.Linde, A. Buzo, and R. M. Gray, “ An
algorithm for vector quantizer design,” IEEE
Trans. Commun., 1980.
[113 J. Bryant, “On the clustering of multi-
dimensional pictorid data,” Pattern Recognition,
1979.
[I21 T. Kohonen, “Self-organizing and
Associative Memory,” Spinger-Verlag, 1989.
[13] M . H. Hasson, “Fundamentals of Artificial
Neural Networks” The MlTpress, 1995.
Cl41 H. Demuth and M. Beale, ‘‘ Neural Network
TOOLBOX for use with Matlab,” The Math
Works Inc, 1994.