Pattern Recognition

Letters
ELSEVIER

Pattern Recognition Letters 16 (1995) 1321-1330

An image detection technique based on morphological edge

detection and background differencing for real-time
traffic analysis
M. Fathy a, M.Y. Siyal b,,
a Dept. of Computer Engineering, Iran University ofSc. & Technology, Tehran, Iran
b School of Electrical and Electronic Engineering, Nanyang Technological University, Nanyang Avenue, Singapore 2263, Singapore
Received 24 March 1995; revised 20 June 1995

Abstract
The real-time vehicle detection from a traffic scene is the major process in image processing based traffic data collection
and analysis techniques. The most common algorithm used for real-time vehicle detection is based on background
differencing and thresholding operations. The efficiency of this method of image detection is heavily dependent on the
background updating and threshold selection techniques. In this paper, a new background updating and a dynamic threshold
selection technique is presented. An alternative image detection technique used in image processing is based on edge
detection techniques. However, an edge detector extracts the edges of the objects of a scene irrespective of whether it
belongs to the background details or the objects. Therefore, to separate these two, extra information is required. We have
developed a new image detection method based on background differencing and edge detection techniques, which separates
the objects from their backgrounds and works well under various lighting and weather conditions. This image detection
technique together with other techniques for calculating traffic parameters e.g. counting number of vehicles, works in
real-time on an 80386-based microcomputer operating at a clock speed of 33 MHz.

1. Introduction
In recent years, extensive research and development efforts have been devoted to image processing
techniques applied to traffic data collection and analysis (Hoose, 1991, 1992). For real-time traffic image
processing applications, the sequence of images have
to be processed at a rate of 25 frames/second, which
produces a large amount of data. Therefore, the
image processing algorithms have to be simple but
effective so that they can be executed in real-time.
* Corresponding author. Email: emysiyal@ntu.ac.sg

The first image processing algorithm required to

extract traffic parameters, is an image detection technique. A common and simple image detection approach used in traffic application is the background
differencing technique. This approach is based on
pixel-by-pixel comparison of a background picture
and the current frame of the scene and has been used
by various researchers in traffic applications (Dickinson and Waterfall, 1984a,b). In practice the effectiveness of this method depends mostly on the accuracy of the background updating technique and the
selection of a suitable threshold value. In this paper
we introduce an effective and low-cost background

0167-8655/95/$09.50 1995 Elsevier Science B.V. All rights reserved

SSDI 0 1 6 7 - 8 6 5 5 ( 9 5 ) 0 0 0 8 1 - X

1322

M. Fathy, M.Y. Siyal / Pattern Recognition Letters 16 (1995) 1321-1330

updating technique and a dynamic threshold selection technique.

An alternative image detection technique used in
image processing is based on the edge detection
technique. Edge-based image detection is generally
more effective than background differencing and has
been used by few researchers in traffic applications
(Hoose, 1991). In fact, the edge information of the
objects remains still significant despite the variation
of the ambient lighting. The conventional gradientbased edge detection operations have found wide
acceptance in image processing applications. However, morphological edge detectors have shown better performance than conventional edge detectors
while having a lower computational cost (Lee et al.,
1987; Fathy et al., 1994). In this paper, a simple and
effective morphological edge detector using the decomposition property of the morphological operators
and a separable median filtering is presented.
An edge detector extracts the edges of the objects
of a scene irrespective of whether it belongs to the
background details or the objects. Therefore, to separate these two, extra information is required. This
extra information can be obtained from the edges of
the background picture. We have developed a new
image detection technique based on background differencing and edge detection techniques, which separates the objects from their backgrounds. This image
detection technique together with other techniques
for calculating traffic parameters e.g. counting the
number of vehicles, works in real-time on an 80386based microcomputer operating at a clock speed of
33 MHz.

ble. Therefore, the updating of the background frame

has to be automatic and concurrent with the processing if real-world traffic scenes are to be analyzed.
In the background differencing approach, the current picture (Cpt) is subtracted from the background
picture (Bpt), resulting in a difference image (Dpl).
The differencing operation is normally an absolute
differencing operation, to detect vehicles having
brighter or darker grey-value than the background.
Each pixel of Dpl is compared with a threshold
value (T) and a binary picture is generated (Bpt). If
the result of subtraction of Dpl and T is greater than
zero, the corresponding pixel of Bpt becomes 1,
otherwise 0. These operations can be summarized as
follows.

2. Background updating

2.1. Frame averaging technique

One of the main problems associated with the

background differencing technique, is the change in
lighting conditions, which makes the current background frame invalid. These temporal changes occur
even on overcast days and the problem becomes
further complicated by spatial changes caused by
emergence and disappearance of shadows on bright
days. Consequently, background updating has to be
performed whenever the illumination level changes.
Interrupting a real-world situation, whenever a background frame has to be updated is not always possi-

In this method the average of a number of input

images in a period of time is referred to as an
updated background image. The general form of
averaging is not easily implementable in real-time.
Alternatively, if the weighting factors of all pictures
used for averaging are identical, then the averaging
of N + 1 frames can be performed using the following equation (Hoose, 1992; Takaba et al., 1984):

For each pixel of Dpl, Cpt, Bpt do

Opl = I Cpt - Bpt I
If Dpl ~> T
Then
Bpt = i
(object)
Else
Bpt = 0
(no object)
To make the background differencing technique
more effective, the changes in ambient lighting must
be compensated by an effective background updating
technique. Currently several operational traffic
surveillance systems have been constructed by using
various background updating techniques (Seed and
Houghton, 1988). The most commonly used background updating techniques are based on averaging
(Siyal et al., 1994; Seed and Houghton, 1988) or
selective techniques (TRIP Project Report, 1986;
Rourke and Bell, 1988).

Bpt = K B p t _ 1 + (1 - K )

Cpt_l,

0 < K < 1 and K = N / ( N + 1)

(1)

1323

M. Fathy, M.Y. Siyal / Pattern Recognition Letters 16 (1995) 1321-1330

where Bpt is the updated background picture, Bpt_ 1

is the previous background picture and Cpt_ 1 is the
previous frame of the scene.
The above equation is also known as the exponential updating technique and is practically implementable in real-time. The value of K determines
the rate of updating. When K approaches to 1, the
new background becomes closer to the previous one
and the effect of the current picture is reduced. The
appropriate value of K depends on the ambient
lighting conditions and is often adjusted manually.

2.2. Selective updating technique

Ambient changes are usually smaller than the
changes due to the objects. This implies that those
sections of the scene which are not covered by
moving objects should only be updated. The updating process is performed simply by replacing the
background pixel values by the current pixel values
at selected pixel points. This selective updating technique can be expressed as follows (Hoose, 1992;
Seed and Houghton, 1988):
If Dpl > T
Then
Bpt = Bpt- 1
Else
Bpt = Cpt 1

(don't update)
(update)

2.3. A new background updating technique

The drawback of the averaging technique is the
quick response to the regions where there is a significant difference between the background and the
current picture, while this difference is mostly due to
the objects. Also, the proper setting of the K value
is not an easy task. The effectiveness of the selective
updating technique depends on the accuracy of the

Table 1
Dpl >/T1
Dp2>/T2
Updating background
Object detected
Status of the trafficat
the pixel to be tested

No
No
Yes
No
Empty road

threshold value. If the threshold value is not selected

properly, the object pixels are miss-classified as the
background pixels, and the background picture becomes quickly unusable.
The proposed updating technique uses the intensity changes between two consecutive frames as a
measure of the ambient lighting variations. If the
difference of the corresponding pixels of two consecutive frames is less than the maximum expected
variation of intensity between the period of two
consecutive frames, the updating is performed, provided that no object is detected. The values of T1
and T2 are selected automatically by analyzing the
histograms of Dpl and Dp2 for a number of frames
(see Section 3). This updating operation can be
expressed as follows.
For each pixel of Cpt and Bpt do
If [Dpl = [ C p t - Bpt 1] < T1
Then If [ Dp2 = [ Cpt - Cpt + 1 [ ] < T2
Then
Bpt +1 = (Bpt + Cpt + 1 ) / 2
(update)
Else
Bpt + 1 = Bpt
(no update)
The operations of the above equation under various traffic conditions are shown in Table 1.
This updating technique is based on replacing the
background of the selected points (see column 2 of
Table 1) by the average of the current and background picture points, instead of directly replacing
the background points by the current picture points.
Thus, the problem of miss-classification of the objects as background in selective updating is significantly reduced, as the averaging of the Cpt and Bpt is
used as a new Bpt and also updating is performed
when no object on the current and the previous
frame is detected. The following points should be
noted regarding the selective-averaging technique.

No
Yes
No
No
Motion (vehicle
leaving the scene)

Yes
No
No
Yes
Stopped
objects

Yes
Yes
No
Yes
Moving
objects

1324

M. Fathy, M.Y. Siyal/Pattern Recognition Letters 16 (1995) 1321-1330

(a) The computation of Dpl and Dp2 imposes no

computational overhead as these two image differences are used for object detection and motion detection respectively.
(b) Instead of linear averaging of the Bpt and Cpt
frames, a weighting factor can be used. However,
linear averaging has a lower computational cost and
in most lighting conditions gives better results, as
both Cpt and Bpt are accounted equally.
(c) This updating method enjoys the advantages
of both averaging and selective techniques, and it is
less sensitive to the threshold selection technique.
However, we use a dynamic threshold selection technique to improve the effectiveness of this updating
technique.

3. A dynamic threshold selection technique

The selection of the threshold value is important
as a low threshold value will classify many objects
as the background and a high threshold value might
not be able to detect some pixels of the objects. The
method proposed here is based on analyzing the
histograms of windows of the difference images (Dpl
or Dp2) for a period of time. The window is located
across the lane, to detect vehicles passing through
the road. This method is expandable to the local
thresholding technique, by locating a window at each
local area of the scene.
Following the computation of Do1 and Dp2, the
histograms of the windows of these images are computed and the left-limit grey-value of each histogram
is extracted. The left-limit value is a grey-value of
the histogram, where there are approximately zero
number of pixels having higher grey-value. When
the window contains an object, the left limit of the
histogram shifts toward the maximum grey-value,
otherwise it shifts towards the origin. The minimum
of these left limits for a period of 10 minutes is
selected for the next period of processing. This
minimum value is used to evaluate the number of
pixels in Dpl or Do2 whose brightness is significant
enough and in most cases can be set arbitrarily
between 0-30 in a 256 grey-level system to speed up
the operation.
For threshold selection, the number of difference
points greater than the left-limit grey-value of each

200

20O

17S

175

150

~.50

(a)

(b)

Fig. 1. The histogram of the left-limit of histogram of 200 frames

(a) before applying median filtering, (b) after applying median

filtering.

window is extracted for a large number of frames

(200 frames). These numbers are used to create an
array (Fig. l(a)), where its horizontal axis corresponds to the number of points and its vertical axis
corresponds to the number of frames. The histogram
of Fig. l(a) is smoothed by using a median filter and
a point in the valley of the smoothed histogram is
selected as the threshold value. The smoothed histogram is shown in Fig. l(b).
The histogram of Fig. l(b) indicates that if we
select the threshold value between 30 and 40, there is
a possibility of detecting vehicles of which only a
small part passes the window. The program selects
the threshold value between 40 and 60 to detect
vehicles of which the most part passes the window.
The above operations are performed on both Dpl and
Dp2 and T1 and T2 are extracted, respectively.
This threshold selection operation consumes very
low computational power due to the small size of the
window required for this purpose. Also, by choosing
the same window for both traffic analysis and threshold selection, the time for generating the array of
Fig. l(a) is significantly reduced to O(N), where N
is the number of frames (200) in a 10 minutes
interval. In other cases, O(N) operation is required
to generate the histogram, where N is the number of
points in the window.

4. A separable morphological edge detector

The background differencing technique discussed
earlier is effective for many full-frame or windowbased image analysis techniques. However, to reduce
the effect of background updating for detecting ob-

M. Fathy,M.Y.Siyal/ PatternRecognitionLetters 16 (1995)1321-1330

jects, an edge-based segmentation technique which is
less sensitive to the variation of ambient lighting can
be used.
Few morphological edge detectors have been developed for both intensity (Krishnapuram and Gupta,
1992) and range images (Lee et al. 1987; Fathy,
1990). In the case of intensity images, a fiat-top
structuring element (Fathy et al., 1994) is considered
to analyze the operations of different edge detectors.
However, for range images, special shapes of structuring elements are used to detect different types of
edges. In this paper, we concentrate on intensity
images.
A suitable noise removal filter which has proven
to preserve edges, is median filtering. A version of
this filter named as "Separable Median Filtering"
has shown comparable performance to the median
filter but has faster computer implementation (Fathy
et al., 1994; Fathy, 1990). Indeed, a separable median filtering can be accomplished by running a
one-dimensional median filtering along any horizontal line and then along any vertical line in the image.
By applying a decomposition technique, the manner
in which a separable median filter is accomplished is
identical to dilation and erosion, except on selecting
median values instead of Max or Min values. Compared with other edge detection operators, the SMED
operator has the lowest computational requirement
and thus is more suitable for real-time image processing applications. This operator can be defined as

[O(S(f)) - g ( s ( f ) ) ]
(2)
where S(f) is the result of applying a separable
SMED =

median filtering to the horizontal image f. D 0 and

E 0 are dilation and erosion operators, respectively.

0.8 .

" /ZI!I11171

0.5---

0.4

1.'66

2:5

-~t-- open-dose

3.'33 4_ 5.'54 5_ 5.55 6/25 6.'66

sC,Cr~~ noCserato
-+-- ATM

-:~ - blv, -rrlln

F i g . 2. P e r f o r m a n c e

comparison

srned ede

( 3 x 3).

conditional probability of the true edge over the

detected edge, P(T/D), and the detected edge over
the true edge, P(D/T). In this method, the threshold values are selected in such a way that P(D/T)
= P(T/D).

The plots of probability versus signal to noise

ratio (SNR) for the edge detectors are shown in Figs.
2 and 3. Fig. 2 shows that the Open-close and SMED
operators have relatively better performance than the
ATM and Blur-min operators. When SNR is large
(SNR > 4), the performance of the SMED operator
is closer to the Open-close operator. Fig. 3 shows
that in the case of 5 x 5 neighborhood size, the
performance of SMED is better than of ATM and
Blur-min and is much closer to the 3 X 3 Open-close
operator (dotted line). It should be noted that the

095 ........... ~

4.1. Evaluation of SMED operator

o~54

To compare the performance of the SMED operator with other edge detectors, a test pattern employed
by other researchers to evaluate the performance of
similar detectors has been used. The test pattern is a
checker board comprising of dark and light squares
with grey-values of 50 and 100 respectively. To
measure the performance of different operators, noise
with gaussian distribution of zero mean and different
standard deviations was added to the test pattern.
The criterion used to compare the operators was

1325

'

....................

OB5

~stnalt~n~atla

open-close - 4 - - ATM

- ~ - blur-rain

- ~ - smed-ede

Fig. 3. Performance comparison (5 5).

1326

M. Fathy, M.Y. Siyal / Pattern Recognition Letters 16 (1995) 1321-1330

Fig. 4. Applyingthe SMEDoperatorto a sectionof a trafficscene.

computational cost of a 3 X 3 Open-close operator is
about 8 times of a 3 X 3 SMED operator. The result
of application of a 3 X 3 SMED operator on a realworld traffic scene is shown in Fig. 4.
4.2. Reasons for choosing SMED

The reasons for adopting the SMED operator in

our application are
(a) SMED is based on summation of erosion-residue (Er) and dilation-residue (Dr) operators (Ede),
which can detect edges at different angles, while

other morphological operators (except Open-close)

use Er, Dr or the minimum of these which are unable
to detect some kind of edges.
(b) The strength of the edges detected by SMED
is twice that of either Dr or Er, which has been used
in other edge detectors. ATM and Blur-min use Er
and are unable to detect thin lines and corners and
the strength of the edges detected is reduced.
(c) SMED uses separable median filtering for
noise removal. Median filtering has proven to preserve edges while removing impulse noises. Separable median filtering has shown to have performance
comparable to the true median filtering but requires
less computer power.
In brief, SMED uses compatible and easily implementable operators and has a lower computational
requirement, compared to the other morphological
edge-detection operators. Open-close has better performance than the SMED operator, but it has about 8
times computational power requirement, so it is not
very suitable for real-time image processing applications.

= l-ea~

Fig. 5. Blockdiagram of the combinedimage detectiontechnique.

5. A combined edge detection and background

differencing technique
The application of the SMED operator on an
image, can be used to extract the edges of the image.

M. Fathy, M.Y. Siyal / Pattern Recognition Letters 16 (1995) 1321-1330

However, to distinguish between the edges of the

desired objects and the edges of the background
details, extra information is required. This extra information can be obtained by using a background
picture containing only the stationary objects. Thus,
to extract the edges of the desired objects, an image
detection technique based on combining the SMED
edge detector and the selective-averaging background differencing technique is used.
In this method the edges of the current scene and
the background scene are obtained by applying a
SMED operator, and then the edges of the desired
objects are obtained by subtracting the edges of the
current and the background scenes. This process is
shown in Fig. 4. The main features of this image
detection technique are as follows.
(a) As the segmentation is based on edge detection, the effect of the ambient lighting is reduced.
(b) Despite the insensitivity of the algorithm to
the background updating technique, we use a low-cost
and effective background updating and threshold selection technique to eliminate the effect of the intensity changes of the scene entirely.
The low computational cost and effectiveness of
the SMED operator make the proposed algorithm

1327

suitable for real-time applications. The proposed algorithm has been applied on full-frame and
window-based image detection techniques using a
DT2867 (Data Translation Manual, 1993) frame
grabber board and an 80386 microcomputer system.
The DT2867 board allows the capture of more than 4
frames/second which is suitable for our applications. It has got special hardware for convolution-type
operations, which were not used in our application.
As stated earlier, the difference of two consecutive frames is used for motion detection and updating
operations. As shown in Table 1, if Dp2 is greater
than T2, a motion is detected for that point. Now, if
the number of moving points in an area is significant, a motion condition is detected, otherwise a stop
condition is detected in the area. Therefore, it can be
said that the proposed algorithm can be used for a
wide range of full-frame or window-based image-detection and motion-detection techniques.

6. Counting number of vehicles

The vehicle-detection operation discussed above
is used to measure other traffic parameters. To count
the number of vehicles passing through a window, a

Fig. 6. Vehicledetectionusing selectivebackgroundupdatingtechnique.

1328

M. Fathy, M.Y. Siyal / Pattern Recognition Letters 16 (1995) 1321-1330

Fig. 7. Vehicle detection using selective-averagingbackground updating technique.

status vector is created for the window. In this
manner, for each frame, if a vehicle is detected in the
window, a ' 1 ' is stored at the status vector of the

(a)

(b)

window, otherwise, a ' 0 ' is stored at the status

vector of the window. A group of ls (ones) corresponds to a vehicle and a group of 0s (zeros) corre-

(c)

(d)
Fig. 8. Operations of the proposed image detection technique.

M. Fathy, M.Y. Siyal / Pattern Recognition Letters 16 (1995) 1321-1330

I III Manual

1329

ra Corfiputer I

Rai~

Sno'

500

loo

Fig. 9. Output of the vehicle count for a period of 90 minutes.

sponds to the distance between two vehicles. A one

alone could be considered as a motorcycle or a fast
moving car.

7. Results

The algorithms were tested under various traffic

and lighting conditions using an 80386-based personal computer system operating at a clock speed of
33 MHz. The algorithm has been implemented in a
user-friendly package written in C language for MS
Windows 3.1 environment. The package allows the
user to define regions and the result of the operations
are reported graphically on-line by user-defined periods.
In the photographs of Figs. 6 and 7, there are four
pictures illustrating the effects of the various processing stages. The first, in the top left comer is the
current frame, while the updated background frame
is given at the bottom left corner. The thresholded
binary picture is shown in the top right comer, while
the final binary picture (without/with reduced noise)
is positioned at the bottom right quadrant.
In Fig. 6, the selective background updating technique was used, while in Fig. 7, the new background
updating technique (selective-averaging) was employed. As can be seen, the selective background
updating technique was unable to remove the shadows, which became part of the vehicle and will

complicate the task of vehicle tracking. On the other

hand, the selective-averaging background updating
technique was able to remove the shadows and detect the vehicles as well.
The operation of the proposed image detection
algorithm on a traffic image is shown in Fig. 8. Fig.
8(a) shows the current picture of the scene. The
thresholded edges of the background are shown in
Fig. 8(b). The binary picture resulting from the
subtraction of the edges of the background and the
current picture is shown in Fig. 8(c). The final binary
image (Fig. 8(d)) is obtained by applying noise
removal operations.
The result of testing shows that the image detection algorithm can detect vehicles passing through
windows nearly free of errors, and if the vehicles
move in their own lane, the counting program also
counts the vehicles with about 100% accuracy. However, in this particular situation, drivers didn't move
in their own lanes, therefore, up to 5% error is
produced due to changing lane of vehicles. Fig. 9
shows the graphical output of the result of counting
of vehicles for a period of 90 minutes under various
weather conditions.

8. Conclusion

The problems associated with the backgroundbased image detection techniques are mainly due to

1330

M. Fathy, M.Y. Siyal/Pattern Recognition Letters 16 (1995) 1321-1330

the variations of ambient lighting and threshold selection operations. In this paper an effective background updating technique based on automatic estimation of the ambient lighting changes and automatic selection of the pixels which should be updated is introduced. This selective-averaging updating technique has the advantages of the two most
commonly used updating techniques, i.e., averaging
and selective, while it eliminates the disadvantages
of these two algorithms. We also implemented a
dynamic threshold selection technique, which can be
used for both local and global thresholding schemes.
The proposed image detection technique has been
tested under various lighting conditions and satisfactory results have been achieved.
In some situations a more robust image detection
technique, which is not sensitive to the variations of
ambient lighting, is preferable to a low-cost image
detection technique. In those applications, a combined edge detection and background differencing
technique can be used. A low-cost approach to this
method using the morphological edge detection in
conjunction with the proposed background updating
technique is implemented. This algorithm has been
used in a wide range for full-frame and window-based
applications in real-time. The information computed
during the operation of the proposed algorithm is
also used for vehicle detection and motion detection,
without imposing significant extra computational
cost. This combination was used to eliminate the
edges of unimportant parts of roads, like shadows of
the trees, stopped cars, white road markings etc.
At present work is underway to extend the proposed vehicle-detection algorithm to calculate other
traffic parameters such as speed of vehicles, type of

vehicles and traffic movements at traffic junctions

e.g. length of the queue, occurrence of the queue etc.
References
Data Translation Manual (1993). Data Translation Inc., USA.
Dickinson, K.W. and R.C. Waterfall (1984a). Image processing
applied to traffic: a general review. Traffic Engineering ControlJ., Jan., 6-13.
Dickinson, K.W. and R.C. Waterfall (1984b). Image processing
applied to traffic: practical experiences. Traffic Engineering
Control J., Feb., 60-67.
Fathy, M. (1990). A RISC-type programmable morphologic image
processor. Ph.D. Thesis, UMIST, England.
Fathy, M., M.Y. Siyal and C.G. Darkin (1994). A low-cost
approach to real-time morphological edge detection. Proc.
IEEE TENCON Conf., Singapore.
Hoose, N. (1991). Computer Image Processing in Traffic Engineering. Research Studies Press, Taunton.
Hoose, N. (1992). IMPACT: an Image analysis tool for motorway
analysis and surveillance. Traffic Engineering ControlJ., Mar.,
140-147.
Krishnapuram, R. and S. Gupta (1992). Morphological methods
and classification of edges in range images. J. Mathematical
Imaging Vision 2, 351-375.
Lee, J. et al. (1987). Morphologic edge detection. IEEE J. Robot.
Automat. 3 (2).
Rourke, A. and M.G.H. Bell (1988). Traffic analysis using low
cost image processing. Proc. Seminar on Transportation Planning Methods, PTRC, Bath, UK.
Seed, N.L. and A.D. Houghton (1988). Background updating for
real-time processing at TV rates. SPIE 901, 73-81.
Siyal, M.Y., M. Fathy and C.G. Darkin (1994). Image processing
algorithms for detecting moving objects. ICARCV'94, Vol. 3,
1719-1723.
Takaba, S. et al. (1984). A traffic flow measuring system using a
solid state sensor. Proc. IEE Conf. on Road Traffic Data
Collection, London, UK.
TRIP Project Report (1986). Investigation into video image processing applied to traffic data collection. Reports, May-March
1983, UMIST, England.