INFRAREDPHYSICS &TECHNOLOGY

ELSEVIER Infrared Physics& Technology37 (1996) 231-238

Performance measurement in point source target detection

Meir Bar-Tal ",b, S.R. Rotman b
"Elisra Electronic Systems Ltd., Airborne Systems Dept., 48 Mivtza Kadesh St., Bene Beraq 51203, Israel bBen-Gurion University, Dept. of Electrical and Computer Engineering, P.O. Box 653, Beer Sheva 84105, Israel

Received 10 March 1995

Abstract
Certain point target detection algorithms applied to digital images can be evaluated through a formalism involving the generation of a threshold picture, i.e., replacing each actual pixel value with a threshold calculated from the background. Methods to evaluate the strengths and weaknesses of each algorithm are presented using this threshold picture. Moreover, the calculation of an electro-optical system sensitivity parameter (minimum detectable signal) is presented, based on the threshold image.

1. Background
It is of great interest to define a measure of performance for point source target detection algorithms. Point source detection is done for many infrared field applications-IRST (Infrared Search and Track), missile approach systems and other surveillance systems. In these applications, it is very important to define performance measures that relate to the operational system requirements - s u c h as the detection range, the probability of true and false target detection, etc. The definition of detection performance is not well established yet, due to the variety of algorithms and the dependence between the algorithm and its performance definition (see [1, 2]). Many point detectors have two stages: a transformation (such as Median, Laplacian, or other filters) and a statistical noise assessment set to a threshold (see [3]).

If the number of real targets is very small with respect to the quantity of data used for the statistical assessment, the threshold level would not be altered by the presence of real targets. This would ideally allow us to evaluate the target acquisition capability of any algorithm based solely on the algorithm itself and the target-less background. This article does not deal with any particular detection algorithm but rather with its performance measurement. The most important performance measure is the minimum detectable signal (MDS). This measure is well established in other areas such as communications and radar. The article examines the MDS definition, its calculation and its applicability to point target detection in electro-optical imagers. In order to measure the MDS, as it is suggested in this article, the algorithm performance must not change in the presence of a real target. This is

01350-4495/96/$15.00 O 1996ElsevierScienceB.V. All rights reserved SSDI 13 50-449 5(95)00050-X

232

M. Bar-Tal, S.R. Rotman/lnfrared Phyics & Technology 37 (1966) 231-238

Intensity

quite natural in detection algorithms. An algorithm usually assesses the noise itself (excluding real targets as much as possible) to set the threshold. It may even use a transformation that excludes the real targets at the very first stage-such as median based filters. For algorithms that satisfy this assumption, we will define the MDS and suggest a method for its measurement. In addition to calculating the MDS, a method to get good insight into the algorithm behavior is also presented. This method uses an image of the threshold distance from the actual background (TABI), which is set by the algorithm. The interpretation and use of this image is explained.

lhreshold

"

~
--

v ~

"

Image
Intensity

Fig. 1. Typical image line and the threshold set for it. used. This image is produced by subtracting each pixel in the original image from the threshold level set for that pixel. The result is a new image labeled TABI (Threshold Above Background Image). Most of the image pixels will be positive if a reasonable low false alarm is desired. Some points may be negative, indicating detections. The negative valued pixels in the TABI can be clamped to zero if one wishes to avoid detections.

2. Definitions

2. I. Point source target definition

A target, whose angular size, viewed by an imager, is much lower than the sensor IFOV (instantaneous field of view) (see Fig. 2). 2.2. MDS definition The minimum intensity of a point source target that is detected for a given Pd (probability of detection) using a threshold that maintains a constant Pfa (probability of false alarm).

3. Point target detection in an electro-optical imager The energy received by a single detector in an image is the integration of the flux impinging on it at the time it was open. The detector receives the flux from a spherical angle-the system IFOV (instantaneous field of view). When a point source target exists in the detector IFOV, the flux originates from the background and the point source targets (see Fig. 2). A point source target angular size is assumed to be much smaller than the imager IFOV. If, for simplicity, the background flux (Hb) is constant over the IFOV and the target (Ht) is constant, the received irradiance is approximately Hb + Ht. This does not take into account that the target energy may be spread over several detectors. The spread over the detectors can be assessed as a statistical process and will be dealt with in the next section. This means that for a point source target, the target flux is approximately added to the background. We observe that each pixel value in the TABI is the required irradiance that a point source target, had it been in this pixel area, would need to cross the threshold. If we apply this interpretation over

2.3. TABI (Threshold Above Background Image) definition

In the detection algorithms to be discussed in this paper, a threshold is set to each and every pixel in the image. The threshold is set by taking into consideration the pixel local surrounding and an assessment of the noise in the image. For most of the image pixels, the threshold is set above the pixels intensity (Fig. 1 illustrates a single typical line in an image and the threshold set for it). The threshold is usually adaptive to the image intensity and tries to be as close to it as possible, without causing excessive false detections. To observe the behavior of the threshold, an image of the threshold above the background is

M. Bar-Tal, S.R. Rotman/Infrared Phyics & Technology 37 (1966) 231-238

Optlcs

233

Detector

q i i q q t

T a r g e t Flux

Background

Flux

Fig. 2. The irradiance received by a detector from background and a point source target.

the TABI, we have a map of the energy that a target needs in order to cross the threshold at each pixel in the image.

4. Quantization effects
As stated above, point source target can fall between detectors. In this case, the energy received by a detector is Hb + k Ht; i.e., only part of the target energy falls on the detector, where k is the proportion parameter (between zero and one). The minimum energy that a detector that receives the maximum energy (compared to all other detectors) from the target can accumulate is a function of the sensor optical blur and the overlap between adjacent detectors. k is a random variable whose C D F (cumulative density function) can be calculated by building a histogram of evenly spaced targets, such as

illustrated in Fig. 3. Several simulation runs are performed. For each run, a target is placed in a different location-the marks in Fig. 3. For each run the energy accumulated at the detector that receives the maximum energy is used to build the C D F histogram. Since an imager is built of closely spaced detectors which overlap in space, the target energy always falls on at least one detector. There is a minimum energy that the detector which accumulated the maximum energy can receive. The minimum energy is received when the target falls exactly between four detectors. The C D F is built by our partitioning the energy range into bins or buckets and counting the number of times that the detector that receives the maximum energy has a lower energy than the bucket energy. The bin represents different values of k (the proportion between the target energy and the energy received by the detector that gets the maximum energy). Since the location where

(a) ~ Target Locatlon

(b)

Histogram
1 ..................

CDF

0.05

....

\ Central Detector

~k
Minimum energy over any detector that the target falls on,

T'

k(p--O.05)

Fig. 3. (a) Evenly spaced targets over a detector. (b) The CDF histogram result.

234

M. Bar-Tal, S.R. Rotman/Infrared Phyics & Technology 37 (1966) 231-238

the target center might fall on any detector is uniformly random, it is appropriate to spread the targets evenly over the detector. After the histogram is built it is normalized to produce a proper CDF (the maximum value is 1).

5. TABI Properties
By observing the TABI one can understand the behavior of the algorithm. For example, for Fig. 4, two algorithms are compared, one based on median to estimate the local background and another based on the maximum. The threshold in both of them was set to allow the same false alarm rate. The maximum based algorithm shows rectangles spread all over the TABI (see Fig. 6). Each rectangle is caused by one pixel in the image that

is higher than its local surrounding. The median based algorithm shows a much more regular TABI (see Fig. 5) which does not have the rectangles phenomenon. This behavior of the algorithm would be very hard to understand using classical probability measures (such as the receiver operating characteristics). Sometimes, the understanding of this behavior can be the most important in the algorithm design, especially when close targets are expected. Using the TABI one can also see where the algorithm gets closer to the actual image and where it far exceeds it. On our example, the image is of a sea and shore background. The thresholds of the two algorithms got closer in the sea and shore areas but were far from the border where they met. The rise of the threshold at the border is expected, and is representative of the good behavior of the

Fig. 4. Original image.

,.o

t~

t~

235

,,,-k

o6

h~

.o
~J

,4
V.

v.:

236

M. Bar-Tal, S.R. Rotman/Infrared Phyics & Technology 37 (1966) 231-238

237

algorithms due to the noisy nature of this area. Again, it is very hard to observe this behavior by using the classical statistical tools. Additional insight can be gained by marking the TABI locations with the highest level (say the top 5 %). This image locates the regions where the algorithm did the worse. The points determine the MDS level, as it will be explained in the next section. The TABI can be used also to compare algorithms. An algorithm is considered better than a second algorithm, if its threshold is closer to the image without having more false detections. To understand the different behavior of the two algorithms, the TABIs from the two algorithms can be subtracted one from another (see Fig. 7, Fig. 8). The resulting image shows the areas where the first or the second algorithm get closer to the image. One can investigate the reasons why one algorithm performs better than the other and what type of image causes this behavior. For example, one algorithm may raise the threshold around lines while the other stays low. It is not obvious that an algorithm that has a threshold closer to the image is necessarily better. This is true only if the two algorithms have the same false detection probability. In order to compare the algorithms, using the TABIs, the algorithm parameters have to be calibrated to have the same probability of false detection. Due to the statistical nature of the false detection probability, its measurement should be made by using many different images. The selection of these images should include as many scenarios as possible that the system may expect in real life operation. These images should have some areas containing the maximum amount of noise we can expect, and the different type of textures which will be found. Urban, rural and sea backgrounds are just some examples of vastly different areas which may need to be analyzed by the algorithm. The statistical moment of the TABI (or many TABIs) can be used to measure the algorithm performance in the statistical sense. The mean can be thought of as the average target energy that can be detected by the algorithm in a given image. The TABI variance indicates how well the algorithm adapt to different areas in the image. The last property is important for comparing algorithms.

6. Calculating the MDS

The calculation of the MDS for a certain image can now be performed. The calculations require several steps. First the threshold parameters of the algorithm have to be set to preserve the required probability of false detection (Pfa). Then the TABI is produced. Next, a threshold is set for TABI. It is placed above a fraction Pd of the TABI pixels. Fig. 9 illustrates this in one dimension. This threshold level (TABI-MDS) is the energy that a point source target needs in order to pass the threshold with Pd probability while preserving Pfa false detection probability. The last step is to account for the point source energy spread over several detectors. If Pd is to be preserved, the intensity of the target energy that falls on the "maximum" pixel has to be higher than the TABI-MDS. The probability of detection is PdTAm.MD s P(TABI-MDS > I), where PdTAaI.MD is the s probability of detection used to set TABI-MDS, assuming that the entire energy of the target was centered on the "maximum" pixel, P(TABI-MDS > I) is the probability that the TABI-MDS is higher than the energy (I) received by the detector that received the maximum energy from the target. P(TABI-MDS > I) can be found in the k CDF histogram. If, for example, one requires a final Pd of 0.9, PdTABI.MD and P(TABI-MDS > I) have to be s approximately 0.95 each (obviously one can select different probabilities that result in a value of 0.9). In order to have 0.95 probability to get an intensity of TABI-MDS, k has to be selected to have only

Intensity

~,///'~ 95%

TABI-MDS threshold ~-~'~./TABI

Fig. 9. Setting the T A B I - M D S for one line in the TAB1 image (where Pd = 95%).

238

M. Bar-Tal, S.R. Rotman/lnfrared Phyics & Technology 37 (1966) 231-238

0.05 probability to get lower values then TABIMDS. This k is found in the k CDF histogram at P of 0.05. The MDS can now be calculated by the formula: MDS =

(MDS) can be calculated for a given probability of detection and false alarm rate. Comparison of different algorithms has been described.

k(p = 0.05) TABI-MDS (Pd = 0.95).

References
[1] B.M. Sabol and K.G. Hall, Image Metrics approach to understanding effects of terrain and environment on performance of thermal target acquisition systems, SPIE V. 131 I, Characterization, Propagation and Simulation of Infrared Scenes (1990) pp. 284-302. [2] M. Burton and C. Benning, Comparison of imaging infrared detection algorithms, SPIE V. 302, Infrared Technology for Target Detection and Classification (1981) pp. 26-33. [3] D.T. Nagel and J. Saniie, Robust CFAR detection using order statistic process for Weibull disturbed clutter, SPIE V. 1841, Signal and data processing of small targets (1991) pp. 49-63.

7. Summary
In this paper, we have outlined a methodology for evaluating point detection algorithms. The key to evaluation is the realization that, for the algorithms we considered, the evaluation of each pixel is based on the difference between the pixel value and the threshold value evaluated solely on basis of the background pixels. Hence, thresholds can be calculated based on the algorithm and the background alone. Minimum Detectable Signal