Higher Order Bilateral Filters and Their Properties

Hiroyuki Takedaa , Sina Farsiua , and Peyman Milanfara

a Electrical

Engineering Department, University of California, 1156 High St., Santa Cruz, CA, USA
ABSTRACT

Bilateral ltering has proven to be a powerful tool for adaptive denoising purposes. Unlike conventional lters, the bilateral lter denes the closeness of two pixels not only based on geometric distance but also based on radiometric (graylevel) distance. In this paper, to further improve the performance and nd new applications, we make contact with a classic non-parametric image reconstruction technique called kernel regression,3 which is based on local Taylor expansions of the regression function. We extend and generalize the kernel regression method and show that bilateral ltering is a special case of this new class of adaptive image reconstruction techniques, considering a specic choice for weighting kernels and zeroth order Taylor approximation. We show improvements over the classic bilateral ltering can be achieved by using higher order local approximations of the signal.

1, 2

1. INTRODUCTION
The bilateral lter was rst proposed in Ref. 1 as a very eective one pass adaptive lter for denoising purposes while keeping the edges relatively sharp. Since its advent, the idea of bilateral ltering has been modied and improved, and its relation to some of the most popular image enhancement and reconstruction algorithms has been established.To take a notable example, Elad2 proved that such lter is a single Jacobi iteration of a weighted least squares minimization, and suggested using more iterations to enhance the smoothing eect of estimation. In related works, Barash et. al.4, 5 established the analogy between the geometric interpretation in anisotropic diusion, mean shift algorithm, and bilateral ltering. To reduce computational complexity and improve compression eciency, a modied implementation of bilateral lter is recently proposed in Ref. 6. In another venue, application of bilateral ltering for color image enhancement has been studied in Refs. 7,8. Moreover, a few papers have reported on a specic class of Maximum a Posteriori (MAP) image reconstruction techniques9, 10 that benet from coupling bilateral ltering with Total Variation11 dening very ecient adaptive regularization terms. The data-adaptive kernel regression framework, proposed in Ref. 12 based on the classic kernel regression framework,3 has wide ranging applications for image and video processing; for example, image denoising, and image interpolation/reconstruction from regularly and irregularly sampled data (i.e. image upscaling, image fusion, and super-resolution). In Ref. 12, we found that the bilateral lter1, 2 is a special case of data-adaptive kernel regression which is called bilateral kernel regression (BKR) or higher order bilateral lter. The purpose of this paper is to review BKR and examine its performance and behavior numerically and visually in ltered images.
Further author information: H.T. : E-mail: htakeda@soe.ucsc.edu S.F. : E-mail: farsiu@soe.ucsc.edu P.M. : E-mail: milanfar@soe.ucsc.edu This work was supported in part by AFOSR Grant F49620-03-1-0387.

2. HIGHER ORDER BILATERAL FILTERS

The kernel regression (KR) framework3 denes its data model in 2-D as yi = z(xi ) + i , i = 1, , P, xi = [x1i , x2i ]T , (1)

where yi is a noisy sample at xi , z() is the (hitherto unspecied) regression function to be estimated, i is an i.i.d zero mean noise, and P is the total number of samples in a neighborhood (window) of interest. As such, the kernel regression framework provides a rich mechanism for computing point-wise estimates of the regression function with minimal assumptions about global signal or noise models. While the specic form of z() may remain unspecied, we can rely on a generic local expansion of the function about a sampling point xi . Specically, if x is near the sample at xi , we have the (N + 1)-term Taylor series 1 T z(xi ) z(x) + {z(x)} (xi x) + (xi x)T {Hz(x)}(xi x) + 2 { } = 0 + T(xi x) + T vech (xi x)(xi x)T + , 2 1

(2)

where and H are the gradient (2 1) and Hessian (2 2) operators, respectively, and vech() is the halfvectorization operator which lexicographically orders the lower triangular portion of a symmetric matrix. Furthermore, 0 is z(x), which is the pixel value of interest, and the vectors 1 and 2 are [ ]T z(x) z(x) 1 = , (3) x1 x2 [ 2 ]T z(x) 2 z(x) 2 z(x) 2 = . (4) 2x2 x1 x2 2x2 1 2 Since this approach is based on local approximations, a logical step to take is to estimate the parameters { n }N from all the samples {yi }P while giving the nearby samples higher weights than samples farther away. n=0 i=1 A formulation of the tting problem capturing this idea is to solve the following optimization problem, min
P [ i=1

{ n }N n=0

]2 { } yi 0 T (xi x) T vech (xi x)(xi x)T KH (xi x) 1 2

(5)

where N is the regression order, KH () is the kernel function (a radially symmetric function), H(= hI) is the (2 2) smoothing matrix, and h is the global smoothing parameter. We control the strength of the smoothing eect by h. For images, the signal of interest often has discontinuities. Although one way to ensure that the estimated signal approximates such discontinuities is to set N large, the larger N we choose, the dramatically higher the computational complexity of the resulting algorithm becomes. Thus, without setting N large, we add an extra parameter to the kernel function, namely radiometric distance. That yields BKR: min
P ]2 [ { } yi 0 T (xi x) T vech (xi x)(xi x)T KHs (xi x)Khr (yi y), 1 2 i=1

{ n }N n=0

(6)

where KHs (xi x)Khr (yi y) is specically a product of two kernels: spatial kernel and radiometric kernel. Hs (= hs I) is the 2 2 smoothing matrix, hs is the spatial smoothing parameter, and hr is the radiometric smoothing parameter. These two smoothing parameters dictate the footprint of the bilateral kernel function. The optimization problem (6) can be expressed in matrix form as a weighted least-squares optimization problem,13, 14
2 b = arg min y Xx b Wx b ( )T ( ) = arg min y Xx b Wx y Xx b , b

(7)

Other localized representations are also possible and may be advantageous.

Figure 1. The footprints of the bilateral equivalent weight function Wi (xi x, yi y, N ) at a variety of image structures; at, strong edge, corner, texture, and weak edge for zeroth and second order (N = 0 and 2). (Note: Each weight function is respectively normalized, and Fig. 2 illustrates the detail of the weight function at the strong edge.)

where

[ ]T y = y1 , y2 , , yP ,

[ ]T T b = 0 , T , , N , 1

(8) (9)

[ ] Wx = diag KHs(x1 x)Khr(y1 y), KHs(x2 x)Khr(y2 y), , KHs(xP x)Khr(yP y) , and Xx = 1 1 . . . 1 (x1 x)T (x2 x)T . . . (xP x)T { } vechT {(x1 x)(x1 x)T } vechT (x2 x)(x2 x)T . . . { } T vech (xP x)(xP x)T . . .

(10)

with diag dening a diagonal matrix. Regardless of the regression order (N ), since our primary interest is to compute an estimate of the image (pixel values), the necessary computations are limited to the ones that estimate the parameter 0 . Therefore, the weighted least-squares estimation is simplied to ( )1 T z (x) = 0 = eT XT Wx Xx Xx Wx y, 1 x (11)

where e1 is a column vector with the rst element equal to one, and the rest equal to zero. This estimator can be summarized the form of the weighted linear combination of all the samples using the bilateral equivalent weight function Wi as follows: P z (x) = 0 = Wi (xi x, yi y, N ) yi . (12)
i=1

For N = 0, the estimator (12) becomes a generalization of the Nadaraya-Watson estimator15 with the bilateral kernel: P P i=1 KHs (xi x)Khr (yi y) yi 0 = z (x) = Wi (xi x, yi y, 0) yi = P , (13) i=1 KHs (xi x)Khr (yi y) i=1

16

x 10

x 10 4

x 10 15

14 Equivalent kernel weight, Wi 12 10 8 6 4 2 0

Zeroth order Second order

10

2 5 1 0

2 30

20

10 0 10 Spatial difference, x x
1i

20

30

(a) Zeroth order

(b) Second order

(c) Horizontal slices of Wi

Figure 2. The footprints of the bilateral equivalent weight function Wi (xi x, yi y, N ) at the strong edge for zeroth and second order (N = 0 and 2): (a)-(b) the footprints of Wi for zeroth and second order, respectively, (c) the horizontal slices pointed by the arrows of Wi of (a) and (b).

which is nothing but the recently well-studied and popular bilateral lter.1, 2 Fig. 1 illustrates the bilateral equivalent weight function Wi at a variety of image structures for the zeroth and second order cases (N = 0 and 2) (Note: Each weight function is respectively normalized), and Fig. 2 illustrates the details of Wi at the strong edge: (a)-(b) the footprints of Wi for zeroth and second order, respectively, (c) the horizontal slices pointed by the arrows of Wi of (a) and (b). The derivations for the bilateral lter (13) and the higher order bilateral lters for N = 1, 2 are fully explained in Ref. 12. We note that in general since the pixel value y at an arbitrary position x might not be available from the given data (e.g. in an interpolation setting), the direct application of (6) is limited to the denoising problem. This limitation, however, can be overcome by using an initial estimate of y by an appropriate interpolation technique (e.g. KR (5)), subsequently followed by the application of data-adaptive kernel regression as elucidated in Ref. 12. The derivation of the zeroth order bilateral lter (13) above indicates that the bilateral lter only consists of the constant term (0 ), and the consideration of only 0 is the cause of ltered image signals being piecewise constant. Such signals are often unsuitable for image processing because image signals of interest have complicated contours which include texture and gradation. Higher order bilateral lters relax the piecewise constancy by the choice of N > 0. For instance, images estimated by BKR with N = 1 and N = 2 become piecewise linear and piecewise quadratic, respectively. As a further extension of the standard bilateral lter, Elad suggested iterative ltering in order to intensify the smoothing eect in Ref. 2. The iterative ltering process is as follows: (i) apply bilateral lter to given noisy data, (ii) apply bilateral lter to the previous estimate, (iii) repeat the step (ii). For N = 0, such estimator can be written as ( ) P i=1 KHs (xi x)Khr z (xi ) z (x) z (xi ) +1 z (x) = (14) ( ) , P i=1 KHs (xi x)Khr z (xi ) z (x) where z 0(xi ) = yi and is the index of iterations. This ltering algorithm is very similar to the Mean-Shift algorithm.16, 17 In the next section, we show a simulation of higher order bilateral lter and its iterative ltering application, and compare to the results of standard (zeroth-order) bilateral ltering.

3. SIMULATION
First, we did denoising simulations by standard and higher order bilateral lters for a variety of SNRs: namely, 10, 15, 20, 25, and 30[dB]. At each noise level, we created 50 noisy images by adding white Gaussian noise with dierent realizations, and denoised by the standard bilateral lter (13) and the second order bilateral lter. The spatial and radiometric smoothing parameters (hs and hr ) in each case were optimized by the cross validation

9 8 7 Root Mean Square Error 6 5 4 3 2 1 Zeroth order Second order

10

12

14

16 18 20 22 24 Signal to Noise Ratio [dB]

26

28

30

Figure 3. Performance comparison between zeroth and second order bilateral lters by Monte Carlo simulations (50 times). The solid (blue) line and dashed (red) line are mean values and standard deviations of RMSE values in dierent SNRs by bilateral lter (N = 0) and second order bilateral lter (N = 2), respectively. The spatial and radiometric smoothing parameters are optimized by the cross validation method.

(CV) method .18 The average root mean square errors (RMSE) are shown in Fig. 3. Numerically, the second order bilateral lter is better than the bilateral lter at all noise levels for this example. Fig. 4 illustrates (a) the original image, (b) a noisy image (SNR = 20[dB]), and (c)-(d) denoised images by the bilateral lter (13) and the second order bilateral lter, respectively. Fig. 5(a) and Fig. 5(b) illustrate absolute residual images of Fig. 4(a) and Fig. 4(b), respectively. It is worth noting that some textures of the original image are visible in Fig. 5(a), and that the residual image in Fig. 5(b) is more noise-like than Fig. 5(a), indicating that the second order bilateral lter was more eective. It is also worth mentioning that in general, the zeroth order bilateral lter will perform better on average for denoising strictly piecewise constant images, but not so for any other general class of images. A further question worth considering is whether a scheme that would change the order N in a spatially varying way would lead to better results for a wider class of images. Next, the graph illustrated in Fig. 6 shows the behavior of iterative ltering using 50 Monte Carlo simulations in the case of SNR = 20[dB]. The solid (blue) line and the dashed (red) line are average RMSE values by the standard bilateral lter (13) and the second order bilateral lter, respectively, versus the number of iterations. We used the optimized spatial and radiometric smoothing parameters by the cross validation method in the previous simulation. The rst ltered images by the bilateral lter and the second order bilateral lter are shown in Fig. 4(a) and Fig. 4(b), respectively, and Fig. 7 shows the ltered images after three and seven iterations. The iterative zeroth order bilateral lter outputs piecewise constant images, and this property can be used for image segmentation, similarly to the output produced by the Mean Shift algorithm. On the other hand, the iterative second order bilateral lter outputs piecewise quadratic images, and the RMSE values for this example become worse as a function of iterations much slower than the iterative zeroth order bilateral lter.
In order to use CV, initial estimates for eliminated pixels are necessary, and we lled up the eliminated pixels by the second order classic kernel regression (5) with the global smoothing parameter h = 0.4.

(a) Original image

(b) Noisy image (SNR = 20[dB])

(c) Bilateral lter

(d) Second order bilateral lter

Figure 4. Denoising example of higher order bilateral lter (single application) in SNR = 20[dB]: (a) original image, (b) noisy image (SNR = 20[dB]), and (c)-(d) denoised images by the standard bilateral lter and the second order bilateral lter, respectively. The RMSE values for the denoised images are (c) 4.15 and (d) 3.84.

4. CONCLUSION AND FUTURE WORKS

We veried that the higher order bilateral lter, which was introduced in Ref. 12, is numerically better than the standard bilateral lter for image denoising, and studied its behavior in an iterative setting. In Ref. 19, we showed that the framework of general data-adaptive kernel regression methods, which includes the higher order bilateral lters, can be extended to other image reconstruction problems such as deblurring, and also indicated its applicability to the multi-frame scenario,12 namely super-resolution.

REFERENCES
1. C. Tomasi and R. Manduchi, Bilateral ltering for gray and color images, Proceeding of the 1998 IEEE International Conference of Compute Vision, Bombay, India , pp. 836846, January 1998. 2. M. Elad, On the origin of the bilateral lter and ways to improve it, IEEE Transactions on Image Processing 11, pp. 11411150, October 2002.

10

12

14

16

18

(a) Bilateral lter

(b) Second order bilateral lter

Figure 5. Absolute residuals images of the standard bilateral lter and the second order bilateral lter: (a) the standard bilateral lter, and (b) the second order bilateral lter.
Zeroth order Second order

11 10 9 8 7 6 5 4 3

Root Mean Square Error

3 4 5 The number of iterations

Figure 6. Behavior demonstration of iterative higher order bilateral lters by Monte Carlo simulations (50 times) in SNR = 20[dB]. The solid (blue) line and the dashed (red) line are average RMSE values by bilateral lter (N = 0) and second order bilateral lter (N = 2), respectively, versus the number of iterations. The spatial and radiometric smoothing parameters are optimized by the cross validation method.

(a) Bilateral lter (3 iterations)

(b) Second order bilateral lter (3 iterations)

(c) Bilateral lter (7 iterations)

(d) Second order bilateral lter (7 iterations)

Figure 7. Behavior of iterative higher order bilateral lters: (a) and (c) are the images by applying bilateral lter 3 times and 7 times, respectively, and (b) and (d) are the images by applying second order bilateral lter 3 times and 7 times, respectively. The noise level SNR = 20[dB] was used.

3. M. P. Wand and M. C. Jones, Kernel Smoothing, Monographs on Statistics and Applied Probability, Chapman and Hall, London; New York, 1995. 4. D. Barash, A fundamental relationship between bilateral ltering, adaptive smoothing and the nonlinear diusion equation, IEEE Transactions on Pattern Analysis and Machine Intelligence 24, pp. 844847, June 2002. 5. D. Barash and D. Comaniciu, A common framework for nonlinear diusion, adaptive smoothing, bilateral ltering and mean shift, Image Vis. Comput. 22(1), pp. 7381, 2004. 6. T. Q. Pham and L. J. van Vliet, Separable bilateral ltering for fast video preprocessing, Proceedings of the IEEE International Conference on Multimedia and Expo (ICME) 2005. , July 2005. 7. W.-C. Kao and Y.-J. Chen, Multistage bilateral noise ltering and edge detection for color image enhancement, IEEE Transactions on Consumer Electronics 51, pp. 13461351, November 2005.

8. S. Morillas, V. Gregori, and A. Sapena, Fuzzy bilateral ltering for color images, in Image analysis and recognition, Lecture Notes in Computer Science 4141, pp. 138145, Springer-Verlag, 2006. 9. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Fast and robust multi-frame super-resolution, IEEE Transactions on Image Processing 13, pp. 13271344, October 2004. 10. W. Shao and Z. Wei, Fast and robust ltering-based image magnication, in Vision Algorithms: Theory and Practice, Image Analysis and Recognition 4141, pp. 5362, 2006. 11. L. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Physica D 60, pp. 259268, November 1992. 12. H. Takeda, S. Farsiu, and P. Milanfar, Kernel regression for image processing and reconstruction, to appear in IEEE Transactions on Image Processing, February, 2007. 13. N. K. Bose and N. Ahuja, Superresolution and noise ltering using moving least squares, IEEE Transactions on Image Processing 15, pp. 22392248, August 2006. 14. S. M. Kay, Fundamentals of Statistical Signal Processing - Estimation Theory -, Signal Processing Series, PTR Prentice-Hall, Englewood Clis, N.J., 1993. 15. E. A. Nadaraya, On estimating regression, Theory of Probability and its Applications , pp. 141142, September 1964. 16. K. Fukunaga and L. D. Hostetler, The estimation of the gradient of a density function, with applications in pattern recognition, IEEE Transactions of Information Theory 21, pp. 3240, 1975. 17. D. Comaniciu and P. Meer, Mean shift: A robust approach toward feature space analysis, IEEE Transactions on Pattern Analysis and Machine Intelligence 24, pp. 603619, May 2002. 18. M. Stone, Cross validatory choice and assessment of statistical predictions, Journal of the Royal Statistical Society, Series B (Methodological) 36(2), pp. 111147, 1974. 19. H. Takeda, S. Farsiu, and P. Milanfar, Regularized kernel regression for image deblurring, Proceedings of the 40th Asilomar Conference on Signals, Systems, and Computers, Pacic Grove, CA , November 2006.