2011 18th IEEE International Conference on Image Processing

VIDEO RESOLUTION ENHANCEMENT BY USING COMPLEX WAVELET TRANSFORM

1

Hasan Demirel1, Gholamreza Anbarjafari2, Cagri Ozcinar3, and Sara Izadpanahi1 Department of Electrical and Electronic Engineering, Eastern Mediterranean University, Gazimausa, via Mersin-10, Turkey email:{hasan.demirel,sara.izadpanahi}@ emu.edu.tr 2 Department of Electrical and Electronic Engineering, Gazikent University, ahinbey, Gaziantep, Turkey email: shahab.jafari @ gazikent.edu.tr 3 Department of Electronics Engineering, University of Surrey, GU2 7XH, Surrey, UK email: c.ozcinar @ surrey.ac.uk
[7]. They have developed a motion estimation algorithm, which considers translations and rotations in spatial domain. Other researchers have also focused on resolution enhancement of low resolution video sequences [8-10]. Vandewalle et al. considered a frequency domain technique to specifically register a set of aliased images [11]. In their method frames were differently considered a frequency domain technique to specifically register a set of aliased frames. Frames are differently considered by a planar motion method. They focused on this problem that an error in the motion estimation translates almost directly into a degradation of the resulting high resolution frame. They proposed that interpolating one of the low resolution frames is better than to create a high resolution frame from the set of frames using incorrect motion parameters [11]. Their proposed algorithm used low frequency information which has the highest signal-to-noise ratio (SNR), and in their setup, the aliasing-free part of the frames. Nowadays, wavelet transform is also widely used in many image processing applications, especially in image and video super resolution techniques [12-15]. A one-level dual tree complex wavelet transform (DT-CWT) of a single frame of a video sequence produces complex-valued low frequency subbands and complex-valued high frequency subbands with more directivity with respect to the DWT [16]. In this work, we have proposed a new video resolution enhancement technique which generates sharper super resolved video sequences. The proposed technique uses DTCWT to decompose low resolution frames of the video sequences into frequency subbands. Then the six complexvalue high frequency subbands with +75, +45, +15, -15, -45, and -75 orientations of the respective frames have been enlarged by using bicubic interpolation. In parallel, the input low resolution frames have been super resolved by using Vandewalle technique separately [11]. In this paper, we have also proposed a new DWT based illumination compensation method by using SVD of low frequency subband of the frame, iteratively. The illumination compensation technique is applied on the frames before the implementation of Vandewalle resolution enhancement technique. Finally, the interpolated high frequency subbands and their respective super resolved input frames have been combined by using inverse DT-CWT (IDT-CWT) to reconstruct a high resolution output video sequence. The

ABSTRACT In this paper, we propose a multi-frame video resolution enhancement technique based on dual tree complex wavelet transform (DT-CWT). Here, before registration step, the respective frames have been passed through an illumination compensation procedure which is based on singular value decomposition (SVD) and discrete wavelet transform (DWT). The frame subject to the resolution enhancement is decomposed into its different frequency subbands by using DT-CWT. The high frequency subbands have been interpolated by using bicubic interpolation. Furthermore, the compensated frames are registered by using Vandewalle registration with structure adaptive normalized convolution reconstruction. Afterwards, the interpolated subbands and the output of the registration technique have been combined by using inverse DT-CWT (IDT-CWT) in order to reconstruct the super resolved frame. For Akiyo video sequence there is 5.04 dB, 4.86 dB, 5.22 dB, and 5.05 dB improvements in the average PSNR values compared to Vandewalle, Marcel, Lucchese, and Keren registration techniques, respectively. 1. INTRODUCTION There are two main approaches in the literature for performing the resolution enhancement. The first approach is multi-frame super resolution based on the combination of frame information from several similar frames taken from a video sequence [1]. The second approach is referred as single-frame super resolution, which uses prior training data to enforce super resolution over a single low resolution input image and usually referred as enlargement [2]. In this work we are following the first approach. Tsai and Huang are the pioneers of super resolution idea [3], who used the frequency domain approach. Further work has been done by Keren et al. [4] who described a spatial domain procedure by using a global translation and rotation model in order to perform image registration. Reddy and Chatterji [5] introduced a frequency domain approach for super resolution. In their method a registration algorithm based on Fourier domain approach to align frames translated and rotated with respect to each other, was proposed [5]. Later on, Cortelazzo and Lucchese [6] presented a method for estimating planar roto-translations that operates in the frequency domain. Another very popular registration technique which is used in many video resolution enhancement applications was proposed by Irani and Peleg

978-1-4577-1303-3/11/$26.00 2011 IEEE

2093

2011 18th IEEE International Conference on Image Processing

The algorithm of the illumination enhancement technique is shown in Fig. 1. In this paper, we have used db.9/7 wavelet function is used in all the steps of the proposed technique. 2.2. Proposed Video Resolution Enhancement Technique In the present work, DT-CWT [16, 17] has been applied in order to preserve the high frequency components of each frame. The DT-CWT has good directional selectivity. It also has limited redundancy [17]. The DT-CWT is approximately shift invariant, unlike the critically sampled DWT. The redundancy and shift invariance of the DT-CWT mean that DT-CWT coefficients are inherently interpolable [17]. The one level DT-CWT process for each frame of the input video generates eight video sequences (i.e. a complex-value low frequency subband sequence which can be divided into a real-valued real sequence and a real-valued imaginary sequence- and six complex-valued high frequency subband sequences with +75, +45, +15, -15, -45, and -75 orientations). In parallel to DT-CWT process, the Vandewalle registration technique with structure adaptive normalized convolution reconstruction method [11] is applied to video sequences in spatial domain. The super resolved frame and the shifted version of the same frame (translated) in horizontal and vertical directions are used as the low frequency of the super resolved subband for reconstruction process. In other word, these two real valued new frames are used as the real and imaginary components of the interpolated complex low frequency frames respectively, for IDT-CWT. Finally, IDT-CWT is used to reconstruct the super resolved subbands to produce the resolution enhanced frame. By super resolving the different subbands of video sequences and then by applying IDT-CWT, the output video sequence contains sharper edges. This is due to the fact that, the proposed super resolution technique isolates high frequency components and preserves more high frequency components after the super resolution of the respective subbands separately than other super resolution technique. Fig. 2 illustrates the block diagram of the proposed video resolution enhancement technique. In all steps of the proposed technique db.9/7 wavelet function and bicubic interpolation are used. In the next section, the result of comparison between the proposed technique with the conventional and state-of-art techniques is discussed. 3. RESULTS AND DISCUSSIONS The proposed resolution enhancement technique is compared with bicubic interpolation as a conventional technique and with the several state-of-art super resolution techniques using Vandewalle [11], Marcel [9], Lucchese [6], and Keren [4] registration followed by interpolation, iterated back projection, robust super resolution, and structure adaptive normalized convolution techniques for reconstruction. The experimental simulations are conducted on four well known video sequences [21], namely Mother daughter, Akiyo, Foreman, and Container.

Table 1 shows the average PSNR values of the super resolution techniques for the aforementioned video sequences. The average PSNR performance of the proposed method reported in Table 1 shows that the proposed method outperforms the conventional and state-of-art techniques. The low resolution video sequences are generated by downsampling and lowpass filtering each frame of the high resolution video sequence [22]. All video sequences have 300 frames and the reported average PSNR values in Table 1 are the average of 300 PSNR values. The low resolution video sequences have the size of 128x128 and the super resolved sequences have the size of 256x256. 4. CONCLUSION
This paper proposed a new video resolution enhancement technique for low resolution video sequences by applying a DWT based illumination compensation technique using SVD before registration process and applying DT-CWT in order to preserve the high frequency components (in various directions) of the frames. The proposed methods benefitted from the Vandewalle registration technique with structure adaptive normalized convolution reconstruction method in order to super resolve the low frequency subbands and bicubic interpolation in order to enlarge the high frequency subbands. Afterwards all these subbands have been combined using IDT-CWT to generate respective super resolved frame.

5. ACKNOWLEDGEMENTS
Authors would like to thank Prof. Dr. Ivan Selesnick from Polytechnic University for providing the DWT codes in MATLAB.

REFERENCES
[1] M. Elad and A. Feuer, Super-resolution reconstruction of image sequences, IEEE Trans. On Pattern Analysis and Machine Intelligence, Vol. 21, 1999, pp. 817 - 834. [2] J. D. van Ouwerkerk, Image super-resolution survey, Image and Vision Computing, Vol. 24, 2006, pp. 1039-1052. [3] R. Y. Tsai and T. S. Huang, Multiframe image restoration and registration, in Advances in Computer Vision and Image Processing, T. S. Huang, Ed. JAI Press, Vol. 1, 1984, pp. 317339. [4] D. Keren, S. Peleg and R. Brada, Image sequence enhancement using subpixel displacements, IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 1988, pp. 742-746. [5] B. S. Reddy and B. N. Chatterji, An fft-based technique for translation, rotation and scale-invariant image registration, IEEE Transactions on Image Processing, Vol. 5, No. 8, August 1996, pp. 12661271. [6] L. Cortelazzo and G. M. Lucchese , A noise-robust frequency domain technique for estimating planar roto translations, IEEE Transactions on Signal Processing, Vol. 48, No. 6, June 2000, pp. 17691786. [7] M. Irani and S. Peleg, Improving resolution by image registration, Graphical Models and Image Processing, Vol. 53, No. 3, May 1991, pp. 231239. [8] H. Demirel, S. Izadpanahi, "Motion-Based Localized Super Resolution Technique for Low Resolution Video Enhancement", 16th European Signal Processing Conference, Switzerland, Aug. 2008.

2095

2011 18th IEEE International Conference on Image Processing

Vandewalle SR with factor /2

Low frequency Subband frame (n/2xm/2)

the low resolution the low input frame low resolution the inputresolution frame (nxm) the low input (nxm) frame resolution

Illumination compensation (as shown in Fig. 1)

the low resolution input frame

DT-CWT
+75

Bicubic interpolation with factor

IDT-CWT
+75

(nxm) input frame (f4)

(nxm)

The respective super resolved frame (2n x 2m)

Bicubic interpolation with factor

+45 +45

+15

Bicubic interpolation with factor

+15

-15

Using Irani and Peleg SR with factor

-15

Bicubic interpolation with factor

-45

-45

Bicubic interpolation with factor

-75

-75

High frequency subbands (n/2xm/2)Super resolved high frequency subbands (n/2xm/2)

Fig. 2: The block diagram of the proposed video resolution enhancement technique.
[9] B. Marcel, M. Briot, and R. Murrieta, Calcul de Translation et Rotation par la Transformation de Fourier, Traitement du Signal, Vol. 14, No. 2, 1997, pp. 135-149. [10] H. Demirel, G. Anbarjafari, and S. Izadpanahi, "Improved Motion-Based Localized Super Resolution Technique Using Discrete Wavelet Transform for Low Resolution Video Enhancement", 17th European Signal Processing Conference, Glasgow, Scotland, Aug. 2009, pp. 1097-1101. [11] P. Vandewalle, S. Ssstrunk and M. Vetterli, A Frequency Domain Approach to Registration of Aliased Images with Application to Super-Resolution, EURASIP Journal on Applied Signal Processing (special issue on Super-resolution), Vol. 2006, Article ID 71459. [12] Y. Piao, l. Shin, and H. W. Park, Image Resolution Enhancement using Inter-Subband Correlation in Wavelet Domain, International Conference on Image Processing (ICIP 2007), Vol. 1, pp. I- 445-448. [13] H. Demirel, C. Ozcinar, and G. Anbarjafari "Satellite Image Contrast Enhancement Using Discrete Wavelet Transform and Singular Value Decomposition", IEEE Geoscience and Remote Sensing Letter, Vol. 7, April 2010, pp. 334-338. [14] A. Temizel and T. Vlachos, Wavelet domain image resolution enhancement using cycle-spinning, Electronics Letters, Vol. 41, Issue 3, Feb. 2005, pp. 119 121. [15] G. Anbarjafari and H. Demirel, "Image Super Resolution Based on Interpolation of Wavelet Domain High Frequency Subbands and the Spatial Domain Input Image", ETRI Journal, Vol. 32, No. 3, Jun 2010, pp. 390-394. [16] N.G. Kingsbury, Image processing with complex wavelets, Phil. Trans. Royal Society London A, Sept. 1999, Vol. 357, pp. 2543-2560. [17] T. H. Reeves, and N. G. Kingsbury, Prediction of coefficients from coarse to fine scales in the complex wavelet transform, IEEE Proceedings International Conference on Acoustics, Speech, and Signal Processing, 2000 (ICASSP 00), 5-9 June, vol. 1, pp. 508 511. [18] A. Zomet, A. Rav-Acha, and S. Peleg, Robust superresolution, Proceedings on international conference on computer vision and pattern recognition (CVPR), 2001, pp. I645 - I-650. [19] T. Q. Pham, L. J. van Vliet, and K. Schutte, Robust Fusion of Irregularly Sampled Data Using Adaptive Normalized Convolution, EURASIP Journal on Applied Signal Processing, Vol. 2006, Article ID 83268. [20] G. Anbarjafari and H. Demirel, Face Recognition by Using Statistical Analysis, LAMBERT Academic Publishing, ISBN13: 978-3-8433-7068-4, Nov. 2010. [21] Xiph.org Test Media, Retrieved on October 2010, from the World Wide Web on http:// media.xiph.org/video/derf/ [22] A. Temizel, Image resolution enhancement using wavelet domain hidden Markov tree and coefficient sign estimation, International Conference on Image Processing (ICIP2007), Vol. 5, pp. V-381-384. Table 1: The average PSNR (dB) values of different resolution enhancement techniques on the test video sequences
RESOLUTION ENHANCEMENT TECHNIQUE REGISTRATION RECONSTRUCTION Bicubic Interpolation Interpolation Iterated Back Projection Vandewalle Robust SR Structure Adaptive Normalized Convolution PSNR (dB) VALUE FOR DIFFERENT SEQUENCES MOTHER DAUGHTER AKIYO FOREMAN CONTAINER

Marcel Lucchese Keren

Structure Adaptive Normalized Convolution Structure Adaptive Normalized Convolution Structure Adaptive Normalized Convolution

22.30 24.29 27.1 27.15 28.95 28.66 29.01 28.63 33.10

26.90 29.45 31.49 31.5 32.98 33.16 32.8 32.97 38.02

26.00 28.01 30.17 30.24 33.46 33.25 33.3 33.25 37.97

22.55 23.6 24.3 24.46 26.38 26.28 26.36 26.15 31.67

Proposed video resolution enhancement technique

2096