Anda di halaman 1dari 14

ARTICLE IN PRESS

Pattern Recognition 43 (2010) 2003–2016

Contents lists available at ScienceDirect

Pattern Recognition
journal homepage: www.elsevier.com/locate/pr

Multi-focus image fusion using PCNN


Zhaobin Wang a,b, Yide Ma a,, Jason Gu b
a
School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China
b
Department of Electrical and Computer Engineering, Dalhousie University, Halifax, Nova Scotia, Canada B3J 2X4

a r t i c l e in fo abstract

Article history: This paper proposes a new method for multi-focus image fusion based on dual-channel pulse coupled
Received 13 May 2009 neural networks (dual-channel PCNN). Compared with previous methods, our method does not
Received in revised form decompose the input source images and need not employ more PCNNs or other algorithms such as
24 November 2009
DWT. This method employs the dual-channel PCNN to implement multi-focus image fusion. Two
Accepted 17 January 2010
parallel source images are directly input into PCNN. Meanwhile focus measure is carried out for source
images. According to results of focus measure, weighted coefficients are automatically adjusted. The
Keywords: rule of auto-adjusting depends on the specific transformation. Input images are combined in the dual-
PCNN channel PCNN. Four group experiments are designed to testify the performance of the proposed
Image fusion
method. Several existing methods are compared with our method. Experimental results show our
Focus measure
presented method outperforms existing methods, in both visual effect and objective evaluation criteria.
Finally, some practical applications are given further.
& 2010 Elsevier Ltd. All rights reserved.

1. Introduction based on the selection of image blocks from source images [14].
The idea of this scheme is to select the better image blocks from
Optics of lenses with a high degree of magnification suffers source images to construct the fused image. Clearly, the latter is
from the problem of a limited depth of field. The larger the focal simpler than the former. However, the latter has some disadvan-
length and magnification of the lens, the smaller the depth of field tages. For instance, the effect of this scheme depends on focus
becomes. As a result, fewer objects in the image are in focus. measurement to a great extent. Some errors in the fused image
However, sometimes people like to get the image in which more are obtained by this scheme. Therefore, the third scheme is
objects are in focus. This is to say that people want to see employed in the paper.
more and clearer objects in one image. The perfect state is that PCNN is a biologically inspired neural network based on the
the whole image is clear or in focus. This directly leads to the work by Eckhorn et al. [4]. Pioneering work in the implementation
appearance of multi-focus image fusion technology. of these algorithms was done by Johnson and his colleagues
Multi-focus image fusion is the process in which different [11,10,9,21]. It has been proven that PCNN is widely applicable in
images with different focus settings are fused to produce a new the field of image processing [18,19] such as image segmentation,
image with extended depth of field. Its purpose is to attempt to image enhancement, pattern recognition, etc. PCNN also plays an
increase the apparent depth of field through the fusion of object important role in image fusion. In fact, Broussard et al. [2] applied
within several different fields of focus. Hence it plays important PCNN into image fusion for object detection as early as 1999. In
roles in many different fields such as biomedical imaging and the same year Johnson and Padgett [9] pointed out that there was
computer vision. Multi-focus image fusion is one of the main the great potential for PCNN in the field of image and data fusion.
research fields of image fusion. Many multi-focus image fusion algorithms based on PCNN
Nowadays, there are two common schemes to be used in the [7,8,13,20,2,16] have been published in different journals or
field of multi-focus image fusion. The one is to use multi- proceedings so far. Although different authors have different
resolution approaches which usually employ the discrete wavelet schemes, most of them employ the same characteristic of PCNN,
transform (DWT) [15,22] or various pyramid algorithms such as namely, the mechanism of synchronous pulse bursts.
contrast pyramid [25], FSD pyramid [1], gradient pyramid [3], For multi-focus image fusion, different from other methods,
morphological pyramid [24], etc. However, this scheme is PCNN has its own advantages. Firstly, PCNN model derives from
complicated and time-consuming to implement. The other is the research on cat visual cortex. Its mode of information
processing is much closer to the mode of human visual
processing. And then PCNN has the flexible structure which can
 Corresponding author. be changed according to different tasks. Additionally, the existing
E-mail addresses: zhaobin_wang@hotmail.com (Z. Wang), PCNN methods also show PCNN has the higher performance.
ydma@lzu.edu.cn (Y. Ma). However, the problem is that the existing PCNN methods are still

0031-3203/$ - see front matter & 2010 Elsevier Ltd. All rights reserved.
doi:10.1016/j.patcog.2010.01.011
ARTICLE IN PRESS
2004 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

complicated and time-consuming. After careful analysis, we find: feeding). The linking receives local stimulus from the output of
(1) almost all the methods with PCNN employ more PCNNs surrounding neurons, while the feeding, besides local stimulus,
[7,8,13,20] or combine with other algorithms such as DWT [2,16]; still receives external stimulus.
(2) almost authors adopt the simplified PCNN model and all In the following expressions, the indexes i and j refer to the
models have one stimulus. Generally an intricate algorithm pixel location in the image, k and l refer to the dislocation in a
usually costs much time to compute and operate intermediate symmetric neighborhood around one pixel, and n denotes the
variables so as to make the whole system inefficient. From (1) we current iteration (discrete time step). Here n varies from 1 to
know that their schemes are very complicated, it is why these N (N is the total number of iterations)
existing methods are not efficient. From (2) we believe that only X
Fij ½n ¼ eaF Fij ½n1 þ VF wijkl Yij ½n1 þSij ð1Þ
having one stimulus is the root cause of complication and k;l
inefficiency of PCNN methods. In other words, to some extent,
X
the standard PCNN structurally limits its application in image Lij ½n ¼ eaL Lij ½n1 þVL mijkl Yij ½n1 ð2Þ
fusion. In order to make PCNN more suitable for image fusion, we k;l
improve the standard PCNN and propose the dual-channel PCNN,
which can solve the problem of complication and inefficiency of Uij ½n ¼ Fij ½nð1 þ bLij ½nÞ ð3Þ
PCNN methods very well. (
Compared with previous methods, this method with the dual- 1; Uij ½n 4 Tij ½n
Yij ½n ¼ ð4Þ
channel PCNN does not decompose the input source images and 0; otherwise
need not employ more PCNNs or combine with other algorithms
such as DWT. So its scheme of image fusion is very simple and Tij ½n ¼ eaT Tij ½n1 þ VT Yij ½n ð5Þ
experimental results also show the proposed method is feasible
The dendritic tree is given by Eqs. (1)–(2). The two main
and efficient.
components F and L are called feeding and linking, respectively.
The rest of the paper is organized as follows. In Section 2, the
wijkl and mijkl are the synaptic weight coefficients and S is the
standard PCNN is briefly reviewed, and then the dual-channel
external stimulus. VF and VL are normalizing constants. aF and aL
PCNN is introduced in details. Section 3 concretely describes our
are the time constants; generally, aF o aL. The linking modulation
proposed multi-focus image fusion algorithm based on the dual-
is given in Eq. (3), where Uij[n] is the internal state of the neuron. b
channel PCNN. It mainly includes focus measurement introduc-
is the linking parameter and the pulse generator determines the
tion, principle and implementation of algorithm, and other
firing events in the model in Eq. (4). Yij[n] depends on the internal
relative contents. Section 4 gives experimental results and
state and threshold. The dynamic threshold of the neuron is
performance evaluation, and then introduces some practical
Eq. (5), where VT and aT are normalized constant and time
applications. Conclusions are summarized in the end.
constant, respectively. Here is a brief review of the standard PCNN.
The detailed description of the implementation of the standard
PCNN model on digital computers can be found in literature [9].
2. PCNN model
More details about PCNN will be found in the literatures [18,19].

Because the dual-channel PCNN is proposed based on the


2.2. Dual-channel PCNN model
PCNN model, firstly PCNN model is briefly introduced. After
analyzing its standard model, we will improve the model
Analysis of the PCNN exposes a defect preventing one PCNN
according to practical demands of multi-focus image fusion.
from fusing multi-focus images. In this case, we modify the
standard model and propose a new improved model which
2.1. Standard PCNN model overcomes some limits of the standard model in multi-focus
image fusion.
In the standard PCNN model, the PCNN neuron consists of The dual-channel neuron model (see Fig. 2) consists of three
three parts: the dendritic tree, the linking modulation, and the parts: the dentritic tree, information fusion pool and the pulse
pulse generator, as shown in Fig. 1. generator. The function of dentritic tree is to receive the stimulus
The role of the dendritic tree is to receive the inputs from two including external inputs and surrounding neuron stimuli;
kinds of receptive fields. Depending on the type of the receptive information fusion pool is the place where all data are fused;
field, it is subdivided into two channels (the linking and the the pulse generator is to generate the output pulse.

surrounding
neurons
Si j
exp( −α F ) Threshold

VF exp( −α T ) VT
W Ti j
Fi j
Feeding

1 Ui j Yi j
Linking
Li j β
M
VL

exp(−α L )

Dendritic tree Linking modulation Pulse generator

Fig. 1. The structure of PCNN.


ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2005

surrounding
neurons SijA β ijA
1 Threshold
A
H ij exp( −α T ) VT
W σ Ti j

Ui j Yi j

M
H ijB
1

S B
ij β B
ij

Dendritic tree Information fusion pool Pulse generator

Fig. 2. The structure of dual-channel PCNN.

In the improved model there are two input channels. So both (5) If all neurons have fired, go to the next step, else go back to
stimuli can be input into the model at the same time. The step (4).
following expressions describe its mathematical model: (6) O is the output of the dual-channel PCNN.
X
HijA ½n ¼ SAij þ wijkl Ykl ½n1 ð6Þ
Note, ‘‘a neuron fires’’ means a PCNN’s neuron generates a pulse.
k;l
Here we would like to explain the principle of data fusion
X
HijB ½n ¼ SBij þ mijkl Ykl ½n1Þ ð7Þ using the dual-channel PCNN. Eqs. (6)–(8) mathematically show
k;l the way of data fusion. For one neuron, the final output value is
A B not completely determined by weighting coefficients (bA and bB).
Uij ½n ¼ ð1þ bij HijA ½nÞð1 þ bij HijB ½nÞ þ s ð8Þ
The output from surrounding neurons also plays a role via the
( mechanism of synchronous pulse bursts and this output is
Uij ½nSurij ½n; Uij ½n 4 Tij ½n1
Yij ½n ¼ ð9Þ unexpected and even random for one neuron. The mechanism
0; otherwise of synchronous pulse bursts, implemented by time-domain
( iterative processing, can make neurons with similar state
eaT Tij ½n1; Yij ½n ¼ 0 generate synchronous pulses; hence, the way of PCNN fusing
Tij ½n ¼ ð10Þ
VT ; otherwise data is not linear but nonlinear.
For the dual-channel PCNN model, it inherits good features
Compared with the standard PCNN, the dual-channel PCNN
from the standard PCNN. For example, the dual-channel model
has few parameters. In the dual-channel model, instead of feed
remains the mechanism of synchronous pulse bursts. The
channel (F) and link channel (L), HA and HB stand for two
exponential attenuation characteristic of the threshold is also
symmetrical input channels. bA and bB are the weighting coeffi-
kept. It is believed that this characteristic is coincident with
cients of two symmetrical channels, s the level factor to adjust the
human visual characteristic. These remaining features are
average level of internal activity. When s =  1, UZ0. Parameters
propitious to image processing.
(U, T, Y, VT, wijkl, mijkl, and aT) have the same meanings as these in
the standard model. Sur denotes the input from surrounding
P
neurons. Generally, kijkl ¼ wijkl ¼ mijkl ; Surij ¼ k;l kijkl Ykl ½n1. 3. Image fusion algorithm
Now we describe the data fusion process of the dual-channel
PCNN. Firstly, two channels of neuron receive external stimuli and In the section, multi-focus image fusion scheme used in the
output of surrounding neurons. Furthermore, the data from these paper is introduced. Image sharpness measure is so important for
channels are weighted and mixed in the information fusion pool multi-focus image fusion that some common evaluation methods
according to the weighting coefficients. Finally, the mixed data of image sharpness are given in order to make the paper
are released by neuron as its output with the attenuation of the understandable and logical.
threshold. The implementation of the dual-channel PCNN in our
experiments is described as follows: 3.1. Image focus measure

(1) Initialize parameters and matrices. U=O = Y= 0, T= 1. The For multi-focus image fusion, we usually select one or more
Initialization of W and M are different from those of other focus measure methods to estimate image sharpness. So, it is
matrices. Here K= W=M. Its values are determined manually. crucial to choose a good focus measure method.
(2) If SA =SB, then O = SA or SB; and go to step (6) The focus measure defined in literatures [12,17] is a maximum
(3) Normalize external stimuli to lie within [0, 1]. for the best focused image and it generally decreases as the
(4) Sur= YK; defocus increases. That is, the focused image should produce
HA = SA +Surij; maximum focus measures, while the defocused should produce
HB = SB + Sur; minimum focus measures.
U= (1 + bA  HA) (1+ bB  HB)+ s; Recently, there are many focus measure techniques. Here are
If Uij 4Tij then Yij = Uij  Surij, else Yij = 0; some focus measure methods: variance, energy of image gradient
If S Aij ¼ S Bij or bAij ¼ bBij , then Oij ¼ S Aij or S Bij ; else Oij = Yij; (EOG), energy of Laplacian of the image (EOL), sum-modified-
If Yij = 0 then Tij =exp(  aT)  Tij, else Tij = VT; Laplacian (SML), and spatial frequency (SF). Huang and Jing [7,8]
ARTICLE IN PRESS
2006 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

assessed these methods according to some objective standards. weighting coefficients [5]. This method is explained in details in
Experiment results show that SML and EOL can provide better the following paragraphs.
performance than other focus measures. However, SML spends Suppose source images IA(i, j) and IB(i, j) turn into measured
more implementation time under the same conditions. Therefore, images MA(i, j) and MB(i, j) after focus measure. The difference
EOL is used in the paper to measure image focus. Because between MA(i, j) and MB(i, j) is defined by: D(i, j)=MA(i, j)–MB(i, j).
evaluation of different focus measure methods is not our topic of Generally speaking, if D(i, j)40, it indicates the pixel value at
this paper, more details can be found in the Refs. [7,8]. For EOL, its location (i, j) in image A should be chosen. Otherwise, its counterpart
corresponding mathematic expressions are also given as follows. from B is selected. However, this measure alone in practice is not
Let f(i, j) denote the gray level intensity of pixel (i, j). sufficient to pick out the better focused image on a pixel-by-pixel
XX basis, since using single pixel information to make decision is
EOL ¼ ½f ði1; j1Þ4f ði1; jÞf ði1; j þ 1Þ vulnerable to wide fluctuations caused by outer environment such
i j
as various noises. So, it is necessary to maintain robustness of the
4f ði; j1Þ þ 20f ði; jÞ4f ði; j þ 1Þf ði þ1; j1Þ algorithm through more information from neighboring pixels.
4f ði þ 1; jÞf ði þ1; j þ1Þ2 ð11Þ In order to make full use of surrounding information, it is
necessary to require summing the D(i, j) s over a (r+ 1)  (r + 1)
This method carries out focus measure by analyzing high region surrounding each decision-point
spatial frequencies associated with image border sharpness, and it r=2
X r=2
X
is implemented through Laplacian operator. Dði; jÞ ¼ Dði þ m; j þ nÞ ð12Þ
m ¼ r=2 n ¼ r=2

Hence their weighting coefficients are


3.2. Principle of multi-focus mage fusion
1
bAij ¼ ð13Þ
Now we explain how to fuse multi-focus image using the dual- 1þ eZDði;jÞ
channel PCNN. Suppose IAand IB are two multi-focus images with the and
same size. IA(i, j) and IB(i, j) denote the pixels of IAand IB in the same
position, respectively. The goal of multi-focus image fusion is to get 1
bBij ¼ ð14Þ
clear one from IA(i, j) and IB(i, j). Actually, from Section 2.2, we know 1þ eZDði;jÞ
the dual-channel PCNN can carry on data fusion. Because the data where Z is a constant. Constant Z has an important influence on
from two channels are weighted according to the weighting weighting coefficients bA and bB (shown in Fig. 3). Larger Z
coefficients, as long as clear pixel get large weighting coefficient markedly increases the difference between bA and bB while
while blurred pixel get small one, the dual-channel PCNN can reducing Z will shrink this difference. Typically, when Z = 0, bA = bB.
achieve the purpose of multi-focus image fusion. Hence, adjusting Z will change the trend of bA and bB and their
Therefore, the core of this method is how to make sure that the difference. Because weighting coefficients play a crucial role in
change of the weighting coefficients depends on the clarity of the multi-focus image fusion, constant Z is usually set via several
input stimuli. Here a new method is employed to implement the experiments to meet practical demands.
transformation from the importance of input stimuli to the

3.3. Implementation of fusion algorithm


1

According to the above statements, we introduce the implemen-


0.8 tation of our proposed multi-focus image fusion algorithm based on
η = 0.05 βA the dual-channel PCNN in this section. The dual-channel PCNN used
0.6 in our experiments is a single layer two-dimensional array of
β ij

laterally linked neurons and all neurons are identical. The number of
0.4
η = 0.01
neurons in the network is equal to the number of pixels in the input
βB

0.2
Table 1
Parameter setting in the experiment.
0
-300 -200 -100 0 100 200 300
Parameters s aT VT r Z
D (i, j )
Their values 1 0.12 1000 14 0.01
Fig. 3. The relation between constant Z and weighting coefficients.

Source IA Focus MA Coefficient βA


image A Measure transformation

IA SA
Dual-channel Fused
PCNN image
IB SB

Source Focus Coefficient


image B IB Measure MB transformation βB

Fig. 4. Procedure of multi-image fusion in the paper.


ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2007

Table 2 image. In terms of position there exists a one-to-one correspondence


Parameters of Huang’s method. between the image pixels (IA(i, j) and IB(i, j)) and the neuron
(N(i, j)). In other words, the external stimuli of N(i, j) are IA(i, j) and
Parameters aL aT VL VT r
IB(i, j).
Their values 1.0 5.0 0.2 20.0 13.0 Now we describe the implementation of algorithm. The
procedure of our proposed multi-focus image fusion is shown in

Fig. 5. Test images which consist of four groups: LETTER, BADGE, TEAPOT, and PHOTO. Each group include two source images (labeled by A and B). (a) LETTER A, (b) LETTER
B, (c) BADGE A, (d) BADGE B, (e) TEAPOT A, (f) TEAPOT B, (g) PHOTO A, and (h) PHOTO B.
ARTICLE IN PRESS
2008 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

Fig. 4. Note that our algorithm is introduced on the assumption 4.1. Parameter setting
that input multi-focus images have been registered.
The proposed multi-focus image fusion method consists of the In the dual-channel PCNN, parameter settings are as follows:
following steps: large K will make computer simulation cost much time, so
synaptic weight matrices K =[1, 0.5, 1; 0.5, 0, 0.5; 1, 0.5, 1]; level
(1) Carry out focus measure with Eq. (11) for two multi-focus images factor, time constant and normalized offset parameter see Table 1.
(IA, IB). Denote measured images by MA and MB, respectively. In addition, parameters in Eqs. (12)–(14) are also in Table 1. Note
(2) Compute the weighting coefficients (bA and bB) via MA and MB that these parameters are manually set by experiments.
according to Eqs. (12)–(14). For comparison, several existing methods are used: contrast
(3) Input IA and IB, taken as two stimuli, into the dual-channel pyramid (CP), FSD pyramid (FSD), gradient pyramid (GP),
PCNN, and then start PCNN. morphological pyramid (MP), SIDWT with Harr (SIDWT), and an
(4) Fuse the multi-focus images via PCNN. existing PCNN method (Huang’s method). In order to make
(5) Obtain the fused image after finishing the process of PCNN. comparison reliable and repeatable, we use the image fusion
toolbox for MATLAB developed by Rockinger [23]. The toolbox
includes all the above methods except Huang’s method and can be
4. Simulations and results downloaded on the web (http://www.metapix.de/toolbox.htm).
Their parameters in toolbox are set by: pyramid level= 4, selection
This section consists of three parts: parameter setting, rules: highpass= select max, lowpass= average. Note, with this
performance evaluation, and practical applications. Different toolbox, our experiment can be reproduced exactly according to
parameters of algorithm can produce different effects. So this parameters provided above. If you want to obtain more details,
section begins with parameter setting. In order to testify the please read related references [6]. For the scheme and PCNN
capability of the proposed method, performance evaluation is model of Huang’s method, Huang and Jing [7,8] give a detailed
done in succession with visual effect and objective evaluation explanation in their paper. Its iteration time is 300 and other
criteria. Finally, we illustrate some applications in practical work. parameters see Table 2.

Fig. 6. Test results (LETTER). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.
ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2009

4.2. Performance evaluation 4.2.1. Objective evaluation


To evaluate objectively those methods mentioned above, we
To evaluate the performance of our proposed fusion method, choose two methods: the root mean squared error (RMSE) and the
extensive experiments with multi-focus image fusion and structural similarity (SSIM) index. Here R and F denote reference
different sensor image fusion have been performed. Here, we image and fused image respectively. RMSE is used to evaluate the
give four groups (LETTER, BADGE, TEAPOT, and PHOTO) experi- performance of the focus measures. RMSE is defined as
ments. All test images used in the experiment are shown in Fig. 5. vffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi
u
u 1 X M X N
Note, images in LETTER, BADGE and TEAPOT are synthetic while RMSE ¼ t ½Fði; jÞRði; jÞ2 : ð15Þ
images in PHOTO are real with synthetic blur. The content of M  Ni¼1j¼1
LETTER is simple; there is more information in BADGE and
Usually, less RMSE signifies better performance for the
TEAPOT; PHOTO is the real picture acquired by camera. Hence, the
algorithm of fusion. The results of RMSE assessment are shown
content of image becomes more and more complicated from
in Figs. 7, 10, 13 and 16.
LETTER to PHOTO.
The structural similarity (SSIM) index is proposed by Wang
Here we give an explanation for figures: images in Figs. 6, 9, 12
et al [26]. Because it is a better approach to image quality
and 15 are experimental results obtained by several methods. In
measure, we use it to assess the performance of different
each figure, there are nine images: one reference image and
methods, too. The mathematical expression of SSIM index is
eight images fused by eight corresponding algorithms. Figs. 7, 8,
10, 11, 13, 14, 16 and 17 show the performance of different ð2mR mF C1 Þð2sRF þ C2 Þ
SSIM ¼ ; ð16Þ
algorithms. ðm2R þ m2F þ C1 Þðs2R þ s2F þ C2 Þ

2.5

1.5

0.5

0
Our Huang's
CP FSD GP MP SIDWT
method method
RMSE 0.0216 0.5914 1.6782 2.8283 2.8288 0.6319 1.2887

Fig. 7. Objective evaluation of LETTER (RMSE).

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0
Huang's
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9999 0.9999 0.9308 0.8291 0.829 0.9372 0.9863

Fig. 8. Objective evaluation of LETTER (SSIM index).


ARTICLE IN PRESS
2010 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

where C1 =(K1L)2, C2 = (K2L)2. Where L is the dynamic range of the Larger value shows both inputs more similar. Experimental
pixel values (255 for 8-bit grayscale images); K1, K2 51 are small results of SSIM index are shown in Figs. 8, 11, 14 and 17.
constants. More descriptions are found in reference. Note that As for objective evaluation, our proposed method is not inferior
SSIM index MATLAB program is downloaded on the website to most methods from Figs. 7, 8, 10, 11, 13 and 14. According to the
(http://www.ece.uwaterloo.ca/  z70wang/research/ssim/). The evaluation rule of RMSE and SSIM index, the better method of multi-
values of SSIM index in the paper are computed with default focus image fusion should have smaller RMSE and larger SSIM index.
parameters. SSIM index describes the similarity of two inputs. In fact, experimental results demonstrate that our method has the

Fig. 9. Test results (BADGE). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.

0
Huang's
Our method CP FSD GP MP SIDWT
method
RMSE 0.5792 1.5996 2.3938 4.6158 4.5491 6.4707 2.7038

Fig. 10. Objective evaluation of BADGE (RMSE).


ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2011

smallest RMSE (in Figs. 7, 10, 13 and 16) and the largest SSIM index 4.2.2. Subjective evaluation
(in Figs. 8, 11 and 17) among several algorithms. However, Fig. 14 As we know, performance evaluation usually includes two
shows SIDWT have the largest SSIM index. At the same time, the sides: subjective (or qualitative) evaluation and objective (or
suggested method, CP, SIDWT, and Huang’s methods have similar quantitative) evaluation. For multi-focus image fusion subjective
performances in Figs 13, 14, 16 and 17. Hence it is essential to carry evaluation means the evaluation of visual effect. Here we use
on subjective evaluation in this case. visual evaluation to evaluate every method.

0.99

0.98

0.97

0.96

0.95

0.94
Huang’s
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9996 0.9981 0.9934 0.9818 0.9821 0.9625 0.9953

Fig. 11. Objective evaluation of BADGE (SSIM index).

Fig. 12. Test results (TEAPOT). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.
ARTICLE IN PRESS
2012 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

From Figs. 6, 9, 12, and 15, it is clear that these two images (3) Obtain error images. E0= XOR(R0,Bs); E1= XOR(R1,Bs); E2=
fused by FSD method and GP method have lower lightness than XOR(R2,Bs);E3= XOR(R3,Bs). Here XOR indicates the exclusive
the others. It shows FSD method and GP method make the fused OR operation. The results see Fig. 18.
images have low contrast. This is not what we want. MP method
has a good contrast but the process of edges in images is bad,
For TEAPOT:
typically such as Figs. 6f, 9f, and 12f.
Although the remaining four methods (the suggested method,
CP, SIDWT, and Huang’s methods) have the similar visual effect, (1) Let R0, R1, R2, and R3 denote the fused image of the suggested
the difference among them still exists. In fact, our method has the method, CP, SIDWT, and Huang’s methods, respectively.
better performance. Here we use one method to make this (2) Make the difference between Rs and R0, R1, R2, R3, respectively.
performance easily seen. We take LETTER (in Fig. 6) and TEAPOT Namely, D0=jRs R0j; D1=jRsR1j; D2=jRs R2j; D3=
for example; our schemes of magnifying the difference among jRs R3j. Here, Rs denotes reference image, and || indicates
these four methods are as follow: computing the absolute value of the difference.
For LETTER: (3) Obtain error images. E0 =D0 40; E1= D140; E2 =D2 40;
E3= D340. The results see Fig. 19.

(1) Let R0, R1, R2, and R3 denote the fused image of the suggested For Figs. 18 and 19, all the images are binary images and light
method, CP, SIDWT, and Huang’s methods, respectively. point means error. The error image of our method in Fig. 18 is
Rs denotes reference image. completely dark, which shows the fused image using our method
(2) Get the binary images. B0= R040; B1 =R140; B2 = R24 0;B3 is more similar to reference image than others. Obviously, CP
= R340; Bs =Rs 40. method and Huang’s method have very bad results in Fig. 19.

0
Huang's
Our method CP FSD GP MP SIDWT
method
RMSE 3.5905 4.5103 3.6549 5.1081 5.12 6.0499 3.7865

Fig. 13. Objective evaluation of TEAPOT (RMSE).

0.99

0.98

0.97

0.96

0.95

0.94

0.93

0.92
Huang's
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9839 0.9884 0.9886 0.9608 0.9619 0.9456 0.9903

Fig. 14. Objective evaluation of TEAPOT (SSIM index).


ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2013

Comparing Fig. 19(1) with Fig. 19(4), you will find that the big By the way, we would like to compare our method with
teapot have the same light while two little teapots in Fig. 19(1) Huang’s method from the references [7,8]; after all, they have
are clearly darker than the ones in Fig. 19(4). It is indicated that similar performance in both objective and subjective standards.
the proposed method is better than SIDWT method. Hence, our Compared with Huang’s method, our method at least has
method is better than others. two advantages. One is that our method has less complexity

Fig. 15. Test results (PHOTO). (a) Our method, (b) CP method, (c) Huang’s method, (d) FSD method, (e) GP method, (f) MP method, (g) SIDWT method, and (h) reference
image.

0
Huang's
Our method CP FSD GP MP SIDWT
method
RMSE 2.3865 2.3951 2.402 6.0212 6.0169 4.0758 2.4105

Fig. 16. Objective evaluation of PHOTO (RMSE).


ARTICLE IN PRESS
2014 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

0.99

0.98

0.97

0.96

0.95

0.94

0.93
Huang's
Our method CP FSD GP MP SIDWT
method
SSIM index 0.9954 0.9934 0.9905 0.9876 0.9877 0.9583 0.9936

Fig. 17. Objective evaluation of PHOTO (SSIM index).

Fig. 18. Error test of LETTER. (a) Our method, (b) CP method, (c) Huang’s method, and (d) SIDWT.

Fig. 19. Error test of TEAPOT. (a) Our method, (b) CP method, (c) Huang’s method, and (d) SIDWT.

because it does not need to divide the input images into many Table 3
image blocks and it also need not compare and choose the best Comparison of time cost between two methods (time unit: second).
image block. Obviously, the other is that our method has less time
Methods LETTER BADGE TEAPOT PHOTO
cost (shown in Table 3), for it has fewer operates than the latter.
Additionally, a large amount of time, spent in the process of Our method 0.3340 0.4131 0.6866 0.6469
determining the adjustable parameter b, greatly decreases the Huang’s method 5.5749 10.1060 18.5210 20.6628
latter’s efficiency. Here, what we need to explain is that data in
Table 3 are obtained in such experimental platform that the
computer is IBM graphics workstation (Intellistation Z Pro) and all
codes have been run in Matlab 7.0. are microscope images required by Motic digital microscope
with focus change (see Figs. 20 and 21). The latter two groups
are a body of photographs (see Figs. 22 and 23) which are
4.3. Practical application obtained from the internet (http://www.imgfsr.com/ifsr_ifs.html;
http://www2.mta.ac.il/  tal/Fusion/). Image A and image B are
Here we take four groups of images as example to demonstrate source images with different focuses. Image C is the fused image
our method’s applications in practice. The first two groups obtained by our proposed method.
ARTICLE IN PRESS
Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016 2015

Fig. 20. Microscope images (1). (a) image A, (b) image B, and (c) image C.

Fig. 21. Microscope images (2). (a) image A, (b) image B, and (c) image C.

Fig. 22. Example of photography (1). (a) image A, (b) image B, and (c) image C.

Fig. 23. Example of photography (2). (a) image A, (b) image B, and (c) image C.

5. Conclusion excels the existing methods in both visual effect and objective
evaluation criteria. In practical applications, it is proved that our
This paper presents a novel multi-focus image fusion algorithm method is feasible. Because the method is simple and easy to
based on the dual-channel PCNN. This method improves the implement, it is suitable to work in real-time system platform, too.
standard PCNN model and simplifies the process of image fusion
using PCNN in comparison with previous methods. Previous
methods usually employ more PCNNs or combination with other Acknowledgements
algorithms such as DWT, while the proposed method in this paper
just uses one dual-channel PCNN to implement multi-focus image We thank the associate editor and the reviewers for their
fusion. Experimental results show that our presented method helpful and constructive suggestions. The authors also thank Ying
ARTICLE IN PRESS
2016 Z. Wang et al. / Pattern Recognition 43 (2010) 2003–2016

Zhu for her support and help. This paper is jointly supported by [12] E. Krotkov, Focusing, International Journal of Computer Vision 1 (1987)
National Natural Science Foundation of China (No.60872109), 223.
[13] M. Li, W. Cai, Z. Tan, A region-based multi-sensor image fusion scheme using
Program for New Century Excellent Talents in University (NCET- pulse-coupled neural network, Pattern Recognition Letter 27 (2006)
06-0900), China Scholarship, and the Fundamental Research 1948–1956.
Funds for the Central Universities of Lanzhou University in China [14] S. Li, J.T. Kwok, Y. Wang, Multifocus image fusion using artificial neural
networks, Pattern Recognition Letter 23 (2002) 985–997.
(lzujbky-2009-129). [15] H. Li, B.S. Manjunath, S.K. Mitra, Multisensor image fusion using the
wavelet transform, Graphical Models and Image Processing 57 (1995)
References 235–245.
[16] W.Li, X.F. Zhu, A new image fusion algorithm based on wavelet packet
analysis and PCNN, in: Proceedings of the Fourth International Conference on
[1] C.H. Anderson, A filter-subtract-decimate hierarchical pyramid signal Machine Learning and Cybernetics. Guangzhou. 2005, pp.5297–5301.
analyzing and synthesizing technique. US Patent 718104. 1987. [17] G. Ligthart, F. Groen, A comparison of different autofocus algorithms, in:
[2] R.P. Broussard, S.K. Rogers, M.E. Oxley, G.L. Tarr, Physiologically motivated Proceedings of International Conference on Pattern Recognition, Munich.
image fusion for object detection using a pulse coupled neural network, IEEE 1982, pp. 597–600.
Transaction Neural Networks 10 (1999) 554–563. [18] T. Lindblad, J.M. Kinser, Image processing using pulse-coupled neural
[3] P.J. Burt, A gradient pyramid basis for pattern-selective image fusion, Society networks (second edition), Springer press, New York, 2005.
for Information Displays (SID) International Symposium Digest of Technical [19] Y.D. Ma, L. Li, Y.F. Wang, R.L. Dai, Principle of Pulse Coupled Neural Network
Papers 23 (1992) 467–470. and Its Applications, Science Press, Beijing, 2006.
[4] R. Eckhorn, H.J. Reitboeck, M. Arndt, P.W. Dicke, Feature linking via [20] Q. Miao, B. Wang, A novel adaptive multi-focus image fusion algorithm based
synchronization among distributed assemblies: Simulation of results from on PCNN and sharpness, in: Proceedings of SPIE-the International Society for
cat cortex, Neural Computation 2 (1990) 293–307. Optical Engineering, Orlando, 2005, pp. 704–712.
[5] H.A. Eltoukhy, S. Kavusi, A computationally efficient algorithm for multi-focus [21] H.S. Ranganath, G. Kuntimad, J.L. Johnson, Pulse coupled neural networks for
image reconstruction, in: Proceedings of SPIE-The International Society for image processing, in: Proceedings of Southeast conference on ‘Visualize the
Optical Engineering, Santa Clara. 2003, pp.332–341. Future’. Raleigh. 1995, pp. 37–43.
[6] R.C. Gonzalez, P. Wintz, Digital image processing, Perason Education, NJ, [22] O. Rockinger, Image sequence fusion using a shift-invariant wavelet trans-
1978. form. In: Proceedings of the IEEE International Conference on Image
[7] W. Huang, Z.L. Jing, Multi-focus image fusion using pulse coupled neural Processing. Santa Barbara. 1997, pp. 288–291.
network, Pattern Recognition Letter 28 (2007) 1123–1132. [23] O. Rockinger, 1999. Image fusion toolbox for Matlab. Technical report,
[8] W. Huang, Z.L. Jing, Evaluation of focus measures in multi-focus image fusion, Metapix. /http://www.metapix.de/toolbox.htmS.
Pattern Recognition Letter 28 (2007) 493–500. [24] A. Toet, A morphological pyramidal image decomposition, Pattern Recogni-
[9] J.L. Johnson, M.L. Padgett, PCNN models and applications, IEEE Transaction tion Letter 9 (1989) 255–261.
Neural Networks 10 (1999) 480–498. [25] A. Toet, L.J. van Ruyven, J.M. Valeton, Merging thermal and visual images by a
[10] J.L. Johnson, H.S. Ranganath, G. Kuntimad, H.J. Caulfield, Pulse coupled neural contrast pyramid, Optical Engineering 28 (1989) 789–792.
networks, Neural Networks and Pattern Recognition (1998) 1–56. [26] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image quality assessment:
[11] J.L. Johnson, D. Ritter, Observation of periodic waves in a pulse-coupled from error visibility to structural similarity, IEEE Transactions Image
neural network, Optical Letter 18 (1993) 1253–1255. Processing 13 (2004) 600–612.

About the Author—ZHAOBIN WANG received the B.A. degree in electronic information science and technology from Yantai University, Yantai, China, in 2004. He is
currently pursuing the Ph.D. degree in radio physics at Lanzhou University, Lanzhou, China. Since 2004, he has been working on image processing techniques, most in the
biological images. In particular, he has developed a special interest in biomedical image segmentation and measurement, pattern recognition and artificial neural network
especially PCNN.

About the Author—YIDE MA received his B.S. degree and M.S. degree in Electronic Engineering from University of Electronic Science and Technology of China, Chengdou
City, China. But his Ph.D. degree was received from Lanzhou University. He is now a professor in School of Information Science and Engineering, Lanzhou University. His
research focuses on image processing and embed-system’s application. He has authored and coauthored more than 100 publications in journals, books and international
conference proceedings.

About the Author—JASON GU received a B. Sc. degree in 1992 from the Department of Electrical Engineering and Information Science (Special Class for the Young Gifted
1987–1990), University of Science and Technology of China, M. Sc. degree in 1995 from Biomedical & Instrumentation Engineering, Jiaotong University (Shanghai), China
and Ph.D. degree from the Department of Electrical and Computer Engineering, University of Alberta, Edmonton, Alberta, Canada. Since September 2000, he has been with
the Department of Electrical and Computer Engineering at Dalhousie University and is presently an associate professor of Electrical Engineering. His research focuses on
Robotics, Biomedical Engineering, Intelligent Systems, etc.