Cluster-Based Feature Extraction and Data Fusion in The Wavelet

Domain

Johannes R. Sveinsson, Magnus Orn Ulfarsson and Jon Atli Benediktsson

Department of Electrical and Computer Engineering, University of Iceland,

Hjardarhagi 2-6, Reykjavik, 1S-107, Iceland
E-mail: sveinssoflhi.is, mouQhi.is, and benedikt(lhi.is

ABSTRACT not lead to a significant decrease in overall classification

accuracy as compared to the one obtained in the origi-
This paper will concentrate on linear feature extraction
nal feature space. In this paper linear feature extraction
methods for neural network classifiers. The considered
method, based on cluster-based feature extraction of the
feature extraction method is based on discrete wavelet
wavelet coefficients for neural networks classifiers are dis-
transformations (.DWTS) and cluster-based procedure, i.e.,
cussed and applied in classification of multisource remote
cluster-based feature extraction of the wavelet coefficients
sensing and geographic data. The method is an extension
of remote sensing and geographic data is considered. The
to a method proposed by Pittner and Kamarthi [3].
cluster-based feature extraction is a preprocessing routine
that computes feature-vectors to group the wavelet coeff-
icients in an unsupervised way. These feature-vectors are 2. FEATURE EXTRACTION
then used as a mask or a filter for the selection of rep-
resentative wavelet coefficients that are used to train the Here we concentrate on linear feature extraction methods
neural network classifiers. In experiments, the proposed for neural networks and then leave the neural networks
feature extraction methods performed well in neural net- with the classification task.
works classifications of multisource remote sensing and
geographic data. 2.1, Wavelets

The discrete wavelet transform (DWT) [4] provides a trans-

1. INTRODUCTION
The selection of variables is a key problem in pattern
recognition and is termed feature selection or feature ex-
traction [1]. However, few feature extraction algorithms
are available for neural networks [2]. Feature extraction
can, thus, be used to transform the input data and in
some way find the best input representation for neural net-
works. For high-dimensional data, large neural networks
(with many inputs and a large number of hidden neurons)
are often used. The training time of a large neural net-
work can be very long. Also, for high-dimensional data
the curse of dimensionality or the Hughes phenomenon
[1] may occur, Hence, it is necessary to reduce the input
dimensionality for the neural network in order to obtain
a smaller network which performs well both in terms of
training and test classification accuracies, This leads to
the importance of feature extraction for neural networks,
that is, to find the best representation of input data in
lower dimensional space where the representation does
This work was supported in part by the Research Fund of the
University of Icelancl and the Icelandic Research Council.
formation of a signal from the time domain to the scale-
frequency domain. The DWT is computed on several with
different time/scale-frequency resolutions. As each level
of the transformation is calculated, there is a decrease in
temporal resolution and a corresponding increase in scale-
frequency resolution. The full DWT for a time domain sig-
nal in ,C2(finite energy), z(t), can be represented in terms
of a shifted version of a scaling function @(t) and shifted
and dilated version of a so-called mother wavelet func-
tion @(t). The connection between the scalar and wavelet
functions with the scalar function at different scale are
given by the two scale equations
~(t) = ~ h(k)@(2t - k) and ~(t) = ~ g(k)@(2t - k)
ke.z? kE2
(1)
where h(k) and g(k) are the finite low-pass and high-pass
impulse responses for the DWT, respectively, The repre-
sentation of the DWT can be written as

0-7803-7031-7/01/$17.00
0-7803-7031-7/01/$10.00(C)
(C)2001
2001IEEE
IEEE 867

0-7803-7033-3/01/$10.00 (C) 2001 IEEE

 where wl,~ are the wavelet coefficients and Ul,k, j < jo
are the scaling coefficients. These coefficients are given
by the inner product in .C2, i.e.
Uj,k =< z(t), +J,k(t)> and Uj,~ =< z(~),+j,~(f)>. (3)
Here +J,k(t) = 2~/2#(2~t k) is a family of scalar func-
tions and +J,~(t) = 2~f2@(2~t k) family of wavelet
function and with a right choices of these mother functions
the family of them form an orthogonal basis for the signal
space. The wavelet coefficients, wj~~,are then measure of
the signal content around time 23k and scaleffrequency
2-~ $0 and the scaling coefficients, uj,~ represent the lo-
cal mean around the time 2Jk. The DWT can be imple-
mented by a tree filter banks. Each stage oft he tree struc-
ture consists then of low-pass, h(t), and high-pass filters,
g(t) each followed by a down-sampling of 2. Every time
down-sampling is performed, the signal length is reduced
by 2. The scale propagation is obtained by the output
from the low-pass branch goes through the same process
of filtering and down-sampling. Thus the DWT has a nat-
ural interpretatio:min terms of tree structure (filter banks)
in the time-scale/frequency domain. The wavelet coeffi-
cients wj,~ are the output of each of the high-pass branch
of the tree structure and the scalar coefficients, U30,~, is
the output of the low-pass branch of the last stage of the
DWT.
2.2. Cluster-Based Feature Extraction
In this section preprocessing routine that computes feature-
vectors to group the wavelet coefficients that are going to
be used to train neural network classifiers. This method
is an extension of a feature extraction method proposed
in [3].
Assume that we have 1 representative signals, xi, of
length 1 = 2n. The DWT is computed for all the 1 rep-
resentative signals, Zt. The wavelet coefficients are rep-
resented by wj,~ and U30,~where j = 1, 2, . . . . j., respec-
tively. Next the DWT coefficients are arranged into a
mat rix B = [bj,k] in the following form:
lwj,~l forj=l,.,,, joandl~k~~
bj,k = ~
{ forj = 1, ...,~oand~<k~$
luj,,~l fork= 1
bjo+l,k =: ~ for $ ~~; + (5)
{ 2
Let p(A) and 0(.A) represent the sample mean and the
standard deviation of the elements of a matrix A. R is
the operator that when applied to any matrix A, reduces
the matrix by its last row. Next calculate the matrix G
as
G = [gj,/c]=
(m:=+))
F:=lBm
-p (R (~:=1 Bin)) I).
0-7803-7031-7/01/$17.00 (C) 2001 IEEE
0-7803-7033-3/01/$10.00 (C) 2001 IEEE
Table 1: Training and Test Samples for Information
Classes in the Experiment on the Anderson River data.
class # In form.t,on Class Tra, nmg S,.. Test S,.,
1 Douglas F,, (31-40rn) 971 1250
2 Dou@as F,, (21-30rn) 551 817
3 Douglas F,r + Other SpecvAs (31-40m) 548 701
3 Douglas F,, + LodEepole P,ne 542 705
5 Helmlock + Cedar (31-40m) 517 405
1625
H 6 I Forest Clearings ~ 1260 ~
Tot al .. .. K,.. u
Then the binary matrix Gb is obtained as
Gb = [~(gj,~ T)] (7)
where @ is the step function @(x) = lforz>O and
@(z) = O for x <0 and T is some threshold to be chosen.
The 1s in the matrix Gb occur where the largest wavelet
coefficients occur in the matrix Bk and we can then use
the row vectors of Gb as a mask or a filter for the se-
lection of representative wavelet coefficients for the data
to be classified. This can be done by only select wavelet
coefficients that correspond to the 1s in the row vectors
of Gb as the inputs for the neural networks. The choice
of the threshold T is used for the reduction of the 1s in
Gb and hence the reduction of wavelet coefficients which
are used as inputs for the neural networks, i.e. feature
extraction,
3. EXAMPLE AND DISCUSSION
The data set used in experiments, the Anderson River
data set, is a multisource remote sensing and geographic
data set made available by the Canada Centre for Re-
mote Sensing (CCRS) [2,5]. Six data sources were used:
Airborne Multispectral Scanner System (11 spectral data
channels), Steep Mode Synthetic Aperture Radar (SAR)
(4 data channels), Shallow Mode SAR (4 data channels),
Elevation data (1 data channel), Slope data (1 data chan-
nel), and Aspect data (1 data channel), The six infor-
(4) mation were used, as listed in Table 1. Here, training
samples were selected uniformly, giving 10% of the total
sample size. Test samples were then selected randomly
from the rest of the labeled data.
The number of features for the data was 22. A con-
jugate gradient perception (CGP) neural network with
one hidden layer was trained on the original data with a
different number of features. The classification results are
listed in Table 2. Since the original data had 22 features it
was necessary, in order to apply the wavelet based feature
extraction method, to add zeros to the data so that the
number of features became 32. The training set is trans-
formed with DWT down to level j.. One masking vector
(6) is found for the training set and used to find the wavelet
868

Table 2: Classification Accuracies for The Original Data,
13Tveazccuracycy
Veraifrc
: on cluster-based feature extraction in the wavelet domain.
Table 3: C Iassification Accuracies for DWT.
coefficients as inputs to the neural networks. Then feature
extraction is done by changing the threshold T and thus
obtaining row vectors with different numbers of 1s. These
row vectors are then used as masks to find the wavelet co-
efficients. The masking vectors for different number of
features are also used for the test set. For the DWT, a
sixt-tab Daubechi es wavelet filter ~G was used [4]. A con-
jugate gradient perception (CGP) neural network with
one hidden layer was trained on the DWT transformed
input data with a,different number of features, The clas-
sification results for the DWT are listed in Table 3, When
the results in Table 2 are compared to the results in Ta-
ble 3, it can be seen that higher overall accuracies were
obtained when 32 DWT features were used instead of the
22 original features. The differences are significant, i.e.,
more than 6 per cent for training data and more than 3
per cent for test data. These results are very interesting
since they show the discrimination capability of the pro-
posed approach. The wavelet transformation by itself is
a linear transformation and should not improve classifi-
cation accuracies. However, the cluster based approach
introduces a technique which improves the accuracies.
From Table 3 it can be seen that the classification accu-
racies decreased when only 10 features are used instead of
the full 32 features. The training accuracies decrease by
almost 10 percent and the test accuracies by nearly 6 per
cent. On the other hand, the accuracies for 10 features
are far higher than the received results when 10 features
are used for the original data in Table 2. The proposed
approach gives more than a 10 per cent improvement in
overall training accuracy and nearly 9 per cent improve-
ment in overall test accuracies when compared to the 10
feature results in Table 2. Similar behavior is seen when 5
and 8 features am used. The accuracies for the proposed
approach are significantly higher than the accuracies ob-
tained when original data are used.
0-7803-7031-7/01/$17.00 (C) 2001 IEEE 869
0-7803-7033-3/01/$10.00 (C) 2001 IEEE
4. CONCLUSION
Feature extraction method for neural network classifiers
are proposed and applied in classification of multisource
remote sensing and geographic data. The method is based
The cluster-based method is an unsupervised preprocess-
ing routine that computes feature-vectors to group the
wavelet coefficients. Then, these feature-vectors are used
as a mask or a filter for the selection of representative
wavelet coefficients, i.e., representative feature, that are
used to train the neural network classifiers. For the appli-
cation of this approach, the data had to be zero-padded.
The method showed a great promise in efficiently extract-
ing important features for the multisource remote sensing
and geographic data, but it should be even more appro-
priate for data which have a number of features that is a
power of 2, i.e., so the data need not to be zero padded.
Also, the choice of the threshold T and hence the reduc-
tion of wavelet coefficients that are used as inputs for the
neural networks after feature extraction, should be such
that the parameters of the classifier and the feature ex-
tractor are jointly optimized. That is a topic of future
research.
Acknowledgments
This work was supported in part by the Research Fund of
the University of Iceland. The Anderson River SAR/MSS
data set was acquired, preprocessed, and loaned by the
Canada Centre for Remote Sensing, Department of En-
ergy Mines, and Resources, of the Government of Canada.
5. REFERENCES
[1] K. Fukunaga, Introduction to Statistical Pattern
Recognition, 2nd edition, Academic Press, NY, 1992.
[2] J.A. Benediktsson and J.R. Sveinsson, Feature ex-
traction for multisource data classification with arti-
ficial neural networks, Int. J. Remote Sensing, vol.
18, no. 4, pp. 727-740, 1997.
[3] S. Pittner and S.V. Kamarthi, Feature Extraction
From Wavelet Coefficients for Pattern Recognition
Tasks, IEEE Trans. of PAMI, vol. 21, pp. 83-88,
1999.
[4] I. Daubechies, Ten Lecture on Wavelets, SIAM, USA,
1992.
[5] G.M. Goodenough, Goldberg, M.P. Plunkett, and J.
Zelek, The CCRS SAR/MSS Anderson River Data
Set, IEEE Transactions on Geoscience and Remote
Sensing, vol. GE-25, pp. 360-367, 1987.