Computers and Electronics in Agriculture 106 (2014) 91101

Contents lists available at ScienceDirect

Computers and Electronics in Agriculture

journal homepage: www.elsevier.com/locate/compag

Identifying blueberry fruit of different growth stages using natural

outdoor color images
Han Li a,b, Won Suk Lee b,, Ku Wang a
a
b

Department of Electrical Engineering, China Agricultural University, Beijing 100083, China

Department of Agricultural & Biological Engineering, University of Florida, Gainesville, FL 32611, United States

a r t i c l e

i n f o

Article history:
Received 5 April 2013
Received in revised form 18 April 2014
Accepted 30 May 2014

Keywords:
Image classication
K-means
K-nearest neighbor
Nave Bayesian classier
Precision agriculture
Yield mapping

a b s t r a c t
This study was conducted to identify blueberry fruit of different growth stages using natural outdoor
images toward the development of a blueberry yield mapping system. As blueberries usually contain different maturity stages in a same branch, identication of blueberry fruit and their maturity stages from
different background is very important for yield mapping. In this study, maturity stages of the fruit were
divided into four categories: mature (m), near-mature (nm), near-young (ny) and young (y). A stepwised
algorithm, termed color component analysis based detection (CCAD) method, was developed and validated to identify blueberry fruit using outdoor color images. Firstly, a dataset was built using manually
cropped pixels from training images. Three color components, red (R), blue (B) and hue (H), were selected
using the forward feature selection algorithm (FFSA), and used to separate all fruit of four maturity stages
from background through different classiers. In this work, not only the traditional classiers such as Knearest neighbor (KNN), and nave Bayesian classication (NBC) were used, but another newly introduced
supervised K-means clustering classier (SK-means) was also developed and applied to the dataset. In
the second step, classiers were built to separate a group of mature & near-mature fruit from a group
of near-young & young fruit from all fruit pixels. Finally, classiers were developed to separate mature
fruit from near-mature fruit, and near-young fruit from young fruit. The classiers obtained from these
different steps were then applied to validation images, resulting in nal identication. Cross validation
was conducted using these different classiers and their results were compared. KNN classier yielded
the highest classication accuracy (8598%) from the validation set of the prebuilt pixel dataset collected
from the training images in all separations. An one-way ANOVA was used to compare the performance of
the three classies, which shows KNN performed signicantly better than other methods. The newly
proposed SK-means classier yielded a fairly high accuracy (90%) for the separation of mature and
near-mature fruit. The newly developed CCAD method for blueberry was proved to be efcient for
identifying blueberry fruit of different growth stages using natural outdoor color images toward the
development of a blueberry yield mapping system.
2014 Elsevier B.V. All rights reserved.

1. Introduction
Highbush blueberry is a good source of ber, and contains antioxidants, which makes it an excellent choice for fresh market fruit
production all over the world (U.S. Highbush Blueberry Council,
2012). In Florida, USA, southern highbush blueberry acreage and
production have increased rapidly. Compared with the 9 million
pounds in 2008, 19 million pounds of blueberry has been harvested
in 2012 (Brazelton, 2013). Since fresh Florida blueberries are
mostly hand-harvested, the primary production cost is harvesting

Corresponding author. Tel.: +1 352 392 1864; fax: +1 352 392 4092.
E-mail address: wslee@u.edu (W.S. Lee).
http://dx.doi.org/10.1016/j.compag.2014.05.015
0168-1699/ 2014 Elsevier B.V. All rights reserved.

labor which can exceed 50% of the total picking, grading, and packing costs. The production window of blueberry in Florida is from
about April 1 until May 15, which is relatively short. The prices
usually drop fast as berries enter the market from northern regions
(Yang et al., 2012). Since blueberry fruit in a same branch usually
do not ripen at the same time before harvesting season, it is important for farmers to estimate the quantity of blueberry fruit on the
bushes at different stages of their growth, so that they can make
proper arrangement for harvesting labor and its distribution to
specic locations in their elds. Also early yield estimation can
be used to provide feedback on how crops respond to certain soil
and crop management practices and to determine recommendation rates for many crop production inputs (Arslan and Colvin,
2002).

92

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

Automated computer vision system is one of the most commonly

used methods for yield estimation of various crops, such as apple
and citrus. Several studies have focused on machine vision-based,
non-destructive fruit yield estimation and mapping. For apple
detection, Alchanatis et al. (2007) developed a method for automatically detecting apples using hyperspectral imaging, and the overall
correct detection rate was 87% with an error rate of 15%.
Aggelopoulou et al. (2011) suggested that apple owering distribution maps could be used for yield mapping, and modeled correlation
between ower density as determined from image analysis and fruit
yield. Their results indicated that potential yield could be predicted
early in the season from owering distribution maps and could be
used for orchard management during the growing season. Zhou
et al. (2012) proposed an apple recognition algorithm with color differences such as red minus blue (RB), and green minus red (GR).
The coefcients of determination (R2) between apples detected by
the fruit counting algorithm and actual harvested yield ranged from
0.57 for young fruit to 0.70 for fruits in a fruit ripening period.
For citrus fruit detection, Annamalai et al. (2004) developed a
color vision system for estimating citrus yield. Chinchuluun and
Lee (2006) developed a fruit identication method using the
Watershed transform to separate and split touching fruit for accurate fruit counts and yield estimation. Kane and Lee (2007)
equipped a monochromatic near-infrared camera with interchangeable optical band pass lters to capture images of citrus
fruit for in-eld immature green citrus identication, and their
results revealed the potential of a multispectral imaging system
to be used in an automated early season yield mapping system.
Okamoto and Lee (2009) developed a hyperspectral image processing method to detect immature green citrus fruit in individual
trees, and the fruit detection tests revealed that 8089% of the fruit
in the foreground of the validation set were correctly identied,
though many occluded or highly contrasted fruit were identied
incorrectly. More recently, Kurtulmus et al. (2011) developed a
new eigenfruit approach to detect immature green citrus fruit
from color images acquired under natural outdoor conditions and
reported a detection accuracy of 75%. Also Bansal et al. (2013)
implemented a novel technique utilizing the fast Fourier transform
leakage values for detecting immature green citrus, and obtained
an accuracy of 82% for correct fruit identication.
For blueberry fruit detection, Zaman et al. (2008) and Zaman
et al. (2010) reported that there was a real potential to estimate
fruit yield and detect wild blueberry plants and bare spots from
digital photography within the elds in Canada, because there
was signicant correlation between percentage of blue pixels and
actual fruit yield in eld 1 (R2 = 0.90) and eld 2 (R2 = 0.97).
Swain et al. (2010) presented the development of an automated
yield mapping system for real-time fruit yield estimation for wild
blueberry. The experiment showed a signicant correlation
between percentage of blue pixels representing ripe fruit in the
eld of view and actual fruit yield. Chang et al. (2012) developed
an automated yield monitoring system consisting of two color
cameras, a real time kinematic-GPS receiver, and custom software.
Highly signicant relationship between the percentage of blue pixels and actual fruit yield in two elds were shown through linear
regression results. Farooque et al. (2013) evaluated the performance of an integrated automated system in two chosen elds.
Plant height, fruit yield, slope and elevation were measured in
real-time simultaneously with harvesting. The results suggested
that this system was reliable for mapping such information in
real-time. Wild blueberry plants are very close to the ground,
and so it is easier to take a picture and estimate yield. However,
southern highbush blueberry plants in Florida, which was studied
in this paper, are taller and bear fruit in different branches.
Therefore, new methods need to be developed to estimate fruit
yield of southern highbush blueberry plants.

The overall objective of this study was to identify blueberry

fruit of different growth stages using red, green, and blue (RGB)
color images acquired outdoors using a regular digital camera,
which would be a rst step for developing a blueberry yield mapping system. Specic objectives were to:
1. develop an efcient method to identify blueberry fruit of four
different growth stages in color images acquired under natural
outdoor conditions,
2. compare the results obtained from different classiers and
choose one classier with best performance to be used in blueberry fruit identication.
2. Materials and methods
2.1. Image acquisition
A total of 46 images were acquired from a commercial blueberry farm (Straughn Farm) in Waldo, Florida, USA from April
12th to April 22nd, 2011, for the variety of Jewel, which was one
of the representative varieties in Florida. A digital SLR camera
(EOS Rebel T2i, Canon Inc., Japan) with an 1855 mm lens was
used, and the camera was set as a full auto mode. The images were
saved in JPEG/Exif format with 3648  2736 pixels, corresponding
to an approximately 13 cm  10 cm actual scene, and the compression rate was 10:1. Image acquisition was performed between
11:00 AM and 2:00 PM local time. The illumination was affected
by the sun light change, wind, and cloud in the sky when capturing
the images.
2.2. Building data library
Among the 46 images, 23 images were randomly selected as
training images, and the other 23 were used as validation images.
A pixel data library including ten classes of different objects mainly
appearing in the images was built by manually cropping from the
23 training images using an image editing software (Corel Painter
Photo Essentials 4, Ottawa, Ontario, Canada). The 10 different classes mostly found in the images are: mature (m), near-mature (nm),
near-young (ny), young (y), leaf (leaf), burgundy branch (bran_b),
green branch (bran_g), older branch (bran_older), other background
objects (bg), and sky (sky). An example image of these classes is
shown in Fig. 1, and a brief description is listed in Table 1. As the
fruit do not ripen at the same time, maturity of the fruit was
divided into four stages: m, nm, ny and y for yield prediction.

bran_g

bg

bran_b
leaf
bran_older
nm

ny

Fig. 1. Example image of blueberry fruit with different growth stages and other
objects.

93

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

Table 1
Brief description of 10 different classes used in this study. 10,000 pixels were randomly chosen from each class.
Class abbreviation

Class description

Color description

Total number of pixels

m
nm
ny
y
leaf
bran_b
bran_g
bran_older
bg
sky

Mature fruit
Near-mature fruit
Near-young fruit
Young fruit
Leaf
Regular branch
Newly developed green branch
Older branch
Other background objects including soil and leaves on the ground
Sky

Deep blue
Reddish purple
Yellowish green
Green
Green
Burgundy
Green
Light brown
Mostly brown
Blue or white

5,712,768
2,334,720
5,326,080
6,555,456
5,647,104
846,336
2,469,696
758,784
5,544,960
5,223,936

After the pixel data library was built for the 10 classes in the
images, ten thousand pixels were randomly chosen for each class
from the training images, and they were named as the whole dataset in this study. A half of the whole dataset (10,000/2 = 5000 pixels from each class) was randomly chosen to be used as a training
dataset, and the other half was used as a validation dataset. The
training dataset was used for analyzing distribution of different
color components of the 10 classes. The other half of the pixels
was used to build a validation dataset. These two datasets were
used for color component selection for training and testing. A fruit
training dataset was created by combining the training datasets of
m, nm, ny and y classes, resulting in a total of 20,000 pixels (4 classes  5000 pixels). Similarly, a background training dataset was
created by combining objects from the other six classes, resulting
in 30,000 pixels (6 classes  5000 pixels). The whole dataset was
used for a cross validation for choosing classiers for different
separations.
2.3. Color component analysis based detection (CCAD) method for
blueberry
The choice of color spaces and color components to classify RGB
images is widely discussed (Gonzalez and Woods, 2002; Szeliski,
2010). Red, green, blue (RGB), hue, saturation, intensity (HSI), luminance, blue-difference chroma, red-difference chroma (YCbCr) and
luminance, in-phase, quadrature-phase (YIQ) are four commonly
used color spaces. Also the normalized color based on maximum
value and 2  greenredblue (2 grb, or excess green index
(EGI)) are widely used for separating green vegetation from the
background (Woebbecke et al., 1995; Sgaard and Olsen, 2003).
In this study, since the images were acquired outdoors, varying
illumination conditions would affect the processing results significantly. Thus, I in HSI, Y in YCbCr, and Y in YIQ color spaces were
not used for fruit detection, which represented intensity or luminance, and directly reected varying illumination of the experimental environment. Therefore, a total of 10 dimensional dataset
was built using the other color components of R, G, B, H, S, I, Q,
Cb, Cr and EGI as a training dataset. To explore the possibility of
utilizing smaller number of color components, the forward feature
selection algorithm (FFSA) as described by Whitney (1971) and
Kumar et al. (2001) was chosen to reduce the number of color components and also to determine the best subset of color components
that would yield the best classication results. FFSA is explained in
a later section.
This section describes an overview of the proposed CCAD
method, and more details are explained in subsequent sections.
The whole algorithm is accomplished in eight steps as shown in
the algorithm owchart in Fig. 2. In the rst step, six color components were selected using FFSA according to their performance
among the 10 color components. It was found that the six components chosen by FFSA and the rst three components (R, B, and H)
performed almost equally well. Thus, only these three color

Fig. 2. Algorithm owchart of the newly proposed method color component

analysis based detection (CCAD) method.

components were used to build subsequent classiers. Next step

was to build a classier to separate all fruit classes from the other
classes (Separation 1). After that, another classier (Separation 2)
was built to separate a group of mature and near-mature fruit
classes from a group of near-young & young fruit classes in the
training set. To further remove leaves from near-young & young
fruit, a leaf_nyy classier was built in Separation 3 step. Then,
another two classiers were developed: one to separate mature
fruit from near-mature fruit, and another to separate near-young
fruit from young fruit using the training dataset. By applying all
these classiers sequentially to the corresponding validation dataset, fruit of four different growth stages were separated from the
complicated background, and individual fruit stages on the validation images were also detected.
To decide which classier yielded the best accuracy, cross validations were conducted by dividing the whole dataset into ve
equal parts: one was used to develop classiers, and the rest was
used for validation. And the classiers were also applied to the
validation images. In this study, all the methods were realized in
Matlab (Ver. R2010a, Mathworks, Natick, MA, USA).
One diagram describing how datasets were used in the paper is
shown in Fig. 3, to help readers better understand how the data
was used in corresponding steps in the proposed CCAD method.
The detailed information will be introduced in the following
sections of the paper.

94

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

Fig. 3. The diagram describing how datasets were used in the paper.

2.4. Forward feature selection algorithm (FFSA)

Whitney (1971) proposed a direct method of nonparametric
measurement selection, and it was named forward feature selection algorithm (FFSA) by Kumar et al. (2001). It is used to determine the best subset of features that would yield the best results
for classication using a suboptimum measurement subset procedure which works as following. For the chosen classication algorithm, the best single feature with the highest accuracy is selected
rst, and then the best pair of features is selected where the pair
includes the best single selected feature. A new feature is added
to the previously selected feature(s) so that the classication accuracy increases maximally when it is added. This procedure is
repeated with the rest of the features until the classication accuracy does not improve considerably or all features are used up.
Zero percent increase was considered the threshold in the study.
This method was chosen in this study to select best set of color
components with best performance. FFSA was repeated 100 times
to decide how many color components should be chosen and
which combination of color components should be chosen.

and then posterior value was calculated through the Bayesian rule.
Based on the generated model, the class label of each sample in the
validation dataset was decided, through the posterior probability
calculated for each class. The label of the class with maximum posterior probability was assigned to the corresponding sample.
2.5.3. Supervised K-means clustering classier using weighted
Euclidian distance (SK-means)
Data clustering nds similarities in data, and then it partitions a
dataset into several groups so that the similarity within a group is
bigger than that of other groups. Different data clustering methods,
such as K-means clustering, fuzzy C-means clustering, and mountain clustering were applied to a variety of areas, including image
and speech data compression (Hammouda and Karray, 2000).
However, it has not traditionally been applied to classication
problems for the reasons that it usually does not assign a meaning
or class label to the groups of data it clustered. In this study, a
novel method, supervised K-means clustering using weighted
Euclidean distance (SK-means), was proposed and applied to the
blueberry fruit classication problem. The algorithm was accomplished in the following four steps:

2.5. Pattern recognition algorithms

2.5.1. K-nearest neighbor (KNN) classier
K-nearest neighbor (KNN) classier has been very successful in
many practical applications. Given an unknown feature vector x
and a distance, the KNN rule classies x by assigning it the label
most frequently represented among the k nearest neighbors and
taking a vote. When k = 1, the feature vector is assigned to the class
of its nearest neighbor (Duda et al., 2001). In this study, one was
used for k, and Euclidean distance was used for the measurement
of distance between the samples.
2.5.2. Nave Bayesian classier (NBC)
When Nave Bayesian classier (NBC) (Duda et al., 2001) is conducted, a Bayesian probability distribution model was developed
for all the classes from the training dataset. The class likelihoods
and priors were estimated from the training dataset in this model,

 (a) Calculation of a squared correlation coefcient (Johnson and

Wichern, 1998; Ye and Li, 2002): In a training dataset, a squared
correlation coefcient is calculated between each predictor variable Xi and the target variable Y (1, 2), where 1 and 2 are the
class labels, to determine how relevant each predictor variable
is for classifying the target class in the target variable, and use
it in the calculation of weighted Euclidean distance.
The squared correlation coefcient r 2iY between the predictor
variable Xi(i e (1, . . ., p)) and the target variable Y is computed as
shown in Eq. (1).

S2iY N

12

B
C
r2iY N @qqA
2
2
Sii N SYY N

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

where

95

2.6. Cross validation

S2ii n

n2 2
1
2
S n  1 X i n  X i n  1
n  1 ii
n

S2iY n

n2 2
1
S n  1 X i n  X i n  1X Y n  X Y n  1
n  1 iY
n

S2YY n

n2 2
1
2
S n  1 X Y n  X Y n  1
n  1 YY
n

X i n

n  1X i n  1 X i n
n

X Y n

n  1X Y n  1 X Y n
n

3. Results

for n 1; . . . ; N:
S2ii n is covariance of X. S2iY n is covariance of X and Y, and S2YY n
is covariance of Y. X i n and X Y n are averages of X and Y.
 (b) Weighted K-means clustering: Using weighted Euclidean distance, k clusters, denoted as Li (i = 1, . . ., k), and the k centroid
coordinates of these clusters, which are denoted as (C1, C2,
. . ., Ck) are obtained through K-means clustering. The weighted
Euclidean distance between the measured data point and the
centroid of the cluster is the following, where p is the dimension
of X.

dX; L

q
Xp
X i  Li 2 r 2iY
i1

To examine efciency of the classication results of the CCAD

method using different classiers, a cross validation was performed on the whole dataset, which were the ten thousand pixels
randomly chosen for each class from the training images, in each
separation. The multiple rounds of cross-validation could reduce
variability caused when only training and validation tests were
used. The total dataset was divided into ve equal parts. In the rst
iteration, the rst part was used for training, and the other four
parts were used for validation. In the next iteration, the second
part was used for training, and the remaining four parts were used
for validation. Similarly each of the ve parts was used for cross
validation. Fig. 4 shows the cross validation diagram for fruit and
background, in which 10 k is an abbreviation for ten thousand, k
is an abbreviation of 1000, T is short for the training set, and V
is short for the validation set.

3.1. FFSA
The results of FFSA using training dataset are shown in Fig. 5.
The number of color features chosen by FFSA during the 100 iterations are shown in Fig. 5a, indicating that six components were
chosen most frequently. Among the 10 color components (R, G,
B, H, S, I, Q, Cb, Cr and EGI), six color components performed best
were R, B, H, Q, Cb and EGI as listed in Fig. 5b, which shows the
number of times chosen for separation of background and fruit.
If the information contained in a smaller subset of variables is
similar with that in a bigger subset of variables, then we could
use a smaller subset instead of a bigger one. It was found that
the rst six color components (R, B, H, Q, Cb and EGI) chosen by

 (c) Classication of clusters: By counting the number of the members Niy (i = 1,. . ., k, y = 1,2) in the clusters belonging to different
groups, the clusters are classied into different classes. In this
case, if Ni1 > Ni2, then, cluster Li is assigned as class 1.
 (d) Testing using an unknown pixel: First, the nearest cluster
number of the testing pixel is found by comparing the weighted
Euclidean distance between the centroids Ci and the unknown
pixel. Then, the same class label of the corresponding cluster
is assigned to this pixel. Thus, the new unknown pixels classication is accomplished.
2.5.4. Classier construction procedure and post processing
For the Separations 1 through 5 as described in Fig. 2, the same
procedure was followed to build ve classiers for each method of
KNN, NBC and SK-means. Let us take a KNN classier for example.
Firstly, using the three color components (R, B, and H) of the training dataset, one KNN classier was built to separate fruit from
background, which was named Separation 1. Then similarly,
another four KNN classiers were built to carry out the Separations
2 through 5. For Separation 3 among these separations, a leaf_nyy
classier was built, using the training dataset of leaf and nearyoung & young (nyy) classes. This classier was used to lter out
more leaves from near-young fruit, since there were some leftover
leaves for nyy fruit resulted from Separation 2.
All these classiers were applied to the validation images as
described in Fig. 2. To lter out small noise after classication, necessary post processing operations, median lter and morphological operations (open and close), were applied sequentially to the
images obtained from the classiers. Five pixels were chosen as a
window size for these three operations based on the results of preliminary experiment. Example images are shown in the result
section.

Fig. 4. Cross validation diagram of one iteration for fruit and background.

96

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

100

25

80

Frequency

Frequency

20
15
10

40
20

5
0

60

9 10

Number of chosen feature color components

Cb Cr EGI

9 10

Color components

(a)

(b)

Fig. 5. Results of FFSA: (a) frequency of the number of color components chosen using FFSA during 100 iterations, and (b) frequency of each feature chosen using FFSA.

Table 2
Preliminary classication results for blueberry fruit from background using different classiers. 20,000 pixels were used for fruit, and 30,000 pixels were used for background.
Classication methods

NBC
KNN

Six color components data

Fruit
Detected pixels
15,038
16,018

Three color components data

Percent (%)

Background
Detected pixels

75.1
80.9

23,440
24,959

FFSA had quite similar classication results compared with the

results using the rst three color components (R, B, and H) based
on a preliminary test, as shown in Table 2. Therefore, these three
color components (R, B, and H) of the dataset were used to achieve
identication of blueberry of different growth status.
Fig. 6 shows a composite image using the R, B, and H color components using the same image in Fig. 1. B was used instead of G,
and H was used instead of B component in the RGB image. In this
gure, young fruit under the shadow exhibit similar color to green
leaves, and could be easily confused with them. However, in the
RBH image, color of those young fruit under the shadow is very different than the color of the green leaves.
Histograms of the manually cropped data library for the chosen
color components are shown in Fig. 7. In Fig. 7a, most fruit have
higher R values than the background. However, there are still some
branches and background (bran_r, bran_g, and bg) pixels overlapped with the young fruit. In Fig. 7b, young fruit has a wide
spread across the whole intensity values, which indicates that
the separation of the young fruit from the background will be very
difcult using only this color component. Fig. 7c, where only a
small part of leaves and young fruit are overlapped together, will
help separate the fruit from the background, but more attention
should be paid to the separation of young fruit and leaves. Mature
fruit is well separated from other objects.
3.2. Cross validation
The results of the cross validation for the validation datasets
corresponding to the 4th box of Fig. 4 are shown in Tables 36.
The numbers of correctly classied pixels and their proportion
(%) relative to the total number of pixels are summarized. Table 3
shows that KNN yielded the highest accuracies for fruit classication in all of the ve iterations, which were over 80%. SK-means
had a better accuracy in one iteration for fruit classication, and
similar accuracies for four iterations for background classications
compared with NBC.
In the cross validation for the classication of mature & nearmature fruit (mnm) from near-young & young fruit (nyy) using
different classiers as shown in Table 4, KNN also performed best

Percent (%)

Fruit
Detected pixels

Percent (%)

Background
Detected pixels

Percent (%)

78.1
83.2

14,732
15,780

73.7
78.9

22,053
25,660

73.5
85.5

bran_g

bg

bran_r
leaf
bran_older
nm

ny

Fig. 6. Composite image using R, B, and H color components of Fig. 1.

among these three methods based on the average accuracy. 16,000

pixels were used for mnm and 16,000 pixels were used for nyy. An
accuracy of near 90% was achieved for both mnm and nyy using
KNN. SK-means yielded slightly higher accuracies in all ve iterations for mnm classication, and lower accuracies in all ve iterations for nyy classication, compared with NBC.
Table 5 also shows that KNN performed best among these three
methods with an average accuracy of above 95% for both m and
nm. SK-means yielded slightly lower accuracies in all ve iterations
for m classication, and similar accuracies in all ve iterations for
nm classication, compared with NBC.
Table 6 shows that KNN also performed best among these three
methods, the average accuracy of which was above 90% for both ny
and y, and SK-means had lower accuracies in all ve iterations for
ny classication, and higher accuracies in all ve iterations for y
classication compared with NBC.
A one-way ANOVA was used to compare the means of the cross
validation results of the three classication methods. A Tukeys
multiple comparison was used to test individual differences in
classication method means. As shown in Table 7, the very small
p-values indicate that differences between column means are

97

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

80000

Number of pixels

70000
60000

m
nm
ny
y
leaf
bran-r
bran-g
bran-older
bg

50000
40000
30000
20000
10000
0
1

51

101

151

201

251

Red

(a)

Number of pixels

140000
120000
m
nm
ny
y
leaf
bran-r
bran-g
bran-older
bg

100000
80000
60000
40000
20000
0
1

51

101

151

201

251

4. Discussion

Blue

(b)

Number of pixels

800000
700000
m
nm
ny
y
leaf
bran-b
bran-g
bran-older
bg

600000
500000
400000
300000
200000
100000
0
1

51

101

151

201

separated from background, classication results of mature &

near-mature fruit, near-young & young fruit with some leaves,
near-young & young fruit after ltering out the leaves, mature
fruit only and near-mature fruit only, respectively.
Fig. 9 shows results after applying the same method to a validation image. Some leaves in Figs. 8d and 9d were classied as young
or near-young fruit due to similar luminance as previously shown
in the histogram. However, after applying the leaf_nyy classier,
most of the leaves were removed as shown in Figs. 8e and 9e.
Although the image classication results from all three classiers (KNN, NBC and SK-Means) have not been listed in the gure,
the results showed that the fruit separation results from the
images using these three methods were consistent with the pixel
classication, as KNN has given the best results. The proposed
CCAD method is proved to be useful and efcient for the identication of blueberry fruit of different growth stages.

251

Hue
(c)
Fig. 7. Histograms of different objects in the manually cropped data library: (a) red,
(b) blue, and (c) hue.

highly signicant. The mean value indicates that KNN performed

best among the three methods, and was signicantly better for
all cases except for mature vs. near-mature fruit, where it was statistically different from SK-means.
Examples of fruit detection in the training and validation
images are shown in Figs. 8 and 9 after applying the KNN classiers
previously developed for cross validation. Fig. 8a shows a typical
training image, which is very complicated, because it contains m,
nm, ny, y, leaf and almost all kinds of other background objects.
Figs. 8bg shows the classication results after applying KNN: fruit

Yield information is very important for blueberry growers to

better plan harvesting and reduce labor cost. A camera based,
non-destructive fruit yield estimation and mapping system has
great potential for providing this information. The goal of this
study was not only to detect the fruit from the complicated background using natural outdoor color images, but also to identify
those fruit of different growth stages, which can help the farmers
know more about yield variability of the eld and make better
management decisions. Color, texture, and edge are usually the
most popular features for object detection or classication. From
Fig. 1, these three features of different fruit could be observed:
the texture and edge for the blueberry fruit were quite similar,
and instead color difference was the most obvious feature for the
separation of fruit of different growth stages. Therefore, color difference was chosen as the standard feature for developing detection algorithm.
Since the images used in this study were acquired outdoors, it
was important to utilize all available color information in different
color spaces. The CCAD was based on the analysis of the ten color
components, R, G, B, H, S, I, Q, Cb, Cr and EGI, in order to nd out
which combination of different color components performed best
for this specic problem, and this was achieved by FFSA. FFSA identied most frequently selected six color components (R, B, H, Q, Cb,
and EGI). It was found that the classication accuracy was similar
for the separation of blueberry fruit and background using the rst
three R, B, and H components. This was probably because Q, Cb and
EGI were converted from R, G, and B components, and so they were
highly correlated. Except for the separation of background and all
fruits, the separations of other fruits were relatively easier based
on the fact that the color differences of the fruits were much more
distinct. The same procedures for selecting the color component
for the separation of mature and near-mature, and young and
near-young fruits were tried. The frequencies of how many times
the component would be chosen indicated RBH components also

Table 3
Cross validation accuracies for the classication of fruit from background using different classiers. 32,000 pixels were used for fruit, and 48,000 pixels were used for background.
Cross validation number

1
2
3
4
5
Average accuracy

NBC

KNN

SK-means

Fruit
Detected pixels (%)

Background
Detected pixels (%)

Fruit
Detected pixels (%)

Background
Detected pixels (%)

Fruit
Detected pixels (%)

Background
Detected pixels (%)

18,156
17,760
17,642
17,461
17,974
17,709

37,336
37,570
37,562
37,581
37,523
37,559

26,522
26,543
26,561
26,338
26,590
26,508

41,041
40,917
41,150
40,886
40,706
40,915

15,226
15,278
23,130
15,180
15,156
17,186

38,064
38,016
33,292
37,967
38,077
36,838

(57)
(56)
(55)
(55)
(56)
(56)

(78)
(78)
(78)
(78)
(78)
(78)

(83)
(83)
(83)
(82)
(83)
(83)

(86)
(85)
(86)
(85)
(85)
(85)

(48)
(48)
(72)
(47)
(47)
(52)

(79)
(79)
(69)
(79)
(79)
(77)

98

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

Table 4
Cross validation accuracies for the classication of mature & near-mature fruit (mnm) from near-young & young fruit (nyy) using different classiers. 16,000 pixels were used
for mnm class, and 16,000 pixels were used for nyy class.
Cross validation
number

1
2
3
4
5
Average accuracy

NBC

KNN

SK-means

Mature & nearmature

Detected pixels (%)

Near-young &
young
Detected pixels (%)

Mature & nearmature

Detected pixels (%)

Near-young &
young
Detected pixels (%)

Mature & nearmature

Detected pixels (%)

Near-young &
young
Detected pixels (%)

11,996
11,939
11,970
11,929
11,803
11,904

13,493
13,586
13,592
13,457
13,450
13,521

14,464
14,551
14,527
14,567
14,498
14,536

14,455
14,296
14,306
14,333
14,477
14,353

12,462
12,399
12,373
12,414
12,393
12,395

10,823
10,853
10,800
10,920
10,821
10,849

(75)
(75)
(75)
(75)
(74)
(75)

(84)
(85)
(85)
(84)
(84)
(84)

(90)
(91)
(91)
(91)
(91)
(91)

(90)
(89)
(89)
(90)
(90)
(90)

(78)
(77)
(77)
(78)
(77)
(77)

(68)
(68)
(68)
(68)
(68)
(68)

Table 5
Cross validation accuracies for the classication of mature fruit (m) from near-mature fruit (nm) using different classiers. 8000 pixels were used for both m and nm classes.
Cross validation number

1
2
3
4
5
Average accuracy

NBC

KNN

SK-means

Mature
Detected pixels (%)

Near-mature
Detected pixels (%)

Mature
Detected pixels (%)

Near-mature
Detected pixels (%)

Mature
Detected pixels (%)

Near-mature
Detected pixels (%)

7598
7669
7620
7606
7645
7635

7257
7135
7234
7306
7212
7222

7672
7709
7732
7684
7705
7708

7745
7661
7700
7722
7679
7691

7142
7115
7266
7066
7123
7143

7177
7144
7065
7189
7167
7141

(95)
(96)
(95)
(95)
(96)
(95)

(91)
(89)
(90)
(91)
(90)
(90)

(96)
(96)
(97)
(96)
(96)
(96)

(97)
(96)
(96)
(97)
(96)
(96)

(89)
(89)
(91)
(88)
(89)
(89)

(90)
(89)
(88)
(90)
(90)
(89)

Table 6
Cross validation accuracies for the classication of young (y) and near-young fruit (ny) using different classiers. 8000 pixels were used for both ny and y classes.
Cross validation number

1
2
3
4
5
Average accuracy

NBC

KNN
Young
Detected pixels (%)

Near-young
Detected pixels (%)

Young
Detected pixels (%)

Near-young
Detected pixels (%)

Young
Detected pixels (%)

6478
6469
6337
6463
6518
6447

6568
6616
6838
6658
6571
6671

7860
7866
7853
7865
7130
7679

7718
7246
7026
7712
7643
7407

5196
5175
5197
5187
5281
5210

7144
7121
7168
7166
7161
7154

(81)
(81)
(79)
(81)
(81)
(81)

(82)
(83)
(85)
(83)
(82)
(83)

Table 7
Means test and one-way ANOVA used to compare the means of the cross validation
results of the three classication methods. The superscripts beside the mean values
indicate a group to which the method belongs.
Method

NBC
KNN
SK-means
p-Value

SK-means

Near-young
Detected pixels (%)

Data group
Fruit vs. background

mnm vs. nyy

m vs. nm

ny vs. y

55.8a
82.8b
52.4a
1.08e05

74.8a
90.8b
77.4c
2.93e15

95.4a
96.2a
89.2b
7.53e09

80.6a
96.2b
65.2c
4.66e10

performed well if not best. Therefore, RBH components were chosen for the other four separations as well besides the separation of
background and all fruits. The good performance of these three
components could be reected on the accuracies of the separations
in Tables 4 through 6.
Five separation steps were conducted sequentially to achieve
the nal goal of identifying blueberry fruit. To show the efciency
of the proposed CCAD method, cross validations were conducted
for the whole dataset by separating them into ve equal parts.
The classiers used were KNN, NBC and SK-means. As shown in
Tables 36, KNN always yielded the best results ranging from
85% to 98%. It was mostly because the datasets obtained from
the raw images contained well represented characteristics of the
classes existed in the images, and nding the nearest neighbor

(98)
(98)
(98)
(98)
(89)
(96)

(96)
(91)
(88)
(96)
(96)
(93)

(65)
(65)
(65)
(65)
(66)
(65)

(89)
(89)
(90)
(90)
(90)
(90)

for the samples in the validation dataset was a good way to classify
them. This result showed that this traditionally used classier was
efcient for the blueberry detection problem as long as a data set
was big enough. SK-means was a new classier proposed in this
study, which intended to take advantage of the clustering of
K-means and a supervised dataset.
From the analyses of this study, some interesting points arise.
First of all, the features of the three components shown in the histograms in Fig. 7 indicated the feasibility and the difculties of the
problem to be solved to detect fruit from the complicated background. Higher R value for the fruit was observed, compared with
that of background, which indicated that the R component could
help with the Separation 1 (building classiers to separate fruit
from background). The separation of young fruit from the background, especially the leaves, was a challenge because the distribution of the young fruit pixels spread over across the whole R and B
intensity values. This indicated that the Separations 3 (building
classiers to lter out additional leaves) was very challenging also.
The H color component could help separate the young fruit from
the leaves because the overlap between young fruit and background was less. However, this separation still remains as a problem because there was some overlap between fruit and leaves even
for the H component. As a result, mature fruit was well separated
from the other objects, and an accuracy of 85% using KNN was
obtained. However, the result of the separation of young fruit
and leaves was still a problem which needs more study in the

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

99

bg

leaf
bran_b

bran_older

ny
y

bran_g

nm

(a)

(b)

(c)

leaf

(d)

(e)

(f)

(g)

Fig. 8. Results of a KNN classier applied to a training image: (a) original image, (b) fruit separated from background, (c) mature and near-mature fruit, (d) near-young and
young fruit with some leaves, (e) near-young and young fruit after ltering out the leaves, (f) mature fruit only, and (g) near-mature fruit only.

future. Texture features might be useful for the separation of young

fruit and leaves.
Secondly, from the image separation results shown in Figs. 8
and 9, we can visually see the separation results showing that
the fruit was well detected from the background using the proposed method. The leaves were also ltered out using a classier
built specically for separation of leaves and young fruit, since it
was much harder to separate them in the Separations 1 and 2
due to their similar colors. These results indicated a great potential

for the CCAD method to detect blueberry fruit using natural outdoor RGB images. However, Figs. 8d and 9d also showed that the
young leaves were not completely removed after applying KNN
classiers, even after morphological operations were used to
remove segmentation errors and noise. The reason for this could
be found from the histogram analysis. A small portion of young
fruit was overlapped with the background, especially the leaves,
even in the H component distribution in Fig. 8c. Further research
is needed to solve this problem.

100

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

bran_g

bg

bran_b
leaf
bran_older
nm

ny

(a)

(b)

(c)

(d)

(e)

(f)

(g)

leaf

Fig. 9. Results of a KNN classier applied to a validation image: (a) original image, (b) fruit separated from background, (c) mature and near-mature fruit, (d) near-young and
young fruit with some leaves, (e) near-young and young fruit after ltering out the leaves, (f) mature fruit only, and (g) near-mature fruit only.

Thirdly, in Table 5, SK-means performed well for the separation

of m and nm with the accuracy of approximately 90%, indicating
the efciency of the newly proposed method. This high performance
was due to the following reasons: (1) it utilized the advantage of Kmeans, which is to partition a dataset to be classied in this study
into several clusters so that the similarity within a cluster is bigger
than that of other groups; (2) it utilized the advantage of supervised
learning when assigning the class label to the clusters of the dataset
to be classied; and (3) it also used weighted Euclidean distance
using a squared correlation coefcient when assigning the class
label to the clusters. The squared correlation coefcient revealed

the different relevance of the components and the class label. However, a performance difference of the SK-means method for detecting ny and y fruit was noticed. The ny pixel detection accuracy was
about 65%, while y detection accuracy was about 90%. This might
be due to the drawback of a K-means method, which is the convergence to a local minimum may produce incorrect results sometimes.
Similarly with any other clustering method, SK-means works well
with some datasets, while failing on others. While we explored the
feasibility of using SK-means for fruit detection in this study, it is
another interesting topic to study in the future, which is how to
make this method perform more stable.

H. Li et al. / Computers and Electronics in Agriculture 106 (2014) 91101

Lastly, most of the source of error for the results came from the
rst separation when the fruit were separated from background,
because the accuracy for the separation was approximately 80%.
This can be explained by the distribution of colors in the histograms as shown in Fig. 7. Even though most of the pixels can be
classied based on these three color components, there are still
some misclassications because of similar color components for
different classes. The errors in the result also might have been
caused by the size of the dataset not being large enough, because
the smaller the overlaps occupied in the dataset is, the higher
the accuracy should be. The next step will be to expand the data
library. In addition, the eld complexity might be another source
of error.

5. Conclusions
Natural outdoor color images were acquired to identify blueberry fruit of different growth stages using machine vision. The
major ndings of this study can be summarized as follows.
(1) This study examined the potential of machine learning
methods for blueberry yield mapping using natural outdoor
color images. A multi-class classication algorithm for separation of blueberries of different growth stages was developed, and the results showed great potential for blueberry
yield mapping with.
(2) To optimally use the information of the acquired color
images, three color components of R, B, and H were selected
through the FFSA method, among different color features of
R, G, B, H, S, I, Q, Cb, Cr and EGI.
(3) In the proposed color component analysis based detection
(CCAD) method, three classiers were applied to the corresponding dataset, including a supervised technique (KNN), a
probabilistic approach (NBC), and a newly proposed method
(SK-means). The proposed classier, SK-means, used Kmeans for supervised classication, which has never been
used in other studies before. SK-means not only took the
advantage of K-means for nding similarities in data and
putting similar dataset into several groups, but also utilized
the target class information for supervised classication, so
that it can assign a class label to each of the groups.
(4) To choose an efcient classier, cross validations were conducted on the validation dataset using the three different
classiers. The statistical analysis results indicated that
KNN classier performed best compared with the other
two. It yielded an average accuracy ranging from 85% to
86% for fruit and background separation, and an average
accuracy ranging from 90% to 98% for the separation of fruit
of mature, near-mature, near-young and young growth
stages.

Acknowledgements
The authors would like to thank Ms. Ce Yang, Mr. Anurag R.
Katti, Ms. Xiuhua Li, Dr. John Schueller, and Mr. James Colee at
the University of Florida for their assistance in this study. The

101

authors also would like to thank the China Scholarship Council

for nancial support.
References
Aggelopoulou, A.D., Bochtis, D., Fountas, S., Swain, K.C., Gemtos, T.A., Nanos, G.D.,
2011. Yield prediction in apple orchards based on image processing. Precision
Agric. 12 (3), 448456.
Alchanatis, V., Safren, O., Levi, O., Ostrovsky, V., 2007. Apple yield mapping using
hyperspectral machine vision. In: Stafford, J. (Ed.), Precision Agriculture.
Wageningen Academic Publishers, Wageningen, Netherlands, pp. 555562.
Annamalai, P., Lee, W.S., Burks, T., 2004. Color Vision Systems for Estimating Citrus
Yield in Real-time. ASAE Paper No. 043054, ASAE, St. Joseph, Mich.
Arslan, S., Colvin, T.S., 2002. Grain yield mapping: yield sensing, yield
reconstruction, and errors. Precision Agric. 3 (2), 135154.
Brazelton, C., 2013. 2012 World blueberry acreage & production report. North
American Blueberry Council. <http://oridablueberrygrowers.com/wp-content/
uploads/2013/03/NABC_2012-World-Blueberry-Acreage-Production.pdf>.
Bansal, R., Lee, W.S., Satish, S., 2013. Green citrus detection using fast Fourier
transform (FFT) leakage. Precision Agric. 14 (1), 5970. http://dx.doi.org/
10.1007/s11119-012-9292-3.
Chang, Y.K., Zaman, Q., Farooque, A.A., Schumann, A.W., Percival, D.C., 2012. An
automated yield monitoring system II for commercial wild blueberry doublehead harvester. Comput. Electron. Agric. 81, 97103.
Chinchuluun, R., Lee, W.S., 2006. Citrus yield mapping system in natural outdoor
scenes using the watershed transform. ASABE Paper No. 063010, ASABE, St.
Joseph, Mich.
Duda, R.O., Hart, P.E., Stork, D.G., 2001. Pattern Classication. John Wiley & Sons Inc.
Farooque, A.A., Chang, Y.K., Zaman, Q.U., Groulx, D., Schumann, A.W., Esau, T.J.,
2013. Performance evaluation of multiple ground based sensors mounted on a
commercial wild blueberry harvester to sense plant height, fruit yield and
topographic features in real-time. Comput. Electron. Agric. 91, 135144.
Gonzalez, R.C., Woods, R.E., 2002. Digital Image Processing, second ed. Prentice
Hall, Englewood Cliffs, NJ.
Hammouda, K., Karray, F., 2000. A comparative study of data clustering techniques.
SYDE 625: Tools of intelligent systems design, course project.
Johnson, R.A., Wichern, D.W., 1998. Applied Multivariate Statistical Analysis.
Prentice-Hall, Englewood Cliffs, NJ.
Kane, K.E., Lee, W.S., 2007. Multispectral imaging for in-eld green citrus
identication. ASABE Paper No. 073025, ASABE, St. Joseph, Mich.
Kumar, S., Ghosh, J., Crawford, M.M., 2001. Best-bases feature extraction algorithms
for classication of hyperspectral data. IEEE Trans. Geosc. Remote Sensing 39
(7), 13681379.
Kurtulmus, F., Lee, W.S., Vardar, A., 2011. Green citrus detection using eigenfruit,
color and circular Gabor texture features under natural outdoor conditions.
Comput. Electron. Agric. 78 (2), 140149.
Okamoto, H., Lee, W.S., 2009. Green citrus detection using hyperspectral imaging.
Comput. Electron. Agric. 66 (2), 201208.
Swain, K.C., Zaman, Q.U., Schumann, A.W., Percival, D.C., Bochtis, D.D., 2010.
Computer vision system for wild blueberry fruit yield mapping. Biosyst. Eng.
106 (4), 389394.
Sgaard, H.T., Olsen, H.J., 2003. Determination of crop rows by image analysis
without segmentation. Comput. Electron. Agric. 38 (2), 141158.
Szeliski, S., 2010. Computer vision: Algorithms and Applications. Springer.
U. S. Highbush Blueberry Council, 2012. Health benets of blueberries: blueberry
nutrition research. <http://www.blueberrycouncil.com/health-benets-ofblueberries/blueberry-nutrition/>.
Whitney, A.W., 1971. A direct method of nonparametric measurement selection.
IEEE Trans. Comput. C-20 (9), 11001103.
Woebbecke, D.M., Meyer, G.E., von Bargen, K., Mortensen, D.A., 1995. Color indices
for weed identication under various soil, residue, and lighting conditions.
Trans. Am. Soc. Agric. Eng. 38, 259269.
Yang, C., Lee, W.S., Williamson, J.G., 2012. Classication of blueberry fruit and leaves
based on spectral signatures. Biosyst. Eng. 113 (4), 351362.
Ye, N., Li, X., 2002. A scalable, incremental learning algorithm for classication
problems. Comput. Ind. Eng. J. 43 (4), 677692.
Zaman, Q.U., Schumann, A.W., Percival, D.C., Gordon, R.J., 2008. Estimation of wild
blueberry fruit yield using digital color photography. Trans. ASABE 51 (5),
15391544.
Zaman, Q.U., Swain, K.C., Schumann, A.W., Percival, D.C., 2010. Automated, low-cost
yield mapping of wild blueberry fruit. Appl. Eng. Agric. 26 (2), 225232.
Zhou, R., Damerow, L., Sun, Y., Blanke, M.M., 2012. Using colour features of cv. Gala
apple fruit in an orchard in image processing to predict yield. Precision Agric. 13
(5), 568580.