158

ECTI TRANSACTIONS ON ELECTRICAL ENG., ELECTRONICS, AND COMMUNICATIONS VOL.5, NO.2 August 2007

Color Based Segmentation of Nuclear Stained Breast Cancer Cell Images

Pornchai Phukpattaranont1 and Pleumjit Boonyaphiphat2 , Non-members
ABSTRACT We present an algorithm for segmenting cells in a microscopic image of immunohistologically stained slides from breast cancer based on color contents. The procedure for the approach consists of color categorization using neural network, noise removal and shape simplication using mathematical morphology, and cell size consideration. In order to obtain the more accurate segmentation, we further separate the attached cancer cells with marker-controlled watershed. Results show that cancer cells in the background with heterogeneous intensities and colors are appropriately segmented using the proposed method. In other words, the visualization of segmented cancer cells is in agreement with their perception in the original red-green-blue (RGB) image. In addition, the resistance of the method to intensity and color variations in the background is obtained without compromising of computational speed. Keywords: Quantitative Immunohistopathology, Image Segmentation, Cancer Cell Images 1. INTRODUCTION Immunohistochemistry is a technique used for detecting in situ a tissue antigen by a specic antibody. An antigen-antibody reaction is visualized by the color development of specic dye and can be seen by light microscope. The tissue antigen is presented at any part of the cell, i.e., cell membrane, cytoplasm or nucleus. Therefore, it is a useful technique to demonstrate the protein markers including cancer cell. Estrogen receptor (ER) and progesterone receptor (PR) are prognostic markers for breast cancer detected by this method. Evaluation of ER and PR positive cells are useful for hormonal therapy. Brown nuclear stains are recognized as positive cells and blue ones are negative cells. Traditionally, the percentage of positive cells of those markers are semiquantitatively counted. It is time consuming, costly, subjective and tedious. In order to overcome these problems, an automation
Manuscript received on March 1, 2007 ; revised on June 27, 2007. 1 The author is with the Department of Electrical Engineering, Prince of Songkla University, Hatyai, Songkhla, Thailand, Emails: pornchai.p@psu.ac.th 2 The author is with the Department of Pathology, Prince of Songkla University, Hatyai, Songkhla, Thailand, Emails: pleamjit.b@psu.ac.th

of image analysis that previously requires manual operations is performed on the basis of the developments in computer capabilities and image processing algorithms [1-3]. There are a number of benets that result from an automated analysis. These include an acceleration of the process, a reduction in cost for image analysis, as well as a decrement in a false inspection due to fatigue. Additionally, the automated analysis provides a quantitative description. Based on this quantitative measurement, the analysis result is objective. Furthermore, the correlation of the quantitative categorization with patient symptoms may allow for an automated diagnostic system [4]. However, it is not expected that automated image analysis will replace pathologists experience. It is only an aid to the pathologist for the repeated routine work and yields quantitative results that complement and enhance interpretations by pathologists. Visual examination by the pathologist is still required where unusual or abnormal cells that the method is not trained to deal with are encountered. Cell image segmentation methods for pathology are largely relied on two image processing techniques: thresholding and region growing. Thresholding identies each pixel in the image into either object or background based on its intensity. Image thresholding is a simple yet often eective means for obtaining a segmentation of images in which cells of interest situate in uniform background intensity across the image. Consequently, it is often used as a part in a sequence of image processing operations. For example, thresholding is one of important steps in segmenting live cell image [5]. For the image with uneven background, local adaptive thresholding method is an eective technique because the threshold value can be adapted to the background intensity variations. In other words, the threshold value is derived in each subregion, which is considered to be composed of both background and objects. Examples of medical image analysis that contain the local adaptive thresholding algorithm are as follows: (1) Segmentation of uorescence tumor cells from tissue sections [2], (2) Segmentation of cell nuclei from the stroma and fat like regions [3], (3) Segmentation of tissue components in liver tissue [4], (4) Segmentation of dead and live hepatocytes (liver cells) in cultures from microscopic images [6], and (5) Segmentation of nuclei from a breast tissue image [7]. One of disadvantages of the thresholding technique is

Color Based Segmentation of Nuclear Stained Breast Cancer Cell Images

159

that it does not take into account the spatial characteristics of the image. This causes it to be sensitive to noise and intensity heterogeneity. Therefore, further image processing algorithms that consider spatial modeling of the image need to be incorporated [8]. Region growing separates objects of interest from background in the image based on some predetermined criteria, i.e. intensity and/or edges [9]. While edge-based methods are sensitive to noise and artifacts, the intensity-based algorithms are usually more computationally expensive. Additionally, one of main disadvantage of region growing is the requirement of manual interaction to obtain the seed point [10]. Similar to thresholding, region growing is partially used in a set of image processing operations [11,12]. An example of region growing used in medical image analysis includes the extraction of noisy cell contour as appear in [13]. In addition to thresholding and region growing, other examples of segmenting techniques are segmentation of white blood cells based on morphological granulometries [14] and the Principle of Least Commitment [15]. We have recently presented the segmentation algorithm for breast cancer cell image based on local adaptive thresholding and mathematical morphology [16]. The method is appropriate for microscopic images with low histological noise, i.e., low variations on background color and intensity. However, the degree of histological noise in breast cancer images varies. For the image with high histological noise, the local adaptive thresholding approach is sensitive to noise and intensity heterogeneity. In addition, the computational time for the approach is quite lengthy. In order to address these problems, we propose a new strategy for segmenting cancer cells in a microscopic image of immunohistological nuclear staining of breast cancer tissue based on the color of pixel. This is motivated from the way that a pathologist determines the positive and negative of tumor cells by using their color contents manually. The remainder of this paper is organized as follows: Section 2 describes the acquisition of microscopic images and features of cancer cells from breast cancer tissue; Section 3 provides our proposed segmentation algorithm; Section 4 gives results and discussion; Section 5 draws conclusions of the paper. 2. MATERIALS AND METHODS 2. 1 Acquisition of the Images The image used in this paper were breast cancer tissue stained with the antibody to ER in the nuclei. The positive cells were marked by brown color and the negative ones were blue color. In addition, the inammatory cells such as lymphocytes, plasma cells and some connective tissue cells were labeled with blue color in the nuclei. However, these non-tumor cells were identied by their contours and pattern

histologic arrangements. The tissue sections were observed under a microscope with a magnifying factor of 40x. The contrast and intensity of staining were manually corrected in such a way that the digitized image became visually acceptable for further manual classication by a specialist. The cell images were acquired in color using the Eclipse 80i advanced research microscope (Nikon Instech Co., Ltd., Japan). The digital image was saved as a color 25603200 JPEG les for processing. 2. 2 Features of Cancer Cells

Fig.1: An example of stained cancer cell image. The brown and blue nuclei of the cells in the right and left circles are representative samples of positive and negative staining of estrogen receptor of breast cancer cells, respectively. Fig. 1 shows an example of stained cancer cell image from microscope with a magnication of 40x. This staining procedure is utilized to demonstrate the existing of estrogen or progesterone receptors in the breast cancer cells. In other words, stained cancer cells are classied into two categories according to their nuclear color contents, i.e. brown (P) and blue (N). The brown color indicates a positive staining while the blue one demonstrates a negative result. The brown and blue cells in the right and left circles shown in Fig. 1 are representative samples of positive and negative staining of estrogen receptor of cancer cells, respectively. The ratio of the total number of positive cancer cells to the total number of cancer cells in the whole image is used by a doctor for medical planning and treatment. It is also noticeable in Fig. 1 that there are color and intensity variations both in the cancer cells and in the background (noncancer cells). The ultimate goal of our research project is to develop an automatic algorithm for counting positive and negative cancer cells on immunohistological stained slides from breast cancer tissue. In order to achieve this objective, the method to segment both negative and positive cancer cells from the back-

160

ECTI TRANSACTIONS ON ELECTRICAL ENG., ELECTRONICS, AND COMMUNICATIONS VOL.5, NO.2 August 2007

ground is investigated. In this paper, we proposed the use of pixel color partitioning based on a neural network classier and morphological operators for segmenting cancer cells microscopically. 3. SEGMENTATION ALGORITHM The originally acquired image is in the red-greenblue (RGB) color space. That is, the color image is formed by the combination of red, green, and blue monochrome images. In the rst step, we classify color pixels in the image into one of three categories, i.e. background, P, or N, based on their RGB components. There are many classiers that can be used for partitioning color of a pixel. However, a neural network is chosen in this paper due to its well known as a successful classier for many applications [17-19]. Subsequently, morphology operations are used for addressing the spatial characteristics of cells. Finally, in order to obtain accurate cell counting results, the marker-controlled watershed is applied for separating attached multiple cells into distinct single cells. An algorithm for segmenting cancer cells based on their colors and sizes is as follows. An algorithm for cancer cell segmentation Read image le Classify pixels using neural network Do threshoding Do morphology opening Fill holes Label objects in the image Classify each object using its size IF size < 1500 THEN Ignore object ELSE IF 1500 size 7500 THEN Object is a distinct single cell Apply morphology closing (1) ELSE size > 7500 THEN Object is attached multiple cells Apply morphology opening Apply marker-controlled watershed (2) ENDIF Combine the results from (1) and (2) Details of the algorithm are given below. 3. 1 Neural Network We use backpropagation neural network to classify pixels in the microscopic image according to their color contents. Backpropagation is created by generalizing the Widrow-Ho learning rule to multiplelayer networks and nonlinear dierentiable transfer functions. Input vectors and the corresponding target vectors are used to train the network until it can classify the dened pattern. The training algorithms use the gradient of the performance function to determine how to adjust the weights to minimize performance. The gradient is determined by a technique called backpropagation, which involves

Fig.2: Diagram of backpropagation neural network. performing computations backwards through the network. The backpropagation computation is derived using the chain rule of calculus [20]. Based on our experiences, the number of neural network layer between two and three is appropriate for classifying color of pixels in cancer cell images. Therefore, a backpropagation neural network of three layers shown in Fig. 2 is chosen to classify image pixels whether they are from background, P, or N regions in this paper. The input vector is composed of 3 elements corresponding to the RGB color vector of pixel. Two hidden layers are determined empirically to be 3 and 4 neurons and the output layer consists of one neuron. In addition, the transfer functions of hidden and output layers are tan-sigmoid and linear, respectively. For the training of neural network, the target is assigned to be -1, 0, and 1, which are corresponding to RGB components from background, P, and N regions, respectively. The networks are trained using the Levenberg-Marquardt (LM) algorithm. The training stops when the maximum number of epochs reaches 100 or the mean square error is less than 1 1012 . The number of samples used for training neural network from each reference region is 1600. 3. 2 Morphology Operations Mathematical morphology is a nonlinear operator based on set theory operating on object shape. It is a powerful tool to numerous image processing problems, for example, image preprocessing, segmentation using object shape, and quantitative description objects [?, ]. We utilize mathematical morphology as a tool for noise ltering and shape simplication in our work. Note that the disk-shaped structuring element (SE) with a radius of R is used for all morphological operations in this paper. As a result, the size of SE matrix is 2R + 1 by 2R + 1. In addition, all mathematical morphology operations are applied once for each stage. After nishing color partition for all pixels, the output image from neural network is transformed to a black and white image by thresholding. That is, while the pixels in background region are transformed to

Color Based Segmentation of Nuclear Stained Breast Cancer Cell Images

161

be zero, the pixels in P and N regions, i.e., objects of interest, are transformed to be one. In order to eliminate spike noise, the binary image is processed using morphological opening. The disk-shaped SE with a radius of 1 is used in this stage. The algorithm based on morphological reconstruction is subsequently used to ll holes in the image before performing size consideration. In the next step, we classify each object in the image into one of three categories according to its size: small, medium, and large. The value of sizes used for cell classication is predetermined from guidance by a specialist. The small object is regarded as noise and is ignored. The medium object is considered to be a distinct single cell. For the large object, it is determined as attached multiple cells. All distinct single cells are processed further with mathematical closing to complete their shape. It is used as spatial compensation for an uneven distribution of color in the cell. The SE used in this stage has a radius of 9. On the other hand, all attached multiple cells are applied with morphological opening. The radius of the SE in this stage is 12. There are two explanations for doing this performing. First, the multiple cells with small degree of attachment can be kept apart. Second, it can be used as a preparation step before marker-controlled watershed processing. 3. 3 Marker-Controlled Watershed In order to separate attached cancer cells into individual objects, we further process the result from last step with marker-controlled watershed. The watershed algorithm is shown to be a powerful tool for dividing attached objects [22]. The marker computation is used as an additional processing because the direct use of watershed transform usually yields the over-segmented result [2]. The computational procedures for marker-controlled watershed are as follows. Step 1. Use the Sobel edge marks to compute the gradient magnitude of all attached cells, Step 2. Determine the marker, which is connected blobs of pixels inside each cell, based on the distant transform. Step 3. Combine the results of Step 1. and Step 2, Step 4. Compute the watershed transform of the result from Step 3. Finally, to obtain whole cancer cells, we combine the image of attached multiple cells with the image of distinct single cells using a logical operator OR. 4. RESULTS AND DISCUSSION 4. 1 Neural Network Fig. 3 shows an original RGB image of cancer cells. Most of cancer cells are located separately, but some of them attach with their neighborhood. We also see an uneven distribution of color and intensity in the background region surrounding cancer cells. It is

Fig.3: Original RGB image of stained cancer cells. Boxes B1, B2, and B3 indicate reference regions from background, P, and N, respectively. Pixel values in the box regions are used for training neural network.)

considered as histological noise. The R-, G- and Bcomponents of pixel values from background, P, and N reference regions (Boxes B1, B2, and B3) shown in Fig. 3 are used as the input vectors for training neural network. After the backpropagation neural network is trained, network responses of pixel values throughout the image are calculated.

Fig.4: Output image from neural network. Pixels from background, P, and N regions are shown in white, gray, and black colors, respectively. Fig. 4 shows the output image from neural network. Pixels from background, P, and N regions are shown in white, gray, and black colors, respectively. Results demonstrate that the neural network can classify color pixels very well. That is, pixel values are appropriately categorized into a connected region corresponding to the cancer cells shown in Fig. 3. Additionally, the results from neural network show that color contents of pixel values for each cancer cell are unevenly distributed. For example, we can notice the appearance of an area with background color inside the biggest brown cancer cell situating at the left hand side of box B1. This is in agreement with the original RGB image.

162

ECTI TRANSACTIONS ON ELECTRICAL ENG., ELECTRONICS, AND COMMUNICATIONS VOL.5, NO.2 August 2007

phological opening. One can see that a slightly attached multiple cell is separated into two distinct single cells as we expect. In addition, the shape of each cell is smoothed and simplied, which make it appropriate for marker-controlled watershed processing. 4. 3 Marker-Controlled Watershed Fig. 5 (f) shows the segmented image after the application of marker-controlled watershed. A more accurate segmented result is achieved. That is, two attached multiple cells are appropriately separated into four distinct single cells.

Fig.5: (a) Binary image after thresholding and morphological operations of the image from neural network. (b) Binary image of distinct single cells before morphological closing. (c) Binary image of distinct single cells after morphological closing. (d) Binary image of attached multiple cells before morphological opening. (e) Binary image of attached multiple cells after morphological opening. (f ) Binary image of attached multiple cells after marker-controlled watershed. 4. 2 Morphology Operations Fig. 5 (a) shows the binary image after thresholding and morphological opening for spike noise removal. Subsequently, hole lling is performed using the algorithm based on morphological reconstruction. Each object in the image is classied into three categories according to its size. Fig. 5 (b) shows the binary image of distinct single cells after size consideration. One can notice that some cells do not have the complete shape due to an uneven distribution of color and intensity. To compensate for this shortcoming, we perform mathematical closing and show result in Fig. 5 (c). It is shown that distinct single cells with perfect round shape are obtained after mathematical closing. Fig. 5 (d) shows the binary image of attached multiple cells after size consideration. The size of cancer cells in this image is large in terms of area compared with the size of distinct single cell. Fig. 5 (e) shows the binary image of attached multiple cells after mor-

Fig.6: Segmented image after the application of a logical operator OR. Fig. 6 shows the segmented image after the application of a logical operator OR. Note that we discard cells at the boundary of image because of their partial appearance. We superimpose this segmented image on the original RGB image and demonstrate the result in Fig. 7. The number of segmented cancer cells is 24. It can be clearly seen that the perception of segmented cancer cells is in agreement with their original visualization.

Fig.7: Segmented image superimposed on the original RGB image.

Color Based Segmentation of Nuclear Stained Breast Cancer Cell Images

163

5. CONCLUSIONS This paper presents the segmentation of cancer cells in a microscopic tissue image from breast cancer based on color contents. The segmenting results of cancer cells from the background are used as a preliminary step before extracting cell features and classifying cell types. The excellent segmentation results from the proposed algorithm are demonstrated with microscopic images under histological noise conditions. However, to make the method automatic, the algorithm that can compensate for the selection of training data by a specialist need to be incorporated. Comparison of automatic cell image classication based on the proposed segmentation algorithm to a pathologist is ongoing research. Result will be reported in the near future. 6. ACKNOWLEDGEMENT This research is supported by Department of Electrical Engineering, Faculty of Engineering and Department of Pathology, Faculty of Medicine, Prince of Songkla University. References [1] J. P. Thiran and B. Macq, Morphological feature extraction for the classication of digital images of cancerous tissues, IEEE Trans. Biomed. Eng., vol. 43, no. 10, pp. 1011-1020, Oct. 1996. B. Fang, W. Hsu, and M. L. Lee, On the accurate counting of tumor cells, IEEE Trans. Nanobiosci., vol. 2, no. 2, pp. 94-103, Jun. 2003. S. Petushi, C. Katsinis, C. Coward, F. Garcia, and A. Tozeren, Automated identication of microstructures on histology slides, in IEEE International Symposium on Biomedical Imaging: Macro to Nano, vol. 1, pp. 424-427., 2004. L. OGorman, A. C. Sanderson, and K. Preston Jr., A system for automated liver tissue image analysis: Methods and results, IEEE Trans. Biomed. Eng., vol. 32, no. 9, pp. 696-706, Sep. 1985. K. Wu, D. Gauthier, and M. D. Levine, Live cell image segmentation, IEEE Trans. Biomed. Eng., vol. 42, no. 1, pp. 1-12, Jan. 1995. H. Refai, L. Li, and T. K. Teague, Automatic count of hepatocytes in microscopic images, inInternational Conference on Image Processing, vol. 2, pp. 1101-1104., 2003. X. Zhao and S. H. Ong, Adaptive local thresholding with fuzzy-validity-guided spatial partitioning, in International Conference on Pattern Recognition, vol. 2, pp. 988-990., 1998 H. D. Li, M. Kallergi, L. P. Clarke, V. K. Jain, and R. A. Clark, Markov random eld for tumor detection in digital mammography, IEEE Trans. Med. Imag., vol. 14, no. 3, pp. 565-576, Sep. 1995.

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[2]

[3]

[17]

[4]

[18]

[5]

[19]

[6]

[20]

[7]

[21] [22]

[8]

R. M. Haralik and L. G. Shapiro, Image segmen- tation techniques, Comput. Vis. Graph. Image Proc., vol. 29, pp. 100-132, 1985. R. Adams and L. Bischof, Seeded region growing, IEEE Trans. Pattern Anal. Mach. Intell., vol. 16, pp. 641-647, Jun. 1994. J. R. Beveridge and et al., Segmenting images using localized histograms and region merging, International Journal of Computer Vision, vol. 2, pp. 311-347, 1989. M. A. Wani and B. G. Batchelor, Edgedregion- based segmentation of range images, IEEE Trans. Pattern Anal. Mach. Intell., vol. 16, pp. 314-319-647, Mar. 1994. H. S. Wu and J. Barba, An algorithm for noisy cell contour extraction via area merging, J. Imag. Sci. Technol., vol. 38, pp. 604-607, Nov. 1994. N. Theera-Umpon and P. D. Gader, Counting white blood cells using morphological granulometries, Journal of Electronic Imaging, vol. 9, no. 2, pp. 170-177, Apr. 2000. J. Park and J. Keller, Fuzzy patch label relaxation in bone marrow cell segmentation, in Proc. IEEE Int. Conf. Syst. Man. Cybern., vol. 2, pp. 1133-1138., 1997. P. Phukpattaranont and P. Boonyaphiphat, Automatic classication of cancer cells in micro- scopic images: Preliminary results, in The 2006 ITC-CSCC International Conference, vol. 1, pp. 113-116., 2006. E. Gelenbe, Y. Feng, and K. R. R. Krishnan, Neural network methods for volumetric magnetic resonance imaging of the human brain, Proc. IEEE, vol. 84, no. 10, pp. 1488-1496, Oct. 1996. W. E. Reddick, G. O. Glass, E. N. Cook, T. D. Elkin, and R. J. Deaton, Automated segmentation and classication of multispectral magnetic resonance images of brain using articial neural network, IEEE Trans. Med. Imag., vol. 16, pp. 911-918, 1997. P. D. Heermann and N. Khazenie, Classication of multispectral remote sensing data using a back-propagation neural network, IEEE Trans. Geosci. Remote Sensing, vol. 30, no. 1, pp. 81-88, Jan. 1992. M. T. Hagan, H. B. Demuth, and M. H. Beale,Neural Network Design, PWS Publishing Com- pany, Boston, MA, 1996. R. C. Gonzalez and R. E. Woods, Digital Image Processing, Prentice-Hall, Inc., 2002. L. Vincent, Morphological grayscale reconstruc- tion in image analysis: applications and ecient algorithms, IEEE Trans. Image Process., vol. 2, no. 2, pp. 176-201, Apr. 1993.

164

ECTI TRANSACTIONS ON ELECTRICAL ENG., ELECTRONICS, AND COMMUNICATIONS VOL.5, NO.2 August 2007

Pornchai Phukpattaranont was born in Songkla, Thailand. He received the B. Eng. and M. Eng. degrees in electrical engineering from Prince of Songkla University in 1993 and 1997, respectively, the Ph.D. degree in electrical engineering from the University of Minnesota, in 2004. He is currently an assistant professor of electrical engineering at Prince of Songkla University. His research interests are ultrasound contrast imaging, ultrasound signal processing, medical image processing, and biomedical signal processing. Dr.Phukpattaranont is a member of the IEEE.

Pleumjit Boonyaphiphat was born in Thailand. She received the B.Sc. degree from Faculty of Nursing, Prince of Songkla University, in 1980, the M.Sc. degree from Faculty of Science, Mahidol University, in 1986. She has been working at Department of Pathology, Faculty of Medicine, Prince of Songkla University since 1986. Her position is an assistant professor in anatomical pathology. Her research eld is head and neck cancer, breast cancer and image analyzer for histopathology.