METODE SEGMENTASI UNTUK TUMOR HATI PADA

GAMBAR CT SCAN MENGGUNAKAN DEEP

LEARNING
Girindra Wardhana

Department of Robotics and Mechatronics.

Faculty of Electrical Engineering, Mathematics, and Computer Science,
Technical Medical Center, University of Twente
1. INTRODUCTION
KonsepDeepLearning

Artificial
Perbedaan antara AI, Machine Learning, dan
Intelligence
Deep Learning

Machine Artificial Intelligence (AI)

Learning Usaha untuk membuat mesin bisa berpikir dan
bertindak seperti manusia

Machine Learning (ML)

Bagian dari AI, dimana suatu mesin dapat
Deep Learning mengembangkan algoritmanya sendiri dari data yang
diberikan

Deep Learning (DL)

Bagian dari ML, algoritma yang menyerupai kerja
otak manusia dalam memproses informasi dalam
bentuk multi layered neural network

3 March 2021 # 3
NeuronModel
Prinsip kerja yang menyerupai otak manusia

Neural Network Neural Component Activation Function

Simulasi dari proses kerja Input dikirim melalui synapse Threshold yang harus
otak dalam mengolah dan diolah di neuron dipenuhi untuk
informasi memperoleh output

3 March 2021 # 4
JenisMachineLearning
Machine Learning dapat dibagi menjadi tiga kategori

Cabang-cabang machine learning

Supervised Learning
Machine learning digunakan untuk memetakan input
data ke label yang sudah diketahui

Unsupervised Learning
Model membentuk kategori sendiri sebagai
outputnya

Reinforcement Learning
Menggunakan system reward and punishment untuk
melatih model saat belajar dengan lingkungannya

https://technovert.com/introduction-to-machine-learning/

3 March 2021 # 5
SupervisedLearning
Kategori-kategori machine learning

https://shuvorahman.com/2020/10/04/understanding-supervised-learning/

3 March 2021 # 6
UnsupervisedLearning
Kategori-kategori machine learning

https://technovert.com/introduction-to-machine-learning/

3 March 2021 # 7
ReinforcementLearning
Kategori-kategori machine learning

https://www.youtube.com/watch?v=C-WbsyfRc0w

AI beats flappy bird world's hardest level

3 March 2021 # 8
PerkembanganDeepLearning
Mengapa deep learning semakin populer

Peningkatan performa
• Hardware yang semakin berkembang
• Jumlah data yang semakin banyak
• Jenis layer yang semakin beragam
• Fitur yang diperoleh secara otomatis

Aplikasi Deep Learning

• Image classification
• Speech recognition
• Natural language processing

3 March 2021 # 9
InteractiveDeepLearning
Contoh-contoh aplikasi deep learning di internet

Quick, Draw! Image-to-Image

3 March 2021 # 10
2. DEEP LEARNING &
COMPUTER VISION
ConvolutionNeuralNetwork
Network layer untuk computer vision

Convolution Layer
• Mengekstrak fitur dari gambar
• Terdiri dari input, kernel, dan
output
• Parameter yang diatur
• Padding
• Strides
• Kernel size

3 March 2021 # 12
3 March 2021 #
ConvolutionNetwork
Feature extraction layer in the neural network

Source: http://cs231n.github.io/convolutional-networks/

3 March 2021 # 14
ImageClassification
ImageNet classification challenge

http://karpathy.github.io/2014/09/02/what-i-learned-from-competing-against-a-convnet-on-imagenet/
ImageNet: Classification of 1.2 million images with 1000 labels
3 March 2021 # 15
AplikasiComputerVision
Berbagai aplikasi deep learning di bidang computer vision

3 March 2021 # 16
AplikasiComputerVision
Berbagai aplikasi deep learning di bidang computer vision

3 March 2021 # 17
AplikasiComputerVision
Berbagai aplikasi deep learning di bidang computer vision

3 March 2021 # 18
3. DEEP LEARNING IN
MEDICAL IMAGE
MedicalApplication
Deep learning in medical images

Ratul, Md Aminur Rab, et al. "Atrous Convolution with Transfer Zhao, Jinyu, et al. "Covid-ct-dataset: a ct scan dataset
Learning for Skin Lesions Classification." BioRxiv (2019): 746388. about covid-19." arXiv preprint arXiv:2003.13865 (2020).

Skin lesion classification COVID-19 Classification

3 March 2021 # 20
MedicalApplication
Deep learning in medical images

Image Registration
Register multimodal image

Yabo Fu, Yang Lei, Tonghe Wang, Jun Zhou, Walter J. Curran, Pretesh Patel, Tian Liu, Xiaofeng Yang, "Deformable MRI-CT liver image
registration using convolutional neural network with modality independent neighborhood descriptors," Proc. SPIE 11597, Medical Imaging
2021: Computer-Aided Diagnosis, 115970G (15 February 2021); https://doi-org.ezproxy2.utwente.nl/10.1117/12.2581082

Image Reconstruction
Reconstruct low resolution image

Knoll, Florian (2020). Chapter 2 - Machine learning for image reconstruction. The Elsevier and MICCAI Society Book Series, , 25-64.

3 March 2021 # 21
MedicalApplication
Deep learning in medical images

Li, W., Li, Y., Qin, W., Liang, X., Xu, J., Xiong, J., & Xie, Y. (2020). Magnetic
resonance image (MRI) synthesis from brain computed tomography (CT)
images based on deep learning methods for magnetic resonance (MR)-
guided radiotherapy. Quantitative Imaging In Medicine And Surgery, 10(6),
1223-1236. doi:10.21037/qims-19-885
Wardhana, Girindra, et al. "Toward reliable automatic liver and tumor
Gibson, Eli, et al. "NiftyNet: a deep-learning platform for medical segmentation using convolutional neural network based on 2.5 D
imaging." Computer methods and programs in biomedicine 158 (2018): 113-122. models." International Journal of Computer Assisted Radiology and
Surgery 16.1 (2021): 41-51.

Image Synthesis Image Segmentation

Reconstruc MRI image from CT Scan Multi organ segmentation from CT Scan

3 March 2021 # 22
4. AUTOMATIC
SEGMENTATION OF
LIVER TUMOR
Introduction
Image segmentation in medical imaging

https://www.emedicinehealth.com/liver/article_em.htm https://www.photonics.com/Article.aspx?AID=57448

Visualization Operation Plan Diagnosis

Help clinicians visualizing Provide a basis for surgery Improve the clinical
the anatomy of the planning in robotic surgeries diagnosis and treatment
patient’s body decision making

3 March 2021 24
SegmentationChallenge
Challenges in the automatic method for liver segmentation

Liver Intensity Tumor Appearance

Large variety of liver intensity due to The intensity of liver is not uniform due to the
different acquisition technique and contrast tumor inside
agent

Organ Contrast
Low contrast among the organ make the Tumor Location
liver boundary fuzzy and hard to detect The location and number of tumor are
varied among the patients

Liver Shape Tumor Number

Liver shapes and sizes are highly varied In contrast with liver number, there is no
between individual specific number of tumor in the patient

Manual Segmentation Automatic Methods Method requirements

It requires a lot of time, Several methods are Method requires prior
laborious and prone to developed such as region knowledge and features
subjectivity. growing, thresholding and that manually extracted
active shape model. from the image.

3 March 2021 25
TrainingStep

Data preparation
- Gathering Data
- Preparing that data
Network Training
- Data Augmentation
- Class Balancing
- Training option
Post-Processing and Evaluation
- Evaluation Metric
- Hyperparameter tuning

3 March 2021 26
DataPreparation

Dataset source
The datasets are taken from Liver and
Tumor Segmentation (LiTS) challenge.
The datasets contain contrast-enhanced
abdominal CT scans from different
clinical site around the world. There are
two part, 130 CT scans for training and
70 CT scan for testing.

3 March 2021 # 27
ModelArchitecture
Layer Architecture
Neural Network model is developed using
Neural Network Toolbox from Matlab 2018a

Convolutional Layer
Two different convolution layer sizes are
used, 3×3 kernel size and 1×1 kernel
size
Pooling Layer
Utilize Max Pooling type for Pooling
layer with size of 2×2 and stride 2

Fusion and Concatenate Layer

In the fusion layer, feature maps are
combined by applying addition
operation. While concatenate layer
combine the feature by stacking the
feature maps.

3 March 2021 # 28
NetworkTraining

Data Augmentation
A minor alteration
like reflection,
rotation, scales and Class Balancing
transition are Implementing data
randomly applied resampling and class
weight to counter the
effect of dominant class
Training Option in the segmentation
result.
Batch size: 8
Total epoch: 20
Learning rate: 10-3
Drop factor: 0.1/5 epoch
Pixel class distribution
Solver function: SGDM
Momentum: 0.8
3 March 2021 # 29
EvaluationMetric
A metric to measure the performance of the network

Dice Score
Statistic for
comparing the
similarity of two
Confusion Matrix samples
2𝑇𝑃
Evaluation Metric 2𝑇𝑃 + 𝐹𝑃 + 𝐹𝑁
The performance is measured by
comparing the ground truth Recall
segmentation (actual value) Measure the
with test segmentation fraction of positive
(predicted value) value that correctly
predicted
𝑇𝑃
𝑇𝑃 + 𝐹𝑃

3 March 2021 # 30
AutomaticSegmentationGUI
Application interface for the proposed network

Application GUI
The graphical user interface for the application has
been developed using Matlab GUI Development
Environment (GUIDE)

Liver Segmentation Tumor Segmentation

3 March 2021 # 31
Hyperparameter | SliceArrangement
Investigation on the number of stacked slices in the input image

CT Image Single Layer Multi Layer

Input Input

3 March 2021 # 32
Hyperparameter | ContrastEnhancement
Investigation on the effect of image contrast

► Four networks were tested:

► Image pre-processing
1 2 3

Basic Contrast Enhancement

1. Raw image
2. Calibrated image
3. Select intensity range (-250,250),
and normalize into (0,255)

3 March 2021 # 33
Hyperparameter | NetworkLayer
Investigation on the number of layer in the network structure

3 March 2021 # 34
SliceArrangementTest
Experiment result of slice arrangement on 20 image dataset

Dice
Score
Slice-1 Slice-3 Slice-5 Slice-7 Slice-9
P
Value*
Segmentation Result
Liver 87.7±5.4% 90.5±4.7% 89.7±5.1% 89.1±5.2% 90.1±4.7% 0.002 Multi layer network are
Tumor 33.5±26.3% 41.1±28.0% 39.9±27.9% 33.4±25.7% 38.4±27.8% <0.0005
performed better than single
layer network

3 March 2021 # 35
ContrastEnhancementTest
Experiment result of contrast enhancement on 20 image dataset

The lowest score

Obtained by histogram
equalization method.
Histogram data are varied
in the image dataset

The highest score

Achieved by bilateral
filtering. But it is risky due
to possibility to remove
small lesion from the image

Statistic Result
The difference was not
statistically significant.
Basic Histogram Gamma Bilateral
Dice Score P Value*
Enhancement Equalization Correction Filtering

Liver 90.5±4.7% 85.1±16.9% 89.1±5.7% 90.6±4.7% 0.117

Tumor 41.2±28.0% 36.4±26.6% 38.6±27.8% 41.4±28.4% 0.094

3 March 2021 # 36
NetworkLayerTest
Experiment result between Net01 and Net02

Segmentation Comparison
Different behavior of segmentation
result from Net01 and Net02

Higher Sensitivity
1 Net02 is able to segment
tumor that are not
detected by Net 01

Lower Accuracy
Net01 Net02 2 More region are falsely
Team
Lesion Liver detected as tumor by
Dice per case Jaccard Dice per case Jaccard
Net02
VGG-FCN 29.8% 19.6% 91.9% 29.8%
ResNet 48.5% 37.2% 95.2% 91.1%
Cascaded ResNet (w Multi-scale Fusion)[15] 50.0% 38.7% 95.9% 92.1%
Larger Area
Net01 – Encoder and Decoder Net
Net02 – Densely Encoder and Decoder Net
56.2%
50.1%
61.9%
56.6%
91.4%
91.1%
84.6%
83.9%
3 Net02 tend to exaggerate
Hans.meine [14] 67.6% 79.6% 96.0% 96.5%
the tumor area
H-DenseUnet [16] 72.2% 82.4% 96.1% 96.5%

3 March 2021 # 37
Hyperparameter | Conclusion
Three steps during the manual segmentation

Automatic Segmentation Method Contrast Enhancement

1 has been successfully developed, using a deep 3 method based on image histogram information
Convolutional Neural Network and utilizing a decrease the performance of segmentation
modified SegNet architecture. network.

Slice Arrangement Expanded Network Architecture

2 shows that stacked slice achieve a higher Dice 4 Increase the network ability to recognize tumor
score than the network with a single slice to some from the image.
extent.

3 March 2021 # 38
5. DEEP LEARNING TIPS
StarterTips
Where should I start?

TIPS
• Choose your platform
• Follow tutorial in GitHub
• Watch deep learning
algorithm explanation on
YouTube
• Follow latest news
regarding Deep Learning
development

3 March 2021 # 40
CloudPlatform
Alternative when resource is limited

Advantages
• Easy to scale up
• Support on Python-based
machine learning ,
TensorFlow or PyTorch
• Provide API for:
• Image recognition
Free Service* Paid Service
• Video analysis
- Google Collab - Google Cloud Platform
• Speech to Text - Kaggle Kernels - Microsoft Azure
• Text to Speech - Amazon Web Services
• Translation * Limited to 24-hour session - IBM Cloud
• Chatbot - Alibaba Cloud

3 March 2021 # 41
MedicalDataset
Useful source for medical dataset

Grand Challenge

• Berisi berbagai sumber

data medis
• Benchmark algoritma yang
dikembangkan
• Platform berbagi
algoritma dengan user lain

https://grand-challenge.org/challenges/

3 March 2021 # 42
MedicalDataset
Useful source for medical dataset

Medical Segmentation
Decathlon

• Data dan label dari

berbagai organ tubuh
manusia
• Fokus di pengembangan
general deep learning untuk
segmentasi berbagai organ
http://medicaldecathlon.com

3 March 2021 # 43
MedicalDataset
Useful source for medical dataset

Classification
Melanoma https://www.kaggle.com/c/siim-isic-melanoma-classification/data
PatchCamelyon https://github.com/basveeling/pcam
ProstateX https://wiki.cancerimagingarchive.net/display/Public/SPIE-AAPM-NCI+
MoN uSAC https://monusac-2020.grand-challenge.org/Home/

Registration
OASIS https://www.oasis-brains.org Brain MRI
AD HD http://fcon_1000.projects.nitrc.org/indi/adhd200/index.html Brain MRI
ABIDE http://fcon_1000.projects.nitrc.org/indi/abide/ Brain MRI
MSD https://decathlon-10.grand-challenge.org/ Liver CT

Covid-19
COVID-CT https://github.com/UCSD-AI4H/COVID-CT
COVID-Chest XRay https://github.com/ieee8023/covid-chestxray-dataset
COVID-19 CT http://medicalsegmentation.com/covid19/
BIMCV-COVID19 http://bimcv.cipf.es/bimcv-projects/bimcv-covid19/
MosMedData Covidl9 https://mosmed.ai/en/

3 March 2021 # 44
MedicalDataset
Useful source for medical dataset

Medical Image Segmentation

Kvasir-SEG https://datasets.simula.no/kvasir-seg/ Polyp Segmentation
CVC-ClinicDB https://polyp.grand-challenge.org/CVCClinicDB/ Polyp Segmentation
iSeg-2017 http://isep2017.web.unc.edu/ MRI Segmentation
CHAOS https://chaos.grand-challenge.org/Combined_Healthy_Abdominal_Organ_Segmentation/ Organ Segmentation
DRIVE https://drive.grand-challenge.org/ Retinal Segmentation
ISIC 2018 https://challenge2018.isic-archive.com/ Lesion Segmentation
BUSIS http://cvprip.cs.usu.edu/busbench/ Ultrasound Segmentation
BraTS https://www.med.upenn.edu/sbia/brats2018/data.html Brain Tumor Segmentation
TCIA Pancreas-CT https://wiki.cancerimagingarchive.net/display/Public/Pancreas-CT 3D Segmentation
ROBUST https://robustmis2019.grand-challenge.org/ Medical Tools Segmentation

3 March 2021 # 45
THANK YOU
For any suggestion or question,
contact to: g.wardhana@utwente.nl