S. Bertoluzza, G.

Maggi

Dr. Tomatis, Istituto Tumori in Milan

Radiation therapy programmed by a machine

which is given
Targets to irradiate
regions to avoid

Identify and isolate anatomical structures to

input such machine

Segment an image = tag each pixel/voxel with a semantic

label

Quo vadis, Atlas based segmentation di T. Rohlfing, R. Brand, R.

Menzel, D.B. Russakoff, C.R. Maurer jr

Intensity based classification

Group the gray level space in clusters
Each cluster is identified with a label
Each pixel is tagged with the label of the

corresponding cluster

Ok for classifying tissue types

Not Ok when classifying anatomical
structures

Make the geometry of the image evolve using

Image properties (gradient)
Constraints (smoothness of the segmented curves)

Level set method

Active contours
Snakes
...

Template image T, already segmented

Find a map transforming the template image

in the image R to be segmented
Tranfer on R the segmentation of T

= image domain
L = label set = {e1,e2,e3,...,eN}

A: >L map tagging each position with a

label
T: >R+: template image

T segmented exactely by superposing A

Atlas: (T,A)

R: -> R+ - image to be segmented

Find mapping : -> such that T0 are as

close as possible

Choose A0 as segmentation of R

Segmentation is reduced to registration

Choice of atlas has substantial impact on the

result of the segmentation algorithm

Four strategies
Single atlas for all images to be segmented
Choice of best atlas for each given image into an

atlas set
Average atlas for all the images to be segmented
Simultaneous use of multiple atlases

Strategia

#Atlas x image

Type of atlas

Assignment of atlas to F

IND

Single

Individual

Fixed

SIM

Single

Individual

Variable

AVG

Single

Average

Fixed

MUL

Multiple

Individual

Fixed

Selection of an individual atlas

Registration of all the raw images with the
selected atlas
(individual atlas = obtained by a single raw
image)

Set of atlases
Segment R using all atlases
Select the atlas that gives the best risult
Maximum similarity
Minimum deformation

Construct an artificial tipical image T from a

set of images
Construct corresponding atlas
Register all images with T

Set of atlases
Register R using each atlas
Tag pixels with labels usin all information
Ex: voting strategy <-> tag with label chosen by

highest number of atlases

Grey scale images <-> real valued functions

R: -> R+, T: -> R+

Find mapping : R -> T such that T0 and R

are as close as possible

Huge number of approaches

Huge number of algorithms

See [Brown 92; Zitova,Flusser 03]

Huge number of algorithms obtained by

combining
Different distance functionals
Different transformation classes
Different image models
Different optimization algorithms

Focus on the distance functional

Rigid transformations (translations, rotations)

Affine transformation

Non rigid transformations

Splines, B-splines [Thevenaz, Unser 98]
Thin plate splines [Bookstein, 89]
Interpolating wavelets [S.B., G. Maggi 13]
...
Parametric transformations

N-dimensional parameter space

Parameter -> mapping

(Ex: = i ei )

= (x;)
x : spatial coordinate (in R2 )
: parameter (in RN )
: R2 x RN function (represents the selected class

of transformations)

Im: space of images

d: Im x Im -> R functional measuring the

discrepancy between the two images

: Im -> R defined as (X) = d(X,R)

c : RN -> R cost functional

c() = d(T0 ,R) = (T0 )

Many different possibilities

Least square error
Human Visual System model distance [Mannos, 74]
Structural similarity index SSI
[Wang et al, 04, Brunet, Vrscay, Wang 11]
Besov functional norm
Besov norm + divisive renormalization (generalised
SSI) [S.B., G. Maggi, 14]
Mutual information [Viola, Wells 97; Thevenaz, Unser 98]
...

Find in RN
= arg min c()

Unconstrained optimization problem

Several possible optimization algorithms

Choice depends on the characteristics of
Transformation
Distance functional

Need rules to
Evaluate images out of their domain of definition

(extrapolation)
Compute values of (deformed) images at pixels
(interpolation)
Compute the values of derivative of images at
pixels (numerical differentiation)

Many different possibilities

[Citare: ....]

X,Y quantized random variables

measure of the mutual dependence of X and Y

MI ( X , Y ) = p[ X = xi , Y = y j ] log
ij

p[ X = xi , Y = y j ]
p[ X = xi ] p[Y = y j ]

Experiment: choose a pixel randomly

T -> XT: gray level of T

R -> XR: gray level of R

Compute MI(XR,XT)

Need to estimate associated probabilities

Quantized RV: histograms

hT (i ) = # { pix : T ( pix) = xi }

hR (i ) = # { pix : R ( pix) = xi }

hR ,T (i, j ) =# { pix : R ( pix) = xi , T ( pix) = x j }

Probabilities

h (i )
p[ X T = xi ] = T
N pix

hR (i )
p[ X R = xi ] =
N pix

hT (i, j )
p[ X T = xi , X R = x j ] =
N pix

Domain of trivial histogram is {0,1,,255}

Trivial histogram gives rise to a discontinuos

cost function

Image is sampling of an continuous image

Domain of continuous histogram is [0,255]

Estimate (a discretized version of) the

continuous histogram and relative probability

 [Parzen, 1962]
x2
Gaussian window ( x) = e
Continuous joint histogram

1
R( x) T ( x)
hR ,T ( , ) = 2
dx

k
k

Joint probability
pR ,T ( , ) =

hR ,T ( , )

h
', '

R ,T

( ' , ' )

Mutual information of the two images

MI ( R, T ) = pR ,T ( , ) log

pR ,T ( , )
pR ( ) pT ( )

Cost function = -MI(R,T)

Extremely robust

Computing the histogram is overly expensive

Straightforward implementation
for cycle over gray level bins
look for pixels contributing to the given bin

Take advantage of matlab vectorial operations

for cycle over pixels
compute contribution of given pixels to different

bins with vectorial operations

is numerically compactly supported

Computational cost of mutual information

64x64

128x128

256x256

512x512

LS

0.0468

0.1248

0.3588

1.56

MI

1.1544

4.4304

17.7061

72.0101

Histogram 1.1856

4.1808

16.6765

65.9104

(the computation also takes into account

computing the Frchet derivative)

[Bovik, Wang 02]

Consider three discrepancies in images

Luminance (mean)
Contrast (variance)
Correlation

Combine the three components in one index

measuring the similarity between R and T

T,R images
1
T =
N

1
R=
N

Ti
i

1
2
s =
(Ti T )

N 1 i
2
T

sT , R

1
2
s =
( Ri R )

N 1 i
2
R

1
=
(Ti T )( Ri R )

N 1 i

Luminance
Contrast
Correlation

2 R T + c1
l ( R, T ) = 2
R + T 2 + c1
2 sR sT + c2
c ( R, T ) = 2
sR + sT2 + c2

2 sR ,T + c3
s ( R, T ) =
sR sT + c3

Structural Similarity Index

SSIM ( R, T ) = l ( R, T )c( R, T ) s( R, T )

Simplification ( c2 = c3 / 2 )
2 sR ,T + c2
2 R T + c1
SSIM ( R, T ) = 2
=
2
2
2
R + T + c1 sR + sT + c2
= S1 ( R, T ) S2 ( R, T )

S1(T,R) and S2(T,R) are not metrics, but

We set
d1 (T , R) = 1 S1 (T , R)
d 2 (T , R) = 1 S 2 (T , R)

d1(T,R) and d2(T,R) are metrics

[Brunet, Vrscay, Wang 12]

Straightforward calculation yields

2

d1 (T , R ) =
2

T R + c1
2

T + R + c1

( T 0 = T T , R0 = R R )

T R
0

d 2 (T , R) =
2

0 2

0 2

+ R

+ c2

0 2

+ c2

Combine d1 and d2 to obtain a class of metrics

d (T , R) = (1d1 ( R, T ) p + 2 d 2 ( R, T ) p )1/ p

Parameters p1 and 1>0 and 2>0

If d1 and d2 are metrics then d is a metric

Functional analysis norm, generalizing Sobolev

norms
Wavelet decomposition of T

T =
j

t = T

j,

(Tj: component at the frequency band [2j,2j+1])

Besov norm
T

B2s , p

= ( 2
j

j (1 / p 1 / 2 )

js

2 Tj

p 1/ p
l2

Combine Besov norm with divisive

renormalization

Wavelet decomposition of T and R

Generalized SSI distance

2

T
R

j
j

d (T , R ) = j
2
2
Tj + R j
j

p / 2 1/ p

SSI metric
T R +c
1

p
d (T , R ) = 1 2
2
T + R + c1

p/2

T0 R +c
1

+ 1
2
2
T 0 + R 0 + c1

0 2

Generalized SSI metric

T R
j
j

d (T , R) p = j
2
Tj + R j
j

p/2

p/2

Code to test new registration algorithms

Flexible algorithm design allowing to switch

one of the ingredient without modifying the
others

New distance functionals / image models /

transformation classes can be incorporated
easily

Most optimization algorithms need to

compute c

Chain-rule

c( ) = D (T ; x)T ( ( x)) J ( x; )dx

II

III

I. Frchet derivative of the functional

D ( X ; x)

II. Gradient of the image

T ( y )

III. Jacobian (w.r. to the parameter) of the

transformation
J ( x; )

Image module

Transformation module

Distance module

Optimization is dealt with by the Matlab

Optimization Toolbox

Input:
The parameter

Output:
The grid G obtained by deforming the reference

grid via the mapping

The values assumed by the jacobian (w.r. to the
parameter ) of the function at the points of the
reference grid

Input:
A sampled image X
A grid G of arbitrary points (representing the

center of the deformed pixels)

Output:
An interpolated (or extrapolated) value of X at the

points of G
A value of X at the points of G

Input
An arbitrary image X sampled at the points of the

reference grid

Output:
The value of the cost function c(X)
The value at the points of the reference grid of

(the Frchet derivative of the cost functional at X)

To evaluate c()
Call the mapping module to obtain the deformed

grid G
Call the image model module to sample T at the
grid G -> T = deformed image sampled at the
reference grid
Call the distance module to evaluate c() = d(T;R).

To evaluate c
Call the mapping module and evaluate J (x; )

at the points of the reference grid

Call the image module and evaluate T at the
points of the deformed grid
Call the distance module and evaluate D (T,x) at
the points of the reference grid
Evaluate the integral by a chosen quadrature rule

Flexible algorithm allowing to switch one of

the ingredient without modifying the others

New distance functionals / image models /

transformation classes can be incorporated
easily

Idea can be extended to the computation of

the Hessian

REFERENCE IMAGE

TEMPLATE IMAGE

R obtained from T by applying deformation with Photoshop

No noise
Distance: LS

No noise
Distance: SSIM
p=2

No noise
Distance: GSSIM
p=2

Gaussian noise
Distance: LS

Gaussian noise
Distance: SSIM
p=2

Gaussian noise
Distance: GSSIM
p=2

Computational cost: evaluation of the cost

functional
64 x 64

128x128 256x256 512x512

LS

0.2815

0.35938

0,73438

3.0156

SSIM

0.23438

0.67188

1.8906

6.5625

G-SSIM

1.125

1.5313

2.9531

12.875

MI(*)

2.106

8.0965

31.902

134.41

(*) computed on a machine twice as fast

REFERENCE IMAGE

TEMPLATE IMAGE

CT studies, of the same patient by different CT machines

Quality of the results

Unreg.

LS

SSIM

G-SSIM

Pearson 0.245

0.858

0.872

0.926

Overlap

0.492

0.992

0.993

0.992

M1

0.47

0.833

0.846

0.85

M2

0.253

0.868

0.861

0.933