!

!
ENCS%6161/4%*%Probability%and%Stochastic%Processes%–%Fall%2015%
!
Instructor:!Dr.!Hassan!Rivaz,!EV5.154,!hrivaz@ece.concordia.ca!
http://users.encs.concordia.ca/~hrivaz/!!
Lectures:!Hall!540,!Mondays!5:45PM!to!8:15PM!
Office%hours:!Fridays!9:30AM!to!12noon.!
Textbook:!Alberto!LeonLGarcia,!Probability,!Statistics,!and!Random!Processes!For!Electrical!
Engineering,!3rd.!Ed.!Prentice!Hall,!2008.!
Syllabus:!
1. Introduction:!Motivation!and!Overview.!Axioms!of!Probability.!Independent!Events.!Total!
Probability.!Conditional!Probability.!Bay’s!Theorem.!Bernoulli!Trials!(Chapters!1!and!2).!
2. Concepts!of!Discrete!and!Continuous!Random!Variables!(RV).!Probability!Mass!Functions!(PMF)!
and!Probability!Density!Functions!(PDF),!Cumulative!Distribution!Functions!(CDF).!Conditional!
Distribution!and!Probability!Density!Functions!of!Random!Variables!(Chapters!3!and!4).!
3. Multiple!random!variables,!joint!PDF,!conditional!probability!and!conditional!expectation,!
correlation!and!covariance,!joint!Gaussian!random!variables!(Chapters!5!and!6).!
4. Vector!Random!Variables.!Sums!of!Random!Variables.!LongLTerm!Averages.!Central!Limit!
Theorem,!Confidence!Intervals!(Chapters!7,8)!
5. Random!Processes,!Mean,!AutoL!correlation!and!CrossLcorrelation,!Autocovariance,!Ergodicity,!
Stationarity!(Chapter!9).!
6. Power!Spectral!Density,!Response!of!linear!systems!to!random!signals!(Chapter!10).!
7. Markov!Chains!(Chapter!11)!

Midterm:!Monday!Oct!26,!in!class.!No!makeLup!tests!offered.!
Grading%Schemes:%
1. Assignments!6%,!Midterm!30%,!Final!64%!
2. Assignments!6%,!Midterm!0%,!Final!94%!

!The!better!mark!will!be!used!to!calculate!the!final!grade.!
Assignments:!
Assignment!1:!posted!on!Mon!Sept!28.!Due!on!Mon!Oct!5!at!the!beginning!of!the!lecture.!!
Assignment!2:!posted!on!Mon!Nov!16.!Due!on!Mon!Nov!23!at!the!beginning!of!the!lecture.!!
Violation!of!the!Academic!Code!of!Conduct!in!any!form!will!be!severely!dealt!with!(see!
http://www.concordia.ca/students/academicLintegrity.html).!You!must!sign!the!following!
form!and!hand!it!in!with!assignment!1:!
http://www.concordia.ca/content/dam/encs/docs/ExpectationsLofLOriginalityLFeb14L2012.pdf!!
 Most%slides%are%graciously%
provided%by%Dr.%Dongyu%Qiu%
 Importance of probability
•  Artificial intelligence (Google, Facebook,
self-driving cars, face recognition)

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz.

Importance of Probability

ultrasound elastography

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz.

9/14/2015 Medical Image Processing

ELEC 498 (Undergraduate), ELEC 691 (Graduate)

Medical Image Processing
WINTER 2016

Instructor: Hassan Rivaz

Course This course covers techniques used in the processing and analysis of medical images. Projects and
description: assignments will provide students experience working with actual medical imaging data.
Assignments: Two assignments + one final project in MATAB or C.
Grading:
Undergraduate (3 credits) Graduate (4 credits)
Assignments 10 10
Project 20 40
Midterm exam 20 20
Final exam 50 30
Outline:
Week Topic
1 Logistics, introduction to X-ray and CT imaging
2 Introduction to Magnetic Resonance (MRI) and ultrasound imaging
3 Images in Matlab, convolution, aliasing in medical images
4 Denoising techniques in medical imaging
5 Machine learning in medical imaging
6 Review and midterm
7 Introduction to medical image registration
8 Introduction to similarity metrics (sum of squared differences, normalized cross correlation)
9 Correlation ratio, joint entropy and Mutual Information (MI)
10 Case study 1: optical flow and ultrasound elastography
11 Case study 1: Bayes rule and regularization in ultrasound elastography
12 Case study 2: rigid and deformable registration of ultrasound and MR
13 Review
Undergraduate students: ELEC 364 Signals and Systems or instructor's permission.
Prerequisites:
Graduate students: none.

file:///Users/hrivaz/Dropbox/Teaching_CV/01_courses/new_medical_imag/application_slot_course/index.html 1/2
 Chapter 2
Basic Concepts of Probability
Theory
ENCS6161 - Probability and Stochastic
Processes
Concordia University
Specifying Random Experiments
Examples of random experiments: tossing a coin,
rolling a dice, the lifetime of a harddisk.
Sample space: the set of all possible outcomes of a
random experiment.
Sample point: an element of the sample space S
Examples: S = {H, T }
S = {1, 2, 3, 4, 5, 6}
S = {t|1 < t < 10}
Event: a subset of a sample space A ⊆ S
A = {H}
A = {2, 4, 6}
A=∅

ENCS6161 – p.1/16
The Axioms of Probability
A probability measure is a set function P (·) that
satisfies the following axioms.
1. P (A) ≥ 0
2. P (S) = 1
3. If A ∩ B = ∅, then P (A ∪ B) = P (A) + P (B)
4. If A1 , A2 , · · · are events s.t Ai ∩ Aj = ∅ for all i ̸= j
! ∞ " #∞
then P ∪k=1 Ak = k=1 P (Ak )
Corollary 1: P (Ac ) = 1 − P (A)
Corollary 2: P (A) ≤ 1 ∀A
Corollary 3: P (∅) = 0

ENCS6161 – p.2/16
The Axioms of Probability
Corollary 4: If A1 · · · An are mutually exclusive, i.e.
Ai ∩ Aj = ∅, ∀i ̸= j then P (∪nk=1 Ak ) = nk=1 P (Ak )
#

Corollary 5: P (A ∪ B) = P (A) + P (B) − P (A ∩ B)

A B

A∩B

Union bound: P (A ∪ B) ≤ P (A) + P (B)

ENCS6161 – p.3/16
 Mutually Exclusive Events

•  Events A and B are mutually exclusive because they

share no common outcomes.
•  They cannot happen together
•  Symbolically, A Π B = Ø

•  Examples of mutually exclusive events:

Throwing a die, 1 and 2 coming together
Turning left and right
•  Examples of NOT mutually exclusive events:
Turning left and scratching your head
Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
 Mutually exclusive

Mutually exclusive NOT mutually exclusive

Pictures from
http://www.mathsisfun.com/data/probability-events-mutually-exclusive.html

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
 Counting Techniques
•  There are three special rules, or counting
techniques, used to determine the number of
outcomes in events.
•  They are :
1.  Multiplication rule
2.  Permutation rule
3.  Combination rule
•  Each has its special purpose that must be
applied properly – the right tool for the right
job.

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
 Counting – Multiplication Rule
•  Multiplication rule:
–  Let an operation consist of k steps and there
are
•  n1 ways of completing step 1,
•  n2 ways of completing step 2, … and
•  nk ways of completing step k.
–  Then, the total number of ways to perform k
steps is:
•  n1 · n2 · … · nk

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Counting Sample Points
Permutation: n distinct objects, how many ways can
we arrange them?
n · (n − 1) · (n − 2) · · · 2 · 1 = n!
Selection with order: select k objects from n ≥ k
objects
n!
n · (n − 1) · · · (n − k + 1) = = n Pk
(n − k)!
Combination (selection without order): select k
objects from n ≥ k objects $ %
n n! n
Ck = =
(n − k)!k! k

ENCS6161 – p.4/16
Counting Sample Points
Example: choose a president and a treasurer from 50
students
1. no restrictions
2. A will serve only if he is president
3. B & C will serve together or not at all
4. D & E will not serve together

ENCS6161 – p.5/16
 Example 2-7 : Hospital Schedule
•  In a hospital, a operating room needs to schedule
three knee surgeries and two hip surgeries in a
day. The knee surgery is denoted as k and the hip
as h.
–  How many sequences are there?
Since there are 2 identical hip surgeries and 3
identical knee surgeries, then
5! 5 ⋅ 4 ⋅ 3!
= = 10
2!⋅3! 2 ⋅1⋅ 3!
–  What is the set of sequences?

{kkkhh, kkhkh, kkhhk, khkkh, khkhk, khhkk, hkkkh,

hkkhk, hkhkk, hhkkk}

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
 Example 2-7 : Hospital Schedule
•  In a hospital, a operating room needs to schedule
three knee surgeries and two hip surgeries in a
day. The knee surgery is denoted as k and the hip
as h.
–  How many sequences are there?
Since there are 2 identical hip surgeries and 3
identical knee surgeries, then
5! 5 ⋅ 4 ⋅ 3!
= = 10
2!⋅3! 2 ⋅1⋅ 3!
–  What is the set of sequences?

{kkkhh, kkhkh, kkhhk, khkkh, khkhk, khhkk, hkkkh,

hkkhk, hkhkk, hhkkk}

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Example 2-8: Sampling w/o Replacement-1

•  A bin of 50 parts contains 3 defectives and

47 non-defective parts. A sample of 6 parts
is selected from the 50 without
replacement. How many samples of size 6
contain exactly 2 defective parts?

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Example 2-8: Sampling w/o Replacement-3

•  How many ways are there to obtain:

–  2 from 3 defectives, and
–  4 from 47 non-defectives?

C23C447 = 3⋅178,365 = 535,095 different ways

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz. All rights reserved.
Conditional Probability
The conditional prob. of A given B has occurred is
defined by :

P (A ∩ B)
P (A|B) = , P (B) > 0
P (B)

Similarly,

P (A ∩ B)
P (B|A) = , P (A) > 0
P (A)

So

P (A ∩ B) = P (B)P (A|B) = P (A)P (B|A)

ENCS6161 – p.6/16
Bayes’ Rule
If B1 · · · Bn are mutually exclusive and ∪Bi = S, we
call these sets a partition of S.
Theorem of total probability
For any event A, if B1 , · · · , Bn is a partition of S,

P (A) = P (A ∩ B1 ) + P (A ∩ B2 ) + · · · + P (A ∩ Bn )
= P (A|B1 )P (B1 ) + P (A|B2 )P (B2 ) + · · · + P (A|Bn )P (Bn )

Bayes’ Rule
Let B1 , · · · , Bn be a partition of S , and P (A) ≥ 0, then
P (A ∩ Bj ) P (A|Bj )P (Bj )
P (Bj |A) = = n
#
P (A) k=1 P (A|Bk )P (Bk )

ENCS6161 – p.7/16
Bayes’ Rule
Example: 2% of people have on type of blood
disease. If a person has the disease and take a
blood test, with 96% probability, the result is positive
and with 4% probability, the result is negtive. If a
person without the disease takes a blood test, then
94% negative and 6% positive. Find

P (have disease|positive)

ENCS6161 – p.11/16
Bayes’ Rule
Solution: Let
D: a person has the disease D′ : no disease
B : blood test positive B ′ : blood test negative
Then P (D) = 2% and
P (B|D) = 0.96 P (B ′ |D) = 0.04
P (B|D′ ) = 0.06 P (B ′ |D′ ) = 0.94
Apply Bayes Rule:

P (D|B) = 0.246 = 24.6%

ENCS6161 – p.12/16
Bayes’ Rule
We can also use tree diagram.

0.96 B 0.02 ∗ 0.96 = 0.0192 (D ∩ B)

D
0.02 0.078 B
0.04 B’ 0.02 ∗ 0.04 = 0.0008 (D ∩ B′)

0.06 B 0.98 ∗ 0.06 = 0.0588 (D ′ ∩ B)

0.98 0.922 B’

D’ ′ ′
0.94 B’ 0.98 ∗ 0.94 = 0.9212 (D ∩ B )

P (D ∩ B) 0.0192
P (D|B) = = = 0.246
P (B) 0.078

ENCS6161 – p.13/16
Independence of Events
A and B are independent iff

P (A ∩ B) = P (A)P (B)

If A and B are independent

P (A ∩ B)
P (A|B) =
P (B)
P (A)P (B)
=
P (B)
= P (A)

Similarly
P (B|A) = P (B)

ENCS6161 – p.15/16
Independence of Events
Events A1 , A2 , · · · , An are independent if

P (Ai1 ∩ Ai2 · · · ∩ Aik ) = P (Ai1 )P (Ai2 ) · · · P (Aik )

where1 ≤ i1 < i2 · · · < ik ≤ n

&n
A special case P (A1 ∩ · · · ∩ An ) = i=1 P (An )
In general if events are not necessarily independent

P (A1 ∩ A2 · · · ∩ An ) = P (A1 )P (A2 |A1 )P (A3 |A1 ∩ A2 ) · · ·

· · · P (An |A1 ∩ · · · ∩ An−1 )

ENCS6161 – p.16/16
 Conditional Probability
•  P(B | A) is the probability of event B
occurring, given that event A has already
occurred.
•  Example:
S={1,2...6} in throwing a die
B={2}. P(B) = 1/6
A={1}. P(B| A) = ?

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz.

 Conditional Probability
•  P(B | A) is the probability of event B
occurring, given that event A has already
occurred.
•  Example:
S={1,2...6} in throwing a die
B={2}. P(B) = 1/6
A={1}. P(B| A) = ?
A={2,4,6}. P(B| A) = ?

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz.

Example 2-22
There are 4 probabilities conditioned on flaws in the
below table. Parts Classified
Surface Flaws
Defective Total
Yes (F ) No (F' )
Yes (D ) 10 18 28
No (D' ) 30 342 372
Total 40 360 400

P(F ) = 40 400 and P(D) = 28 400

P(D | F ) = P(D ∩ F ) P(F ) = 400
10 40 = 10
400 40

( ) (
P D ' | F = P D '∩ F ) P (F ) = 30
400
40
400
= 30
40

P ( D | F ') = P ( D ∩ F ') P ( F ') = 18

400
360
400
18
= 360
P ( D ' | F ') = P ( D '∩ F ') P ( F ') = 342
400
360
400
= 342
360

Copyright © 2015 John Wiley & Sons, Inc and H Rivaz.

 Sec 2-
Fig_2-14 Copyright © 2015 John Wiley & Sons, Inc and H Rivaz.