Lecture01 Intro Probability Theory

EE800 Stochastic Systems
Welcome and Introduction
Dr. Muhammad Usman Ilyas

School of Electrical Engineering & Computer Science (SEECS)
National University of Sciences & Technology (NUST)
Copyright Syed Ali Khayam 2009
Course Information
Lecture Timings:
My Office:
Room # A-312
Office Hours
Tuesday: 5:00pm-6:50pm, IAEC CR# 20

Thursday: 6:00pm-7:50pm, IAEC CR# 20
Tuesday, 4:00-4:30pm, or by appointment.

usman.ilyas@seecs.edu.pk
The course will be managed through LMS and facebook
NUST LMS: www.lms.nust.edu.pk

Facebook group: https://www.facebook.com/groups/839067969546757/
Timetable - MS EE-6 (Telecom & Comp Networks)

MS EE(Digital System and Signal Processing)-7
First Semester (7 Sep 2015 - 15 Jan 2016)
TIME / DAYS
Monday
Tuesday
5:00pm-5:50pm
Advanced Digital
System Design
Stochastic
Systems
IAEC (CR#20)
Wednesday
Advanced Digital
System Design
6:00pm-6:50pm
7:00pm-7:50pm
8:00pm-8:50pm
Advanced Digital
Communication
Systems
IAEC Lec Hall
Library/Make-up
Class/Seminar
Advanced Digital
Signal Processing
Library/Make-up
Class/Seminar
Thursday
Friday
Advanced Digital
Communication
Systems
IAEC Lec Hall
Advanced Digital Signal

Processing
Library/Make-up
Class/Seminar
Stochastic Systems
IAEC (CR#20)
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Saturday
Library/Make-up
Class/Seminar
MS EE(Telecommnication & Computer Networks)-7

TIME / DAYS
5:00pm-5:50pm
6:00pm-6:50pm
7:00pm-7:50pm
8:00pm-8:50pm
Monday
Tuesday
Adv. Computer
Networks
Stochastic
Systems
IAEC (CR#20)
Advanced Digital
Communication
Systems
IAEC Lec Hall
EMC/EMI
RIMMS CR#22
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Wednesday
Adv. Computer
Networks
EMC/EMI
RIMMS CR#22
Library/Make-up
Class/Seminar
Thursday
Friday
Saturday
Library/Make-up
Class/Seminar
Library/Make-up
Class/Seminar
Advanced Digital
Communication
Systems
IAEC Lec Hall
Stochastic Systems
IAEC (CR#20)
Library/Make-up
Class/Seminar
Textbooks
Probability & Random Processes

for Electrical Engineers, 2nd or 3rd
ed.
Albert Leon-Garcia
Introduction to Probability Models,

9th ed.
Sheldon M. Ross
Elements of Information Theory
Thomas M. Cover and Joy

A. Thomas
Chaos Theory Tamed
Garnett P. Williams
Course Outline
Syllabus
Introduction to Probability Theory

Random Variables
Limits and Inequalities
Central Limit Theorem
Application Area: Information Theory
Stochastic Processes
Prediction and Estimation
Markov Chain
Counting processes, queueing theory (time permitting)
Application Area: Chaos Theory
Grading
(subject to change)
Final Exam:
Midterm Exam:
Quizzes:
Homework Assignments:
50%
30%
10%
10%
Policies
Quizzes will be unannounced
Late homework submissions will be accepted for up to 24 hours

with a 50% penalty of the total.
I will take strong disciplinary action in cases of plagiarism or

cheating in exams, homework or quizzes. There are no second
chances if plagiarism is proven.
Attendance:
Will be taken at any time during class.
The current rules of the school will be followed (min. 75%
attendance requirement).
Boilerplate Disclaimers
About your 20s (or 30s)

About graduate school
About questions
About cheating and plagiarism
About attendance
About history
What will we cover in this lecture?
This lecture is intended to be an introduction to elementary

probability theory
We will cover:
Random Experiments and Random Variables

Axioms of Probability
Mutual Exclusivity
Conditional Probability
Independence
Law of Total Probability
Bayes Theorem
Definition of Probability
Probability:
1 : the quality or state of being possible
2 : something (as an event or circumstance) that is possible
3 : the ratio of the number of outcomes in an exhaustive set of
equally likely outcomes that produce a given event to the
total number of possible outcomes, the chance that a given
event will occur
We will revisit these definitions in a little bit
10
Definition of a Random Experiment
A random experiment comprises of:
A procedure
An outcome
Procedure
(e.g., flipping a coin)
Outcome
(e.g., the value
observed [head, tail] after
flipping the coin)
Sample Space
(Set of All Possible
Outcomes)
11
Definition of a Random Experiment:

Outcomes, Events and the Sample Space
An outcome cannot be further decomposed into other outcomes

{s1 = the value 1}, , {s6 = the value 6}
An event is a set of outcomes that are of interest to us

A = {s: such that s is an even number}
The sample space is the set of all possible outcomes, called S

= {1, 2, 3, 4, 5, 6}
12

s1
s5
s6
s4
s2
s3
13

Example of a Random Experiment:
Experiment:
Sample space:
Roll a fair dice once and record the number of dots

on the top face
= {1, 2, 3, 4, 5, 6}
Events:
= the outcome is even = {2, 4, 6}

= the outcome is greater than 4 = {5, 6}
14
Axioms of Probability
Probability of any event A is non-negative:

Pr{} 0
The probability that an outcome belongs to the sample

space is 1:
Pr{} = 1
The probability of the union of mutually exclusive events

is equal to the sum of their probabilities:
If,
1 2 = ,
Then,
Pr{1 2} = Pr{1} + Pr{2}

15
Mutual Exclusivity
Are 1 and 2 mutually exclusive?

For mutually exclusive events 1, 2 , we have:
A1
s5
s1
s6
A2
s2
s4
s3
S
Find Pr{1 2} and Pr{1} + Pr{2} in the fair dice
example
16
Mutual Exclusivity
Discarding the condition of exclusivity, in general, we have:
Pr{1 2} =? ?
s5
s1
s6
s4
s2
s3
S
17
Mutual Exclusivity
Discarding the condition of exclusivity, in general, we have:
Pr{1 2} = Pr{1} + Pr{2} Pr{1 2}

s5
s1
s6
s4
s2
s3
S
18
Given that event B has already occurred, what is the probability

that event A will occur?
Given that event B has already occurred, reduces the sample
space of A
s5
s1
Event B has
already occurred
s6
s2
s4
s3
=> s2, s4, s3

cannot occur
s5
s1
s6
s2
s4
s3
19
Given that event has already occurred, we define a new

conditional sample space that only contains s outcomes
The new event space for is the intersection of and :
Event space | =
s5
s1
s6
s2
s4
Event B has
already
occurred
s3
Whats missing here?

s5
s1
s6
s4
s2
s3
= {1 , 5 , 6 , }
| = = {6 }
20
The probability of an event in the conditional sample space is:
Pr
s5
s1
s4
s3
s5
s1
s6
s2
Pr
Pr{|} =
Pr{}
Pr{6 }
1/6
1
=
=
=
Pr{}
3/6
3
Event has
already
occurred
s6
s4
s2
s3
= {1, 5, 6}
21
Independence
Two events are independent if they do not provide any

information about each other:
(|) = ()
In other words, the fact that B has already happened does not
affect the probability of As outcomes
Implications:
(|) = ()

= ()
()
( ) = () ()
22
Independence: Example
Are events and independent?
Assume that all outcomes are equally likely
s2
s6
s4
s5
s1
s3
S
23
Check if
( ) = () ()
s2
s6
s4
s5
s1
s3
S
24
Are events A and C independent?

1
Pr{ } = Pr{5} =
6
3 2
1
Pr Pr = =
6 6
6
Yes!
s2
s6
s4
s5
s1
s3
S
25
Assume that all outcomes are equally likely
s2
s6
s4
s5
s1
s3
S
26

1
Pr{ } = Pr{5} =
6
3 3 1
Pr{}Pr{} = =
6 6 4
No!
s2
s6
s4
s5
s1
s3
S
27
Mutual Exclusivity and Independence

Experiment:
Roll a fair dice twice and record the dots on the top face:
= {(1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),
(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),
(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }
28

Define three events:
1 = first roll gives an odd number
2 = second roll gives an odd number
= the sum of the two rolls is odd
Find the probability of using probability of 1 and 2
29

A2
S={
(1,1), (1,2), (1,3), (1,4), (1,5), (1,6),

(2,1), (2,2), (2,3), (2,4), (2,5), (2,6),
(3,1), (3,2), (3,3), (3,4), (3,5), (3,6),
A1
(4,1), (4,2), (4,3), (4,4), (4,5), (4,6),

(5,1), (5,2), (5,3), (5,4), (5,5), (5,6),
(6,1), (6,2), (6,3), (6,4), (6,5), (6,6) }
30
31
32
33
34
35
36
37
Recap
1.
Outcomes, events and sample space:
2.
For mutually exclusive events A1, A2,, AN, we have:
3.
In general, we have:
38
Recap
4.
Conditional probability reduces the sample space:
5.
Two events A and B are independent only if
6.
For independent events:
39
Four Rules of Thumb

1. Whenever you see two events which have an OR relationship (i.e., event A or
event B), their joint event will be their union, { }
Example: On a binary channel, find the probability of error?
An error occurs when
A: a 0 is transmitted and a 1 is received OR
B: a 1 is transmitted and a 0 is received
Thus probability of error is: Pr{ }
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
40
Four Rules of Thumb

2. Whenever you see two events which have an AND relationship (i.e., both
event A and event B), their joint event will be their intersection, { }
Example: On a binary channel, find the probability that a 0 is transmitted and a
1 is received?
A: a 0 is transmitted AND
B: a 1 is received
Thus probability of above event is: Pr{ }
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
41
Four Rules of Thumb

3. Whenever you see two events which have an OR relationship (i.e., ), check if they
are mutually exclusive. If so, set
Example: On a binary channel, find the probability of error?
A: a 0 is transmitted and a 1 is received OR
B: a 1 is transmitted and a 0 is received
Thus probability of error is:
Are A and B are mutually exclusive?
YES!
A and B are mutually exclusive; transmission of a 0 precludes the possibility of
transmission of a 1, and vice versa. Therefore, we can set
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
42
Four Rules of Thumb

4. Whenever you see two events which have an AND relationship (i.e., ), check if
they are independent. If so, set Pr{ } = Pr{}Pr{}
Example: On a binary channel, find the probability that a 0 is transmitted and a 1 is
received?
A: a 0 is transmitted AND
B: a 1 is received
Probability of above event is: Pr{ }
Are A and B independent?
No.
Pr{R0|T0}
T0
T1
R0
Pr{R1|T1}
R1
1 01
Pr = Pr Pr = 01 =
2
2
1 1 1
Pr Pr = =
2 2 4
43
Total Probability
1, 2, , form a partition of a sample space we

have:
= 1 2
= ,
B2
B1
s1
B3
B4
s2
s5
s4
s6
s3
44
Total Probability
If 1, 2, , form a mutually exclusive partition:

What does this imply?
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
45
Total Probability

1 2 = and 1 2 =
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
46
Total Probability
If B1, B2,, BN form a mutually exclusive partition:

1 2 . . = and 1 2 =
How to express A in term of Bi?
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
47
Total Probability

1 2 . . = and 1 2 =
= ( 1) ( 2) ( )
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
48
Total Probability

1 2 . . = and 1 2 =
= ( 1) ( 2) ( )

What is the probability of A?
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
49
Total Probability

1 2 . . = and 1 2 =
= ( 1) ( 2) ( )
What is the probability of A?

Pr{ }
Pr{} = Pr{ 1} + Pr{ 2} + +
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
50
Total Probability
Using the definition of conditional probability:

Pr } = Pr{ } / Pr{ }
=> Pr = Pr } Pr{ }
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
51
The Law of Total Probability

The Law of Total Probability states:
If 1, 2, , form a partition then for any event
Pr{} = Pr{|1} Pr{1} + Pr{|2} Pr{2} + + Pr{| } Pr{ }
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
52
Bayes Theorem
Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular event, the
so-called inverse probability.
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
53
Bayes Theorem
Based on the Law of Total Probability, Thomas Bayes decided to

look at the probability of a partition given a particular event
Pr
Pr =
Pr
Pr = Pr Pr
Pr Pr
Pr =
Pr
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
54
Bayes Theorem
Pr Pr
Pr =
Pr{}
From the Law of Total Probability, we have:
Pr{} = Pr{|1} Pr{1} + Pr{|2} Pr{2} + + Pr{|} Pr{}
Bayes Rule
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
55
Bayes Theorem
Think of the event A as making a certain observation or taking a

certain value of a measurement.
B2
s1
B1
s5
A
B4
s2
s4
s6
B3
s3
56
Bayes Theorem
Pr | Pr

Pr =
=
=1 Pr | Pr
Pr{ |}
posterior / a posteriori probability, i.e. the probability

you know after making an observation or taking a
measurement
Pr{| }
likelihood probability
Pr{ }
prior / a priori probability, i.e. the probability you knew

before taking into consideration any observations or
measurements
Pr{}
evidence probability, i.e. the probability of obtaining a

certain observation or measurement
57
A Fishy Problem
Asad is a fisherman.
Asad is an educated person.
Asad builds a fishing robot that will do his work for him.
B2
B3
B1
B4
58
A Fishy Problem
Question: If Asads robot catches a fish that is detected red, what
species is it, 1 , 2 , 3 or 4 ?
Answer: It could be any of four species in the sea.
B2
B3
B1
B4
59
A Fishy Problem
Lets change the question:
What is the probability that a red fish is a species B1, B2,
B3 and B4?
B2
B3
B1
B4
60
A Fishy Problem
B2
B3
B1
B4
61

Lecture01 Intro Probability Theory

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

Lecture01 Intro Probability Theory

Diunggah oleh

Hak Cipta:

Format Tersedia

EE800 Stochastic Systems

Welcome and Introduction

Dr. Muhammad Usman Ilyas

Copyright Syed Ali Khayam 2009

Tuesday: 5:00pm-6:50pm, IAEC CR# 20

Tuesday, 4:00-4:30pm, or by appointment.

The course will be managed through LMS and facebook

NUST LMS: www.lms.nust.edu.pk

Copyright Syed Ali Khayam 2008

Timetable - MS EE-6 (Telecom & Comp Networks)

Advanced Digital Signal

MS EE(Telecommnication & Computer Networks)-7

Copyright Syed Ali Khayam 2008

Copyright Syed Ali Khayam 2008

Probability & Random Processes

Introduction to Probability Models,

Elements of Information Theory

Thomas M. Cover and Joy

Chaos Theory Tamed

Introduction to Probability Theory

Application Area: Information Theory

Application Area: Chaos Theory

Copyright Syed Ali Khayam 2008

Copyright Syed Ali Khayam 2008

Quizzes will be unannounced

Late homework submissions will be accepted for up to 24 hours

I will take strong disciplinary action in cases of plagiarism or

Copyright Syed Ali Khayam 2008

About your 20s (or 30s)

Copyright Syed Ali Khayam 2008

What will we cover in this lecture?

This lecture is intended to be an introduction to elementary

Random Experiments and Random Variables

Copyright Syed Ali Khayam 2008

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment

A random experiment comprises of:

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment:

An outcome cannot be further decomposed into other outcomes

An event is a set of outcomes that are of interest to us

The sample space is the set of all possible outcomes, called S

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment:

Copyright Syed Ali Khayam 2008

Definition of a Random Experiment:

Example of a Random Experiment:

Roll a fair dice once and record the number of dots

= the outcome is even = {2, 4, 6}

Copyright Syed Ali Khayam 2008

Probability of any event A is non-negative:

The probability that an outcome belongs to the sample

The probability of the union of mutually exclusive events

Copyright Syed Ali Khayam 2008

Pr{1 2} = Pr{1} + Pr{2}

Are 1 and 2 mutually exclusive?

Discarding the condition of exclusivity, in general, we have:

Discarding the condition of exclusivity, in general, we have:

Pr{1 2} = Pr{1} + Pr{2} Pr{1 2}

Given that event B has already occurred, what is the probability

Copyright Syed Ali Khayam 2008

=> s2, s4, s3

Given that event has already occurred, we define a new