Distribution Functions
( ) ( ) ( )
u x 1 + u x 2 + u x 3
() ( )
FX x = 1 / 6
( ) ( ) ( )
+ u x 4 + u x 5 + u x 6
Distribution Functions
A possible distribution function for a continuous random
variable
Probability Density
The derivative of the distribution function is the probability
density function (PDF)
pX x ()d
dx
( ( ))
FX x
()
p X x 0 , < x < + p ( x ) dx = 1
X
x x2
( ) p ( ) d
FX x = X ()
P x1 < X x2 = p X x dx
x1
Expectation and Moments
( )
E X = lim xi = lim ri xi = P X = xi xi .
i=1 N
N N
i=1 i=1
Expectation and Moments
( )
E X = P xi
2
< X xi +
2 ( )
xi xi p X xi x
i=1 i=1
( ) x p ( x ) dx.
In the limit as x approaches zero, E X = X
( ( )) = g ( x ) p ( x ) dx.
Similarly E g X
X
( )
The nth moment of a random variable is E X n = ()
x n p X x dx.
Expectation and Moments
is its mean-squared value (which is the mean of its square, not the
square of its mean).
( )
E X2 = ()
x 2 p X x dx
Expectation and Moments
A central moment of a random variable is the moment of
that random variable after its expected value is subtracted.
( )
E X E X = x E ( X ) ()
n n
p X x dx
( )
= E X E X = x E ( X ) ()
2 2
2
p X x dx
X
Properties of Expectation
() ( )
E a = a , E aX = a E X ( ) , E Xn = E Xn
n
( )
n
F ( , ) = 1
XY
( )
FXY x, y does not decrease if either x or y increases or both increase
( ) ( ) (
FXY , y = FY y and FXY x, = FX x ) ()
Joint Probability Density
p XY ( )
x, y =
2
x y
( ( ))
FXY x, y
( )
p XY x, y 0 , < x < , < y <
y x
p ( x, y ) dx dy = 1
XY ( ) p ( , ) d d
FXY x, y =
XY
pX x =( ) p ( x, y ) dy
XY ( ) p ( x, y ) dx
and pY y = XY
y2 x2
P x1 < X x2 , y1 < Y y2 = p ( x, y ) dx dy
y1 x1
XY
((
E g X ,Y )) = g ( x, y ) p ( x, y ) dx dy
XY
Independent Random Variables
( ) xy p ( x, y ) dx dy = y p ( y ) dy x p ( x ) dx = E ( X ) E (Y )
E XY = XY Y X
Independent Random Variables
Covariance
( ) ( )
E X E X Y E Y
*
XY
( x E ( X ))( y E (Y )) p ( x, y ) dx dy
XY = * *
XY
XY = E ( XY ) E ( X ) E (Y )
* *
( ) ( ) ( ) ( )
If X and Y are independent, XY = E X E Y * E X E Y * = 0
Independent Random Variables
Correlation Coefficient
XY
X E X
= E
( )
Y* E Y*
( )
X Y
XY
=
( ) ( )
x E X y* E Y *
( )
p XY x, y dx dy
X Y
XY =
( ) ( ) ( )=
E XY * E X E Y * XY
XY XY
If X and Y are independent = 0. If they are perfectly positively
correlated = +1 and if they are perfectly negatively correlated
= 1.
Independent Random Variables
Unfortunately, this does not also imply that their correlation is zero.
If their correlation is zero they are said to be orthogonal.
( )
X and Y are "Uncorrelated" E XY = 0
Independent Random Variables
The variance of a sum of random variables X and Y is
2X +Y = 2X + Y2 + 2 XY = 2X + Y2 + 2 XY X Y
( )
then E Z = a0 + ai E X i ( )
i=1
N N N N N
Z2 = ai a j X X = ai2 2X + ai a j X X
i j i i j
i=1 j=1 i=1 i=1 j=1
i j
Independent Random Variables
If Z is simply the sum of the Xs, and the Xs are all independent
of each other, then the variance of the sum is the sum of the
variances.
N
Z2 = 2X
i
i=1
Probability Density of a Sum
of Random Variables
( ) p ( x, y ) dx dy
FZ z = XY
z y
If X and Y are independent, FZ z =( ) p ( ) Y ()
y p X x dx dy
( ) p ( y ) p ( z y ) dy = p ( z ) p ( z )
and it can be shown that p Z z = Y X Y X
The Central Limit Theorem
() () () ()
p Z z = p X z p X z p X z p X
1 2 2 N
(z)
and it can be shown that, under very general conditions, the PDF
of a sum of a large number of independent random variables
with continuous PDFs approaches a limiting shape called the
Gaussian PDF regardless of the shapes of the individual PDFs.
The Central Limit Theorem
The Central Limit Theorem
( )
X = E X and X = E X E X
( )
2
The Central Limit Theorem
1
( )
x x = lim e (
x X )2 /2 2X
X 0
X 2
Correlation
The correlation between two signals is a measure of how similarly
shaped they are. The definition of correlation R12 for two signals
x1 ( t ) and x 2 ( t ) , at least one of which is an energy signal, is the area
under the product of x1 ( t ) and x*2 ( t )
R12 = 1 2 (t ) dt .
( ) *
x t x
T /2
Correlation
For two energy signals notice the similarity of correlation to signal energy.
1 2 (t ) dt
( ) x1 ( t ) dt x 2 ( t ) dt
2 2
R12 = x t x *
E1 = E2 =
T /2 T /2 T /2
both non-zero.
In either case, x1 ( t ) and x 2 ( t ) are orthogonal.
Correlation
x (t ) + y (t ) a x (t ) + z (t )
*
R12 = dt
= a x ( t ) x* ( t ) + x ( t ) z* ( t ) + a y ( t ) x* ( t ) + y ( t ) z* ( t ) dt
= a x ( t ) x* ( t ) dt = aR11
Correlation
Positively Correlated Uncorrelated Random Negatively Correlated
Random Signals with Signals with Random Signals
Zero Mean Zero Mean with Zero Mean
Correlation
Positively Correlated Negatively Correlated
Uncorrelated Sinusoids
Sinusoids with Sinusoids with
with Non-Zero Mean
Non-Zero Mean Non-Zero Mean
Correlation
Let v ( t ) be a power signal, not necessarily real-valued or periodic, but
with a well-defined average signal power
Pv v ( t ) = v ( t ) v* ( t ) 0
2
z* ( t ) = z ( t ) z ( t t d ) = z ( t ) for
*
Time averaging has the properties ,
any t d and a1 z1 ( t ) + a2 z 2 ( t ) = a1 z1 ( t ) + a2 z 2 ( t ) . If v ( t ) and w ( t ) are
power signals, v ( t ) w * ( t ) is the scalar product of v ( t ) and w ( t ) . The
scalar product is a measure of the similarity between two signals.
Correlation
Pz = Pv + a 2 Pw 2aRvw
Now find the value of a that minimizes Pz by differentiating with
respect to a and setting the derivative equal to zero.
R
Pz = 2aPw 2Rvw = 0 a = vw
a Pw
Therefore, to make v and aw as similar as possible (minimizing z)
set a to the correlation of v and w divided by the signal power of w.
If v ( t ) = w ( t ) , a = 1. If v ( t ) = w ( t ) then a = 1.
If Rvw = 0, Pz = Pv + a 2 Pw .
Correlation
The correlation between two energy signals x and y is the area under
the product of x and y* .
Rxy = x ( t ) y* ( t ) dt
R xy ( ) = x ( t ) y *
(t ) dt = x ( t + ) y *
(t ) dt
R xy ( ) = x (t ) y (t ) dt = x (t + ) y (t ) dt
Correlation
The correlation function for two real-valued energy signals is very
similar to the convolution of two real-valued energy signals.
x (t ) y (t ) = x (t ) y ( ) d = x ( ) y (t ) d
R xy ( ) = x ( ) y ( ) d = x ( ) y ( ( )) d = x ( ) y ( )
If the two signals are both periodic and their fundamental periods have
a finite least common period, where T is any integer multiple of that
least common period.
1
R xy ( ) = x ( t ) y*( t ) dt
T T
For real-valued periodic signals this becomes
1
R xy ( ) = x ( t ) y ( t ) dt
T T
Correlation
( jAB / 4 ) ( [ k + 1] [ k 1])
AB
R12 ( ) = sin ( 2 f0 )
FS
T0
2
Correlation
Correlation
R12 ( ) = x (t ) x (t ) dt
1 2
4 3 , 0 < t < 2 +
For 2 < < 0, x1 ( t ) x 2 ( t ) = R12 ( ) = 12 ( 2 + )
0 , otherwise
4 ( 3) , 0 < t <
For 0 < < 2 , x1 ( t ) x 2 ( t ) = 4 3 , < t < 2 + R12 ( ) = 12 + 24 = 12 ( 2 )
0 , otherwise
Correlation
4 ( 3) , 2 < t <
For 2 < < 4 , x1 ( t ) x 2 ( t ) = 4 3 , <t <4 R12 ( ) = 12 ( 2 )
0 , otherwise
4 ( 3) , 2 < t < 4
For 4 < < 6 , x1 ( t ) x 2 ( t ) = R12 ( ) = 12 ( 6 )
0 , otherwise
2 1 0 1 2 3 4 5 6
R12 ( ) 0 12 24 12 0 12 24 12 0
Correlation
Find the correlation function between these two functions.
3 , 2 < t < 0
4 , 0 < t < 4
x1 ( t ) = , x2 (t ) = 3 , 0 < t < 2
0 , otherwise 0 , otherwise
Alternate Solution:
t 2 t + 1 t 1 t + 1 t 1
x1 ( t ) = 4 rect x 2 ( t ) = 3 rect + rect ( t ) =
4
, , x 3 rect rect
2 2 2
2 2
a + b 2t a b 2t
Using rect ( t / a ) rect ( t / b ) = tri tri
2 a + b 2 ab
1 3
R12 ( ) = x1 ( ) x 2 ( ) = 12 3tri tri ( 1) 3tri + tri ( 3)
3 3
Checking some values of :
2 1 0 1 2 3 4 5 6
R12 ( ) 0 12 24 12 0 12 24 12 0
These answers are the same as in the previous solution.
Autocorrelation
A very important special case of correlation is autocorrelation.
Autocorrelation is the correlation of a function with a shifted
version of itself. For energy signals,
R x ( ) = R x x ( ) = x ( t ) x *
(t ) dt
At a shift of zero,
Rx (0) = x (t ) x (t ) dt = x (t )
2
*
dt = Ex
At a shift of zero,
1
R x ( 0 ) = lim x ( t ) dt
2
T T T
x (t ) X( f )
2 2
Ex = dt = df
R x ( t )
F
x ( f )
The signal energy of a signal is the area under the energy spectral density
and is also the value of the autocorrelation at zero shift.
Ex = R x ( 0 ) = ( f ) df
x
Energy Spectral Density
t 3
Find the energy spectral density of x ( t ) = 10 rect .
4
Using x ( f ) = X ( f ) = X ( f ) X* ( f ) ,
2
X ( f ) = 40 sinc ( 4 f ) e j 6 f
X ( f ) X* ( f ) = 40 sinc ( 4 f ) e j 6 f 40 sinc ( 4 f ) e j 6 f
X ( f ) X* ( f ) = 1600 sinc 2 ( 4 f )
Power Spectral Density
Power spectral density (PSD) applies to power signals in the same way
that energy spectral density applies to energy signals. The PSD of a
signal x is conventionally indicated by the notation G x ( f ) whose units
are signal power/Hz. In an LTI system,
G y ( f ) = H ( f ) G x ( f ) = H ( f ) H* ( f ) G x ( f )
2
R x ( )
FS
T
c x [ k ] c x [k]
*
Using T = T0 = 0.01 s,
c x [ k ] = 30 ( j / 2 ) ( [ k + 1] [ k 1]) (1 / 2 ) ( [ k 1000 ] + [ k + 1000 ])
cx [ k ] = j
15
2
( [ k 999 ] + [ k + 1001] [ k 1001] [ k + 999 ])
2
15
c x [ k ] c*x [ k ] = ( [ k 999 ] + [ k + 1001] + [ k 1001] + [ k + 999 ])
2
15 2
R x ( ) = cos (199800 ) + cos ( 200200 )
2
15 2
G( f ) = ( f 99900 ) + ( f + 9900 ) + ( f 100100 ) + ( f + 100100 )
4
Random Processes
A random process maps experimental outcomes into real functions
of time. The collection of time functions is known as an ensemble and
each member of the ensemble is called a sample function. The ensemble
will be represented by the notation v ( t, s ) in which t is time and s is
the sample function.
Random Processes
To simplify notation, let v ( t, s ) become just v ( t ) where it will be
understood from context that v ( t ) is a sample function from a random
process. The mean value of v ( t ) at any arbitrary time t is E ( v ( t )) .
This is an ensemble mean, not a time average. It is the average of
( )
all the sample function values at time t, E v ( t1 ) = V1 . Autocorrelation
( )
is defined by R v ( t1 ,t 2 ) E v ( t1 ) v ( t 2 ) . If V1 and V2 are statistically
independent then R v ( t1 ,t 2 ) = V1V2 . If t1 = t 2 , then V1 = V2 and
( )
R v ( t1 ,t 2 ) = V12 and, in general, R v ( t,t ) = E v2 ( t ) = v2 ( t ), the mean-
squared value of v ( t ) as a function of time.
Random Processes
A generalization of autocorrelation to the relation between two
different random processes is cross - correlation defined by
( )
R vw ( t1 ,t 2 ) E v ( t1 ) w ( t 2 ) . The covariance function is defined by
( ( ) ( ))
Cvw ( t1 ,t 2 ) E v ( t1 ) E v ( t1 ) w ( t1 ) E w ( t1 ) .
g ( vi ( t ) ) g ( vi ( t )) dt
1
= lim
T T
T /2
( )
So, for an ergodic process, g ( vi ( t )) = E g ( v ( t )) . By definition
g ( vi ( t )) is independent of time because it is an average over all time.
It then follows that ensemble averages of ergodic processes are
independent of time. If a random process v ( t ) is ergodic then
( )
E ( v ( t )) = v = mv and E v2 ( t ) = v2 = v2 + mv2 where m and v2 are
the mean and variance of v ( t ) .
Random Processes
For an ergodic random process representing an electrical signal we can
identify some common terms as follows:
Mean value mv is the "DC" component vi ( t ) .
The square of the mean m is the "DC" power vi ( t ) (the power in the
2 2
v
average).
The mean-squared value v2 is the total average power v2i ( t ) .
The variance v2 is the "AC" power (the power in the time-varying part).
The standard deviation v is the RMS value of the time-varying part.
Be sure not to make the common mistake of confusing "the square of
the mean" with "the mean-squared value", which means "the mean of
the square". In general the square of the mean and the mean of the
square are different.
Random Processes
Actually proving that a random process is ergodic is usually very
difficult, if not impossible. A much more common and useful
requirement on a random process that is much easier to prove is that
it be wide - sense stationary ( WSS ) . A random process is wide-sense
stationary when the mean E ( v ( t )) is independent of time and the
autocorrelation function R v ( t1 ,t 2 ) depends only on the time difference
t1 t 2 . So wide-sense stationarity requires E ( v ( t )) = mv and
R v ( t1 ,t 2 ) = R v ( t1 t 2 ) and we usually write autocorrelation functions
with the notation R v ( ) in which = t1 t 2 . So
R v ( ) = E ( v ( t ) v ( t )) = E ( v ( t + ) v ( t ))
and R v ( ) has the properties R v ( ) = R v ( ) , R v ( 0 ) = v2 = mv2 + v2
and R v ( ) R v ( 0 ) .
Random Processes
v
G ( f ) df = R v ( 0 ) = v 2
= P , Gv ( f ) 0 , Gv ( f ) = Gv ( f )
Random Signals
If two random signals v ( t ) and w ( t ) are jointly stationary such that
R vw ( t1 ,t 2 ) = R vw ( t1 t 2 ) and if z ( t ) = v ( t ) w ( t ) , then
R z ( ) = R v ( ) + R w ( ) R vw ( ) + R wv ( )
and
G z ( f ) = G v ( f ) + G w ( f ) G vw ( f ) + G wv ( f )
where R vw ( )
F
G vw ( f ) and G vw ( f ) is cross - spectral density
(also known as "cross power spectral density (CPSD)"). If v ( t ) and w ( t )
are uncorrelated and mv mw = 0, then R vw ( ) = R wv ( ) = 0,
R z ( ) = R v ( ) + R w ( ) , G z ( f ) = G v ( f ) + G w ( f ) and z 2 = v2 + w 2 .
Random Signals
Let z ( t ) = v ( t ) cos ( ct + ) in which v ( t ) is a stationary random signal
and is a random angle independent of v ( t ) and uniformly distributed
over the range . Then
( ) (
R z ( t1 ,t 2 ) = E z ( t1 ) z ( t 2 ) = E v ( t1 ) cos ( ct1 + ) v ( t 2 ) cos ( ct 2 + ) )
( ( () ))
R z ( t1 ,t 2 ) = E v ( t1 ) v ( t 2 ) (1 / 2 ) cos c ( t1 t 2 ) + cos c ( t1 + t 2 ) + 2
E ( v ( t ) v ( t ) cos ( ( t t )))
(t ,t ) = (1 / 2 )
1 2 c 1 2
Rz
+ E ( v ( t ) v ( t ) cos ( ( t + t ) + 2 ))
1 2
1 2 c 1 2
E ( v ( t ) v ( t )) cos ( ( t t ))
1 2 c 1
2
Rz (t ,t ) = (1 / 2 ) + E ( v(t ) v(t )) E ( cos ( (t + t ) + 2 ))
1 2
1 2 c 1 2
=0
( ) (
R z ( t1 ,t 2 ) = (1 / 2 ) E v ( t1 ) v ( t 2 ) cos c ( t1 t 2 ) )
Random Signals
( ) ( )
In R z ( t1 ,t 2 ) = (1 / 2 ) E v ( t1 ) v ( t 2 ) cos c ( t1 t 2 ) since
R v ( t1 ,t 2 ) = R v ( ) we can say that R z ( ) = (1 / 2 ) R v ( ) cos ( c ) .
Then G z ( f ) = (1 / 2 ) G v ( f ) (1 / 2 ) ( f fc ) + ( f + fc )
G z ( f ) = (1 / 4 ) G v ( f fc ) + G v ( f + fc ) .
In general, if v ( t ) and w ( t ) are independent and jointly stationary
and z ( t ) = v ( t ) w ( t ) , then
R z ( ) = R v ( ) R w ( ) and G z ( f ) = G v ( f ) G w ( f )
Random Signals
y (t ) = x (t ) h (t ) = x ( ) h (t ) d .
Every signal in every system has noise on it and may also have interference.
Noise is a random signal occurring naturally and interference is a non-random
signal produced by another system. In some cases the noise is small enough
to be negligible and we need not do any formal analysis of it, but it is never
zero. In communication systems the relative powers of the desired signal and
the undesirable noise or interference are always important and the noise is often
not negligible in comparison with the signal. The most important naturally
occurring random noise is thermal noise (also called Johnson noise). Thermal
noise arises from the random motion of electrons in any conducting medium.
Random Signals
A resistor of resistance R ohms at an absolute temperature of T kelvins
produces a random gaussian noise at its terminals with zero mean and variance
2 ( kT )
2
v = v =
2 2
R V2 where k is Boltzmann's constant 1.38 10 23 J/K
3h
and h is Planck's constant 6.62 10 34 J s. The power spectrum of this voltage
2Rh f
is G v ( f ) = V 2
/ Hz. To get some idea of how this power spectrum
e h f /kT
1
varies with frequency let T = 290 K (near room temperature). Then
kT = 4 10 21 J and h f / kT = f / 6.0423 1012. So at frequencies below
about 1 THz, h f / kT << 1 and eh f /kT 1 + h f / kT . Then
2Rh f
Gv ( f ) = 2kTR V2 / Hz and the power spectrum is approximately
h f / kT
constant.
Random Signals
In analysis of noise effects due to the thermal noise of resistors we can
model a resistor by a Thevenin equivalent, the series combination of a
noiseless resistor of the same resistance and a noise voltage source whose
power spectrum is G v ( f ) = 2kTR V2 / Hz. Alternately we could also use a
Norton equivalent of a noiseless resistor of the same resistance in parallel
2kT
with a noise current source whose power spectrum is G i ( f ) = A 2 / Hz.
R
Random Signals
We have seen that at frequencies below about 1 THz the power spectrum of
thermal noise is essentially constant. Noise whose power spectrum is constant
over all frequencies is called white noise. (The name comes from optics in
which light having a constant spectral density over the visible range appears
white to the human eye.) We will designate the power spectrum of white
noise as G ( f ) = N 0 / 2 where N 0 is a density constant. (The factor of 1/2
is there to account for half the power in positive frequencies and half in
negative frequencies.) If the power spectrum is constant, the autocorrelation
N0
must be R ( ) = ( ) , an impulse at zero time shift. This indicates that
2
white noise is completely uncorrelated with itself at any non-zero shift.
In analysis of communication systems we normally treat thermal noise as
white noise because its power spectrum is virtually flat over a very wide
frequency range.
Random Signals
Some random noise sources have a power spectrum that is white and
unrelated to temperature. But we often assign them a noise temperature
anyway and analyze them as though they were thermal. This is convenient
for comparing white random noise sources. The noise temperature of a
2 Ga ( f ) N0
non-thermal random noise source is TN = = . Then, if we know
k k
a random noise source's noise temperature its density constant is N 0 = kTN .
Baseband Signal Transmission with Noise