Information Theory

Goal of Todays Lecture

Information Theorysome introduction
Information measure
Function determination for information
Average information per symbol
Information rate
Coding
Shannon-Fano coding

Information Theory
It is a study of Communication Engineering
plus Maths.

A Communication Engineer has to Fight with
Limited Power
Inevitable Background Noise
Limited Bandwidth

Information Theory deals with
The Measure of Source Information
The Information Capacity of the channel
Coding
If the rate of information from a source does not exceed the capacity of
the channel, then there exist a coding scheme such that information can
be transmitted over the communication channel with arbitrary small
amount of errors despite the presence of noise
Source
Encoder
Channel
Encoder
Noisy
Channel
Channel
Decoder
Source
Decoder
Equivalent noiseless
Channel
Information Measure
This is utilized to determine the information rate of
discrete Sources

Consider two Messages

A Dog Bites a Man High probability Less information

A Man Bites a Dog Less probability High Information

So we can say that

Information (1/Probability of Occurrence)

Information Measure
Also we can state the three law from Intution

Rule 1: Information I(m
k
) approaches to 0 as P
k

approaches infinity.

Mathematically I(m
k
) = 0 as P
k
1

e.g. Sun Rises in East


Information Measure

Rule 2: The Information Content I(m
k
) must be Non
Negative contity.

It may be zero

Mathematically I(m
k
) >= 0 as 0 <= P
k
<=1

e.g. Sun Rises in West.


Information Measure

Rule 3: The information content of message
having higher probability is less than the
information content of message having lower
probability

Mathematically I(m
k
) > I(m
j
) if P
k
< P
j



Information Measure
Also we can state for the Sum of two messages that the
information content in the two combined messages
is same as the sum of information content of each
message Provided the occurrence is mutually
independent.

e.g. There will be sunny weather today.
There will be cloudy weather tomorrow.

Mathematically

I (m
k
and m
j
) = I(m
k
m
j
)
= I(m
k
)+I(m
j
)
Information measure
So Question is which function that we can use that measure the
Information?

Information = F(1/Probability)

Requirement that function must satisfy
1. Its output must be non negative Quantity.
2. Minimum Value is 0.
3. It should convert product into summation.

Information I(m
k
) = Log
b
(1/ P
k
)

Here b may be 2, e or 10

If b = 2 then unit is bits
b = e then unit is nats
b = 10 then unit is decit
Conversion Between Units
10
2
10
log ln
log
ln 2 log 2
v v
v = =
Example
A source generates one of four symbols
during each interval with probabilities P
1
=1/2,
P
2
=1/4, P
3
= P
4
=1/8. Find the information
content of three messages.
Average Information Content
It is necessary to define the information content of
the particular symbol as communication channel
deals with symbol.

Here we make following assumption..

1. The source is stationary, so probability remains
constant with time.

2. The successive symbols are statistically
independent and come out at average rate of r
symbols per second
Average Information Content
Suppose a source emits M Possible symbols s
1
, s
2
,
..S
M
having probability of occurrence
P
1
, P
2
,.P
M



For a long message having symbols N (>>M)

s
1
will occur P
1
N times, like also
s
2
will occur P
2
N times and so on.
1
1
M
i
Pi
=
=

Average Information Content

Since s
1
occurs P
1
N times so information Contribution by s
1

is P
1
N log (1/P
1
).
Similarly information contribution by s
2
is
P
2
N log (1/P
2
) and so on.

Hence the total information content is


And average information is obtained by
1
1
log
M
total i
i
i
I NP
P
=
| |
=
|
\ .

1
1
log
M
total
i
i
i
I
H P
N P
=
| |
= =
|
\ .

Bits/Symbol
It means that in a long message we can expect H bits of information
per symbol. H is called entropy.
Information Rate
Information rate = total information/ time taken

Here, time taken

N bits are transmitted with r symbols per second. Total
information is nh.


Information rate

n
Tb
r
=
nH
R
n
r
R rH
=
| |
|
\ .
=
Bits/sec
Some Maths
H satisfies following Equation
2
0 log H M s s
Maximum H will occur when all the messages have equal probability.
Hence h also shows the uncertainty that which of the symbol will occur. As
h approaches to its maximum value we cant determine which message will
occur.
Consider a system which transmits only 2 messages having equal
probability of occurrence 0.5. Then, H=1. And at every instant we cant say
which one of the two message will occur. So what would happen if there are
more than two symbol sources?
Variation of H Vs. p
Lets Consider a Binary Source,
means M=2

Let the two symbols occur at the probability
p and
1-p Respectively.
Where o < p < 1.
So Entropy is

2 2
1 1
log (1 ) log
1
H p p
p p
| | | |
= +
| |

\ . \ .
( ) p = O
Horse Shoe Function
Variation of H Vs. P
( )
0
dH d p
dp dp
O
= =
2
2
1 1
0
1
d H
dp p p
= <

0.5 1 0
1
Now we want to obtain the shape of the curve
1
log 0
p
p
| |
=
|
\ .
Verify it by Double differentiation
Example

Maximum Information rate
R rH =
2
max log H M =
We Know that
Also
2
max log R r M =
Hence
Coding for Discrete memoryless Source
Here Discrete means The Source is emitting
different symbols that are fixed.

Memoryless = Occurrence of present symbol is
independent of previous symbol.

Average Code Length
1
i i
M
i
N pN
=
=

Where
Ni=Code length in Binary
digits (binits)
Coding for Discrete memoryless Source
1
b
R H
r
N
q = = s
Efficiency
Coding for Discrete memoryless Source
Krafts inequality
1
2 1
M
Ni
i
K

=
= s

If this is satisfied then only the Coding is uniquely Decipherable

or Separable.
Example
Find The efficiency and Krafts inequality
m
i
p
i
Code I

Code II

Code III

Code IV

A

B

C

D








00

01

10

11
0

1

10

11
0

01

011

0111
0

10

110

111
This Code is not
Uniquely Decipherable
Shannon Fano Coding Technique
Algorithm.
Step 1: Arrange all messages in descending
order of probability.

Step 2: Devide the Seq. in two groups in such a
way that sum of probabilities in each
group is same.

Step 3: Assign 0 to Upper group and 1 to Lower
group.

Step 4: Repeat the Step 2 and 3 for Group 1 and 2 and
So on..
Example
Messages
Mi
Pi No. Of
Bits
Code
M1
M2
M3
M4
M5
M6
M7
m8

1/8/
1/8
1/16
1/16
1/16
1/32
1/32
0
1
1
1
1
1
1
1

0
0
1
1
1
1
1

0
1
0
0
1
1
1
0
1
0
1
1
0
1
Coding Procedure
1
3
3
4
4
4
5
5
0
100
101
1100
1101
1110
11110
11111
Questions
Thank You