Mathematical Preliminaries of

Lossless Compression

Review of information theory

Self Information
Entropy

Coding

Prefix Codes
Huffman Codes

Example

Example codes for alphabets with four

symbols
Symbol

Prob

Code1

Code2

Code3

Code3

a1

0.5

a2

0.25

10

01

a3

0.125

00

110

011

a4

0.125

10

11

111

0111

Test for unique decodability

Two code words a & b

a k bits long, b n bits long and k < n
If first k bits of b are identical to a then
a is a prefix of b.
Last n-k bits of b dangling suffix
Example: a = 010, b = 01011
a is prefix of b and dangling suffix is 11

Test

Construct list of all code words

Examine them pair wise to see if any code word is a prefix
of another codeword.
If found, add the dangling suffix to the list unless we have
added the same dangling suffix in the previous iteration
Repeat the procedure using the larger list
Continue this until we get a dangling suffix that is a
codeword or there are no more unique dangling suffixes
If we get first outcome, the code is not uniquely decodable.
If we get the second outcome, the code is uniquely
decodable.

Examples

1.
2.
3.
4.

{0, 01,11}
{0,01,10}
Determine whether the following codes are
uniquely decodable:
{0,01,11,111}
{0,01,110,111}
{0,10,110,111}
{1,10,110,111}

Kraft-McMillan Inequality

Necessary condition on the codeword lengths

of uniquely decodable codes

Let C be a code with N words with lengths l 1, l2,

..lN. If C is uniquely decodable, then
iN

K (C ) 2 li 1.
i 1

Proof - See the text (by Sayood) page 32

Prefix Code

No codeword is a prefix of any other

codeword
Prefix codes are always uniquely decodable
For any uniquely decodable non-prefix code
there exists an equivalent prefix code

Test for Prefix Codes

Binary Tree

A simple way to check whether a code is a

prefix code
Draw a binary tree starting from a single node
(root) with two branches.
One branch corresponds to a 1 and other to a 0.
Notation used here left branch 0 and right
branch 1.

Prefix Codes

A code that is not a prefix code such as

code 4, will have code words associated
with internal nodes.

The code for any symbol can be obtained

by traversing the tree from the root to the
node corresponding to that symbol.

Examples

In a prefix code all code words are associated with

the external nodes or leaves only

a1

a1
a1
a3

a2
a4

a2

a2

a3

a3
a4

a4

Prefix Codes

If we restrict ourselves to Prefix codes, do

we lose something?

No

For any non prefix uniquely decodable

code, we can always find a Prefix code
with the same codeword lengths
Proof - See the text (by Sayood) pages 33 and
34

Prefix Codes

We can always find a prefix code that satisfies

necessary condition, Kraft McMillan Inequality

Hence, we can restrict ourselves to Prefix

codes alone.

Huffman codes..