Greedy

Huffman codes

R. Inkulu
http://www.iitg.ac.in/rinkulu/

(Huffman codes)

1 / 15

Encoding symbols using bits

Given a set of symbols S, the code of S is a one-to-one function

: S N, where each element of N is a binary number. The codeword

of a symbol x S is (x).
The fixed-length code does not take frequency of occurrence of

individual symbols into account; hence, not space-efficient.

The variable-length code helps in improving the space-efficiency: assign

longer code to less frequently used symbols and vice versa.

(Huffman codes)

2 / 15

Prefix code

Difficulty in decoding text with an arbitrary variable-length code:

ex. how to decode 01, when (a) = 0, (b) = 1, (c) = 01
The variable-length code in which no codeword is a prefix of another is

termed as a prefix code.

with code 1 (a) = 11, 1 (b) = 01, 1 (c) = 001, 1 (d) = 10, 1 (e) = 000
decoding 0010000011101 yields cecab

(Huffman codes)

3 / 15

Optimal prefix codes

Given a set of symbols S with their frequency of occurrences, fx for every

x S, determine a space-efficient prefix code that assigns a unique

codeword for each symbol x in S.
The average number of bits required per letter (ABL) is

xS fx .|(x)|.

Hence, the objective is to choose a code that minimizes ABL.

(Huffman codes)

4 / 15

Optimal prefix code example

For fa = 0.32, fb = 0.25, fc = 0.20, fd = 0.18, fe = 0.15,

with fixed-length code, ABL is 3
with 1 (a) = 11, 1 (b) = 01, 1 (c) = 001, 1 (d) = 10, 1 (e) = 000,

ABL(1 ) = 2.25
with 2 (a) = 11, 2 (b) = 10, 2 (c) = 01, 2 (d) = 001, 2 (e) = 000,

ABL(2 ) = 2.23

(Huffman codes)

5 / 15

Representing prefix codes using binary trees

0

a
0

0
e

1
b

1
d

0
c

1
b

0
e

0
d

1
a

1
c

c
0
e

0
b

1
a

1
d

(a) = 11, (b) =

(a) = 1, (b) =

(a) = 11, (b) =

10, (c) = 01, (d) =

011, (c) = 010, (d) =

01, (c) = 001, (d) =

001, (e) = 000

001, (e) = 000

10, (e) = 000

Consider a binary tree T with each leaf of T is labeled with a distinct

letter in S. For each symbol x S, the path from the root to the leaf
labeled x; each time the path goes from a node to its left (resp. right)
child, write down a 0 (resp. 1) to get the encoding of x.

(Huffman codes)

6 / 15

Representing prefix codes using binary trees

0

a
0

0
e

1
b

1
d

0
c

1
b

0
e

0
d

1
a

1
c

0
0
e

0
b

1
a

1
d

(a) = 11, (b) =

(a) = 1, (b) =

(a) = 11, (b) =

10, (c) = 01, (d) =

011, (c) = 010, (d) =

01, (c) = 001, (d) =

001, (e) = 000

001, (e) = 000

10, (e) = 000

Consider a binary tree T with each leaf of T is labeled with a distinct

letter in S. For each symbol x S, the path from the root to the leaf
labeled x; each time the path goes from a node to its left (resp. right)
child, write down a 0 (resp. 1) to get the encoding of x.
The encoding of S constructed from T is a prefix code.

(Huffman codes)

6 / 15

Representing prefix codes using binary trees (cont)

a
0

0
e

1
d

0
c

1
b

0
e

0
d

1
a

1
c

1
b

0
e

0
b

1
a

1
d

(a) = 11, (b) =

(a) = 1, (b) =

(a) = 11, (b) =

10, (c) = 01, (d) =

011, (c) = 010, (d) =

01, (c) = 001, (d) =

001, (e) = 000

001, (e) = 000

10, (e) = 000

Given a prefix code, we can build a binary tree recursively.

(Huffman codes)

7 / 15

Objective in terms of binary trees

Constructing an optimal prefix code involves

(Huffman codes)

8 / 15

Objective in terms of binary trees

Constructing an optimal prefix code involves
searching for a binary tree T

(Huffman codes)

8 / 15

Objective in terms of binary trees

Constructing an optimal prefix code involves
searching for a binary tree T
labeling the leaves of T

(Huffman codes)

8 / 15

Objective in terms of binary trees

Constructing an optimal prefix code involves
searching for a binary tree T
labeling the leaves of T

so that together they minimize ABL =

(Huffman codes)

xS fx .|(x)|

xS fx . depthT (x).

8 / 15

Objective in terms of binary trees

Constructing an optimal prefix code involves
searching for a binary tree T
labeling the leaves of T

so that together they minimize ABL =

xS fx .|(x)|

xS fx . depthT (x).

a
0

0
e

1
d

0
c

1
b

0
e

0
d

1
a

1
c

1
b

c
0
e

0
b

1
a

1
d

(a) = 11, (b) =

(a) = 1, (b) =

(a) = 11, (b) =

10, (c) = 01, (d) =

011, (c) = 010, (d) =

01, (c) = 001, (d) =

001, (e) = 000

001, (e) = 000

10, (e) = 000

For fa = 0.32, fb = 0.25, fc = 0.20, fd = 0.18, fe = 0.15, the rightmost one gives an optimal prefix code.

(Huffman codes)

8 / 15

Optimal binary tree is full

The binary tree corresponding to the optimal prefix code is full.

(Huffman codes)

9 / 15

Objective in terms of full binary trees

Constructing an optimal prefix code involves

(Huffman codes)

10 / 15

Objective in terms of full binary trees

Constructing an optimal prefix code involves

searching for a full binary tree T

(Huffman codes)

10 / 15

Objective in terms of full binary trees

Constructing an optimal prefix code involves

searching for a full binary tree T
labeling the leaves of T

(Huffman codes)

10 / 15

Objective in terms of full binary trees

Constructing an optimal prefix code involves

searching for a full binary tree T
labeling the leaves of T

so that together they minimize ABL =

(Huffman codes)

xS fx .|(x)|

xS fx . depthT (x).

10 / 15

Labeling leaves of a given optimal full binary tree

For any two leaves u and v with depth(u) < depth(v) in an optimal full

binary tree T , the symbol associated with u must be more frequent than
the symbol associated with v.
- proof using an exchange argument

(Huffman codes)

11 / 15

Labeling leaves of a given optimal full binary tree

For any two leaves u and v with depth(u) < depth(v) in an optimal full

binary tree T , the symbol associated with u must be more frequent than
the symbol associated with v.
- proof using an exchange argument
With the above in place, choice of assignment of symbols among leaves

of the same depth does not affect the ABL.

(Huffman codes)

11 / 15

Algorithm to label the leaves of a given optimal full

binary tree

take leaves of least depth and label them with the highest-frequency

symbols in any order

take leaves of next least depth and label them with the highest-frequency

symbols in any order

etc.,

(Huffman codes)

12 / 15

Observations to construct an optimal full binary tree

There is an optimal prefix code, with corresponding tree T , in which the

two lowest-frequency letters, say x and y, are assigned to leaves that are
sibilings in T .

(Huffman codes)

13 / 15

Observations to construct an optimal full binary tree

There is an optimal prefix code, with corresponding tree T , in which the

two lowest-frequency letters, say x and y, are assigned to leaves that are
sibilings in T .
Let x and y be the two lowest-frequency letters. Let T be a full binary

tree corresponding to an optimal prefix code for S {y, z} {w} with

fw = fy + fz . Also, let T be the tree obtained by attaching leaves y and z
as children of node w of T .
Then, ABL(T) = ABL(T ) + fw .

(Huffman codes)

13 / 15

Huffman algorithm
Recursively find two symbols y, z with lowest frequency and make them

siblings of the binary tree T to be constructed, before setting S to

S {y, z} {w} with fw = fy + fz . The resultant codewords for all the
symbols together is known as the Huffman code.

0
6/21
1

0
3/21
0

0
12/21
1

21/21
1
0

d:6/21 e:4/21

9/21
1
f:5/21

c:3/21

a:1/21 b:2/21 (freq)

(a) = 0000, (b) = 0001, (c) = 001, (d) = 01, (e) = 10, (f ) = 11

(Huffman codes)

14 / 15

Huffman algorithm
Recursively find two symbols y, z with lowest frequency and make them

siblings of the binary tree T to be constructed, before setting S to

S {y, z} {w} with fw = fy + fz . The resultant codewords for all the
symbols together is known as the Huffman code.

0
6/21
1

0
3/21
0

0
12/21
1

21/21
1
0

d:6/21 e:4/21

9/21
1
f:5/21

c:3/21

a:1/21 b:2/21 (freq)

(a) = 0000, (b) = 0001, (c) = 001, (d) = 01, (e) = 10, (f ) = 11

using priority queue, takes O(|S| lg |S|) time

(Huffman codes)

14 / 15

Correctness

feasibility: every symbol got a symbol

(Huffman codes)

15 / 15

Correctness

feasibility: every symbol got a symbol

optimality: induction on the size of the alphabet

(Huffman codes)

15 / 15