6.046J/18.401J
LECTURE 1
Analysis of Algorithms
Insertion sort
Merge sort
Course information
1.
2.
3.
4.
5.
6.
Staff
Prerequisites
Lectures
Recitations
Handouts
Textbook (CLRS)
7. Extra help
8. Registration
9.Problem sets
10.Describing algorithms
11.Grading policy
12.Collaboration policy
September 8, 2004
Introduction to Algorithms
L1.2
Analysis of algorithms
The theoretical study of computer-program
performance and resource usage.
Whats more important than performance?
modularity
user-friendliness
correctness
programmer time
maintainability
simplicity
functionality
extensibility
robustness
reliability
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.3
September 8, 2004
Introduction to Algorithms
L1.4
September 8, 2004
Introduction to Algorithms
L1.5
Insertion sort
pseudocode
INSERTION-SORT (A, n)
A[1 . . n]
for j 2 to n
do key A[ j]
ij1
while i > 0 and A[i] > key
do A[i+1] A[i]
ii1
A[i+1] = key
September 8, 2004
Introduction to Algorithms
L1.6
Insertion sort
INSERTION-SORT (A, n)
A[1 . . n]
for j 2 to n
do key A[ j]
ij1
while i > 0 and A[i] > key
do A[i+1] A[i]
ii1
A[i+1] = key
pseudocode
A:
sorted
September 8, 2004
key
Introduction to Algorithms
L1.7
September 8, 2004
Introduction to Algorithms
L1.8
September 8, 2004
Introduction to Algorithms
L1.9
September 8, 2004
Introduction to Algorithms
L1.8
September 8, 2004
Introduction to Algorithms
L1.11
September 8, 2004
Introduction to Algorithms
L1.12
September 8, 2004
Introduction to Algorithms
L1.13
September 8, 2004
Introduction to Algorithms
L1.14
September 8, 2004
Introduction to Algorithms
L1.15
September 8, 2004
Introduction to Algorithms
L1.16
September 8, 2004
Introduction to Algorithms
L1.17
9 done
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.18
Running time
The running time depends on the input: an
already sorted sequence is easier to sort.
Parameterize the running time by the size of
the input, since short sequences are easier to
sort than long ones.
Generally, we seek upper bounds on the
running time, because everybody likes a
guarantee.
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.19
Kinds of analyses
Worst-case: (usually)
T(n) = maximum time of algorithm
on any input of size n.
Average-case: (sometimes)
T(n) = expected time of algorithm
over all inputs of size n.
Need assumption of statistical
distribution of inputs.
Best-case: (bogus)
Cheat with a slow algorithm that
works fast on some input.
September 8, 2004
Introduction to Algorithms
L1.20
Machine-independent time
What is insertion sorts worst-case time?
It depends on the speed of our computer:
relative speed (on the same machine),
absolute speed (on different machines).
BIG IDEA:
Ignore machine-dependent constants.
Look at growth of T(n) as n .
Asymptotic Analysis
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.21
-notation
Math:
Engineering:
Drop low-order terms; ignore leading constants.
Example: 3n3 + 90n2 5n + 6046 = (n3)
September 8, 2004
Introduction to Algorithms
L1.22
Asymptotic performance
When n gets large enough, a (n2) algorithm
always beats a (n3) algorithm.
T(n)
n
September 8, 2004
n0
We shouldnt ignore
asymptotically slower
algorithms, however.
Real-world design
situations often call for a
careful balancing of
engineering objectives.
Asymptotic analysis is a
useful tool to help to
structure our thinking.
Introduction to Algorithms
L1.23
T ( n) =
2)
(
(
j
)
=
[arithmetic series]
j =2
T ( n) =
( j / 2) = (n 2 )
j =2
September 8, 2004
Introduction to Algorithms
L1.24
Merge sort
MERGE-SORT A[1 . . n]
1. If n = 1, done.
2. Recursively sort A[ 1 . . n/2 ]
and A[ n/2+1 . . n ] .
3. Merge the 2 sorted lists.
Key subroutine: MERGE
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.25
September 8, 2004
Introduction to Algorithms
L1.26
1
1
September 8, 2004
Introduction to Algorithms
L1.27
20 12
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.28
20 12
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.29
20 12
20 12
13 11
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.30
20 12
20 12
13 11
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.31
20 12
20 12
20 12
13 11
13 11
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.32
20 12
20 12
20 12
13 11
13 11
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.33
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
September 8, 2004
Introduction to Algorithms
L1.34
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
11
September 8, 2004
Introduction to Algorithms
L1.35
20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
13
11
September 8, 2004
Introduction to Algorithms
L1.36
20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
13
11
12
September 8, 2004
Introduction to Algorithms
L1.37
20 12
20 12
20 12
20 12
20 12
13 11
13 11
13 11
13 11
13 11
13
11
12
September 8, 2004
Introduction to Algorithms
L1.38
September 8, 2004
Introduction to Algorithms
L1.39
(1) if n = 1;
2T(n/2) + (n) if n > 1.
September 8, 2004
Introduction to Algorithms
L1.40
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
September 8, 2004
Introduction to Algorithms
L1.41
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
T(n)
September 8, 2004
Introduction to Algorithms
L1.42
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
T(n/2)
T(n/2)
September 8, 2004
Introduction to Algorithms
L1.43
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2
cn/2
T(n/4)
T(n/4)
T(n/4)
T(n/4)
September 8, 2004
Introduction to Algorithms
L1.44
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn/4
(1)
September 8, 2004
Introduction to Algorithms
L1.45
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn/2
cn/2
cn/4
cn/4
cn/4
h = lg n cn/4
(1)
September 8, 2004
Introduction to Algorithms
L1.46
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
h = lg n cn/4
(1)
September 8, 2004
Introduction to Algorithms
L1.47
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
h = lg n cn/4
cn
(1)
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.48
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2
cn/2
cn/4
cn/4
cn/4
cn
h = lg n cn/4
cn
(1)
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.49
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2
cn/2
cn/4
cn/4
(1)
cn/4
cn
h = lg n cn/4
cn
#leaves = n
(n)
20014 by Charles E. Leiserson
September 8, 2004
Introduction to Algorithms
L1.50
Recursion tree
Solve T(n) = 2T(n/2) + cn, where c > 0 is constant.
cn
cn
cn/2
cn/2
cn/4
cn/4
(1)
cn/4
cn
h = lg n cn/4
cn
#leaves = n
(n)
Total = (n lg n)
September 8, 2004
Introduction to Algorithms
L1.51
Conclusions
(n lg n) grows more slowly than (n2).
Therefore, merge sort asymptotically
beats insertion sort in the worst case.
In practice, merge sort beats insertion
sort for n > 30 or so.
Go test it out for yourself!
September 8, 2004
Introduction to Algorithms
L1.52
Introduction to Algorithms
6.046J/18.401J
LECTURE 2
Asymptotic Notation
O-, -, and -notation
Recurrences
Substitution method
Iterating the recurrence
Recursion tree
Master method
Prof. Charles E. Leiserson
Asymptotic notation
O-notation (upper bounds):
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
Introduction to Algorithms
L2.2
Asymptotic notation
O-notation (upper bounds):
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
EXAMPLE: 2n2 = O(n3)
(c = 1, n0 = 2)
Introduction to Algorithms
L2.3
Asymptotic notation
O-notation (upper bounds):
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
EXAMPLE: 2n2 = O(n3)
functions,
not values
September 13, 2004
(c = 1, n0 = 2)
Introduction to Algorithms
L2.4
Asymptotic notation
O-notation (upper bounds):
We
We write
write f(n)
f(n) == O(g(n))
O(g(n)) ifif there
there
exist
such
exist constants
constants cc >> 0,
0, nn00 >> 00 such
that
that 00 f(n)
f(n) cg(n)
cg(n) for
for all
all nn nn00..
EXAMPLE: 2n2 = O(n3)
(c = 1, n0 = 2)
funny, one-way
equality
functions,
not values
Introduction to Algorithms
L2.5
Introduction to Algorithms
L2.6
Introduction to Algorithms
L2.7
Introduction to Algorithms
L2.8
Macro substitution
Convention: A set in a formula represents
an anonymous function in the set.
Introduction to Algorithms
L2.9
Macro substitution
Convention: A set in a formula represents
an anonymous function in the set.
EXAMPLE:
f(n) = n3 + O(n2)
means
f(n) = n3 + h(n)
for some h(n) O(n2) .
Introduction to Algorithms
L2.10
Introduction to Algorithms
L2.11
Introduction to Algorithms
L2.12
(g(n))
(g(n)) == {{ f(n)
f(n) :: there
there exist
exist constants
constants
cc >> 0,
such
0, nn00 >> 00 such
that
that 00 cg(n)
cg(n) f(n)
f(n)
for
for all
all nn nn00 }}
EXAMPLE:
n = (lg n)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.13
Introduction to Algorithms
L2.14
1 2
2
EXAMPLE: n 2n = (n )
2
Introduction to Algorithms
L2.15
1 2
n
2
2 n = ( n )
Introduction to Algorithms
L2.16
Solving recurrences
The analysis of merge sort from Lecture 1
required us to solve a recurrence.
Recurrences are like solving integrals,
differential equations, etc.
o Learn a few tricks.
Lecture 3: Applications of recurrences to
divide-and-conquer algorithms.
Introduction to Algorithms
L2.17
Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
Introduction to Algorithms
L2.18
Substitution method
The most general method:
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
EXAMPLE: T(n) = 4T(n/2) + n
[Assume that T(1) = (1).]
Guess O(n3) . (Prove O and separately.)
Assume that T(k) ck3 for k < n .
Prove T(n) cn3 by induction.
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.19
Example of substitution
T (n) = 4T (n / 2) + n
4c ( n / 2 ) 3 + n
= ( c / 2) n 3 + n
desired residual
= cn3 ((c / 2)n3 n)
cn3 desired
whenever (c/2)n3 n 0, for example,
if c 2 and n 1.
residual
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.20
Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have (1) cn3, if we
pick c big enough.
Introduction to Algorithms
L2.21
Example (continued)
We must also handle the initial conditions,
that is, ground the induction with base
cases.
Base: T(n) = (1) for all n < n0, where n0
is a suitable constant.
For 1 n < n0, we have (1) cn3, if we
pick c big enough.
This bound is not tight!
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.22
Introduction to Algorithms
L2.23
Introduction to Algorithms
L2.24
Introduction to Algorithms
L2.25
Introduction to Algorithms
L2.26
Introduction to Algorithms
L2.27
Introduction to Algorithms
L2.28
Introduction to Algorithms
L2.29
Recursion-tree method
A recursion tree models the costs (time) of a
recursive execution of an algorithm.
The recursion-tree method can be unreliable,
just like any method that uses ellipses ().
The recursion-tree method promotes intuition,
however.
The recursion tree method is good for
generating guesses for the substitution method.
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.30
Introduction to Algorithms
L2.31
Introduction to Algorithms
L2.32
T(n/2)
Introduction to Algorithms
L2.33
T(n/8)
(n/2)2
T(n/8)
T(n/4)
Introduction to Algorithms
L2.34
(n/8)2
(n/4)2
(n/16)2
(n/2)2
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.35
(n/2)2
(n/8)2
(n/4)2
(n/16)2
n2
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.36
5 n2
16
(n/2)2
(n/8)2
(n/4)2
(n/16)2
n2
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.37
(n/8)2
(n/8)2
(n/16)2
(n/2)2
(n/4)2
5 n2
16
25 n 2
256
(n/4)2
n2
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.38
(n/8)2
(1)
September 13, 2004
(n/4)2
( ) +( )
2
5
5
1 + 16 + 16
Total = n
= (n2)
Introduction to Algorithms
5 n2
16
25 n 2
256
(n/4)2
(n/16)2
n2
5 3
16
+L
geometric series
20014 by Charles E. Leiserson
L2.39
Introduction to Algorithms
L2.40
Introduction to Algorithms
L2.41
Introduction to Algorithms
L2.42
Introduction to Algorithms
L2.43
Examples
EX. T(n) = 4T(n/2) + n
a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 ) for = 1.
T(n) = (n2).
Introduction to Algorithms
L2.44
Examples
EX. T(n) = 4T(n/2) + n
a = 4, b = 2 nlogba = n2; f (n) = n.
CASE 1: f (n) = O(n2 ) for = 1.
T(n) = (n2).
EX. T(n) = 4T(n/2) + n2
a = 4, b = 2 nlogba = n2; f (n) = n2.
CASE 2: f (n) = (n2lg0n), that is, k = 0.
T(n) = (n2lg n).
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.45
Examples
EX. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = (n2 + ) for = 1
and 4(n/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = (n3).
Introduction to Algorithms
L2.46
Examples
EX. T(n) = 4T(n/2) + n3
a = 4, b = 2 nlogba = n2; f (n) = n3.
CASE 3: f (n) = (n2 + ) for = 1
and 4(n/2)3 cn3 (reg. cond.) for c = 1/2.
T(n) = (n3).
EX. T(n) = 4T(n/2) + n2/lg n
a = 4, b = 2 nlogba = n2; f (n) = n2/lg n.
Master method does not apply. In particular,
for every constant > 0, we have n = (lg n).
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.47
f (n)
a
f (n/b) f (n/b) f (n/b)
a
f (n/b2) f (n/b2) f (n/b2)
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.48
f (n)
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
f (n/b2) f (n/b2) f (n/b2)
f (n)
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.49
f (n)
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
f (n)
(1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.50
f (n)
(1)
#leaves = ah
= alogbn
= nlogba
a f (n/b)
a2 f (n/b2)
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
f (n)
nlogba (1)
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.51
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 1:
1: The
The weight
weight increases
increases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The leaves
leaves hold
hold aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
September 13, 2004
Introduction to Algorithms
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
(nlogba)
L2.52
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
(1)
September 13, 2004
C
CASE
ASE 2:
2: (k
(k == 0)
0) The
The weight
weight
isis approximately
approximately the
the same
same on
on
each
each of
of the
the log
logbbnn levels.
levels.
Introduction to Algorithms
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
(nlogbalg n)
L2.53
a
f (n/b) f (n/b) f (n/b)
a
h = logbn
f (n/b2) f (n/b2) f (n/b2)
C
CASE
ASE 3:
3: The
The weight
weight decreases
decreases
geometrically
geometrically from
from the
the root
root to
to the
the
(1) leaves.
leaves. The
The root
root holds
holds aa constant
constant
fraction
fraction of
of the
the total
total weight.
weight.
September 13, 2004
Introduction to Algorithms
f (n)
a f (n/b)
a2 f (n/b2)
Recursion tree:
nlogba (1)
( f (n))
L2.54
for x 1
1 + x + x2 + L + xn =
1 x
1
1+ x + x +L =
for |x| < 1
1 x
2
Return to last
slide viewed.
20014 by Charles E. Leiserson
Introduction to Algorithms
L2.55
Introduction to Algorithms
6.046J/18.401J
LECTURE 3
Divide and conquer
Binary search
Powering a number
Fibonacci numbers
Matrix multiplication
Strassens algorithm
VLSI tree layout
Prof. Charles E. Leiserson
The divide-and-conquer
design paradigm
1. Divide the problem (instance)
into subproblems.
2. Conquer the subproblems by
solving them recursively.
3. Combine subproblem solutions.
Introduction to Algorithms
L3.2
Merge sort
1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Linear-time merge.
Introduction to Algorithms
L3.3
Merge sort
1. Divide: Trivial.
2. Conquer: Recursively sort 2 subarrays.
3. Combine: Linear-time merge.
T(n) = 2 T(n/2) + (n)
# subproblems
subproblem size
work dividing
and combining
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.4
Introduction to Algorithms
L3.5
Introduction to Algorithms
L3.6
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Introduction to Algorithms
L3.7
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
12
15
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.8
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
12
15
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.9
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
12
15
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.10
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
12
15
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.11
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
12
15
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.12
Binary search
Find an element in a sorted array:
1. Divide: Check middle element.
2. Conquer: Recursively search 1 subarray.
3. Combine: Trivial.
Example: Find 9
3
12
15
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.13
work dividing
and combining
Introduction to Algorithms
L3.14
work dividing
and combining
Introduction to Algorithms
L3.15
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: (n).
Introduction to Algorithms
L3.16
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: (n).
Divide-and-conquer algorithm:
an
a n/2 a n/2
a (n1)/2 a (n1)/2 a
if n is even;
if n is odd.
Introduction to Algorithms
L3.17
Powering a number
Problem: Compute a n, where n N.
Naive algorithm: (n).
Divide-and-conquer algorithm:
an
a n/2 a n/2
a (n1)/2 a (n1)/2 a
if n is even;
if n is odd.
Introduction to Algorithms
L3.18
Fibonacci numbers
Recursive definition:
0
if n = 0;
if n = 1;
Fn = 1
Fn1 + Fn2 if n 2.
0
8 13 21 34 L
Introduction to Algorithms
L3.19
Fibonacci numbers
Recursive definition:
0
if n = 0;
if n = 1;
Fn = 1
Fn1 + Fn2 if n 2.
0
8 13 21 34 L
Introduction to Algorithms
L3.20
Computing Fibonacci
numbers
Bottom-up:
Compute F0, F1, F2, , Fn in order, forming
each number by summing the two previous.
Running time: (n).
Introduction to Algorithms
L3.21
Computing Fibonacci
numbers
Bottom-up:
Compute F0, F1, F2, , Fn in order, forming
each number by summing the two previous.
Running time: (n).
Naive recursive squaring:
Fn = n/ 5 rounded to the nearest integer.
Recursive squaring: (lg n) time.
This method is unreliable, since floating-point
arithmetic is prone to round-off errors.
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.22
Recursive squaring
Fn +1
Theorem:
Fn
Fn 1 1
.
=
Fn 1 1 0
Introduction to Algorithms
L3.23
Recursive squaring
Fn +1
Theorem:
Fn
Fn 1 1
.
=
Fn 1 1 0
Introduction to Algorithms
L3.24
Recursive squaring
Fn +1
Theorem:
Fn
Fn 1 1
.
=
Fn 1 1 0
F1 F0 1 0
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.25
Recursive squaring
Inductive step (n 2):
Fn +1
F
n
.
Fn Fn
Fn 1 1 1
=
Fn 1 Fn 1 Fn 2 1 0
n1
1 1
1 1
=
.
1 0
1 0
n
1 1
=
1
0
Introduction to Algorithms
L3.26
Matrix multiplication
Input: A = [aij], B = [bij].
Output: C = [cij] = A B.
c11 c12
c c
21 22
M M
c c
n1 n 2
i, j = 1, 2, , n.
O M M M
L ann bn1 bn 2
L b1n
L b2 n
O M
L bnn
Introduction to Algorithms
L3.27
Standard algorithm
for i 1 to n
do for j 1 to n
do cij 0
for k 1 to n
do cij cij + aik bkj
Introduction to Algorithms
L3.28
Standard algorithm
for i 1 to n
do for j 1 to n
do cij 0
for k 1 to n
do cij cij + aik bkj
Introduction to Algorithms
L3.29
Divide-and-conquer algorithm
IDEA:
nn matrix = 22 matrix of (n/2)(n/2) submatrices:
r s a b e f
t u = c d g h
C
r
s
t
u
= ae + bg
= af + bh
= ce + dg
= cf + dh
Introduction to Algorithms
L3.30
Divide-and-conquer algorithm
IDEA:
nn matrix = 22 matrix of (n/2)(n/2) submatrices:
r s a b e f
t u = c d g h
r
s
t
u
= ae + bg
= af + bh
= ce + dh
= cf + dg
September 15, 2004
C = A B
recursive
8 mults of (n/2)(n/2) submatrices
^
4 adds of (n/2)(n/2) submatrices
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.31
work adding
submatrices
Introduction to Algorithms
L3.32
work adding
submatrices
Introduction to Algorithms
L3.33
work adding
submatrices
Introduction to Algorithms
L3.34
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
Introduction to Algorithms
L3.35
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.36
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
r
s
t
u
= P5 + P4 P2 + P6
= P1 + P2
= P3 + P4
= P5 + P1 P3 P7
Introduction to Algorithms
L3.37
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
r
s
t
u
= P5 + P4 P2 + P6
= P1 + P2
= P3 + P4
= P5 + P1 P3 P7
77 mults,
mults, 18
18 adds/subs.
adds/subs.
Note:
Note: No
No reliance
reliance on
on
commutativity
commutativity of
of mult!
mult!
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.38
Strassens idea
Multiply 22 matrices with only 7 recursive mults.
P1 = a ( f h)
P2 = (a + b) h
P3 = (c + d) e
P4 = d (g e)
P5 = (a + d) (e + h)
P6 = (b d) (g + h)
P7 = (a c) (e + f )
r = P5 + P4 P2 + P6
= (a + d) (e + h)
+ d (g e) (a + b) h
+ (b d) (g + h)
= ae + ah + de + dh
+ dg de ah bh
+ bg + bh dg dh
= ae + bg
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.39
Strassens algorithm
1. Divide: Partition A and B into
(n/2)(n/2) submatrices. Form terms
to be multiplied using + and .
2. Conquer: Perform 7 multiplications of
(n/2)(n/2) submatrices recursively.
3. Combine: Form C using + and on
(n/2)(n/2) submatrices.
Introduction to Algorithms
L3.40
Strassens algorithm
1. Divide: Partition A and B into
(n/2)(n/2) submatrices. Form terms
to be multiplied using + and .
2. Conquer: Perform 7 multiplications of
(n/2)(n/2) submatrices recursively.
3. Combine: Form C using + and on
(n/2)(n/2) submatrices.
Introduction to Algorithms
L3.41
Analysis of Strassen
T(n) = 7 T(n/2) + (n2)
Introduction to Algorithms
L3.42
Analysis of Strassen
T(n) = 7 T(n/2) + (n2)
nlogba = nlog27 n2.81 CASE 1 T(n) = (nlg 7).
Introduction to Algorithms
L3.43
Analysis of Strassen
T(n) = 7 T(n/2) + (n2)
nlogba = nlog27 n2.81 CASE 1 T(n) = (nlg 7).
The number 2.81 may not seem much smaller than
3, but because the difference is in the exponent, the
impact on running time is significant. In fact,
Strassens algorithm beats the ordinary algorithm
on todays machines for n 32 or so.
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.44
Analysis of Strassen
T(n) = 7 T(n/2) + (n2)
nlogba = nlog27 n2.81 CASE 1 T(n) = (nlg 7).
The number 2.81 may not seem much smaller than
3, but because the difference is in the exponent, the
impact on running time is significant. In fact,
Strassens algorithm beats the ordinary algorithm
on todays machines for n 32 or so.
Best to date (of theoretical interest only): (n2.376L).
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.45
VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.
Introduction to Algorithms
L3.46
VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.
W(n)
H(n)
Introduction to Algorithms
L3.47
VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.
W(n)
H(n)
H(n) = H(n/2) + (1)
= (lg n)
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.48
VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.
W(n)
H(n)
H(n) = H(n/2) + (1)
= (lg n)
Introduction to Algorithms
L3.49
VLSI layout
Problem: Embed a complete binary tree
with n leaves in a grid using minimal area.
W(n)
H(n)
H(n) = H(n/2) + (1) W(n) = 2 W(n/2) + (1)
= (lg n)
= (n)
Area = (n lg n)
Introduction to Algorithms
L3.50
H-tree embedding
L(n)
L(n)
Introduction to Algorithms
L3.51
H-tree embedding
L(n)
L(n)
Introduction to Algorithms
L3.52
H-tree embedding
L(n)
L(n) = 2 L(n/4) + (1)
= ( n )
L(n)
Area = (n)
L(n/4) (1) L(n/4)
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.53
Conclusion
Divide and conquer is just one of several
powerful techniques for algorithm design.
Divide-and-conquer algorithms can be
analyzed using recurrences and the master
method (so practice this math).
The divide-and-conquer strategy often leads
to efficient algorithms.
20014 by Charles E. Leiserson
Introduction to Algorithms
L3.54
Introduction to Algorithms
6.046J/18.401J
Lecture 4
Prof. Piotr Indyk
Today
Randomized algorithms: algorithms that flip
coins
Matrix product checker: is AB=C ?
Quicksort:
Example of divide and conquer
Fast and practical sorting algorithm
Other applications on Wednesday
September 20, 2004
L4.2
Randomized Algorithms
Algorithms that make random decisions
That is:
Can generate a random number x from
some range {1R}
Make decisions based on the value of x
Why would it make sense ?
L4.3
L4.4
Randomized Algorithms
Two basic types:
Typically fast (but sometimes slow):
Las Vegas
Typically correct (but sometimes output
garbage): Monte Carlo
The probabilities are defined by the random
numbers of the algorithm! (not by random
choices of the problem instance)
September 20, 2004
L4.5
Matrix Product
Compute C=AB
Simple algorithm: O(n3) time
Multiply two 22 matrices using 7 mult.
O(n2.81) time [Strassen69]
Multiply two 70 70 matrices using 143640
multiplications O(n2.795) time [Pan78]
O(n2.376) [Coppersmith-Winograd]
September 20, 2004
L4.6
L4.7
The algorithm
Algorithm:
Choose a random binary vector x[1n] ,
such that Pr[xi=1]= , i=1n
Check if ABx=Cx
Does it run in O(n2) time ?
YES, because ABx = A(Bx)
L4.8
Correctness
Let D=AB, need to check if D=C
What if D=C ?
Then Dx=Cx ,so the output is YES
What if DC ?
Presumably there exists x such that
DxCx
We need to show there are many such x
September 20, 2004
L4.9
DC
L4.10
Vector product
(d-c):
x:
d1-c1 d2-c2
x1
x2
di-ci
dn-cn
xi
xn
= ji(dj-cj)xj + (di-ci)xi
L4.11
Analysis, ctd.
If xi=0, then (c-d)x=S1
If xi=1, then (c-d)x=S2S1
So, 1 of the choices gives (c-d)x0
Pr[cx=dx]
L4.12
Is AB=C ?
We have an algorithm that:
If answer=YES, then Pr[output=YES]=1
If answer=NO, then Pr[output=YES]
What if we want to reduce to ?
Run the algorithm twice, using independent random numbers
Output YES only if both runs say YES
Analysis:
If answer=YES, then Pr[output1=YES, output2=YES ]=1
If answer=NO, then
Pr[output=YES] = Pr[output1=YES, output2=YES]
= Pr[output1=YES]*Pr[output2=YES]
L4.13
Quicksort
Proposed by C.A.R. Hoare in 1962.
Divide-and-conquer algorithm.
Sorts in place (like insertion sort, but not
like merge sort).
Very practical (with tuning).
Can be viewed as a randomized Las Vegas
algorithm
L4.14
L4.15
L4.16
Partitioning subroutine
PARTITION(A, p, r) A[ p . . r]
x A[ p]
pivot = A[ p]
ip
for j p + 1 to r
do if A[ j] x
then i i + 1
exchange A[i] A[ j]
exchange A[ p] A[i]
return i
Invariant: xx
p
September 20, 2004
xx
xx
i
??
j
r
L4.17
Example of partitioning
66 10
10 13
13 55
i
j
88
33
22 11
11
L4.18
Example of partitioning
66 10
10 13
13 55
i
j
88
33
22 11
11
L4.19
Example of partitioning
66 10
10 13
13 55
i
j
88
33
22 11
11
L4.20
Example of partitioning
66 10
10 13
13 55
66
88
33
22 11
11
55 13
13 10
10 88
i
j
33
22 11
11
L4.21
Example of partitioning
66 10
10 13
13 55
66
88
33
22 11
11
55 13
13 10
10 88
i
j
33
22 11
11
L4.22
Example of partitioning
66 10
10 13
13 55
66
88
33
22 11
11
55 13
13 10
10 88
i
33
j
22 11
11
L4.23
Example of partitioning
66 10
10 13
13 55
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
i
j
L4.24
Example of partitioning
66 10
10 13
13 55
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
i
j
L4.25
Example of partitioning
66 10
10 13
13 55
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
i
88 13
13 10
10 11
11
j
L4.26
Example of partitioning
66 10
10 13
13 55
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
i
88 13
13 10
10 11
11
j
L4.27
Example of partitioning
66 10
10 13
13 55
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
i
88 13
13 10
10 11
11
L4.28
Example of partitioning
66 10
10 13
13 55
88
33
22 11
11
66
55 13
13 10
10 88
33
22 11
11
66
55
33 10
10 88 13
13 22 11
11
66
55
33
22
88 13
13 10
10 11
11
22
55
33
66
i
88 13
13 10
10 11
11
L4.29
Analysis of quicksort
Assume all input elements are distinct.
In practice, there are better partitioning
algorithms for when duplicate input
elements may exist.
What is the worst case running time of
Quicksort ?
xx
September 20, 2004
xx
(c) Piotr Indyk & Charles Leiserson
xx
L4.30
Worst-case of quicksort
Input sorted or reverse sorted.
Partition around min or max element.
One side of partition always has no elements.
(arithmetic series)
L4.31
L4.32
L4.33
L4.34
L4.35
O
(1)
L4.36
n
k = (n 2 )
k =1
O
(1)
L4.37
(1) c(n2)
(1)
n
k = (n 2 )
k =1
O
(1)
L4.38
Nice-case analysis
If were lucky, PARTITION splits the array evenly:
T(n) = 2T(n/2) + (n)
(same as merge sort)
= (n lg n)
What if the split is
1 9
always 10 : 10 ?
L4.39
L4.40
T (101 n )
T (109 n )
L4.41
cn
9
10
cn
9
9
81
1
T (100
n ) T (100
n ) T (100
n )T (100
n)
L4.42
cn
9
100
9
10
cn
cn
log10/9n
9
81
cn
cn
100
100
1
100
cn
cn
(1)
cn
cn
cn
(1)
September 20, 2004
L4.43
9
100
9
10
cn
cn
log10/9n
9
81
cn
cn
100
100
log10n
1
cn
100
cn
cn
(1)
cn
cn
cn
(1)
cn log10n T(n) cn log10/9n + (n)
September 20, 2004
L4.44
Randomized quicksort
Partition around a random element. I.e.,
around A[t] , where t chosen uniformly
at random from {pr}
We will show that the expected time is
O(n log n)
L4.45
Paranoid quicksort
Will modify the algorithm to make it easier to
analyze:
Repeat:
Choose the pivot to be a random element
of the array
Perform PARTITION
Until the resulting split is lucky, i.e., not
worse than 1/10: 9/10
Recurse on both sub-arrays
September 20, 2004
L4.46
Analysis
Let T(n) be an upper bound on the expected
running time on any array of n elements
Consider any input of size n
The time needed to sort the input is bounded
from the above by a sum of
The time needed to sort the left subarray
The time needed to sort the right subarray
The number of iterations until we get a
lucky split, times cn
September 20, 2004
L4.47
Expectations
By linearity of expectation:
L4.48
Final bound
Can use the recursion tree argument:
Tree depth is (log n)
Total expected work at each level is at most
10/8 cn
The total expected time is (n log n)
L4.49
Lucky partitions
The probability that a random pivot induces
lucky partition is at least 8/10
(we are not lucky if the pivot happens to be
among the smallest/largest n/10 elements)
If we flip a coin, with heads prob. p=8/10 ,
the expected waiting time for the first head
is 1/p = 10/8
L4.50
Quicksort in practice
Quicksort is a great general-purpose
sorting algorithm.
Quicksort is typically over twice as fast
as merge sort.
Quicksort can benefit substantially from
code tuning.
Quicksort behaves well even with
caching and virtual memory.
Quicksort is great!
September 20, 2004
L4.51
More intuition
Suppose we alternate lucky, unlucky,
lucky, unlucky, lucky, .
L(n) = 2U(n/2) + (n) lucky
U(n) = L(n 1) + (n) unlucky
Solving:
L(n) = 2(L(n/2 1) + (n/2)) + (n)
= 2L(n/2 1) + (n)
= (n lg n) Lucky!
How can we make sure we are usually lucky?
September 20, 2004
L4.52
Randomized quicksort
analysis
Let T(n) = the random variable for the running
time of randomized quicksort on an input of size
n, assuming random numbers are independent.
For k = 0, 1, , n1, define the indicator
random variable
Xk =
L4.53
Analysis (continued)
T(0) + T(n1) + (n) if 0 : n1 split,
T(1) + T(n2) + (n) if 1 : n2 split,
M
T(n1) + T(0) + (n) if n1 : 0 split,
T(n) =
n 1
X k (T (k ) + T (n k 1) + (n)) .
k =0
L4.54
Calculating expectation
n 1
L4.55
Calculating expectation
n 1
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
Linearity of expectation.
L4.56
Calculating expectation
n 1
=
=
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
n 1
E[ X k ] E[T (k ) + T (n k 1) + (n)]
k =0
L4.57
Calculating expectation
n 1
=
=
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
n 1
E[ X k ] E[T (k ) + T (n k 1) + (n)]
k =0
n 1
n 1
n 1
= 1 E [T (k )] + 1 E [T (n k 1)] + 1 (n)
n k =0
n k =0
n k =0
L4.58
Calculating expectation
n 1
E[T (n)] = E X k (T (k ) + T ( n k 1) + (n) )
k =0
=
=
n 1
E[ X k (T (k ) + T (n k 1) + (n) )]
k =0
n 1
E[ X k ] E[T (k ) + T (n k 1) + (n)]
k =0
n 1
n 1
n 1
= 1 E [T (k )] + 1 E [T (n k 1)] + 1 (n)
n k =0
n k =0
n k =0
n 1
= 2 E [T (k )] + (n)
n k =1
September 20, 2004
Summations have
identical terms.
L4.59
Hairy recurrence
n 1
Use fact:
1 n 2 lg n 1n 2
k
lg
k
(exercise).
2
8
k =2
September 20, 2004
L4.60
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
Substitute inductive hypothesis.
L4.61
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
Use fact.
L4.62
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
= an lg n an (n)
4
L4.63
Substitution method
n 1
E [T (n)] 2 ak lg k + (n)
n k =2
= 2a 1 n 2 lg n 1 n 2 + (n)
n 2
8
= an lg n an (n)
4
an lg n ,
if a is chosen large enough so that
an/4 dominates the (n).
September 20, 2004
L4.64
Running
Running time
time
== O(n)
O(n) for
for nn
elements.
elements.
Assume
L4.65
Randomized Algorithms
Algorithms that make decisions based on
random coin flips.
Can fool the adversary.
The running time (or even correctness) is a
random variable; we measure the expected
running time.
We assume all random choices are
independent .
This is not the average case !
September 20, 2004
L4.66
Introduction to Algorithms
6.046J/18.401J
Lecture 5
Prof. Piotr Indyk
Today
Order statistics (e.g., finding median)
Two O(n) time algorithms:
Randomized: similar to Quicksort
Deterministic: quite tricky
Both are examples of divide and conquer
Introduction to Algorithms
L5.2
Order statistics
Select the ith smallest of n elements (the
element with rank i).
i = 1: minimum;
i = n: maximum;
i = (n+1)/2 or (n+1)/2: median.
How fast can we solve the problem ?
Min/max: O(n)
General i : O(n log n) by sorting
We will see how to do it in O(n) time
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.3
p
If i<k, recurse on the left
If i>k, recurse on the right
Otherwise, output x
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
xx
L5.4
Randomized Divide-andConquer
RAND-SELECT(A, p, r, i)
if p = r then return A[ p]
q RAND-PARTITION(A, p, r)
k=rank(A[q])
kqp+1
if i = k then return A[ q]
if i < k
then return RAND-SELECT(A, p, q 1, i )
else return RAND-SELECT(A, q + 1, r, i k )
k
A[q]
A[q]
Introduction to Algorithms
A[q]
A[q]
L5.5
Example
Select the i = 7th smallest:
66 10
10 13
13 55
pivot
Partition:
22 55
33
66
88
33
22 11
11
i=7
88 13
13 10
10 11
11
k=4
Introduction to Algorithms
L5.6
Analysis
What is the worst-case running time ?
Unlucky:
arithmetic series
T(n) = T(n 1) + (n)
= (n2)
Recall that a lucky partition splits into arrays
with size ratio at most 9:1
What if all partitions are lucky ?
Lucky:
n log10 / 9 1 = n 0 = 1
T(n) = T(9n/10) + (n)
CASE 3
= (n)
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.7
Introduction to Algorithms
L5.8
Digression: 9 to 1
Do we need to define the lucky partition as
9:1 balanced ?
No. Suffices to say that both sides have size
n , for 0< <
Probability of getting a lucky partition is
1-2
Introduction to Algorithms
L5.9
Introduction to Algorithms
L5.10
Partitioning subroutine
PARTITION(A, p, r) A[ p . . r]
x A[ p]
pivot = A[ p]
ip
for j p + 1 to r
do if A[ j] x
then i i + 1
exchange A[i] A[ j]
exchange A[ p] A[i]
return i
Invariant: xx
p
Piotr Indyk and Charles Leiserson
xx
xx
i
Introduction to Algorithms
??
j
L5.11
Summary of randomized
order-statistic selection
Works fast: linear expected time.
Excellent algorithm in practice.
But, the worst case is very bad: (n2).
Q. Is there an algorithm that runs in linear
time in the worst case?
A. Yes, due to [Blum-Floyd-Pratt-RivestTarjan73].
IDEA: Generate a good pivot recursively.
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.12
Introduction to Algorithms
Same as
RANDSELECT
L5.13
Introduction to Algorithms
L5.14
Introduction to Algorithms
L5.15
Introduction to Algorithms
L5.16
Introduction to Algorithms
L5.17
Analysis
lesser
greater
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.18
Analysis
lesser
greater
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.19
Analysis
Introduction to Algorithms
lesser
greater
L5.20
T(7n/10)
SELECT(i, n)
1. Divide the n elements into groups of 5. Find
the median of each 5-element group by rote.
2. Recursively SELECT the median x of the n/5
group medians to be the pivot.
3. Partition around the pivot x. Let k = rank(x).
4. if i = k then return x
elseif i < k
then recursively SELECT the ith
smallest element in the lower part
else recursively SELECT the (ik)th
smallest element in the upper part
Introduction to Algorithms
L5.21
1
7
T (n) cn + cn + (n)
5
10
18
cn + (n)
=
20
2
= cn cn (n)
20
cn
Introduction to Algorithms
L5.22
Minor simplification
For n 50, we have 3 n/10 n/4.
Therefore, for n 50 the recursive call to
SELECT in Step 4 is executed recursively
on 3n/4 elements.
Thus, the recurrence for running time
can assume that Step 4 takes time
T(3n/4) in the worst case.
For n < 50, we know that the worst-case
time is T(n) = (1).
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.23
Conclusions
Since the work at each level of recursion
is a constant fraction (18/20) smaller, the
work per level is a geometric series
dominated by the linear work at the root.
In practice, this algorithm runs slowly,
because the constant in front of n is large.
The randomized algorithm is far more
practical.
Exercise: Why not divide into groups of 3?
Piotr Indyk and Charles Leiserson
Introduction to Algorithms
L5.24
Introduction to Algorithms
6.046J/18.401J
Lecture 6
Prof. Piotr Indyk
Today: sorting
Show that (n lg n) is the best possible
running time for a sorting algorithm.
Design an algorithm that sorts in O(n) time.
Hint: different models ?
Introduction to Algorithms
L6.2
Comparison sort
All the sorting algorithms we have seen so far
are comparison sorts: only use comparisons to
determine the relative order of elements.
E.g., insertion sort, merge sort, quicksort,
heapsort.
Introduction to Algorithms
L6.3
Partitioning subroutine
PARTITION(A, p, r) A[ p . . r]
x A[ p]
pivot = A[ p]
ip
for j p + 1 to r
do if A[ j] x
then i i + 1
exchange A[i] A[ j]
exchange A[ p] A[i]
return i
Invariant: xx
p
Charles E. Leiserson and Piotr Indyk
xx
xx
i
Introduction to Algorithms
??
j
r
September 27, 2004
L6.4
Comparison sort
All of our algorithms used comparisons
All of our algorithms have running time
(n lg n)
Is it the best that we can do using just
comparisons ?
Answer: YES, via decision trees
Introduction to Algorithms
L6.5
Decision-tree example
Sort a1, a2, , an
(n=3)
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321
Introduction to Algorithms
L6.6
Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321
Introduction to Algorithms
L6.7
Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
94
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321
Introduction to Algorithms
L6.8
Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
96
2:3
2:3
231
231
321
321
Introduction to Algorithms
L6.9
Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213 4 6 2:3
2:3
1:3
1:3
132
132
312
312
231
231
321
321
Introduction to Algorithms
L6.10
Decision-tree example
Sort a1, a2, a3
= 9, 4, 6 :
1:2
1:2
2:3
2:3
123
123
1:3
1:3
213
213
1:3
1:3
132
132
312
312
2:3
2:3
231
231
321
321
469
Each leaf contains a permutation (1), (2),, (n) to
indicate that the ordering a(1) a(2) L a(n) has been
established.
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.11
Decision-tree model
A decision tree can model the execution of
any comparison sort:
One tree for each input size n.
View the algorithm as splitting whenever it compares two
elements.
The tree contains the comparisons along all possible
instruction traces.
The number of comparisons done by the algorithm on a given
input =
the length of the path taken.
Worst-case number of comparisons =
max path length = height of tree.
Worst-case time worst-case number of comparisons
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.12
Introduction to Algorithms
L6.13
Introduction to Algorithms
L6.14
Proof, ctd.
2h
n!
n*(n-1)** n/2
(n/2)n/2
lg( (n/2)n/2 )
(n/2) (lg n lg 2)
(n lg n) .
Introduction to Algorithms
L6.15
Introduction to Algorithms
L6.16
1:2
1:2
2:3
2:3
123
123
213
213
1:3
1:3
132
132
1:3
1:3
312
312
Introduction to Algorithms
2:3
2:3
231
231
321
321
L6.17
Introduction to Algorithms
L6.18
Counting sort
for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
for i 2 to k
do C[i] C[i] + C[i1]
C[i] = |{key i}|
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.19
Counting-sort example
A:
44
11
33
44
33
C:
B:
Introduction to Algorithms
L6.20
Loop 1
A:
44
11
33
44
33
C:
00
00
00
00
B:
for i 1 to k
do C[i] 0
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.21
Loop 2
A:
44
11
33
44
33
C:
00
00
00
11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.22
Loop 2
A:
44
11
33
44
33
C:
11
00
00
11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.23
Loop 2
A:
44
11
33
44
33
C:
11
00
11
11
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.24
Loop 2
A:
44
11
33
44
33
C:
11
00
11
22
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.25
Loop 2
A:
44
11
33
44
33
C:
11
00
22
22
B:
for j 1 to n
do C[A[ j]] C[A[ j]] + 1 C[i] = |{key = i}|
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.26
Loop 3
A:
44
11
33
44
33
B:
C:
11
00
22
22
C':
11
11
22
22
for i 2 to k
do C[i] C[i] + C[i1]
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.27
Loop 3
A:
44
11
33
44
33
B:
C:
11
00
22
22
C':
11
11
33
22
for i 2 to k
do C[i] C[i] + C[i1]
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.28
Loop 3
A:
44
11
33
44
33
B:
C:
11
00
22
22
C':
11
11
33
55
for i 2 to k
do C[i] C[i] + C[i1]
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.29
Loop 4
A:
44
11
33
44
33
B:
C':
11
11
33
55
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.30
Loop 4
A:
B:
44
11
33
44
33
33
C':
11
11
33
55
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.31
Loop 4
A:
B:
44
11
33
44
33
33
C':
11
11
22
55
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.32
Loop 4
A:
B:
44
11
33
44
33
33
44
C':
11
11
22
55
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.33
Loop 4
A:
B:
44
11
33
44
33
33
44
C':
11
11
22
44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.34
Loop 4
A:
B:
44
11
33
44
33
33
33
44
C':
11
11
22
44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.35
Loop 4
A:
B:
44
11
33
44
33
33
33
44
C':
11
11
11
44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.36
Loop 4
A:
B:
44
11
33
44
33
11
33
33
44
C':
11
11
11
44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.37
Loop 4
A:
B:
44
11
33
44
33
11
33
33
44
C':
00
11
11
44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.38
Loop 4
A:
B:
44
11
33
44
33
11
33
33
44
44
C':
00
11
11
44
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.39
Loop 4
A:
B:
44
11
33
44
33
11
33
33
44
44
C':
00
11
11
33
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.40
B vs C
B:
11
33
33
44
44
C':
11
11
33
55
Introduction to Algorithms
L6.41
Analysis
(k)
(n)
(k)
(n)
for i 1 to k
do C[i] 0
for j 1 to n
do C[A[ j]] C[A[ j]] + 1
for i 2 to k
do C[i] C[i] + C[i1]
for j n downto 1
do B[C[A[ j]]] A[ j]
C[A[ j]] C[A[ j]] 1
(n + k)
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.42
Running time
If k = O(n), then counting sort takes (n) time.
But, sorting takes (n lg n) time!
Why ?
Answer:
Comparison sorting takes (n lg n) time.
Counting sort is not a comparison sort.
In fact, not a single comparison between
elements occurs!
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.43
Stable sorting
Counting sort is a stable sort: it preserves
the input order among equal elements.
A:
44
11
33
44
33
B:
11
33
33
44
44
Introduction to Algorithms
L6.44
Sorting integers
We can sort n integers from {1, 2, , k} in
O(n+k) time
This is nice if k=O(n)
What if, say, k=n2 ?
Introduction to Algorithms
L6.45
Radix sort
Origin: Herman Holleriths card-sorting
machine for the 1890 U.S. Census. (See
Appendix .)
Digit-by-digit sort.
Holleriths original (bad) idea: sort on
most-significant digit first.
Good idea: Sort on least-significant digit
first with auxiliary stable sort.
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.46
720
355
436
457
657
329
839
720
329
436
839
355
457
657
Introduction to Algorithms
329
355
436
457
657
720
839
September 27, 2004
L6.47
Introduction to Algorithms
720
329
436
839
355
457
657
329
355
436
457
657
720
839
L6.48
Introduction to Algorithms
720
329
436
839
355
457
657
329
355
436
457
657
720
839
L6.49
Introduction to Algorithms
720
329
436
839
355
457
657
329
355
436
457
657
720
839
L6.50
Introduction to Algorithms
L6.51
Analysis (continued)
Recall: Counting sort takes (n + k) time to
sort n numbers in the range from 0 to k 1.
If each b-bit word is broken into r-bit pieces,
each pass of counting sort takes (n + 2r) time.
Since there are b/r passes, we have
T (n, b) = b (n + 2 r ) .
r
Choose r to minimize T(n, b):
Increasing r means fewer passes, but as
r >> lg n, the time grows exponentially.
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.52
Choosing r
T (n, b) = b (n + 2 r )
r
Minimize T(n, b) by differentiating and setting to 0.
Or, just observe that we dont want 2r >> n, and
theres no harm asymptotically in choosing r as
large as possible subject to this constraint.
Choosing r = lg n implies T(n, b) = (bn/lg n) .
For numbers in the range from 0 to n d 1, we
have b = d lg n radix sort runs in (d n) time.
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.53
Conclusions
In practice, radix sort is fast for large inputs, as
well as simple to code and maintain.
Example (32-bit numbers):
At most 3 passes when sorting 2000 numbers.
Merge sort and quicksort do at least lg 2000 =
11 passes.
Downside: Unlike quicksort, radix sort displays
little locality of reference.
Introduction to Algorithms
L6.54
Appendix: Punched-card
technology
Herman Hollerith (1860-1929)
Punched cards
Holleriths tabulating system
Operation of the sorter
Origin of radix sort
Modern IBM card
Web resources on punchedcard technology
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
Return to last
slide viewed.
September 27, 2004
L6.55
Herman Hollerith
(1860-1929)
The 1880 U.S. Census took almost
10 years to process.
While a lecturer at MIT, Hollerith
prototyped punched-card technology.
His machines, including a card sorter, allowed
the 1890 census total to be reported in 6 weeks.
He founded the Tabulating Machine Company in
1911, which merged with other companies in 1924
to form International Business Machines.
Image removed due to copyright considerations.
Introduction to Algorithms
L6.56
Punched cards
Punched card = data record.
Hole = value.
Algorithm = machine + human operator.
Introduction to Algorithms
L6.57
Holleriths
tabulating
system
Pantograph card
punch
Hand-press reader
Dial counters
Sorting box
Charles E. Leiserson and Piotr Indyk
Introduction to Algorithms
L6.58
Introduction to Algorithms
L6.59
Introduction to Algorithms
L6.60
Introduction to Algorithms
L6.61
Introduction to Algorithms
L6.62
Introduction to Algorithms
6.046J/18.401J
Lecture 7
Prof. Piotr Indyk
Data Structures
Role of data structures:
Encapsulate data
Support certain operations (e.g., INSERT,
DELETE, SEARCH)
What data structures do we know already ?
Yes, heap:
INSERT(x)
DELETE-MIN
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.2
Dictionary problem
Dictionary T holding n records:
x
record
key[x]
key[x]
Other fields
containing
satellite data
Operations on T:
INSERT(T, x)
DELETE(T, x)
SEARCH(T, k)
Introduction to Algorithms
L7.3
Assumptions
Assumptions:
The set of keys is K U = {0, 1, , u1}
Keys are distinct
What can we do ?
Introduction to Algorithms
L7.4
T[k] =
x
NIL
if k K and key[x] = k,
otherwise.
Benefit:
Each operation takes constant time
Drawbacks:
The range of keys can be large:
64-bit numbers (which represent
18,446,744,073,709,551,616 different keys),
character strings (even larger!)
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.5
Hash functions
Solution: Use a hash function h to map the
universe U of all keys into
T
{0, 1, , m1}:
0
k1
K
k2
k4
h(k1)
h(k4)
k5
h(k2) = h(k5)
k3
h(k3)
m1
Introduction to Algorithms
L7.6
Collisions resolution by
chaining
Records in the same slot are linked into a list.
T
49
49
86
86
52
52
Introduction to Algorithms
L7.7
Hash functions
Designing good functions is quite nontrivial
For now, we assume they exist. Namely, we
assume simple uniform hashing:
Each key k K of keys is equally likely
to be hashed to any slot of table T,
independent of where other keys are
hashed
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.8
Analysis of chaining
Let n be the number of keys in the table, and
let m be the number of slots.
Define the load factor of T to be
= n/m
= average number of keys per slot.
Introduction to Algorithms
L7.9
Search cost
Expected time to search for a record with
a given key = (1 + ).
apply hash
function and
access slot
search
the list
Introduction to Algorithms
L7.10
Other operations
Insertion time ?
Constant: hash and add to the list
Deletion time ? Recall that we defined
DELETE(T, x)
Also constant, if x has a pointer to the
collision list and list doubly linked
Otherwise, do SEARCH first
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.11
Delete
key[x]
key[x]
T
49
49
86
86
Introduction to Algorithms
52
52
L7.12
Hashing in practice
Introduction to Algorithms
L7.14
Division method
Define
h(k) = k mod m.
Deficiency: Dont pick an m that has a small
divisor d. A preponderance of keys that are
congruent modulo d can adversely affect
uniformity.
Extreme deficiency: If m = 2r, then the hash
doesnt even depend on all the bits of k:
If k = 10110001110110102 and r = 6, then
h(k) = 0110102 .
h(k)
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.15
Introduction to Algorithms
L7.16
Multiplication method
Assume that all keys are integers, m = 2r, and our
computer has w-bit words. Define
h(k) = (Ak mod 2w) rsh (w r),
where rsh is the bit-wise right-shift operator
and A is an odd integer in the range 2w1 < A < 2w.
Dont pick A too close to 2w.
Multiplication modulo 2w is fast.
The rsh operator is fast.
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.17
Multiplication method
example
h(k) = (Ak mod 2w) rsh (w r)
Suppose that m = 8 = 23 and that our computer
has w = 7-bit words:
1011001 =A
1101011 =k
10010100110011
A
h(k)
0
7 1
5 4 3
Introduction to Algorithms
.
2A
Modular wheel
Charles Leiserson and Piotr Indyk
3A
L7.18
Introduction to Algorithms
L7.19
A weakness of hashing as we
saw it
Problem: For any hash function h, a set
of keys exists that can cause the average
access time of a hash table to skyrocket.
An adversary can pick all keys from
h-1(i) ={k U : h(k) = i} for a slot i.
There is a slot i for which |h-1(i)| u/m
Introduction to Algorithms
L7.20
Solution
Randomize!
Choose the hash function at random from
some family of function, and independently
of the keys.
Even if an adversary can see your code, he
or she cannot find a bad set of keys, since
he or she doesnt know exactly which hash
function will be chosen.
What family of functions should we select ?
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.21
Introduction to Algorithms
L7.22
Universal hashing
Idea #2: Universal Hashing
Let H be a finite collection of hash
functions, each mapping U to {0, 1, , m1}
We say H is universal if for all x, y U,
where x y, we have
PrhH{h(x) = h(y)}| = 1/m.
Introduction to Algorithms
L7.23
Universality is good
Theorem. Let h be a hash function chosen
(uniformly) at random from a universal set H
of hash functions. Suppose h is used to hash
n arbitrary keys into the m slots of a table T.
Then, for a given key x, we have
E[#collisions with x] < n/m.
Introduction to Algorithms
L7.24
Proof of theorem
Proof. Let Cx be the random variable denoting
the total number of collisions of keys in T with
x, and let
1 if h(x) = h(y),
cxy =
0 otherwise.
Note: E[cxy] = 1/m and C x =
Introduction to Algorithms
cxy .
yT {x}
L7.25
Proof (continued)
E[C x ] = E c xy
yT { x}
Take expectation
of both sides.
Introduction to Algorithms
L7.26
Proof (continued)
E[C x ] = E c xy
yT { x}
=
E[cxy ]
yT { x}
Take expectation
of both sides.
Linearity of
expectation.
Introduction to Algorithms
L7.27
Proof (continued)
E[C x ] = E c xy
yT { x}
=
E[cxy ]
Linearity of
expectation.
1/ m
E[cxy] = 1/m.
yT { x}
Take expectation
of both sides.
yT { x}
Introduction to Algorithms
L7.28
Proof (continued)
E[C x ] = E c xy
yT { x}
=
E[cxy ]
Linearity of
expectation.
1/ m
E[cxy] = 1/m.
yT { x}
Take expectation
of both sides.
yT { x}
= n 1 .
m
Charles Leiserson and Piotr Indyk
Algebra.
Introduction to Algorithms
L7.29
Define ha (k ) = ai ki mod m
i =0
Introduction to Algorithms
L7.30
Universality of dot-product
hash functions
Theorem. The set H = {ha} is universal.
Introduction to Algorithms
L7.31
Proof (continued)
r
i =0
i =0
ha ( x) = ha (b) ai xi ai yi
r
(mod m)
ai ( xi yi ) 0
(mod m)
a0 ( x0 y0 ) + ai ( xi yi ) 0
(mod m)
i =0
r
i =1
a0 ( x0 y0 ) ai ( xi yi )
(mod m. )
i =1
Introduction to Algorithms
L7.32
Recall PS 2
Theorem. Let m be prime. For any z Zm
such that z 0, there exists a unique z1 Zm
such that
z z1 1 (mod m).
Introduction to Algorithms
L7.33
a0 ( x0 y0 ) ai ( xi yi )
(mod m) ,
i =1
a0 ai ( xi yi ) ( x0 y0 ) 1
i =1
(mod m) .
Introduction to Algorithms
L7.34
Proof (completed)
Q. What is the probability that x and y
collide?
A. There are m choices for a0, but exactly one
choice for a0 causes x and y to collide,
namely
1
a0 = ai ( xi yi ) ( x0 y0 ) mod m .
i =1
Introduction to Algorithms
L7.35
Recap
Showed how to implement dictionary so
that INSERT, DELETE, SEARCH work in
expected constant time under the uniform
hashing assumption
Relaxed the assumption to universal
hashing
Constructed universal hashing for keys in
{0mr -1}
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.36
Perfect hashing
Given a set of n keys, construct a static hash
table of size m = O(n) such that SEARCH takes
(1) time in the worst case.
IDEA: Twolevel scheme
with universal
hashing at
both levels.
No collisions
at level 2!
Charles Leiserson and Piotr Indyk
T
0
1 44 31
31
2
3
4 11 00
00
5
6 99 86
86
m a
S1
14
27
1427
S4
26
26
h31(14) = h31(27) = 1
S6
40
22
40 37
37
22
0 1 2 3 4 5 6 7 8
Introduction to Algorithms
L7.37
Collisions at level 2
Theorem. Let H be a class of universal hash
functions for a table of size m = n2. Then, if we
use a random h H to hash n keys into the table,
the expected number of collisions is at most 1/2.
Proof. By the definition of universality, the
probability that 2 given keys in the table collide
n
2
under h is 1/m = 1/n . Since there are (2 ) pairs
of keys that can possibly collide, the expected
number of collisions is
n 1
n(n 1) 1
2 < 1.
2 =
2
2
n
2 n
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.38
No collisions at level 2
Corollary. The probability of no collisions
is at least 1/2.
Introduction to Algorithms
L7.39
Analysis of storage
For the level-1 hash table T, choose m = n, and
let ni be random variable for the number of keys
that hash to slot i in T. By using ni2 slots for the
level-2 hash table Si, the expected total storage
required for the two-level scheme is therefore
m1
2
E (ni ) = (n) ,
i =0
Introduction to Algorithms
L7.40
Introduction to Algorithms
L7.41
0. Probe h(496,0)
586
133
204
collision
481
m1
Introduction to Algorithms
L7.42
0. Probe h(496,0)
1. Probe h(496,1)
586
133
collision
204
481
m1
Introduction to Algorithms
L7.43
0. Probe h(496,0)
1. Probe h(496,1)
2. Probe h(496,2)
586
133
204
496
481
insertion
m1
Introduction to Algorithms
L7.44
586
133
204
496
481
Introduction to Algorithms
L7.45
Probing strategies
Linear probing:
Given an ordinary hash function h(k), linear
probing uses the hash function
h(k,i) = (h(k) + i) mod m.
This method, though simple, suffers from primary
clustering, where long runs of occupied slots build
up, increasing the average search time. Moreover,
the long runs of occupied slots tend to get longer.
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.46
Probing strategies
Double hashing
Given two ordinary hash functions h1(k) and h2(k),
double hashing uses the hash function
h(k,i) = (h1(k) + ih2(k)) mod m.
This method generally produces excellent results,
but h2(k) must be relatively prime to m. One way
is to make m a power of 2 and design h2(k) to
produce only odd numbers.
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.47
Introduction to Algorithms
L7.48
<
= for i = 1, 2, , n.
Observe that
mi m
Charles Leiserson and Piotr Indyk
Introduction to Algorithms
L7.49
Proof (continued)
Therefore, the expected number of probes is
2
1
n
n
n
1
1 + 1 +
L
L 1 +
1 +
m m 1 m 2 m n + 1
1 + (1 + (1 + (L (1 + )L)))
1+ + 2 +3 +L
=
i =0
= 1 .
1
Charles Leiserson and Piotr Indyk
L7.50
Introduction to Algorithms
L7.51
Dot-product method
Randomized strategy:
Let m be prime. Decompose key k into r + 1
digits, each with value in the set {0, 1, , m1}.
That is, let k = k0, k1, , km1, where 0 ki < m.
Pick a = a0, a1, , am1 where each ai is chosen
randomly from {0, 1, , m1}.
Define ha (k ) =
ai ki mod m.
i =0
Introduction to Algorithms
L7.52
Introduction to Algorithms
6.046J/18.401J
Lecture 8
Data structures
Introduction to Algorithms
October 6, 2004
L7.2
key[x]
Pointers:
left[x]
right[x]
p[x]
9
5
1
12
6
7
8
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.3
Introduction to Algorithms
9
5
1
12
6
7
8
October 6, 2004
L7.4
No uniqueness
5
1
9
6
5
12
12
6
7
8
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.5
Sort !
Inorder-Walk(x):
If xNIL then
9
5
Inorder-Walk( left[x] )
print key[x]
Inorder-Walk( right[x] )
Output:
Piotr Indyk
12
6
7
Introduction to Algorithms
12
October 6, 2004
L7.6
Sorting, ctd.
Introduction to Algorithms
9
5
1
12
6
7
8
October 6, 2004
L7.7
Sorting, ctd.
9
5
1
12
6
7
8
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.8
Operations:
Insert(x)
Delete(x)
Search(k)
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.9
Search
Search(x):
If xNIL then
If key[x] = k then return x
If k < key[x] then return
Search( left[x] )
Search( right[x] )
Piotr Indyk
Introduction to Algorithms
9
5
1
12
6
7
Search(8):
8
Search(8.5):
October 6, 2004
L7.10
Predecessor/Successor
x=Search(k)
If key[x]<k, then return Successor(x)
Else return x
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.11
Successor
Successor(x):
yx
If right[x] NIL then
5
12
return Minimum( right[x] )
yx
Otherwise
1
6
y p[x]
While yNIL and x=right[y] do
7 xy
xy
8
y p[y]
Return y
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.12
Minimum
Minimum( x )
While left[x]NIL do
x left[x]
Return x
9
5
1
12
6
7
8
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.13
Nearest Neighbor
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.14
Analysis
9
5
1
12
6
7
8
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.15
Constructing BST
Insert(z):
y NIL
x root
While x NIL do
y x
then x left[x]
else x right[x]
p[z] y
then left[y] z
else right[y] z
Piotr Indyk
9
5
12
1
6 y
z 5.5
7
Insert(8.5)
Insert(5.5)
8
Introduction to Algorithms
8.5
October 6, 2004
L7.16
Analysis
1
After we insert n elements,
what is the worst possible
BST height ?
Pretty bad: n-1
Piotr Indyk
Introduction to Algorithms
2
3
4
5
6
October 6, 2004
L7.17
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.18
1 2 3 4 5 6 7 8 9
1 2
4 5 6 7 8 9
Piotr Indyk
4 5
7 8 9
7
7
Introduction to Algorithms
9
9
October 6, 2004
L7.19
Observations
partition
Introduction to Algorithms
October 6, 2004
L7.20
Partitions
Introduction to Algorithms
October 6, 2004
L7.21
Concentration inequalities
t
Pr[hx >t] (k) /2t-k
(et/k)k/2t-k
= (ce)log n/2(c-1) log n
= 2lg(ce) log n/2 (c-1) log n
= 2 [lg(ce) (c-1)] * (lg n)/ lg(4/3)
2 -1.1 lg n = 1/n1.1, for sufficient c
4/3
4/3
4/3
Piotr Indyk
4/3
Introduction to Algorithms
October 6, 2004
L7.22
Final Analysis
n * 1/n1.1
= 1/n0.1
Introduction to Algorithms
October 6, 2004
L7.23
Summing up
Piotr Indyk
Introduction to Algorithms
October 6, 2004
L7.24
Introduction to Algorithms
6.046J/18.401J
Lecture 9
Today
2
3
4
5
6
Introduction to Algorithms
L9.2
Examples:
AVL trees
2-3 trees
2-3-4 trees
B-trees
Red-black trees
Introduction to Algorithms
L9.3
Red-black trees
Introduction to Algorithms
L9.4
18
18
NIL
10
10
88
22
22
11
11
Introduction to Algorithms
NIL
26
26
NIL
NIL
L9.5
Introduction to Algorithms
L9.6
18
18
10
10
88
22
22
11
11
26
26
7.5
7.5
Introduction to Algorithms
L9.7
black trees ?
Introduction to Algorithms
L9.8
Rotations
RIGHT-ROTATE(B)
BB
LEFT-ROTATE(A)
AA
AA
BB
a , b , c a A b B c.
Introduction to Algorithms
L9.9
B
2
A 1
LEFT-ROTATE(A)
AA
BB
B
3
BB
AA
Introduction to Algorithms
L9.10
O(log n) re-colorings
1 rotation
Introduction to Algorithms
L9.11
2-3 Trees
Introduction to Algorithms
L9.12
2-3 Trees
12
6
5
Introduction to Algorithms
8
6
12
8
9 12
L9.13
Internal nodes
Internal nodes:
Values:
Leaves:
x.max : the key
Piotr Indyk and Charles E. Leiserson
Introduction to Algorithms
L9.14
Introduction to Algorithms
L9.15
Searching
key k ?
Search(x,k):
If x=NIL then return NIL
Else if x is a leaf then
If x.max=k then return x
Else return NIL
Else
If k left[x].max
then Search(left[x],k)
Else if k mid[x].max
then Search(mid[x],k)
Else Search(right[x],k)
12
6
1
8
6
12
8
9 12
Search(8)
Search(13)
Introduction to Algorithms
L9.16
Insertion
12
13
How to insert x ?
key of x
6
8
node
Insert x into y in a
1 5 6 7 8
sorted order
5.5
7.5
max values on the path
to root
Insert(7.5)
slide)
12
13
9 12
13
Insert(5.5)
Introduction to Algorithms
L9.17
Insertion, ctd.
12
13
previous slide)
If y has 4 children,
then Split(y)
6 y
1
6
5.5
x
Introduction to Algorithms
8
7
12
13
8
7.5
9 12
13
L9.18
Split
y1, y2
y
a
y1
d
z
*If
y2
c
L9.19
Split
12
13
6
1
8
6
5.5
12
13
8
9 12
7.5
Introduction to Algorithms
13
L9.20
Split
12
13
5.5
1
5 5.5 6
8
7
12
13
8
9 12
7.5
Introduction to Algorithms
13
L9.21
Split
13
6
5.5
1
13
5 5.5 6
8
7
12
13
8
9 12
7.5
13
Insert and Split preserve heights, unless new root is created, in which case all heights are
increased by 1
Introduction to Algorithms
L9.22
Delete
12
How to delete x ?
6
12
Let y=p(x)
z
Remove x from y
5.5
6
8 y 12
If y has 1 child:
Remove y
1 5 5.5 6 7 8 9 12
Attach x to ys sibling z
x
Delete(8)
Introduction to Algorithms
L9.23
Delete
12
How to delete x ?
6
12
Let y=p(x)
z
Remove x from y
5.5
6
12
If y has 1 child:
Remove y
1 5 5.5 6 7
9 12
Attach x to ys sibling z
If z has 4 children, then
Delete(8)
Split(z)
INCOMPLETE SEE THE END FOR FULL VERSION
Introduction to Algorithms
L9.24
Summing up
2-3 Trees:
O(log n) depth Search in O(log n) time
Introduction to Algorithms
L9.25
2-3-4 Trees
5 9
1 2 4
7 8
Introduction to Algorithms
10 12
L9.26
h 2 lg(n + 1).
INTUITION:
Introduction to Algorithms
L9.27
h 2 lg(n + 1).
INTUITION:
Introduction to Algorithms
L9.28
h 2 lg(n + 1).
INTUITION:
Introduction to Algorithms
L9.29
h 2 lg(n + 1).
INTUITION:
Introduction to Algorithms
L9.30
h 2 lg(n + 1).
INTUITION:
Introduction to Algorithms
L9.31
h 2 lg(n + 1).
INTUITION:
Introduction to Algorithms
L9.32
Summing up
We have seen:
Red-black trees
2-3 trees (in detail)
2-3-4 trees
Red-black trees are undercover 2-3-4 trees
In most cases, does not matter what you use
Introduction to Algorithms
L9.33
Problem: there is an
6
5.5
12
5 5.5 6
Introduction to Algorithms
12
12
7
9 12
L9.34
Introduction to Algorithms
L9.35
Let y=p(x)
Remove x
If yroot then
Let z be the sibling of y.
Assume z is the right sibling of y, otherwise the code is
symmetric.
Else
*Note
Introduction to Algorithms
L9.36
Example
12
6
5.5
1
5 5.5 6
Introduction to Algorithms
12
8
7
12
8
9 12
L9.37
Example, ctd.
12
6
5.5
1
5 5.5 6
Introduction to Algorithms
12
8
7
12
9 12
L9.38
Example, ctd.
12
12
5.5
1
5 5.5 6
Introduction to Algorithms
12
12
7
9 12
L9.39
Example, ctd.
12
5.5
1
5 5.5 6
Introduction to Algorithms
12
7
9 12
L9.40
Introduction to Algorithms
6.046J/18.401J/SMA5503
Lecture 10
Prof. Piotr Indyk
Today
A data structure for a new problem
Amortized analysis
Introduction to Algorithms
L10.2
12
6
12
Problem: there is
an internal node
that has only 1
5.5
6
12
child
Solution: delete 1 5 5.5 6 7
9 12
recursively
Introduction to Algorithms
L10.3
Example
12
6
5.5
1
5 5.5 6
Introduction to Algorithms
12
8
7
12
8
9 12
L10.4
Example, ctd.
12
6
5.5
1
5 5.5 6
Introduction to Algorithms
12
8
7
12
9 12
L10.5
Example, ctd.
12
12
5.5
1
5 5.5 6
Introduction to Algorithms
12
12
7
9 12
L10.6
Example, ctd.
12
5.5
1
5 5.5 6
Introduction to Algorithms
12
7
9 12
L10.7
Let y=p(x)
Remove x
If yroot then
Let z be the sibling of y.
Assume z is the right sibling of y, otherwise the code is
symmetric.
If y has only 1 child w left
Case 1: z has 3 children
Attach left[z] as the rightmost child of y
Update y.max and z.max
Case 2: z has 2 children:
Attach the child w of y as the leftmost child of z
Update z.max
Delete(y) (recursively*)
Else
Update max of y, p(y), p(p(y)) and so on until root
Else
If root has only one child u
Remove root
Make u the new root
*Note
Introduction to Algorithms
L10.8
2-3 Trees
The simplest balanced trees on the planet!
(but, nevertheless, not completely trivial)
Introduction to Algorithms
L10.9
Introduction to Algorithms
1
3
2
6
L10.10
Introduction to Algorithms
L10.11
Quiz
If we have a WEAKUNION( x, y) that works
only if x, y are representatives, how can we
implement UNION that works for any x, y ?
UNION( x, y)
=WEAKUNION( FIND-SET(x) , FIND-SET(y) )
Introduction to Algorithms
L10.12
Representation
x
Data
Other fields containing
data of our choice
Introduction to Algorithms
L10.13
Applications
1
Data clustering
Killer App: Minimum
Spanning Tree
(Lecture 13)
Amortized analysis
2
6
Introduction to Algorithms
L10.14
Ideas ?
How can we implement this data structure
efficiently ?
MAKE-SET
UNION
FIND-SET
Introduction to Algorithms
L10.15
Introduction to Algorithms
n+1
2n
L10.16
x1
rep[Si]
x2
xk
How
it ?x as a lone node.
M
AKEcan
-SETwe
(x)improve
initializes
(1)
FIND-SET(x) walks left in the list containing x
(n)
until it reaches the front of the list.
UNION(x, y) concatenates the lists containing
(n)
x and y, leaving rep. as FIND-SET[x].
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
L10.17
Si :
x1
x2
xk
Introduction to Algorithms
L10.18
Example of
augmented linked-list solution
rep
Sx :
x1
rep[Sx]
x2
rep
Sy :
y1
rep[Sy]
Introduction to Algorithms
y2
y3
L10.19
Example of
augmented linked-list solution
Sx Sy :
x1
rep[Sx]
rep
x2
rep
y1
rep[Sy]
Introduction to Algorithms
y2
y3
L10.20
Example of
augmented linked-list solution
rep
Sx Sy :
x1
rep[Sx Sy]
x2
y1
Introduction to Algorithms
y2
y3
L10.21
Si :
x1
x2
xk
Introduction to Algorithms
L10.22
Amortized analysis
So far, we focused on worst-case time of each operation.
E.g., UNION takes (n) time for some operations
Amortized analysis: count the total time spent by any
sequence of operations
Total time is always at most
worst-case-time-per-operation * #operations
but it can be much better!
E.g., if times are 1,1,1,,1,n,1,,1
Can we modify the linked-list data structure so that any
sequence of m MAKE-SET, FIND-SET, UNION
operations cost less than m*(n) time?
Introduction to Algorithms
L10.23
Alternative
UNION(x, y) :
concatenates the lists containing y and x, and
update the rep pointers for all elements in the
list containing y x
rep
rep
Sy :
y1
rep[Sy]
y2
Sx :
x1
rep[Sx]
x2
y3
Introduction to Algorithms
L10.24
Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep
Sx Sy :
y1
rep[Sy]
x1
rep[Sx]
rep
y2
x2
y3
Introduction to Algorithms
L10.25
Alternative concatenation
UNION(x, y) could instead
concatenate the lists containing y and x, and
update the rep pointers for all elements in the
list containing x.
rep
Sx Sy :
y1
rep[Sx Sy]
x1
rep
y2
x2
y3
Introduction to Algorithms
L10.26
Introduction to Algorithms
L10.27
Introduction to Algorithms
L10.28
Final Result
We have a data structure for dynamic sets which
supports:
MAKE-SET: O(1) worst case
FIND-SET: O(1) worst case
UNION:
Any sequence of any m operations* takes O(m log n) time, or
the amortized complexity of the operations* is O(log n)
*
Introduction to Algorithms
L10.29
Amortized vs Average
What is the difference between average case
complexity and amortized complexity ?
Average case assumes random
distribution over the input (e.g., random
sequence of operations)
Amortized means we count the total
time taken by any sequence of m
operations (and divide it by m)
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
L10.30
Can we do better ?
One can do:
MAKE-SET: O(1) worst case
FIND-SET: O(lg n) worst case
WEAKUNION: O(1) worst case
Thus, UNION: O(lg n) worst case
Introduction to Algorithms
L10.31
Introduction to Algorithms
L10.32
Time Analysis
MAKE-SET(x) initializes x
O(1)
as a lone node.
FIND-SET(x) walks up the
O(depth) = ?
tree containing x until it
reaches the root.
WEAKUNION(x, y)
O(1)
concatenates
the trees containing x and y
Introduction to Algorithms
L10.33
Introduction to Algorithms
y1
y4
y3
L10.34
Introduction to Algorithms
L10.35
Tree implementation
We have:
MAKE-SET: O(1) worst case
FIND-SET: O(depth) = O(lg n) worst case
WEAKUNION: O(1) worst case
Can amortized analysis buy us anything ?
Need another trick
Introduction to Algorithms
L10.36
Introduction to Algorithms
y2
y5
L10.37
Introduction to Algorithms
y2
y5
L10.38
Introduction to Algorithms
L10.39
The Theorem
Theorem: In general, amortized cost is O((n)),
where (n) grows really, really, really slow.
Introduction to Algorithms
L10.40
Ackermanns function A
j + 1 if k = 0
Define Ak ( j ) = ( j +1)
Ak 1 ( j ) if k 1 -iterate Ak-1() j+1 times
A0( j) = j + 1
A1( j) = A0((A0( j)) ~2j
A2( j) = A1(A1( j)) ~2j 2j
.2
A3( j ) > 22
..
A0(1) = 2
A1(1) = 3
A2(1) = 7
A3(1) = 2047
.2
A4(1) > 2
A4( j) is a lot bigger.
Define (n) = min {k : Ak(1) n}.
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
2047
..
2048
L10.41
The Theorem
Theorem: In general, amortized cost is O((n)),
where (n) grows really, really, really slow.
Proof: Really, really, really long (CLRS, p. 509)
Introduction to Algorithms
L10.42
Application:
Dynamic connectivity
Suppose a graph is given to us incrementally by
ADD-VERTEX(v)
ADD-EDGE(u, v)
and we want to support connectivity queries:
CONNECTED(u, v):
Are u and v in the same connected component?
For example, we want to maintain a spanning forest,
so we check whether each new edge connects a
previously disconnected pair of vertices.
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
L10.43
Application:
Dynamic connectivity
Sets of vertices represent connected components.
Suppose a graph is given to us incrementally by
ADD-VERTEX(v) MAKE-SET(v)
ADD-EDGE(u, v) if not CONNECTED(u, v)
then UNION(v, w)
and we want to support connectivity queries:
CONNECTED(u, v): FIND-SET(u) = FIND-SET(v)
Are u and v in the same connected component?
For example, we want to maintain a spanning forest,
so we check whether each new edge connects a
previously disconnected pair of vertices.
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
L10.44
Introduction to Algorithms
x4
x2
x3
x5
L10.45
Plan of attack
We will build a simple disjoint-union data structure
that, in an amortized sense, performs significantly
better than (lg n) per op., even better than
(lg lg n), (lg lg lg n), etc., but not quite (1).
To reach this goal, we will introduce two key tricks.
Each trick converts a trivial (n) solution into a
simple (lg n) amortized solution. Together, the
two tricks yield a much better solution.
First trick arises in an augmented linked list.
Second trick arises in a tree structure.
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
L10.46
Introduction to Algorithms
L10.47
Introduction to Algorithms
L10.48
Introduction to Algorithms
L10.49
UNION(x, y)
Every tree has a rank
Rank is an upper bound for height
When we take UNION(x, y):
If rank[x] >rank[y] then link y to x
If rank[x] <rank[y] then link x to y
If rank[x]=rank[y] then
link x to y
rank[y]=rank[y]+1
Can show that 2rank(x) #elements in x (Exercise 21.4-2)
Therefore, height is O(log n)
2004 by Erik Demaine and Piotr Indyk
Introduction to Algorithms
L10.50
Introduction to Algorithms
6.046J/18.401J
LECTURE 11
Amortized analysis
Dynamic tables
Aggregate method
Accounting method
Potential method
Prof. Charles E. Leiserson
Introduction to Algorithms
L14.2
1
overflow
Introduction to Algorithms
L14.3
11
overflow
Introduction to Algorithms
L14.4
11
2
Introduction to Algorithms
L14.5
11
22
overflow
Introduction to Algorithms
L14.6
1
2
overflow
Introduction to Algorithms
L14.7
1
2
Introduction to Algorithms
L14.8
INSERT
INSERT
INSERT
INSERT
1
2
3
4
Introduction to Algorithms
L14.9
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
overflow
Introduction to Algorithms
L14.10
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
overflow
Introduction to Algorithms
L14.11
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
Introduction to Algorithms
L14.12
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
INSERT
1
2
3
4
5
6
7
Introduction to Algorithms
L14.13
Worst-case analysis
Consider a sequence of n insertions. The
worst-case time to execute one insertion is
(n). Therefore, the worst-case time for n
insertions is n (n) = (n2).
WRONG! In fact, the worst-case cost for
n insertions is only (n) (n2).
Lets see why.
Introduction to Algorithms
L14.14
Tighter analysis
Let ci = the cost of the i th insertion
i if i 1 is an exact power of 2,
=
1 otherwise.
i
sizei
16 16
ci
Introduction to Algorithms
10
1
L14.15
Tighter analysis
Let ci = the cost of the i th insertion
i if i 1 is an exact power of 2,
=
1 otherwise.
i
sizei
16 16
1
1
1
2
1
4
1
8
ci
Introduction to Algorithms
10
1
L14.16
Cost of n insertions = ci
i =1
n+
lg( n 1)
2j
j =0
3n
= ( n ) .
Thus, the average cost of each dynamic-table
operation is (n)/n = (1).
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.17
Amortized analysis
An amortized analysis is any strategy for
analyzing a sequence of operations to
show that the average cost per operation is
small, even though a single operation
within the sequence might be expensive.
Even though were taking averages, however,
probability is not involved!
An amortized analysis guarantees the
average performance of each operation in
the worst case.
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.18
Introduction to Algorithms
L14.19
Accounting method
Charge i th operation a fictitious amortized cost
i, where $1 pays for 1 unit of work (i.e., time).
This fee is consumed to perform the operation.
Any amount not immediately consumed is stored
in the bank for use by subsequent operations.
The bank balance must not go negative! We
must ensure that n
n
ci ci
i =1
i =1
for all n.
Thus, the total amortized costs provide an upper
bound on the total true costs.
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.20
Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
$0
$0 $0
$0 $0
$0 $2
$2 $2
$2 $2 $2 overflow
$0 $0
Introduction to Algorithms
L14.21
Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
overflow
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0
$0 $0
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.22
Accounting analysis of
dynamic tables
Charge an amortized cost of i = $3 for the i th
insertion.
$1 pays for the immediate insertion.
$2 is stored for later table doubling.
When the table doubles, $1 pays to move a
recent item, and $1 pays to move an old item.
Example:
$0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $0
$0 $2 $2 $2
$0 $0
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.23
Accounting analysis
(continued)
Key invariant: Bank balance never drops below 0.
Thus, the sum of the amortized costs provides an
upper bound on the sum of the true costs.
i
sizei
16 16
ci
2* 3
banki
10
*Okay, so I lied. The first operation costs only $2, not $3.
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.24
Potential method
IDEA: View the bank account as the potential
energy ( la physics) of the dynamic set.
Framework:
Start with an initial data structure D0.
Operation i transforms Di1 to Di.
The cost of operation i is ci.
Define a potential function : {Di} R,
such that (D0 ) = 0 and (Di ) 0 for all i.
The amortized cost i with respect to is
defined to be i = ci + (Di) (Di1).
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.25
Understanding potentials
i = ci + (Di) (Di1)
potential difference i
Introduction to Algorithms
L14.26
i =1
i =1
ci = (ci + ( Di ) ( Di1 ))
Summing both sides.
Introduction to Algorithms
L14.27
i =1
i =1
n
ci = (ci + ( Di ) ( Di1 ))
= ci + ( Dn ) ( D0 )
i =1
Introduction to Algorithms
L14.28
i =1
i =1
n
ci = (ci + ( Di ) ( Di1 ))
= ci + ( Dn ) ( D0 )
i =1
n
ci
i =1
Introduction to Algorithms
L14.29
= 26 23 = 4
$0
$0 $0
$0 $0
$0 $2
$2 $2
$2
$0 $0
accounting method)
Introduction to Algorithms
L14.30
Introduction to Algorithms
L14.31
Calculation (Case 1)
Case 1: i 1 is an exact power of 2.
= i + 2 (2(i 1) (i 1))
= i + 2 2i + 2 + i 1
=3
Introduction to Algorithms
L14.32
Calculation (Case 2)
Case 2: i 1 is not an exact power of 2.
Introduction to Algorithms
L14.33
Conclusions
Amortized costs can provide a clean abstraction
of data-structure performance.
Any of the analysis methods can be used when
an amortized analysis is called for, but each
method has some situations where it is arguably
the simplest.
Different schemes may work for assigning
amortized costs in the accounting method, or
potentials in the potential method, sometimes
yielding radically different bounds.
20014 by Charles E. Leiserson
Introduction to Algorithms
L14.34
Introduction to Algorithms
6.046J/18.401J
LECTURE 12
Dynamic programming
Longest common
subsequence
Optimal substructure
Overlapping subproblems
Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
Introduction to Algorithms
L12.2
Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
a not the
Introduction to Algorithms
L12.3
Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
a not the
x: A B
C
B
D A B
y: B
Introduction to Algorithms
L12.4
Dynamic programming
Design technique, like divide-and-conquer.
Example: Longest Common Subsequence (LCS)
Given two sequences x[1 . . m] and y[1 . . n], find
a longest subsequence common to them both.
a not the
x: A B
C
B
D A B
BCBA =
LCS(x, y)
y: B
D C
A B
A
functional notation,
but not a function
20014 by Charles E. Leiserson
Introduction to Algorithms
L12.5
Introduction to Algorithms
L12.6
Introduction to Algorithms
L12.7
Introduction to Algorithms
L12.8
Introduction to Algorithms
L12.9
Introduction to Algorithms
L12.10
Recursive formulation
Theorem.
c[i, j] =
c[i1, j1] + 1
if x[i] = y[j],
max{c[i1, j], c[i, j1]} otherwise.
Introduction to Algorithms
L12.11
Recursive formulation
Theorem.
c[i1, j1] + 1
if x[i] = y[j],
c[i, j] = max{c[i1, j], c[i, j1]} otherwise.
Proof. Case x[i] = y[ j]:
x:
1
1
2
2
y:
L
j
Introduction to Algorithms
L12.12
Recursive formulation
Theorem.
c[i1, j1] + 1
if x[i] = y[j],
c[i, j] = max{c[i1, j], c[i, j1]} otherwise.
Proof. Case x[i] = y[ j]:
x:
1
1
2
2
y:
L
j
Introduction to Algorithms
L12.13
Proof (continued)
Claim: z[1 . . k1] = LCS(x[1 . . i1], y[1 . . j1]).
Suppose w is a longer CS of x[1 . . i1] and
y[1 . . j1], that is, | w | > k1. Then, cut and
paste: w || z[k] (w concatenated with z[k]) is a
common subsequence of x[1 . . i] and y[1 . . j]
with | w || z[k] | > k. Contradiction, proving the
claim.
Introduction to Algorithms
L12.14
Proof (continued)
Claim: z[1 . . k1] = LCS(x[1 . . i1], y[1 . . j1]).
Suppose w is a longer CS of x[1 . . i1] and
y[1 . . j1], that is, | w | > k1. Then, cut and
paste: w || z[k] (w concatenated with z[k]) is a
common subsequence of x[1 . . i] and y[1 . . j]
with | w || z[k] | > k. Contradiction, proving the
claim.
Thus, c[i1, j1] = k1, which implies that c[i, j]
= c[i1, j1] + 1.
Other cases are similar.
20014 by Charles E. Leiserson
Introduction to Algorithms
L12.15
Dynamic-programming
hallmark #1
Optimal substructure
An optimal solution to a problem
(instance) contains optimal
solutions to subproblems.
Introduction to Algorithms
L12.16
Dynamic-programming
hallmark #1
Optimal substructure
An optimal solution to a problem
(instance) contains optimal
solutions to subproblems.
If z = LCS(x, y), then any prefix of z is
an LCS of a prefix of x and a prefix of y.
20014 by Charles E. Leiserson
Introduction to Algorithms
L12.17
Introduction to Algorithms
L12.18
Introduction to Algorithms
L12.19
Recursion tree
m = 3, n = 4:
3,4
3,4
2,4
2,4
1,4
1,4
3,3
3,3
2,3
2,3
1,3
1,3
3,2
3,2
2,3
2,3
2,2
2,2
1,3
1,3
Introduction to Algorithms
2,2
2,2
L12.20
Recursion tree
m = 3, n = 4:
3,4
3,4
2,4
2,4
1,4
1,4
3,3
3,3
2,3
2,3
1,3
1,3
3,2
3,2
2,3
2,3
2,2
2,2
1,3
1,3
m+n
2,2
2,2
Introduction to Algorithms
L12.21
Recursion tree
m = 3, n = 4:
3,4
3,4
2,4
2,4
1,4
1,4
3,3
3,3
same
subproblem
2,3
2,3
1,3
1,3
3,2
3,2
2,3
2,3
2,2
2,2
1,3
1,3
m+n
2,2
2,2
Introduction to Algorithms
L12.22
Dynamic-programming
hallmark #2
Overlapping subproblems
A recursive solution contains a
small number of distinct
subproblems repeated many times.
Introduction to Algorithms
L12.23
Dynamic-programming
hallmark #2
Overlapping subproblems
A recursive solution contains a
small number of distinct
subproblems repeated many times.
The number of distinct LCS subproblems for
two strings of lengths m and n is only mn.
Introduction to Algorithms
L12.24
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check the table to avoid redoing work.
Introduction to Algorithms
L12.25
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check the table to avoid redoing work.
LCS(x, y, i, j)
if c[i, j] = NIL
then if x[i] = y[j]
then c[i, j] LCS(x, y, i1, j1) + 1
else c[i, j] max{ LCS(x, y, i1, j),
LCS(x, y, i, j1)}
Introduction to Algorithms
same
as
before
L12.26
Memoization algorithm
Memoization: After computing a solution to a
subproblem, store it in a table. Subsequent calls
check the table to avoid redoing work.
LCS(x, y, i, j)
if c[i, j] = NIL
then if x[i] = y[j]
then c[i, j] LCS(x, y, i1, j1) + 1
else c[i, j] max{ LCS(x, y, i1, j),
LCS(x, y, i, j1)}
same
as
before
Introduction to Algorithms
L12.27
Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
Introduction to Algorithms
A B
00 00
11 11
22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
October 25, 2004
L12.28
Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).
A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
Introduction to Algorithms
A B
00 00
11 11
22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
October 25, 2004
L12.29
Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).
Reconstruct
LCS by tracing
backwards.
A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
Introduction to Algorithms
A B
00 00
11 11
22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
October 25, 2004
L12.30
Dynamic-programming
algorithm
IDEA:
Compute the
table bottom-up.
Time = (mn).
Reconstruct
LCS by tracing
backwards.
Space = (mn).
Exercise:
O(min{m, n}).
20014 by Charles E. Leiserson
A B C B D
00 00 00 00 00 00
B 00 00 11 11 11 11
D 00 00 11 11 11 22
C 00 00 11
A 00 11 11
B 00 11 22
A 00 11 22
Introduction to Algorithms
A B
00 00
11 11
22 22
22 22 22 22 22
22 22 22 33 33
22 33 33 33 44
22 33 33 44 44
October 25, 2004
L12.31
Introduction to Algorithms
6.046J/18.401J
LECTURE 13
Graph algorithms
Graph representation
Minimum spanning trees
Greedy algorithms
Optimal substructure
Greedy choice
Prims greedy MST
algorithm
Prof. Charles E. Leiserson
Graphs (review)
Definition. A directed graph (digraph)
G = (V, E) is an ordered pair consisting of
a set V of vertices (singular: vertex),
a set E V V of edges.
In an undirected graph G = (V, E), the edge
set E consists of unordered pairs of vertices.
In either case, we have | E | = O(V 2). Moreover,
if G is connected, then | E | | V | 1, which
implies that lg | E | = (lg V).
(Review CLRS, Appendix B.)
20014 by Charles E. Leiserson
Introduction to Algorithms
L13.2
Adjacency-matrix
representation
The adjacency matrix of a graph G = (V, E), where
V = {1, 2, , n}, is the matrix A[1 . . n, 1 . . n]
given by
1 if (i, j) E,
A[i, j] =
0 if (i, j) E.
Introduction to Algorithms
L13.3
Adjacency-matrix
representation
The adjacency matrix of a graph G = (V, E), where
V = {1, 2, , n}, is the matrix A[1 . . n, 1 . . n]
given by
1 if (i, j) E,
A[i, j] =
0 if (i, j) E.
22
11
33
44
A 1 2 3 4
1 0 1 1 0
2 0 0 1 0
3 0 0 0 0
4 0 0 1 0
Introduction to Algorithms
(V 2) storage
dense
representation.
October 27, 2004
L13.4
Adjacency-list representation
An adjacency list of a vertex v V is the list Adj[v]
of vertices adjacent to v.
Adj[1] = {2, 3}
22
11
33
44
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
Introduction to Algorithms
L13.5
Adjacency-list representation
An adjacency list of a vertex v V is the list Adj[v]
of vertices adjacent to v.
Adj[1] = {2, 3}
22
11
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
33
44
For undirected graphs, | Adj[v] | = degree(v).
For digraphs, | Adj[v] | = out-degree(v).
Introduction to Algorithms
L13.6
Adjacency-list representation
An adjacency list of a vertex v V is the list Adj[v]
of vertices adjacent to v.
Adj[1] = {2, 3}
22
11
Adj[2] = {3}
Adj[3] = {}
Adj[4] = {3}
33
44
For undirected graphs, | Adj[v] | = degree(v).
For digraphs, | Adj[v] | = out-degree(v).
Handshaking Lemma: vV = 2 |E| for undirected
graphs adjacency lists use (V + E) storage
a sparse representation (for either type of graph).
20014 by Charles E. Leiserson
Introduction to Algorithms
L13.7
Introduction to Algorithms
L13.8
Introduction to Algorithms
L13.9
Example of MST
6
12
9
5
14
8
3
15
10
Introduction to Algorithms
L13.10
Example of MST
6
12
9
5
14
8
3
15
10
Introduction to Algorithms
L13.11
Optimal substructure
MST T:
(Other edges of G
are not shown.)
Introduction to Algorithms
L13.12
Optimal substructure
MST T:
(Other edges of G
are not shown.)
Introduction to Algorithms
L13.13
Optimal substructure
MST T:
(Other edges of G
are not shown.)
Introduction to Algorithms
L13.14
Optimal substructure
MST T:
(Other edges of G
are not shown.)
T2
T1
v
Introduction to Algorithms
L13.15
Optimal substructure
MST T:
(Other edges of G
are not shown.)
T2
T1
v
Introduction to Algorithms
L13.16
Introduction to Algorithms
L13.17
Introduction to Algorithms
L13.18
Introduction to Algorithms
L13.19
Introduction to Algorithms
L13.20
Introduction to Algorithms
L13.21
Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:
v
A
VA
u
(u, v) = least-weight edge
connecting A to V A
Introduction to Algorithms
L13.22
Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:
v
A
VA
u
(u, v) = least-weight edge
connecting A to V A
Introduction to Algorithms
L13.23
Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T:
v
A
VA
u
(u, v) = least-weight edge
connecting A to V A
Introduction to Algorithms
L13.24
Proof of theorem
Proof. Suppose (u, v) T. Cut and paste.
T :
A
VA
v
u
(u, v) = least-weight edge
connecting A to V A
Introduction to Algorithms
L13.25
Prims algorithm
IDEA: Maintain V A as a priority queue Q. Key
each vertex in Q with the weight of the leastweight edge connecting it to a vertex in A.
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v)
DECREASE-KEY
[v] u
Introduction to Algorithms
L13.26
14
8
3
12
9
7
15
00
10
Introduction to Algorithms
L13.27
14
8
3
12
9
7
15
00
10
Introduction to Algorithms
L13.28
14
77
7
8
3
12
00
10
10
15
15
15
10
Introduction to Algorithms
L13.29
14
77
7
8
3
12
00
10
10
15
15
15
10
Introduction to Algorithms
L13.30
12
12
55
14
77
7
8
3
12
00
10
10
99
15
15
15
10
Introduction to Algorithms
L13.31
12
12
55
14
77
7
8
3
12
00
10
10
99
15
15
15
10
Introduction to Algorithms
L13.32
66
55
14
14
14
77
7
8
3
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.33
66
55
14
14
14
77
7
8
3
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.34
66
55
14
14
14
77
7
8
3
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.35
66
55
14
3
77
7
33
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.36
66
55
14
3
77
7
33
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.37
66
55
14
3
77
7
33
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.38
66
55
14
3
77
7
33
12
00
88
99
15
15
15
10
Introduction to Algorithms
L13.39
Analysis of Prim
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v)
[v] u
Introduction to Algorithms
L13.40
Analysis of Prim
(V)
total
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v)
[v] u
Introduction to Algorithms
L13.41
Analysis of Prim
(V)
total
|V |
times
QV
key[v] for all v V
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
do if v Q and w(u, v) < key[v]
then key[v] w(u, v)
[v] u
Introduction to Algorithms
L13.42
Analysis of Prim
QV
(V)
key[v] for all v V
total
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
|V |
do if v Q and w(u, v) < key[v]
times degree(u)
times
then key[v] w(u, v)
[v] u
Introduction to Algorithms
L13.43
Analysis of Prim
QV
(V)
key[v] for all v V
total
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
|V |
do if v Q and w(u, v) < key[v]
times degree(u)
times
then key[v] w(u, v)
[v] u
Handshaking Lemma (E) implicit DECREASE-KEYs.
Introduction to Algorithms
L13.44
Analysis of Prim
QV
(V)
key[v] for all v V
total
key[s] 0 for some arbitrary s V
while Q
do u EXTRACT-MIN(Q)
for each v Adj[u]
|V |
do if v Q and w(u, v) < key[v]
times degree(u)
times
then key[v] w(u, v)
[v] u
Handshaking Lemma (E) implicit DECREASE-KEYs.
Introduction to Algorithms
L13.45
Introduction to Algorithms
L13.46
TEXTRACT-MIN TDECREASE-KEY
Introduction to Algorithms
Total
L13.47
TEXTRACT-MIN TDECREASE-KEY
O(V)
O(1)
Introduction to Algorithms
Total
O(V2)
L13.48
TEXTRACT-MIN TDECREASE-KEY
Total
array
O(V)
O(1)
O(V2)
binary
heap
O(lg V)
O(lg V)
O(E lg V)
Introduction to Algorithms
L13.49
TEXTRACT-MIN TDECREASE-KEY
Total
array
O(V)
O(1)
O(V2)
binary
heap
O(lg V)
O(lg V)
O(E lg V)
Fibonacci O(lg V)
heap
amortized
20014 by Charles E. Leiserson
O(1)
O(E + V lg V)
amortized worst case
Introduction to Algorithms
L13.50
MST algorithms
Kruskals algorithm (see CLRS):
Uses the disjoint-set data structure (Lecture 10).
Running time = O(E lg V).
Introduction to Algorithms
L13.51
MST algorithms
Kruskals algorithm (see CLRS):
Uses the disjoint-set data structure (Lecture 10).
Running time = O(E lg V).
Best to date:
Karger, Klein, and Tarjan [1993].
Randomized algorithm.
O(V + E) expected time.
Introduction to Algorithms
L13.52
Introduction to Algorithms
6.046J/18.401J
LECTURE 14
Shortest Paths I
Properties of shortest paths
Dijkstras algorithm
Correctness
Analysis
Breadth-first search
Prof. Charles E. Leiserson
Paths in graphs
Consider a digraph G = (V, E) with edge-weight
function w : E R. The weight of path p = v1
v2 L vk is defined to be
k 1
w( p ) = w(vi , vi +1 ) .
i =1
Introduction to Algorithms
November 1, 2004
L14.2
Paths in graphs
Consider a digraph G = (V, E) with edge-weight
function w : E R. The weight of path p = v1
v2 L vk is defined to be
k 1
w( p ) = w(vi , vi +1 ) .
i =1
Example:
vv11
vv22
vv33
vv44
Introduction to Algorithms
vv55
w(p) = 2
November 1, 2004
L14.3
Shortest paths
A shortest path from u to v is a path of
minimum weight from u to v. The shortestpath weight from u to v is defined as
(u, v) = min{w(p) : p is a path from u to v}.
Note: (u, v) = if no path from u to v exists.
Introduction to Algorithms
November 1, 2004
L14.4
Optimal substructure
Theorem. A subpath of a shortest path is a
shortest path.
Introduction to Algorithms
November 1, 2004
L14.5
Optimal substructure
Theorem. A subpath of a shortest path is a
shortest path.
Introduction to Algorithms
November 1, 2004
L14.6
Optimal substructure
Theorem. A subpath of a shortest path is a
shortest path.
Introduction to Algorithms
November 1, 2004
L14.7
Triangle inequality
Theorem. For all u, v, x V, we have
(u, v) (u, x) + (x, v).
Introduction to Algorithms
November 1, 2004
L14.8
Triangle inequality
Theorem. For all u, v, x V, we have
(u, v) (u, x) + (x, v).
Proof.
(u, v)
uu
(u, x)
vv
(x, v)
xx
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.9
Well-definedness of shortest
paths
If a graph G contains a negative-weight cycle,
then some shortest paths may not exist.
Introduction to Algorithms
November 1, 2004
L14.10
Well-definedness of shortest
paths
If a graph G contains a negative-weight cycle,
then some shortest paths may not exist.
Example:
<0
uu
20014 by Charles E. Leiserson
vv
Introduction to Algorithms
November 1, 2004
L14.11
Introduction to Algorithms
November 1, 2004
L14.12
Dijkstras algorithm
d[s] 0
for each v V {s}
do d[v]
S
QV
Q is a priority queue maintaining V S
Introduction to Algorithms
November 1, 2004
L14.13
Dijkstras algorithm
d[s] 0
for each v V {s}
do d[v]
S
QV
Q is a priority queue maintaining V S
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)
Introduction to Algorithms
November 1, 2004
L14.14
Dijkstras algorithm
d[s] 0
for each v V {s}
do d[v]
S
QV
Q is a priority queue maintaining V S
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
relaxation
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)
step
Implicit DECREASE-KEY
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.15
Example of Dijkstras
algorithm
Graph with
nonnegative
edge weights:
10
AA
1 4
3
BB
Introduction to Algorithms
CC
2
8
D
D
7 9
EE
November 1, 2004
L14.16
Example of Dijkstras
algorithm
BB
Initialize:
10
0 AA
Q: A B C D E
0
1 4
3
CC
2
8
D
D
7 9
EE
S: {}
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.17
Example of Dijkstras
algorithm
A EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
BB
1 4
3
CC
2
8
D
D
7 9
EE
S: { A }
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.18
Example of Dijkstras
algorithm
Relax all edges leaving A:
10
0 AA
Q: A B C D E
0
10
10
BB
1 4
3
CC
3
2
8
D
D
7 9
EE
S: { A }
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.19
Example of Dijkstras
algorithm
C EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
10
BB
2
8
1 4
3
CC
3
D
D
7 9
EE
S: { A, C }
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.20
Example of Dijkstras
algorithm
Relax all edges leaving C:
10
0 AA
Q: A B C D E
0
10
7
11
7
BB
2
8
1 4
3
CC
3
11
D
D
7 9
EE
5
S: { A, C }
Introduction to Algorithms
November 1, 2004
L14.21
Example of Dijkstras
algorithm
E EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
7
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C, E }
Introduction to Algorithms
November 1, 2004
L14.22
Example of Dijkstras
algorithm
Relax all edges leaving E:
10
0 AA
Q: A B C D E
0
10
7
7
11
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C, E }
Introduction to Algorithms
November 1, 2004
L14.23
Example of Dijkstras
algorithm
B EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
7
7
11
11
7
BB
1 4
3
CC
3
2
8
11
D
D
7 9
EE
5
S: { A, C, E, B }
Introduction to Algorithms
November 1, 2004
L14.24
Example of Dijkstras
algorithm
Relax all edges leaving B:
10
0 AA
Q: A B C D E
0
10
7
7
11
11
9
7
BB
1 4
3
CC
3
9
D
D
2
8
7 9
EE
5
S: { A, C, E, B }
Introduction to Algorithms
November 1, 2004
L14.25
Example of Dijkstras
algorithm
D EXTRACT-MIN(Q):
10
0 AA
Q: A B C D E
0
10
7
7
11
11
9
7
BB
1 4
3
CC
3
2
8
9
D
D
7 9
EE
5
S: { A, C, E, B, D }
Introduction to Algorithms
November 1, 2004
L14.26
Correctness Part I
Lemma. Initializing d[s] 0 and d[v] for all
v V {s} establishes d[v] (s, v) for all v V,
and this invariant is maintained over any sequence
of relaxation steps.
Introduction to Algorithms
November 1, 2004
L14.27
Correctness Part I
Lemma. Initializing d[s] 0 and d[v] for all
v V {s} establishes d[v] (s, v) for all v V,
and this invariant is maintained over any sequence
of relaxation steps.
Proof. Suppose not. Let v be the first vertex for
which d[v] < (s, v), and let u be the vertex that
caused d[v] to change: d[v] = d[u] + w(u, v). Then,
d[v] < (s, v)
supposition
(s, u) + (u, v) triangle inequality
(s,u) + w(u, v) sh. path specific path
d[u] + w(u, v)
v is first violation
Contradiction.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.28
Correctness Part II
Lemma. Let u be vs predecessor on a shortest
path from s to v. Then, if d[u] = (s, u) and edge
(u, v) is relaxed, we have d[v] = (s, v) after the
relaxation.
Introduction to Algorithms
November 1, 2004
L14.29
Correctness Part II
Lemma. Let u be vs predecessor on a shortest
path from s to v. Then, if d[u] = (s, u) and edge
(u, v) is relaxed, we have d[v] = (s, v) after the
relaxation.
Proof. Observe that (s, v) = (s, u) + w(u, v).
Suppose that d[v] > (s, v) before the relaxation.
(Otherwise, were done.) Then, the test d[v] >
d[u] + w(u, v) succeeds, because d[v] > (s, v) =
(s, u) + w(u, v) = d[u] + w(u, v), and the
algorithm sets d[v] = d[u] + w(u, v) = (s, v).
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.30
Introduction to Algorithms
November 1, 2004
L14.31
uu
S, just before
adding u.
20014 by Charles E. Leiserson
ss
Introduction to Algorithms
xx
yy
November 1, 2004
L14.32
uu
xx
yy
Introduction to Algorithms
November 1, 2004
L14.33
Analysis of Dijkstra
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)
Introduction to Algorithms
November 1, 2004
L14.34
Analysis of Dijkstra
|V |
times
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
do if d[v] > d[u] + w(u, v)
then d[v] d[u] + w(u, v)
Introduction to Algorithms
November 1, 2004
L14.35
Analysis of Dijkstra
|V |
times
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
degree(u)
do if d[v] > d[u] + w(u, v)
times
then d[v] d[u] + w(u, v)
Introduction to Algorithms
November 1, 2004
L14.36
Analysis of Dijkstra
|V |
times
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
degree(u)
do if d[v] > d[u] + w(u, v)
times
then d[v] d[u] + w(u, v)
Introduction to Algorithms
November 1, 2004
L14.37
Analysis of Dijkstra
|V |
times
while Q
do u EXTRACT-MIN(Q)
S S {u}
for each v Adj[u]
degree(u)
do if d[v] > d[u] + w(u, v)
times
then d[v] d[u] + w(u, v)
Introduction to Algorithms
November 1, 2004
L14.38
Analysis of Dijkstra
(continued)
Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY
Q
TEXTRACT-MIN TDECREASE-KEY
Introduction to Algorithms
Total
November 1, 2004
L14.39
Analysis of Dijkstra
(continued)
Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY
Q
array
TEXTRACT-MIN TDECREASE-KEY
O(V)
O(1)
Introduction to Algorithms
Total
O(V2)
November 1, 2004
L14.40
Analysis of Dijkstra
(continued)
Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY
Q
TEXTRACT-MIN TDECREASE-KEY
Total
array
O(V)
O(1)
O(V2)
binary
heap
O(lg V)
O(lg V)
O(E lg V)
Introduction to Algorithms
November 1, 2004
L14.41
Analysis of Dijkstra
(continued)
Time = (V)TEXTRACT-MIN + (E)TDECREASE-KEY
Q
TEXTRACT-MIN TDECREASE-KEY
Total
array
O(V)
O(1)
O(V2)
binary
heap
O(lg V)
O(lg V)
O(E lg V)
Fibonacci O(lg V)
heap
amortized
20014 by Charles E. Leiserson
O(1)
O(E + V lg V)
amortized worst case
Introduction to Algorithms
November 1, 2004
L14.42
Unweighted graphs
Suppose that w(u, v) = 1 for all (u, v) E.
Can Dijkstras algorithm be improved?
Introduction to Algorithms
November 1, 2004
L14.43
Unweighted graphs
Suppose that w(u, v) = 1 for all (u, v) E.
Can Dijkstras algorithm be improved?
Use a simple FIFO queue instead of a priority
queue.
Introduction to Algorithms
November 1, 2004
L14.44
Unweighted graphs
Suppose that w(u, v) = 1 for all (u, v) E.
Can Dijkstras algorithm be improved?
Use a simple FIFO queue instead of a priority
queue.
Breadth-first search
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)
Introduction to Algorithms
November 1, 2004
L14.45
Unweighted graphs
Suppose that w(u, v) = 1 for all (u, v) E.
Can Dijkstras algorithm be improved?
Use a simple FIFO queue instead of a priority
queue.
Breadth-first search
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)
Introduction to Algorithms
November 1, 2004
L14.46
Example of breadth-first
search
aa
ff
hh
dd
bb
gg
ee
ii
cc
Q:
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.47
Example of breadth-first
search
0
aa
ff
hh
dd
bb
gg
ee
ii
cc
0
Q: a
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.48
Example of breadth-first
search
0
aa
ff
hh
dd
1
bb
gg
ee
ii
cc
1 1
Q: a b d
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.49
Example of breadth-first
search
0
aa
ff
hh
dd
1
bb
gg
ee
cc
ii
2
1 2 2
Q: a b d c e
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.50
Example of breadth-first
search
0
aa
ff
hh
dd
1
bb
gg
ee
cc
ii
2
2 2
Q: a b d c e
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.51
Example of breadth-first
search
0
aa
ff
hh
dd
1
bb
gg
ee
cc
ii
2
2
Q: a b d c e
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.52
Example of breadth-first
search
0
aa
dd
1
bb
cc
ff
1
3
hh
gg
ee
ii
3
3 3
Q: a b d c e g i
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.53
Example of breadth-first
search
4
0
aa
dd
1
bb
cc
ff
1
3
hh
gg
ee
ii
3
3 4
Q: a b d c e g i f
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.54
Example of breadth-first
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
3
4 4
Q: a b d c e g i f h
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.55
Example of breadth-first
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
3
4
Q: a b d c e g i f h
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.56
Example of breadth-first
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
Q: a b d c e g i f h
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.57
Example of breadth-first
search
0
aa
dd
1
bb
cc
ff
hh
gg
ee
ii
Q: a b d c e g i f h
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.58
Correctness of BFS
while Q
do u DEQUEUE(Q)
for each v Adj[u]
do if d[v] =
then d[v] d[u] + 1
ENQUEUE(Q, v)
Key idea:
The FIFO Q in breadth-first search mimics
the priority queue Q in Dijkstra.
Invariant: v comes after u in Q implies that
d[v] = d[u] or d[v] = d[u] + 1.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 1, 2004
L14.59
Introduction to Algorithms
6.046J/18.401J
LECTURE 15
Shortest Paths II
Bellman-Ford algorithm
DAG shortest paths
Linear programming and
difference constraints
VLSI layout compaction
Prof. Charles E. Leiserson
Negative-weight cycles
Recall: If a graph G = (V, E) contains a negativeweight cycle, then some shortest paths may not exist.
Example:
<0
uu
vv
Introduction to Algorithms
November 3, 2004
L15.2
Negative-weight cycles
Recall: If a graph G = (V, E) contains a negativeweight cycle, then some shortest paths may not exist.
Example:
<0
uu
vv
Introduction to Algorithms
November 3, 2004
L15.3
Bellman-Ford algorithm
d[s] 0
for each v V {s}
do d[v]
initialization
for i 1 to | V | 1
do for each edge (u, v) E
do if d[v] > d[u] + w(u, v)
relaxation
then d[v] d[u] + w(u, v)
step
for each edge (u, v) E
do if d[v] > d[u] + w(u, v)
then report that a negative-weight cycle exists
At the end, d[v] = (s, v), if no negative-weight cycles.
Time = O(VE).
20014 by Charles E. Leiserson
Introduction to Algorithms
November 3, 2004
L15.4
Example of Bellman-Ford
BB
1
3
AA
4
CC
1
5
2
2
D
D
Introduction to Algorithms
EE
3
November 3, 2004
L15.5
Example of Bellman-Ford
BB
AA
4
CC
1
5
EE
D
D
Initialization.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 3, 2004
L15.6
Example of Bellman-Ford
BB
0
AA
4
5
CC
1
3
1
5
2
8
D
D
EE
Introduction to Algorithms
November 3, 2004
L15.7
Example of Bellman-Ford
BB
0
AA
4
5
CC
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.8
Example of Bellman-Ford
BB
0
AA
4
5
CC
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.9
Example of Bellman-Ford
BB
0
AA
4
5
CC
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.10
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.11
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
4
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.12
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
4
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.13
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
4
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.14
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.15
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
2
8
D
D
EE
End of pass 1.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 3, 2004
L15.16
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.17
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
Introduction to Algorithms
EE
November 3, 2004
L15.18
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
1
Introduction to Algorithms
EE
November 3, 2004
L15.19
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
1
Introduction to Algorithms
EE
November 3, 2004
L15.20
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
1
Introduction to Algorithms
EE
November 3, 2004
L15.21
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
1
Introduction to Algorithms
EE
November 3, 2004
L15.22
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
1
Introduction to Algorithms
EE
November 3, 2004
L15.23
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
2
1
Introduction to Algorithms
EE
November 3, 2004
L15.24
Example of Bellman-Ford
1
BB
0
AA
4
5
CC
2
1
3
1
5
2
8
D
D
2
EE
Introduction to Algorithms
November 3, 2004
L15.25
Correctness
Theorem. If G = (V, E) contains no negativeweight cycles, then after the Bellman-Ford
algorithm executes, d[v] = (s, v) for all v V.
Introduction to Algorithms
November 3, 2004
L15.26
Correctness
Theorem. If G = (V, E) contains no negativeweight cycles, then after the Bellman-Ford
algorithm executes, d[v] = (s, v) for all v V.
Proof. Let v V be any vertex, and consider a shortest
path p from s to v with the minimum number of edges.
s
p: vv0
0
vv11
vv22
vv33
vvkk
Introduction to Algorithms
November 3, 2004
L15.27
Correctness (continued)
s
p: vv0
0
vv11
vv22
vv33
v
vvkk
Introduction to Algorithms
November 3, 2004
L15.28
Detection of negative-weight
cycles
Corollary. If a value d[v] fails to converge after
|V| 1 passes, there exists a negative-weight
cycle in G reachable from s.
Introduction to Algorithms
November 3, 2004
L15.29
Linear programming
Let A be an mn matrix, b be an m-vector, and c
be an n-vector. Find an n-vector x that maximizes
cTx subject to Ax b, or determine that no such
solution exists.
n
m
.
A
x b
maximizing
Introduction to Algorithms
cT
November 3, 2004
x
L15.30
Linear-programming
algorithms
Algorithms for the general problem
Simplex methods practical, but worst-case
exponential time.
Interior-point methods polynomial time and
competes with simplex.
Introduction to Algorithms
November 3, 2004
L15.31
Linear-programming
algorithms
Algorithms for the general problem
Simplex methods practical, but worst-case
exponential time.
Interior-point methods polynomial time and
competes with simplex.
Feasibility problem: No optimization criterion.
Just find x such that Ax b.
In general, just as hard as ordinary LP.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 3, 2004
L15.32
Introduction to Algorithms
November 3, 2004
L15.33
Introduction to Algorithms
November 3, 2004
L15.34
vvii
wij
Introduction to Algorithms
vvjj
(The A
matrix has
dimensions
|E | |V |.)
November 3, 2004
L15.35
Unsatisfiable constraints
Theorem. If the constraint graph contains
a negative-weight cycle, then the system of
differences is unsatisfiable.
Introduction to Algorithms
November 3, 2004
L15.36
Unsatisfiable constraints
Theorem. If the constraint graph contains
a negative-weight cycle, then the system of
differences is unsatisfiable.
Proof. Suppose that the negative-weight cycle is
v1 v2 L vk v1. Then, we have
x2 x1
x3 x2
w12
w23
M
xk xk1 wk1, k
x1 xk wk1
Introduction to Algorithms
November 3, 2004
L15.37
Unsatisfiable constraints
Theorem. If the constraint graph contains
a negative-weight cycle, then the system of
differences is unsatisfiable.
Proof. Suppose that the negative-weight cycle is
v1 v2 L vk v1. Then, we have
x2 x1
x3 x2
w12
w23
M
xk xk1 wk1, k
x1 xk wk1
0
weight of cycle
<0
Introduction to Algorithms
Therefore, no
values for the xi
can satisfy the
constraints.
November 3, 2004
L15.38
Introduction to Algorithms
November 3, 2004
L15.39
vv11
vv44
vv77
20014 by Charles E. Leiserson
vv99
vv33
Introduction to Algorithms
November 3, 2004
L15.40
vv11
vv44
vv77
vv99
vv33
Introduction to Algorithms
Note:
No negative-weight
cycles introduced
shortest paths exist.
November 3, 2004
L15.41
Proof (continued)
Claim: The assignment xi = (s, vi) solves the constraints.
Consider any constraint xj xi wij, and consider the
shortest paths from s to vj and vi:
ss
(s, vi)
(s, vj)
vvii
wij
vvjj
Introduction to Algorithms
November 3, 2004
L15.42
Introduction to Algorithms
November 3, 2004
L15.43
Introduction to Algorithms
November 3, 2004
L15.44
x1
x2
x2 x1 d 1 +
Bellman-Ford minimizes maxi{xi} mini{xi},
which compacts the layout in the x-dimension.
Constraint:
Introduction to Algorithms
November 3, 2004
L15.45
Introduction to Algorithms
6.046J/18.401J
LECTURE 16
Shortest Paths III
All-pairs shortest paths
Matrix-multiplication
algorithm
Floyd-Warshall algorithm
Johnsons algorithm
Prof. Charles E. Leiserson
Shortest paths
Single-source shortest paths
Nonnegative edge weights
General
DAG
Introduction to Algorithms
November 8, 2004
L16.2
Shortest paths
Single-source shortest paths
Nonnegative edge weights
General
Bellman-Ford: O(VE)
DAG
General
Introduction to Algorithms
November 8, 2004
L16.3
Introduction to Algorithms
November 8, 2004
L16.4
Introduction to Algorithms
November 8, 2004
L16.5
Dynamic programming
Consider the n n adjacency matrix A = (aij)
of the digraph, and define
dij(m) = weight of a shortest path from
i to j that uses at most m edges.
Claim: We have
0 if i = j,
(0)
dij =
if i j;
and for m = 1, 2, , n 1,
dij(m) = mink{dik(m1) + akj }.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 8, 2004
L16.6
Proof of claim
ks
es
g
d
1e
ii
s
e
g
d
e
1
m
m
1
edg
es
jj
M
m 1 edges
Introduction to Algorithms
November 8, 2004
L16.7
Proof of claim
ks
es
g
d
1e
ii
Relaxation!
s
e
g
d
e
1
m
m
1
edg
es
for k 1 to n
do if dij > dik + akj
then dij dik + akj
Introduction to Algorithms
jj
M
m 1 edges
November 8, 2004
L16.8
Proof of claim
ks
es
g
d
1e
ii
Relaxation!
s
e
g
d
e
1
m
m
1
edg
es
for k 1 to n
do if dij > dik + akj
then dij dik + akj
jj
M
m 1 edges
Introduction to Algorithms
November 8, 2004
L16.9
Matrix multiplication
Compute C = A B, where C, A, and B are n n
matrices:
n
cij = aik bkj .
k =1
Introduction to Algorithms
November 8, 2004
L16.10
Matrix multiplication
Compute C = A B, where C, A, and B are n n
matrices:
n
cij = aik bkj .
k =1
Introduction to Algorithms
November 8, 2004
L16.11
Matrix multiplication
Compute C = A B, where C, A, and B are n n
matrices:
n
cij = aik bkj .
k =1
0
0
0
0
Introduction to Algorithms
= D0 = (dij(0)).
November 8, 2004
L16.12
Matrix multiplication
(continued)
The (min, +) multiplication is associative, and
with the real numbers, it forms an algebraic
structure called a closed semiring.
Consequently, we can compute
D(1) = D(0) A = A1
D(2) = D(1) A = A2
M
M
D(n1) = D(n2) A = An1 ,
yielding D(n1) = ((i, j)).
Time = (nn3) = (n4). No better than n B-F.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 8, 2004
L16.13
Improved matrix
multiplication algorithm
Repeated squaring: A2k = Ak Ak.
lg(n1)
2
4
2
.
Compute A , A , , A
O(lg n) squarings
Note: An1 = An = An+1 = L.
Time = (n3 lg n).
To detect negative-weight cycles, check the
diagonal for negative values in O(n) additional
time.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 8, 2004
L16.14
Floyd-Warshall algorithm
Also dynamic programming, but faster!
Define cij(k) = weight of a shortest path from i
to j with intermediate vertices
belonging to the set {1, 2, , k}.
ii
kk
kk
kk
kk
jj
Introduction to Algorithms
November 8, 2004
L16.15
Floyd-Warshall recurrence
cij(k) = min {cij(k1), cik(k1) + ckj(k1)}
cik
ii
(k1)
cij(k1)
ckj(k1)
jj
Introduction to Algorithms
November 8, 2004
L16.16
relaxation
Notes:
Okay to omit superscripts, since extra relaxations
cant hurt.
Runs in (n3) time.
Simple to code.
Efficient in practice.
20014 by Charles E. Leiserson
Introduction to Algorithms
November 8, 2004
L16.17
Transitive closure of a
directed graph
Compute tij =
Introduction to Algorithms
November 8, 2004
L16.18
Graph reweighting
Theorem. Given a function h : V R, reweight each
edge (u, v) E by wh(u, v) = w(u, v) + h(u) h(v).
Then, for any two vertices, all paths between them are
reweighted by the same amount.
Introduction to Algorithms
November 8, 2004
L16.19
Graph reweighting
Theorem. Given a function h : V R, reweight each
edge (u, v) E by wh(u, v) = w(u, v) + h(u) h(v).
Then, for any two vertices, all paths between them are
reweighted by the same amount.
Proof. Let p = v1 v2 L vk be a path in G. We
k 1
have
wh ( p ) =
=
wh ( vi ,vi+1 )
i =1
k 1
( w( vi ,vi+1 )+ h ( vi ) h ( vi+1 ) )
i =1
k 1
w( vi ,vi+1 ) + h ( v1 ) h ( vk ) Same
i =1
= w ( p ) + h ( v1 ) h ( v k ) .
20014 by Charles E. Leiserson
Introduction to Algorithms
amount!
November 8, 2004
L16.20
Introduction to Algorithms
November 8, 2004
L16.21
Introduction to Algorithms
November 8, 2004
L16.22
Johnsons algorithm
1. Find a function h : V R such that wh(u, v) 0 for
all (u, v) E by using Bellman-Ford to solve the
difference constraints h(v) h(u) w(u, v), or
determine that a negative-weight cycle exists.
Time = O(V E).
2. Run Dijkstras algorithm using wh from each vertex
u V to compute h(u, v) for all v V.
Time = O(V E + V 2 lg V).
3. For each (u, v) V V, compute
(u, v) = h(u, v) h(u) + h(v) .
Time = O(V 2).
Introduction to Algorithms
November 8, 2004
L16.23
Introduction to Algorithms
6.046J/18.401
Lecture 18
Today
Introduction to Algorithms
L17.2
Computational Geometry
vision,.
lecture
Introduction to Algorithms
L17.3
Closest Pair
Introduction to Algorithms
L17.4
Divide:
Introduction to Algorithms
L17.5
Combine
2d
Let d=min(d1,d2)
Observe:
Need to check only pairs
which cross the dividing
line
Only interested in pairs
within distance < d
Suffices to look at points in the
2d-width strip around the
median line
2003 by Piotr Indyk
Introduction to Algorithms
d2
d1
L17.6
j=i-1
Introduction to Algorithms
d
d
L17.7
Analysis
Correctness: easy
Proof by packing
argument
Introduction to Algorithms
L17.8
Analysis, ctd.
Introduction to Algorithms
qi
L17.9
Packing bound
Will prove 5
Draw a disk of radius d/2
around each point
Disks are disjoint
Introduction to Algorithms
L17.10
Running time
Divide: O(n)
Combine: O(n log n) because we sort by y
However, we can:
Sort all points by y at the beginning
T(n)=O(n log n)
Introduction to Algorithms
L17.11
Close pair
Given: P={p1pn}
Goal: check if there is any pair pi pj within
distance R from each other
Will give an O(n) time algorithm, using
radix sort !
(assuming coordinates are small integers)
Introduction to Algorithms
L17.12
Algorithm
an R R square
and exit
Let c =b(p)
adjacent to c
For all points q from buckets in C {c}
If ||p-q||R, then answer YES and exit
Answer NO
Introduction to Algorithms
L17.13
Bucket access
Introduction to Algorithms
L17.14
Analysis
Running time:
Putting points into the buckets: O(n) time
Checking if there is a heavy bucket: O(n)
Checking the cells: 9 4 n = O(n)
Overall: linear time
Introduction to Algorithms
L17.15
Computational Model
Introduction to Algorithms
L17.16
Introduction to Algorithms
6.046J/18.401
Lecture 17
by coordinates of endpoints
intersecting segments
Introduction to Algorithms
L18.2
Segment intersection
However:
We will see we can do better for
the detection problem
Moreover, the number of
intersections P is usually small.
Then, we would like an output
sensitive algorithm, whose
running time is low if P is small.
Introduction to Algorithms
L18.3
Result
We will show:
O(n log n) time for detection
O( (n +P) log n) time for reporting
We will use
Introduction to Algorithms
L18.4
Orthogonal segments
V-segment
horizontal or vertical
are distinct
Introduction to Algorithms
H-segment
L18.5
Orthogonal segments
Sweep line:
Introduction to Algorithms
L18.6
We maintain sorted y-
coordinates of H-segments
BST V)
coordinate to V
coordinate from V
Introduction to Algorithms
L18.7
ytop
17
ybot
12
Introduction to Algorithms
L18.8
Algorithm
Sort all V-segments and endpoints of Hsegments by their x-coordinates this gives
the trajectory of the sweep line
Scan the elements in the sorted list:
Left endpoint: add segment to tree V
Right endpoint: remove segment from V
V-segment: report intersections with the
H-segments stored in V
2003 by Piotr Indyk
Introduction to Algorithms
L18.9
Analysis
Processing V-segments:
O(log n) per intersection - SEE NEXT SLIDE
O(P log n) total
Overall: O( (P+ n) log n) time
Can be improved to O(P +n log n)
2003 by Piotr Indyk
Introduction to Algorithms
L18.10
Analyzing intersections
Given:
A BST V containing y-coordinates
An interval I=[ybot,ytop]
Goal: report all ys in V that belong to I
Algorithm:
y=Successor(ybot)
While yytop
Report y
y:=Successor(y)
End
Introduction to Algorithms
L18.11
Assumption: all
coordinates of endpoints
and intersections distinct
In particular:
No vertical segments
No three segments
intersect at one point
2003 by Piotr Indyk
Introduction to Algorithms
L18.12
Sweep line
Introduction to Algorithms
L18.13
Sweep line
Introduction to Algorithms
L18.14
Algorithm
// Update V
Introduction to Algorithms
L18.15
Algorithm ctd.
// Update H
For each new pair of neighbors s and s in V:
Introduction to Algorithms
L18.16
Analysis
Updating V:
O(log n) per operation
O( (P+n) log n) total
Updating H:
O(log n) per intersection
O(P log n) total
Overall: O( (P+ n) log n) time
2003 by Piotr Indyk
Introduction to Algorithms
L18.17
Correctness
Introduction to Algorithms
L18.18
Changes
Introduction to Algorithms
L18.19
Introduction to Algorithms
6.046J/18.401J
LECTURE 19
Take-home exam
Instructions
Academic honesty
Strategies for doing well
Take-home quiz
1 hard
1 very hard
Introduction to Algorithms
L19.2
End of quiz
Introduction to Algorithms
L19.3
Planning
Introduction to Algorithms
L19.4
Format
Introduction to Algorithms
L19.5
Executive summary
Introduction to Algorithms
L19.6
Solutions
Introduction to Algorithms
L19.7
Solutions
solutions.
Introduction to Algorithms
L19.8
Assumptions
Introduction to Algorithms
L19.9
Bugs, etc.
Introduction to Algorithms
L19.10
Academic honesty
Introduction to Algorithms
L19.11
Academic honesty
Introduction to Algorithms
L19.12
Academic honesty
Introduction to Algorithms
L19.13
76 No.
1 Yes.
1 Abstain.
Introduction to Algorithms
L19.14
72 None.
2 3 people compared answers.
1 Either 0 or 2.
1 Abstain.
1 10 (the cheater).
20014 by Charles E. Leiserson
Introduction to Algorithms
L19.15
Reread instructions
Introduction to Algorithms
L19.16
Test-taking strategies
Introduction to Algorithms
L19.17
Introduction to Algorithms
L19.18
Introduction to Algorithms
L19.19
Brainstorm
Introduction to Algorithms
L19.20
Introduction to Algorithms
L19.21
Positive attitude
Introduction to Algorithms
L19.22
Introduction to Algorithms
6.046J/18.401J
LECTURE 20
Network Flow I
Flow networks
Maximum-flow problem
Flow notation
Properties of flow
Cuts
Residual networks
Augmenting paths
Prof. Charles E. Leiserson
Flow networks
Introduction to Algorithms
L20.2
Flow networks
2
3
3
1
ss
2
20014 by Charles E. Leiserson
3
Introduction to Algorithms
tt
2
2
L20.3
Flow networks
p(u, v) p(v,
u ) = 0 .
vV
vV
Introduction to Algorithms
L20.4
Flow networks
p(u, v) p(v, u ) = 0 .
vV
vV
vV
vV
Introduction to Algorithms
L20.5
A flow on a network
positive
flow
capacity
2:2
2:3
1:3
0:1
ss
2:2
Introduction to Algorithms
tt
1:2
1:2
L20.6
A flow on a network
positive
flow
capacity
2:2
2:3
1:3
0:1
ss
2:2
2:3
tt
1:2
1:2
Flow into u is 2 + 1 = 3.
Flow out of u is 0 + 1 + 2 = 3.
The value of this flow is 1 0 + 2 = 3.
20014 by Charles E. Leiserson
Introduction to Algorithms
L20.7
2:3
0:1
ss
2:2
tt
1:2
2:2
Introduction to Algorithms
L20.8
Flow cancellation
vv
2:3
vv
1:2
uu
1:3
0:2
uu
Introduction to Algorithms
L20.9
A notational simplification
Introduction to Algorithms
L20.10
A notational simplification
f (u, v) = 0. One
instead of two.
vV
Skew symmetry: For all u, v V,
f (u, v) = f (v, u).
20014 by Charles E. Leiserson
Introduction to Algorithms
L20.11
Equivalence of definitions
Introduction to Algorithms
L20.12
Equivalence of definitions
Flow conservation:
( p(u, v) p(v, u ) )
f (u , v) =
vV
= p (u , v) p (v, u )
vV
vV
vV
Skew symmetry:
f (u, v) = p(u, v) p(v, u)
= (p(v, u) p(u, v))
= f (v, u).
20014 by Charles E. Leiserson
Introduction to Algorithms
L20.13
Proof (continued)
() Let
p(u, v) =
0
if f(u, v) 0.
vV
vV
Introduction to Algorithms
vV
L20.14
Notation
= f ( s, V ) .
Introduction to Algorithms
L20.15
Lemma.
f (X, X) = 0,
f (X, Y) = f (Y, X),
f (XY, Z) = f (X, Z) + f (Y, Z) if XY = .
Introduction to Algorithms
L20.16
Lemma.
f (X, X) = 0,
f (X, Y) = f (Y, X),
f (XY, Z) = f (X, Z) + f (Y, Z) if XY = .
Proof.
|f| =
=
=
=
=
f (s, V)
f (V, V) f (Vs, V)
Omit braces.
f (V, Vs)
f (V, t) + f (V, Vst)
f (V, t).
Introduction to Algorithms
L20.17
2:2
2:3
2:3
0:1
ss
2:2
3:3
| f | = f (s, V) = 4
Introduction to Algorithms
tt
0:2
2:2
f (V, t) = 4
L20.18
Cuts
2:3
0:1
ss
2:2
3:3
tt
0:2
2:2
f (S, T) = (2 + 2) + ( 2 + 1 1 + 2)
=4
S
T
Introduction to Algorithms
L20.19
Another characterization of
flow value
Lemma. For any flow f and any cut (S, T), we
have | f | = f (S, T).
Introduction to Algorithms
L20.20
Another characterization of
flow value
Lemma. For any flow f and any cut (S, T), we
have | f | = f (S, T).
Proof.
Introduction to Algorithms
L20.21
Capacity of a cut
2:2
2:3
2:3
0:1
ss
2:2
tt
0:2
3:3
S
T
2:2
c(S, T) = (3 + 2) + (1 + 2 + 3)
= 11
20014 by Charles E. Leiserson
Introduction to Algorithms
L20.22
Introduction to Algorithms
L20.23
f = f (S ,T )
= f (u , v)
uS vT
c(u , v)
uS vT
= c( S , T ) .
Introduction to Algorithms
L20.24
Residual network
Introduction to Algorithms
L20.25
Residual network
G:
0:1
uu
vv
Gf :
3:5
Introduction to Algorithms
uu
vv
2
L20.26
Residual network
G:
0:1
uu
vv
Gf :
3:5
uu
vv
2
Lemma. | Ef | 2| E |.
20014 by Charles E. Leiserson
Introduction to Algorithms
L20.27
Augmenting paths
Definition. Any path from s to t in Gf is an augmenting path in G with respect to f. The flow
value can be increased along an augmenting
path p by c f ( p) = min {c f (u , v)}.
(u ,v )p
Introduction to Algorithms
L20.28
Augmenting paths
Definition. Any path from s to t in Gf is an augmenting path in G with respect to f. The flow
value can be increased along an augmenting
path p by c f ( p) = min {c f (u , v)}.
(u ,v )p
Ex.:
3:5
G:
0:2
2:5
ss
cf (p) = 2
Gf :
2:6
tt
2
5:5
7
2:3
2
ss
tt
3
2
Introduction to Algorithms
L20.29
1. f is a maximum flow.
2. Gf contains no augmenting paths.
3. | f | = c(S, T) for some cut (S, T) of G.
Introduction to Algorithms
L20.30
1. f is a maximum flow.
2. Gf contains no augmenting paths.
3. | f | = c(S, T) for some cut (S, T) of G.
Proof (and algorithms). Next time.
Introduction to Algorithms
L20.31
Introduction to Algorithms
6.046J/18.401J
LECTURE 21
Network Flow II
Max-flow, min-cut theorem
Ford-Fulkerson algorithm
and analysis
Edmonds-Karp algorithm
and analysis
Best algorithms to date
Prof. Charles E. Leiserson
Introduction to Algorithms
L21.2
Introduction to Algorithms
L21.3
Introduction to Algorithms
L21.4
Introduction to Algorithms
L21.5
Proof (continued)
ss
path in Gf
uu
vv
S
Introduction to Algorithms
L21.6
Ford-Fulkerson max-flow
algorithm
Algorithm:
Introduction to Algorithms
L21.7
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
109
G:
ss
109
109
tt
109
Introduction to Algorithms
L21.8
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
0:109
G:
0:1
ss
0:109
0:109
tt
0:109
Introduction to Algorithms
L21.9
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
0:109
G:
0:1
ss
0:109
0:109
tt
0:109
Introduction to Algorithms
L21.10
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
1:109
G:
1:1
ss
0:109
0:109
tt
1:109
Introduction to Algorithms
L21.11
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
1:109
G:
1:1
ss
0:109
0:109
tt
1:109
Introduction to Algorithms
L21.12
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
1:109
G:
0:1
ss
1:109
1:109
tt
1:109
Introduction to Algorithms
L21.13
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
1:109
G:
0:1
ss
1:109
1:109
tt
1:109
Introduction to Algorithms
L21.14
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
2:109
G:
1:1
ss
1:109
1:109
tt
2:109
Introduction to Algorithms
L21.15
Ford-Fulkerson max-flow
algorithm
Algorithm:
Can be slow:
2:109
G:
1:109
1:1
ss
1:109
tt
2:109
Introduction to Algorithms
L21.16
Edmonds-Karp algorithm
Introduction to Algorithms
L21.17
Monotonicity lemma
Introduction to Algorithms
L21.18
Monotonicity lemma
Introduction to Algorithms
L21.19
Introduction to Algorithms
L21.20
Case: (u, v) Ef .
Thus, we have
(v) = (u) 1
(breadth-first path)
(u) 1
(induction)
= (v) 2
(breadth-first path)
< (v) ,
thereby establishing monotonicity for this case, too.
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.21
Introduction to Algorithms
L21.22
Introduction to Algorithms
L21.23
Gf :
ss
tt
3
1
Introduction to Algorithms
2
November 29, 2004
L21.24
Introduction to Algorithms
L21.25
Example:
uu
ss
tt
vv
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.26
Example:
(u) = 5
uu
ss
tt
vv
(v) = 6
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.27
Example:
(u) = 5
uu
ss
tt
vv
(v) = 6
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.28
Example:
(u) 7
uu
ss
tt
vv
(v) 6
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.29
Example:
(u) 7
uu
ss
tt
vv
(v) 6
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.30
Example:
(u) 7
uu
ss
tt
vv
(v) 8
20014 by Charles E. Leiserson
Introduction to Algorithms
L21.31
Introduction to Algorithms
L21.32
Introduction to Algorithms
L21.33
Introduction to Algorithms
L21.34
Best to date
Introduction to Algorithms
L21.35
Introduction to Algorithms
6.046J/18.401J
Lecture 22
Today
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.2
String Matching
T[118]=to be or not to be
P[1..2]=be
Goal: find all shifts 0 s n-m such that
T[s+1s+m]=P
E.g. 3, 16
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.3
Simple Algorithm
for s 0 to n-m
Match 1
for j 1 to m
Introduction to Algorithms
December 1, 2004
L22.4
Results
Introduction to Algorithms
December 1, 2004
L22.5
Worst-case
Knuth-Morris-Pratt77: deterministic
Karp-Rabin81: randomized
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.6
Karp-Rabin Algorithm
Introduction to Algorithms
December 1, 2004
L22.7
Attempt 0
Introduction to Algorithms
December 1, 2004
L22.8
Attempt 1
Assume ={0,1}
number p represented by P
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.9
How to transform
ts=T[s+1]2m-1+T[s+2]2m-2++T[s+m]20
into
ts+1=T[s+2]2m-1+T[s+3]2m-2++T[s+m+1]20 ?
Three steps:
Subtract T[s+1]2m-1
Introduction to Algorithms
December 1, 2004
L22.10
Algorithm
Introduction to Algorithms
December 1, 2004
L22.11
Problem
Introduction to Algorithms
December 1, 2004
L22.12
Attempt 2: Hashing
Introduction to Algorithms
December 1, 2004
L22.13
Problem
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.14
False positives
Represent x=p1e1p2e2pkek
, pi prime, ei1
Since 2 pi , we have x 2k
But x 2m
k m
Introduction to Algorithms
December 1, 2004
L22.15
Algorithm
Algorithm:
Let be a set of 2nm primes, each having O(log n) bits
Choose q uniformly at random from
Compute t0 mod q, t1 mod q, ., and p mod q
Report s such that ts mod q = p mod q
Analysis:
For each s, the probability that TsP but
ts mod q =p mod q
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.16
Details
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.17
Prime density
|PRIMES(N)| N/(2ln N)
Introduction to Algorithms
December 1, 2004
L22.18
N/(2ln N) 2mn
(Note: for such N we have PRIMES(N) 2mn )
Proof:
Introduction to Algorithms
December 1, 2004
L22.19
Prime selection
Piotr Indyk
Introduction to Algorithms
December 1, 2004
L22.20
Introduction to Algorithms
December 1, 2004
L22.21
Final Algorithm
Repeat
Choose q uniformly at random from
{1N}
Until q is prime
Introduction to Algorithms
December 1, 2004
L22.22
Introduction to Algorithms
6.046J/18.401J
Lecture 24
What to do if:
Divide and conquer
Dynamic programming
Greedy
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.2
Introduction to Algorithms
December 8, 2004
L24.3
Traveling Salesman
Problem (TSP)
exactly once
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.4
Set Covering
Set Cover:
Input: subsets S1Sn of X,
i Si = X, |X|=m
Output: C {1n} , such
that iC Si = X, and |C|
minimal
Best known algorithm:
O(2n m) time(?)
Piotr Indyk
Introduction to Algorithms
Sets:
December 8, 2004
L24.5
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.6
Approximation Algorithms
Introduction to Algorithms
December 8, 2004
L24.7
Comments on Approximation
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.8
Compute MST T
An edge between any pair of points
Compute a tree-walk W of T
Each edge visited twice
shortcuts
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.9
2-approximation: Proof
Cost(T) Cost(COPT )
Removing an edge from C gives a spanning
Cost(W) = 2 Cost(T)
Each edge visited twice
Cost(C) Cost(W)
Triangle inequality
Cost(C) 2 Cost(COPT )
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.10
Greedy algorithm:
Initialize C=
Repeat until all elements are covered:
Introduction to Algorithms
December 8, 2004
L24.11
X={1,2,3,4,5,6}
Sets:
S1={1,2}
S2={3,4}
S3={5,6}
S4={1,3,5}
Algorithm picks C={4,1,2,3}
Not optimal!
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.12
ln(m)-approximation
Notation:
k=|COPT |
Introduction to Algorithms
December 8, 2004
L24.13
ln(m)-approximation
ui+1 ui (1-1/k)
u0=m
Therefore, after t=k ln m steps, we have
ut u0 (1-1/k)t m (1-1/k)k ln m < m 1/eln m =1
I.e., all elements are covered by the k ln m sets
chosen by greedy algorithm
Opt size is k greedy is ln(m)-approximate
Piotr Indyk
Introduction to Algorithms
December 8, 2004
L24.14
Approximation Algorithms
Introduction to Algorithms
December 8, 2004
L24.15