15-150 Fall 2014

Lecture 5

Tuesday, 9 September

Announcements
HOMEWORK 1

MAX

MEAN

MEDIAN

ST DEV

50

45.3

48.0

9.9

Homework 2 due tonight

Homework 3 out tomorrow

NO CHEATING

Today
Work, (sequential) runtime

Recurrence relations

Exact and approximate solutions

Improving efficiency
program

recurrence

work

asymptotic
Runtime, for large inputs

work (sequential)

span (parallel)

Assuming basic ops take constant time

Give big-O classification
f(n) is O(g(n))
if there are N and c such that
nN, f(n) c g(n)

motivation
Why make an

exponential time

function

when we can make a

quadratic time

function

thats

extensionally

equivalent

asymptotic
Ignore additive constants

n5+1000000 is O(n5)

Absorb multiplicative constants

1000000n5 is O(n5)

Be as accurate as you can

O(n2) O(n3) O(n4)

common terminology
Use and learnlogarithmic,
linear,

polynomial, exponential

recurrences
A recursive function definition suggests a
recurrence relation for work,
i.e. (sequential) runtime

W(n) = work on inputs of size n

counts the number of basic operations
Base cases
Inductive cases

W(0) = k0
W(n) = W(n-1)
W(n) = W(n div 2)

example
A recurrence for Fibonacci numbers
Fib(0) = 1

Fib(1) = 1

Fib(n) = Fib(n-1) + Fib(n-2) for n>1

A recurrence for the work to evaluate Fib(n)

WFib(0) = c0

WFib(1) = c0

WFib(n) = WFib(n-1) + WFib(n-2) + c1
for some constants c0, c1

finding solutions
Try to find a closed form solution for W(n)
(usually, by guessing and induction)

OR Code the recurrence in ML and look for a
common pattern

OR Find solution to a simplified recurrence with
the same asymptotic properties

OR Appeal to table of standard recurrences

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4

=>(1)
=>(4)

M4
2 * (M 3)

M 4 => if 4=0 then ...

=> 2 * exp (4-1)

=> 2 * M (4-1)

=> 2 * M 3

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))

M 3 => if 3=0 then ...

=> 2 * exp (3-1)

=> 2 * M (3-1)

=> 2 * M 2

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))
=>(4) 2 * (2 * (2 * (M 1)))

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))
=>(4) 2 * (2 * (2 * (M 1)))
(4)
=> 2 * (2 * (2 * (2 * (M 0)))

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))
=>(4) 2 * (2 * (2 * (M 1)))
(4)
=> 2 * (2 * (2 * (2 * (M 0)))
=>(2) 2 * (2 * (2 * (2 * 1)))

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))
=>(4) 2 * (2 * (2 * (M 1)))
(4)
=> 2 * (2 * (2 * (2 * (M 0)))
=>(2) 2 * (2 * (2 * (2 * 1)))
=>(4) 16

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let M be (fn n => if n=0 then 1 else 2 * exp(n-1))
exp 4 =>(1) M 4
=>(4) 2 * (M 3)
=>(4) 2 * (2 * (M 2))
exp
=>(4) 2 * (2 * (2 * (M 1)))
(4)
=> 2 * (2 * (2 * (2 * (M 0)))
=>(2) 2 * (2 * (2 * (2 * 1)))
=>(4) 16

(23)
=>

16

exp
Its not hard to prove that for all n0,
(5n+3)
=>

exp n
k,

where k is the numeral for 2n
But do we need to be so accurate?
And does 5n+3 tell us about

actual runtime in milliseconds?
No! But it does tell us runtime is linear.

big-O
Its useful to classify runtimes asymptotically

This abstracts away from additive and
multiplicative constants

(which may be machine-dependent, so not very significant)

And ignores runtime on small inputs

(which may be special-cased in the code, so dont imply much)

For f, g : int -> int we say that

f is O(g)

if there is a constant c and an integer N such
that for all n N,

|f(n)| c * |g(n)|.

(usually f(n) and g(n)

are always positive

so we omit the |-| symbols)

exp
fun exp (n:int):int =

if n=0 then 1 else 2 * exp (n-1)
Let Wexp(n) be the runtime for exp(n)

Wexp(0) = c0

Wexp(n) = Wexp(n-1) + c1

for n>0

for some constants c0 and c1

c0
c1

cost of n=0
cost of n=0, n-1, mult by 2

solution
Can prove by induction on n that
Wexp(n) = c0 + n c1 for n0

the work of
exp(n) is linear in n

classification
W
W

exp(n)

= c0 + n c1

exp(n)

is O(n)

O-class for Wexp(n)

independent of

c0, c1

Let N=0, c = max(c0,c1)+1

For all n0,

Wexp(n) c n

classification
W
W

exp(n)

= c0 + n c1

exp(n)

is O(n)

O-class for Wexp(n)

independent of

c0, c1

summary
Weve shown that for n0,

exp n computes the value of

in O(n) time

n
2

fact is independent of machine details

This
(provided basic operations are constant time)

Can we do better?

faster exp?
The definition of exp relies on the fact that

n
2

=2

n-1
(2 )

Everybody knows that

2n = (2n div 2)2

if n is even

fastexp
fun square(x:int):int = x * x
fun fastexp (n:int):int =

if n=0 then 1 else

if n mod 2 = 0 then square(fastexp (n div 2))

else 2 * fastexp(n-1)

fastexp
fun square(x:int):int = x * x
fun fastexp (n:int):int =

if n=0 then 1 else

if n mod 2 = 0 then square(fastexp (n div 2))

else 2 * fastexp(n-1)
fastexp 4 = square(fastexp 2)

= square(square (fastexp 1))

= square(square (2 * fastexp 0))

= square(square (2 * 1))

= square 4 =16

is it faster?
fun fastexp (n:int):int =

if n=0 then 1 else

if n mod 2 = 0 then square(fastexp (n div 2))

else 2 * fastexp(n-1)

Let Wfastexp(n) be the runtime for fastexp(n)

Wfastexp(0) = k0

Wfastexp(n) = Wfastexp(n div 2) + k1 for n>0, even

Wfastexp(n) = Wfastexp(n-1) + k2
for some constants k0, k1, k2

for n>0, odd

is it faster?
fun fastexp (n:int):int =

if n=0 then 1 else

if n mod 2 = 0 then square(fastexp (n div 2))

else 2 * fastexp(n-1)

Let Wfastexp(n) be the runtime for fastexp(n)

Wfastexp(0) = c0

Wfastexp(1) = c1

Wfastexp(n) = Wfastexp(n div 2) + c2

for n>1, even

Wfastexp(n) = Wfastexp(n div 2) + c3

for n>1, odd

for some constants c0, c1, c2, c3

solution?
Not so obvious how to solve for W (n)

A closed form would involve c , c , c , c
But we only care about asymptotic behavior

So we can work with a simpler recurrence
fastexp

that has the same asymptotic properties

simplification:

choose each constant to be 1

simplified recurrence
Let Tfastexp(n) be given by
Tfastexp(0) = 1

Tfastexp(1) = 1

Tfastexp(n) = Tfastexp(n div 2) + 1 for n>1
Wfastexp(n) and Tfastexp(n) are

asymptotically equivalent
(belong to the same big-O class)

solution
For n>1, T

fastexp(n)

is defined like log(n)

fun log n =

if n=1 then 0 else log(n div 2) + 1

We know that log(n) = log (n) for all n>0

Can show that there is a constant c such that
2

Tfastexp(n) c log2(n)
for all large enough n

classification
T (n) is O(log n)

W (n) depends on c , c , c , c

We can find constants c and c
fastexp

fastexp

low

high

such that
clow Tfastexp(n) Wfastexp(n) chigh Tfastexp(n)
and this implies that

Wfastexp(n) is also O(log2(n))

its faster
Work of exp(n) is O(n)

Work of fastexp(n) is O(log n)

O(log n) is a proper subset of O(n)

So fastexp is faster than exp, asymptotically

even faster?
The definition of fastexp relies on

2n = (2n div 2)2
2n = 2 (2n-1)

if n is even
if n is odd

A moments thought tells us that

2n = 2 (2(n div 2))2

if n is odd

pow
fun pow (n:int):int =

case n of

0 => 1
| 1 => 2

| _ => let
val k = pow(n div 2)

in

if n mod 2 = 0 then k*k else 2*k*k

end

work of pow(n)
Wpow(0) = c0

Wpow(1) = c1

Wpow(n) = c2 + Wpow(n div 2) for n>1
Same recurrence as Wfastexp
Same asymptotic behavior
pow(n) is O(log n)

comparison
fastexp(n) and pow(n) have O(log n) work.

For n0, fastexp(n) = pow(n).

For n<0, fastexp(n) and pow(n) fail to terminate.
So fastexp and pow are extensionally equivalent
and have the same asymptotic work classification.

badpow
fun badpow (n:int):int =

case n of

0 => 1
| 1 => 2

| _ => let
val k2 = badpow(n div 2)*badpow(n div 2)

in

if n mod 2 = 0 then k2 else 2*k2

end

work of badpow(n)
Wbadpow(0) = c0

Wbadpow(1) = c1

Wbadpow(n) = c2 + 2 Wbadpow(n div 2)

for n>1
Same asymptotic class as
Tbadpow(0) = 1

Tbadpow(1) = 1

Tbadpow(n) = 1 + 2 Tbadpow(n div 2)

for n>1

examples
Tbadpow(20) = 1
Tbadpow

1
(2 )

= 1 + 2*Tbadpow
= 1 + 2*1 = 3

0
(2 )

Tbadpow(22) = 1 + 2*Tbadpow(21)

= 1 + 2*3 = 7
Tbadpow(2m) = 2m+1 - 1

analysis
Tbadpow(2m) is O(2m)
is
W
This implies that W
badpow

m
(2 )

m
O(2 )

badpow(n)

is O(n)

Wpow(n) is O(log n)
O(log n) O(n)
pow is asymptotically faster than badpow

standard results
T(n) = c + T(n-1)
T(n) = c + n + T(n-1)
T(n) = c + T(n div 2)
T(n) = c + 2 T(n div 2)
T(n)= c + n + 2T(n div 2)
T(n) = c + k T(n-1)

O(n)

2
O(n )

O(log n)

O(n)

O(n log n)

n
O(k )