Relevant Mathematics
Logic Quantifiers The Time Complexity of an Algorithm Classifying Functions Adding Made Easy Recurrence Relations Jeff Edmonds York University Lecture ?
COSC 3101
Some Math
Logic Quantifiers g "b Loves(b,g) "b g Loves(b,g) Done here
Time Complexity t(n) = Q(n2)
In Iterative Algs (GCD) Repeated here
Some Math
Logs and Exps
2a × 2b = 2a+b 2log n = n
Classifying Functions f(i) = nQ(n)
Time
You are on your own.
Done here
Input Size
Some Math
Adding Made Easy ∑i=1 f(i). In Iterative Algs (Insertion Sort) In Recursive Algs (Multiplying)
Repeated here.
Recurrence Relations T(n) = a T(n/b) + f(n)
In Recursive Algs (Multiplying, Towers of Hanoi, Merge Sort) Repeated here.
Quantifers
Quantifiers,
Do you understand them?
g, " b, Loves(b, g)
“A(I)=P(I)” means algorithm A gives the required output for problem P on instance I. Time(A,I) is the running time of algorithm A on instance I. T(n) some function like n2. Express: • Problem P is computable by some algorithm. • Problem P is computable in time T(n). • Problem P is computable in polynomial time. • The computational class “Exponential Time” is strictly bigger than the computational class “Polynomial Time”.
Understand Quantifiers!!!
" b, Loves(b, MM) A relation/predicate •returns True/False •depending whether objects b and MM have relation Loves.
Sam Mary Beth Marilin Monro Ann
Loves(Sam,Mary) = False Loves(Sam,Beth) = True
Bob John Fred
Understand Quantifiers!!!
" b, Loves(b, MM) “Every boy loves Marilin Monro”
Sam Bob John
Fred
Mary Beth Marilin Monro
Ann
Sam Bob John Fred
Mary Beth Marilin Monro Ann
True
False
Understand Quantifiers!!!
" b, Loves(b, MM) “Every boy loves Marilin Monro” b vs MM?
b is a variable. MM is an object.
(cannot appear in or ")
The statement is about MM.
Understand Quantifiers!!!
" b, Loves(b, g) “Every boy loves g” b vs g?
b is a free variable.
(appears in or ")
g is a bound variable.
(does not appear in or ")
The statement is about the set of all boys.
The statement is about g.
Bound variable Object x=5 Free variable loop i = 1..9
++x end loop
Understand Quantifiers!!!
" b, Loves(b, g)
g is a bound variable.
A relation/predicate •returns True/False •depending whether object g has the property Loved.
(does not appear in or ")
The statement is about g.
Loved(g) = " b, Loves(b, g)
Understand Quantifiers!!!
g, " b, Loves(b, g) “There is a girl whom every boy loves”
Sam Bob John
Fred
Mary Beth Marilin Monro
Ann
Sam Bob John Fred
Mary Beth Marilin Monro Ann
True
False
Understand Quantifiers!!!
" b, g, Loves(b, g) “Every boy loves some girl.”
Sam Bob John
Fred
Mary Beth Marilin Monro
Ann
Sam Bob John Fred
Mary Beth Marilin Monro Ann
True Still True
False True
Understand Quantifiers!!!
g, " b, Loves(b, g) One girl
Sam Bob John
Fred
vs
" b, g, Loves(b, g)
Could be a separate girl for each boy.
Sam Bob John Fred Mary Beth Marilin Monro Ann
Mary Beth Marilin Monro
Ann
Understand Quantifiers!!!
Proof: g, " b, Loves(b, g) We read this left to right. I produce the object when it is a .
I produce the object when it is a ". I can always win if and only if statement is true.
Sam Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Easy Proof: b, Loves(b, MM) I claim that there is a boy that does not love MM. Oh yeah! Prove it. I produce Sam And prove Sam does not loves MM. I can win. Hence, statement is true.
Sam Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, Loves(b, MM) I don’t like proof by contradiction. By way of contradiction assume b, Loves(b, MM) Let b’ be the object that is assumed to exist. Prove Loves(b’,MM) Contradiction. Hence, true.
Sam Bob John Fred
Mary Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, Loves(b, MM) Instead play the game. I claim that every boy loves MM. No! I produce Sam Sam does not loves MM. I cannot win. Hence, statement is false.
Sam Bob John Fred Mary Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, Loves(b, MM) I claim that every boy loves MM. No! I produce Sam I prove Sam does loves MM. I can always win. Hence, statement is true.
Sam Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, Loves(b, MM) I claim that every boy loves MM. No! I produce boy x. I prove x does loves MM. This proof needs to work without knowing x I can always win. Hence, statement is true.
Sam
Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, Loves(b, MM) I claim that every boy loves MM. No! I produce boy x. I give an “alg” with input x, and output a proof that x loves MM. I can always win. Hence, statement is true.
Sam
Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: g, " b, Loves(b, g) I claim “there is a girl whom every boy loves.” Oh yeah! Prove it. I produce MM I proceed to prove “every boy loves MM”
Sam Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: g, " b, Loves(b, g) I proceed to prove “every boy loves MM” No! I produce boy x. I give an “alg” with input x, and output a proof that x loves MM. I can always win. Hence, statement is true.
Sam
Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, g, Loves(b, g) I claim “Every boy loves some girl.” No! I produce boy x.
I proceed to prove “x loves some girl.”
Sam
Mary
I produce girl y I prove x loves y. I can always win. Hence, statement is true.
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, g, Loves(b, g) I claim “Every boy loves some girl.” No! I produce boy x.
I proceed to prove “x loves some girl.”
Sam
Mary
I give an “alg” with input x, and output an y and a proof that
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
Proof: " b, g, Loves(b, g) I give an “alg” with input y, and output an x and a proof that x loves y. You give Sam, I give Beth. You give Bob, I give MM. …
Sam
Mary
Bob John Fred
Beth Marilin Monro Ann
Understand Quantifiers!!!
[ g, ] " b, Loves(b, g) = " g, [ " b, Loves(b, g) ] = " g, b, [ Loves(b, g) ] = " g, b, Loves(b, g) moves right "
Understand Quantifiers!!!
Proof: " x, y, x+y=5 Let x be an arbitrary real number. Let y = 5-x.
The relation is true. x+y = x + (5-x) = 5 I can always win. Hence, the statement is true.
Understand Quantifiers!!!
Proof: " x, y, x×y=5 Let x be an arbitrary real number. Let y = 5/x.
The relation is true. x×y = x × 5/x = 5 I can always win. Hence, the statement is true.
What a minute the statement is false!
Understand Quantifiers!!!
Proof: [ " x, y, x×y=5 ] = x, " y, x×y =5 Let x=0.
Let y be an arbitrary real number. The negated relation is true. x×y = 0×y = 0 = 5
I can always win. Hence, the negated statement is true and the original statement is false.
Classifying Functions
Theta BigOh
f(n) = θ(g(n)) f(n) = O(g(n)) f(n) = Ω(g(n))
f(n) ≈ c g(n) f(n) ≤ c g(n) f(n) ≥ c g(n)
Omega
Little Oh
f(n) = o(g(n)) f(n) << c g(n)
Little Omega
f(n) = ω(g(n)) f(n) >> c g(n)
Giving an idea of how fast a function grows without going into too much detail.
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
f(n) is sandwiched between c1g(n) and c2g(n)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
f(n) is sandwiched between c1g(n) and c2g(n) for some sufficiently small c1 (= 0.0001) for some sufficiently large c (= 1000)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
For all sufficiently large n
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
For all sufficiently large n For some definition of “sufficiently large”
Definition of Theta
3n2 + 7n + 8 = θ(n2)
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
Definition of Theta
3n2 + 7n + 8 = θ(n2)
True
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
3 4 8 "n 8 3·n2 3n2 + 7n + 8 4·n2
7n + 8 1·n2 7 + 8/n 1·n
Definition of Theta
3n2 + 7n + 8 = θ(n)
The reverse statement
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
" c1, c2, " n0, n n0, c1g( n) f ( n) or f ( n) c2 g( n)
Definition of Theta
3n2 + 7n + 8 = nθ(1)
Polynomial time
c1, c2, n0, " n n0 nc1 f(n) nc2
Definition of Theta
3n2 + 7n + 8 = 2θ(n)
Exponential time
c1, c2, n0, " n n0
2c1n
f(n) 2c2n
Order of Quantifiers f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n) n0, " n n0, c1, c2, c1g( n) f ( n) c2 g( n)
?
Order of Quantifiers f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n) n0, " n n0, c1, c2, c1g( n) f ( n) c2 g( n)
No! It cannot be a different c1 and c2 for each n.
The Time Complexity of a Problem P
Merge, Quick, and Heap Sort can sort N numbers using O(N log N) comparisons between the values.
Theorem: No algorithm can sort faster.
The Time Complexity of a Problem P: The minimum time needed by an algorithm to solve it.
The Time Complexity of a Problem P
The minimum time needed by an algorithm to solve it. Upper Bound: Problem P is computable in time Tupper(n) if there is an algorithm A which • outputs the correct answer • in this much time given any input instance I. A, " I, A(I)=P(I) and Time(A,I) Tupper(|I|) Eg: Sorting computable in Tupper(n) = O(n2) time.
The Time Complexity of a Problem P
The minimum time needed by an algorithm to solve it.
A, " I, A(I)=P(I) and Time(A,I) Tupper(|I|)
" I, A, A(I)=P(I) and Time(A,I) Tupper(|I|) What does this say? One algorithm A, that works for every instance I vs A different algorithm AI for each instance I.
The Time Complexity of a Problem P
The minimum time needed by an algorithm to solve it.
A, " I, A(I)=P(I) and Time(A,I) Tupper(|I|)
" I, A, A(I)=P(I) and Time(A,I) Tupper(|I|) What does this say? True for any problem P and time Tupper. Given fixed I. Its output is P(I). Let AP(I) be the algorithm that outputs the string P(I).
The Time Complexity of a Problem P
The minimum time needed by an algorithm to solve it. Lower Bound: Time Tlower(n) is a lower bound for problem p if no algorithm solve the problem faster.
There may be algorithms that give the correct answer or run quickly on some inputs instance.
The Time Complexity of a Problem P
The minimum time needed by an algorithm to solve it. Lower Bound: Time Tlower(n) is a lower bound for problem p if no algorithm solve the problem faster. But for every algorithm, there is at least one instance I for which either the algorithm gives the wrong answer or it runs in too much time. " A, I, A(I) = P(I) or Time(A,I) Tlower(|I|) Eg: No algorithm can sort N values in T = sqrt(N) time.
The Time Complexity of a Problem P
The minimum time needed by an algorithm to solve it. Upper Bound: A, " I, A(I)=P(I) and Time(A,I) Tupper(|I|) Lower Bound: " A, I, A(I) ≠ P(I) or Time(A,I) Tlower(|I|) “There is” and “there isn’t a faster algorithm” are almost negations of each other.
Prover-Adversary Game
Upper Bound: A, " I, A(I)=P(I) and Time(A,I) Tupper(|I|) I have an algorithm A that I claim works and is fast. Oh yeah, I have an input I for which it does not.
I win if A on input I gives •the correct output •in the allotted time. What we have been doing all along.
Prover-Adversary Game
Lower Bound: " A, I, [ A(I) = P(I) or Time(A,I) Tlower(|I|)] I have an algorithm A that I claim works and is fast. Oh yeah, I have an input I for which it does not . I win if A on input I gives •the wrong output or •runs slow.
Prover-Adversary Game
Lower Bound: " A, I, [ A(I) = P(I) or Time(A,I) Tlower(|I|)] I have an algorithm A that I claim works and is fast. Lower bounds are very hard to prove, because I must consider every algorithm no matter how strange.
Quantifiers, Do you understand them?
g, " b, Loves(b, g)
“A(I)=P(I)” means algorithm A gives the required output for problem P on instance I. Time(A,I) is the running time of algorithm A on instance I. T(n) some function like n2. Express: • Problem P is computable by some algorithm. • Problem P is computable in time T(n). • Problem P is computable in polynomial time. • The computational class “Exponential Time” is strictly bigger than the computational class “Polynomial Time”.
• Problem P is computable by some algorithm. A, " I, A(I)=P(I) • Problem P is computable in time T(n). A, " I, A(I)=P(I) & Time(A,I) ≤ T(|I|) • Problem P is computable in polynomial time. A, c, n0," I, A(I)=P(I) & (|I| < n0 or Time(A,I) ≤ |I|c) •Problem P is not computable in polynomial time. " A, c, n0, I, A(I)≠P(I) or (|I| ≥ n0 & Time(A,I) > |I|c) • Problem P is computable in exponential time. A, c, n0," I, A(I)=P(I) & (|I| < n0 or Time(A,I) ≤ 2c|I|) •The computational class “Exponential Time” is strictly bigger than the computational class “Polynomial Time”.
P, [" A, c, n0, I, A(I)≠P(I) or (|I| ≥ n0 & Time(A,I) > |I|c)] & [ A, c, n0," I, A(I)=P(I) & (|I| < n0 or Time(A,I) ≤ 2c|I|)]
Some Math
Logs and Exps
2a × 2b = 2a+b 2log n = n
You are on your own.
The Time Complexity of an Algorithm
Specifies how the running time depends on the size of the input.
Purpose?
Purpose
• To estimate how long a program will run. • To estimate the largest input that can reasonably be given to the program. • To compare the efficiency of different algorithms. • To help focus on the parts of code that are executed the largest number of times. • To choose an algorithm for an application.
Time Complexity Is a Function
Specifies how the running time depends on the size of the input.
A function mapping “size” of input “time” T(n) executed .
Definition of Time?
Definition of Time
• # of seconds (machine dependent). • # lines of code executed. • # of times a specific operation is performed (e.g., addition).
Which?
Definition of Time
• # of seconds (machine dependent). • # lines of code executed. • # of times a specific operation is performed (e.g., addition). These are all reasonable definitions of time, because they are within a constant factor of each other.
Size of Input Instance?
83920
Size of Input Instance
5
83920
2’’
• • • • Size of paper # of bits # of digits Value - n = 2 in2 - n = 17 bits - n = 5 digits - n = 83920
1’’
Which are reasonable?
Size of Input Instance
5
83920
2’’
• • • • Size of paper # of bits # of digits Value - n = 2 in2 - n = 17 bits - n = 5 digits - n = 83920
1’’
• Intuitive
Size of Input Instance
5
83920
2’’
• • • • Size of paper # of bits # of digits Value - n = 2 in2 - n = 17 bits - n = 5 digits - n = 83920
1’’
• Intuitive • Formal
Size of Input Instance
5
83920
2’’
• • • • Size of paper # of bits # of digits Value - n = 2 in2 - n = 17 bits - n = 5 digits - n = 83920
1’’
• Intuitive • Formal • Reasonable # of bits = 3.32 * # of digits
Size of Input Instance
5
83920
2’’
• • • • Size of paper # of bits # of digits Value - n = 2 in2 - n = 17 bits - n = 5 digits - n = 83920
1’’
• • • •
Intuitive Formal Reasonable Unreasonable # of bits = log2(Value) Value = 2# of bits
Two Example Algorithms
• Sum N array entries:
A(1) + A(2) + A(3) + …
• Factor value N:
Input N=5917 & Output N=97*61. Algorithm N/2, N/3, N/4, …. ?
Time = N
Time = N
Time? Is this reasonable? No! One is considered fast and the other slow!
Two Example Algorithms
• Sum N array entries:
A(1) + A(2) + A(3) + …
• Factor value N:
N = hard drive = 60G < 236 Time = N
Input N=5917 & Output N=97*61. N = crypto key = 2100 Algorithm N/2, N/3, N/4, …. ?
Time = N
Standard input size?
Two Example Algorithms
• Sum N array entries:
A(1) + A(2) + A(3) + …
size n = N Time = N
size n = log N Time = N
• Factor value N:
Input N=5917 & Output N=97*61. Algorithm N/2, N/3, N/4, …. ?
Size of Input Instance?
Two Example Algorithms
• Sum N array entries:
A(1) + A(2) + A(3) + …
size n = N Time = N = n
• Factor value N:
Input N=5917 & Output N=97*61. Algorithm N/2, N/3, N/4, …. ?
size n = log N Time = N = 2n Time as function of input size? Time for you to solve problem instance as a function of time for me to give you the problem instance.
Two Example Algorithms
• Sum N array entries:
A(1) + A(2) + A(3) + …
size n = N Time = N = n
size n = log N Time = N = 2n
• Factor value N:
Input N=5917 & Output N=97*61. Algorithm N/2, N/3, N/4, …. ?
Linear vs Exponential Time!
Size of Input Instance?
14,23,25,30,31,52,62,79,88,98
Size of Input Instance
10
14,23,25,30,31,52,62,79,88,98
• # of elements = n = 10 elements
Size of Input Instance
10
14,23,25,30,31,52,62,79,88,98
• # of elements = n = 10 elements
Is this reasonable?
Size of Input Instance
10
14,23,25,30,31,52,62,79,88,98
• # of elements = n = 10 elements
Reasonable
If each element has size c # of bits = c * # of elements
Size of Input Instance
10
14,23,25,30,31,52,62,79,88,98
• # of elements = n = 10 elements
~ Reasonable
If each element is in [1..n] each element has size log n # of bits = n log n ≈ n
Time Complexity Is a Function
Specifies how the running time depends on the size of the input.
A function mapping # of bits n needed to represent the input # of operations T(n) executed .
Which Input of size n?
There are 2n inputs of size n. Which do we consider for the time T(n)?
Which Input of size n?
Typical Input
Average Case Worst Case
Which Input of size n?
Typical Input But what is typical?
Average Case Worst Case
For what distribution? •Time for all inputs is bound. •Easiest to compute
What is the height of tallest person in the class?
Bigger than this? Smaller than this?
Need to look at only one person
Need to look at every person
Time Complexity of Algorithm
The time complexity of an algorithm is the largest time required on any input of size n.
O(n2): Prove that for every input of size n,
the algorithm takes no more than cn2 time. Ω(n2): Find one input of size n, for which the algorithm takes at least this much time. θ (n2): Do both.
Time Complexity of Problem
The time complexity of a problem is the time complexity of the fastest algorithm that solves the problem.
O(n2): Provide an algorithm that solves the
problem in no more than this time. Ω(n2): Prove that no algorithm can solve it faster. θ (n2): Do both.
Classifying Functions
T(n) 10 100 1,000 10,000
log n
3
6
10
9
31
13
100
amoeba
bird
n1/2 3
n 10
n log n 30
100
600
1,000 10,000 108 1012 103000
human elephant dinosaur the universe
9,000 130,000 my father
10,000 106 n2 100 109 n3 1,000 106 2n 1,024 1030 10300
Note: The universe contains approximately 1050 particles.
Classifying Functions
Functions
Poly Logarithmic
(log n)θ(1)
Exponential
Polynomial
Exp
Double Exp
θ(1)
Constant
5 << (log n)5 << n5 <<
25n <<
2
n5
<<
2
25n
nθ(1)
2θ(n)
2
nθ(1)
2
2θ(n)
Classifying Functions
Polynomial = nθ(1) Linear
θ(n)
θ(n2)
Quadratic
?
log(n) not absorbed because not Mult-constant
θ(n3) θ(n4) θ(n3 log7(n))
Cubic
Others
Classifying Functions
We consider two (of many) levels in this hierarchy
Functions
θ(n2) θ(n3 log(n)) 8·n2 7·n3log(n)
Poly. nθ(1)
θ(24n / n100)
8·24n / n100
Exp.
2θ(n)
θ(24n)
Ignore Power-constant Ignore Mult-constant Individuals
8·24n + n3
BigOh and Theta?
•5n2 + 8n + 2log n = Q(n2) •5n2 log n + 8n + 2log n = Q(n2 log n)
Drop low-order terms. Drop multiplicative constant.
• nc • n0.0001 • n10000 • 5n2 + 8n + 2log n • 5n2 log n • 5n2.5 Drop low-order terms. Drop constant, log, (and poly) factors. Ignore power constant. Write nθ(1)
Which Functions are Polynomials?
Which Functions are Exponential?
• 2n • 20.0001 n • 210000 n • 8n • 2n / n100 •2n · n100 Drop low-order terms. Drop constant, log, and poly factors. Ignore power constant. Write 2θ(n)
Notations
Theta BigOh Omega Little Oh Little Omega
f(n) = θ(g(n)) f(n) = O(g(n)) f(n) = Ω(g(n))
f(n) ≈ c g(n) f(n) ≤ c g(n) f(n) ≥ c g(n)
f(n) = o(g(n)) f(n) << c g(n) f(n) = ω(g(n)) f(n) >> c g(n)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
f(n) is sandwiched between c1g(n) and c2g(n)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
f(n) is sandwiched between c1g(n) and c2g(n) for some sufficiently small c1 (= 0.0001) for some sufficiently large c (= 1000)
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
For all sufficiently large n
Definition of Theta
f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
For all sufficiently large n For some definition of “sufficiently large”
Definition of Theta
3n2 + 7n + 8 = θ(n2)
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n)
Order of Quantifiers f(n) = θ(g(n))
c1, c2, n0, " n n0, c1g( n) f ( n) c2 g( n) n0, " n n0, c1, c2, c1g( n) f ( n) c2 g( n)
No! It cannot be a different c1 and c2 for each n.
Understand Quantifiers!!!
g, " b, loves( b, g )
One girl
Sam Mary
" b, g, loves( b, g )
Could be a separate girl for each boy.
Sam Mary
Bob John Fred
Beth
Marilin Monro
Bob John Fred
Beth
Marilin Monro
Ann
Ann
End of first part of math review
Recursive Algorithms
Adding The Classic Techniques
Evaluating ∑i=1 f(i).
n
Arithmetic Sum
Gauss
∑i=1..n i = 1 + 2 + 3 + . . . + n = ?
Adding Made Easy
1 + 2 + 3 + ... + n-1 + n = S
= number of white dots
1
2........n
Adding Made Easy
1 n + + 2 + 3 + ... ... + + n-1 + 2 + n 1 = = S S
= number of white dots = number of yellow dots
n-1 +
n-2 +
n ....... 2 1
1
2........n
Adding Made Easy
1 + 2 + 3 + ... + n-1 + n = S
= number of white dots = number of yellow dots
n n
n
+
n-1 +
n-2 +
...
+
2
+
1
=
S
2S dots
n
= n(n+1) dots in the grid
n
n
n (n+1)
n
S=
2
n+1 n+1
n+1
n+1 n+1
Arithmetic Sum
Gauss
∑i=1..n i = 1 + 2 + 3 + . . . + n
= Q(# of terms · last term)
True when ever terms increase slowly
Geometric Sum
∑i=0..n 2i
= 1 + 2 + 4 + 8 +. . . + 2n
=
?
Geometric Sum
Geometric Sum
∑i=0..n 2i
= 1 + 2 + 4 + 8 +. . . + 2n = 2 · last term - 1
Geometric Sum
∑i=0..n ri
= r0 + r1 + r2 +. . . + rn
=
?
Geometric Sum
S = 1 + r + r + r +... +r
2 3
n n +1
Sr =
r + r + r +... +r + r
2 3 n
S - Sr = 1
-1 S= r -1 r
n +1
-r
n +1
Geometric Sum
∑i=0..n
-1 ri = r -1
r
n +1
When r>1?
Geometric Sum
∑i=0..n
-1 ri = = θ(rn) r -1
r
n +1
When r>1
Biggest Term
Geometric Increasing
∑i=0..n ri
=
= r0 + r1 + r2 +. . . + rn
Q(biggest term)
True when ever terms increase quickly
Geometric Sum
∑i=0..n
ri
1- r = 1-r
n +1
When r<1?
Geometric Sum
∑i=0..n
ri
1- r = 1-r
n +1
When r<1
= θ(1)
Biggest Term
Bounded Tail
∑i=0..n ri
=
= r0 + r1 + r2 +. . . + rn
Q(1)
True when ever terms decrease quickly
Harmonic Sum
∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n = ?
Sum of Shrinking Function
f(i) = 1 ∑i=1..n f(i) = n
n
Sum of Shrinking Function
f(i) = ? ∑i=1..n f(i) = n1/2
n
Sum of Shrinking Function
f(i) = 1/2i ∑i=1..n f(i) = 2
Sum of Shrinking Function
f(i) = 1/i ∑i=1..n f(i) = ?
n
Harmonic Sum
Harmonic Sum
∑i=1..n 1/i = 1/1 + 1/2 + 1/3 + 1/4 + 1/5 + …+ 1/n = Q(log(n))
Approximating Sum by Integrating
∑i=1..n f(i) ≈ ∫x=1..n f(x) dx
The area under the curve approximates the sum
Adding Made Easy
(For +, -, ×, , exp, log functions f(n))
Adding Made Easy
Adding Made Easy
• Geometric Like: If f(n) 2Ω(n), then ∑i=1..n f(i) = θ(f(n)). • Arithmetic Like: If f(n) = nθ(1)-1, then ∑i=1..n f(i) = θ(n · f(n)). • Harmonic: If f(n) = 1/n , then ∑i=1..n f(i) = logen + θ(1). • Bounded Tail: If f(n) n-1-Ω(1), then ∑i=1..n f(i) = θ(1).
(For +, -, , , exp, log
functions f(n))
This may seem confusing, but it is really not. It should help you compute most sums easily.
Recurrence Relations
T(1) = 1 T(n) = a T(n/b) + f(n)
Recurrence Relations Time of Recursive Program
procedure Eg(In) Recurrence relations n = |In| ? arise from the timing if(n1) then of recursive programs. put “Hi” else Let T(n) be the # of “Hi”s loop i=1..f(n) on an input of “size” n. put “Hi” loop i=1..a In/b = In cut in b pieces Eg(In/b)
Recurrence Relations Time of Recursive Program
procedure Eg(In) n = |In| if(n1) then put “Hi” else loop i=1..f(n) put “Hi” loop i=1..a In/b = In cut in b pieces Eg(In/b)
Given size 1, the program outputs T(1)=1 Hi’s.
Recurrence Relations Time of Recursive Program
procedure Eg(In) n = |In| if(n1) then put “Hi” else loop i=1..f(n) put “Hi” loop i=1..a In/b = In cut in b pieces Eg(In/b) Given size n, the stackframe outputs f(n) Hi’s.
Recurrence Relations Time of Recursive Program
procedure Eg(In) n = |In| if(n1) then put “Hi” else loop i=1..f(n) Recursing put “Hi” on an instances of size n/b loop i=1..a generates T(n/b) “Hi”s. In/b = In cut in b pieces Eg(In/b)
Recurrence Relations Time of Recursive Program
procedure Eg(In) n = |In| if(n1) then put “Hi” else loop i=1..f(n) Recursing put “Hi” a times loop i=1..a generates a·T(n/b) “Hi”s. In/b = In cut in b pieces Eg(In/b)
Recurrence Relations Time of Recursive Program
procedure Eg(In) n = |In| if(n1) then put “Hi” else loop i=1..f(n) put “Hi” loop i=1..a In/b = In cut in b pieces Eg(In/b) For a total of T(1) = 1 T(n) = a·T(n/b) + f(n) “Hi”s.
Evaluating: T(n) = aT(n/b)+f(n)
Level 0 1 2 i
h = log n/log b
Instance size n
n/b n/b2
Work in stack frame
f(n) f(n/b) f(n/b2)
# stack frames 1 a a2 ai n
log a/ log b
Work in Level
1 · f(n) a · f(n/b) a2 · f(n/b2)
n/bi
f(n/bi)
ai · f(n/bi)
log a/
n/bh
T(1)
n
log b ·
T(1)
Dominated by Top Level or Base Cases
Evaluating: T(n) = aT(n/b)+f(n)
Time: T(n) = 2T(n/2) + Q(n) = Q(n log(n))
Merge Sort Sort
Total work at any level = Q(n) # levels = Q(log(n))
2T(n/2) + Q(n)
Evaluating: T(n) = aT(n/b)+f(n)
Level 0 1 2 i
h = log n/log b
Instance size n
n/b n/b2
Work in stack frame
f(n) f(n/b) f(n/b2)
# stack frames 1 a a2 ai
Work in Level
1 · f(n) a · f(n/b) a2 · f(n/b2)
n/bi
f(n/bi)
ai · f(n/bi)
n/bh
T(1)
2i·Q(n/2i) = Q(n) log a/ log a/ log b n n log b · T(1) All levels basically the same.
Total Work T(n) = ∑i=0..h aif(n/bi) = Q(n) · log(n)
Evaluating: T(n) = aT(n/b)+f(n)
Logs
log 27 = log 9
Logs
log? 27 = log? 9 Which base?
Logs
log2 27 log2 c · logc 27 logc 27 = = log2 9 log2 c · logc 9 logc 9 It does not matter. Which is easiest?
Logs
3 log3 27 log3 33 = 2 = 2 log3 9 log3 3 Please no calculators in exams. And I wont ask log 25 log 9
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level?:
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level: f(n) = n3/log n Time for base cases?:
5
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level: f(n) = n3 /log n log ?/ log a/ log b θ(n log ? ) )= Time for base cases: θ(n
5
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level: f(n) = n3 /log n log 4/ log a/ log b θ(n log 2 ) = ? )= Time for base cases: θ(n
5
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level: f(n) = n3/log n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = ? log a/log b = ?
5
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level: f(n) = n3/log n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = 3 > 2 = log a/log b
5
Hence, T(n) = ?
Evaluating: T(n) = 4T(n/2)+ n3/log n
5
Time for top level: f(n) = n3/log n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = 3 > 2 = log a/log b
5
Hence, T(n) = θ(top) = θ(f(n)) = θ(n3/log n).
5
Evaluating: T(n) = 4T(n/2)+ 2n
bigger
Time for top level: f(n) = 2n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n >big Dominated?: c = ? > 2 = log a/log b
Hence, T(n) = θ(top) = θ(f(n)) = θ(2n).
The time is even more dominated by the top level of recursion
Evaluating: T(n) = 4T(n/2)+ n log n
5
Time for top level: f(n) = n log5n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = 1< 2 = log a/log b ?
Hence, T(n) = ?θ(base cases) = θ(n =
log a/
log b
) = θ(n2).
Evaluating: T(n) = 4T(n/2)+ log5n
Time for top level: f(n) = log5n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = 0< 2 = log a/log b ?
Hence, T(n) = ?θ(base cases) = θ(n =
log a/
log b
) = θ(n2).
Evaluating: T(n) = 4T(n/2)+ n2
Time for top level: f(n) = n2 log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = 2= 2 = log a/log b ?
Hence, T(n) = ?θ(f(n) log(n) ) = θ(n2 log(n)). =
Evaluating: T(n) = 4T(n/2)+ n2 log n
5
Time for top level: f(n) = n2 log5n log 4/ log a/ log b θ(n log 2 ) = θ(n2) )= Time for base cases: θ(n Dominated?: c = 2 = 2 = log a/log b 5> d = ? ? -1
Hence, T(n) = ? = θ(f(n) log(n) ) = θ(n2 log6(n)).
Evaluating: T(n) = aT(n/b)+f(n)
Evaluating: T(n) = aT(n-b)+f(n)
Level
Instance size n
n-b n-2b
Work in stack frame
f(n)
f(n-b) f(n-2b)
# stack frames 1 a a2 ai a
n/ b
Work in Level
1 · f(n)
a · f(n-b) a2 · f(n-2b)
0 1 2 i h = n?b h= /
n-ib
f(n-ib)
ai · f(n-ib)
n/
n-hb
T(0)
a b · T(0) Exponential
|base case| = 0 = n-hb Likely dominated by base cases n/ h= b
Evaluating: T(n) = 1T(n-b)+f(n)
Level
Instance size n
n-b n-2b
Work in stack frame
f(n)
f(n-b) f(n-2b)
# stack frames 1 1 1
1
Work in Level
f(n)
f(n-b) f(n-2b)
0 1 2 i h = n?b h= /
n-ib
f(n-ib)
f(n-ib)
n-hb
T(0)
1
f(0)
Total Work T(n) = ∑i=0..h f(b·i) = θ(f(n)) or θ(n·f(n))
Evaluating: T(n) = aT(n/b)+f(n)
End of second part of math review
More Math Graphs Searching Algorithms