Advanced Counting Techniques
Yen (NTUEE) Discrete Mathematics 2009 1 / 38
Recurrence Relations
Definition
A recurrence relation for {an } is an equation that expresses an in terms
of a0 , . . . , an−1 . A sequence is called a solution of a recurrence relation if
its terms satisfy the recurrence relation.
For instance, the relation fn = fn−1 + fn−2 of Fibonacci numbers is a
recurrence relation.
Yen (NTUEE) Discrete Mathematics 2009 2 / 38
Tower of Hanoi
Example
(The Tower of Hanoi) Consider moving a stack of disks with different sizes
with three pegs. Initially, all disks are sorted and placed on the first peg.
One can only move the topmost disk from a peg to another peg. At any
time, disks on each peg must be sorted as well. How many steps does it
take to move all disks from one peg to another?
Solution: Let Hn be the number of steps to solve the problem of n disks.
Clearly, H1 = 1.
Now consider moving n disks from peg 1 to peg 2. If we can move the
topmost n − 1 disks from peg 1 to peg 3, we can solve the puzzle by
moving the bottom disk from peg 1 to peg 2 and then the n − 1 disks
from peg 3 to peg 2. In other words, Hn = Hn−1 + 1 + Hn−1 . We have
Hn = 2Hn−1 + 1.
Yen (NTUEE) Discrete Mathematics 2009 3 / 38
Tower of Hanoi
Hence
Hn = 2Hn−1 + 1
= 2(2Hn−2 + 1) + 1
= 22 Hn−2 + 21 + 20
= 22 (2Hn−3 + 1) + 21 + 20
= 23 Hn−3 + 22 + 21 + 20
···
= 2n−1 H1 + 2n−2 + · · · + 21 + 20
n−1
= 2i
i=0
2n −
1
= = 2n − 1
2−1
Yen (NTUEE) Discrete Mathematics 2009 4 / 38
Catalan numbers
Example
Find the number of ways to parenthesize the product of n + 1 numbers
x0 , x1 , . . . , xn .
Solution: Let Cn denote the number of ways to parenthesize the product
of n + 1 numbers. For instance, C3 = 5 because
x0 · (x1 · (x2 · x3 )) x0 · ((x1 · x2 ) · x3 ) (x0 · (x1 · x2 )) · x3 ((x0 · x1 ) · x2 ) · x3
are all the possible ways of multiplying x0 · x1 · · · · · x3 .
Clearly C0 = 1. Consider x0 · x1 · · · · · xn . We can compute it by
(x0 · · · · · xk ) · (xk+1 · · · · · xn ) for any k = 0, . . . , n. Therefore
Cn = C0 Cn−1 + C1 Cn−2 + · · · + Cn−1 C0 .
The sequence {Cn }, Cn = n−1 Ck Cn−k−1 , is called Catalan numbers.
k=0
We’ll give a closed form for Catalan numbers later.
Yen (NTUEE) Discrete Mathematics 2009 5 / 38
Homogeneous Linear Recurrence Relations with Constant
Coefficients
Definition
A linear homogeneous recurrence relation of degree k with constant
coefficients is of the form
an = c1 an−1 + c2 an−2 + · · · + ck an−k ,
where c1 , c2 , . . . , ck ∈ R with ck = 0.
For instance, the recurrence for Fibonacci numbers fn = fn−1 + fn−2 is a
linear homogeneous recurrence relation of degree two.
Theorem
Let c1 , c2 ∈ R. Suppose x 2 = c1 x + c2 has two distinct roots r1 and r2 .
Then {an } is a solution of the recurrence relation an = c1 an−1 + c2 an−2 if
n n
and only if an = α1 r1 + α2 r2 for n ∈ N and some constants α1 , α2 .
Yen (NTUEE) Discrete Mathematics 2009 6 / 38
Solving Linear Recurrence Relations
Proof.
n n
(⇐) Suppose an = α1 r1 + α2 r2 . We want to verify an = c1 an−1 + c2 an−2 .
n−1 n−1
c1 an−1 = c1 α1 r1 + c1 α2 r2
n−2 n−2
c2 an−2 = c2 α1 r1 + c2 α2 r2
n−2 n−2
c1 an−1 + c2 an−2 = α1 r1 (c1 r1 + c2 ) + α2 r2 (c1 r2 + c2 )
n−2 2 n−2 2
= α1 r1 r1 + α2 r2 r2
n n
= α1 r1 + α2 r2
= an
(Cont’d on Next Page)
Yen (NTUEE) Discrete Mathematics 2009 7 / 38
Solving Linear Recurrence Relations
Proof.
n
(Cont’d) (⇒) From the first part of proof, we know an = α1 r1 + α2 r2 n
satisfies the recurrence relation an = c1 an−1 + c2 an−2 . It remains to show
n n
an = α1 r1 + α2 r2 satisfies initial conditions for some α1 , α2 . The theorem
then follows from the uniqueness of solution to linear homogeneous
recurrence relation.
n n
To see an = α1 r1 + α2 r2 satisfies the initial conditions for some α1 , α2 .
Consider a1 = α1 r1 + α2 r2 and a0 = α1 + α2 .
This is a linear system of two variables α1 and α2 . The solutions are
a1 − a0 r2
α1 = ,
r1 − r2
and
a0 r1 − a1
α2 = .
r1 − r2
Yen (NTUEE) Discrete Mathematics 2009 8 / 38
Fibonacci numbers
Example
Recall the recurrence relation for Fibonacci numbers fn = fn−1 + fn−2 with
f0 = 0 and f1 = 1. Find an explicit formula for√the Fibonacci numbers.
Solution: The solutions to x 2 = x + 1 are 1±2 5 . Hence
√ n √ n
1+ 5 1− 5
fn = α1 + α2
2 2
√ √
for some α1 and α2 . Solving 0 = α1 + α2 , and 1 = α1 1+2 5
+ α2 1−2 5 , we
1 1
have α1 = √5 and α2 = − √5 . Therefore
√ n √ n
1 1+ 5 1 1− 5
fn = √ −√ .
5 2 5 2
Yen (NTUEE) Discrete Mathematics 2009 9 / 38
Characteristic Equations with Multiple Roots
Theorem
Let c1 , c2 ∈ R with c2 = 0. Suppose x 2 = c1 x + c2 has only one root r .
Then {an } is a solution of the recurrence relation an = c1 an−1 + c2 an−2 if
and only if an = α1 r n + α2 nr n for n ∈ N and some constants α1 , α2 .
Proof.
(⇐) 2r = c1 and c1 r + 2c2 = 0. Let an = α1 r n + α2 nr n . Then
c1 an−1 = c1 α1 r n−1 + c1 α2 (n − 1)r n−1
c2 an−2 = c2 α1 r n−2 + c2 α2 (n − 2)r n−2
∴ c1 an−1 + c2 an−2 = α1 r n−2 (c1 r + c2 ) + α2 r n−2 (c1 (n − 1)r + c2 (n − 2))
= α1 r n−2 r 2 + α2 r n−2 (c1 nr − c1 r + c2 n − 2c2 )
= α1 r n + α2 r n−2 (c1 nr + c2 n)
= α1 r n + α2 r n−2 n(c1 r + c2 )
= α1 r n + α2 nr n−2 r 2
Yen (NTUEE) n
nDiscrete Mathematics 2009 10 / 38
Characteristic Equations with Multiple Roots
Proof.
(Cont’d)
(⇒) It remains to show there are α1 and α2 satisfying the initial
conditions. We have a0 = α1 . Therefore
a1 − a0 r
α2 = .
r
Yen (NTUEE) Discrete Mathematics 2009 11 / 38
An Example
Example
What is the solution of the recurrence relation
an = 6an−1 − 9an−2
with a0 = 1 and a1 = 6?
Solution: Since 3 is the multiple root of x 2 = 6x − 9, we have
an = α1 3n + α2 n3n . Moreover, α1 = a0 = 1 and α2 = (6 − 1 · 3)/3 = 1.
We have an = 3n + n3n .
Yen (NTUEE) Discrete Mathematics 2009 12 / 38
Characteristic Equations with Distinct Roots
Theorem
Let c1 , c2 , . . . , ck ∈ R. Suppose the characteristic equation
r k − c1 r k−1 − · · · − ck = 0
has k distinct roots r1 , . . . , rk . Then {an } is a solution of the recurrence
relation
an = c1 an−1 + c2 an−2 + · · · + ck an−k
if and only if
n n n
an = α1 r1 + α2 r2 + · · · + αk rk
for n ∈ N and constants α1 , α2 , . . . , αk .
Yen (NTUEE) Discrete Mathematics 2009 13 / 38
Characteristic Equations with Multiple Roots
Theorem
Let c1 , c2 , . . . , ck ∈ R. Suppose the characteristic equation
r k − c1 r k−1 − · · · − ck = 0
has t distinct roots r1 , . . . , rt with multiplicities m1 , . . . , mt respectively
such that mi ≥ 1 and m1 + m2 + · · · + mk = k. Then {an } is a solution of
the recurrence relation an = c1 an−1 + c2 an−2 + · · · + ck an−k if and only if
an = (α1,0 + α1,1 n + · · · + α1,m1 −1 nm1 −1 )r1
n
+ (α2,0 + α2,1 n + · · · + α2,m2 −1 nm2 −1 )r2
n
+ ···
+ (αt,0 + αt,1 n + · · · + αt,mt −1 nmt −1 )rtn
for n ∈ N and constants αi,j .
Yen (NTUEE) Discrete Mathematics 2009 14 / 38
Solving Nonhomogeneous Linear Recurrence Relations
Definition
A linear nonhomogeneous recurrence relation with constant
coefficients is of the form
an = c1 an−1 + c2 an−2 + · · · + ck an−k + F (n),
where c1 , c2 , . . . , ck ∈ R and F : N → Z. The recurrence relation
an = c1 an−1 + c2 an−2 + · · · + ck an−k
is called the associated homogeneous recurrence relation.
Yen (NTUEE) Discrete Mathematics 2009 15 / 38
Solving Linear Recurrence Relations
Theorem
(p)
Let {an } be a particular solution of
an = c1 an−1 + c2 an−2 + · · · + ck an−k + F (n),
(p) (h) (h)
Then every solution is of the form {an + an }, where {an } is a solution
of
an = c1 an−1 + c2 an−2 + · · · + ck an−k .
(Cont’d on Next Page)
Yen (NTUEE) Discrete Mathematics 2009 16 / 38
Solving Nonhomogeneous Linear Recurrence Relations
Proof.
(p)
Since {an } is a particular solution, we have
(p) (p) (p) (p)
an = c1 an−1 + c2 an−2 + · · · + ck an−k + F (n).
Suppose {bn } be any solution of the nonhomogeneous recurrence relation
such that
bn = c1 bn−1 + c2 bn−2 + · · · + ck bn−k + F (n).
Then
(p) (p) (p) (p)
bn − an = c1 (bn−1 − an−1 ) + c2 (bn−2 − an−2 ) + · · · + ck (bn−k − an−k ).
(p)
That is, {an − bn } is a solution of the associated homogeneous
(p) (h)
recurrence relation. Thus, bn = an + an .
Yen (NTUEE) Discrete Mathematics 2009 17 / 38
An Example
Example
Find all solutions of an = 3an−1 + 2n with a1 = 3.
(p)
Solution: Let us guess an = cn + d. Then
cn + d = 3(c(n − 1) + d) + 2n. We have cn + d = (3c + 2)n + (3d − 3c).
3
Solve c = 3c + 2 and d = 3d − 3c. We obtain c = −1 and d = − 2 .
(p) 3
Therefore an = −n − 2 is a particular solution.
(h)
By Theorem 9, we have an = α3n as solutions to the associated
homogeneous recurrence relation. Therefore all solutions are of the form
an = −n − 3 + α · 3n where α is a constant.
2
3
Finally, 3 = a1 = −1 − 2 + 3α. α = 11 . Therefore
6
3 11 n
an = −n − + 3 .
2 6
Yen (NTUEE) Discrete Mathematics 2009 18 / 38
An Example
Example
Find solutions of an = 5an−1 − 6an−2 + 7n .
(p)
Solution: Let us guess an = c · 7n . Then
49
c · 7n = 5c · 7n−1 − 6c · 7n−2 + 7n . Hence 49c = 35c − 6c + 49, c = 20 .
(p) 49
an = 20 · 7n is a particular solution.
(h)
By Theorem 9, we have an = α1 2n + α2 3n as solutions of the associated
homogeneous recurrence relation. Therefore
49 n
an = α1 2n + α2 3n + 7
20
are all solutions.
Yen (NTUEE) Discrete Mathematics 2009 19 / 38
Solving Particular Solutions
Theorem
Suppose {an } satisfies an = c1 an−1 + c2 an−2 + · · · + ck an−k + F (n), where
c1 , c2 , . . . , ck ∈ R and
F (n) = (bt nt + bt−1 nt−1 + · · · + b1 n + b0 )s n ,
where s, b0 , b1 , . . . , bt ∈ R.
When s is not a root of the characteristic equation of the associated linear
homogeneous recurrence relation, there is a particular solution of the form
(pt nt + pt−1 nt−1 + · · · + p1 n + p0 )s n .
When s is a root of the characteristic equation and its multiplicity is m,
there is a particular solution of the form
nm (pt nt + pt−1 nt−1 + · · · + p1 n + p0 )s n .
Yen (NTUEE) Discrete Mathematics 2009 20 / 38
Solving Nonhomogeneous Linear Recurrence Relations
Example
Solve an = an−1 + 3n2 − 3n + 1. with a0 = 0.
Solution: Since an = an−1 + 3n2 − 3n + 1, F (n) = (3n2 − 3n + 1)1n .
(h)
By Theorem 9, we have an = α · 1n = α as homogeneous solutions.
(p)
By Theorem 15, we let an = n(p2 n2 + p1 n + p0 ) as a particular solution.
(p)
Substitute an in the recurrence relation, we have
n(p2 n2 + p1 n + p0 ) = (n − 1)(p2 (n − 1)2 + p1 (n − 1) + p0 ) + 3n2 − 3n + 1.
Simplify both sides of the equation and compare their coefficients:
p2 = p2
p1 = −3p2 + p1 + 3
p0 = 3p2 − 2p1 + p0 − 3
0 = −p2 + p1 − p0 + 1
Yen (NTUEE) Discrete Mathematics 2009 21 / 38
Solving Nonhomogeneous Linear Recurrence Relations
(Cont’d)
Solve the linear system and get p0 = 0, p1 = 0, p2 = 1. Hence
(p)
an = n(1 · n2 + 0 · n + 0) = n3 is a particular solution.
Thus, an = n3 + α are all solutions. Since a0 = 0, we have α = 0. And we
conclude an = n3 .
Notice that n3 − (n − 1)3 = 3n2 − 3n + 1. Hence
n 2 3
k=1 3k − 3k + 1 = n . Alternatively, note that
n
an = 3k 2 − 3k + 1,
k=1
you can solve it by computing the summation.
Yen (NTUEE) Discrete Mathematics 2009 22 / 38
Solving Recurrence Relations by Domain Transformation
Example
T (n) = 2T (n/2) + nlog2 n, n > 1
where n is a power of 2.
Solutions: Replace n by 2k , we have
T (2k ) = 2T (2k−1 ) + k2k
Let tk = T (2k ). The above can be rewritten as
tk = 2tk−1 + k2k
Hence,
tk = c1 2k + c2 k2k + c3 k 2 2k
T (n) = c1 n + c2 nlogn + c3 nlog 2 n
Yen (NTUEE) Discrete Mathematics 2009 23 / 38
Solving Recurrence Relations by Range Transformation
Example
2
an = 3an−1 subject to a0 = 1
Solution: Let bn = log an . Then the above can be rewritten as
bn = 2bn−1 + log 3 subject to b0 = 0.
Hence
bm = (2n − 1)log 3
n −1)log 3 n −1
an = 2(2 = 32
Yen (NTUEE) Discrete Mathematics 2009 24 / 38
Divide-and-Conquer Algorithms and Recurrence Relations
Recall the merge sort algorithm. The algorithm divides the sequence in
halves, solves each half recursively, and then merges sorted results to get
the solution. This type of algorithms are called divide-and-conquer
algorithms. In the complexity analysis of divide-and-conquer algorithms,
we obtain the recurrence relation of the form
n
f (n) = af ( ) + g (n),
b
where f (n) is the number of steps needed in solving an instance of size n.
We will prove a theorem to help us solve the recurrence relation.
Yen (NTUEE) Discrete Mathematics 2009 25 / 38
Divide-and-Conquer Algorithms and Recurrence Relations
Theorem
Let f be a non-decreasing function satisfying
n
f (n) = af ( ) + c
b
whenever b|n, where a ≥ 1, b ∈ Z+ , b > 1, and c ∈ R+ . Then
O(nlogb a ) if a > 1
f (n) =
O(lg n) if a = 1
Furthermore, when n = b k and a > 1, k ∈ Z+ ,
f (n) = C1 nlogb a + C2 ,
where C1 = c/(a − 1) + f (1) and C2 = −c/(a − 1).
Yen (NTUEE) Discrete Mathematics 2009 26 / 38
Divide-and-Conquer Algorithms and Recurrence Relations
Proof: We first consider n = b k . Then
bk
f (n) = af ( )+c
b
= af (b k−1 ) + c
= a(af (b k−2 ) + c) + c
= a2 f (b k−2 ) + ac + c
= ···
= ak f (1) + ak−1 c + ak−2 c + · · · + ac + c
k−1
k
= a f (1) + c ai
i=0
When a = 1, we have f (n) = f (1) + ck = f (1) + c logb n. Furthermore, if
b k 1 and n = b k , then nlogb a = b k logb a = ak . We have
ak − 1
f (n) = ak f (1) + c
a−1
c c
= ak f (1) + −
a−1 a−1
= nlogb a C1 + C2
For b k 1, and c, d ∈ R with
c > 0, d ≥ 0. Then
O(nd ) if a b d
Therefore, the complexity of merge sort algorithm is O(n lg n).
Yen (NTUEE) Discrete Mathematics 2009 30 / 38
Inclusion-Exclusion
Theorem
Let A0 , A1 , . . . , An be finite sets. Then
|A0 ∪ A1 ∪ · · · ∪ An | = |Ai | − |Ai ∩ Aj |+
0≤i≤n 0≤i
|Ai ∩ Aj ∩ Ak | + · · · + (−1)n+1 |A0 ∩ A1 ∩ · · · ∩ An |
0≤i
Yen (NTUEE) Discrete Mathematics 2009 31 / 38
Inclusion-Exclusion
Proof.
We will count the number of times for an element a such that a belongs to
r
Ai1 , Ai2 , . . . , Air . The element is counted C1 times by i |Ai |. It is
r
counted C2 times by i,j |Ai ∩ Aj |, and so on. Hence a is counted
r r r
C1 − C2 + C3 − · · · + (−1)r +1 Crr
times by the right hand side. Now
C0 − C1 + C2 − C3 + · · · + (−1)r Crr = 0.(why?)
r r r r
We have
r r r r
C0 = 1 = C1 − C2 + C3 − · · · + (−1)r +1 Crr .
Since each element belongs to r of A0 , A1 , . . . , An , we compute each
individually and obtain the result.
Yen (NTUEE) Discrete Mathematics 2009 32 / 38
Applications of Inclusion-Exclusion
Example
Find the number of primes no more than 100.
Solution: Observe that for n ≤ 100, n = 2, 3, 5, 7 and n is composite if
and only if 2|n, 3|n, 5|n or 7|n. (why?) Let Mi = {n : i|n}. Then the
number of primes is equal to 4 + (99 − |M2 ∪ M3 ∪ M5 ∪ M7 |).
100
|M2 | = = 50
2
100
|M3 | = = 33
3
100
|M5 | = = 20
5
100
|M7 | = = 14
7
100
|M2 ∩ M3 | = = 16
6
Yen (NTUEE) Discrete Mathematics 2009 33 / 38
Applications of Inclusion-Exclusion
100
|M2 ∩ M5 | = = 10
10
100
|M2 ∩ M7 | = =7
14
100
|M3 ∩ M5 | = =6
15
100
|M3 ∩ M7 | = =4
21
100
|M5 ∩ M7 | = =2
35
100
|M2 ∩ M3 ∩ M5 | = =3
30
100
|M2 ∩ M3 ∩ M7 | = =2
42
100
|M2 ∩ M5 ∩ M7 | = =1
70
Yen (NTUEE) Discrete Mathematics 2009 34 / 38
Applications of Inclusion-Exclusion
100
|M3 ∩ M5 ∩ M7 | = =0
105
100
|M2 ∩ M3 ∩ M5 ∩ M7 | = =0
210
|M2 ∪ M3 ∪ M5 ∪ M7 | = (50 + 33 + 20 + 14) − (16 + 10 + 7 + 6 + 4 + 2)
+(3 + 2 + 1 + 0) − 0 = 117 − 45 + 6 = 78.
And the number of primes no more than 100 is (4 + (99 − 78)) = 25.
The method in Example 23 can be used to compute all primes up to n. We
define a Boolean array of size n. Starting from 2, we mark out all multiples
of 2. Then the next unmarked number 3 is a prime. We then mark out all
multiples of 3. The next unmarked number 5 is a prime, and so on. This
algorithm is called the sieve of Eratosthenes. Can you implement it?
Yen (NTUEE) Discrete Mathematics 2009 35 / 38
Applications of Inclusion-Exclusion
Theorem
Let m, n ∈ Z+ with m ≥ n. Then there are
n n n
nm − C1 (n − 1)m + C2 (n − 2)m − · · · + (−1)n−1 Cn−1 · 1m
onto functions from {1, 2, . . . , m} to {1, 2, . . . , n}.
Yen (NTUEE) Discrete Mathematics 2009 36 / 38
Applications of Inclusion-Exclusion
Proof.
Let M = {1, 2, . . . , m}, N = {1, 2, . . . , n} and Ni = N − {i}. Define
Fi = {f ∈ M → Ni }, that is, Fi consists of all functions that do not map
to i. The onto functions belong to the set (M → N) − (∪n Fi ). We
i=1
compute | ∪n Fi | by Theorem 22. Observe that
i=1
|Fi1 ∩ Fi2 ∩ · · · ∩ Fi | = (n − )m . Hence
|Fi1 ∩ Fi2 ∩ · · · ∩ Fi | = C n (n − )m .
1≤i1
And
n n n
| ∪n Fi | = C1 (n − 1)m − C2 (n − 2)m + · · · + (−1)n Cn−1 · 1m .
i=1
So the number of onto functions from M to N is
n n n
nm − C1 (n − 1)m + C2 (n − 2)m + · · · + (−1)n−1 Cn−1 · 1m .
Yen (NTUEE) Discrete Mathematics 2009 37 / 38
Applications of Inclusion-Exclusion
Theorem
How many ways are there to permute n objects such that none of them
stays in the original position?
Proof.
Let Pi be the set of permutations such that the object i stays. Then the
number we’re seeking is n! − | ∪n Pi |. Observe that
i=1
|Pi1 ∩ Pi2 ∩ · · · ∩ Pi | = (n − )!
n!
|Pi1 ∩ Pi2 ∩ · · · ∩ Pi | = C n (n − )! =
!
1≤i1
n! n! n!
| ∪n Pi | =
i=1 − + · · · + (−1)n−1
1! 2! n!
1 1 n−1 1
= n! − + · · · + (−1)
1! 2! n!
Yen (NTUEE) Discrete Mathematics 2009 38 / 38