# CSCI 210 Data Structures & Algorithms by J74oS5w

VIEWS: 4 PAGES: 49

• pg 1
```									           CSCE 210
Data Structures and Algorithms

Prof. Amr Goneid
AUC
Part 3. Introduction to the
Analysis of Algorithms

Prof. Amr Goneid, AUC   1
Introduction to the Analysis of
Algorithms
 Algorithms
 Analysis of Algorithms
 Time Complexity
 Bounds and the Big-O
 Types of Complexities
 Rules for Big-O
 Examples of Algorithm Analysis

Prof. Amr Goneid, AUC   2
1. Algorithms

   The word Algorithm comes from the name of Abu
Ja’afar Mohamed ibn Musa Al Khowarizmi (c.
825 A.D.)
   An Algorithm is a procedure to do a certain task
   An Algorithm is supposed to solve a general, well-
specified problem
Prof. Amr Goneid, AUC             3
Algorithms
   Example: Sorting Problem:
Input: A sequence of keys {a1 , a2 , …, an}
output: A permutation (re-ordering) of the input,
{a’1 , a’2 , …, a’n} such that a’1 ≤ a’2 ≤ …≤ a’n
   An instance of the problem might be sorting an
array of names or sorting an array of integers.
   An algorithm is supposed to solve all instances of
the problem

Prof. Amr Goneid, AUC             4
Example: Selection Sort Algorithm

 Solution:
“From those elements that are currently unsorted,
find the smallest and place it next in the sorted list”
 Algorithm:

for each i = 0 .. n-2
   find smallest element in sub-array a[i] to a[n-1]
   swap that element with that at the start of the sub-
array

Prof. Amr Goneid, AUC                  5
Example: Selection Sort
Algorithm
void selectsort (itemType a[ ], int n)
{
int i , j , m;
for (i = 0; i < n-1; i++) {
m=i;
for ( j = i+1; j < n; j++)
if (a[j]  a[m]) m = j ;
swap (a[i] , a[m]);
}
}

Prof. Amr Goneid, AUC   6
Algorithms
   Example: Euclide’s Algorithm for the GCD

ALGORITHM Euclid(m, n)
//Computes gcd(m, n) by Euclid’s algorithm
//Input: Two nonnegative, not-both-zero integers m and n
//Output: Greatest common divisor of m and n
while n ≠ 0 do
r ←m mod n
m←n
n←r
return m

Prof. Amr Goneid, AUC          7
Algorithms
   Euclide’s Algorithm for the GCD
(Recursive Version)
function gcd(m, n)
if n = 0   return m
else    return gcd (n, m mod n)

"The Euclidean algorithm is the granddaddy of all
algorithms, because it is the oldest nontrivial algorithm
that has survived to the present day”.
Donald Knuth, The Art of Computer Programming, Vol. 2

Prof. Amr Goneid, AUC             8
Algorithms should be:
   Transparent
   Correct
   Complete
   Writeable
   Maintainable
   Easy to use
   Efficient

Prof. Amr Goneid, AUC   9
Algorithms
 An algorithm should have a clear and
Transparent purpose
 An Algorithm should be Correct, i.e. solves
the problem correctly.
 An Algorithm should be Complete, i.e.
solve all instances of the problem
 An algorithm should be Writeable, i.e. we
should be able to express its procedure
with available implementation language.

Prof. Amr Goneid, AUC       10
Algorithms
 An  algorithm should be Maintainable, i.e.
easy to debug and modify.
 An algorithm is supposed to be Easy to
use.
 An Algorithm is supposed to be Efficient.
 An efficient algorithm uses minimum of
resources (space and time).
 In particular, it should solve the problem in
the minimum amount of time.

Prof. Amr Goneid, AUC         11
2. Analysis of Algorithms
 The main goal is to determine the cost of
running an algorithm and how to reduce
that cost. Cost is expressed as
Complexity

 Time Complexity

 Space Complexity

Prof. Amr Goneid, AUC     12
Analysis of Algorithms
 Time Complexity
Depends on:
- Machine Speed
- Size of Data and Number of Operations
needed (n)

 Space Complexity
Depends on:
- Size of Data
- Size of Program
Prof. Amr Goneid, AUC    13
3. Time Complexity
 Expressed as T(n) = number of
operations required.

 (n) is the Problem Size:
n could be the number of specific
operations, or the size of data (e.g. an
array) or both.

Prof. Amr Goneid, AUC        14
Number of Operations T(n)
Example (1): Factorial Function

int factorial (int n)
{
int i , f ;
f = 1;
if ( n > 0 )
for (i = 1; i <= n; i++) f * = i ;
return f ;
}

Let T(n) = Number of multiplications.
For a given n , then T(n) = n (always)
Prof. Amr Goneid, AUC                 15
Complexity of the Factorial
Algorithm
Because T(n) = n always, then T(n) = (n)
T(n)

(n)

n

Prof. Amr Goneid, AUC       16
Number of Operations T(n)
Example (2): Linear Search

int linSearch (const int a[ ], int target, int n)
{
for (int i = 0; i < n; i++)
if (a[i] == target) return i;
return -1;
}
T(n) = number of array element comparisons.
Best case:         T(n) = 1
Worst case:        T(n) = n

Prof. Amr Goneid, AUC       17
Complexity of the Linear Search
Algorithm
T(n) = 1 in the best case. T(n) = n in the worst case
We write that as: T(n) = (1) and T(n) = O(n)

T(n)
O(n)

(1)
n

Prof. Amr Goneid, AUC               18
4. Bounds and the Big-O
 If an algorithm always costs T(n) = f(n)
for the same (n) independent of the
data, it is an Exact algorithm. In this
case we say T(n) = (f(n)), or Big .
 The factorial function is an example
where T(n) = (n)

Prof. Amr Goneid, AUC        19
Bounds
 If the cost T(n) of an algorithm for a given size
(n) changes with the data, it is not an exact
algorithm. In this case, we find the Best Case
(Lower Bound) T(n) = (f(n)) or Big  and
the Worst Case (Upper Bound) T(n) = O(f(n))
or Big O
 The linear search function is an example
where T(n) = (1) and T(n) = O(n)

Prof. Amr Goneid, AUC         20
Constants do not matter
If T(n) = constant * function of (n), i.e.
T(n) = c f(n)
we still say that T(n) is O(f(n)) or (f(n)) or
(f(n)).

Prof. Amr Goneid, AUC          21
Constants do not matter
T(n) = 4   (best case)               then T(n) = (1)
T(n) = 6 n2 (worst case)             then T(n) = O(n2)
T(n) = 3 n (always)                  then T(n) = (n)

is of

Number of operations
Complexity

Prof. Amr Goneid, AUC                   22
5. Types of Complexities
 Constant Complexity
 T(n) = constant independent of (n)
 Runs in constant amount of time  O(1)
 Example: cout << a[0][0]
 Polynomial Complexity
 T(n)=amnm+…+ a2n2 + a1n1 + a0
 If m=1, then O(a1n+a0)  O(n)
 If m > 1, then  O(nm) as nm dominates

Prof. Amr Goneid, AUC       23
Polynomials Complexities

O(n3)

O(n2)
Log T(n)

O(n)

n
Prof. Amr Goneid, AUC      24
Types of Complexities
 Logarithmic Complexity
 Log2n=m   is equivalent to n=2m
 Reduces the problem to half  O(log2n)
 Example: Binary Search
T(n) = O(log2n)
Much faster than Linear Search which has
T(n) = O(n)

Prof. Amr Goneid, AUC          25
Linear vs Logarithmic Complexities
16

14

12

10                                         O(n)
T(n)
x
8
logx

6

4                                             O(log2n)
2

0
1   2   3   4   5    6    7       8   9     10   11   12   13   14

n
x

Prof. Amr Goneid, AUC                                26
Types of Complexities
 Exponential
 Example:    List all the subsets of a set of n
elements {a,b,c}
{a,b,c}, {a,b},{a,c},{b,c},{a},{b},{c},{}
 Number of operations T(n) = O(2n)
 Exponential expansion of the problem 
O(an) where a is a constant greater than 1

Prof. Amr Goneid, AUC              27
Exponential Vs Polynomial

O(2n)
Log T(n)

O(n3)

O(n)

n
Prof. Amr Goneid, AUC           28
Types of Complexities
 Factorial time Algorithms
 Example:
 Traveling salesperson problem (TSP):
Find the best route to take in visiting n
cities away from home. What are the
number of possible routes? For 3 cities:
(A,B,C)

Prof. Amr Goneid, AUC         29
Possible routes in a TSP
Home

Amsterdam                    NY                        Montreal

NY         Montreal   Amsterdam          Montreal   Amsterdam          NY

Montreal           NY    Montreal        Amsterdam       NY        Amsterdam

–Number of operations = 3!=6, Hence T(n) = n!
–Expansion of the problem  O(n!)
Prof. Amr Goneid, AUC                                  30
Exponential Vs Factorial

O(nn)
Log T(n)

O(n!)

O(2n)

n
Prof. Amr Goneid, AUC              31
Execution Time Example

 Example:
 For the exponential algorithm of listing all
subsets of a given set, assume the set size
to be of 1024 elements
 Number of operations is 21024 about
1.8*10308
 If we can list a subset every nanosecond
the process will take 5.7 * 10291 yr!!!

Prof. Amr Goneid, AUC         32
Polynomial & Non-polynomial
Times
 P (Polynomial) Times:
O(1), O(log n), O(log n)2, O(n) , O(n logn),
O(n2), O(n3), ….

 NP (Non-Polynomial) Times:
O(2n) , O(en) , O(n!) , O(nn) , …..

Prof. Amr Goneid, AUC     33
6. Rules for Big-O
Rule                                  Example
For constant k, O(k) < O(n)           O(7) = O(1) < O(n)

For constant k, O(kf) = O(f)          O(2n) = O(n)

O(|f|+|g|) = O(|f|) + O(|g|) =        O(6n2+n)=O(6n2)+O(n) = O(n2)
Max (O(|f|) , O(|g|)
Nesting of loop O(g) within a         O(n4*n2)=O(n6)
loop O(f) gives O(f*g)
O(nm-1) < O(nm)                       O(n2) < O(n3)

O(log n) O(n)                        O(log2 n) < O(n)
Prof. Amr Goneid, AUC                     34
Exercises
Which function has smaller complexity ?
 f = 100 n4            g = n5
 f = log(log n3)       g = log n
 f = n2                g = n log n
 f = 50 n5 + n2 + n    g = n5
 f = en                g = n!

Prof. Amr Goneid, AUC      35
7. Examples of Algorithm Analysis
for (i = 1; i <= n/2; i++)
{
O(1);
for (j = 1; j <= n*n; j++)
{ O(1); }
}

T(n) = (n2 + 1) * n/2 = n3/2 + n/2 Hence T(n) = O(n3)
_________________________________________________________
for (i = 1; i <= n/2; i++)
{ O(1); }
for (j = 1; j <= n*n; j++)
{ O(1); }

T(n) = (n/2) + n2     Hence T(n) = O(n2)

Prof. Amr Goneid, AUC       36
Infinite Loop?
int k , n , d ;
cout << “Enter n : “ ;        cin >> n ;
k = n;
for ( ; ; )
{
d = k % 2;       cout << d;     k /= 2;
if (k == 0) break;
}

Each iteration cuts the value of (k) by half and the final
iteration reduces k to zero. Hence the number of iterations
T(n) =  log2 n  + 1
Hence T(n) = O(log2 n)

Prof. Amr Goneid, AUC                  37
A function to reverse an array(1)

 A function to reverse an array a[ ].
Version(1): using a temporary array                   b[ ].

int a[N];
void reverse_array (int a[ ], int n)
{
int b[N];
for (int i = 0; i < n ; i++) b[i] = a[n-i-1];
for (int i = 0; i < n ; i++) a[i] = b[i];
}

Consider T(n) to be the number array accesses, then
T(n) = 2n + 2n = 4n
Hence       T(n) = O(n), with extra space b[N]

Prof. Amr Goneid, AUC             38
A function to reverse an array(2)
 A function to reverse an array a[ ].
Version(2): using swapping.

int a[N];
void reverse_array (int a[ ], int n)
{
int temp;
for (int i = 0; i < n/2 ; i++)
{ temp = a[i]; a[i] = a[n-i-1]; a[n-i-1] = temp; }
}

Consider T(n) to be the number array accesses, then
T(n) = 4(n/2) = 2n
Hence       T(n) = O(n), without extra space

Prof. Amr Goneid, AUC             39
Index of the minimum element
   A function to return the index of the minimum element

int index_of_min ( int a[ ] , int s , int e )
{
int imin = s;
for (int i = s+1; i <= e ; i++)
if (a[i] < a[imin]) imin = i ;
return imin ;
}

Consider T(n) to be the number of times we compare array
elements. When invoked as index_of_min (a, 0 , n-1 ), then
T(n) = e – (s+1) +1 = e - s = n-1
Hence        T(n) = O(n)

Prof. Amr Goneid, AUC             40
Analysis of Selection Sort
void SelSort(int a[ ], int n)
{
int m, temp;
for (int i = 0; i < n-1; i++)                         Costs n-1-i
{
m = index_of_min(a, i, n-1);
temp = a[i]; a[i] = a[m]; a[m] = temp;
}
}

T(n) = number of array comparisons. For a given iteration (i),
Ti(n) = n-1-i and T(n) = T0(n) + T1(n) + …. Tn-2(n)
= (n-1) + (n-2) + …+ 1 = n(n-1)/2 = 0.5 n2 – 0.5 n = O(n2)
This cost is the same for any data of size (n) (exact algorithm)
Prof. Amr Goneid, AUC                 41
SelectSort vs QuickSort

Selectsort
O(n2)

T(n)

Quicksort
O(n log2n)

n
Prof. Amr Goneid, AUC                 42
Analysis of Binary Search
int BinSearch( int a[ ], int n, int x)
{ int L, M, H; bool found = false;
L = 0; H = n;
while ( (L < H) && ! found)
{
M = (L + H)/2;                         // Approximate Middle
if (x == a[M]) found = true;           // Match
else if (x > a[M]) L = M+1;            // Discard left half
else H = M – 1;                        // Discard right half
}
if (found) return M; else return -1;
}

In the best case, a match occurs in the first iteration, thus T(n) = (1). In the
Hence T(n) = log2 n +1 = O(log2 n)

Prof. Amr Goneid, AUC                                43
Linear vs Binary Search
16

14

12

10
O(n)
T(n)
x
8
logx

6

4
O(log2n)
2

0
1   2   3   4   5    6    7       8   9     10   11   12   13   14
n
x

Prof. Amr Goneid, AUC                                44
Polynomial Evaluation
 A polynomial of degree n be be evaluated directly
 A polynomial of degree n can
can
as:
evaluated directly as
P(x) = an xn + an-1 xn-1 + …..+ ai xi + ….+
a1 x a a
P(x)+= 0 n xn + an-1 xn-1 + …..+ ai xi + ….+ a1 x + a0
xi is computed by a function pow(x,i)
using i-1 multiplications.The direct
xi is computed by a function pow(x,i) using (i-1)
algorithm is (consider x and a[ ] to be of
multiplications. The direct algorithm is: (consider x
type double):
and a[ ] to be of type double):
double p = a[0];
for (int = = i <= n;
doublei p 1; a[0]; i++)
p = a[i] * pow(x,i);
for (int=ip +1; i <= n; i++)
p = p + a[i] * pow(x,i);

Prof. Amr Goneid, AUC              45
Polynomial Evaluation
The number of double arithmetic
operations inside the loop is 2 + (i-1),
Hence,
n
T (n)   (i  1)  0.5n 2  1.5n  O(n 2 )
i 1

We never use this method because it
Horner’s method.
Prof. Amr Goneid, AUC       46
Horner’s Algorithm
William George Horner (1819) introduced a
factorization in the form:
P(x) = (…(((an)x + an-1)x + an-2)x +..+a1)x + a0

The corresponding algorithm is:

double p = a[n];
for ( i = n-1; i >= 0; i--)
p = p * x + a[i];

Prof. Amr Goneid, AUC      47
Horner’s Algorithm
Analysis of this algorithm gives for the
number of double arithmetic operations:
n 1
T (n)   2 2n  O(n)
i 0

This is a faster linear algorithm

Prof. Amr Goneid, AUC   48