VIEWS: 4 PAGES: 49 POSTED ON: 2/11/2013 Public Domain
CSCE 210 Data Structures and Algorithms Prof. Amr Goneid AUC Part 3. Introduction to the Analysis of Algorithms Prof. Amr Goneid, AUC 1 Introduction to the Analysis of Algorithms Algorithms Analysis of Algorithms Time Complexity Bounds and the Big-O Types of Complexities Rules for Big-O Examples of Algorithm Analysis Prof. Amr Goneid, AUC 2 1. Algorithms The word Algorithm comes from the name of Abu Ja’afar Mohamed ibn Musa Al Khowarizmi (c. 825 A.D.) An Algorithm is a procedure to do a certain task An Algorithm is supposed to solve a general, well- specified problem Prof. Amr Goneid, AUC 3 Algorithms Example: Sorting Problem: Input: A sequence of keys {a1 , a2 , …, an} output: A permutation (re-ordering) of the input, {a’1 , a’2 , …, a’n} such that a’1 ≤ a’2 ≤ …≤ a’n An instance of the problem might be sorting an array of names or sorting an array of integers. An algorithm is supposed to solve all instances of the problem Prof. Amr Goneid, AUC 4 Example: Selection Sort Algorithm Solution: “From those elements that are currently unsorted, find the smallest and place it next in the sorted list” Algorithm: for each i = 0 .. n-2 find smallest element in sub-array a[i] to a[n-1] swap that element with that at the start of the sub- array Prof. Amr Goneid, AUC 5 Example: Selection Sort Algorithm void selectsort (itemType a[ ], int n) { int i , j , m; for (i = 0; i < n-1; i++) { m=i; for ( j = i+1; j < n; j++) if (a[j] a[m]) m = j ; swap (a[i] , a[m]); } } Prof. Amr Goneid, AUC 6 Algorithms Example: Euclide’s Algorithm for the GCD ALGORITHM Euclid(m, n) //Computes gcd(m, n) by Euclid’s algorithm //Input: Two nonnegative, not-both-zero integers m and n //Output: Greatest common divisor of m and n while n ≠ 0 do r ←m mod n m←n n←r return m Prof. Amr Goneid, AUC 7 Algorithms Euclide’s Algorithm for the GCD (Recursive Version) function gcd(m, n) if n = 0 return m else return gcd (n, m mod n) "The Euclidean algorithm is the granddaddy of all algorithms, because it is the oldest nontrivial algorithm that has survived to the present day”. Donald Knuth, The Art of Computer Programming, Vol. 2 Prof. Amr Goneid, AUC 8 Algorithms should be: Transparent Correct Complete Writeable Maintainable Easy to use Efficient Prof. Amr Goneid, AUC 9 Algorithms An algorithm should have a clear and Transparent purpose An Algorithm should be Correct, i.e. solves the problem correctly. An Algorithm should be Complete, i.e. solve all instances of the problem An algorithm should be Writeable, i.e. we should be able to express its procedure with available implementation language. Prof. Amr Goneid, AUC 10 Algorithms An algorithm should be Maintainable, i.e. easy to debug and modify. An algorithm is supposed to be Easy to use. An Algorithm is supposed to be Efficient. An efficient algorithm uses minimum of resources (space and time). In particular, it should solve the problem in the minimum amount of time. Prof. Amr Goneid, AUC 11 2. Analysis of Algorithms The main goal is to determine the cost of running an algorithm and how to reduce that cost. Cost is expressed as Complexity Time Complexity Space Complexity Prof. Amr Goneid, AUC 12 Analysis of Algorithms Time Complexity Depends on: - Machine Speed - Size of Data and Number of Operations needed (n) Space Complexity Depends on: - Size of Data - Size of Program Prof. Amr Goneid, AUC 13 3. Time Complexity Expressed as T(n) = number of operations required. (n) is the Problem Size: n could be the number of specific operations, or the size of data (e.g. an array) or both. Prof. Amr Goneid, AUC 14 Number of Operations T(n) Example (1): Factorial Function int factorial (int n) { int i , f ; f = 1; if ( n > 0 ) for (i = 1; i <= n; i++) f * = i ; return f ; } Let T(n) = Number of multiplications. For a given n , then T(n) = n (always) Prof. Amr Goneid, AUC 15 Complexity of the Factorial Algorithm Because T(n) = n always, then T(n) = (n) T(n) (n) n Prof. Amr Goneid, AUC 16 Number of Operations T(n) Example (2): Linear Search int linSearch (const int a[ ], int target, int n) { for (int i = 0; i < n; i++) if (a[i] == target) return i; return -1; } T(n) = number of array element comparisons. Best case: T(n) = 1 Worst case: T(n) = n Prof. Amr Goneid, AUC 17 Complexity of the Linear Search Algorithm T(n) = 1 in the best case. T(n) = n in the worst case We write that as: T(n) = (1) and T(n) = O(n) T(n) O(n) (1) n Prof. Amr Goneid, AUC 18 4. Bounds and the Big-O If an algorithm always costs T(n) = f(n) for the same (n) independent of the data, it is an Exact algorithm. In this case we say T(n) = (f(n)), or Big . The factorial function is an example where T(n) = (n) Prof. Amr Goneid, AUC 19 Bounds If the cost T(n) of an algorithm for a given size (n) changes with the data, it is not an exact algorithm. In this case, we find the Best Case (Lower Bound) T(n) = (f(n)) or Big and the Worst Case (Upper Bound) T(n) = O(f(n)) or Big O The linear search function is an example where T(n) = (1) and T(n) = O(n) Prof. Amr Goneid, AUC 20 Constants do not matter If T(n) = constant * function of (n), i.e. T(n) = c f(n) we still say that T(n) is O(f(n)) or (f(n)) or (f(n)). Prof. Amr Goneid, AUC 21 Constants do not matter T(n) = 4 (best case) then T(n) = (1) T(n) = 6 n2 (worst case) then T(n) = O(n2) T(n) = 3 n (always) then T(n) = (n) is of Number of operations Complexity Prof. Amr Goneid, AUC 22 5. Types of Complexities Constant Complexity T(n) = constant independent of (n) Runs in constant amount of time O(1) Example: cout << a[0][0] Polynomial Complexity T(n)=amnm+…+ a2n2 + a1n1 + a0 If m=1, then O(a1n+a0) O(n) If m > 1, then O(nm) as nm dominates Prof. Amr Goneid, AUC 23 Polynomials Complexities O(n3) O(n2) Log T(n) O(n) n Prof. Amr Goneid, AUC 24 Types of Complexities Logarithmic Complexity Log2n=m is equivalent to n=2m Reduces the problem to half O(log2n) Example: Binary Search T(n) = O(log2n) Much faster than Linear Search which has T(n) = O(n) Prof. Amr Goneid, AUC 25 Linear vs Logarithmic Complexities 16 14 12 10 O(n) T(n) x 8 logx 6 4 O(log2n) 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 n x Prof. Amr Goneid, AUC 26 Types of Complexities Exponential Example: List all the subsets of a set of n elements {a,b,c} {a,b,c}, {a,b},{a,c},{b,c},{a},{b},{c},{} Number of operations T(n) = O(2n) Exponential expansion of the problem O(an) where a is a constant greater than 1 Prof. Amr Goneid, AUC 27 Exponential Vs Polynomial O(2n) Log T(n) O(n3) O(n) n Prof. Amr Goneid, AUC 28 Types of Complexities Factorial time Algorithms Example: Traveling salesperson problem (TSP): Find the best route to take in visiting n cities away from home. What are the number of possible routes? For 3 cities: (A,B,C) Prof. Amr Goneid, AUC 29 Possible routes in a TSP Home Amsterdam NY Montreal NY Montreal Amsterdam Montreal Amsterdam NY Montreal NY Montreal Amsterdam NY Amsterdam –Number of operations = 3!=6, Hence T(n) = n! –Expansion of the problem O(n!) Prof. Amr Goneid, AUC 30 Exponential Vs Factorial O(nn) Log T(n) O(n!) O(2n) n Prof. Amr Goneid, AUC 31 Execution Time Example Example: For the exponential algorithm of listing all subsets of a given set, assume the set size to be of 1024 elements Number of operations is 21024 about 1.8*10308 If we can list a subset every nanosecond the process will take 5.7 * 10291 yr!!! Prof. Amr Goneid, AUC 32 Polynomial & Non-polynomial Times P (Polynomial) Times: O(1), O(log n), O(log n)2, O(n) , O(n logn), O(n2), O(n3), …. NP (Non-Polynomial) Times: O(2n) , O(en) , O(n!) , O(nn) , ….. Prof. Amr Goneid, AUC 33 6. Rules for Big-O Rule Example For constant k, O(k) < O(n) O(7) = O(1) < O(n) For constant k, O(kf) = O(f) O(2n) = O(n) O(|f|+|g|) = O(|f|) + O(|g|) = O(6n2+n)=O(6n2)+O(n) = O(n2) Max (O(|f|) , O(|g|) Nesting of loop O(g) within a O(n4*n2)=O(n6) loop O(f) gives O(f*g) O(nm-1) < O(nm) O(n2) < O(n3) O(log n) O(n) O(log2 n) < O(n) Prof. Amr Goneid, AUC 34 Exercises Which function has smaller complexity ? f = 100 n4 g = n5 f = log(log n3) g = log n f = n2 g = n log n f = 50 n5 + n2 + n g = n5 f = en g = n! Prof. Amr Goneid, AUC 35 7. Examples of Algorithm Analysis for (i = 1; i <= n/2; i++) { O(1); for (j = 1; j <= n*n; j++) { O(1); } } T(n) = (n2 + 1) * n/2 = n3/2 + n/2 Hence T(n) = O(n3) _________________________________________________________ for (i = 1; i <= n/2; i++) { O(1); } for (j = 1; j <= n*n; j++) { O(1); } T(n) = (n/2) + n2 Hence T(n) = O(n2) Prof. Amr Goneid, AUC 36 Infinite Loop? int k , n , d ; cout << “Enter n : “ ; cin >> n ; k = n; for ( ; ; ) { d = k % 2; cout << d; k /= 2; if (k == 0) break; } Each iteration cuts the value of (k) by half and the final iteration reduces k to zero. Hence the number of iterations T(n) = log2 n + 1 Hence T(n) = O(log2 n) Prof. Amr Goneid, AUC 37 A function to reverse an array(1) A function to reverse an array a[ ]. Version(1): using a temporary array b[ ]. int a[N]; void reverse_array (int a[ ], int n) { int b[N]; for (int i = 0; i < n ; i++) b[i] = a[n-i-1]; for (int i = 0; i < n ; i++) a[i] = b[i]; } Consider T(n) to be the number array accesses, then T(n) = 2n + 2n = 4n Hence T(n) = O(n), with extra space b[N] Prof. Amr Goneid, AUC 38 A function to reverse an array(2) A function to reverse an array a[ ]. Version(2): using swapping. int a[N]; void reverse_array (int a[ ], int n) { int temp; for (int i = 0; i < n/2 ; i++) { temp = a[i]; a[i] = a[n-i-1]; a[n-i-1] = temp; } } Consider T(n) to be the number array accesses, then T(n) = 4(n/2) = 2n Hence T(n) = O(n), without extra space Prof. Amr Goneid, AUC 39 Index of the minimum element A function to return the index of the minimum element int index_of_min ( int a[ ] , int s , int e ) { int imin = s; for (int i = s+1; i <= e ; i++) if (a[i] < a[imin]) imin = i ; return imin ; } Consider T(n) to be the number of times we compare array elements. When invoked as index_of_min (a, 0 , n-1 ), then T(n) = e – (s+1) +1 = e - s = n-1 Hence T(n) = O(n) Prof. Amr Goneid, AUC 40 Analysis of Selection Sort void SelSort(int a[ ], int n) { int m, temp; for (int i = 0; i < n-1; i++) Costs n-1-i { m = index_of_min(a, i, n-1); temp = a[i]; a[i] = a[m]; a[m] = temp; } } T(n) = number of array comparisons. For a given iteration (i), Ti(n) = n-1-i and T(n) = T0(n) + T1(n) + …. Tn-2(n) = (n-1) + (n-2) + …+ 1 = n(n-1)/2 = 0.5 n2 – 0.5 n = O(n2) This cost is the same for any data of size (n) (exact algorithm) Prof. Amr Goneid, AUC 41 SelectSort vs QuickSort Selectsort O(n2) T(n) Quicksort O(n log2n) n Prof. Amr Goneid, AUC 42 Analysis of Binary Search int BinSearch( int a[ ], int n, int x) { int L, M, H; bool found = false; L = 0; H = n; while ( (L < H) && ! found) { M = (L + H)/2; // Approximate Middle if (x == a[M]) found = true; // Match else if (x > a[M]) L = M+1; // Discard left half else H = M – 1; // Discard right half } if (found) return M; else return -1; } In the best case, a match occurs in the first iteration, thus T(n) = (1). In the worst case (not found) half of the array is discarded in each iteration. Hence T(n) = log2 n +1 = O(log2 n) Prof. Amr Goneid, AUC 43 Linear vs Binary Search 16 14 12 10 O(n) T(n) x 8 logx 6 4 O(log2n) 2 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 n x Prof. Amr Goneid, AUC 44 Polynomial Evaluation A polynomial of degree n be be evaluated directly A polynomial of degree n can can as: evaluated directly as P(x) = an xn + an-1 xn-1 + …..+ ai xi + ….+ a1 x a a P(x)+= 0 n xn + an-1 xn-1 + …..+ ai xi + ….+ a1 x + a0 xi is computed by a function pow(x,i) using i-1 multiplications.The direct xi is computed by a function pow(x,i) using (i-1) algorithm is (consider x and a[ ] to be of multiplications. The direct algorithm is: (consider x type double): and a[ ] to be of type double): double p = a[0]; for (int = = i <= n; doublei p 1; a[0]; i++) p = a[i] * pow(x,i); for (int=ip +1; i <= n; i++) p = p + a[i] * pow(x,i); Prof. Amr Goneid, AUC 45 Polynomial Evaluation The number of double arithmetic operations inside the loop is 2 + (i-1), Hence, n T (n) (i 1) 0.5n 2 1.5n O(n 2 ) i 1 We never use this method because it is quadratic. Instead, we use Horner’s method. Prof. Amr Goneid, AUC 46 Horner’s Algorithm William George Horner (1819) introduced a factorization in the form: P(x) = (…(((an)x + an-1)x + an-2)x +..+a1)x + a0 The corresponding algorithm is: double p = a[n]; for ( i = n-1; i >= 0; i--) p = p * x + a[i]; Prof. Amr Goneid, AUC 47 Horner’s Algorithm Analysis of this algorithm gives for the number of double arithmetic operations: n 1 T (n) 2 2n O(n) i 0 This is a faster linear algorithm Prof. Amr Goneid, AUC 48 Learn on your own about: Proving Correctness of algorithms Analyzing Recursive Functions Standard Algorithms in C++ Prof. Amr Goneid, AUC 49