Document Sample
algorithms Powered By Docstoc
Modified from slides by Sanjoy Dasgupta
 Why you should know Algorithms
• Courses: CSE 21 (Discrete Math), CSE 100 (Data
  Structures), CSE 101 (Algorithms)
• Motivation? If you want to be a good programmer, why
  should you know this stuff?
• Definition: Algorithms are recipes for faster, smarter
  computation in lieu of brute force (more HW)
• Apps!: Many real world problems and systems (Roomba,
  Google, Facebook, Internet) have a few core algorithms
• Bottom line: Knowing algorithms can help you make
  some spectacular speed wins in your problem area.
       Some big algorithm wins
• Compression: Lempel and Ziv. Gzip for files  faster
  downloads, less disk storage
• Fast Fourier Transforms (FFTs): Cooley-Tukey. Signal
  processing  modern cell phones
• Shortest Path Algorithm: Dijkstra. Computing Internet
  routes  fast response when comm links crash
• Primality Testing: Rabin-Miller. Finding large primes for
  public key encryption  PGP
• String Matching: Boyer-Moore. Fast search for a
  keyword in a file  grep
              Algorithm Areas
Useful recipes in different areas
• Mathematical Computing
   – App: Solid modeling, more interactive
• String matching
   – App: Google search, faster search
• Graph
   – App: Computing Internet Routes, fast failure recovery
• Scientific Computation
   – App: Drug Design, faster design
Area 1: Computing Mathematics
              Counting rabbits
A royal mathematical
 challenge (1202):

Suppose that rabbits take
  exactly one month to
  become fertile, after
  which they produce one
  child per month, forever.
  Starting with one rabbit,
  how many are there after
  n months?
                              Leonardo da Pisa, aka Fibonacci
   The proliferation of rabbits
               Fertile   Not fertile

One month
Two months
Three months
Four months
Five months
           Formula for rabbits
Let Fn = number of rabbits at month n
             F1 = 1
             F2 = 1
             Fn = Fn-1 + Fn-2
These are the Fibonacci numbers:
  1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, …
They grow very fast: F30 > 106 !
In fact, Fn  20.694n, exponential growth.
Useful in CSE: merge sort, compression
  Computing Fibonacci numbers
function F(n)                            F(5)
if n = 1 return 1
if n = 2 return 1
return F(n-1) + F(n-2)            F(4)          F(3)

                           F(3)     F(2)        F(2)   F(1)

                        F(2)   F(1)

A recursive algorithm
  Two questions we always ask
Does it work correctly? (Proofs)

Yes – it directly implements the definition of
 Fibonacci numbers.

How long does it take? (Time analysis)

This is not so obvious…
          Running time analysis
function F(n)
if n = 1 return 1
if n = 2 return 1
return F(n-1) + F(n-2)

Let T(n) = number of steps needed to compute F(n).

       T(n) > T(n-1) + T(n-2)
But recall that Fn = Fn-1 + Fn-2. Therefore T(n) > Fn  20.694n !

Exponential time.
  How bad is exponential time?
Need 20.694n operations to compute Fn.

Eg. Computing F200 needs about 2140

How long does this take on a fast computer?
NEC Earth Simulator
      NEC Earth Simulator

Can perform up to 40 trillion operations per second.
Is exponential time all that bad?
The Earth simulator needs 295 seconds for F200.

Time in seconds       Interpretation
  210                 17 minutes
  220                 12 days
  230                 32 years
  240                 cave paintings
                  Post mortem
What takes so long? Let’s unravel the recursion…


             F(n-1)                   F(n-2)

  F(n-2)           F(n-3)        F(n-3)            F(n-4)

F(n-3) F(n-4) F(n-4) F(n-5) F(n-4) F(n-5) F(n-5) F(n-6)

  The same subproblems get solved over and over again!
               A better algorithm
There are n subproblems F1, F2, …, Fn. Solve them in order
from low to high (unlike recursion, from high to low)
          function F(n)
          Create an array fib[1..n]
          fib[1] = 1
          fib[2] = 1
          for i = 3 to n:
              fib[i] = fib[i-1] + fib[i-2]
          return fib[n]

 The usual two questions:
 1. Does it return the correct answer?
 2. How fast is it?
          Running time analysis
           function F(n)
           Create an array fib[1..n]
           fib[1] = 1
           fib[2] = 1
           for i = 3 to n:
               fib[i] = fib[i-1] + fib[i-2]
           return fib[n]

The number of operations is proportional to n.
  [Previous method: 20.7n]
F200 is now reasonable to compute, as are F2000 and F20000.

Motto: the right algorithm makes all the difference.
                Big-O notation
function F(n)
Create an array fib[1..n]            Running time is
fib[1] = 1                           proportional to n.
fib[2] = 1
for i = 3 to n:                      But what is the constant:
    fib[i] = fib[i-1] + fib[i-2]     is it 2n or 3n or what?
return fib[n]

The constant depends upon:
      The units of time – minutes, seconds, milliseconds,…
      Specifics of the computer architecture.
It is much too hairy to figure out exactly. Moreover it is
nowhere as important as the huge gulf between n and 2n.
So we simply say the running time is O(n).
       A more careful analysis
           function F(n)
           Create an array fib[1..n]
           fib[1] = 1
           fib[2] = 1
           for i = 3 to n:
               fib[i] = fib[i-1] + fib[i-2]
           return fib[n]

Dishonest accounting: the O(n) operations aren’t all
The numbers get very large: Fn  20.7n has 0.7n bits.
Adding numbers of this size is hardly a unit operation.
Adding two n-bit numbers

  [22]           1       0       1       1      0
  [13]                   1       1       0      1
  [35]    1       0      0       0       1      1

Takes O(n) operations… and we can’t hope for
  better. Addition takes linear time.
         Revised time analysis
            function F(n)
            Create an array fib[1..n]
            fib[1] = 1
            fib[2] = 1
            for i = 3 to n:
                fib[i] = fib[i-1] + fib[i-2]
            return fib[n]

Takes O(n) simple operations and O(n) additions… a total
of O(n2), quadratic time.

The inefficient algorithm: not O(20.7n) but O(n20.7n).
     Polynomial vs. exponential
Running times like
      n, n2, n3, are polynomial.
Running times like
      2n, en, 2n are exponential.

To an excellent first approximation:
  polynomial is reasonable
  exponential is not reasonable

This is the most fundamental dichotomy in algorithms.
Intuitively harder than addition… a time analysis lets us quantify this.

[22]                                  1          0           1          1         0
[5]                                                          1          0         1
                                       1          0          1          1         0
                            0          0          0          0          0
                 1          0          1          1          0
[110]            1          1          0          1          1          1         0

To multiply two n-bit numbers: create an array of n intermediate sums,
   and add them up. Each addition is O(n)… a total of O(n2).

It seems that addition is linear, while multiplication is quadratic.
There are other ways to multiply!

            22                     5
            11                     10
            5                      20
            2                      40
            1                      80

Still quadratic, but other methods are close to linear!
Area 2:   String Matching
         Imaginary Scenario
• Context: You agree to help a historian
  build a web site of famous speeches
• Problem: Historian wants a keyword
  facility: e.g., all lines with keyword “truth”
• Measure: Need fast response to search so
  that the site feels “interactive”
      Keyword Search in File
• Naïve: search for keyword at every offset
  in file, one character at a time
             truths            truths
          We hold these truths

          Usual Questions
• How fast does it run? If file length is F
  and keyword is K, can take F * K time.
• e.g. F = 10 Mbyte, K = 10: 10 M 10 sec
             aaaaz        aaaaz
            aaaaaa . . . . aaaaz

  Boyer-Moore String Matching
• Ideas: Compare last character first; on
  failure consult table to find next offset to try
• e.g. F = 10 Mbyte, K = 10: 0.2M 0.2 sec

                            TABLE COMPUTED AT START
             aaaaz aaaaz             ...
                              Failure a for z  Skip 5
Area 3:   Graph Algorithms
A cartographer’s problem
            Color this map (one
             color per country)
             using as few colors as

            Neighboring countries
             must be different
                              6       5   4    3    2

                          8       7       1
                           13                  11

Graph specified by nodes and edges.
             node          =      country
             edge          =      neighbors
The graph coloring problem: color the nodes of the graph
  with as few colors as possible, such that there is no edge
  between nodes of the same color.
            Exam scheduling
Schedule final exams:
- use as few time slots as possible
- can’t schedule two exams in the same slot if
  there’s a student taking both classes.

This is also graph coloring!
  Node = exam                                 3
  Edge = some student is taking
          both endpoint-exams
  Color = time slot               1
      Building short networks

Albuquerque     Amarillo                    Tulsa

                                                    Little Rock

              Abilene                 Dallas

  El Paso                               ?

                        San Antonio

Minimum spanning tree
Finding the minimum spanning tree

Kruskal’s algorithm (1957):

Repeat until nodes are connected:
      Add the shortest edge which doesn’t
      create a cycle

 Greedy strategy: at each stage, make the move which
        gives the greatest immediate benefit.
       Greedy: a popular algorithmic paradigm.
 Network Example: Shortest Path
• Sally to Jorge via ISP ATT
                                              New York
 Seattle                                 2
            3                  Chicago                   12
                          14             15

        Los Angeles                                  Boston

  Dikstra’s Algorithm (Greedy)
• Add shortest node to tree each step
           3                    New York
                      Chicago              12

        Los Angeles                    Boston

 Network Example: Shortest Path

          3                     New York
                      Chicago              12

        Los Angeles                    Boston

Area 4:   Scientific Computation
   A slight variation of MST
     Chicago              Boston

                                   = “Steiner point”


Minimum Steiner tree
              A slight variation

Albuquerque      Amarillo                  Tulsa

                                                   Little Rock

               Abilene                 Dallas

  El Paso

                         San Antonio

Minimum Steiner tree
                Bad news
The existing methods for Steiner tree fall
   into two categories:

1. Correct answer, but exponential time.
2. Efficient, but often the wrong answer.

Let’s look at the second category…
         Biological inspiration
What are the underlying algorithmic paradigms
  behind evolution?

1. Mutation
2. Crossover (during reproduction)
3. Survival of the fittest

Can these ideas be used to design algorithms?
               Hill climbing
Start with a solution
      in this case: minimum spanning tree, but
       more generally scientific optimizations
Produce a mutation
      add/remove/move a Steiner point and
      its connections
Keep it if it works well
      ie. if it reduces the total length

Length keeps improving at each step…
Hill climbing/Simulated Annealing

                                     Local optimum

                  Optimal solution

             Genetic algorithms
“Breed” good solutions…

Maintain a population of candidate solutions.
At each time step, have them reproduce:

  Solution 1
  [set of Steiner points]
                                         “Child” solution
  Solution 2
  [set of Steiner points]

Routinely eliminate “unfit” solutions.
Applications to scientific computing
• Problems in physics, chemistry and biology boil
  down to searching for an “optimal” solution
• For example, recall Bafna’s lecture: finding best
  arrangement of primers for cancer
• Simulated annealing and genetic algorithms are
  very useful for such problems
• No strong bounds on performance: yet they do
  very well in practice.
1. Write in big-O notation.

(a) 24n
(b) 320n2

2. Draw the graph which
     captures the adjacency
     relationships in this map
     (the countries are
     Guatemala, Belize, El
     Salvador, Honduras,
     Nicaragua, Costa Rica).
• Most programming has a few parts of
  performance critical code that can benefit
  from a knowledge of algorithms
• Beyond application, it’s a fundamental way
  of thinking
• This way of thinking is impacting all the
  sciences: physics, chemistry, linguistics
• Even if you love to program, take CSE 21
  and 101 seriously. It will help you!
                          Specs for Earth simulator

  Distributed Memory Parallel Computing System which 640 processor nodes
               interconnected by Single-Stage Crossbar Network

          Processor Node: 8 vector processors with shared memory

                        Peak Performance: 40 Tflops

                          Total Main Memory: 10 TB
                        Purpose: --- Climate change ---

IBM said that Blue Gene/L is special for its size compared to the Yokohama,
Japan-based Simulator, which gauges climate changes. Blue Gene/L is one-
hundredth the physical size (320 vs. 32,500 square feet) and consumes one-
twentieth the power (216 kilowatts vs. 6,000 KW) compared to the Earth
Leonardo da Pisa a.k.a. Fibonacci

             His Liber abaci (1202) begins:

             These are the nine figures of the Indians:

                     9 8 7 6 5 4 3 2 1.

             With these nine figures, and with the
             sign 0 which in Arabic is called
             zephirum, any number can be written, as
             will be demonstrated.

Shared By: