comp

Document Sample
comp Powered By Docstoc
					Indexing and Complexity
                Agenda
• Inverted indexes
• Computational complexity
    Some Interesting Questions
• How long will it take to find a document?
  – Is there any work we can do in advance?
     • If so, how long will that take?
• How big a computer will I need?
  – How much disk space? How much RAM?
• What if more documents arrive?
  – How much of the advance work must be repeated?
  – Will searching become slower?
  – How much more disk space will be needed?
            A Cautionary Tale
• Searching is easy - just ask Microsoft!
  – “Find” can search my 1 GB disk in 30 seconds
     • Well, actually it only looks at the file names...
• How long do you think find would take for
  – The 100 GB disk we just got?
  – For the World Wide Web?
• Computers are getting faster, but…
  – How does AltaVista give answers in 5 seconds?
     The “Inverted File” Trick
• Organize the bag of words matrix by terms
  – You know the terms that you are looking for
• Look up terms like you search phone books
  – For each letter, jump directly to the right spot
     • For terms of reasonable length, this is very fast
  – For each term, store the document identifiers
     • For every document that contains that term
• At query time, use the document identifiers
  – Consult a “postings file”
                An Example




                            Doc 3
                            Doc 2
                            Doc 1


                            Doc 4
                            Doc 5
                            Doc 6
                            Doc 7
                            Doc 8
Inverted File     Term                                      Postings
             AI      aid    0   0   0   1   0   0   0   1       4, 8
         A   AL       all   0   1   0   1   0   1   0   0      2, 4, 6
             BA    back     1   0   1   0   0   0   1   0      1, 3, 7
         B   BR   brown     1   0   1   0   1   0   1   0    1, 3, 5, 7
         C        come      0   1   0   1   0   1   0   1    2, 4, 6, 8
         D          dog     0   0   1   0   1   0   0   0       3, 5
         F           fox    0   0   1   0   1   0   1   0      3, 5, 7
         G         good     0   1   0   1   0   1   0   1    2, 4, 6, 8
         J         jump     0   0   1   0   0   0   0   0         3
         L          lazy    1   0   1   0   1   0   1   0    1, 3, 5, 7
         M          men     0   1   0   1   0   0   0   1      2, 4, 8
         N          now     0   1   0   0   0   1   0   1      2, 6, 8
         O         over     1   0   1   0   1   0   1   1   1, 3, 5, 7, 8
         P         party    0   0   0   0   0   1   0   1       6, 8
         Q        quick     1   0   1   0   0   0   0   0       1, 3
             TH    their    1   0   0   0   1   0   1   0      1, 5, 7
         T   TI     time    0   1   0   1   0   1   0   0      2, 4, 6
The Finished Product
  Inverted File     Term Postings
               AI      aid        4, 8
           A   AL       all      2, 4, 6
               BA    back        1, 3, 7
           B   BR   brown      1, 3, 5, 7
           C        come       2, 4, 6, 8
           D          dog         3, 5
           F           fox       3, 5, 7
           G         good      2, 4, 6, 8
           J         jump           3
           L          lazy     1, 3, 5, 7
           M          men        2, 4, 8
           N          now        2, 6, 8
           O         over     1, 3, 5, 7, 8
           P         party        6, 8
           Q        quick         1, 3
               TH    their       1, 5, 7
           T   TI     time       2, 4, 6
  What Goes in a Postings File?
• Boolean retrieval
  – Just the document number
• Ranked Retrieval
  – Document number and term weight (TF*IDF, ...)
• Proximity operators
  – Word offsets for each occurrence of the term
     • Example: Doc 3 (t17, t36), Doc 13 (t3, t45)
  How Big Is the Postings File?
• Very compact for Boolean retrieval
  – About 10% of the size of the documents
     • If an aggressive stopword list is used!
• Not much larger for ranked retrieval
  – Perhaps 20%
• Enormous for proximity operators
  – Sometimes larger than the documents!
• But access is fast - you know where to look
    Building an Inverted Index
• Simplest solution is a single sorted array
  – Fast lookup using binary search
  – But sorting large files on disk is very slow
  – And adding one document means starting over
• Tree structures allow easy insertion
  – But the worst case lookup time is linear
• Balanced trees provide the best of both
  – Fast lookup and easy insertion
  – But they require 45% more disk space
Starting a B+ Tree Inverted File
           Now is the time for all good …

                  aaaaa   now




  all   good                         now    time
      Adding a New Term
      Now is the time for all good men …

               aaaaa    now



       aaaaa   men




all   good     men                now      time
  How Big is the Inverted Index?
• Typically smaller than the postings file
  – Depends on number of terms, not documents
• Eventually almost all terms will be indexed
  – But the postings file will continue to grow
• Postings dominate asymptotic space complexity
  – Linear in the number of documents
     • Assuming that the documents remain about the same size
         Some Facts About Disks
• It takes a long time to get the first byte
   – A Pentium can do 1,000,000 operations in 10 ms
• But you can get 1,000 bytes just about as fast
   – 40 MB/sec transfer rates are typical
• So it pays to put related stuff in each “block”
   – M-ary trees B+ are better than binary B+ trees
• Time complexity is measured in disk blocks read
   – Since computing time is negligible by comparison
             Time Complexity
• Indexing
  – Walk the inverted file, splitting if needed
  – Insert into the postings file in sorted order
  – Hours or days for large collections
• Query processing
  – Walk the inverted file
  – Read the postings file
  – Seconds, even for enormous collections
                 Summary
• Slow indexing yields fast query processing
• We use extra disk space to save query time
  – Index space is in addition to document space
  – Time and space complexity must be balanced
• Disk block reads are the critical resource
  – Fast disks are more useful than fast computers
               A Question
• If insertions are more common than queries
  (for example, filtering news stories as they
  arrive and then never looking at them
  again), what kind of an index should you
  build?

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:3
posted:9/9/2011
language:English
pages:17