Docstoc

Preliminary

Document Sample
Preliminary Powered By Docstoc
					                     Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Lecture 9: Rank Aggregation in
MetaSearch

   MetaSearch Engine
   Social Choice Rules
   Rank Aggregation
                         Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Requirement: Rank aggregation
protocols

   Majority rule
   Borda’s rule
   Optimal Spearman’s footrule distance solution
   Kemeny optimal aggregation.
       Kendall tau distance
                     Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Choices of Search Engines
   Many search engines exist to compete for
    users
       The results are not necessarily the same
       Different users prefer different search engines
       Search results may, in the future, be biased
        towards paid advertisements.
                   Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




     GOOGLE: City University
1.   City University London - Top ranking UK university
     for employment www.city.ac.uk/
2.   City University Home www.cityu.edu
3.   City University of Hong Kong * www.cityu.edu.hk/
4.   Welcome to Dublin City University - DCU www.dcu.ie/
5.   Oklahoma City University www.okcu.edu/ -
6.   The University of Iowa www.uiowa.edu/
7.   UMKC The University of Missouri-Kansas City (UMKC)
     www.umkc.edu
8.   The City College of New York www.ccny.cuny.edu/
9.   New Jersey City University [NJCU] www.njcu.edu/
                    Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




      YAHOO
1.    City University London - Top ranking UK university
      for employment www.city.ac.uk
2.    City University City University - www.cityu.edu
3.    Dublin City University www.dcu.ie
4.    City University of Hong Kong www.cityu.edu.hk
5.    UMKC www.umkc.edu
6.    New Jersey City University www.njcu.edu
7.    Oklahoma City University www.okcu.edu
8.    Department of Music www.city.ac.uk/music
9.    Oakland City University www.oak.edu
10.   City University of New York www.cuny.edu
                                       Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




        AllTheWeb
1.   Apartment Listings in University City (sponsored)
     St. Louis, Missouri area apartment listings at Apartments.com. Free nationwide apartment search with
     visual rental listings online.
     http://www.apartments.com
2.   City University - Washington (sponsored)
     Contact information and resources such as yellow page information, phone number, address, maps and
     directions as provided by QwestDex.
     http://service.bfast.com
3.   City University Apartments - Rent.com (sponsored)
     Rent.com has millions of free apartment listings nationwide. Get $100 when you sign a lease near your
     school - it's easy.
     http://www.rent.com
4.   City University London - the University for business and the professions
     ... Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help ...
     Description: Official site with information about courses, research, schools, and departments. Includes
     details of news and events.
     more hits from: http://www.city.ac.uk/ - 13 KB
5.   Flash Upgrade
     Description: A University which believes in forward-thinking business and leadership skills, exposing
     students to the latest technology in all courses of study.
     http://www.cityu.edu/ - 27 KB
6.   Welcome to Dublin City University
     ... KnowledgeWorks, wins DCU Mallin-invent award Full text you can go anywhere in the world from here...
     Dublin City University, Dublin 9, Ireland. Tel. +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated:
     02/09/03 legal search ...
     Description: Information on facilities, services, degree courses, research, the campus, student life, the
     library and DCU news.
                                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




        MSN Search
1.   City University of New York Details the CUNY system's campuses and admissions
     requirements. Eye photos of notable graduates such as actor Judd Hirsch.
     www.cuny.edu
2.   City University Offers prospectus, admissions, academic calendar, courses and degrees, and
     news.
     www.cityuniversity.net
3.   City University Private, nonprofit institution located in Belleview, Washington, offers higher
     education opportunities to working professionals and lifelong learners.
     www.cityu.edu
4.   Valley City University Admissions Site covers financial aid and tuition and offers a virtual
     campus tour. Site also provides access to an online application.
     www.vcsu.nodak.edu/admissions
5.   Oklahoma City University The Oklahoma City University site contains information about
     admissions, academic departments and student life. The site also provides library resources,
     faculty information and news.
     frodo.okcu.edu
6.   Dublin City University, Dublin - TripAdvisor Browse reviews of the Dublin City University in
     Dublin, County Dublin, Ireland. Check consumer reviews, articles and opinions about the
     Dublin City University.
     www.tripadvisor.com/Hotel_Review-g186605-d209093-Reviews-Dublin_City_U...
                                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




       AOL Search
1.   City University London - the University for business and the ... - ... The University for
     business and the professions. Contact Us | About City University | Maps & Directions, AZ
     Index | Site Map | Help. Prospective Students. ...
     http://www.city.ac.uk/
2.   City University Home - A University which believes in forward-thinking business and
     leadership skills, exposing students to the latest technology in all courses of study.
     http://www.cityu.edu/
3.   Welcome to Dublin City University - ... here... Dublin City University, Dublin 9, Ireland. Tel.
     +353 (0) 1 700 5000, Fax. +353 (0) 1 836 0830. Page updated: 02/09/03 legal. ...
     http://www.dcu.ie/
4.   http://www.cuny.edu/ - The University's Main Website.
     http://www.cuny.edu/
5.   City University of Hong Kong - AD Working Group lends staff, students an ear. Several
     issues still evoked strong emotions as staff members and students voiced their ...
     http://www.cityu.edu.hk/
6.   Welcome to Oklahoma City University - ... At Oklahoma City University, our students come
     first. Our ... Oklahoma City University offers a quality, values-centered education. United ...
     http://www.okcu.edu/
                             Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




       Ask Jeeves
1.   City University of Hong Kong Formerly the City Polytechnic of Hong Kong.
     Includes information on university, links to learning resources, research and
     departmental information.....From:www.cityu.edu.hk/
2.   City University Text Version. The University for business and the professions
     ... Contact Us | About City University | Maps & Directions A-Z Index | Site
     Map | Help...From:www.city.ac.uk/
3.   New Jersey City University Official site includes graduate and undergraduate
     information, news, events, online learning, calendars, and
     athletics.From:www.njcu.edu/
4.   Elizabeth City State University Admission Application Introduction 2004-
     2005 APPLICATION FOR ADMISSION. Welcome to the Elizabeth City State
     University Online Application. New
     Freshmen...From:www.ncmentor.org/applications/unc/apply/elizabeth_city
     _state_univ...
5.   City University A University which believes in forward-thinking business and
     leadership skills, exposing students to the latest technology in all courses of
     study.From:www.cityu.edu/
6.   The City College of New York City College of New York (CUNY) A senior
     college of The City University of New York. Located in Manhattan. 138th
     Street at Convent Avenue. A Harlem masterpiece in the neo-
     Gothic...From:www.ccny.cuny.edu/
                                    Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




          HotBot
1. City University of New York
        Details the CUNY system's campuses and admissions requirements. Eye photos of notable
        graduates such as actor Judd Hirsch.
        www.cuny.edu/ - August 10, 2003 - 25 KB
2. City University
        Private, nonprofit institution located in Belleview, Washington, offers higher education
        opportunities to working professionals and lifelong learners.
        www.cityu.edu/ - August 23, 2003 - 27 KB
3. New Jersey City University
        Explore the course requirements for this liberal arts institution. Link to admissions and
        financial aid information. ... New Jersey City University. 2039 Kennedy Boulevard Jersey
        City, New Jersey 07305-1597 ...
        www.njcu.edu/ - November 1, 2003 - 20 KB
4. City University London - the University for business and the...
        ... Contact Us | About City University | Maps & Directions. A-Z Index | Site Map | Help ...
        www.city.ac.uk/ - September 28, 2003 - 14 KB
5. Oklahoma City University
        Take a tour of the campus, and explore a roster of academic programs. ... At Oklahoma
        City University, our students come first. ...
        www.okcu.edu/ - October 26, 2003 - 20 KB
6. University of Missouri, Kansas City
        University based in Kansas City, Missouri, presents and overview of its academic
        programs, and offers campus news. ... UNIVERSITY OF MISSOURI-KANSAS CITY. Bulletin.
        Winter 2004 Fee Update, more... ...
        www.umkc.edu/ - October 23, 2003 - 16 KB
                                   Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




         Lycos
1. City University London - the University for business and the...
        … Contact Us | About City University | Maps & Directions A-Z Index | Site Map | Help …
        More results from: www.city.ac.uk September 16, 2003 - 14 KB
2. Flash Upgrade
        A University which believes in forward-thinking business and leadership skills, exposing
        students to the latest technology in all courses of study.
        www.cityu.edu November 1, 2003 - 27 KB
3. The City University of New York
        The University's Main Website.
        www.cuny.edu December 31, 1969 - 121 B
4. Welcome to Dublin City University
        … KnowledgeWorks, wins DCU Mallin-invent award Full text you can go anywhere in the
        world from here... Dublin City University , Dublin 9, Ireland. Tel. +353 (0) 1 700 5000,
        Fax. +353 (0) 1 836...
        More results from: www.dcu.ie October 29, 2003 - 5 KB
5. City University of Hong Kong
        Formerly the City Polytechnic of Hong Kong. Includes information on university, links to
        learning resources, research and departmental information and student information.
        More results from: www.cityu.edu.hk October 29, 2003 - 26 KB
6. HCU HomePage -English-
        … contact us For any suggestion and requests to this web site, www-admin@hiroshima-
        cu.ac.jp Feel free to link this web site. Hiroshima City University
        More results from: www.hiroshima-cu.ac.jp May 11, 2003 - 26 KB
                                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




       Teoma
1.   City University of Hong Kong Formerly the City Polytechnic of Hong Kong. Includes
     information on university, links to learning resources, research and departmental
     information...www.cityu.edu.hk/
     [Related Pages]
     [More Results from www.cityu.edu.hk]

2.   City University Text Version. The University for business and the professions ... Contact Us
     | About City University | Maps & Directions A-Z Index | Site Map | Help...www.city.ac.uk/
     [More Results from www.city.ac.uk]

3.   New Jersey City University Official site includes graduate and undergraduate information,
     news, events, online learning, calendars, and athletics.www.njcu.edu/
     [Related Pages]
     [More Results from www.njcu.edu]

4.   Elizabeth City State University Admission Application Introduction 2004-2005
     APPLICATION FOR ADMISSION. Welcome to the Elizabeth City State University Online
     Application. New Freshmen...www.ncmentor.org/applications/unc/apply/el...

5.   City University A University which believes in forward-thinking business and leadership
     skills, exposing students to the latest technology in all courses of study.www.cityu.edu/

6.   The City College of New York City College of New York (CUNY) A senior college of The City
     University of New York. Located in Manhattan. 138th Street at Convent Avenue. A Harlem
     masterpiece in the neo-Gothic...www.ccny.cuny.edu/
     [More Results from www.ccny.cuny.edu]
                     Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




        MetaSearch Engine
   Metasearch Engines are designed to
    increase the coverage of web by
    forwarding users’ queries to multiple search
    engines
       Users’ requests are sent to multiple search
        engines such as AlltheWeb, Google, MSN.
   Then the results from the individual search
    engine are combined into a single result set
    to present to users.
                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




     Different Forms of MetaSearch
   Submit different representations of the
    same query to the same search engine,
    then combine the results.

   Submit the same query to several search
    engine adopting different information
    retrieval models, then combine the results.
               Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng

    Issues
   How to combine the results retrieved by
    different source search engines is
    crucial for the success of a metasearch
    engine.
   And this is the problem that social
    choice theory has been trying to
    answer.
                      Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




         Search Engine Watch

   Interesting metat search engines are listed at
       http://www.searchenginewatch.com/links/article.p
        hp/2156241
                                   Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng

    Nielsen NetRatings Search Engine Ratings
    Source: http://searchenginewatch.com/reports/article.php/2156451


    Google: Shows searches at any Google-branded web site such as Google.com or Google
     Images.

    Yahoo: Shows searches at any Yahoo-branded web site such as Yahoo.com or Yahoo
     Local. Does NOT include searches at the Yahoo-owned sites of AltaVista, AllTheWeb and
     Overture.

    MSN: Shows searches at any MSN-branded web site such as MSN Search.

    AOL: Shows searches at any AOL-branded web site such as AOL Search. Does NOT
     include searches at AOL-owned Netscape Search (this is on the chart separately) or
     probably other AOL-owned sites.

    Ask: Shows searches at Ask Jeeves but not Ask Jeeves-owned Teoma. Does NOT show
     searches at the Ask Jeeves-owned sites of MyWay.com, iWon and My Search. These are
     shown on the chart separately.

    Earthlink: Shows searches at Earthlink.

    Other: Shows searches that occur at other search sites not named on the chart. Any site
     not listed on the chart can be assumed to have a share that is less than the smallest
     named site (IE, My Search is the lowest named search site with a 0.7% share. Other
     sites will have shares less than this).
                                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng

    Award Winners
    Source:   http://searchenginewatch.com/awards/article.php/3494141

    Outstanding Search Service
     Winner: Yahoo
     Best Meta Search Engine
     Winner: Jux2
     Best News Search Engine
     Winner: Google News
     Best Blog/Feed Search Engine
     Winner: Bloglines
     Best Image Search Engine
     Winner: Yahoo Images
     Best US Shopping Search Engine
     Winner: Google's Froogle
     Most SEO/Webmaster Friendly Search Provider
     Winner: Google
     Best Search Ads Provider
     Winners: Google AdWords
     Best Search Feature
     Honorable Mention: Clusty's Clustering, MSN's Search Builder, Ask Jeeves Binoculars Site
     Preview, Surfwax LookAhead, Pinpoint Shopping Suggestions, A9 Search History, My Jeeves,
     My Yahoo Search, Google Desktop Web History Feature, AOL Snapshots, Ask Jeeves Smart
     Search, Google OneBox Results, Yahoo Shortcuts
    Best Specialty Search Engine
     Honorable Mention: Google Local, Yahoo Local, Google Scholar, Scirus, Citeseer,
     Librarians' Index To The Internet
                           Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng

Best Meta Search Engine
Source: www.searchenginewatch.com


     Winner: Jux2
     Just 2 percent of the 490 votes cast in this category went to
      Jux2, but we thought it deserved to win over the more popular
      choices. It provides a clean, simple way to see all the results
      from the most important major search engines. You can easily
      view what the big three crawlers of Google, Yahoo and Ask
      Jeeves agree on -- plus the results that are specific to each
      search engine, as well. There's no better way to understand the
      unique voices that each search engine speaks with.
     Jux2 takes its name from initially juxtaposing results from
      Google and Yahoo. Ask Jeeves was later added, and we hope
      that MSN will be included down the line. We also wouldn't mind
      seeing paid listings being added to the service, perhaps in a
      right-hand column display. Paid listings can often be relevant to
      searchers as well, so we'd like to have them.
                          Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng
Best Meta Search Engine Second Place: Dogpile
Source: www.searchenginewatch.com



      Dogpile was the clear winner in the popular vote, gaining 37
       percent of the votes. Add in votes for sister-site MetaCrawler,
       which essentially is Dogpile in all but name, and the share goes
       up to 52 percent.
      However, we thought a winning meta search engine should
       make it easy to pick and choose exactly which search engines
       you want to search against. At Jux2, this can easily be done
       right from the home page or saved as a permanent choice. At
       Dogpile, it's not an option at all, whether you use the advanced
       search page or the preferences option.
      In addition, we also dislike the fact that sponsored listings are
       mixed in among the regular results. The paid listings are all
       clearly labeled as "Sponsored By," so there's no confusion on
       that front. But we'd like to see them physically segregated from
       the editorial results. For these reasons, Dogpile didn't make the
       winning spot this year.
                            Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng

Best Meta Search Engine: Honorable Mention: Clusty & Mamma
Source: www.searchenginewatch.com
      Clusty was a newcomer to meta search in 2004, though it comes from
       an old hand in the area, Vivisimo. Vivisimo has long run meta search on
       its site to demo its clustering technology. Clusty was designed to move
       past being a demonstration and into a site that can better help searchers
       every day.
      We like the clustering options that Clusty takes its name from, as well as
       the clear distinction between sponsored and editorial results. We also
       love that you can permanently pick and choose which web search
       engines you wish to search against. Unfortunately, the collection doesn't
       include either Google or Yahoo, something we think would be essential to
       a winner in this category.
      In terms of popular voting, Clusty drew 14 percent of the votes following
       Dogpile and MetaCrawler, a very strong showing for a new service.
      Mamma lacks the clustering that Clusty offers, but it does have all the
       other things we like: sponsored listings segregated from editorial results
       and the ability to permanently pick and choose search engines. But like
       Clusty, it also lacks access to results from either Google or Yahoo. In the
       popular voting, it came after Clusty, picking up 9 percent of those cast.
       We thought that was strong enough to warrant an honorable mention, as
       well.
               Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Social Choice Theory

   Studies on protocols that help a group
    of people make collective decisions,
    such as vote.
                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




A Fundamental problem
   Given a collection of agents (voters)
       with preferences over different alternatives
        (allocations, outcomes),
   How should society evaluate these
    alternatives and make a decision for all?
       that may be for the will of some voters but
        against that of others.
              Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Applications
   Voters elect president from several
    candidates.
   National polls for economic or political
    policy of the government
   The procedure or rule of election
   The rank of metasearch engine
    obtained from those of search engines
                Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Group Descisions
How do we make decisions
 Flip a coin?
 Dictatorship?

 Democracy (Majority rule)?

in a group with
       Heterogeneity of individual preference
                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Group Decision Rules
   Majority rule ,
       Condorcet paradox (voting cycle)
   Borda rule
                      Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Mathematical model
   A set of voters V={v1,v2,v3,…,Vn}

   A set of alternatives or outcomes
    S={s1,s2,s3,…Sm}, with |S|=m; and

   A set of preference relation P={R1,R2,R3…Rn},
    called a preference profile,
       the preference relation Ri for each voter i is a
        permutation (order) of elements in S.
                         Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Example 1 Majority Rule
   3 rational people have rational preferences
    over 2 alternatives {x,y}
                  Person
              1      2        3

        1st   X      Y        X                 1 : X>Y
Pref.                              i.e.Person 2 : Y>X
        2nd   Y      X       Y                  3 : X>Y


How to Aggregate their preferences? How to choose?
              Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Using majority rule.

   Since more than ½ people (two out of
    three) prefer x to y.
   Then the group prefers x to y
                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Example 2 Condorcet Paradox
   3 rational people have rational
    preferences over 3 alternatives {x,y,z}
                 Person
             1        2         3
       1st   X        Y         Z                     1 : X>Y>Z
Pref. 2nd    Y        Z         X i.e. Person 2 : Y>Z>X
       3rd   Z        X          Y                    3 : Z>X>Y
                     Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng


    Binary/paired Comparison With Majority rule
   Person
             1   2   3
     1st     X Y      Z             1 : X>Y
Pref. 2nd Y Z         X   for (x,y) 2 : Y>X X>Y
      3rd    Z X       Y            3 : X>Y
 Similarly, for (Y,Z) we can get Y>Z; for (Z,X) we

  can get Z>X.
 Then X>Y>Z>X (cycling) , Intransitive  Not

  rational
                Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   It was noted by Condorcet in the 18 century
    that no alternative can win a majority against
    all other alternatives.

   Pairwise majority is not satisfactory in all
    cases.
                Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Example 3 Borda Rule
   For each voter,
       associate the number 1 with the most
        preferred alternative,
       2 with the second and so on,
   Assign to each alternative the number
    equal to
       the sum of the numbers the individual
        voters assigned to the alternative.
                    Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




                   Person
               1       2           3
     1st    X(1)   Y(1)       X(1)            X(4)            X
Pref. 2nd   Y(2)   X(2) W(2)  Y(7)  Y
     3rd    Z(3)   W(3) Z(3)   Z(10)  W
     4th    W(4)   Z(4) Y(4)   W(9)    Z

  Then We get choice X>Y>W>Z
               Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   For above example, if we use binary/paired
    comparison With majority rule . We can get

    X>Y in 2 out of 3, Y>W in 2 out of 3,
    W>Z in 2 out of 3, X>W in 3 out of 3,
    X>Z in 3 out of 3, Y>Z in 2 out of 3

    Then we can achieve same choice
          X>Y>W>Z
                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   For the previous example we
    had trouble with majority rule
    via binary/paired comparison,
    we get a tie between all
    three alternatives with the
    Borda’s rule:
       All three alternatives get a sum
        of 6.
                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Some variations
1 with relevant scores available
  allotting each input system a point p to be
  distributed according to relevance scores of the
  documents.

2 Weighted Borda-rule
  Each voter may not have equal effectiveness to
  the final result. We may set more weight to good
  quality input systems.
                         Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




    Rank Aggregation in MetaSearch
    Here we discussed two cases which using
    algorithm rooted at social choice theory for
    MetaSearch rank aggregation.
   Data fusion track in TREC
[Javed A. Aslam, Mark Montague 2001] Models for Metasearch
 in SIGIR2001


   Rank aggregation for web search engine
    [Cynthia Dwork, Ravi Kumar, Moni Naor, D.Sivakumar 2001]
Rank Aggregation Methods for the Web in WWW10
               Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Data fusion track in TREC
   TREC (Text Retrieval Conference ,see
    http://trec.nist.gov/) maintains about 6Gb of
    SGML tagged text, queries and respective
    answers for evaluation purposes.
   The TREC organizers distribute data sets in
    advance and 50 new queries each year.
   The competing teams then submit ranked
    lists of documents that their system gave in
    response to each query. And these retrieval
    systems will be evaluated.
                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   These ranked lists are available for
    metasearch researchers to download and
    fuse.
   For each query, every retrieval system will
    return top 1000 documents and relevant
    score is available.
   Then given these results retrieved by many
    different retrieval systems, how to
    aggregate them for better performance?
                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Previous algorithms
   Min, Max and Average Models
    [Fox and Shaw,1995]
   Linear Combination Model
    [Bartell 1995]


   Logistic Regression Model
                   Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng


     Example
   Min, Max and Average model
    The final score of each document d is based on
    the scores given to d by each input systems
    (voters).
Algorithm                      Final score
CombMin         minimum of individual relevance scores
CombMed         median of individual relevance scores
CombMax         maximum of individual relevance scores
CombSum        sum of individual relevance scores
CombANZ        CombSum / num non-zero relevance scores
CombMNZ        CombSum * num non-zero relevance scores
                Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Linear Combination Model

        S LC (d )   ai si (d )
                        i

    The final score of document d is a simply
     linearly (each weighted differently)
     combining the normalized relevance
     scores given to each document.
                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Experiment result on TREC Model
      The performance of rank aggregation is
       evaluated by average precision over the
       queries
      Score-based borda-fuse is usually the best
       method among several borda variant
       algorithms.
      It is better than best input system over most
       of data collection. Such as TREC3, TREC5
                       Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




       Experiment result II
   The performance of rank aggregation is evaluated by
    average precision over the queries.

   Condorcet-fusion is the only algorithm that , without
    training data, ever matches the performance of the best
    input system over TREC 9.

   Condorcet-fusion seems particularly sensitive to the
    dependence of input systems. If the input systems (voters)
    are too similar, the performance will decrease.
                    Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




Rank aggregation methods for web
 New Challenges: Different from the case in
       TREC data fusion,
        The coverage of various search engine is
         different
        Thus some highly relevant web pages may not
         be ranked by some search engines.
        Therefore, each voter ranks a partial
         candidate list
                         Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




        Preliminaries
   Given a universe U, an ordered list  with
    respect to U is an ordering of a subset S  U,
    i.e.,  [ x1  x2  ...  xd ] ,with each xi  S , and  is
    some ordering relation on S.
   If  contains
       all the elements in U, then it is said to be a full list,
       otherwise it is called partial list.
                     Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Distance measures between two full lists with
    respect to a set S
      The Kendall tau distance
       It counts the number of pairwise disagreements between
        two lists.
       The distance is given by

    K ( , )  {(i, j) | i  j,  (i)   ( j), but  (i)   ( j)}
       Normalize it by dividing the maximum possible value
        2
     S /2
                         Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Spearman footrule distance

   Given two full lists  and  , the distance is given by

             F ( ,  )  i 1  (i)   (i)
                              s




    Normalize it by dividing the maximum value
                                                                                  2
                                                                              S /2
                                  Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng



   Distance measures for more than 2 list

    Given several full lists  1 , 2 ,...,  k , for
    instance, the normalized Footrule distance of
    to σ is given by
       F ( , 1,  2 ,..., k )  (1 / k )i 1 F ( ,  i )
                                              k



    An optimal Spearman’s footrule distance
    solution σ is one that minimizes the above
                                          i


    normalized footrule distance.
                           Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




       Kemeny optimal aggregation
The question is
 Given (full or partial) lists  1 , 2 ,...,  k , find a  such that 
 is a full list with respect to the union of the elements of
   , ,...,  and  minimizes K ( , , ,...,  )
   1   2     k                                     1   2      k




 The aggregation obtained by optimizing Kendall distance is
  called Kemeny optimal aggregation.
                 Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   When k>=4,computing the Kemeny
    optimal aggregation is NP-hard.

    (please refer to [Cynthia Dwork, Ravi Kumar, Moni
     Naor, D.Sivakumar 2001] for detailed proof )



    We can use Spearman footrule distance to
    approximate the Kendall distance.
                    Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Proposition 1 For any two full lists  ,  ,

       K ( , )  F ( , )  2 K ( , )

   Proposition 2 if  is the Kemeny optimal
    aggregation of full lists  1 ,  2 ,...,  k , and  
    optimizes the footrule aggregation, then

       K ( , 1 ,...,  k )  2 K ( , 1 ,...,  k )
                       Social Choice and Rank Aggregation in MetaSearch/ Xiaotie Deng




   Proposition 3 Given full lists  1 ,  2 ,...,  k if the
    median positions of the candidates in the lists
    form a permutation, then this permutation is a
    footrule optimal aggregation.

   Proposition 4 Footrule optimal aggregation of full
    lists can be computed in polynomial time,
    specifically, the time to find a minimum cost
    perfect matching in a bipartite graph.

				
DOCUMENT INFO
Shared By:
Categories:
Tags:
Stats:
views:5
posted:10/12/2012
language:English
pages:54