Docstoc

Uncertainty

Document Sample
Uncertainty Powered By Docstoc
					Recommender Systems
Collaborative Filtering Process
           Challenge - Sparsity
• Active users may have purchased well under 1% of the
  items (1% of 2 million books is 20,000 books).

• Solution: Use sparse representations of the rating matrix.
            Ratings in a hashtable
critics = {
   'Lisa Rose':       {'Lady in the Water': 2.5,
                  'Snakes on a Plane': 3.5,
                  'Just my Luck': 3.0,
                  'Superman Returns': 3.5,
                  'You, Me and Dupree': 2.5,
                  'The Night Listener': 3.0},

  'Gene Seymour': {'Lady in the Water': 3.0,
             'Snakes on a Plane': 3.5,
             'Just my Luck': 1.5,
             'Superman Returns': 5.0,
             'The Night Listener': 3.0,
             'You, Me and Dupree': 3.5},
          Ratings in a hashtable
'Michael Phillips': {'Lady in the Water': 2.5,
              'Snakes on a Plane': 3.0,
              'Superman Returns': 3.5,
              'The Night Listener': 4.0},
'Claudia Puig': {'Snakes on a Plane': 3.5,
              'Just my Luck': 3.0,
              'The Night Listener': 4.5,
              'Superman Returns': 4.0,
              'You, Me and Dupree': 2.5},
'Mick LaSalle': {'Lady in the Water': 3.0,
              'Snakes on a Plane': 4.0,
              'Just my Luck': 2.0,
              'Superman Returns': 3.0,
              'The Night Listener': 3.0,
              'You, Me and Dupree': 2.0},
          Ratings in a hashtable
'Jack Matthews': {'Lady in the Water': 3.0,
            'Snakes on a Plane': 4.0,
            'Superman Returns': 5.0,
            'The Night Listener': 3.0,
            'You, Me and Dupree': 3.5},

'Toby':       {'Snakes on a Plane': 4.5,
            'Superman Returns': 4.0,
            'You, Me and Dupree': 1.0}
}
          Finding Similar Users
• Simple way to calculate a similarity score is to use
  Euclidean distance, which considers the items that people
  have ranked in common.




                   People in preference space
  Computing Euclidean Distance
def sim_distance(prefs, person1, person2):
  #Get the list of shared items
  si=[]
  for item in prefs[person1]:
     if item in prefs[person2]:
         si += [item]

  if len(si) == 0: return 0

  sum_of_squares = sum(
       [ (prefs[person1][item]-prefs[person2][item])**2
        for item in si]
     )
  return 1/(1+sqrt(sum_of_squares))
     Pearson Correlation Score
• The correlation coefficient is a measure of how well two
  sets of data fit on a straight line.
                                           Best fit line
        Pearson Correlation Score
                                             • Corrects for grade
                                               inflation.
                                                – E.g., Jack Matthews
                                                  tends to give higher
                                                  scores than Lisa Rose,
                                                  but the line still fits
                                                  because they have
                                                  relatively similar
                                                  preferences.


                                             • Euclidean distance
Two critics with a high correlation score.     score will say they are
                                               quite dissimilar...
Pearson Correlation Formula


               xy   x y
  r                     N
       
        x2   x   y 2   y  
                    2
                                  2
                                      
              N 
                               N 
                                   
         Geometric Interpretation
• For centered data (i.e., data which have been shifted by the sample
   mean so as to have an average of zero), the correlation coefficient can
   also be viewed as the cosine of the angle between two vectors.
E.g.,
• suppose a critic rated five movies by 1, 2, 3, 5, and 8, respectively,
• and another critic rated those movies by .11, .12, .13, .15, and .18.

• These data are perfectly correlated: y = 0.10 + 0.01 x.
   – Pearson correlation coefficient must therefore be exactly one.
• Centering the data (shifting x by E(x) = 3.8 and y by E(y) = 0.138)
  yields
      x = (−2.8, −1.8, −0.8, 1.2, 4.2) and
      y = (−0.028, −0.018, −0.008, 0.012, 0.042), from which
  as expected.
          Pearson Correlation Code
def sim_pearson(prefs, person1, person2):
  si=[]
  for item in prefs[person1]:
      if item in prefs[person2]:
              si += [item]
  n = len(si)
  if n == 0: return 0

  #Add up all the preferences
  sum1 = sum([prefs[person1][item] for item in si])
  sum2 = sum([prefs[person2][item] for item in si])

  #Sum up the squares
  sum1Sq = sum([prefs[person1][item]**2 for item in si])
  sum2Sq = sum([prefs[person2][item]**2 for item in si])

  #Sum up the products
  pSum=sum([ prefs[person1][item] * prefs[person2][item] for item in si ])

  #Calculate Pearson Score
  numerator = pSum-(sum1*sum2/n)
  denumerator = sqrt( (sum1Sq-sum1**2/n) * (sum2Sq-sum2**2/n) )
  if denumerator == 0: return 0
  return numerator/denumerator
                        Top Matches
def topMatches(critics, person, n=5, similarity=sim_pearson):

   scores=[ (similarity(critics,person,other), other)
          for other in critics if other!=person]

   scores.sort()
   scores.reverse()
   return scores[0:n]

>> recommendations.topMatches(recommendations.critics,'Toby',n=3)
   [(0.99124070716192991, 'Lisa Rose'),
    (0.92447345164190486, 'Mick LaSalle'),
   (0.89340514744156474, 'Claudia Puig')]
Recommending Items
              Recommending Items
def getRecommendations(prefs, person, similarity=sim_pearson):
 totals={}
 simSums={}
 for other in prefs:
  if other==person: continue
  sim=similarity(prefs,person,other)

  if sim<=0: continue
  for item in prefs[other]:

    # only score movies I haven't seen yet
  if item not in prefs[person]:
      # Similarity * Score
      totals.setdefault(item,0)
       totals[item]+=prefs[other][item]*sim
      # Sum of similarities
      simSums.setdefault(item,0)
      simSums[item]+=sim
             Recommending Items
 # Create the normalized list
 rankings=[(total/simSums[item],item) for item,total in totals.items()]

 # Return the sorted list
 rankings.sort( )
 rankings.reverse( )
 return rankings



>>> recommendations.getRecommendations(recommendations.critics,'Toby')
   [(3.3477895267131013, 'The Night Listener'),
    (2.8325499182641614, 'Lady in the Water'),
    (2.5309807037655645, 'Just My Luck')]
          Matching Products
• Recall Amazon…
               Transform the data
{'Lisa Rose': {'Lady in the Water': 2.5, 'Snakes on a Plane': 3.5},
'Gene Seymour': {'Lady in the Water': 3.0, 'Snakes on a Plane': 3.5}}
to:
{'Lady in the Water':{'Lisa Rose':2.5,'Gene Seymour':3.0},
'Snakes on a Plane':{'Lisa Rose':3.5,'Gene Seymour':3.5}} etc..

def transformPrefs(prefs):
 result={}
 for person in prefs:
  for item in prefs[person]:
    result.setdefault(item,{})
    # Flip item and person
    result[item][person]=prefs[person][item]
 return result
            Getting Similar Items
>> movies=recommendations.transformPrefs(recommendations.critics)
>> recommendations.topMatches(movies,'Superman Returns')
   [(0.657, 'You, Me and Dupree'),
    (0.487, 'Lady in the Water'),
    (0.111, 'Snakes on a Plane'),
    (-0.179, 'The Night Listener'),
    (-0.422, 'Just My Luck')]
  Whom to invite to a premiere?
>>recommendations.getRecommendations(movies,'Just
  My Luck')
  [(4.0, 'Michael Phillips'),
   (3.0, 'Jack Matthews')]

• For another example, reversing the products with the
  people, as done here, would allow an online retailer to
  search for people who might buy certain products.
                 Building a Cache
def calculateSimilarItems(prefs,n=10):
 # Create a dictionary of items showing which other items they
 # are most similar to.
 result={}

 # Invert the preference matrix to be item-centric
 itemPrefs=transformPrefs(prefs)

 for item in itemPrefs:
   # Find the most similar items to this one
  scores=topMatches(itemPrefs,item,n=n,similarity=sim_distance)
  result[item]=scores
 return result