Document Sample
Clustering Powered By Docstoc
					                                 Conceptual Clustering

• Unsupervised, spontaneous - categorizes or postulates
  concepts without a teacher
• Conceptual clustering forms a classification tree - all
  initial observations in root - create new children using
  single attribute (not good), attribute combinations (all),
  information metrics, etc. - Each node is a class
• Should decide quality of class partition and significance
• Many models use search to discover hierarchies which
  fulfill some heuristic within and/or between clusters -
  similarity, cohesiveness, etc.
Machine Learning - Conceptual Clustering - 2/10/2012     Page 1

• Cobweb is an incremental hill-climbing strategy with bidirectional
  operators - not backtrack, but could return in theory
• Starts empty. Creates a full concept hierarchy (classification tree) with
  each leaf representing a single instance/object. You can choose how deep
  in the tree hierarchy you want to go for the specific application at hand
• Objects described as nominal attribute-value pairs
• Each created node is a probabilistic concept (a class) which stores
  probability of being matched (count/total), and for each attribute,
  probability of being on, P(a=v|C), only counts need be stored.
• Arcs in tree are just connections - nodes store info across all attributes
  (unlike ID3, etc.)

Machine Learning - Conceptual Clustering - 2/10/2012                     Page 2
               Category Utility: Heuristic Measure

• Tradeoff between intra-class similarity and inter-class
  dissimilarity - sums measures from each individual
• Intra-class similarity a function of P(Ai = Vij|Ck),
  Predictability of C given V - Larger P means if class is
  C, A likely to be V. Objects within a class should have
  similar attributes.
• Inter-class dissimilarity a function of P(Ck|Ai = Vij),
  Predictiveness of C given V - Larger P means A=V
  suggests instance is member of class C rather than some
  other class. A is a stronger predictor of class C.

Machine Learning - Conceptual Clustering - 2/10/2012   Page 3
                             Category Utility Intuition

• Both should be high over all (most) attributes for a
  good class breakdown
   – Predictability: P(V|C) could be high for multiple
     classes, giving a relatively low P(C|V), thus not
     good for discrimination
   – Predictiveness: P(C|V) could be high for a class,
     while P(V|C) is relatively low, due to V occurring
     rarely, thus good for discrimination, but not intra-
     class similarity
   – When both are high, get best categorization balance
     between discrimination and intra-class similarity
Machine Learning - Conceptual Clustering - 2/10/2012      Page 4
                                          Category Utility

• For each category sum predictability times predictiveness for
  each attribute weighted by P(Ai = Vij), with k proposed
  categories, i attributes, j values/attribute
                   P( Ai  Vij )P(Ck | Ai  Vij )P( Ai  Vij |Ck )
               k 1 i     j

Bayes Rule - P( Ai  Vij )P(Ck | Ai  Vij )  P(Ck )P( Ai  Vij |Ck )
                         Thus,           P(Ck )  P(Ai  Vij |Ck )
                                        k 1           i   j
The expected number of attribute
values one could guess given C
                                                             P( Ai  Vij |Ck )
                                                               i   j

Machine Learning - Conceptual Clustering - 2/10/2012                                   Page 5
                                           Category Utility

• Category Utility is the increase in expected attributes that could be
  guessed, given a partitioning of categories - leaf nodes.
• CU({C1, C2, ... Ck}) =

              P(C )[ P( A
             k 1
                                 i     j
                                                  i     Vij | Ck ) 2   P ( Ai  Vij | Ck ) 2 ]
                                                                       i   j


• K normalizes CU for different numbers of categories in the
  candidate partition
• Since incremental, there is a limited number of possible
  categorization partitions
• If Ai = Vij is independent (irrelevant) of class membership, CU = 0
Machine Learning - Conceptual Clustering - 2/10/2012                                                 Page 6
                         Cobweb Learning Algorithm

1. Incrementally add a new training example
2. Recurse down the at root until new node with just this example is added.
   Update appropriate probabilities at each level.
3. At each level of the tree calculate the scores for all valid modifications
   using category utility (CU)
4. Depending on which of the following gives the best score:
    – Classify into an existing class - then recurse
    – Create a new class node – done, can get next example
    – Combine two classes into a single class (Merging) - then recurse
    – Divide a class into multiple classes (Splitting) - then recurse

Machine Learning - Conceptual Clustering - 2/10/2012                      Page 7
                      Cobweb Learning Mechanisms

• Classifying (Matching) - calculate overall CU for each
  case of putting the example in a node at current level
• New Class - calculate overall CU for putting example
  into a single new class- Note gradient descent (greedy)
  nature. Does not go back and try all possible new
        – If created from internal node, simply add
        – If created from leaf node, split into two, one for new and old

• These alone are quite order dependent - splitting and
  merging allow bi-directionality - ability to undo

Machine Learning - Conceptual Clustering - 2/10/2012                Page 8
                      Cobweb Learning Mechanisms

• Merging - For best matching node (the one that would
  be chosen for classification) and the second best
  matching node at that level, calculate CU when both are
  merged into one node, with two children
• Splitting - For best matching node, calculate CU if that
  node were deleted and it’s children added to the current
• Both schemes could be extended to test other nodes, at
  the cost of increased computational complexity
• Can overcome initial “misconceptions”
Machine Learning - Conceptual Clustering - 2/10/2012   Page 9
                                      Cobweb Comments

• Generalization done by just executing recursive classification step
• Could use different variations on CU and search strategy
• Complexity: O(AVB2logK) for each example, where B is branching
  factor, A (attributes), V (average number of values), K (classes)
• Empirically, B usually between 2 and 5
• Does not directly handle noise - no defined significance
• Tends to make “bushy” trees, however high levels should be most
  important class categories (because of merge/split causing best
  breaks to float up, though no optimal guarantee), and one could
  just use nodes highest in the tree for classification
• Does not support continuous values
Machine Learning - Conceptual Clustering - 2/10/2012            Page 10
                                      Extensions - Classit

• Cannot store probability counts for continuous data
• Classit uses a scheme similar to Cobweb, but assumes
  normal distribution around an attribute and thus can just
  store a mean and variance - not always a reasonable
• Also uses a formal cut-off (significance) mechanism to
  better support generalization and noise handling (a class
  node can then include outliers)
• More work needed

Machine Learning - Conceptual Clustering - 2/10/2012         Page 11

Shared By: