Performance Evaluation of Grouping Algorithms by Uel8ee

VIEWS: 8 PAGES: 25

									Design and Perceptual Validation
        of Performance Measures
     for Salient Object Segmentation


              Vida Movahedi, James H. Elder
                  Centre for Vision Research
                    York University, Canada
Evaluation of Salient Object Segmentation




              Source: Berkeley Segmentation Dataset




2         Centre for Vision Research, York University
Evaluation of Salient Object Segmentation



       How do we measure success?




3         Centre for Vision Research, York University
Existing literature
       Salient object segmentation
           [Liu07, Zhang07, Park07, Zhuang09, Achanta09, Pirnog09, …]


       Evaluation of salient object segmentation algorithms
           [Ge06,?]

       Evaluation of segmentation algorithms
           [Huang95, Zhang96, Martin01, Monteiro06, Goldmann08, Estrada09]




    4                     Centre for Vision Research, York University
Contributions
       Analysis of previously suggested measures
       Contour Mapping Measure (CM)
           Order-preserving matching

       A new dataset of salient objects (SOD)
       Psychophysics experiments
           Evaluation of above measures
       Matching paradigm in Precision and Recall measures




    5                   Centre for Vision Research, York University
Evaluation measures in literature
       Region-based error measures
           Based on false positive/ false negative pixels
           [Young05], [Ge06], [Goldmann08], ...

       Boundary-based error measures
           Based on distance between boundaries
           [Huttenlocher93], [Monteiro06], ...

       Mixed measures
           Based on distance of misclassified pixels to the boundaries
           [Young05], [Monteiro06], ...




    6                      Centre for Vision Research, York University
Region-based error measures
[Young05], [Ge06], [Goldmann08], ...

       A and B two boundaries
       RA the region corresponding to a boundary A and |RA| the
        area of this region,
                                              False Positives         False Negatives

                           RA  RB          | RA |  RA  RB        | RB |  RA  RB
        RI ( A, B)  1                                        
                           RA  RB              RA  RB                  RA  RB




                                                      Not sensitive to shape differences
    7                      Centre for Vision Research, York University
Boundary-based error measures
[Huang95],[Huttenlocher93], [Monteiro06], ...

       A and B two boundaries
       Distance of one point a on A from B is d B (a)  mind (a, b)
                                                                             bB

                                                   
        Hausdorff distance: HD( A, B)  max max d B (a), max d A (b)               
                                                                                      
                                            aA          bB

       Mean distance:            MD( A, B)  mean mean d B (a), mean d A (b)
                                                        aA            bB




           a                                     Not sensitive to shape differences
    8                    Centre for Vision Research, York University
Mixture error measures
[Young05], [Monteiro06], ...

       Penalizing the over-detected and under-detected regions by
        their distances to intersection
                                False Negatives       False Positives

                        1       1    N fn
                                                   1
                                                         N fp
                                                                      
        MM ( A, B) 
                     2 Ddiag
                               
                               N     dA( pj )  N       d B ( qk ) 
                                                                      
                                fn   j 1          fp   k 1         




                                                  Not sensitive to shape difference
    9                   Centre for Vision Research, York University
Another example




       Different shapes with low errors



10      Centre for Vision Research, York University
   Comparing two boundaries


    Small False
Negative Region

                                                                    Small False
                                                                    Positive Region



       The two boundaries need to follow each other
       Thus it is not sufficient to map points to the closest
        point on the other boundary
       The ordering of mapped points must be preserved

       11             Centre for Vision Research, York University
Order-preserving Mapping
   The order of mapped points on the two boundaries must be
    monotonically non-decreasing.
     If ai  bm , a j  bn and i  j then m  n


   Allowing for different levels of detail:
       One-to-one
       Many-to-one
       One-to-many




12                    Centre for Vision Research, York University
Contour Mapping Measure
    Given two contours A=a1a2..an and B=b1b2..bm,
    Find the correct order-preserving mapping


                  Contour mapping error measure:
           Average distance between matched pairs of points




    Bimorphism [Tagare02]
    Elastic Matching [Geiger95, Basri98, Sebastian03, ..]



13                   Centre for Vision Research, York University
Contour Mapping Measure

   A dynamic programming implementation to
    find the optimum mapping
        Closed contours  point indices are assigned cyclically


        Based on string correction techniques [Maes90]
        Complexity: O(nm log m)
         if m<n and m, n points on two boundaries




    14                  Centre for Vision Research, York University
Contour Mapping Example
     Ground Truth Boundary                        Algorithm Boundary




                   Matched pairs shown as line segments



                                                               CM= average length of
                                                             line segments connecting
                                                                   matched pairs




15             Centre for Vision Research, York University
Contour Mapping Measure
    Order- preserving mapping avoids problems experienced
     by other measures




    16           Centre for Vision Research, York University
    SOD: Salient Object Dataset
        A dataset of salient objects
        Based on Berkeley Segmentation Dataset (BSD) [Martin01]
        300 images
        7 subjects


                                                            1
                                                        1




                                                    1

                                                1


                                                            1


Source: Berkeley Segmentation Dataset                                        Available in SOD



        17                     Centre for Vision Research, York University
Psychophysical experiments
    Which error measure is closer to human judgements of
     shape similarity?

    9 subjects
    5 error measures
             Regional Intersection (RI)
             Mean distance (MD)
             Hausdorff distance (HD)
             Mixed distance (MM)
             Contour Mapping (CM)


    18              Centre for Vision Research, York University
  Psychophysical Experiments
         Experiment 1 - SOD                              Experiment 2 - ALG
      Reference & test shapes all                        Reference from SOD,
             from SOD                              test shapes algorithm-generated


 Reference:                                                                      Reference:
   Human                                                                           Human
segmentation                                                                    segmentation




                 Test cases:                                      Test cases:
                  Human                                          Algorithm-g
               segmentations                                       enerated




    19                         Centre for Vision Research, York University
Agreement with Human Subjects
   Human subject chooses Left or                           Reference
    Right
   An error measure M also
    chooses Left or Right, based on
    their error w.r.t. the reference
    shape
   If M chooses the same as the
    human, it is a case of agreement


   Human-Human consistency:
                                                    Left                Right
    defined based on agreement
    between human subjects
    20             Centre for Vision Research, York University
Psychophysical Experiments
       Experiment 1- SOD                                       Experiment 2 - ALG
   Reference & tests shapes all                                Reference from SOD,
           from SOD                                      test shapes algorithm-generated




RI: region intersection, MD: mean distance, HD: Hausdorff distance, MM: mixed measure, CM: contour mapping

 21                        Centre for Vision Research, York University
Precision and Recall measures
    For algorithm boundary A and ground truth boundary B
                                                                      matched( A, B)
    Precision: proportion of true positives on A                   
                                                                           | A|
                                                                      matched( B, A)
    Recall:       proportion of detected points on B               
                                                                           |B|


    Martin’s PR (M-PR)[Martin04]
        Minimum cost bipartite matching, cost proportional to distance
    Estrada’s PR (E-PR)[Estrada09]
        ‘No intervening contours’ and ‘Same side’ constraints
    Contour Mapping PR (CM-PR)
        Order-preserving mapping

    22                Centre for Vision Research, York University
Matching paradigm in Precision/Recall
    Experiment 1- SOD                          Experiment 2 - ALG
 Reference & test shapes all                   Reference from SOD,
        from SOD                         test shapes algorithm-generated




23              Centre for Vision Research, York University
Summary
    Analysis of available measures for evaluation of salient object
     segmentation algorithms
    A new measure- contour mapping measure (CM)
        Code available online: http://elderlab.yorku.ca/ContourMapping

    A new dataset of salient objects
        Dataset available online: http://elderlab.yorku.ca/SOD

    Psychophysical Experiment
        CM has a higher agreement with human subjects

    Order-preserving matching paradigm in Precision/Recall
     analysis
        Code available online: http://elderlab.yorku.ca/ContourMapping


    24                 Centre for Vision Research, York University
                    Thank You!




25   Centre for Vision Research, York University

								
To top