Merging Judgments and the Problem of Truth-Tracking by sdfsb346f

VIEWS: 6 PAGES: 30

More Info
									     Merging Judgments and the Problem of
                Truth-Tracking

           Stephan Hartmann (with Gabriella Pigozzi)

                   Center for Logic and Philosophy of Science
                      Tilburg University, The Netherlands


progic07: 3rd Workshop on Combining Probability and Logic
         University of Kent, UK, 6 September 2007



  Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model
                                  Some Results
                                    Conclusions


Judgment Aggregation
     A committee has to make a decision on logically
     interconnected propositions.
     Example: A jury has to decide whether a defendant should be
     sent to prison (R). This is the case iff he is guilty (P) and
     liable (Q). Each jury member submits a judgment on all three
     propositions (P, Q and R), respecting their logical inter-
     relations. The task is then to obtain a collective judgment.
     Related problems: multi agent systems, sensors, decision
     making in medicine, etc.
     Mirroring the literature in Social Choice Theory, the literature
     on Judgment Aggregation formulates conditions on acceptable
     aggregation rules (i.e. a mapping of individual judgments into
     a collective judgment).
     This leads to various impossibility theorems.
      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model
                                  Some Results
                                    Conclusions


Assessing Aggregation Rules

      The impossibility theorems raise the question which of the
      conditions have to be given up to arrive at a possibility
      theorem.
      There is a literature that examines these conditions and
      explores which should be given up in real decision situations.
      We believe that (i) this question cannot be answered in
      general; it depends on the case in question. (ii) Instead of
      focusing on the conditions, we should step back and ask what
      one wants from an aggregation procedure. One might then
      choose the procedure that is conducive to this purpose, such
      as tracking the truth or maximizing the utility of the group.


      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                                                  An Example
                            Discursive Dilemma
                                                  Ways out 1: The Premise-Based Procedure (PBP)
                                     Our Model
                                                  Ways out 2: The Conclusion-Based Procedure (CBP)
                                  Some Results
                                                  Ways out 3: Distance-Based Procedures
                                    Conclusions


The Discursive Dilemma
     Back to our example: A jury with 7 members has to decide
     whether a defendant should be sent to prison (R). This is the
     case iff he is guilty (P) and liable (Q). They vote as follows:

                                                   P        Q         R
                         Members 1,2,3            Yes      Yes       Yes
                         Members 4,5              Yes      No        No
                         Members 6,7              No       Yes       No
                         Majority                 Yes      Yes       No

     We learn: Propositionwise majority voting leads to an
     inconsistency on the collective level.
     Some terminology: P and Q are the premises, R is the
     conclusion of the argument (P ∧ Q) ↔ R.
      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                                                  An Example
                            Discursive Dilemma
                                                  Ways out 1: The Premise-Based Procedure (PBP)
                                     Our Model
                                                  Ways out 2: The Conclusion-Based Procedure (CBP)
                                  Some Results
                                                  Ways out 3: Distance-Based Procedures
                                    Conclusions


Ways out 1: The Premise-Based Procedure (PBP)

     Propositionswise majority voting on the premises, then infer
     the conclusion.
     Motivating intuition: The reasons for a decision are important.

                                                   P        Q         R
                         Members 1,2,3            Yes      Yes        –
                         Members 4,5              Yes      No         –
                         Members 6,7              No       Yes        –
                         Majority                 Yes      Yes       Yes

     Decision: The defendant is sent to prison.
     Note: This procedure is manipulable.

      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                                                  An Example
                            Discursive Dilemma
                                                  Ways out 1: The Premise-Based Procedure (PBP)
                                     Our Model
                                                  Ways out 2: The Conclusion-Based Procedure (CBP)
                                  Some Results
                                                  Ways out 3: Distance-Based Procedures
                                    Conclusions


Ways out 2: The Conclusion-Based Procedure (CBP)

     Propositionwise majority voting on the conclusion.
     Motivating intuition: Jurors make up their minds on the
     premises privately and then submit their judgment on the
     conclusion.

                                                  P      Q       R
                         Members 1,2,3            –      –      Yes
                         Members 4,5              –      –      No
                         Members 6,7              –      –      No
                         Majority                 –      –      No

     Decision: The defendant is not sent to prison.
     Note: This procedure is not manipulable.
      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                                                  An Example
                            Discursive Dilemma
                                                  Ways out 1: The Premise-Based Procedure (PBP)
                                     Our Model
                                                  Ways out 2: The Conclusion-Based Procedure (CBP)
                                  Some Results
                                                  Ways out 3: Distance-Based Procedures
                                    Conclusions


Ways out 3: Distance-Based Procedures
     Avoid dilemmas and find an aggregation method that ensures
     consistency in every step of the aggregation procedure.
     Motivating intuition: Find some kind of average.
     How can this idea be made more precise?
     Four situations (or “judgment sets”): (YES: 1, NO: 0)
                        S1 := (1, 1, 1)           ,       S2 := (1, 0, 0)
                        S3 := (0, 1, 0)           ,       S4 := (0, 0, 0)
     Identify the situation that has minimal distance to the
     judgments expressed by the voters.
     One option: Hamming distance
     Example: The Hamming distance btw. S1 and S1 is 0, btw.
     S1 and S2 is 2, btw. S1 and S3 2, and btw. S1 and S4 3.
      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                                                  An Example
                            Discursive Dilemma
                                                  Ways out 1: The Premise-Based Procedure (PBP)
                                     Our Model
                                                  Ways out 2: The Conclusion-Based Procedure (CBP)
                                  Some Results
                                                  Ways out 3: Distance-Based Procedures
                                    Conclusions


Distance-Based Procedures: How It Works

     The total distances di of all submitted judgments to situation
     Si can be expressed in terms of the numbers ni of voters for
     each situation Sj (j = 1, . . . , 4):

            d1 = 2n2 + 2n3 + 3n4                   ;       d2 = 2n1 + 2n3 + n4
              d3 = 2n1 + 2n2 + n4                  ;       d4 = 3n1 + n2 + n3

     In our example, we have: n1 = 3, n2 = n3 = 2 and n4 = 0.
     Hence, d1 = 8, d2 = 10, d3 = 10 and d4 = 13.
     Result: Situation 1 has minimal distance and is selected by
     this procedure. Hence, the defendant is sent to prison.
     Note: One gets a ranking: S1                      S2 , S3     S4 .

      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                                                  An Example
                            Discursive Dilemma
                                                  Ways out 1: The Premise-Based Procedure (PBP)
                                     Our Model
                                                  Ways out 2: The Conclusion-Based Procedure (CBP)
                                  Some Results
                                                  Ways out 3: Distance-Based Procedures
                                    Conclusions


Distance-Based Procedures: Another illustration

      Show: Distance minimization = averaging
                                                       P        Q         R
                             Members 1,2,3             1        1         1
                             Members 4,5               1        0         0
                             Members 6,7               0        1         0
                             Average                  5/7      5/7       3/7
      Distance of the average to S1 : D1 = 2/7 + 2/7 + 4/7 = 8/7
      Distance of the average to S2 and S3 : D2 = D3 = 10/7
      Distance of the average to S4 : D4 = 13/7
      S1 has minimal distance to the average.
      This insight can be generalized: di = N · Di

      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                                                   An Example
                             Discursive Dilemma
                                                   Ways out 1: The Premise-Based Procedure (PBP)
                                      Our Model
                                                   Ways out 2: The Conclusion-Based Procedure (CBP)
                                   Some Results
                                                   Ways out 3: Distance-Based Procedures
                                     Conclusions


Which procedure is best?


  This depends on what our goals are. We may want, for example,
  that the aggregation procedure . . .
      avoids dimemmas,
      maximizes the total utility of the group
      tracks the truth, or
      satisfies a combination of these goals.
  Distance-based approaches (a.k.a. “belief fusion”, “belief
  merging”) satisfy the first goal, but how well does it do with
  regard to the others? Let us focus here on truth-tracking.



       Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                        Motvation
                               Discursive Dilemma
                                        Our Model
                                     Some Results
                                       Conclusions


Truth tracking



      We follow the account of Nozick (1981).
      An aggregation procedure tracks the truth if the following
      four subjunctive conditionals are true:
        (i)   If   S1   were   the   true   state    of   the   world,   then   S1   would   be   chosen.
       (ii)   If   S2   were   the   true   state    of   the   world,   then   S2   would   be   chosen.
      (iii)   If   S3   were   the   true   state    of   the   world,   then   S3   would   be   chosen.
      (iv)    If   S4   were   the   true   state    of   the   world,   then   S4   would   be   chosen.




      Stephan Hartmann (with Gabriella Pigozzi)            Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                             Discursive Dilemma
                                      Our Model
                                   Some Results
                                     Conclusions


The Condorcet framework
  We consider the case of P ∧ Q ↔ R and compare the performance
  of fusion with the PBP and the CBP studied by Bovens &
  Rabinowicz (2006). Assumptions:
      P and Q are logically and probabilistically independent.
      All N voters are equally competent and independent.
      The chance that a voter correctly judges the truth or falsity of
      proposition P (her competence) is p. The same for Q.
      The prior probability that P is true is q. The same for Q.
      There are four possible situations:
             S1   = {P, Q, R} = (1, 1, 1)
             S2   = {P, ¬Q, ¬R} = (1, 0, 0)
             S3   = {¬P, Q, ¬R} = (0, 1, 0)
             S4   = {¬P, ¬Q, ¬R} = (0, 0, 0)

       Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                             Discursive Dilemma
                                      Our Model
                                   Some Results
                                     Conclusions


Truth tracking in the limit



  In probabilistic terms, truth tracking can be explicated as follows:
   (i) P(The procedure choses S1 |S1 ) → 1 if N → ∞
   (ii) P(The procedure choses S2 |S2 ) → 1 if N → ∞
  (iii) P(The procedure choses S3 |S3 ) → 1 if N → ∞
  (iv) P(The procedure choses S4 |S4 ) → 1 if N → ∞
  Construct a probabilistic model to test these claims.




       Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model
                                  Some Results
                                    Conclusions


Our framework


     We want to calculate the probability of the proposition F :
     Fusion ranks the right judgment set first.
     Note that
                                              4
                               P(F ) =            P(F |Si ) · P(Si ),
                                            i=1

     so we have to calculate the prior probabilities P(Si ) and the
     conditional probabilities P(F |Si ) for i = 1, . . . , 4.
     The prior probabilities of the situations are (with x := 1 − x):
                                                         ¯
              P(S1 ) = q 2 ; P(S2 ) = P(S3 ) = q¯; P(S4 ) = q 2
                                                q           ¯



      Stephan Hartmann (with Gabriella Pigozzi)    Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model
                                  Some Results
                                    Conclusions


Our framework (cont’d)
     Let’s assume that S1 is the right judgment set. Then the right
     judgment set ranks first if d1 ≤ min(d2 , d3 , d4 ).
     The distances di can be expressed in terms of the numbers ni
     of voters for each judgment set Si (i = 1, . . . , 4):
           d1 = 2n2 + 2n3 + 3n4                    ;       d2 = 2n1 + 2n3 + n4
               d3 = 2n1 + 2n2 + n4                 ;       d4 = 3n1 + n2 + n3
     We now calculate:
                            N
                                             N
     P(F |S1 ) =                                       p 2n1 (p¯)n2 +n3 p 2n4 C(n1 , . . . , n4 )
                                                               p        ¯
                                       n1 , . . . , n4
                      n1 ,...,n4 =0

     The sum is constrained: C(n1 , . . . , n4 ) = 1 if (i) 4 ni = N
                                                                i=1
     and (ii) d1 ≤ min(d2 , d3 , d4 ). Otherwise C(n1 , . . . , n4 ) = 0.
      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Fusion ranks the right judgment set first (R) compared
with PBP (G), and CBP (B) for N = 3 and q = .5

                1

           0.8

           0.6

            0.4

            0.2

                0
                    0         0.2           0.4       0.6          0.8            1


      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Same for N = 11

                1

           0.8

           0.6

            0.4

            0.2

                0
                    0         0.2           0.4       0.6          0.8            1



      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Same for N = 21

                1

           0.8

           0.6

            0.4

            0.2

                0
                    0         0.2           0.4       0.6          0.8            1



      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                             Discursive Dilemma
                                      Our Model    How does fusion compare to PBP and CBP?
                                   Some Results
                                     Conclusions


Truth tracking




  All three procedures track the truth in the limit if p > 1/2.




       Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                             Discursive Dilemma
                                      Our Model     How does fusion compare to PBP and CBP?
                                   Some Results
                                     Conclusions


Right result

      Let us now calculate the probability of G : Fusion ranks a
      judgment set with the right result (not necessarily for the
      right reasons) first.
      In this case, we have

                          S1 true         →        d1 ≤ min(d2 , d3 , d4 )
                          S2 true         →        min(d2 , d3 , d4 ) ≤ d1
                          S3 true         →        min(d2 , d3 , d4 ) ≤ d1
                          S4 true         →        min(d2 , d3 , d4 ) ≤ d1

      With this, we obtain . . .


       Stephan Hartmann (with Gabriella Pigozzi)    Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                             Discursive Dilemma
                                      Our Model    How does fusion compare to PBP and CBP?
                                   Some Results
                                     Conclusions


Fusion ranks a judgment set with the right result (not nec.
for the right reasons) first (R) comp. with PBP (G), and
CBP (B) for N = 3 and q = .5
                 1

            0.9

            0.8

             0.7

            0.6

            0.5
                     0         0.2           0.4       0.6          0.8            1

       Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Same for N = 11

                1

           0.9

           0.8

            0.7

           0.6

           0.5
                    0         0.2           0.4       0.6          0.8            1



      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Same for N = 31

                1

           0.9

           0.8

            0.7

           0.6

           0.5
                    0         0.2           0.4       0.6          0.8            1



      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                      Motvation
                             Discursive Dilemma
                                      Our Model    How does fusion compare to PBP and CBP?
                                   Some Results
                                     Conclusions


Fusion ranks a judgment set with the right result (not nec.
for the right reasons) first (R) comp. with PBP (G), and
CBP (B) for N = 3 and q = .2
                 1
            0.9
            0.8
             0.7
            0.6
            0.5
             0.4

                     0         0.2           0.4       0.6          0.8            1

       Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Same for N = 21

                1
           0.9
           0.8
            0.7
           0.6
           0.5
            0.4

                    0         0.2           0.4       0.6          0.8            1



      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                     Motvation
                            Discursive Dilemma
                                     Our Model    How does fusion compare to PBP and CBP?
                                  Some Results
                                    Conclusions


Same for N = 51

                1
           0.9
           0.8
            0.7
           0.6
           0.5
            0.4

                    0         0.2           0.4       0.6          0.8            1



      Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                       Motvation
                              Discursive Dilemma
                                       Our Model    How does fusion compare to PBP and CBP?
                                    Some Results
                                      Conclusions


Understanding the dip
  We decompose:
                                   4
             P(G ) =                   P(G |Si ) · P(Si )
                                 i=1
                                  4
                          =            [P< (G |Si ) + P> (G |Si )] · P(Si )
                                 i=1
                                                     4
                          = P> (G |S1 ) +                 P< (G |Si ) · P(Si )
                                                    i=1

  P< (G |Si ): the probability that fusion selects the right outcome
  given that the minority votes for the right situation Si
  P> (G |Si ): the probability that fusion selects the right outcome
  given that the majority votes for the right situation Si
       Stephan Hartmann (with Gabriella Pigozzi)    Merging Judgments and the Problem of Truth-Tracking
                                       Motvation
                              Discursive Dilemma
                                       Our Model     How does fusion compare to PBP and CBP?
                                    Some Results
                                      Conclusions


Understanding the dip (N = 21)
                          1

                       0.8

                       0.6

                       0.4

                       0.2


                              0      0.2       0.4     0.6       0.8        1


  Probability that fusion selects the right outcome given that:
  – the majority votes for the right situation, S1 true (R)
  – the minority votes for the right situation, S1 true (G)
  – the minority votes for the right situation, S2 [S3 ] true (B)
  – the minority votes for the right situation, S4 true (T)
       Stephan Hartmann (with Gabriella Pigozzi)     Merging Judgments and the Problem of Truth-Tracking
                               Motvation
                      Discursive Dilemma
                               Our Model
                            Some Results
                              Conclusions



The fusion approach does especially well for middling values
of the competence p (p ≈ .5).
For other values of p, the fusion approach is often in between
PBP and CBP (whichever is better in the case at hand).
                                            (∨)             (∧)
Disjunctive case analogous: Pq (F ) = Pq (F ) with
                                       ¯
                                   (∨)            (∧)
q = 1 − q. Similarly, Pq (G ) = Pq (G ).
¯                                ¯
Open questions
      Generalization to more than two premises (⇒ Monte Carlo)
      Dependencies between voters: social networks
      Introduce weights for the premises and the conclusion and fix
      them such that merging does best in terms of truth tracking
      Introduce reliabilities
      Can this framework be used to model deliberation? Under
      which conditions does a group reach consensus? (cf. the
      Lehrer-Wagner model)
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
                                    Motvation
                           Discursive Dilemma
                                    Our Model
                                 Some Results
                                   Conclusions




Thanks for your attention!




     Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking

								
To top