VIEWS: 6 PAGES: 30 POSTED ON: 3/17/2010 Public Domain
Merging Judgments and the Problem of Truth-Tracking Stephan Hartmann (with Gabriella Pigozzi) Center for Logic and Philosophy of Science Tilburg University, The Netherlands progic07: 3rd Workshop on Combining Probability and Logic University of Kent, UK, 6 September 2007 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Judgment Aggregation A committee has to make a decision on logically interconnected propositions. Example: A jury has to decide whether a defendant should be sent to prison (R). This is the case iﬀ he is guilty (P) and liable (Q). Each jury member submits a judgment on all three propositions (P, Q and R), respecting their logical inter- relations. The task is then to obtain a collective judgment. Related problems: multi agent systems, sensors, decision making in medicine, etc. Mirroring the literature in Social Choice Theory, the literature on Judgment Aggregation formulates conditions on acceptable aggregation rules (i.e. a mapping of individual judgments into a collective judgment). This leads to various impossibility theorems. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Assessing Aggregation Rules The impossibility theorems raise the question which of the conditions have to be given up to arrive at a possibility theorem. There is a literature that examines these conditions and explores which should be given up in real decision situations. We believe that (i) this question cannot be answered in general; it depends on the case in question. (ii) Instead of focusing on the conditions, we should step back and ask what one wants from an aggregation procedure. One might then choose the procedure that is conducive to this purpose, such as tracking the truth or maximizing the utility of the group. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions The Discursive Dilemma Back to our example: A jury with 7 members has to decide whether a defendant should be sent to prison (R). This is the case iﬀ he is guilty (P) and liable (Q). They vote as follows: P Q R Members 1,2,3 Yes Yes Yes Members 4,5 Yes No No Members 6,7 No Yes No Majority Yes Yes No We learn: Propositionwise majority voting leads to an inconsistency on the collective level. Some terminology: P and Q are the premises, R is the conclusion of the argument (P ∧ Q) ↔ R. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions Ways out 1: The Premise-Based Procedure (PBP) Propositionswise majority voting on the premises, then infer the conclusion. Motivating intuition: The reasons for a decision are important. P Q R Members 1,2,3 Yes Yes – Members 4,5 Yes No – Members 6,7 No Yes – Majority Yes Yes Yes Decision: The defendant is sent to prison. Note: This procedure is manipulable. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions Ways out 2: The Conclusion-Based Procedure (CBP) Propositionwise majority voting on the conclusion. Motivating intuition: Jurors make up their minds on the premises privately and then submit their judgment on the conclusion. P Q R Members 1,2,3 – – Yes Members 4,5 – – No Members 6,7 – – No Majority – – No Decision: The defendant is not sent to prison. Note: This procedure is not manipulable. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions Ways out 3: Distance-Based Procedures Avoid dilemmas and ﬁnd an aggregation method that ensures consistency in every step of the aggregation procedure. Motivating intuition: Find some kind of average. How can this idea be made more precise? Four situations (or “judgment sets”): (YES: 1, NO: 0) S1 := (1, 1, 1) , S2 := (1, 0, 0) S3 := (0, 1, 0) , S4 := (0, 0, 0) Identify the situation that has minimal distance to the judgments expressed by the voters. One option: Hamming distance Example: The Hamming distance btw. S1 and S1 is 0, btw. S1 and S2 is 2, btw. S1 and S3 2, and btw. S1 and S4 3. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions Distance-Based Procedures: How It Works The total distances di of all submitted judgments to situation Si can be expressed in terms of the numbers ni of voters for each situation Sj (j = 1, . . . , 4): d1 = 2n2 + 2n3 + 3n4 ; d2 = 2n1 + 2n3 + n4 d3 = 2n1 + 2n2 + n4 ; d4 = 3n1 + n2 + n3 In our example, we have: n1 = 3, n2 = n3 = 2 and n4 = 0. Hence, d1 = 8, d2 = 10, d3 = 10 and d4 = 13. Result: Situation 1 has minimal distance and is selected by this procedure. Hence, the defendant is sent to prison. Note: One gets a ranking: S1 S2 , S3 S4 . Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions Distance-Based Procedures: Another illustration Show: Distance minimization = averaging P Q R Members 1,2,3 1 1 1 Members 4,5 1 0 0 Members 6,7 0 1 0 Average 5/7 5/7 3/7 Distance of the average to S1 : D1 = 2/7 + 2/7 + 4/7 = 8/7 Distance of the average to S2 and S3 : D2 = D3 = 10/7 Distance of the average to S4 : D4 = 13/7 S1 has minimal distance to the average. This insight can be generalized: di = N · Di Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation An Example Discursive Dilemma Ways out 1: The Premise-Based Procedure (PBP) Our Model Ways out 2: The Conclusion-Based Procedure (CBP) Some Results Ways out 3: Distance-Based Procedures Conclusions Which procedure is best? This depends on what our goals are. We may want, for example, that the aggregation procedure . . . avoids dimemmas, maximizes the total utility of the group tracks the truth, or satisﬁes a combination of these goals. Distance-based approaches (a.k.a. “belief fusion”, “belief merging”) satisfy the ﬁrst goal, but how well does it do with regard to the others? Let us focus here on truth-tracking. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Truth tracking We follow the account of Nozick (1981). An aggregation procedure tracks the truth if the following four subjunctive conditionals are true: (i) If S1 were the true state of the world, then S1 would be chosen. (ii) If S2 were the true state of the world, then S2 would be chosen. (iii) If S3 were the true state of the world, then S3 would be chosen. (iv) If S4 were the true state of the world, then S4 would be chosen. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions The Condorcet framework We consider the case of P ∧ Q ↔ R and compare the performance of fusion with the PBP and the CBP studied by Bovens & Rabinowicz (2006). Assumptions: P and Q are logically and probabilistically independent. All N voters are equally competent and independent. The chance that a voter correctly judges the truth or falsity of proposition P (her competence) is p. The same for Q. The prior probability that P is true is q. The same for Q. There are four possible situations: S1 = {P, Q, R} = (1, 1, 1) S2 = {P, ¬Q, ¬R} = (1, 0, 0) S3 = {¬P, Q, ¬R} = (0, 1, 0) S4 = {¬P, ¬Q, ¬R} = (0, 0, 0) Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Truth tracking in the limit In probabilistic terms, truth tracking can be explicated as follows: (i) P(The procedure choses S1 |S1 ) → 1 if N → ∞ (ii) P(The procedure choses S2 |S2 ) → 1 if N → ∞ (iii) P(The procedure choses S3 |S3 ) → 1 if N → ∞ (iv) P(The procedure choses S4 |S4 ) → 1 if N → ∞ Construct a probabilistic model to test these claims. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Our framework We want to calculate the probability of the proposition F : Fusion ranks the right judgment set ﬁrst. Note that 4 P(F ) = P(F |Si ) · P(Si ), i=1 so we have to calculate the prior probabilities P(Si ) and the conditional probabilities P(F |Si ) for i = 1, . . . , 4. The prior probabilities of the situations are (with x := 1 − x): ¯ P(S1 ) = q 2 ; P(S2 ) = P(S3 ) = q¯; P(S4 ) = q 2 q ¯ Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Our framework (cont’d) Let’s assume that S1 is the right judgment set. Then the right judgment set ranks ﬁrst if d1 ≤ min(d2 , d3 , d4 ). The distances di can be expressed in terms of the numbers ni of voters for each judgment set Si (i = 1, . . . , 4): d1 = 2n2 + 2n3 + 3n4 ; d2 = 2n1 + 2n3 + n4 d3 = 2n1 + 2n2 + n4 ; d4 = 3n1 + n2 + n3 We now calculate: N N P(F |S1 ) = p 2n1 (p¯)n2 +n3 p 2n4 C(n1 , . . . , n4 ) p ¯ n1 , . . . , n4 n1 ,...,n4 =0 The sum is constrained: C(n1 , . . . , n4 ) = 1 if (i) 4 ni = N i=1 and (ii) d1 ≤ min(d2 , d3 , d4 ). Otherwise C(n1 , . . . , n4 ) = 0. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Fusion ranks the right judgment set ﬁrst (R) compared with PBP (G), and CBP (B) for N = 3 and q = .5 1 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Same for N = 11 1 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Same for N = 21 1 0.8 0.6 0.4 0.2 0 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Truth tracking All three procedures track the truth in the limit if p > 1/2. Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Right result Let us now calculate the probability of G : Fusion ranks a judgment set with the right result (not necessarily for the right reasons) ﬁrst. In this case, we have S1 true → d1 ≤ min(d2 , d3 , d4 ) S2 true → min(d2 , d3 , d4 ) ≤ d1 S3 true → min(d2 , d3 , d4 ) ≤ d1 S4 true → min(d2 , d3 , d4 ) ≤ d1 With this, we obtain . . . Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Fusion ranks a judgment set with the right result (not nec. for the right reasons) ﬁrst (R) comp. with PBP (G), and CBP (B) for N = 3 and q = .5 1 0.9 0.8 0.7 0.6 0.5 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Same for N = 11 1 0.9 0.8 0.7 0.6 0.5 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Same for N = 31 1 0.9 0.8 0.7 0.6 0.5 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Fusion ranks a judgment set with the right result (not nec. for the right reasons) ﬁrst (R) comp. with PBP (G), and CBP (B) for N = 3 and q = .2 1 0.9 0.8 0.7 0.6 0.5 0.4 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Same for N = 21 1 0.9 0.8 0.7 0.6 0.5 0.4 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Same for N = 51 1 0.9 0.8 0.7 0.6 0.5 0.4 0 0.2 0.4 0.6 0.8 1 Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Understanding the dip We decompose: 4 P(G ) = P(G |Si ) · P(Si ) i=1 4 = [P< (G |Si ) + P> (G |Si )] · P(Si ) i=1 4 = P> (G |S1 ) + P< (G |Si ) · P(Si ) i=1 P< (G |Si ): the probability that fusion selects the right outcome given that the minority votes for the right situation Si P> (G |Si ): the probability that fusion selects the right outcome given that the majority votes for the right situation Si Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model How does fusion compare to PBP and CBP? Some Results Conclusions Understanding the dip (N = 21) 1 0.8 0.6 0.4 0.2 0 0.2 0.4 0.6 0.8 1 Probability that fusion selects the right outcome given that: – the majority votes for the right situation, S1 true (R) – the minority votes for the right situation, S1 true (G) – the minority votes for the right situation, S2 [S3 ] true (B) – the minority votes for the right situation, S4 true (T) Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions The fusion approach does especially well for middling values of the competence p (p ≈ .5). For other values of p, the fusion approach is often in between PBP and CBP (whichever is better in the case at hand). (∨) (∧) Disjunctive case analogous: Pq (F ) = Pq (F ) with ¯ (∨) (∧) q = 1 − q. Similarly, Pq (G ) = Pq (G ). ¯ ¯ Open questions Generalization to more than two premises (⇒ Monte Carlo) Dependencies between voters: social networks Introduce weights for the premises and the conclusion and ﬁx them such that merging does best in terms of truth tracking Introduce reliabilities Can this framework be used to model deliberation? Under which conditions does a group reach consensus? (cf. the Lehrer-Wagner model) Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking Motvation Discursive Dilemma Our Model Some Results Conclusions Thanks for your attention! Stephan Hartmann (with Gabriella Pigozzi) Merging Judgments and the Problem of Truth-Tracking