# Merging Judgments and the Problem of Truth-Tracking by sdfsb346f

VIEWS: 6 PAGES: 30

• pg 1
```									     Merging Judgments and the Problem of
Truth-Tracking

Stephan Hartmann (with Gabriella Pigozzi)

Center for Logic and Philosophy of Science
Tilburg University, The Netherlands

progic07: 3rd Workshop on Combining Probability and Logic
University of Kent, UK, 6 September 2007

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Judgment Aggregation
A committee has to make a decision on logically
interconnected propositions.
Example: A jury has to decide whether a defendant should be
sent to prison (R). This is the case iﬀ he is guilty (P) and
liable (Q). Each jury member submits a judgment on all three
propositions (P, Q and R), respecting their logical inter-
relations. The task is then to obtain a collective judgment.
Related problems: multi agent systems, sensors, decision
making in medicine, etc.
Mirroring the literature in Social Choice Theory, the literature
on Judgment Aggregation formulates conditions on acceptable
aggregation rules (i.e. a mapping of individual judgments into
a collective judgment).
This leads to various impossibility theorems.
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Assessing Aggregation Rules

The impossibility theorems raise the question which of the
conditions have to be given up to arrive at a possibility
theorem.
There is a literature that examines these conditions and
explores which should be given up in real decision situations.
We believe that (i) this question cannot be answered in
general; it depends on the case in question. (ii) Instead of
focusing on the conditions, we should step back and ask what
one wants from an aggregation procedure. One might then
choose the procedure that is conducive to this purpose, such
as tracking the truth or maximizing the utility of the group.

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

The Discursive Dilemma
Back to our example: A jury with 7 members has to decide
whether a defendant should be sent to prison (R). This is the
case iﬀ he is guilty (P) and liable (Q). They vote as follows:

P        Q         R
Members 1,2,3            Yes      Yes       Yes
Members 4,5              Yes      No        No
Members 6,7              No       Yes       No
Majority                 Yes      Yes       No

We learn: Propositionwise majority voting leads to an
inconsistency on the collective level.
Some terminology: P and Q are the premises, R is the
conclusion of the argument (P ∧ Q) ↔ R.
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

Ways out 1: The Premise-Based Procedure (PBP)

Propositionswise majority voting on the premises, then infer
the conclusion.
Motivating intuition: The reasons for a decision are important.

P        Q         R
Members 1,2,3            Yes      Yes        –
Members 4,5              Yes      No         –
Members 6,7              No       Yes        –
Majority                 Yes      Yes       Yes

Decision: The defendant is sent to prison.
Note: This procedure is manipulable.

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

Ways out 2: The Conclusion-Based Procedure (CBP)

Propositionwise majority voting on the conclusion.
Motivating intuition: Jurors make up their minds on the
premises privately and then submit their judgment on the
conclusion.

P      Q       R
Members 1,2,3            –      –      Yes
Members 4,5              –      –      No
Members 6,7              –      –      No
Majority                 –      –      No

Decision: The defendant is not sent to prison.
Note: This procedure is not manipulable.
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

Ways out 3: Distance-Based Procedures
Avoid dilemmas and ﬁnd an aggregation method that ensures
consistency in every step of the aggregation procedure.
Motivating intuition: Find some kind of average.
How can this idea be made more precise?
Four situations (or “judgment sets”): (YES: 1, NO: 0)
S1 := (1, 1, 1)           ,       S2 := (1, 0, 0)
S3 := (0, 1, 0)           ,       S4 := (0, 0, 0)
Identify the situation that has minimal distance to the
judgments expressed by the voters.
One option: Hamming distance
Example: The Hamming distance btw. S1 and S1 is 0, btw.
S1 and S2 is 2, btw. S1 and S3 2, and btw. S1 and S4 3.
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

Distance-Based Procedures: How It Works

The total distances di of all submitted judgments to situation
Si can be expressed in terms of the numbers ni of voters for
each situation Sj (j = 1, . . . , 4):

d1 = 2n2 + 2n3 + 3n4                   ;       d2 = 2n1 + 2n3 + n4
d3 = 2n1 + 2n2 + n4                  ;       d4 = 3n1 + n2 + n3

In our example, we have: n1 = 3, n2 = n3 = 2 and n4 = 0.
Hence, d1 = 8, d2 = 10, d3 = 10 and d4 = 13.
Result: Situation 1 has minimal distance and is selected by
this procedure. Hence, the defendant is sent to prison.
Note: One gets a ranking: S1                      S2 , S3     S4 .

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

Distance-Based Procedures: Another illustration

Show: Distance minimization = averaging
P        Q         R
Members 1,2,3             1        1         1
Members 4,5               1        0         0
Members 6,7               0        1         0
Average                  5/7      5/7       3/7
Distance of the average to S1 : D1 = 2/7 + 2/7 + 4/7 = 8/7
Distance of the average to S2 and S3 : D2 = D3 = 10/7
Distance of the average to S4 : D4 = 13/7
S1 has minimal distance to the average.
This insight can be generalized: di = N · Di

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
An Example
Discursive Dilemma
Ways out 1: The Premise-Based Procedure (PBP)
Our Model
Ways out 2: The Conclusion-Based Procedure (CBP)
Some Results
Ways out 3: Distance-Based Procedures
Conclusions

Which procedure is best?

This depends on what our goals are. We may want, for example,
that the aggregation procedure . . .
avoids dimemmas,
maximizes the total utility of the group
tracks the truth, or
satisﬁes a combination of these goals.
Distance-based approaches (a.k.a. “belief fusion”, “belief
merging”) satisfy the ﬁrst goal, but how well does it do with
regard to the others? Let us focus here on truth-tracking.

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Truth tracking

We follow the account of Nozick (1981).
An aggregation procedure tracks the truth if the following
four subjunctive conditionals are true:
(i)   If   S1   were   the   true   state    of   the   world,   then   S1   would   be   chosen.
(ii)   If   S2   were   the   true   state    of   the   world,   then   S2   would   be   chosen.
(iii)   If   S3   were   the   true   state    of   the   world,   then   S3   would   be   chosen.
(iv)    If   S4   were   the   true   state    of   the   world,   then   S4   would   be   chosen.

Stephan Hartmann (with Gabriella Pigozzi)            Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

The Condorcet framework
We consider the case of P ∧ Q ↔ R and compare the performance
of fusion with the PBP and the CBP studied by Bovens &
Rabinowicz (2006). Assumptions:
P and Q are logically and probabilistically independent.
All N voters are equally competent and independent.
The chance that a voter correctly judges the truth or falsity of
proposition P (her competence) is p. The same for Q.
The prior probability that P is true is q. The same for Q.
There are four possible situations:
S1   = {P, Q, R} = (1, 1, 1)
S2   = {P, ¬Q, ¬R} = (1, 0, 0)
S3   = {¬P, Q, ¬R} = (0, 1, 0)
S4   = {¬P, ¬Q, ¬R} = (0, 0, 0)

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Truth tracking in the limit

In probabilistic terms, truth tracking can be explicated as follows:
(i) P(The procedure choses S1 |S1 ) → 1 if N → ∞
(ii) P(The procedure choses S2 |S2 ) → 1 if N → ∞
(iii) P(The procedure choses S3 |S3 ) → 1 if N → ∞
(iv) P(The procedure choses S4 |S4 ) → 1 if N → ∞
Construct a probabilistic model to test these claims.

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Our framework

We want to calculate the probability of the proposition F :
Fusion ranks the right judgment set ﬁrst.
Note that
4
P(F ) =            P(F |Si ) · P(Si ),
i=1

so we have to calculate the prior probabilities P(Si ) and the
conditional probabilities P(F |Si ) for i = 1, . . . , 4.
The prior probabilities of the situations are (with x := 1 − x):
¯
P(S1 ) = q 2 ; P(S2 ) = P(S3 ) = q¯; P(S4 ) = q 2
q           ¯

Stephan Hartmann (with Gabriella Pigozzi)    Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Our framework (cont’d)
Let’s assume that S1 is the right judgment set. Then the right
judgment set ranks ﬁrst if d1 ≤ min(d2 , d3 , d4 ).
The distances di can be expressed in terms of the numbers ni
of voters for each judgment set Si (i = 1, . . . , 4):
d1 = 2n2 + 2n3 + 3n4                    ;       d2 = 2n1 + 2n3 + n4
d3 = 2n1 + 2n2 + n4                 ;       d4 = 3n1 + n2 + n3
We now calculate:
N
N
P(F |S1 ) =                                       p 2n1 (p¯)n2 +n3 p 2n4 C(n1 , . . . , n4 )
p        ¯
n1 , . . . , n4
n1 ,...,n4 =0

The sum is constrained: C(n1 , . . . , n4 ) = 1 if (i) 4 ni = N
i=1
and (ii) d1 ≤ min(d2 , d3 , d4 ). Otherwise C(n1 , . . . , n4 ) = 0.
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Fusion ranks the right judgment set ﬁrst (R) compared
with PBP (G), and CBP (B) for N = 3 and q = .5

1

0.8

0.6

0.4

0.2

0
0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Same for N = 11

1

0.8

0.6

0.4

0.2

0
0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Same for N = 21

1

0.8

0.6

0.4

0.2

0
0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Truth tracking

All three procedures track the truth in the limit if p > 1/2.

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model     How does fusion compare to PBP and CBP?
Some Results
Conclusions

Right result

Let us now calculate the probability of G : Fusion ranks a
judgment set with the right result (not necessarily for the
right reasons) ﬁrst.
In this case, we have

S1 true         →        d1 ≤ min(d2 , d3 , d4 )
S2 true         →        min(d2 , d3 , d4 ) ≤ d1
S3 true         →        min(d2 , d3 , d4 ) ≤ d1
S4 true         →        min(d2 , d3 , d4 ) ≤ d1

With this, we obtain . . .

Stephan Hartmann (with Gabriella Pigozzi)    Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Fusion ranks a judgment set with the right result (not nec.
for the right reasons) ﬁrst (R) comp. with PBP (G), and
CBP (B) for N = 3 and q = .5
1

0.9

0.8

0.7

0.6

0.5
0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Same for N = 11

1

0.9

0.8

0.7

0.6

0.5
0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Same for N = 31

1

0.9

0.8

0.7

0.6

0.5
0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Fusion ranks a judgment set with the right result (not nec.
for the right reasons) ﬁrst (R) comp. with PBP (G), and
CBP (B) for N = 3 and q = .2
1
0.9
0.8
0.7
0.6
0.5
0.4

0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Same for N = 21

1
0.9
0.8
0.7
0.6
0.5
0.4

0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Same for N = 51

1
0.9
0.8
0.7
0.6
0.5
0.4

0         0.2           0.4       0.6          0.8            1

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model    How does fusion compare to PBP and CBP?
Some Results
Conclusions

Understanding the dip
We decompose:
4
P(G ) =                   P(G |Si ) · P(Si )
i=1
4
=            [P< (G |Si ) + P> (G |Si )] · P(Si )
i=1
4
= P> (G |S1 ) +                 P< (G |Si ) · P(Si )
i=1

P< (G |Si ): the probability that fusion selects the right outcome
given that the minority votes for the right situation Si
P> (G |Si ): the probability that fusion selects the right outcome
given that the majority votes for the right situation Si
Stephan Hartmann (with Gabriella Pigozzi)    Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model     How does fusion compare to PBP and CBP?
Some Results
Conclusions

Understanding the dip (N = 21)
1

0.8

0.6

0.4

0.2

0      0.2       0.4     0.6       0.8        1

Probability that fusion selects the right outcome given that:
– the majority votes for the right situation, S1 true (R)
– the minority votes for the right situation, S1 true (G)
– the minority votes for the right situation, S2 [S3 ] true (B)
– the minority votes for the right situation, S4 true (T)
Stephan Hartmann (with Gabriella Pigozzi)     Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

The fusion approach does especially well for middling values
of the competence p (p ≈ .5).
For other values of p, the fusion approach is often in between
PBP and CBP (whichever is better in the case at hand).
(∨)             (∧)
Disjunctive case analogous: Pq (F ) = Pq (F ) with
¯
(∨)            (∧)
q = 1 − q. Similarly, Pq (G ) = Pq (G ).
¯                                ¯
Open questions
Generalization to more than two premises (⇒ Monte Carlo)
Dependencies between voters: social networks
Introduce weights for the premises and the conclusion and ﬁx
them such that merging does best in terms of truth tracking
Introduce reliabilities
Can this framework be used to model deliberation? Under
which conditions does a group reach consensus? (cf. the
Lehrer-Wagner model)
Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking
Motvation
Discursive Dilemma
Our Model
Some Results
Conclusions

Stephan Hartmann (with Gabriella Pigozzi)   Merging Judgments and the Problem of Truth-Tracking

```
To top