Response to reviewers by md820pSH


									            Shahar E, Shahar DJ. Causal diagrams and change variables
    Journal of Evaluation in Clinical Practice: Epub ahead of print Sep 12, 2010

An original article that was rejected by the Editor of The European Journal of Epidemiology and
by several other editors. Read the article before reading this document.

We post here a public response to the reviewer of the European Journal of Epidemiology whose
identity is unknown to us (and does not matter, of course). At the end of the document you will
find what other reviewers and editors said on this paper. This is a long, but worth reading,
document (at least for those who have genuine interest in Science and in the idol called “peer

Reviewer: “The authors consider different ways in which one might represent a change variable
on a causal diagram.”

Response: That’s false. First, causal structures are not "ways in which one might represent...".
They are not a menu to choose from to one's liking. Causal diagrams are theories about the
existing causal structure. Second, we asked: what is the true structure? And we gave an
answer—after a lengthy deliberation. We did not ask how one might represent… Third, that’s
not only a false representation of our paper, but also a wrong question to ask.

Reviewer: “Some of the discussion is of interest but the manuscript in its present form is subject
to numerous problems.”

Response: That’s the typical writing style of a biased reviewer. A manuscript may contain minor
problems and major problems, and a reviewer should not mix them to create the impression of
“numerous problems”. Minor problems are not a reason for rejection. Only major problems
could substantiate a recommendation to reject a manuscript.

So let’s investigate these “numerous problems”. We have re-organized the reviewer’s 12
remarks into major comments (points that argue against the main thesis) and minor comments
(semantics and minor mistakes), and renumbered them. Of the 12 points, nine are minor; we
respond to them first.

Minor comment #1: “p. 3-4: It seems somewhat inappropriate to cite comments from prior
reviewers here.”

Really? Previous reviewers expressed their scientific opinions in a scientific exchange. Why not
cite those opinions, especially since their authors safely hide behind the anonymity of peer
review? If anything, the reviewer’s inclination to conceal scientific views from the public eye is
inappropriate. Our article was belittled by these and other reviewers and—funny enough—the
reviewers unknowingly belittled each other as well... The present reviewer is no different, with
one exception. He (assuming “he”) did not state which causal structure he settled in his mind
(fearing he would be quoted, too?). Is ΔX a cause of X2 or vice versa? Does the reviewer think
that the question itself should also be suppressed?

Minor comment #2: “p. 4, three lines from the bottom: What do the authors mean by a
"mandatory effect"?”

Response: Mathematical-deterministic. We changed it to “an effect.”

Minor comment #3: “p. 5, 2 lines from bottom: The expression, "these effects" should be
clarified i.e. are direct effects or total effects in view?”

Response: “These effects” refer to arrows 1, 2 and 3, and they are shown in the diagram. There
is no ambiguity here.

Minor comment #4: “p. 8: The discussion on the bottom of p. 8 is unnecessary. If the relation
Y=6X describes a functional relationship then a change in X by 1 unit changes Y by 6 units.
There is nothing more that need be said here.”

Response: Describing how Y changes when X changes is an associational claim—not a causal
claim! The causal relations between X, 2X, 3X, and Y should be described in a diagram. Does
the reviewer suggest that the question we asked should be suppressed?

Minor comment #5: “p. 11-12: The authors have made a technical error here. They claim that
"According to the causal structures in Figure 2, V and dX should be associated, not only
marginally, but also after conditioning on X1 and X2." They go on to show 0 correlation and thus
argue that the arrows must be absent. However, their claim "According to the causal structures
in Figure 2, V and dX should be associated, not only marginally, but also after conditioning on
X1 and X2" only necessarily holds under an assumption of faithfulness (see Pearl, 2009,
chapter 2, 2nd edition). It is well known that faithfulness is generally violated when there are
deterministic relations, as is the case in the present context. The argument on these pages

Response: Yes. The reviewer is correct. This claim is only valid if the faithfulness condition
holds. We deleted this point. It is just one of many arguments; our main thesis does not rise or
fall on this point.

Minor comment #6: “p. 11/14: The authors repeatedly call statements involving causal
relationships with dX a "metaphysical assertion." I believe the authors use this expression to
suggest that such statements should be dismissed on this account. I am not sure what to make
of this position. Most philosophers would agree that ALL statements about causation are
metaphysical assertions.”

Response: By metaphysical assertions, we meant untestable causal claims. We have changed
the wording. There are different kinds of statements about causation, and not all of them are
metaphysical. We haven’t tested that “most philosophers would agree” on this point, nor do we
believe that it was tested by the reviewer.

Minor comment #7: “p. 14: The point about considering the change measured on a difference
scale versus a ratio scale or some other scale is a good one. However, this is arguably simply a
point about the importance of considering which scale is relevant rather than an argument
against using change measures per se.”

Response: The relevant scale is the causal one. We propose that none of these terms is a cause
of X2!. Each of them, along with X1, perfectly predicts X2 (and we explained why in Figure 5).
None of these variables causes X2, and an infinite number of different complementary variables
can be proposed. Is there any way by which one can decide which of an infinite number of
derived variables effects X2, and which are derived from (causes by) X2? The reviewer did not
state which variable should be chosen even in one particular case—simply avoiding a discussion
of causation. Again, the reviewer shows lack of understanding of (or inability to grasp) the main
question: What is the TRUE causal structure in this world? Unfortunately, no writing can force a
mind to confront a question it chooses to ignore.

Minor comment #8: “p. 17: The authors suggest that it is always assumed that arrows go from
the true variable to the measured variable. This is not the case. Some of the social science
and econometric literature effectively consider the arrows going in the other direction.”

Response: First, this point is tangential to our main thesis, so even if we are wrong it makes little
difference. Second, the reviewer is wrong. The variable X and the variable X* exist at distinct
time points, and X* always follows X on the time scale. (By definition, you measure a value that

has already been realized, not a value that will be realized). Therefore, X* X never exists. Of
course, after measuring X at one point in time, X* can then affect future values of X:
X1X1*X2. Note that measurement has many steps, the last of which is the variable one fits in
the analysis. If X* is defined as that last variable, its values realize after X has already realized,
and therefore, it can never be the cause of the X of interest.

Minor comment #9: “p. 19: The authors note that sometimes using change variables is not
always wrong. It would be good (and more useful to readers) to describe, in a more balanced
manner than the present commentary, when change variables are of use.”

Response: At the end of the article we described the two cases where change variables are of
use. How are we supposed to write this article in a more balanced way? We argued that ΔX is
not a cause of X2, and is not the cause of any other variable, either. Then, we mentioned that the
computation of ΔX can be used to test other causal theories. What is “balanced” in the context of
critical examination of truth and falsehood? If an argument we have made is false, show us
where it is false (e.g., comment #5). If you have no more arguments to offer, say nothing and
don’t hide your prejudice behind “balanced”.

The reviewer has made only three major points (buried in a list of “numerous problems”). Two
of them repeat the same theme.

Major comment #1: “p. 5, line 7: The authors suggest that a change variable is never of intrinsic
interest (does not fall in what they call category 1). Would not "change in weight" be of intrinsic
interest to individuals evaluating different weight loss programs?”

Major comment #2: “p. 9-10: I do not find the arguments here convincing. Whether or not it is
appropriate to delete a particular arrow depends on the question and the underlying structures.
Suppose that a weight loss program received payment (Y=1) if either the participant lost 10kg or
if the final weight of the participant was less than 100kg. We would then have arrows 2 and 3
but no arrow 1. What is important is that at least one of the arrows 1, 2 or 3 is eliminated
because, as the authors point out, having all three is redundant.”

Response: The main claim in both comments is that arrow 3 could exist (ΔX Y). We gave
numerous arguments as to why we thought it is the only arrow that should be deleted (and does
not exist). To which the author responds “I do not find the arguments here convincing.”—
without stating what mistakes we have made in our arguments. This kind of reaction reminds me
of the quote “These are my ideas upon which I base my data” (Michael Marmot).

Rather than responding to our arguments, the reviewer gave examples of his own to convince us
(and himself) that arrow 3 exists. 1) “Would not ‘change in weight’ be of intrinsic interest to
individuals evaluating different weight loss programs?”; 2) “Suppose that a weight loss program
received payment (Y=1) if either the participant lost 10kg or if the final weight of the participant
was less than 100kg. We would then have arrows 2 and 3 but no arrow 1.”

Well, the reviewer surely missed the distinction we have made between ΔX and ΔX *. The
reviewer’s examples show that the measured change, ΔX*, can have an effect. Had he read our
paper carefully through the end, the reviewer would have noticed that we have addressed this
topic. Here is what we wrote. It is shown in bold print to make sure that the editor and the
reviewer don’t miss it again.

“We think that ΔX, the true change, cannot cause anything, but ΔX*, the computed change, may be a

cause—for example, via behavioral pathways. Knowledge of the computed change, ΔX*, can affect

behavior and its consequences, just like knowledge of any X*. For instance, people who are told that

they gained 5kg, or that they weigh 100kg, might choose a different diet, regardless of whether they

indeed gained 5kg or weigh 100kg. Other derivations could affect diet as well, for instance telling

people that their current weight is 1.1 times their previous weight (X2*/X1*) or even telling them the

ratio of their current weight to their height squared (X2*/ H2*2). Human beings, perhaps different

from most other forms of life, can learn how to allow a computed derivation to affect their behavior.”

When choosing a weight loss program, the results do not tell us the value of the actual (true)
change (ΔX), only the calculated change (ΔX*)!!! Is that clear, or should we say it again? Thus,
the arrow the reviewer is describing is the arrow from ΔX* to the choice of weight loss program:
ΔX*(calculated change)weight loss program. (Note that perceived, calculated change could
affect the outcome, even if it has zero correlation with actual change.) Likewise, in the second
example: ΔX*(calculated change) payment. Since ΔX (true change) is not a cause of any
variable (including ΔX*), these examples do not contradict our argument that arrow 3 does not
exist. Therefore, the reviewer’s example-based claim that ΔXY exists is baseless. (“Thus, the
argument fails”—to use his eloquent jargon.) We note again that the reviewer chose to not
answer our most fundamental question: which is the true causal structure in this world: ΔXX2
or X2ΔX?

Major comment #3: “p. 6-7: Upon a first reading, my immediate reaction to the discussion on
these page was that the problem described here is resolved from Figure 1 DAGs A and B by
removing arrow 3 or by removing arrow 2 from the diagram. The effects are then identified. The
authors consider this a few pages later but it seems important to raise here. In my view there

would be two ways to conceptualize what is going on: either Y=f(X1, dX) or Y=f(X1, X2). The
two are functionally equivalently. How one approaches this will depend on the substantive
question of interest.”

The first part of this comment is the same baseless claim (arrow 3 sometimes exists). In the
second part, the reviewer replaced the question we asked about the true causal structure with his
erroneous idea of representation. We repeat the point to those who choose to turn a deaf ear:
there is only one substantive question of interest—what is the true causal structure in this world?
If you choose to represent the truth with an arrow that does not exist, you are wrong. Science is
not about “your representation” of causal reality; it is about causal reality itself. “Functionally
equivalent” is not “causally equivalent”, just as “associated with” is not equivalent to “a cause
of”. It may turn out that modeling possession of a cigarette lighter is functionally equivalent to
modeling smoking status, but that does not mean that possession of a cigarette lighter causes
lung cancer. We are puzzled why the reviewer is unable to grasp a basic causal idea. Many
laypersons should have no problem understanding the distinction.


So, what do we have so far? One reviewer who says that there is no doubt: ΔX definitely does
not affect X2 (quoted in the paper). Another reviewer who confidently says that there is no
doubt: ΔX definitely affect X2 (quoted in the paper). And another reviewer who says nothing on
this question, but has a confident answer to another question. There is no doubt: ΔX can
definitely affect Y directly (but is actually saying that ΔX * can definitely affect Y, as we have
said as well.)

Now, what should a thoughtful editor do when he reads such an indigestible set of key
statements about an unusual, thought-provoking article? We think he or she should keep in mind
the following two quotes:
"All truth passes through three stages: First, it is ridiculed; Second, it is violently opposed; and Third, it is
accepted as self-evident."-- Arthur Schopenhauer

"If you restrict the journal to publishing only what pleases the referees, you end up publishing what is
popular, and while it does make everyone feel more comfortable, you are guaranteed to miss the
occasional breakthrough." -- A. Dessler

What did other editors and other reviewers say about this paper?

We have a nice collection of text written by editors and reviewers who have rejected this
manuscript. (The list of journals, associate editors and editors-in-chief is saved in my files.) A
critical review of that collection would make a nice masters thesis on the sociology of peer
review (and on the competence of some reviewers and some editors). But that’s too much for us
to pursue. We note below three key points:

   1. Almost every editor of a biostatistics journal found an easy way out: don’t tell me about
      causation; my journal is just interested in “statistics”. They conveniently forgot that most
      of their statistics is about causation. Their readers estimate coefficients involving change
      variables, and write methodological papers on change variables. Time to expand your
      horizons, biostatistics editors! And by the way, DAGs are math, too.

   2. About half a dozen reviewers of this paper shared only one theme: this is a useless paper
      that should not be published. As for the main reason, they had close to zero agreement,
      often disagreeing with each other on the true causal structure. Two reviewers are quoted
      in the paper: one confidently said that our conclusion was wrong; the other confidently
      said that our conclusion was right, but trivial (not realizing the implications). The
      reviewer we quoted above claimed ΔX Y, but confused ΔX with ΔX *. Two reviewers
      knew some statistics but refused to think about causation; therefore, their reviews were
      irrelevant. Another reviewer spent most of her (assuming “her”, for a change) critique on
      stating her beliefs that no single structure fits all situations, and gave examples of ΔX
      Y, which were actually examples of ΔX*Y. Here is a typical claim of that
      reviewer, which is similar to the main claim of the reviewer of the European Journal of
      Epidemiology. It also contains the very same mistake.

   “we may imagine that depression results from a recognition of cognitive losses, thus, an
   individual who perceived that his faculties are declining will become depressed at this loss.”

       First, to “imagine” the existence of a causal arrow is not evidence for its existence.
   Second, perception of change is ΔX*, not ΔX (and you can imagine that you will get
   depressed even if your perception is completely false.) Third, you perceive a loss of
   cognition by comparing your ability to think now (X2*) with your ability to think in the past
   (X1*): X1*ΔX*X2*. Fourth, since ΔX does not cause ΔX*, your example does not show
   that true cognitive loss (ΔX) can cause depression (Y). Fifth, as we have shown, you cannot
   test that ΔX (true cognitive loss) has caused Y (depression), so your claim is baseless.

   3. We have spent many months on every aspect of the question, and reaching the conclusion
      was a lengthy process. Our starting point was not very far from that of those reviewers,
      where gut feelings and preconceived ideas dominate the mind. The paper clearly shows
      that we have examined the topic thoroughly, and lined up many arguments. We seriously
      doubt that any reviewer had even conceived the central question before reading the
      article, yet they all quickly displayed their prejudice (or the fruits of a couple hours of
      thinking). They produced a series of incompatible answers to the key question we asked,
      not knowing how bad their confident (arrogant?) claims sound when they are lined up,
      one after the other. They argued poorly, belittled, or simply ignored the arguments that
      created problems for their prejudice. Reading the entire collection of these reviews is a
      lot more illuminating than reading the two quotes we added to the introduction. If it were
      not the serious business of science, some people would find them funny.

We conclude: X1ΔX X2 is always the causal structure, and ΔX causes nothing—until
someone argues differently in a formal publication (with or without “peer review”). That

publication should include a point-by-point refutation of all of the arguments we presented. No
“picking and choosing” allowed. We suspect that we will be waiting forever. After all, writing
and publishing a rebuttal is much more difficult than writing the reviews we have read, or typing
“I regret to inform you”.

Think differently? Go ahead and prove us wrong—in the public domain.


To top