Originally Posted by purveyors of woo
Absence of evidence is not evidence of absence.
This assertion has long puzzled me, since its falsehood seems so obvious. Nevertheless, we hear it all the time:
"No evidence that homeopathic treatments are more effective than placebo? That's okay, absence of evidence is not evidence of absence! No evidence for alien abductions...?"
Most reasonable people cringe when they hear statements like these... but they nevertheless concede the point that absence of evidence is notindeed, cannot beevidence of absence. This thread is about why this is nonsense.
The statement above really has two interpretations, a weak version and a strong version. The weak version is trivially true, and the strong version is simply wrong. The reason that so many reasonable people make the above concession is that they hear someone make the strong version of the statement, but they hear it as the weak version, and then in intending to concede the weak version, they unintentionally concede the strong version. The weak version goes like this: absence of evidence is not proof of absence. Notice the subtle but important difference here: in this case we are concerned with establishing proof of absence. It is notoriously difficult to prove anything, regardless of the state of the evidence, which is what makes this version of the statement weak.
The strong version is strong because it is concerned with evidence rather than proof: "absence of evidence of not evidence of absence." The strong version is wrong because it defies the basic axioms of probability theory. To demonstrate this, we will view probability theory through the lens of Bayes' Rule, which is a normative and deductively valid interpretation of probability. Although Bayes' Rule is at its heart a mathematical theorem, it can be perfectly well understood on an intuitive level through a simple example. I will not be the first person by far to bring Bayes' Rule to bear on this annoyingly persistent misbelief, but hopefully my explanation will serve as a clear first introduction. For a superb introduction to Bayesian reasoning in general, see the link above.
Let's say that I have misplaced my cell phone. Consider two hypotheses concerning where my cell phone is. The first hypothesis, H, represents the hypothesis that my cell phone is somewhere in my bedroom, and is true with some probability P(H). The competing hypothesis, that my cell phone is anywhere but in my bedroom (i.e., that H is false), is true with probability P(~H) or 1  P(H). Note that these are exhaustive and exclusive. In order to gather evidence for and against these two hypotheses, I decide to call my cell phone from my house phone and see if there is a ringing coming from my bedroom. We will call this evidence E, and it will obtain with probability P(E) or not obtain with probability P(~E).
I call my cell phone from my house phone. No ring comes from my bedroom. The evidence E did not obtain. What does this mean for our hypotheses H and ~H? We are ultimately interested in two conditional probabilities: the probability that my cell phone is in my bedroom given that no ring came from the bedroom, P(H~E), and the probability that my cell phone is not in my bedroom given that no ring came from my bedroom, P(~H~E).
(From this point on it is crucial to recognize that in this format for representing conditional probabilities, the events following the  symbol are assumed to have obtained. What we are considering is the probability of a certain event conditional on a different event being true. Keeping this format in mind will allow me to save a lot of space in the following paragraphs and will save you a lot of confusion.)
These two probabilities of ultimate interest are in turn dependent on the two conditional probabilities: P(~EH), the probability that no ring comes from my bedroom given that my cell phone is in my bedroom, and P(~E~H), the probability that no ring comes from my bedroom given that my cell phone is not in my bedroom. That these latter two probabilities make a difference makes sense, because it's basically saying that the reliability of our particular evidence has an important bearing on what kinds of conclusions we can make about the hypotheses. If my cell phone has a faulty speaker and only rings about half of the times that it's called, then failing to hear a ring coming from the bedroom will be less convincing evidence against hypothesis H. On the other hand, if my cell phone is in perfect condition and has never been observed not to ring when called, this will be much more damning evidence against H.
Bayes' Rule precisely specifies how to combine these probabilities in order to get the exact values for P(H~E) and P(~H~E), but we don't need to be precise for the purposes of this example. We need only think about the matter intuitively. We saw above that as the reliability of my cell phone's speaker increases, it becomes stronger and stronger evidence against H in the event that we fail to hear its ring. In other words, as P(~EH) decreases, P(H~E) increases and P(~H~E) therefore decreases, since the two hypotheses are exclusive and exhaustive. On the other hand, higher values of P(~E~H) tip the odds in favor of P(~H~E) and against P(H~E).
What we need to do then is compare P(~EH) and P(~E~H). I should point out here that we are making simplifying assumptions in order to keep things intuitive, but we are not fundamentally altering the nature of the problem. Recall that P(~EH) represents the probability of not hearing the ring coming from my bedroom given that the cell phone is actually in my bedroom, while P(~E~H) represents the probability of not hearing the cell phone's ring given that it is not in my bedroom. There are various reasons to believe that P(~EH) is not terribly low. The phone's speaker may be unreliable, the battery may have run out, I could have poor hearing, etc. But it should be obvious that P(~E~H) is incredibly high, almost 1. If the cell phone is not actually in my bedroom, the odds of hearing a cell phone ring which sounds exactly like mine coming from my bedroom at the exact instant that I happen to be calling it are impossibly low, whereas it is a near certainty that I will not hear such a ring. No matter how many reasons we might dream up that would increase P(~EH), it will never be as high as P(~E~H).
Thus, failing to hear the ring coming from my bedroom will always decrease the probability that H is true and increase the probability that H is false. In other words, absence of evidence for my cell phone being in my bedroom is always evidence that my cell phone is not in my bedroom. This is true for any set of hypotheses where the correlation between E and H is different from 0, in other words, when evidence is actually evidence. Therefore, absence of evidence is evidence of absence.


Bookmarks