Thread: Probabilities, Frequencies, and Base Rate Neglect

Consider the following hypothetical situation involving a medical diagnosis:

• A given disease (let's call it "belief in intelligent design"... just kidding!) has a prevalence rate of 0.1% - meaning that 0.1% of the people in your country have this disease.
• Your doctor decides to test you for this disease. He obtains a blood sample from you and runs a particular test on it; this test has a false positive rate of 5%, which is to say that if a person does not have this disease, 5% of the time they will test positive for the disease anyway. The "true positive" rate is 100% - if you have the disease, you will test positive.
• Your test comes up positive.
• The question: what is the probability that you actually have the disease, assuming you know nothing about the signs and symptoms?

Spoiler for the answer:

The answer, according to Bayes' rule, is 1.96%. The most common answer given is 95%.

We don't find probability calculations to be very intuitive. We simply have not evolved to handle probability information. What we have evolved to handle is frequency information. Frequencies are the format of information as it is naturally present in our environment, whereas probabilities are not directly available to our perception. Consider again the medical diagnosis scenario above, which only 12% of participants answered correctly when it was first asked by Cascells, Schoenberger and Graboys in 1978. The answer is most easily explained by imagining that there are 1000 people, of whom 51 will test positive (50 false positives and one true positive) and of whom one actually has the disease (prevalence is 1/1000). See how much easier it is to understand when we express it using frequencies?

When we are forced to use probability information, we make mistakes. In this medical diagnosis scenario, most people fail to take into account that even though an average of about 51 people out of every 1000 administered the test will test positive, this doesn't change the fact that only 1 out of every 1000 people actually has the disease. In other words, they neglect the base rate (expressed as prevalence in this case) of 0.1%.

Luckily, probability statistics can often be equally well-expressed using frequencies, and they will be better understood this way. Take advantage of this fact.

Let's go back and tackle the medical diagnosis problem again, this time using frequencies rather than probabilities. For consistency's sake, let's use a sample of 1000.

• First we determine how many people out of 1000 are going to have this disease; with a prevalence (i.e., base rate) of 0.1%, this is 1/1000.
• Now we determine how many people out of 1000 are going to receive positive results from the test; this is going to be the sum of the numbers of false positives and true positives.
  • Since the rate of false positives is 5% and there will be 999 healthy people, the number of false positives will be about 50.
  • We have already determined that the number of true positives will be one.
  • One true positive plus 50 false positives equals 51 positive results.
• Since 51/1000 people will receive positive results while only 1/1000 people actually have the disease, the probability of any given person who tested positive actually having the disease is 1/51, which is close to 1/50, which is 2% (1/51 = 1.96%, as we saw earlier).

Congratulations - you just calculated Bayes' theorem in your head!

Here is a different problem which follows a similar format:

• In predicting terrorist attacks for a given day, your government has a false positive rate of 20% (for any given day on which there is not a terrorist attack, there is a 20% chance that the government predicted that there would be one) and a true positive rate of 40% (if a terrorist attack does occur, there is a 40% chance that the government predicted it for that day).
• On any given day, the probability that there will be a terrorist attack is 5% (hint: this is the base rate).
• The government has predicted that there will be a terrorist attack today.
• The question: What is the probability that there will be a terrorist attack today?

How did you do this time around?

What are some of the implications of our inability to handle probability information?

What are some other applications of the technique that you just learned for handling probabilities by converting them to frequencies?

This isn't really directly connected, but perhaps inability to handle probability statistics has something to do with gambling addictions. Yes, taking the risk gives an adrenaline rush, but why? Maybe this mechanic, inability to handle probability intuitively is related to the common fear of the unknown.

...reading that last sentence, I'm thinking that being inept at probability is an evolutionary advantage. It makes us more likely to think an action through.

wooo..hold up. your first example has me seriously confused. maybe I should just go to bed . What's confused me is the percentage you gave of the positive error. Why would that be wrong?

you just told me 95% of the people who test positive have this disease. That there is only a 5% of error.

Given that percentage, if only 1/1000 people actually have the disease, wouldn't only 1.05 people test positive? Or to make it a whole number of human beings, only 21 people out of 20,000 will test positive. With a 5% error, meaning 20/20000, or 1/1000 actually are sick?

I mean..its one thing to talk about the percentage of people who have the disease. It's one thing to ask what is your percentage of having the disease, or your chances of testing positive. Which is what, like (.105%)?

Its a whole different question that once you test positive, that your chances jump up to 95%. Regardless of the population, because its simply a 5% test error, having nothing to do with how many people actually get sick???

at least that's how I understood the problem. I would still answer 95%, because I am assuming the error of the test is technical, having nothing to do with how many test positive in the total population (.105%). But rather, how many who test positive who actually are............which you told me.......95% o_o .....or .1%

am I missing something?

seriously

you can yell at me

I haven't taken a math class in years

otherwise I understand how problematic percentages can be. but its the wording that you have to be careful of. I mean, I understand if 20% of the population has a disease, it doesn't mean that is also your chances.

Originally Posted by juroara

wooo..hold up. your first example has me seriously confused. maybe I should just go to bed . What's confused me is the percentage you gave of the positive error. Why would that be wrong?

you just told me 95% of the people who test positive have this disease. That there is only a 5% of error.

Given that percentage, if only 1/1000 people actually have the disease, wouldn't only 1.05 people test positive? Or to make it a whole number of human beings, only 21 people out of 20,000 will test positive. With a 5% error, meaning 20/20000, or 1/1000 actually are sick?

I mean..its one thing to talk about the percentage of people who have the disease. It's one thing to ask what is your percentage of having the disease, or your chances of testing positive. Which is what, like (.105%)?

Its a whole different question that once you test positive, that your chances jump up to 95%. Regardless of the population, because its simply a 5% test error, having nothing to do with how many people actually get sick???

at least that's how I understood the problem. I would still answer 95%, because I am assuming the error of the test is technical, having nothing to do with how many test positive in the total population (.105%). But rather, how many who test positive who actually are............which you told me.......95% o_o .....or .1%

am I missing something?

seriously

you can yell at me

I haven't taken a math class in years

otherwise I understand how problematic percentages can be. but its the wording that you have to be careful of. I mean, I understand if 20% of the population has a disease, it doesn't mean that is also your chances.

If 1000 people test positive, 950 of them have the disease. 50 of them don't but return a positive anyway.

But those 950 positive tests are taken FROM that original 0.1% chance, the other 50 serve no purpose but to make the problem a better example of the challenge of probability.
1,000 positives out of 1,000,000 with 950 out of the 1,000 really being positive.

The two percentages are only slightly related. When you test positive, your chances do not jump up to 95% from 0.1%, simply testing positive won't randomly infect you. You have to look at the combined percentage, the one the math problem solved for.

Does that make sense?

Originally Posted by juroara

wooo..hold up. your first example has me seriously confused. maybe I should just go to bed . What's confused me is the percentage you gave of the positive error. Why would that be wrong?

The problem lies in the way that I explained the percentages; I was highly misleading. I edited the OP to be more accurate. I apologize for the confusion - thanks for pointing this out to me.