Cognitive Bias: Base-Rate Fallacy

mig-19 fighter plane
Image: MiG 19. Public Domain.

The base-rate fallacy happens when available statistical data is ignored in favor of specific data to make a probability judgment.

The C.I.A. gives this example to illustrate the problem:

During the Vietnam War, a fighter plane made a non-fatal strafing attack on a US aerial reconnaissance mission at twilight. Both Cambodian and Vietnamese jets operate in the area. You know the following facts:

(a) Specific case information: The US pilot identified the fighter as Cambodian. The pilot’s aircraft recognition capabilities were tested under appropriate visibility and flight conditions. When presented with a sample of fighters (half with Vietnamese markings and half with Cambodian) the pilot made correct identifications 80 percent of the time and erred 20 percent of the time.

(b) Base rate data: 85 percent of the jet fighters in that area are Vietnamese; 15 percent are Cambodian.

Question: What is the probability that the fighter was Cambodian rather than Vietnamese?

A common procedure in answering this question is to reason as follows: We know the pilot identified the aircraft as Cambodian. We also know the pilot’s identifications are correct 80 percent of the time; therefore, there is an 80 percent probability the fighter was Cambodian. This reasoning appears plausible but is incorrect. It ignores the base rate–that 85 percent of the fighters in that area are Vietnamese. The base rate, or prior probability, is what you can say about any hostile fighter in that area before you learn anything about the specific sighting.

The correct way to do this is to use Bayesian reasoning:

If we suppose that there are 100 enemy fighter planes total, that means that 85 are Vietnamese and 15 are Cambodian.

From paragraph (a), we know that the eye-witness identifies correctly enemy planes 80% of the time, so out of 85 Vietnamese planes, he would identify 68 correctly (85 * 0.80 = 68) and erroneously identify 17 (85 * 0.20 = 17).

Out of the 15 Cambodian aircrafts, he would identify correctly 12 of them (15 * 0.80 = 12) and be mistaken about 3 (15 * 0.20 = 3).

This makes a total of 71 Vietnamese and 29 Cambodian sightings, of which only 12 of the 29 Cambodian sightings are correct; the other 17 are incorrect sightings of Vietnamese aircraft. Therefore, when the pilot claims the attack was by a Cambodian fighter, the probability that the craft was actually Cambodian is only 12/29ths or 41 percent, despite the fact that the pilot’s identifications are correct 80 percent of the time.

Ignore the base-rate in favor of specific data at your own risks! In some cases, it can make a huge difference.

mammograph
Normal (left) versus cancerous (right) mammography image. Public Domain image.

Another example to make this kind of reasoning clearer:

1% of women at age forty who participate in routine screening have breast cancer. 80% of women with breast cancer will get positive mammographies. 9.6% of women without breast cancer will also get positive mammographies. A woman in this age group had a positive mammography in a routine screening. What is the probability that she actually has breast cancer?

So, based on the numbers above, what is the probability that a woman who gets a positive mammography really has breast cancer?

Lets go through it. If there are 10,000 women screened, 1% will have breast cancer. So that’s 100. 80% of those will get a positive result, so that’s 80. That leaves us with 9,900 women who don’t have breast cancer. Out of those, 9.6% will get a false-positive result, so that’s 950 women.

You see where this is going?

So out of 10,000 women who get tested, 80 will have a real positive result and 950 will have a false positive, for a total of 1,030 positive results. Out of those, only 80 really have cancer, that’s 7.76% (80/1,030 * 100 = 7.76).

So if, with these numbers, you were to get a positive result to your mammography test, that would still mean that you only had a 7.76% chance of really having breast cancer.

Counter-intuitive, but true.

Sources:

See also: Rationality Resources

About these ads

4 Responses to “Cognitive Bias: Base-Rate Fallacy”

  1. Nu Says:

    Mike,

    I have two observations. In the first example, you don’t actually give the Bayesian probabilities for false alarms from the actual data, and assume that in each case it is 0.8. More likely, they will have differnent false alarm rates. Imagine for instance that all of the Migs were correctly identified and that 30% of the Cambodian a/c. This would bring your probability of a correct Cambodian sighting down to about 10%.

    In the second example, the statistic that scares me is that of those who get a second mamogram (standard procedure when you have an initial positive as you show) about 90 women out of 10,000 will have TWO false positives, and then be put on the track for having to treat breast cancer until found otherwise. Furthermore, 7 of those with breast cancer will have a second false negative. You are thus left with a conundrum in either case after the first positive.

  2. Michael Graham Richard Says:

    Hi Nu,

    I’m using the data from the CIA’s example (see link at the end). I’m sure that in a real-life situation things would be more complex. Also, I think that what they imply is that both the Vietnamese and Cambodian use the same planes (Soviet-made), so the witness is tested on how well he recognizes the markings on the planes and not plane types.

    I put a picture of a MiG 19 on top because it is one of the planes used in the war, but I didn’t do the research to see if both Cambodia and Vietnam used those. I know MiG 17 and 21 were more frequent, though.

    Very good observation about the second example. Scary stuff, really.

  3. why base rates matter – Small Gray Matters Says:

    [...] For a nice overview of empirical data on the base rate fallacy, see this article in BBS. For more blogospheric bloviation on base rates, see here, here, and here. [...]

  4. AmPahn Says:

    Hi everyone. I’m new here. Just wanted to say hi

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 66 other followers

%d bloggers like this: