Lies, damn lies, and statistics

[squirell nutkin]

I was going to tag this into the thread Look! A missile! Blowing up an airplane! but I figured it was a thread drift two generations removed, and I didn't want to compete with CF's and UT's discussion.

A friend of mine recently had a mammogram that came back with a request for a biopsy or further testing or something. I didn't hear what. What I did hear about was that she spent the weekend (the news came on a Friday--nice!) in agony and torment and fear that she had breast cancer. Despite there being no history of it in her family and her being in a low risk population.

My BIL, who is a doctor and my SIL who is an oncology nurse both advise against routine mammography. Here is an excerpt from a a NYT review of a book by Gerd Gigerenzer "Calculated Risks: How To Know When Numbers Deceive You"

In one study, Gigerenzer and his colleagues asked doctors in Germany and the United States to estimate the probability that a woman with a positive mammogram actually has breast cancer, even though she’s in a low-risk group: 40 to 50 years old, with no symptoms or family history of breast cancer. To make the question specific, the doctors were told to assume the following statistics — couched in terms of percentages and probabilities — about the prevalence of breast cancer among women in this cohort, and also about the mammogram’s sensitivity and rate of false positives:

The probability that one of these women has breast cancer is 0.8 percent. If a woman has breast cancer, the probability is 90 percent that she will have a positive mammogram. If a woman does not have breast cancer, the probability is 7 percent that she will still have a positive mammogram. Imagine a woman who has a positive mammogram. What is the probability that she actually has breast cancer?

Gigerenzer describes the reaction of the first doctor he tested, a department chief at a university teaching hospital with more than 30 years of professional experience:

“[He] was visibly nervous while trying to figure out what he would tell the woman. After mulling the numbers over, he finally estimated the woman’s probability of having breast cancer, given that she has a positive mammogram, to be 90 percent. Nervously, he added, ‘Oh, what nonsense. I can’t do this. You should test my daughter; she is studying medicine.’ He knew that his estimate was wrong, but he did not know how to reason better. Despite the fact that he had spent 10 minutes wringing his mind for an answer, he could not figure out how to draw a sound inference from the probabilities.”

When Gigerenzer asked 24 other German doctors the same question, their estimates whipsawed from 1 percent to 90 percent. Eight of them thought the chances were 10 percent or less, 8 more said 90 percent, and the remaining 8 guessed somewhere between 50 and 80 percent. Imagine how upsetting it would be as a patient to hear such divergent opinions.

As for the American doctors, 95 out of 100 estimated the woman’s probability of having breast cancer to be somewhere around 75 percent.

The right answer is 9 percent.

How can it be so low? Gigerenzer’s point is that the analysis becomes almost transparent if we translate the original information from percentages and probabilities into natural frequencies:

Eight out of every 1,000 women have breast cancer. Of these 8 women with breast cancer, 7 will have a positive mammogram. Of the remaining 992 women who don’t have breast cancer, some 70 will still have a positive mammogram. Imagine a sample of women who have positive mammograms in screening. How many of these women actually have breast cancer?

Since a total of 7 + 70 = 77 women have positive mammograms, and only 7 of them truly have breast cancer, the probability of having breast cancer given a positive mammogram is 7 out of 77, which is 1 in 11, or about 9 percent.

Notice two simplifications in the calculation above. First, we rounded off decimals to whole numbers. That happened in a few places, like when we said, “Of these 8 women with breast cancer, 7 will have a positive mammogram.” Really we should have said 90 percent of 8 women, or 7.2 women, will have a positive mammogram. So we sacrificed a little precision for a lot of clarity.

another example from the book

Although reformulating the data in terms of natural frequencies is a huge help, conditional probability problems can still be perplexing for other reasons. It’s easy to ask the wrong question, or to calculate a probability that’s correct but misleading.

Both the prosecution and the defense were guilty of this in the O.J. Simpson trial of 1994-95. Each of them asked the jury to consider the wrong conditional probability. The prosecution spent the first 10 days of the trial introducing evidence that O.J. had a history of violence toward his ex-wife, Nicole. He had allegedly battered her, thrown her against walls and groped her in public, telling onlookers, “This belongs to me.” But what did any of this have to do with a murder trial? The prosecution’s argument was that a pattern of spousal abuse reflected a motive to kill. As one of the prosecutors put it, “A slap is a prelude to homicide.”

Alan Dershowitz countered for the defense, arguing that even if the allegations of domestic violence were true, they were irrelevant and should therefore be inadmissible. He later wrote, “We knew we could prove, if we had to, that an infinitesimal percentage — certainly fewer than 1 of 2,500 — of men who slap or beat their domestic partners go on to murder them.”

In effect, both sides were asking the jury to consider the probability that a man murdered his ex-wife, given that he previously battered her. But as the statistician I. J. Good pointed out, that’s not the right number to look at.

The real question is: What’s the probability that a man murdered his ex-wife, given that he previously battered her and she was murdered by someone? That conditional probability turns out to be very far from 1 in 2,500.

To see why, imagine a sample of 100,000 battered women. Granting Dershowitz’s number of 1 in 2,500, we expect about 40 of these women to be murdered by their abusers in a given year (since 100,000 divided by 2,500 equals 40). We can estimate that an additional 5 of these battered women, on average, will be killed by someone else, because the murder rate for all women in the United States at the time of the trial was about 1 in 20,000 per year. So out of the 40 + 5 = 45 murder victims altogether, 40 of them were killed by their batterer. In other words, the batterer was the murderer about 90 percent of the time.

The upshot for my friend was that she was fine and endured three days of anguish for no real good reason.

[Undertoad]

It's an awesome point. The same is true of drug testing, where cheaper tests will find more false positives than true positives.

[Griff]

Originally Posted by squirell nutkin

The upshot for my friend was that she was fine and endured three days of anguish for no real good reason.

[Clodfobble]

I'm curious whether all the doctors they surveyed were practicing gynecologists. Most of the doctors I've seen always put things in terms not of real statistics, but of their own experiences. My OB didn't tell me the numerical risks of having a normal delivery after having a caesarian section, for example, because he didn't know them--instead he described the possible outcome, and said that he'd never had it happen for any of his patients, but that it did happen "sometimes." On the other hand, when telling me the risks of an amniocentesis, he very gravely told me that he had indeed seen it happen a handful of times in his career. If most women over 35 in his practice are getting mammograms every year, why wouldn't a physician have a general sense that 9 out of 10 of the positive mammograms he'd seen in his practice had turned out to be negative after all?

[Lamplighter]

My BIL, who is a doctor and my SIL who is an oncology nurse both advise against routine mammography.

Eight out of every 1,000 women have breast cancer. Of these 8 women with breast cancer, 7 will have a positive mammogram. Of the remaining 992 women who don’t have breast cancer, some 70 will still have a positive mammogram. Imagine a sample of women who have positive mammograms in screening. How many of these women actually have breast cancer?

OK, but this is a problem with all tests of all kinds.
Every kind of test has some probability of false-positives and false-negatives.
So it is always a matter of RISK vs BENEFIT

BENEFIT of having the mammogram:
Dx leads to treatment and longer survival
(7 women in the example)

RISK of having the mammogram:
First, does the test CAUSE any problems ? (a mammogram = finite, but very low probability)
False Positives cause anxiety/expense until further testing is done
False Negative may lead to early death
(1 woman in example)

RISK of NOT having the mammogram:
High probability of early death due to lack of treatment
(all 8 women in example)

Being male, I can't make the decision.
But for my wife and daughters and G-daughters, a few days of anxiety vs early death:
"No matter how much it hurts, Go get the damn mammogram !"

My wife's reply: OK, but no matter how embarrassing, Go get your damn prostate exam !"

[squirell nutkin]

Actually, there was part of the book that I didn't quote which also goes on to mention that there is a type of breast cancer that a mammogram can detect that a regular exam cannot, Ductal carcinoma in situ, that is non invasive and rarely needs treatment. Read here:
http://cllr.me/NcE

Here is a PDF of the article that explains the cost benefit more clearly.
http://www.psychologicalscience.org/..._2_article.pdf

[ZenGum]

I have seen this issue before, couched in terms of doctors not understanding the statistics of false positives.

Part of the problem is that the simple statement "false positive rate of 7%" is ambiguous.
It might be takes as:
7% of positive tests are false,
7% of false tests are positive or
7% of tests of negative subjects will falsely show positive.
The third is the correct interpretation but the first is the most common one.

The further point about comparing the likelihood of a false positive with the likelihood of actually having the condition is still valid.