The Statistics of Health Decision Making-Statistical Illiteracy

statanalysis_03This will be the first post of a three-part series. They will all be based on a 2008 monograph that appeared in Psychological Science in the Public Interest, entitled:  “Helping Doctors and Patients Make Sense of Health Statistics.” Gerd Gigerenzer was the primary author.  I have started posts on numeracy, especially health numeracy, based on journal articles, but they tended to be focused on the public’s generally poor numeracy skills.  This always stopped me in my tracks, because it left out the doctors who have skills much like the public, and it left out the researchers, screening test providers, drug makers, and device makers who tend to have excellent skills.  They sometimes use those skills to take advantage of the doctors, patients, and the public(for my purposes journalists are just part of the public).  This monograph puts  the pieces together.

The Public — Basic Numeracy
To analyze the prevalence of low numeracy and gauge the extent to which it impairs communication about health risks, Schwartz, Woloshin, Black, and Welch (1997) developed a simple three question scale.  The test was applied to a random sample of female veterans in New England, 96% of whom were high school graduates, and whose average age was 68. Forty-six percent were unable to convert 1% to 10 in 1,000, 80% were unable to convert 1 in 1,000 to 0.1%, and 46% were unable to correctly estimate how many times a coin would likely come up heads in 1,000 flips, with the most common incorrect answers being 25, 50, and 250.  Specific populations do better than this, but the performances are not great.

The Public -The Illusion of Certainty
The first item in minimal statistical literacy is learning to live with  uncertainty. To appreciate the importance of health statistics, patients need to understand that there is no certainty in the first place. The term illusion of certainty refers to an emotional need for certainty when none exists. This feeling can be attached to test results that are taken to be absolutely certain and to treatments that appear to guarantee a cure.

The Public – Understanding That Screening Tests May Have Benefits and Harms
Sir Muir Gray, knighted by the British Queen for his contribution to health-care issues, is known for saying that ‘‘All screening programs do harm; some do good as well, and, of these, some do more good than harm at reasonable cost.’’

The Public – Understanding the Difference Between Relative and Absolute Risk

In October 1995, the U.K. Committee on Safety of Medicines issued a warning that third-generation oral contraceptive pills increased the risk of potentially life-threatening blood clots in the legs or lungs twofold—that is, by 100%.  How big is 100%? The studies on which the warning was based had shown that of every 7,000 women who took the earlier, second-generation oral contraceptive pills, about 1 had a thrombosis; this number increased to 2 among women who took third-generation pills. That is, the absolute risk increase was only 1 in 7,000, whereas the relative increase was indeed 100%. Absolute risks are typically small numbers while the corresponding relative changes tend to look big—particularly when the base rate is low. Had the committee and the media reported the absolute risks, few women would have panicked and stopped taking the pill. The pill scare led to an estimated 13,000 additional abortions and a similar number of additional births.  Ironically, both abortions and pregnancy carry greater risk of thrombosis than the third generation birth control pill.

The Doctors – Basic Numeracy

Doctors do beat the public in that three question test mentioned above.  But only between 60% and 72% got all three correct in a couple of testings. The minority who did not get them all correct are probably not too good at critically assessing the findings of a study in the relevant literature,  If unable to do so, doctors are more dependent on hearsay or leaflets provided by the pharmaceutical industry to update their knowledge.

The Doctors – The Illusion of Certainty
Physicians need to inform patients that even the best tests are not perfect and that every test result therefore needs to be interpreted with care or the test needs to be repeated. Some test results are more  threatening than others and need to be handled particularly carefully.

The HIV test is one example. Out of every 10,000 men, it is expected that one will be infected and will test positive with high probability; out of the other, uninfected men, it is expected that one will also test positive (the complement to the specificity of 99.99%). Thus, two test positive, and one of these is infected.  Thus, if you test positive, your chances of having HIV is 50-50.  There were reports of suicides when testing positive in the early years of HIV.

From one who has much math education, good numeracy skills, and has been exposed to all the tricks, I continue to struggle with interpreting the results of screening tests on rare diseases like HIV.  I know to turn everything into an absolute frequency and to see how many are in each of the false negative, false positive, actual positive, and actual negative boxes. Still, I think that I should have some intuitive feel, and avoid using that painful analysis part of the brain.  I can’t.  They are much like financial derivatives.  You need a pencil.

The Doctors –  Screening Tests

Mammography screening reduces mortality from breast cancer by about 25%. Assume that 1,000 women age 40 and over participate in mammography screening. How many fewer women are likely to die of breast cancer?
 1 [66%]
 25 [16%]
 100 [3%]
 250 [15%]
The numbers in the brackets show the percentage of gynecologists who gave the respective answer. Two thirds understood that the best answer was 1 in 1,000. Yet 16% believed that the figure meant 25 in 1,000, and 15% responded that 250 fewer women in 1,000 die of breast cancer.

Improved statistical skills might provide doctors and patients with the momentum to reduce the unwanted geographical  and medical specialty variation, and to practice shared decision making based on the best scientific evidence, a huge and necessary step toward evidence-based medicine

Doctors influence patients’ understanding of health issues, and the media influence both. In this way, shared statistical illiteracy becomes a stable phenomenon whose existence is rarely noticed.