The bullet holes show where a plane could be hit and survive. The places without holes are where the lost planes were hit.
Illustration: Martin Grandjean, McGeddon, and Cameron Moll. Wikimedia Commons, CC BY-SA 4.0. Source: File:Survivorship-bias.svg
Wald at the Statistical Research Group
In 1943, the U.S. military was about to armor its bombers in exactly the wrong places. The analysts had examined every bomber that came back from combat, mapped the bullet holes across the airframe, and proposed reinforcing the spots where the holes clustered. The data was right there. You could see it on the planes.
They referred the question to the Statistical Research Group at Columbia University, a small team of mathematicians assembled for the war effort. The Hungarian-born statistician Abraham Wald examined the bullet-hole maps and gave the opposite recommendation. Armor the engines and the cockpit. The places where the returning planes had no bullet holes.
The military analysts had committed a logical error so simple they could not see it. They were studying the planes that came back. The planes shot through the engines and the cockpit did not come back. They had gone down across two oceans and the territory between them. The returning planes did not reveal where bombers were vulnerable. They revealed where bombers could be hit and still fly home.
The bullet holes on the survivors mapped survivable damage, not dangerous damage. Armoring where the holes were meant armoring the places that did not need armoring. The damage that mattered was on the planes you could not examine because they were destroyed.
Wald’s memorandum was classified. Decades later, when his work was declassified and republished, the principle he had identified came to be called survivorship bias.¹ ² It is the most pervasive and least understood logical error in any field that draws conclusions from a visible population of survivors.
The same error sits at the center of the modern screening-and-treatment industry.
The Error Generalized
Survivorship bias operates wherever a process selects what gets seen. The destroyed do not file reports, and the audience reads only what made it through.
Mutual fund performance averages exclude funds that closed. Hedge fund return statistics quietly drop the funds that liquidated. The historical returns of “the market” routinely omit bankruptcies, delistings, and total losses. Funds that lived publish their numbers; funds that died publish nothing. The retail investor reads the winners.
The literature on entrepreneurship was built the same way. CEOs who succeeded wrote memoirs about their habits and their early-morning routines. The thousands of equally disciplined founders who failed wrote nothing because their companies went under. The “success habits” identified by reading the survivors are, in many cases, just habits, shared by the dead and the living alike, with no causal relationship to outcomes.
In architecture, the buildings that survive are studied for their construction methods. The buildings that collapsed in storms, earthquakes, or fires are no longer there to be examined. Old buildings appear well-built because the badly-built ones are gone.
In each case, the visible sample is selected by the very property you are trying to measure. You cannot learn about plane vulnerability from intact planes. You cannot learn about cancer survival from cancer survivors.
Survivorship Bias in Medicine
A woman undergoes mammography in her early fifties. The scan finds a small lesion. She receives a biopsy, a lumpectomy, six weeks of radiation, and five years of tamoxifen. She is alive ten years later. She becomes an advocate, walking in Race for the Cure and telling her sister, her daughter, and the women in her neighborhood that screening saved her life. Get the test.
Her experience is real, her gratitude genuine. The conclusion she draws does not follow from either.
What she does not know, what she cannot know, is what would have happened to her without the screening, the biopsy, the surgery, the radiation, and the years of medication. She cannot run the counterfactual on herself. The version of her that did not get the treatment does not exist, and she has no way to consult it.
The institutions that promoted her screening have access to data she does not. Population-level data, accumulated across decades of randomized and observational studies, is consistent with four overlapping forms of survivorship bias, each of which inflates the apparent success of the system. Together they are sufficient to explain most of what the industry presents as the triumph of “early detection.”
Survivorship Bias Proper
The most direct form: only the living testify.
Eight months after her mastectomy, a woman dies of chemotherapy-induced sepsis. She does not appear at the October fundraiser. Six weeks after his prostatectomy, a man dies of cardiac complications. He writes no op-eds about prostate health. The radiation that “cured” the first cancer induces a second one five years later, and the patient’s family attends a funeral, not a marathon. The chemotherapy regimen that ostensibly drove the tumor into remission also drove the patient’s bone marrow into failure, and she dies of sepsis a year later, recorded by quiet bureaucratic convention as a “cancer death.”
When the public hears about cancer treatment, it hears from the patients who survived. Those who did not survive are statistically invisible. They are counted in mortality columns nobody reads, while their grateful surviving counterparts address the television cameras. The audience for screening campaigns sees a heavily filtered population, filtered by the treatments themselves.
The filtering goes further than visibility. It reaches into the mortality data itself. When a patient on chemotherapy dies of cardiotoxic heart failure, the death is typically coded as a cancer death. When a patient with treatment-induced bone marrow failure dies of sepsis, it is coded as a cancer death. When the surgical complication kills the patient on the operating table, it is generally coded as a cancer death. When the second cancer induced by radiation given for the first cancer kills the patient ten years later, the second cancer is frequently recorded as primary, the radiation that caused it noted in passing if at all. The coding conventions tilt systematically in one direction: failures of treatment are folded back into the column labeled “disease.” The treatment is shielded from blame. The cancer absorbs it. The mortality statistics that institutions cite to justify aggressive treatment are themselves an artifact of how treatment failures are recorded.
Lead-Time Bias
Finding a cancer earlier does not mean treating it earlier extends life. It means knowing about it longer.
Two women with identical lesions, identical biology, and identical eventual outcomes. Both die at age seventy. Woman A is screened at fifty, her cancer is detected, and she is “treated” for the next twenty years. Woman B is unscreened. She develops symptoms at sixty-five, is diagnosed, and dies at seventy.
The standard reporting metric is five-year survival from diagnosis. By that measure, Woman A counts at 100% survival. Woman B counts at 0%. The treatment looks miraculous. Nothing has actually changed. Both women died at seventy. Woman A simply spent twenty years as a patient.
The five-year survival statistic is the standard currency of cancer reporting. In the presence of widespread screening, it is also a metric that can rise to 100% without saving a single life. Between 1950 and 1995, the five-year survival rate for prostate cancer in the United States rose from 43% to 93%. The age-adjusted mortality rate from prostate cancer over the same period barely moved.³ The screened population learned about their cancer earlier. They did not die later.
When you read that “early detection” has improved five-year survival rates for breast, prostate, or thyroid cancer, you are reading a statistic structurally biased toward the appearance of benefit even when no benefit exists.³
Length-Time Bias
Screening preferentially detects slow-growing lesions. Aggressive cancers grow rapidly between screening intervals and present symptomatically, not through the scan. Indolent lesions sit for years, available to be detected at the next mammogram or PSA test.
The population of cancers caught by screening is therefore enriched for slow biology, for lesions that were less likely to kill in the first place. Patients with these lesions tend to do well, not because the screening saved them but because their cancers were not going to kill them on any rapid timescale. The aggressive cancers, the ones that genuinely threaten life, frequently arise and progress in the gaps between scans.
Screening catches the cancers least in need of catching. The system then takes credit for the favorable outcomes of patients who would have done well regardless.
Overdiagnosis
The fourth and most powerful form. Many of the lesions detected by screening are not, in any meaningful sense, going to harm the patient. They are stable, non-progressive anatomical findings that medicine has chosen to label as cancer.
Bleyer and Welch, examining three decades of U.S. mammography data, estimated that 31% of breast cancers detected by screening represented overdiagnosis: disease that would never have produced symptoms or shortened life.⁵ The Cochrane systematic review of mammography trials concluded that for every life potentially saved by screening, ten women receive treatment for a cancer that would not have harmed them. The same review found no reduction in all-cause mortality from screening.⁶
In South Korea, the introduction of widespread thyroid ultrasound produced a fifteen-fold increase in thyroid cancer diagnoses over two decades. Mortality from thyroid cancer did not change. The country had not experienced a thyroid cancer epidemic. It had begun finding microscopic lesions that had always existed in the population, at autopsy in people who died of other causes, and that had never killed anyone before they were found and treated.⁷
Autopsy studies of men who died of unrelated causes have found prostate cancer cells in roughly a third of those in their forties, rising to two-thirds by their late sixties.¹⁵ The lifetime mortality from prostate cancer is approximately 3%. Most older men carry the disease into a natural death from something else; they die with prostate cancer, not from it. The PSA test cannot distinguish between the cancer that would have killed and the cancer that would have been silently carried into the grave. It detects both. It produces a diagnosis in both. The men with non-threatening lesions, who vastly outnumber the rest, are subjected to surgery, radiation, and hormonal therapy for a condition that was never going to harm them. They survive what was never threatening. They credit the system. They tell other men to get tested.
The ERSPC trial, the largest prostate cancer screening trial ever conducted, found that PSA screening reduced prostate cancer mortality by a small absolute amount over thirteen years. To prevent one death, approximately twenty-seven men had to be diagnosed and treated, most of whom would not have died from their disease and many of whom were rendered incontinent, impotent, or both by the intervention.⁸
Every overdiagnosed patient is, by definition, a successful “treatment outcome.” She survived a treatment for a condition that was never going to harm her. The system takes credit. She testifies on its behalf.
Why the System Selects for Evangelism
The four biases would matter less if the visible patients were a representative sample. They are not. The system that produces them also amplifies them.
Hospitals run survivor outreach programs. Pharmaceutical companies fund patient advocacy organizations. The pink ribbon ecology, Susan G. Komen, the National Breast Cancer Foundation, the dozens of subsidiary charities, operates almost entirely on survivor testimony. October fills American mailboxes with pink-ribboned testimonials. The American Cancer Society’s national publicity is built on survivor stories. The patient who survived is the asset.
The corporate machinery built around the survivor is substantial. Estée Lauder co-founded the pink ribbon symbol with Self magazine in 1992 and now sells pink-ribbon cosmetics each October. The National Football League runs an annual “Crucial Catch” campaign with players wearing pink cleats and accessories. Major League Baseball stages pink-bat games on Mother’s Day. Yoplait produced pink-lidded yogurt for two decades under the “Save Lids to Save Lives” campaign. Ford sold pink-ribbon merchandise through “Warriors in Pink.” General Mills, KitchenAid, the National Hockey League, the airlines, the cosmetics counters at every major department store all participate. The advertising spend on these campaigns runs into hundreds of millions of dollars annually, dwarfing the portion of the proceeds that ever reaches research and dwarfing many times over the portion of research funding that addresses environmental causes of breast cancer rather than treatment. The campaigns sell screening. The screening produces patients. Most patients survive, because most of what is found in screening was not going to kill them, and they testify. The testimony funds the next round of campaigns.
The patient who died is, from a marketing perspective, a problem. Her death cannot be celebrated. Her family is often grieving and angry. Her doctors generally do not call the local newspaper. She becomes a statistic in a column nobody reads, while the surviving patient in the next room becomes the face of the cause.
This selection is not a conspiracy. It is a structural feature of how the industry communicates. Living patients can be photographed; dead patients cannot. The living speak at events; the dead are credited, by quiet convention, to “the disease.”
The grateful survivor is also psychologically necessary for the treatment to continue being offered in its current form. The patient who has undergone radical mastectomy, six rounds of chemotherapy, weeks of radiation, and years of endocrine therapy must believe, on pain of intolerable cognitive dissonance, that this was necessary and life-saving. To accept that she may have been treated for a lesion that would not have harmed her, that she lost her breast, her hair, her fertility, her cardiac reserve, perhaps her marriage, to a system that misjudged the threat, is psychologically devastating. The mind protects itself. She becomes an advocate.
The advocate then promotes the system to other women, who undergo screening, get diagnosed, get treated, and become advocates in turn. Each cycle generates more survivors, each of whom credits the system that produced them. The dead and the harmed are silent by definition.
The financial scale of this ecosystem is not small. The United States spends roughly two hundred billion dollars annually on cancer-related medical care. Mammography alone is a multi-billion-dollar industry. The PSA test, despite repeated expert task force recommendations against routine screening, generates billions in downstream procedures. The pink ribbon charities raise hundreds of millions per year, much of which goes to “awareness,” that is, to producing more screening, more diagnoses, more treatment, and more survivors. The asset class, the patient, is manufactured by the process that then takes credit for her survival.
In 2018, a Goldman Sachs equity research report posed the question explicitly to its biotechnology clients: “Is curing patients a sustainable business model?”⁹ The analysts noted that one-time cures undermine recurring revenue streams. The pharmaceutical industry’s most profitable customers are chronic patients, not cured ones. The screening-and-treatment cancer model is, from a financial perspective, an excellent business. It produces patients. It treats them for years. Many of them survive, which is what the model needs them to do, because survivors testify and dead patients do not.
Inside the Testimonial
The cancer survivor has four pieces of information. She was screened. Something was found. She received treatment. She is alive years later. From these facts, she draws a single conclusion: the screening and treatment saved her life.
The inference is intuitive but unsupported. The same four facts admit at least three other explanations.
The screening detected a lesion that would never have harmed her. She survived not because of the treatment but in spite of it. The treatment did damage that she absorbed because the rest of her body was healthy enough to recover.
The lesion was real but slow-growing. She would have lived equally long, with less suffering, by doing nothing.
The lesion was real and biologically active, but her body’s repair mechanisms, what the establishment calls her constitution and what terrain medicine recognizes as her terrain, would have managed it. The treatment was incidental to her survival.
She cannot distinguish between these explanations from her own experience. None of them is available to her introspectively. The only way to determine which is correct, at the level of a population, is the randomized controlled trial, the kind of trial that, in most screening contexts, has either not been done with adequate follow-up or has produced equivocal results that the institutions promoting screening do not publicize.
Her testimonial is sincere. It is also, with respect to the question of whether the treatment worked, evidence of nothing in particular. The dead woman two beds down the hall, who received identical screening and identical treatment and died of cardiotoxic chemotherapy, would have a different testimonial if she could give one. The system that asks the survivor to speak does not ask the dead woman’s family to speak. The asymmetry produces the appearance of a treatment success rate the underlying data does not support.
What Survives the Error
Once you see survivorship bias, you cannot unsee it. What remains is not paranoia but a discipline: asking, in every medical context, which population you are looking at and which population is missing.
When a screening campaign reports that “five-year survival rates have improved,” ask whether overall mortality has changed. Five-year survival can rise to 100% without saving a single life if all the increase comes from earlier detection of lesions that were going to be survived anyway. Overall mortality, deaths per hundred thousand population per year, is much harder to manipulate. It is also the only figure that answers the question the survival rate appears to answer.
A cancer survivor telling you her treatment saved her life can be sincere and still wrong about causation. The conviction is real. The causal claim it carries is not derived from anything she has direct access to. You can be glad she is alive without accepting her account of why.
An oncology center’s published survivor outcomes report a filtered population. Ask about the patients who did not complete treatment. Ask about the patients who died of treatment-related complications and were classified as cancer deaths. Ask about the patients whose follow-up was lost because they moved into hospice care or stopped responding to calls. The shape of the population that gets reported is the shape of the population that survived long enough to be counted.
Facing a screening recommendation yourself, the question to ask is the one the military analysts in 1943 did not ask: what does this examination fail to show me? The bullet holes on the surviving bombers concealed the bullet holes on the destroyed ones. The success stories of the screening industry conceal the women treated for lesions they did not have, the men rendered impotent by surgery for cancers that would never have grown, the second cancers induced by the radiation, and the patients whose treatments killed them and who are now counted, by quiet bureaucratic convention, as having died from their disease.
The full investigative case on these screenings, what they detect, what they miss, what they manufacture, and what they cost, has been documented at length in earlier work.¹⁰ ¹¹ ¹² ¹³ ¹⁴
What the Trials Actually Show
Defenders of mammography routinely cite two trials. The Health Insurance Plan of New York trial, begun in 1963, and the Swedish Two-County Trial, conducted between 1977 and 1985, both reported reductions in breast cancer mortality among screened women. Both have been criticized on methodological grounds. The HIP trial’s randomization was uneven, with baseline differences between arms and exclusion rules applied asymmetrically. The Two-County Trial used cluster randomization that did not consistently balance comparison groups and lacked blinded cause-of-death assessment. Its mortality estimates shifted across successive reanalyses. The Cochrane systematic review of mammography trials classified both as carrying significant risk of bias.⁶
The most rigorously conducted breast cancer screening trial, the Canadian National Breast Screening Study, followed nearly ninety thousand women for twenty-five years. It found no reduction in breast cancer mortality from mammography screening, and no reduction in all-cause mortality.¹⁶
All-cause mortality is the figure that resists the gaming. When a study reports that breast cancer deaths fell among screened women but all-cause deaths did not, the women who avoided a death coded as breast cancer died of something else within the same window: heart failure from chemotherapy-induced cardiotoxicity, second cancers induced by the radiation given for the first, complications from the surgery, strokes after years of endocrine therapy. The Cochrane review and meta-analyses across multiple cancers have repeatedly found that all-cause mortality is essentially identical in screened and unscreened populations. The treatment that prevents one death produces another. The cancer-specific column improves; the death column does not. Survivorship bias gives the illusion of a saved life. All-cause mortality data shows that the life, where treatment did anything at all, was traded rather than added.
The Bombers and the Patients
Abraham Wald died in 1950 in a plane crash in the Nilgiri mountains of southern India. The principle he identified outlived him by three-quarters of a century and now sits, unrecognized by the institutions that depend on it, at the center of the modern medical industry.
The bombers that came back showed where a bomber could be shot and still come back. They did not show where a bomber was vulnerable. The cancer survivors who give interviews, walk in fundraisers, and tell their friends to get screened show where the modern oncology machine is not lethal. They do not show where it works.
The patients who died of their treatments are not at the marathon. The patients treated for lesions that would never have killed them have no way to know they were never in danger. The patients whose cancers were going to be survived regardless have no way to credit their own bodies rather than the chemicals introduced into them. They are the bullet holes on the wings: survivable damage, mapped and celebrated, while the damage that mattered remains invisible because the people who suffered it are no longer in the room.
Wald told the military to armor the engines and the cockpit, the places the surviving planes were not hit. The same instruction applies to medicine: look at what the survivors do not show you. The damage that matters is on the planes that did not come back, and on the patients who can no longer testify.
How to Explain This to a Six-Year-Old
Imagine all your friends drink a magic potion that’s supposed to keep them safe. Half of them disappear. The other half come back and tell you the potion worked great.
If you only listen to the friends who came back, you’ll be sure the potion is wonderful. You’ll tell other kids to drink it too.
But the other half are gone. They can’t tell you whether the potion hurt them. You don’t know whether the potion saved the friends who came back, or whether those friends would have been fine without ever drinking it.
When a cancer survivor tells you the treatment saved her life, listen to her kindly. But remember the friends who disappeared. They are part of the story too.
Quick Reference: The Four Biases
Survivorship bias. Only the living testify. Cancer patients who died of their treatments do not give interviews. The visible patient population is filtered by the treatments themselves.
Lead-time bias. Finding a cancer earlier does not extend life. It extends the time you know about it. Five-year survival rates can rise to 100% without saving a single life.
Length-time bias. Screening preferentially catches slow-growing lesions that were less likely to kill in the first place. Aggressive cancers arise and progress between scans, often undetected until they present symptomatically.
Overdiagnosis. Many screen-detected lesions would never have caused harm. The patient is treated for something that was not a threat, survives easily, and credits the treatment.
References
Mangel, M., & Samaniego, F. J. (1984). Abraham Wald’s work on aircraft survivability. Journal of the American Statistical Association, 79(386), 259–267.
Wald, A. (1943). A Method of Estimating Plane Vulnerability Based on Damage of Survivors. Statistical Research Group, Columbia University. (Declassified and republished by the Center for Naval Analyses, 1980.)
Welch, H. G., Schwartz, L. M., & Woloshin, S. (2000). Are increasing 5-year survival rates evidence of success against cancer? JAMA, 283(22), 2975–2978.
Welch, H. G., & Black, W. C. (2010). Overdiagnosis in cancer. Journal of the National Cancer Institute, 102(9), 605–613.
Bleyer, A., & Welch, H. G. (2012). Effect of three decades of screening mammography on breast-cancer incidence. New England Journal of Medicine, 367(21), 1998–2005.
Gøtzsche, P. C., & Jørgensen, K. J. (2013). Screening for breast cancer with mammography. Cochrane Database of Systematic Reviews, Issue 6, CD001877.
Vaccarella, S., Franceschi, S., Bray, F., Wild, C. P., Plummer, M., & Dal Maso, L. (2016). Worldwide thyroid-cancer epidemic? The increasing impact of overdiagnosis. New England Journal of Medicine, 375(7), 614–617.
Schröder, F. H., et al. (2014). Screening and prostate-cancer mortality in a randomized European study: results of the ERSPC at 13 years of follow-up. The Lancet, 384(9959), 2027–2035.
Richter, S., et al. (2018). The Genome Revolution (biotechnology equity research report). Goldman Sachs Global Investment Research, April 10, 2018.
Unbekoming. The 12 Screenings That Manufacture the Patients They Claim to Find.
Unbekoming. The Screening Trap.
Unbekoming. Breast Cancer: What They Didn’t Tell You.
Unbekoming. The PSA Trap: How a Flawed Test Built a Billion-Dollar Industry and Destroyed Millions of Men.
Unbekoming. The Unbekoming Cancer Compendium.
Sakr, W. A., Grignon, D. J., Crissman, J. D., Heilbrun, L. K., Cassin, B. J., Pontes, J. J., & Haas, G. P. (1994). High grade prostatic intraepithelial neoplasia (HGPIN) and prostatic adenocarcinoma between the ages of 20-69: an autopsy study of 249 cases. In Vivo, 8(3), 439–443.
Miller, A. B., Wall, C., Baines, C. J., Sun, P., To, T., & Narod, S. A. (2014). Twenty five year follow-up for breast cancer incidence and mortality of the Canadian National Breast Screening Study: randomised screening trial. BMJ, 348, g366.
Survivorship Bias: The Logical Error at the Heart of Modern Medicine
Reply With Quote
Bookmarks