Love Your Work 251. Survivorship Bias's Fatal Flaw
There’s an important bias to avoid: Survivorship bias. Unfortunately, people who might otherwise do something with their lives hide behind survivorship bias. Just as important as knowing when survivorship bias matters is knowing when survivorship bias does not matter. Survivorship bias has a fatal flaw.
Example: Abraham Wald avoided survivorship bias to bring back more survivorsIn WWII the US military was trying to improve their planes. Each time a plane came back from a mission, they made a record of the bullet holes. Since most bullet holes were on the wings and tails of the planes, the military concluded they needed to add more armor in the wings and tails.
But statistician Abraham Wald said, No – that’s not where you want to add more armor. You want more armor around the engine.
That seemed weird. Their map of bullet holes showed very little damage to the engine compartment.
What Wald noticed that the military hadn’t noticed is they were only seeing bullet holes on planes that returned from missions. The bullet holes they weren’t seeing were the bullet holes on planes that did not return. And the bullet holes on planes that did not return were the ones bringing the planes down.
Abraham Wald was cleverly taking into account what would become known as survivorship bias.
Example: How survivorship bias can be used by an investing con artistIn his book, Fooled by Randomness, Nassim Taleb tells a story of a con artist. He’d send out 10,000 letters. Half the letters predicted the stock market would go up in the next month. Half the letters, down.
The next month, the con artist would send not 10,000 letters, but only 5,000. The following month, 2,500. Then 1,250, and on and on.
Why did he keep sending fewer and fewer letters? Because he only sent follow-up letters to those who had received correct predictions. After enough letters, he had 150 or so victims hanging on his every word, eager to have this mystery genius invest money for them. Of course once the con artist received their money, they never heard from him again. They had been “fooled by randomness.” They had been fooled by survivorship bias.
Survivorship bias doesn’t account for ergodicityBoth these stories are useful examples of survivorship bias. In the first case, Abraham Wald used an awareness of survivorship bias to avoid getting a false signal from the data. In the second example, the recipients of the letters didn’t realize they could be getting a false signal from the letters.
Survivorship bias is an important phenomenon to understand, but survivorship bias has a fatal flaw: Survivorship bias doesn’t account for ergodicity.
What is ergodicity?What is ergodicity? Imagine you enter a dimly-lit bar just as it opens. A table of patrons across the room light up cigarettes. You can see the cascading trails of smoke rising. When they’re done with their cigarettes, they don’t smoke anymore the rest of the night.
When you get home, you realize your clothes smell like smoke. How could this be? You were nowhere near the trails of smoke.
Well, after the trails of smoke rose from the cigarettes, they dissipated around the room, until a faint haze of smoke filled the entire room.
Randomness eventually touches everythingThat’s ergodicity. The smoke was rising from the cigarettes in a random pattern. But when a random pattern continues for long enough, that random pattern eventually fills the entire space it could have filled. The smoke spread randomly, until it filled the whole room.
Ergodicity is why it’s not only 1% of Americans who are in the top 1% of income. As time passes, people enter and leave the top 1% of income. In a lifetime, 10% of Americans spend a year in the top 1%. More than half will spend a year in the top 10%.
Ergodicity is why – even though life expectancy is about 76 – a 76-year-old only has a 4% chance of dying. The small risks of dying each year of life accumulate over time.
Not every game is do-or-die
Next time some entrepreneur or creative gives advice, or is profiled in an article, look at the comments or responses. You’ll probably see something like this: Don’t forget about survivorship bias! You didn’t hear from the thousands of others who followed that same advice, but didn’t succeed!
Sometimes this is useful. More often than not, this is as damaging as survivorship bias itself.
Example: Survivorship bias in Russian RouletteImagine Russian Roulette was a spectator sport (thank God it’s not, but imagine). Chances are, there would be some “Michael Jordan” of Russian Roulette. Through mere chance, this person has survived hundreds of Russian Roulette matches.
It just so happens that of the thousands of times this “champion” has spun the cylinder on the revolver, pointed the gun at their temple, and pulled the trigger, the chamber hasn’t had a bullet in it once. If there were millions of Russian Roulette players in the world – playing college Russian Roulette and little league Russian Roulette, hoping to make it to the Russian Roulette big leagues – a person like this would probably exist.
Just like there is today in entrepreneurship and creativity, there would be an entire cottage industry of journalists and courses and Russian Roulette podcasts, all touting the advice from this Russian Roulette champion. How to spin the chamber, what thoughts to think while pulling the trigger, how much pressure to use, what gear like like gloves and jerseys to wear, and exercise programs for strength and conditioning.
That would be survivorship bias at its finest – or worst. It’s all random. This “champion” has no skill. All their advice is useless.
The Queen’s Gambit was not a “survivor” – then it wasIn 1983, Walter Tevis published a novel. He soon after optioned the screenplay rights to Jesse Kornbluth. Then Tevis died, and the project was cancelled.
Nine years later, in 1992, Kornbluth could no longer afford to keep the option. Allan Scott bought the screen rights.
Fifteen years later, in 2007, plans were underway to make a feature film out of this novel. Then the director died. (That director, Heath Ledger.)
Finally, in 2020, the story from this novel was released as a Netflix series. At least 60 million people watched it. It’s Netflix’s most popular limited series ever.
That series: The Queen’s Gambit. For the first time, the novel, The Queen’s Gambit, became a New York Times bestseller.
This overnight success was almost 40 years in the making. For 37 years, The Queen’s Gambit was one of the “thousands of others who never made it.” In 2011, Kornbluth – who had the screenplay option before Scott had rights – said the rights had been bought by “people who will never get the film made.”
The Queen’s Gambit was not a “survivor.” In fact, it went out of print. Not to mention at least two people literally did not survive to see it on the screen.
Creative work is not Russian RouletteWhen you play Russian Roulette and lose, you are out of the game forever. Fortunately, as creatives, we are not playing Russian Roulette.
If you build a company that fails, you can try again. If you write a blog post that falls flat, you can try hundreds or thousands more times.
Creative work happens in ExtremistanWhen someone says “Don’t forget survivorship bias,” what they’re really saying is, “Show me the exact steps to follow that guarantee success.” In creative work, there are no exact steps that guarantee success. Those only exist in Mediocristan.
Even The Queen’s Gambit, which was wildly successful, wasn’t guaranteed success. It could have just as easily stayed out of print.
Creative work happens not in Mediocristan, but Extremistan. No failure will come from pure lack of skill. No success will come from pure good luck.
Creatives are like poker championsI have a friend who is a professional poker player. He knows if he plays poker online eight hours a day, he’ll average 100 dollars an hour.
But he also knows for long stretches of time he’ll be losing money. It will look as if his career is over. On the contrary, he’ll also sometimes be flush with cash. It will look like he’s making way more than 100 dollars an hour.
His career only works if he has one critical thing: “bankroll.” He needs a certain amount of money – a certain amount of padding – to help him weather losing streaks. If he goes bust, he’s out of the game entirely – he’s lost Russian Roulette.
He has tremendous skill. That’s how he can survive as a professional poker player. But there’s no fighting randomness. His bankroll allows him to let randomness run its course long enough for ergodicity to even things out.
The creative career is riding randomnessYour results in creative work are not a direct reflection of your skill. Even if you’re “So good they can’t ignore you,” you could be toiling in obscurity for a while as you wait for your big break.
If you aren’t cut out for that, fine. But admit it to yourself and don’t use survivorship bias as your scapegoat.
Stay in the game long enough to surviveBut if you’re willing to try something that, as Seth Godin says, “might not work,” go ahead and try that advice. Maybe it will improve your odds a little.
What’s important is you stay in the game long enough to let ergodicity give you more shots at a win. That could be literally having the bankroll to stay in the game. That could be making sure you make small enough bets – with high enough potential upside – that you don’t go bust, but have a chance to hit the jackpot. You have to watch your eggs long enough for a Black Swan to hatch.
My Weekly Newsletter: Love MondaysStart off each week with a dose of inspiration to help you make it as a creative. Sign up at: kadavy.net/mondays.
About Your Host, David KadavyDavid Kadavy is author of Mind Management, Not Time Management, The Heart to Start and Design for Hackers. Through the Love Your Work podcast, his Love Mondays newsletter, and self-publishing coaching David helps you make it as a creative.
Follow David on:
Subscribe to Love Your Work Support the show on PatreonPut your money where your mind is. Patreon lets you support independent creators like me. Support now on Patreon »
Show notes: http://kadavy.net/blog/posts/survivorship-bias/