Holes In Your Theory

I absolutely love this picture:

Here’s the story: in WWII, the RAF was losing a lot of planes to German anti-aircraft fire and they wanted to increase the armor on them. But increasing armor over the whole plane would hinder their ability to fly well and also be really costly, so they were trying to be efficient by only armoring the places where a little extra armor would have the highest impact.

The way they figured out where to armor was to look at every single plane that came back from a battle and make a graph of where the bullet holes were. They’d then only armor those spots.

If you don’t see the hole (ha!) in this plan, don’t worry – the entire project team didn’t either. Only a mathematician named Abraham Wald spotted the error. You see, they were charting the bullet holes on the planes that came back. They couldn’t look at the ones that didn’t, of course. But that means literally every single bullet hole they charted was non-fatal. Where they needed the armor, Wald pointed out, was any place on their chart that didn’t have any bullet holes. Because that meant that shots to that area were fatal every time.

This is symptomatic of a larger blind spot we tend to have, which is that we measure what we can measure, and then draw conclusions from the measurements without realizing what we didn’t or couldn’t measure that may be really, really important.

This is what I was talking about just a few days ago when I wrote about companies that say that every person on their sales team is hitting quota. Sure, every salesperson on your team is making their numbers – just like how every plane that comes back has no bullet holes in the middle of the wing. You’re not measuring the planes that don’t come back – or the salespeople who get fired.

This is also the reason why a LOT of polls, on just about every topic from politics to consumer preferences, are garbage. If you ask people to reach out to you if they like a particular thing, and then say “nearly 100% of people who reached out to us said they liked X,” then you’re not measuring the people who didn’t reach out – and thus probably don’t like X. Or the reverse.

Don’t accept partial information. When making a decision you really need…

…the hole story.