We need data, lots of it, to tell where we are and where we need to go. But, could there be so much data that we’ve become biased towards using the data no matter what? We must be able to account for everything....or should we? Here is a quick story about what data did not show and how important that was to the future of the free world.
Meet Abraham Wald - master of noticing what did not happen in order to better understand how success happens. During World War II, Wald applied his statistical expertise to various wartime problems. One of the problems he examined was the distribution of damage to aircraft returning from missions so as to minimize bomber losses. The Air Force saw an opportunity for efficiency; you can get the same protection with less armor if you concentrate the armor on the places with the greatest need –armoring areas that take the most fire. But Wald’s answer to the problem was unexpected. “The armor,” said Wald, “doesn’t go where the bullet holes are. It goes where the bullet holes aren’t.”
Wald’s insight was simply to ask: where are the missing holes? Wald was pretty sure he knew. The missing bullet holes were on the missing planes. If the plane made it back, it survived – this was actually success. Wald suggested that if you study the surface failure (bullet holes) and reinforce those areas, you’re missing the bigger picture; the parts that actually keep the plane flying.
We can easily become biased; pouring a lot of effort and “armor” into areas the data would indicate as important. But, we’re only scratching the surface and potentially missing the bigger picture. We still have to fly; too much armor in the wrong places makes us too heavy and too slow. This yields inefficacy and frustration with little to no impact on actually reducing catastrophic failure. Data helps us understand where the system fails; in doing so it also reveals to us something of much greater value – where the system succeeds. Where there is success, the learning can be especially valuable. Investigating the deeper roots of success allows us a unique opportunity to “armor” and reinforce things that promote successful outcomes without incurring the negative outcome of a failure.