Puzzles and Mysteries

Malcolm Gladwell wrote a great piece in the New Yorker: OPEN SECRETS: Enron, intelligence, and the perils of too much information. It’s being blogged all over, but since I’m still thinking about it 12 hours after reading, I’m posting too.

He distinguishes between puzzles and mysteries, and the role of data in both.

The national-security expert Gregory Treverton has famously made a distinction between puzzles and mysteries. Osama bin Laden’s whereabouts are a puzzle. We can’t find him because we don’t have enough information. The key to the puzzle will probably come from someone close to bin Laden, and until we can find that source bin Laden will remain at large.

The problem of what would happen in Iraq after the toppling of Saddam Hussein was, by contrast, a mystery. It wasn’t a question that had a simple, factual answer. Mysteries require judgments and the assessment of uncertainty, and the hard part is not that we have too little information but that we have too much. The C.I.A. had a position on what a post-invasion Iraq would look like, and so did the Pentagon and the State Department and Colin Powell and Dick Cheney and any number of political scientists and journalists and think-tank fellows. For that matter, so did every cabdriver in Baghdad.

Adding data to a mystery doesn't make it less of a mystery, but smart analysts can bring experience to bear that will point towards possible answers that are more or less likely to be true.

The Enron disaster qualifies as a mystery, and Gladwell's summary of the confused and complex financial details is disquieting. Even financial experts couldn't make heads or tails out of what was going on with the money. I'm unsettled by how easily corporate environments breed mysteries requiring complex analysis both of their internal and external behavior; and how hard it can be to understand and correct corporate mismanagement. People lose their livelihoods all the time now-- how many other Enrons are actually happening with no one noticing?

As a side note, as I read the recap of the Woodward and Bernstein investigation I actually thought, "This is surely going on all the time now. Would anyone be shocked at this in this political era?"