The Probabilistic Age

Q: Why are people so uncomfortable with Wikipedia? And Google? And, well, that whole blog thing?

A: Because these systems operate on the alien logic of probabilistic statistics, which sacrifices perfection at the microscale for optimization at the macroscale.

Q: Huh?

A: Exactly. Our brains aren't wired to think in terms of statistics and probability. We want to know whether an encyclopedia entry is right or wrong. We want to know that there's a wise hand (ideally human) guiding Google's results. We want to trust what we read.

    When professionals--editors, academics, journalists--are running the show, we at least know that it's someone's job to look out for such things as accuracy. But now we're depending more and more on systems where nobody's in charge; the intelligence is simply emergent. These probabilistic systems aren't perfect, but they are statistically optimized to excel over time and large numbers. They're designed to scale, and to improve with size. And a little slop at the microscale is the price of such efficiency at the macroscale.

    But how can that be right when it feels so wrong?

    There's the rub. This tradeoff is just hard for people to wrap their heads around. There's a reason why we're still debating Darwin. And why Jim Suroweicki's book on Adam Smith's invisible hand is still surprising (and still needed to be written) more than 200 years after the great Scotsman's death. Both market economics and evolution are probabilistic systems, which are simply counterintuitive to our mammalian brains. The fact that a few smart humans figured this out and used that insight to build the foundations of our modern economy, from the stock market to Google, is just evidence that our mental software has evolved faster than our hardware.

    Probability-based systems are, to use Kevin Kelly's term, "out of control". His seminal book by that name looks at example after example, from democracy to bird-flocking, where order arises from what appears to be chaos, seemingly reversing entropy's arrow. The book is more than a dozen years old and decades from now we'll still find the insight surprising. But it's right.

    Is Wikipedia "authoritative"? Well, no. But what really is? Britannica is reviewed by a smaller group of reviewers with higher academic degrees on average. There are, to be sure, fewer (if any) total clunkers or fabrications than in Wikipedia. But it's not infallible either; indeed, it's a lot more flawed that we usually give it credit for.

    Britannica's biggest errors are of omission, not commission. It's shallow in some categories and out of date in many others. And then there are the millions of entries that it simply doesn't--and can't, given its editorial process--have. But Wikipedia can scale to include those and many more. Today Wikipedia offers 860,000 articles in English - compared with Britannica's 80,000 and Encarta's 4,500. Tomorrow the gap will be far larger.

    The good thing about probabilistic systems is that they benefit from the wisdom of the crowd and as a result can scale nicely both in breadth and depth. But because they do this by sacrificing absolute certainty on the microscale, you need to take any single result with a grain of salt. As Zephoria puts it in this smart post, Wikipedia "should be the first source of information, not the last. It should be a site for information exploration, not the definitive source of facts."

    The same is true for blogs, no single one of which is authoritative. As I put it in this post, "blogs are a Long Tail, and it is always a mistake to generalize about the quality or nature of content in the Long Tail--it is, by definition, variable and diverse." But collectively they are proving more than an equal to mainstream media. You just need to read more than one of them before making up your own mind.

    Likewise for Google, which seems both omniscient and inscrutable. It makes connections that you or I might not, because they emerge naturally from math on a scale we can't comprehend. Google is arguably the first company to be born with the alien intelligence of the Web's large-N statistics hard-wired into its DNA. That's why it's so successful, and so seemingly unstoppable.

    Paul Graham puts it beautifully:

"The Web naturally has a certain grain, and Google is aligned with it.  That's why their success seems so effortless.  They're sailing with the wind, instead of sitting becalmed praying for a business model, like the print media, or trying to tack upwind by suing their customers, like Microsoft and the record labels. Google doesn't try to force things to happen their way.  They try to figure out what's going to happen, and arrange to be standing there when it does."

The Web is the ultimate marketplace of ideas, governed by the laws of big numbers. That grain Graham sees is the weave of statistical mechanics, the only logic that such really large systems understand. Perhaps someday we will, too.

[Update: Nicholas Carr, who seems to have inherited the Clifford Stoll chair of reliable techno-skepticism, has a clever and well-written response here.]

Comments

[sanateseri]

Wired editor Chris Anderson offers a spirited defense of internet "systems" like Wikipedia, Google, and the blogosphere. Criticism of these systems, he argues, stems largely from our incapacity to comprehend their "alien logic." Built on the mathematic...

[DFDF]

http://www.hugechoiceof.com/

[ben]

I don't buy the "not wired" argument, that seems more a matter of how we're trained to think. Many people are quite good at figuring out things intuitively, and I'd be inclined to guess that what we call intuition is an innate capacity for sorting out imprecise probabilities without consciously working out special algorithms.http://damen.big-teens.com/

[nice]

http://www.anzwers.net/hot/nicesite/

[Joseph Dunphy]

So dismayed was I, as somebody who actually has studied Probability at the PhD level, to learn that the logic of Probability and Statistics was inaccessible to the Mammalian brain! It was all that I could do to not look up a certain Biostatistician with whom I have had many interesting discussions and other meaningful moments and ask her for clarity. "Honey, are we mammals?" But it was all too easy to picture her opening her blouse and asking "what do you think, Joe", and there was no way I'd be able to focus on work after that, so I had to learn to live with my self-doubt, if only for the moment.

But pray, do tell which branch of the animal kingdom claims those of us who have pursued studies in and contributed to the building of this field, which mere Mammalian brains can't take in. On witnessing the pallor of a measure theorist of my acquaintence, I imagined that we might be fish, raising disturbing questions about those midautumn clambakes some of us spoke of, for that is surely no way to treat one's kin. But then I remembered how much I enjoyed my strolls up out of the Sonoran Desert, and thought "I must surely not be a fish, or even an amphibian. Perhaps a reptile?"

But as I remembered climbing higher and higher, I started remembering how pleasant the crisp air atop the Santa Catalinas felt, and I was sure that no lizard could possibly find such pleasure at the sight of snow, no matter how hot he had been hours before. No, I thought, I must surely be a bird. But then I got one look over the side of the cliff I had just gotten done hiking along, and about four bowel loosening minutes later, I was fairly sure that I couldn't fly. Though I was sure that some of my ex-students, in their scientific zeal, would have urged me to test that hypothesis by experiment, I decided to accept my flightless status, and as I am decidely too short to be an emu or an ostrich, and decidedly too tall to be a penguin, the whole avian lineage was lost to me and mine.

So as you can see, our entire department, graduate students and faculty alike, is now left in crisis, with no biological sense of itself, all hoping that you might help us find resolution. I can only pray that you can find time in your busy schedule in which to do so.

Your humble servant,

Joseph Dunphy

[porno]

http://frauen-porno.oral-teeny.com/

[jeu en ligne]

I find it amusing that you are endorsing so strongly an article that says that creationists are people whose brain aren't able to grasp the truth about evolution.

This must be an example of the liberal media bias I've been hearing about.