The Long Tail

The origins of "The Long Tail"

One of the signs of a great idea is that people feel like they've known it forever. That, at least, is what I tell myself when people suggest that the concept of The Long Tail predates my article of the same name last year. This first came up in the crafting of the Wikipedia entry on The Long Tail following my original article, and it again rose in my conversations with Tom Standage of The Economist last week. I'll take a moment here to explain the derivation and perhaps we can clear up this issue once and for all.

It is not new that powerlaw (and other) distributions have "heads" and "tails". It is also not new that some tails are longer than others. Indeed, if you search for "long tail" in the context of statistics (as opposed to animals with long tails and long tail boats [shown])  before September 2004, when my original piece came out, you will find quite a few references to various curves that have long tails. But the only one that even comes close to talking about the consequences of the abundance boom created by technology is a passing reference in a 2003 essay by Clay Shirky about powerlaws distributions in the blogosphere ("Meanwhile, the long tail of weblogs with few readers will become conversational...").

Given my intellectual debt to Clay for his groundbreaking powerlaw research, I was delighted to see him comment on this in the discussion behind the Wikipedia entry. He writes:

"For my part, when I used the term in "Powerlaws, Weblogs, and Inequality", I didn't think of it as a coinage at all -- linear distributions have heads and tails and the weblog tail is long and flat....Chris and I and lots of other people use the phrase to describe a particular kind of distribution, but Chris has taken it in the direction of Tipping Point, a phrase that conjures up a whole complex of related issues, particularly issues of the business aspects of media and culture, that I didn't. So from my pov, Chris should get credit for originality, not of the phrase but of its current application and vividness.

To be precise, what I coined was the notion of looking at the tail itself as a new market. The use of the proper noun (including "The") is not incidental, but is intrinsic to the observation that we have historically looked at the market at the head of the curve in isolation, and we can now shift our gaze to the right and see that the tail is another market.

The notion of two markets--The Head and The Long Tail; one familiar, the other long ignored but now emerging--is at the core of the thesis and explains the initial-caps TLT construction I've used. To get a sense of its resonance, compare  a long tail search before September 15th, 2004 (1,570 results) with one after (619,000 results). Long tails are not new; The Long Tail is.

The first rule of the blogosphere...

...is not to generalize about the blogosphere. I am reminded of this every time I read a hand-wringing article about what blogs are or aren't, such as this one by Adam Cohen in today's NYT.

Jeff Jarvis unloads on Cohen at length, but this to me is the key point:

He is, again, trying to turn blogs into an institution, like journalism. But they are not. Blogs are all individual.

You can find responsible and irresponsible blogs, journalistic and non-journalistic ones, male-dominated clusters and female-dominated clusters, snarky blogs and serious blogs, and superficial, derivative, navel-gazing ones alongside ones of such depth and scholarship that they put even the serious press to shame.

In short, blogs are a Long Tail, and it is always a mistake to generalize about the quality or nature of content in the Long Tail--it is, by definition, variable and diverse.

The presumption that things that are less popular are less good is a classic fallacy; indeed,tail content is often of higher quality than head content, simply because it is uncompromising. Content that is perfectly targeted for some people will be, by definition, wrong for other people--and that's okay.

Head content satisfies us in areas where we are similar (watching beautiful, talented people interact amusingly); Tail content satisfies us in areas where we are different (watching geeks talk about remotely programming DVRs). Me, I'd rather watch the latter than the former, but the beauty of the Long Tail is that you don't have to agree with me to make that geek show worth making. Vive la difference!

Bollywood's distributed demand

I've often pointed to the Indian film industry as a perfect Long Tail candidate. Each year Bollywood makes at least 800 films. In American alone, there are 1.7m people who are first or second generation Indians, most of whom can speak Hindi (the main language of Bollywood films).

Yet each year only a handful of those films get distribution in the few US theaters that even consider showing Indian movies. Although there's a huge potential audience here, it's too spread out to fill theaters anywhere but in a few urban Indian communities. This is the tyranny of geography at work: bricks-and-mortar distribution requires customers to be geographically concentrated, which is usually not the case for niche goods. The alternative is corner DVD rental shops, but the ones that specialize in Indian films are also found mostly in urban Indian neighborhoods, which still leaves much if not most of the potential market underserved.

The reason I bring this up is that I've noticed a lot of Hindi-language films appearing in my Netflix new release RSS feed. I just checked, and they now have more than 600 Bollywood DVDs, which is twice the size of even their French film inventory. That's even more than the Bollywood specialists online.

Further evidence that Netflix really gets the Long Tail. I note that they now carry 40,000 titles, of which 98% are rented at least once each quarter. Which, as it happens, is exactly the percentage of the top 10,000 music titles rented each month at Ecast, the digital jukebox company whose CEO first blew my mind 18 months ago with his observation of a long, long tail of demand. Neat.

Experiencing technical difficulties

I'm afraid I've temporarily screwed up the domain forwarding from thelongtail.com to this Typepad site, which results in visitors to www.thelongtail.com seeing a web hosting page. You can still get to this site at longtail.typepad.com and all the permalinks should still work, but until I sort out my web hosting transfer the usual front door will be farkled.

If you're curious, this came about because I was trying to fix a bug that caused people who came in via thelongtail.com to not see my RSS feed, while those who were pointed to longtail.typepad.com did. The solution is something called "domain mapping" and it involves a complicated and synchronized dance between my domain registrar and Typepad. One missed handoff and you get the embarrassment I'm now experiencing. But I'm working on it and hope to have it sorted, or at least patched, by the end of the day.

[UPDATE: semifixed, which is to say no worse than it was. But browsers still aren't seeing the RSS feed from the main domain, so work continues....]

The Lost Boys

Adding to the evidence of a shift from mass culture to niche culture, the latest Nielsen numbers (link coming it turns out this was custom research for us, so no external link I'm afraid) confirm that network TV is losing young men at an accelerating pace. For the September 04-April 05 period, broadcast (network) TV viewership by 18-34 year old males was down by 7.8%.

Where are they going instead? Ad-supported cable TV (ie, Comedy Central and the Cartoon Network, but not premium channels such as HBO), which were collectively up 5.3% over the same period, and videogames, up an astounding 16.7% in equivalent viewership terms.

Source: Nielsen Media Research

The Economist does the Long Tail

Tom Standage writes a great piece in The Economist this week on the Long Tail.  If you don't know Tom's work, check out his books, including the classic The Victorian Internet: The Remarkable Story of the Telegraph and the Nineteenth Century's On-Line Pioneers and brand new A History Of The World In Six Glasses. Here's an excerpt from his one-page Economics Focus column in the current issue:

Perhaps the most profound implication of the long tail, however, is its impact on popular culture. As choice expands and people can more easily find niche content that particularly interests them, hits will be less important: so what will people talk about when gathered around the water cooler? In fact, says Mr Anderson, the idea of a shared popular culture is a relatively recent phenomenon: before radio and television, he notes, countries did not operate in "cultural lockstep". And the notion of shared culture is already in decline, thanks to the rise of cable television and other forms of market fragmentation. The long tail will merely accelerate the effect. There will still be blockbuster movies, albums and books, but there will be fewer of them. The companies that will prosper, says Mr Anderson, will be "those that switch out of lowest-common-denominator mode and figure out how to address niches."

The three-body problem

In a previous life (somewhere between the punk rock bands and The Economist) I was in training to become a physicist. As the math got progressively gnarlier, we eventually encountered the "three-body problem", which is the challenge of computing the mutual gravitational interaction of three planets or other masses. It's practically impossible to solve, but does prove to be a handy metaphor to use in later life. Such as now...

I have, you might have noticed, failed to post for three weeks (other than adding sidebar shoutouts). The reason is that I've got my own three-body problem, with the three bodies being my job (editing Wired), my book, and my blog. (To say nothing of my family, complete with four little bodies who have their own gravitational attractions.) Two of those three are at least full-time jobs, so the challenge for me is to figure out how to fit the blog into the zero-sum game of time.

In principle, this shouldn't be too hard; the blog is about the book, after all, and my expectation was that the book would spin off plenty of tangents, sidechannel discussions and overmatter to feed a steady flow of posts. And indeed it has so far and probably will for a long time to come. But it turns out that just as Wired has a monthly cycle, of which the last two weeks demand maximum capacity, the book writing also has plant and reap cycles. And the last three weeks have been a lot of planting.

Right now I'm in the heavy research phase. That means lots of interviews and data negotiation, compilation and analysis. The good news is that they're both going brilliantly. Yesterday, in particular, was a landmark day for data. I'm not going to give away too much yet, but we basically compared two very  solid (20,000+ titles) data sets of music sales, one drawing from traditional retail (with shelf-space scarcity and poor searchability) and one online (huge inventory and good filters and recommendations).

My thesis had been a Long Tail business (massive variety plus good filters) would show a much flatter demand curve, with a more even balance between hits and niches. And, mirabile dictu, that's exactly what this data analysis showed. Even over the same inventory, when the two data sets were normalized (hits set to the same levels) the online business showed 2x demand for niches. Overall for the same sales of the top-1,000, the online stats showed 50% higher sales of the next 19,000.

So the  Long Tail doesn't just theoretically or anecdotally shift demand from hits and niches, I can now show it does so statistically, too. Next I hope to extend this to other markets. Oslo awaits!