The Long Tail

The Cricket Diaspora

From Paid Content:

India’s big media group TV18 Group is becoming the Fox Interactive of the space there…it has announced three new acquisitions in the online space...including cricket portal CricketNext.com...

Why is this a big deal? By itself it isn't. But what it hints at is something potentially huge: the Long Tail of global audiences for regional content. In sports, news and entertainment we're shifting away from thinking only about concentrated audiences in one geography to thinking about distributed audiences around the world.

One country's hits are another country's niches. In Japan Anime and Manga are mainstream; here in the United States they're niche. Telenovelas are mainsteam in Latin America, but niche elsewhere. And then there's that whole soccer thing, which is seemingly mainstream everywhere but the US, where outside of World Cup time it can hardly be found on TV at all.

Take cricket. It's huge in Commonwealth countries, as well as the Subcontinent and the rest of the former British Empire. In India, Pakistan, Australia, South Africa, New Zealand the the UK, all the big cricket matches are broadcast live on TV. Elsewhere they're virtually impossible to find. According to the economics of broadcast TV, this makes sense: you can only devote your scarce airwaves to content of mass appeal. But there are millions of cricket fans outside of those few concentrated markets. And they're as eager to watch live (or even recorded) cricket matches as their home-bound countrymen. Now, thanks to streaming web video, they can.

There are about 25 million people in the Indian Diaspora, most of whom are in countries that don't broadcast cricket on TV. Think of them as the Cricket Diaspora, a distributed audience of potentially immense scale. The same for the Rugby Diaspora, the Soccer Diaspora, the Sumo Diaspora and so on. Then turn the tables and do the same for the potential global audience for US sports: the Football Diaspora, the Baseball Diaspora the Basketball Diaspora. Then extend that to news, TV shows, music and more. See what I mean?

Today, as more and more TV migrates from the scheduled world of live broadcast to the on-demand world of streaming Web video, we're about to enter an era where distributed markets are as good as concentrated ones. Long Tail video will reunite disaporas through their common culture, even if they are seen as a niche culture in the world around them. We already see this in the rise of Latino radio in the US; soon that will extend to every other immigrant culture, here and abroad. We often think of the Long Tail as a force of fragmentation, but it can be a force of unification for the already-fragmented, too.

How to say "Long Tail" in Japanese (and Chinese and...)

ロングテール.

Pronounced "Rongu tee-ru."

(Thanks to Kevin Kelleher for the pronunciation correction, and YF Juan for the further correction in the comments)

UPDATE: Other languages:

In Chinese it's:

长尾

Pronounced "Chaang Weei"

(Thanks to Ola Spannar)

In French, there's some debate about whether it should be:

La Longue Traîne

or

La Longue Queue

The first of those has something of the double-entendre semantics of "Tail" in English and is the cheekier option. So that's the one I prefer, of course.

In Brazilian Portuguese it's:

A Longa Cauda,

A Cauda Longa, or

A Cauda Comprida

(Thanks to Josh Bancroft; correction by André U. Manoel in the comments)

In Greek it's:

Μακριά Ουρά (Makria Oura)
Μεγάλη Ουρά (Megali Oura)

(Thanks to Yiorgos Adamopoulos)

In Dutch it's:

De Lange Staart

(Thanks to Tijmen in the comments)

In Romanian it's

Coada Lungă

(Thanks to JS Bangs)

And see the comments for Hindi, Russian, Korean and more....

The Long Tail of idiocy and masturbation

From The Economist this week, an article on the legacy of Larry Summers that includes this novel application of the theory:

"...Mr Summers may have been right. In most intellectual areas, such as vocabulary and verbal reasoning, the differences between men and women are statistically insignificant. But the long tail of mathematical genius does tend to be male, along with higher rates of idiocy and masturbation."

Thanks to Ian Murdock for the spot

Long Tail Meetup in NYC on Aug 24th

The book party last month in NYC was a blast, but we didn't have room for everyone and it was too crowded and noisy to talk much or meet everyone. So I'm going to have a lower-key event next Thursday, August 24th.

It's a Long Tail Meetup at Stone Creek, a lovely bar/restaurant on East 27th Street in Manhattan. We'll be meeting at 7:00 and it's an opportunity to chat with me and each other, get your book signed, buy a book if you want and otherwise hang out and talk about Long Tail stuff.

You can sign up for it here. We're capping it at around 100 people, so sign up soon!

A billion dollar question

This weekend I ran a session at the "SciFoo" camp (an interdisciplinary meeting of scientists and technologists) at Google that was focused on an interesting statistical problem in the Long Tail. I've long argued that the "natural" shape of most markets is a powerlaw, and that any deviation from that shape is due to some bottleneck in distribution. Get rid of the bottleneck and you can tap the latent demand in the market, unlocking the potential of the Long Tail.

The usual example I give is this one, which shows US box office revenues over a three-year period (2003-2005). Remember that a powerlaw looks like a straight line in a log-log plot, so the key part of the below is where the real world data drops off the line (around rank 350).

 

In this case, the explanation for the fall-off is simple. They just ran out of screens. The carrying capacity of the US megaplex theater network is about 100 films per year, or 300 over three years. Over the same period about 13,000 films are shown in film festivals, although only a tiny fraction of them get mainstream commercial distribution. But if you can distribute niche films as easily as the blockbusters, the curve would look like the straight line predicted by the theory, which is, as it happens, exactly what we see with the Netflix data.

However, there is another common distribution that looks a lot like a powerlaw at first, but then deviates from the straight line on its own, even without scarcity effects and other distortions. It's called a lognormal distribution and it looks a little like this:

How can you tell one from the other? This is obviously an important question, since if the theory has any real predicative power, you've got to be able to say ahead of time whether the "natural" shape of the market is a powerlaw or a lognormal. Otherwise you can't tell if the fall-off is due to a removable bottleneck, such as inefficient distribution, or not.

The difference between those two curves is the subject of a lot of research at the cutting edge of complexity theory, and the simple answer seems to be that it comes down to the nature of the network effects that create unequal ("rich get richer") distributions such as the powerlaw and lognormal in the first place. I lay out the basics of that research in this presentation, which I gave on Saturday at Google. (Make sure you're displaying the notes field in that Powerpoint so you can read my narration)

One final note: This doesn't affect any of the conclusions of the book, which are based on real-world data rather than predictions (although I discuss the problem briefly in the notes on page 229). Instead, it's an issue when people attempt to make predictions based on the theory, such as estimating the latent value of a television or film archive based on the assumption that the natural shape of demand would be a powerlaw.  But billions of dollars in valuation depend on getting that calculation right. If you're into this stuff, check out the presentation and see if you can see a path to a solution.

UPDATE: See related discussion: Jakob Neilsen analyzes the observed fall-off in web traffic on one site here. Chris Edwards gives another perspective on that here. And the always astute Nick Carr is following the conversations and adds commentary here.

In the academic world, there are good posts here (""how not to fit a straight line") and here (powerlaws versus lognormals in web tagging). Meanwhile, the two must-read papers in this domain are:

Mitzenmacher, M. (2003). "A brief history of generative models for power law and lognormal distributions". Internet Mathematics 1: 226–251.

Log-normal Distributions across the Sciences: Keys and Clues, E. Limpert, W. Stahel and M. Abbt,. BioScience, 51 (5), p. 341–352 (2001).

Peter Moore on Long Tail gaming

[UPDATE: See Microsoft announcement at bottom]

I call it "hitism" (or "headism"); Peter Moore calls it "elitism". In either case, it's the sin of seeing the world only through the lens of megahits. At a games conference this week, Moore, the former Sega US boss and now Microsoft Xbox chief, talked about how to take the game industry into a world beyond the blockbuster.

Here are Moore's first seven rules "to confront elitism and open the doors of our industry", which could just as easily be called Seven Rules Of Long Tail Game Development:

1. Beyond The Boys In Their Bedrooms.  Appeal to a much wider audience.
2. Don’t Pass The Buck On Rising Development Costs.  We need to find ways to reduce costs and restructure our revenue models.
3. Rebel Without A Platform:  Bring Aspiring Developers Into The Fold. For too long we’ve expected the developers of the future to claw their way up to us … we have to start coming to them and proactively develop a farm team of future stars.
4. It’s A YouTube World:  Embrace Community Created Content. We’re control freaks when it comes to how games are delivered to consumers.  We need more spaces where garage designers can get noticed.
5. Set Us Free.  We should look at delivering new IP in new ways that recognize how powerful a concept shaping your own gaming experience is.
6. We’re Too Cool For School:  Make Ourselves More Approachable.  Games are more powerful, but less approachable.  We need to make more games for more people.  Expand demographics, online gameplay, strong family settings. 
7. Lower The Total Cost Of Ownership With Choice.  Consumer should have choices, starting at entry level, purchases should be upgradable, don’t lock consumers in.

It's worth noting that Microsoft's Xbox Live Arcade platform, which is an on-demand marketplace for small and inexpensive casual games, many of them created by small teams on even smaller budgets, is a great example of the Long Tail of games. In an industry where the cost of developing a traditional console game is now measured in the tens of millions of dollars, a way to reach those same consumers via a online marketplace with infinite shelf space and a place for niche titles of all sort (include re-releases of classic titles from gaming's past) is a perfect way to "scale down" to the market lost to the hits.

UPDATE. Today (8/14) Microsoft announced XNA Game Studios Express, designed to stimulated exactly what Moore was talking about. From the press release:

In the 30 years of video game development, the art of making console games has been reserved for those with big projects, big budgets and the backing of big game labels. Now Microsoft Corp. is bringing this art to the masses with a revolutionary new set of tools, called XNA Game Studio Express, based on the XNA™ platform. XNA Game Studio Express will democratize game development by delivering the necessary tools to hobbyists, students, indie developers and studios alike to help them bring their creative game ideas to life while nurturing game development talent, collaboration and sharing that will benefit the entire industry.

Thanks to Gamerscoreblog for the transcription of Moore's presentation.

Another awesome Long Tail video

My British publisher, Random House, has put together a new site to promote the book in the UK, which includes the above cool video made by Apt Studio in Edinburgh. Those clever Scots have morphed a series of spheres (which symbolize niches) into pie charts, then a powerlaw and then my face. Like this:

It's terrific and I'm totally impressed. The site also has two free sample chapters--the Introduction and Chapter 7, The New Tastemakers--amounting to some 33 pages. Even better, they've released the component graphics and animation files used to make the video under a Creative Commons license so that anyone can use them, misuse them, mess with them or otherwise remix them for any purpose they want. Check it out.

A top ten bestseller!

 

If you happen to follow the bestseller sweepstakes in the New York Times Sunday book review section each week, you'll have a sense of what a very cool day today is for me. The Long Tail, in its second week of eligibility, has broken the top 10 of the NYT non-fiction bestseller list. (last week it was #13). It's #13 on the WSJ list and is the #3 business book.

Many thanks to all of you who bought the book, reviewed it, blogged about it, talked about it and even debated it. I owe you all a great debt of gratitude for making this such a fantastic day, and I hope to partially repay it by continuing the conversation the book began here on the blog, free and open to all.

Part of the reason the book is successful, I believe, is because as I was writing it the smart readers of this blog helped improve the ideas, catch my errors and suggest dozens of applications and dimensions of the Long Tail I never would have thought of myself. So today's recognition is also a recognition of the power of tapping collective intelligence. I couldn't have done it without you!

Andrew Odlyzko responds to Gomes so I don't have to

[UPDATE: the below was, as I mentioned, an excerpt from Andrew's lengthy post. He's now asked me to include the rest of it, too. His full post continues at the bottom.]

Andrew Odlyzko, a mathematician, long-time analyst of digital economics and the director of the University of Minnesota's Digital Technology Center, has written a response to Lee Gomes' latest WSJ column, which says hits still matter. (I know!) 

Andrew kindly sent me a post he just made to an economics mailing list he participates in. Here's an excerpt (I've underlined the data that he quotes from Lee's column):

Lee Gomes' Aug. 2 WSJ column does make a valid point that hits still matter.  But at the same time the evidence it cites actually confirm a quantitative form of the "long tail" hypothesis.  If we assume that popularity of items in an online store follows the ubiquitous Zipf's Law, in which the k-th most popular item is 1/k times as popular as the most popular one, we find that (by approximating 1 + 1/2 + 1/3 + ... + 1/k by log(k)) that the most popular k items out of a total of n items should be bought/viewed/... 

log(k) / log(n) 

fraction of the time. 

Now let's look at the numbers in Lee's column:

(a)  For Amazon, Lee cites estimates that the top 100,000 sellers account for 60% to 80% of all sales, among the "millions of books" that Amazon lists.  Let's assume that those millions of books are 2 million.  Then the rule above suggests that the top 100,000 should account for

log(100000) / log(2000000) =  0.793...

which is at the boundary of the range Lee gave (and the boundary that corresponds to the least contribution of the "long tail").

If those "millions of books" are 6 million, the number we get is 73.7%, still safely inside the 60-80% range.

(b)  Netflix: 50 out of 60,000 titles account for 30% of rentals. The rule above predicts

log(50) / log(60000) = 0.355...

which is even more than what we see (and so the "long tail" is even bigger than might be expected).

(c)  YouTube: Top 10% of 5.1 million videos account for 79% of plays, and top 20% for 89%.  The rule listed predicts

log(510000) / log(5100000) = 0.8509...

and

log(1020000) / log(5100000) = 0.8957...

so that in the first case the "long tail" is again bigger than predicted, while in the second case it is almost exactly on target.

So the conclusion is that yes, Lee is right, one should not go overboard with the "long tail" thesis, and that hits do continue to play a major role, and should be expected to do so in the future.  But at the same time, the long tail is there, and can be expected to play an increasing role, it's just that it will take a while.

Andrew's post continues:

Consider the Amazon example.  With the current 3.7 million titles, the top 100,000 should account (according to the logarithmic ratio rule) for 76% of sales.  But how much larger can the 3.7 million figure grow?  Books are not easy to write, and so even if every would-be author who manages to write a complete manuscript gets "published" in some form, we are unlikely to increase the total number of books by more than a factor of 10, say.  So suppose that Amazon goes to 37 million books from 3.7 million.

Then the quantitative rule would suggest that the top 100,000 titles would account for 66% of the sales.  That is a noticeable drop from the 76% today, but hardly earth-shattering.

On the other hand, the difference can be substantial in other settings.

For example, if historical patterns repeat, then home-made videos will become key to the growth in penetration of broadband.  And with improved cameras, editing tools, and high-speed connectivity, it is easy to imagine billions of videos available on the Net.  Let's assume we end up with a relatively modest figure of 6 billion videos (we already have over 5 million on YouTube).

Then the top 50 titles on Netflix might drop from the 35% predicted by the rule for today to 17%, and the entire current inventory of 60,000 titles might account for just

log(60000) / log(6000000000) = 0.488...

or 49% of the total.  That would be a major change.

The quantitative version of the "long tail" hypothesis is developed in my paper with Ben Tilly, "A refutation of Metcalfe's Law and a better estimate for the value of networks and network interconnections" (which also gives references for Zipf's Law and related issues), and in a shorter form in the paper with Bob Briscoe and Ben Tilly, "Metcalfe's Law is wrong" which appeared in the July 2006 issue of IEEE Spectrum,

It can also be used to provide a quantitative justification for the observation that connectivity has traditionally been valued more highly than content, as was shown in my Feb. 2001 paper "Content is not king".

Basically the huge mass of trivial communications (such as your making a dinner reservation), mostly of very little importance to anyone beyond the two people involved, and so at the extreme tail of the long tail, outweighs the blockbusters.  (Ordinary voice telephony in the US, wired and wireless, still produces well over $300 billion a year in revenues, while Hollywood brings in something like $80 billion, and much of that from overseas.)

How a Canadian folk song became a Japanese ringtone

I was on a local NPR affliate show yesterday, and the host ended our segment with a lovely story that I wanted to pass along. She's Mary Ambrose of KALW, and this is how she summarized the Long Tail, while some charming music played in the background:

I have two friends in Canada who run a Canadian folk music label called Borealis. This is music from their album Six Strings North of the Border. While folk music fans are loyal, there simply aren't enough of them in Canada to support this label, even though Borealis has a solid back catalog, which never goes out of style.

Then the Internet came along and they were able to find folk music fans who didn't know the lyricism of Penny Lang or the songwriting of Ron Hynes. There's little room on the retail shelves for any niche music, and most music--including folk--is niche music. The new reality of interconnectedness means Borealis is thriving with new audiences they never could have found before. Recently one of my friends told me that they just received a check for ringtone downloads in Japan, where the hip thing is to have acoustic music on your cell phone. That is the Long Tail.

Exactly. Thanks, Mary, for saying it so well.